CN115904369A - Method and system for efficient aggregation and correlation analysis of network security source data - Google Patents
Method and system for efficient aggregation and correlation analysis of network security source data Download PDFInfo
- Publication number
- CN115904369A CN115904369A CN202211362682.1A CN202211362682A CN115904369A CN 115904369 A CN115904369 A CN 115904369A CN 202211362682 A CN202211362682 A CN 202211362682A CN 115904369 A CN115904369 A CN 115904369A
- Authority
- CN
- China
- Prior art keywords
- data
- aggregation
- time
- strategy
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a system for efficient aggregation and correlation analysis of network security source data, which comprises the steps of accessing a data source by using a real-time queue and storing results, configuring and storing an aggregation strategy by using a storage medium, aggregating the data based on the source data and the aggregation strategy and then storing the aggregated data, starting an asynchronous thread to read XML file attributes at regular time by adopting a main program of the method by using an observer mode, reading file contents if the last modification time of the file is changed, and informing an observer of the latest aggregation strategy to update a set of aggregation strategies in a memory of the observer. Compared with the traditional method which takes the server time as the time reference, the method takes the data time as the time reference, and aggregates and sends the data with the same aggregation characteristics to the downstream in a non-delay way without waiting for the step length of a fixed time window, thereby ensuring the real-time property of data processing and the data truth.
Description
Technical Field
The invention relates to the technical field of industrial network security, in particular to a method and a system for high-efficiency aggregation and correlation analysis of network security source data.
Background
The invention relates to the field of industrial network security, in particular to a correlation analysis early warning method and a data high-efficiency aggregation method based on network flow source data and log source data.
With the rapid development of industrial internet in China and frequent occurrence of network security accidents internationally, the construction of network security monitoring systems in enterprises becomes more and more important. The source data rule early warning module in the network security system occupies an extremely important position, but due to the large scale of an enterprise network and the numerous network devices, the realization of the module faces the problems of large quantity of source data, complex flow characteristics, numerous early warning rules and the like, so that the early warning module has the defects of instantaneity, high efficiency and accuracy. In order to solve the problems, a high-efficiency aggregation method is adopted to carry out homofeature sampling pretreatment on mass data so as to solve the delay problem of data backlog, and a source data correlation analysis method is adopted to achieve the purposes of simple and rapid online early warning rule and high accuracy of early warning events.
In industrial networks, there are security devices such as firewalls, IDS, forward isolation devices, longitudinal encryption devices, etc.; network devices such as switches, routers, etc.; host equipment such as an operator station, an engineer station, an interface machine and the like needs to analyze a port mirror image of a convergence switch besides log monitoring on the equipment, so that the data volume is huge, and under the condition of considering hardware cost, the processing capacity of a single equipment product is very poor when regular early warning is carried out, so that real-time preprocessing aggregation needs to be carried out on source data. The method has a large influence on the real-time performance of the data, and the aggregation operation can be performed only after the time window is triggered, so that the data can reach a downstream processing module with a certain delay, and the delay influence is more obvious along with the increase of the time window, and the occurrence time of the data is shielded by taking the server time as the reference, thereby causing data shortage.
Abundant equipment types lead to multi-type log formats, and complex flow characteristics require that early warning rules are more flexible and accurate. The current main early warning method fixes a data source data structure in a code form, and realizes early warning judgment on a source data set basically through a simple black and white list, confidence, threshold configuration and the like. The data structure cannot be flexibly configured, the new data source and the new online rule need to be recoded and restarted for application, the method is very inefficient, the original early warning method cannot be used for complex rule configuration, when the rule is matched, an alarm is output, the capability of performing evidence adjustment on the alarm of source data for correlation analysis is not provided, and false alarm is easy to generate.
The current method for aggregating data is mainly realized based on server time waiting for a fixed time interval, and the main realization process is as follows:
step one, opening a time window task according to an aggregation strategy;
step two, receiving data and storing the data into a corresponding window cache;
step three, judging whether a time window triggering execution condition is met, if not, waiting, repeating the step two, and if so, executing the step four;
and step four, executing aggregation logic.
The implementation of the early warning rule is mainly implemented by a recoding method and an SQL statement making method at present.
The SQL statement making method is realized by compiling SQL statements corresponding to early warning requirements through SQL-CLI according to a query language mode given by the early warning module.
The recoding method mainly comprises the following implementation processes:
step one, receiving a requirement of an early warning rule;
secondly, encoding a source data structure and an early warning rule;
compiling the codes, uploading the codes to a server, and restarting the application;
step four, receiving source data to perform rule processing, and directly sending an alarm to downstream when the rule is satisfied;
and step five, if a new online requirement exists or an existing requirement is modified, repeating the step one to the step four.
The aggregation method has the problems of large data delay, data distortion and the like; the early warning rule method has the problems of needing to have certain SQL knowledge, low rule making efficiency, low accuracy rate and the like, and cannot meet the operation scene with large data volume, high real-time requirement, various rules, needing to be made quickly and high warning accuracy rate.
In order to solve the problems, the invention provides an efficient polymerization method and a correlation analysis early warning method. The invention realizes dynamic loading of aggregation rules by configuring any data storage mode (XML configuration, MYSQL, REDIS and the like), performs time window trigger calculation based on the time of the data and sends the data with the same aggregation characteristics to the downstream without delay, thereby solving the problems of data delay processing, data distortion and the like; for the rule early warning of source data, the invention can formulate (including a source data structure and an early warning rule) an association analysis strategy through a UI interface, all formulations can be dynamically loaded without encoding and restarting, and complex association, time sequence analysis and association evidence of multiple data sources are supported, so that the problems of low efficiency, single early warning rule formulation, high false alarm rate and the like of online new rules and modification of the existing rules are solved.
Disclosure of Invention
The invention provides a method and a system for efficient aggregation and association analysis of network security source data, which aim to overcome the defects of the prior art.
In one aspect, the present invention provides a method for efficient aggregation and association analysis of network security source data, which includes the following steps:
s1: selecting a storage medium to configure an aggregation strategy of network security source data, selecting a message queue to access the network security source data and store a result, starting a file monitoring asynchronous thread to read file attributes of the storage medium at regular time, and starting an observer asynchronous thread to wait for a notification containing the aggregation strategy;
s2: if the last modification time of the storage medium is read to be changed, reading the content of the storage medium, informing an observer of the latest strategy in the storage medium, and then continuously monitoring the modification time of the storage medium, wherein the observer analyzes the latest strategy and updates the configured aggregation strategy according to the latest strategy;
s3: reading a field corresponding value in received source data, merging the field corresponding value into a key field key, comparing the key field key with a key of an existing aggregation strategy, matching the corresponding aggregation strategy, combining a timestamp carried by the source data and start time and end time of the aggregation strategy, obtaining a window object from a time window manager, if the window does not exist, establishing and maintaining the window object to the window manager, and obtaining an execution handle of the window based on the obtained window object;
s4: calculating an aggregation characteristic according to the received characteristic of the key field of the data source, triggering the time window based on the execution handle, and executing the following aggregation processes based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and sending the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
The method comprises the steps of accessing a data source by using a real-time queue, storing results, configuring and storing an aggregation strategy by using a storage medium, aggregating and storing data based on source data and the aggregation strategy, starting an asynchronous thread to read XML file attributes at regular time by adopting an observer mode as a main program of the method, reading file contents if the last modification time of the file is changed, and informing an observer of a latest aggregation strategy to update a set of the aggregation strategies in a memory of the observer. Compared with the traditional method which takes the server time as the time reference, the method takes the data time as the time reference, and aggregates and sends the data with the same aggregation characteristics to the downstream in a non-delay way without waiting for the fixed time window step length, thereby ensuring the real-time property of data processing and the data truth.
In a particular embodiment, the storage medium is a storage component that includes an XML file and Mysql.
In a particular embodiment, the message queue includes Kafka and RabbitMQ.
In a specific embodiment, the S3 specifically includes:
reading received source data and matching a corresponding aggregation strategy, extracting a timestamp of the received source data, calculating and obtaining the starting time and the ending time of a time window to which the received source data belongs according to aggregation time characteristics in the timestamp, executing a manager handle of the time window, and inquiring whether the time window exists or not;
if yes, directly acquiring all execution handles of the time window;
and if not, creating a new time window based on the starting time and the ending time, saving the new time window into a time window manager, and acquiring all execution handles of the new time window.
In a specific embodiment, the S4 specifically includes:
calculating according to the corresponding aggregation strategy and based on the characteristics of the key fields to obtain the aggregation characteristics of the received data source, and operating all the execution handles obtained in the step S3 so as to execute the corresponding aggregation strategy to judge whether the data with the same aggregation characteristics exist in the window storage body or not;
if the data source exists, the received data source is the aggregated data, the received data source is stored in a window storage body, and the received data source is sent to a downstream auxiliary data processing body;
and if the data source does not exist, sending the received data source to a downstream main data processing body.
In a specific embodiment, the method further includes performing association analysis on the network security source data, which specifically includes:
a1: performing dynamic injection of a source data structure through a user interface, selecting a certain number of data sources in the input source data, and selecting a plurality of key fields from the selected data sources according to association requirements to generate an association strategy;
a2: the association strategy is used for reading asynchronous data when the system is started so as to be matched with the aggregation strategy, and if the association strategy is successfully matched with the aggregation strategy, a pre-warning event is generated;
a3: and automatically associating original logs in a query message queue according to the labels of the key fields in the pre-alarm event, performing evidence on the pre-alarm event, and if the evidence result conforms to the alarm characteristics, generating a quasi-alarm event and sending the quasi-alarm event to the downstream.
And the correlation analysis provides a user interface for realizing data source data structure hot loading and drawing a rule graph hot deployment early warning strategy. The strategy can be formulated by selecting one or more data sources, performing parallel association analysis or time sequence association analysis according to conditions, and automatically performing association evidence on the early warning information when the source data is successfully matched with the rule, so that the online efficiency and the warning accuracy of the rule are ensured, and the method has the multi-scene applicability of rule writing.
In a specific embodiment, the user interface is built based on a Java language design UI user interaction interface.
In a specific embodiment, the A1 specifically includes:
maintaining information including a data source and a data structure in the data source through the user interface, storing the information in a data source table, selecting a plurality of key fields from the selected data source according to association requirements, constructing an association diagram according to the selected key fields, and storing the association diagram in an association analysis model table.
In a specific embodiment, the A2 specifically includes:
and the monitoring thread scans the data source table and the association analysis model table at regular time, if the data source table and the association analysis model table are added or modified, a new asynchronous thread is automatically started, an association analysis task is executed or an existing task thread is restarted, and the change of the data source and the association strategy is updated to the execution task in real time.
In a particular embodiment, the performing of the correlation analysis includes:
reading the correlation analysis model from the correlation analysis model table, converting the correlation analysis model into an executable logic diagram by using a rule compiler, converting the executable logic diagram into an execution task diagram by using a rule executor, and acquiring execution physical resources for starting related threads so as to start a correlation analysis task.
According to a second aspect of the present invention, a computer-readable storage medium is proposed, on which a computer program is stored, which computer program, when being executed by a computer processor, carries out the above-mentioned method.
According to a third aspect of the present invention, a system for efficient aggregation and association analysis of network security source data is provided, the system comprising:
an aggregation policy configuration module: configuring an aggregation strategy for selecting a storage medium to configure network security source data, selecting a message queue to access and store results of the network security source data, starting a file monitoring asynchronous thread to read file attributes of the storage medium at regular time, and starting an observer asynchronous thread to wait for a notification containing the aggregation strategy;
an aggregation policy update module: the configuration is used for reading the content of the storage medium if the last modification time read to the storage medium is changed, informing an observer of the latest strategy in the storage medium, and the observer analyzes the latest strategy and updates the configured aggregation strategy according to the latest strategy;
an aggregation policy matching module: configuring an aggregation strategy used for reading received source data and matching the received source data, and acquiring an execution handle of a time window based on a timestamp of a received data source;
an aggregation policy enforcement module: the system is configured to calculate an aggregation feature according to the feature of the received key field of the data source, trigger the time window based on the execution handle, and execute the following aggregation processes based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and transmitting the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
The invention uses real-time queue to access data source and store result, uses storage medium to configure and store aggregation strategy, and stores the aggregated strategy after aggregating data based on source data and aggregation strategy, the main program of the method adopts observer mode, starts asynchronous thread to read XML file attribute at regular time, reads file content if the file is modified at the last time, and informs observer of the latest aggregation strategy to update aggregation strategy in the memory. Compared with the traditional method which takes the server time as the time reference, the method takes the data time as the time reference, and aggregates and sends the data with the same aggregation characteristics to the downstream in a non-delay way without waiting for the fixed time window step length, thereby ensuring the real-time property of data processing and the data truth. In addition, the correlation analysis method provides a user interface for realizing data source data structure hot loading and drawing a rule graph hot deployment early warning strategy. The strategy can be formulated by selecting one or more data sources, performing parallel association analysis or time sequence association analysis according to conditions, and automatically performing association evidence on the early warning information when the source data is successfully matched with the rule, so that the online efficiency and the warning accuracy of the rule are ensured, and the method has the multi-scene applicability of rule writing.
Drawings
The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and together with the description serve to explain the principles of the invention. Other embodiments and many of the intended advantages of embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow diagram of a method for efficient aggregation and correlation analysis of cyber-security source data according to one embodiment of the invention;
FIG. 3 is a flow diagram of an aggregation method hot-load aggregation policy of a particular embodiment of the present invention;
FIG. 4 is a flowchart of aggregate policy enforcement for a specific embodiment of the present invention;
FIG. 5 is a flow diagram of a method for association analysis according to a specific embodiment of the present invention;
FIG. 6 is a diagram of an overall implementation architecture of the polymerization process of a specific embodiment of the present invention;
FIG. 7 is a diagram of an association analysis overall implementation architecture for a specific embodiment of the present invention;
FIG. 8 is a block diagram of a system for efficient aggregation and correlation analysis of cyber-secure source data, in accordance with an embodiment of the present invention;
FIG. 9 is a block diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 illustrates an exemplary system architecture 100 to which a method for efficient aggregation and association analysis of network security source data according to an embodiment of the present application may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various applications, such as a data processing application, a data visualization application, a web browser application, etc., may be installed on the terminal devices 101, 102, 103.
The terminal apparatuses 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices including, but not limited to, smart phones, tablet computers, laptop portable computers, desktop computers, and the like. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The server 105 may be a server that provides various services, such as a background information processing server that provides support for source data presented on the terminal devices 101, 102, 103. The backend information processing server may process the obtained aggregation policy and generate a processing result (e.g., an aggregation feature).
It should be noted that the method provided in the embodiment of the present application may be executed by the server 105, or may be executed by the terminal devices 101, 102, and 103, and the corresponding apparatus is generally disposed in the server 105, or may be disposed in the terminal devices 101, 102, and 103.
The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. And is not particularly limited herein.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 shows a flowchart of a method for efficient aggregation and association analysis of network security source data according to an embodiment of the present invention. As shown in fig. 2, the method comprises the steps of:
s1: selecting a storage medium to configure an aggregation strategy of network security source data, selecting a message queue to access the network security source data and store results, starting a file monitoring asynchronous thread to read file attributes of the storage medium at regular time, and starting an observer asynchronous thread to wait for a notification containing the aggregation strategy.
In a particular embodiment, the storage medium is a storage component that includes an XML file and Mysql.
In a particular embodiment, the message queue includes Kafka and RabbitMQ.
S2: and if the last modification time of the storage medium is read to be changed, reading the content of the storage medium, informing an observer of the latest strategy in the storage medium, continuously monitoring the modification time of the storage medium, and analyzing the latest strategy by the observer and updating the configured aggregation strategy according to the latest strategy.
S3: reading a field corresponding value in received source data, merging the field corresponding value into a key field key, comparing the key field key with a key of an existing aggregation strategy, matching the corresponding aggregation strategy, combining a timestamp carried by the source data and the start time and the end time of the aggregation strategy, obtaining a window object from a time window manager, if the window does not exist, establishing and maintaining the window manager, and obtaining an execution handle of the window based on the obtained window object. For example, according to field combinations selected on the UI interface, such as f1, f2, and the like, the specific values corresponding to the fields in the source data, such as f1, f2, and the like, are read and combined into the key field key.
In a specific embodiment, the S3 specifically includes:
reading received source data and matching a corresponding aggregation strategy, extracting a timestamp of the received source data, calculating and obtaining the starting time and the ending time of a time window to which the received source data belongs according to aggregation time characteristics in the timestamp, executing a manager handle of the time window, and inquiring whether the time window exists or not;
if yes, directly acquiring all execution handles of the time window;
and if not, creating a new time window based on the starting time and the ending time, saving the new time window into a time window manager, and acquiring all execution handles of the new time window.
S4: calculating an aggregation characteristic according to the received characteristic of the key field of the data source, triggering the time window based on the execution handle, and executing the following aggregation processes based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and transmitting the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
In a specific embodiment, the S4 specifically includes:
calculating according to the corresponding aggregation strategy and based on the characteristics of the key fields to obtain the aggregation characteristics of the received data source, and operating all the execution handles obtained in the step S3 so as to execute the corresponding aggregation strategy to judge whether the data with the same aggregation characteristics exist in the window storage body or not;
if the data source exists, the received data source is the aggregated data, the received data source is stored in a window storage body, and the received data source is sent to a downstream auxiliary data processing body;
and if the data source does not exist, sending the received data source to a downstream main data processing body.
In a specific embodiment, the method further includes performing association analysis on the network security source data, specifically including:
a1: performing dynamic injection of a source data structure through a user interface, selecting a certain number of data sources in the input source data, and selecting a plurality of key fields from the selected data sources according to association requirements to generate an association strategy;
a2: the association strategy is used for reading asynchronous data when the system is started so as to be matched with the aggregation strategy, and if the association strategy is successfully matched with the aggregation strategy, a pre-warning event is generated;
a3: and automatically associating original logs in a query message queue according to the labels of the key fields in the pre-alarm event, performing evidence on the pre-alarm event, and if the evidence result conforms to the alarm characteristics, generating a quasi-alarm event and sending the quasi-alarm event to the downstream.
In a specific embodiment, the user interface is built based on a Java language design UI user interaction interface.
In a specific embodiment, the A1 specifically includes:
maintaining information including a data source and a data structure in the data source through the user interface, storing the information in a data source table, selecting a plurality of key fields from the selected data source according to association requirements, constructing an association diagram according to the selected key fields, and storing the association diagram in an association analysis model table.
In a specific embodiment, the A2 specifically includes:
and the monitoring thread scans the data source table and the association analysis model table at regular time, if the data source table and the association analysis model table are newly added or modified, a new asynchronous thread is automatically started, an association analysis task is executed or an existing task thread is restarted, and the change of the data source and the association strategy is updated into the execution task in real time.
In a particular embodiment, the performing of the correlation analysis includes:
reading the correlation analysis model from the correlation analysis model table, converting the correlation analysis model into an executable logic diagram by using a rule compiler, converting the executable logic diagram into an execution task diagram by using a rule executor, and acquiring execution physical resources for starting related threads so as to start a correlation analysis task.
Fig. 3 shows a flowchart of an aggregation method hot-loading aggregation policy of a specific embodiment of the present invention, where configuration and update of an aggregation policy are implemented in this embodiment as shown in the following: the aggregation policy may be configured via any storage medium, most conveniently via an XML file. The main program of the method adopts an observer mode, starts an asynchronous thread to read XML file attributes at regular time, reads file contents if the last modification time of the file is changed, informs an observer of the latest strategy, and changes the strategy set in the memory of the observer.
Fig. 4 shows an aggregation policy execution flowchart of a specific embodiment of the present invention, as shown in the drawing, the implementation of the aggregation policy in this embodiment specifically includes matching and executing the aggregation policy, and as shown in fig. 4, the steps are as follows:
reading source data, and inquiring a corresponding aggregation strategy according to the source data; extracting a timestamp (st) of received data, calculating to obtain the starting time (bt) and the ending time (et) of a time window to which the data belongs according to the aggregation time characteristics, executing a time window manager handle, inquiring whether the attribute window exists or not, directly acquiring all the execution handles of the window if the attribute window exists, newly building the time window if the attribute window does not exist, storing the time window with the starting time being bt and the ending event being et into the time window manager, and obtaining all the execution handles; and extracting each field value with the aggregated field characteristics to combine the field values into an aggregated characteristic, executing a query handle of a time window, judging whether the value with the same aggregated characteristic exists or not, if not, sending the data to a processing body of the downstream main data, and if so, sending the aggregated data to a downstream secondary data.
Fig. 5 shows a flow chart of a correlation analysis method according to a specific embodiment of the present invention, where the flow of the correlation analysis method in this embodiment is as follows:
supporting interaction between a UI interface and a user, maintaining a data source through the UI interface, wherein the data source comprises a data source and a data structure, and storing the data source into a data source table;
selecting one or more data sources required by the association requirement through an interface, selecting some fields according to the association strategy, and constructing an association diagram and storing the association diagram in an association analysis model table;
the monitoring thread scans the data source table and the association analysis model table at regular time, if the data source table and the association analysis model table are added or modified, a new asynchronous thread is automatically started, an association analysis task is executed or an existing task thread is restarted, the change of the data source or the association strategy is ensured, and the data source or the association strategy is updated to the execution task in real time;
and after the correlation analysis method is read from the model table, converting the model into an executable logic diagram by using a rule compiler, converting the logic execution diagram into an execution task diagram by using a rule executor, acquiring execution physical resources, starting a related thread, and starting a correlation analysis task. And the task execution reads a configuration data source, converts the configuration data source into a POJO type data stream, performs rule judgment according to a drawn association strategy, automatically retrieves an original log within a period of time according to a key field label if a pre-alarm event is generated, judges whether the pre-alarm event accords with the alarm characteristic, and generates an alarm event and sends the alarm event to the downstream if the pre-alarm event accords with the alarm characteristic.
Fig. 6 shows an architecture diagram for overall implementation of the aggregation method according to a specific embodiment of the present invention, where the aggregation method may perform dynamic configuration aggregation policy based on other storage components such as XML file configuration, and improve a conventional manner that server time is used as a time reference, data time is used as a time reference, and data with the same characteristics are aggregated and sent to downstream in a delay-free manner, and there is no need to wait for a fixed window step size, so as to ensure real-time performance and data fidelity of data processing.
Fig. 7 shows an overall implementation architecture diagram of association analysis according to a specific embodiment of the present invention, where the association analysis provides a UI interface for implementing hot loading of a data source data structure and drawing a rule diagram hot deployment early warning policy. The strategy can be formulated by selecting one or more data sources, performing parallel association analysis or time sequence association analysis according to conditions, and automatically performing association evidence on the early warning information when the source data is successfully matched with the rule, so that the online efficiency and the warning accuracy of the rule are ensured, and the method has the multi-scene applicability of rule writing.
The invention has the following advantages:
1) The method provided by the invention can be deployed in different complex industrial networks, and has strong adaptability;
2) The method adopts the central manager to execute tasks, can realize multi-host joint deployment, and has strong expansibility;
3) The aggregation method ensures the real-time performance and the reality degree of the source data, the association method provides user interaction, the operation is simple and convenient, and the rule definition is flexible and rich.
FIG. 8 is a block diagram of a system for efficient aggregation and association analysis of cyber-security source data according to an embodiment of the present invention. The system includes an aggregation policy configuration module 801, an aggregation policy update module 802, an aggregation policy matching module 803, and an aggregation policy enforcement module 804.
In a specific embodiment, the aggregation policy configuration module 801 is configured to select a storage medium to configure an aggregation policy for network security source data, select a message queue to access and store a result for the network security source data, start a file monitoring asynchronous thread to read a file attribute of the storage medium at regular time, and start an observer asynchronous thread to wait for a notification including the aggregation policy;
the aggregation policy update module 802 is configured to read the content of the storage medium if the last modification time read for the storage medium is changed, notify the latest policy in the storage medium to an observer, and then continue to monitor the storage medium modification time, where the observer analyzes the latest policy and updates the configured aggregation policy according to the latest policy;
the aggregation policy matching module 803 is configured to read a field corresponding value in the received source data, merge the field corresponding value into a key field key, compare the key field key with a key of an existing aggregation policy, match the aggregation policy corresponding to the key field key, acquire a window object from the time window manager in combination with a timestamp carried by the source data and start time and end time of the aggregation policy, establish and maintain the window object to the window manager if the window does not exist, and acquire an execution handle for the window based on the acquired window object;
the aggregation policy execution module 804 is configured to calculate an aggregation feature according to a feature of the received key field of the data source, trigger the time window based on the execution handle, and execute the following aggregation procedures based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and transmitting the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
The system accesses a data source by using a real-time queue and stores results, configures and stores an aggregation strategy by using a storage medium, aggregates and stores the data based on source data and the aggregation strategy, adopts an observer mode for a main program of the method, starts an asynchronous thread to read XML file attributes at regular time, reads file contents if the file is modified at the last modification time, and informs an observer of the latest aggregation strategy to update a set of the aggregation strategies in a memory of the observer. Compared with the traditional method which takes the server time as the time reference, the method takes the data time as the time reference, and aggregates and sends the data with the same aggregation characteristics to the downstream in a non-delay way without waiting for the step length of a fixed time window, thereby ensuring the real-time property of data processing and the data truth.
Referring now to FIG. 9, shown is a block diagram of a computer system 900 suitable for use in implementing the electronic device of an embodiment of the present application. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 9, the computer system 900 includes a Central Processing Unit (CPU) 901 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.
The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including a display such as a Liquid Crystal Display (LCD) and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary so that a computer program read out therefrom is mounted into the storage section 908 as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program performs the above-described functions defined in the method of the present application when executed by a Central Processing Unit (CPU) 901. It should be noted that the computer readable storage medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present application may be implemented by software or hardware. The units described may also be provided in a processor, and the names of the units do not in some cases constitute a limitation of the unit itself.
Embodiments of the invention also relate to a computer-readable storage medium having stored thereon a computer program which, when executed by a computer processor, implements the method above. The computer program comprises program code for performing the method illustrated in the flow chart. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable medium or any combination of the two.
The invention uses real-time queue to access data source and store result, uses storage medium to configure and store aggregation strategy, and stores the aggregated strategy after aggregating data based on source data and aggregation strategy, the main program of the method adopts observer mode, starts asynchronous thread to read XML file attribute at regular time, reads file content if the file is modified at the last time, and informs observer of the latest aggregation strategy to update aggregation strategy in the memory. Compared with the traditional method which takes the server time as the time reference, the method takes the data time as the time reference, and aggregates and sends the data with the same aggregation characteristics to the downstream in a non-delay way without waiting for the step length of a fixed time window, thereby ensuring the real-time property of data processing and the data truth. In addition, the correlation analysis method provides a user interface for realizing hot loading of the data source data structure and drawing the rule graph hot deployment early warning strategy. The strategy can be formulated by selecting one or more data sources, performing parallel association analysis or time sequence association analysis according to conditions, and automatically performing association evidence on the early warning information when the source data is successfully matched with the rule, so that the online efficiency and the warning accuracy of the rule are ensured, and the method has the multi-scene applicability of rule writing.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (12)
1. A method for efficiently aggregating network security source data is characterized by comprising the following steps:
s1: selecting a storage medium to configure an aggregation strategy of network security source data, selecting a message queue to access the network security source data and store results, starting a file monitoring asynchronous thread to read file attributes of the storage medium at regular time, and starting an observer asynchronous thread to wait for a notification containing the aggregation strategy;
s2: if the last modification time of the storage medium is read to be changed, reading the content of the storage medium, informing an observer of the latest strategy in the storage medium, and then continuously monitoring the modification time of the storage medium, wherein the observer analyzes the latest strategy and updates the configured aggregation strategy according to the latest strategy;
s3: reading a field corresponding value in received source data, merging the field corresponding value into a key field key, comparing the key field key with a key of an existing aggregation strategy, matching the corresponding aggregation strategy, combining a timestamp carried by the source data and start time and end time of the aggregation strategy, obtaining a window object from a time window manager, if the window does not exist, establishing and maintaining the window object to the window manager, and obtaining an execution handle of the window based on the obtained window object;
s4: calculating an aggregation characteristic according to the received characteristic of the key field of the data source, triggering the time window based on the execution handle, and executing the following aggregation processes based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and transmitting the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
2. The method of claim 1, wherein the storage medium is a storage component comprising an XML file and Mysq.
3. The method of claim 1, wherein the message queue comprises Kafka and RabbitMQ.
4. The method according to claim 1, wherein S3 specifically comprises:
reading received source data and matching a corresponding aggregation strategy, extracting a timestamp of the received source data, calculating and obtaining the starting time and the ending time of a time window to which the received source data belongs according to aggregation time characteristics in the timestamp, executing a manager handle of the time window, and inquiring whether the time window exists or not;
if yes, directly acquiring all execution handles of the time window;
and if the new time window does not exist, creating a new time window based on the starting time and the ending time, saving the new time window into a time window manager, and acquiring all execution handles of the new time window.
5. The method according to claim 1, wherein the S4 specifically comprises:
calculating according to the corresponding aggregation strategy and based on the characteristics of the key fields to obtain the aggregation characteristics of the received data source, and operating all the execution handles obtained in the step S3 so as to execute the corresponding aggregation strategy to judge whether the data with the same aggregation characteristics exist in the window storage body or not;
if the data source exists, the received data source is the aggregated data, the received data source is stored in a window storage body, and the received data source is sent to a downstream auxiliary data processing body;
and if the data source does not exist, the received data source is sent to a downstream main data processing body.
6. A method for analyzing association of network security source data, after the method for efficiently aggregating according to any one of claims 1 to 5, further comprising performing association analysis on the network security source data, specifically comprising:
a1: performing dynamic injection of a source data structure through a user interface, selecting a certain number of data sources in the input source data, and selecting a plurality of key fields from the selected data sources according to association requirements to generate an association strategy;
a2: the association strategy is used for reading asynchronous data when the system is started, so that the association strategy is matched with the aggregation strategy, and if the association strategy is successfully matched with the aggregation strategy, a pre-warning event is generated;
a3: and automatically associating original logs in a query message queue according to the labels of the key fields in the pre-alarm event, performing evidence on the pre-alarm event, and if the evidence result conforms to the alarm characteristics, generating a quasi-alarm event and sending the quasi-alarm event to the downstream.
7. The method according to claim 6, wherein the user interface is constructed based on a Java language design UI user interaction interface.
8. The method according to claim 6, wherein A1 specifically comprises:
and maintaining information including the source and the data structure of the data source in the data source through the user interface, storing the information in a data source table, selecting a plurality of key fields from the selected data source according to the association requirements, constructing an association diagram according to the selected key fields, and storing the association diagram in an association analysis model table.
9. The method according to claim 8, wherein A2 specifically comprises:
and the monitoring thread scans the data source table and the association analysis model table at regular time, if the data source table and the association analysis model table are added or modified, a new asynchronous thread is automatically started, an association analysis task is executed or an existing task thread is restarted, and the change of the data source and the association strategy is updated to the execution task in real time.
10. The method of claim 6, wherein performing the correlation analysis comprises:
reading the correlation analysis model from the correlation analysis model table, converting the correlation analysis model into an executable logic diagram by using a rule compiler, converting the executable logic diagram into an execution task diagram by using a rule executor, and acquiring execution physical resources for starting related threads so as to start a correlation analysis task.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a computer processor, carries out the method of any one of claims 1 to 10.
12. A system for efficient aggregation of network security source data, comprising:
an aggregation policy configuration module: configuring an aggregation strategy for selecting a storage medium to configure network security source data, selecting a message queue to access the network security source data and store results, starting a file monitoring asynchronous thread to read file attributes of the storage medium at regular time, and starting an observer asynchronous thread to wait for a notification containing the aggregation strategy;
an aggregation policy update module: the configuration is used for reading the content of the storage medium if the last modification time of the storage medium is read and changed, informing an observer of the latest strategy in the storage medium, and then continuously monitoring the modification time of the storage medium, wherein the observer analyzes the latest strategy and updates the configured aggregation strategy according to the latest strategy;
an aggregation policy matching module: configuring a key field key used for reading and merging a field corresponding value in received source data, comparing the key field key with a key of an existing aggregation strategy, matching the corresponding aggregation strategy, obtaining a window object from a time window manager by combining a timestamp carried by the source data and start time and end time of the aggregation strategy, if the window does not exist, newly building and maintaining the window manager, and obtaining an execution handle of the window based on the obtained window object;
an aggregation policy enforcement module: the configuration is used for calculating aggregation characteristics according to the characteristics of the received key fields of the data source, triggering the time window based on the execution handle, and executing the following aggregation processes based on the time window: and sequentially judging whether the aggregation characteristics exist in a window storage body or not according to the time window, and if so, executing the corresponding aggregation strategy by the received data source and sending the corresponding aggregation strategy to a downstream auxiliary data processing body in real time.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211362682.1A CN115904369B (en) | 2022-11-02 | 2022-11-02 | Method and system for efficiently aggregating and associated analysis of network security source data |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211362682.1A CN115904369B (en) | 2022-11-02 | 2022-11-02 | Method and system for efficiently aggregating and associated analysis of network security source data |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115904369A true CN115904369A (en) | 2023-04-04 |
CN115904369B CN115904369B (en) | 2023-10-13 |
Family
ID=86471785
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211362682.1A Active CN115904369B (en) | 2022-11-02 | 2022-11-02 | Method and system for efficiently aggregating and associated analysis of network security source data |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115904369B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116756197A (en) * | 2023-08-23 | 2023-09-15 | 中国电信股份有限公司 | Method, system and communication equipment for realizing dynamic window and aggregation parameters |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100146522A1 (en) * | 2008-12-09 | 2010-06-10 | Microsoft Corporation | Handling exceptions in a data parallel system |
US20160036903A1 (en) * | 2014-07-31 | 2016-02-04 | Splunk Inc. | Asynchronous processing of messages from multiple servers |
CN107729559A (en) * | 2017-11-08 | 2018-02-23 | 携程旅游网络技术(上海)有限公司 | Method, system, equipment and the storage medium of data base read-write asynchronous access |
CN110597891A (en) * | 2018-06-12 | 2019-12-20 | 武汉斗鱼网络科技有限公司 | Device, system, method and storage medium for aggregating MySQL into PostgreSQL database |
CN110612716A (en) * | 2017-01-20 | 2019-12-24 | 十位数通信有限责任公司 | Intermediate device for network routing of data messages |
CN110806958A (en) * | 2019-10-24 | 2020-02-18 | 长城计算机软件与***有限公司 | Monitoring method, monitoring device, storage medium and electronic equipment |
CN113179267A (en) * | 2021-04-27 | 2021-07-27 | 长扬科技(北京)有限公司 | Network security event correlation analysis method and system |
CN115221116A (en) * | 2022-07-25 | 2022-10-21 | 深圳市网心科技有限公司 | Data writing method, device and equipment and readable storage medium |
-
2022
- 2022-11-02 CN CN202211362682.1A patent/CN115904369B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100146522A1 (en) * | 2008-12-09 | 2010-06-10 | Microsoft Corporation | Handling exceptions in a data parallel system |
US20160036903A1 (en) * | 2014-07-31 | 2016-02-04 | Splunk Inc. | Asynchronous processing of messages from multiple servers |
CN110612716A (en) * | 2017-01-20 | 2019-12-24 | 十位数通信有限责任公司 | Intermediate device for network routing of data messages |
CN107729559A (en) * | 2017-11-08 | 2018-02-23 | 携程旅游网络技术(上海)有限公司 | Method, system, equipment and the storage medium of data base read-write asynchronous access |
CN110597891A (en) * | 2018-06-12 | 2019-12-20 | 武汉斗鱼网络科技有限公司 | Device, system, method and storage medium for aggregating MySQL into PostgreSQL database |
CN110806958A (en) * | 2019-10-24 | 2020-02-18 | 长城计算机软件与***有限公司 | Monitoring method, monitoring device, storage medium and electronic equipment |
CN113179267A (en) * | 2021-04-27 | 2021-07-27 | 长扬科技(北京)有限公司 | Network security event correlation analysis method and system |
CN115221116A (en) * | 2022-07-25 | 2022-10-21 | 深圳市网心科技有限公司 | Data writing method, device and equipment and readable storage medium |
Non-Patent Citations (2)
Title |
---|
T. LI 等: "RAMSYS: Resource-Aware Asynchronous Data Transfer with Multicore SYStems", 《IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS》, vol. 28, no. 05, pages 1430 - 1444, XP011645234, DOI: 10.1109/TPDS.2016.2619344 * |
张天庆 等: "一种异步多线程的Web数据流高效处理模型", 《四川大学学报(自然科学版)》, vol. 42, no. 02, pages 264 - 269 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116756197A (en) * | 2023-08-23 | 2023-09-15 | 中国电信股份有限公司 | Method, system and communication equipment for realizing dynamic window and aggregation parameters |
CN116756197B (en) * | 2023-08-23 | 2023-11-07 | 中国电信股份有限公司 | Method, system and communication equipment for realizing dynamic window and aggregation parameters |
Also Published As
Publication number | Publication date |
---|---|
CN115904369B (en) | 2023-10-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
CN110704290A (en) | Log analysis method and device | |
US11934287B2 (en) | Method, electronic device and computer program product for processing data | |
CN110532322B (en) | Operation and maintenance interaction method, system, computer readable storage medium and equipment | |
CN110196790A (en) | The method and apparatus of abnormal monitoring | |
WO2023103390A1 (en) | Task processing method, task processing apparatus, electronic device and storage medium | |
CN114625597A (en) | Monitoring operation and maintenance system, method and device, electronic equipment and storage medium | |
CN114091704A (en) | Alarm suppression method and device | |
CN115904369B (en) | Method and system for efficiently aggregating and associated analysis of network security source data | |
CN116226189A (en) | Cache data query method, device, electronic equipment and computer readable medium | |
CN111241189A (en) | Method and device for synchronizing data | |
EP3011456B1 (en) | Sorted event monitoring by context partition | |
CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
CN113672671A (en) | Method and device for realizing data processing | |
CN114661807A (en) | Method, device, equipment and medium for processing abnormity of flight management system | |
CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
CN114756301A (en) | Log processing method, device and system | |
CN113077201B (en) | Method, device and system for analyzing service parameters | |
US20130097622A1 (en) | Framework for system communication for handling data | |
CN114546780A (en) | Data monitoring method, device, equipment, system and storage medium | |
CN114201508A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
CN111382057B (en) | Test case generation method, test method and device, server and storage medium | |
CN112749204A (en) | Method and device for reading data | |
CN112214500A (en) | Data comparison method and device, electronic equipment and storage medium | |
CN115563183B (en) | Query method, query device and program product |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |