CN113704226A - Plug-in heterogeneous Syslog access processing system and method - Google Patents
Plug-in heterogeneous Syslog access processing system and method Download PDFInfo
- Publication number
- CN113704226A CN113704226A CN202110969343.9A CN202110969343A CN113704226A CN 113704226 A CN113704226 A CN 113704226A CN 202110969343 A CN202110969343 A CN 202110969343A CN 113704226 A CN113704226 A CN 113704226A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- data stream
- external
- variable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a plug-in heterogeneous Syslog access processing system and a method. The data stream input module supports access of various heterogeneous Syslog data, processing operations such as data cleaning, conversion, formatting, enrichment and the like are carried out through the data stream processing module, and the data streams are output to a database, a Logstash server, a log server or a message queue through the data stream output module. And the plug-in script management module manages the plug-in script. The flow processing task management module is responsible for monitoring and managing the execution state of each processing module. The external data management module supports the introduction of external data sources and supports data processing related operations. The invention realizes flexible access, processing and output of multi-source heterogeneous Syslog data, effectively filters useless alarm data and provides technical support for processing threat events more quickly.
Description
Technical Field
The invention belongs to the field of data processing, and particularly relates to a plug-in heterogeneous Syslog access processing system and method.
Background
The network battle is the main battle style of future high-technology war, and the network attack can effectively paralyze the enemy battle system, disturb the logistic supply of the enemy and induce the public opinion guidance of the enemy people. The network attack scenes are various and invisible and untouchable, and strategic and tactical combat mission execution is greatly promoted once the attack is effectively implemented. The modern military network topology is complex, the number of safety devices is large, the number of safety logs is explosively increased, along with the continuous enhancement of network threat means, the safety logs of a single device are not enough to support the study and judgment of high-level threat events, in order to effectively discover the threat events, correlation analysis needs to be carried out on multiple types of safety logs, however, the input data formats of various types of safety devices are various, the data preprocessing difficulty is high, a system needs to be customized again when one device is accessed, the development workload is large, and the flexibility is insufficient.
Disclosure of Invention
The purpose of the invention is as follows: the invention provides a plug-in heterogeneous Syslog access and processing system and method aiming at the defects of the prior art.
In order to solve the above technical problems, a first aspect of the present invention provides a plug-in heterogeneous Syslog access processing system, which includes a data stream input module, a data stream processing module, a data stream output module, an external data management module, a plug-in script management module, and a stream processing task management module;
the data stream input module is used for accessing the heterogeneous Syslog data, preprocessing the accessed heterogeneous Syslog data and then sending the preprocessed data to the data stream processing module;
the data stream processing module is used for carrying out data cleaning, conversion, formatting and enriching processing operations;
the data stream output module is used for outputting the processed data to a database, a data collection engine Logstash server, a log server or a message queue;
the external data management module is used for introducing an external data source and supporting data processing related operations of the data stream processing module;
the plug-in script management module is used for managing a data input script to be executed by the data stream input module, a data processing script to be executed by the data stream processing module, a data output script to be executed by the data stream output module and an external data management script to be executed by the external data management module; the scripts managed by the plug-in script management module comprise variable sections and code sections;
the stream processing task management module is used for managing data stream processing tasks, and each data stream processing task consists of more than one data stream input module, more than one data stream processing module, more than one data stream output module and more than zero external data management modules; the stream processing task management module can monitor and manage the running state of the composition modules.
With reference to the first aspect, in an implementation manner, an external data source of the external data management module includes an external database and a configuration file, where the external database supports mysql, and the configuration file adopts a properties format;
the variable section of the external data management script comprises an external data management module ID, an external data source address, a user name and password, an external variable name list and a mapping configuration,
the external data management module ID is used for uniquely identifying one external data management module;
the external data source address and the username password are used for connecting the external data management module with an external database or reading a configuration file, and the format is as follows:
external data source address and username password of external database: database internal names, database URL;
external data source address and username password of configuration file: "internal name of configuration file: file URL";
the external variable name list and the mapping configuration are used for mapping the values in the external database or the configuration file to the user-defined variables by the external data management module according to the external variable name list and the mapping configuration, and the format is as follows;
external variable name list and mapping configuration of external database: "variable name: database internal name: SQL statement";
external variable name list and mapping configuration of configuration file: the variable name, the file internal name, the field name.
With reference to the first aspect, in one implementation, the variable segment of the data input script includes a data stream input module ID, a data stream processing module ID list, and a data output variable list,
the data stream input module ID is used for uniquely identifying one data stream input module;
a data stream processing module ID list for representing a data stream processing module list to which the data stream input module transmits heterogeneous Syslog data, the IDs being separated by commas;
a data output variable list used for defining internal variables sent to each data stream processing module, and the format is as follows: the name of the internal variable is data stream processing module ID.
With reference to the first aspect, in an implementation manner, the stream processing task management module may monitor and manage an operation state of the component module, including monitoring resource consumption, alarm data, and call times, and perform management operations of starting, suspending, and stopping the component module;
the variable section of the data processing script includes a data stream processing module ID, a data stream input module ID list, a data stream output module ID list, a data input variable list, a data output variable list, and an external data management module ID list,
the data stream processing module ID is used for uniquely identifying one data stream processing module;
a data stream input module ID list for indicating data to be received from a plurality of data stream input modules, the plurality of IDs being separated by commas;
a data stream output module ID list for indicating a data stream output module list to which the processed data is to be forwarded, the plurality of IDs being separated by commas;
a data input variable list for defining internal variables received from each data stream input module in the format: "internal variable name: data stream input module ID";
a data output variable list for defining internal variables sent to each data stream output module in the format: "internal variable name: data stream output module ID";
and the external data management module ID list is used for defining variables in the external data management module to be referred, and the format is 'internal variable name:: external data management module ID:: external variable name'.
With reference to the first aspect, in one implementation, the variable segment of the data output script includes a data stream output module ID, a data stream processing module ID list, an output address and username password configuration, an input variable list, and an output variable mapping configuration,
the data stream output module ID is used for uniquely identifying one data stream output module;
a data stream processing module ID list for indicating data to be received from a plurality of data stream processing modules, the plurality of IDs being separated by commas;
the output address and the user name password configuration are used for representing the processed data to be stored in a database or a message queue, a Logstash server or a log server to be sent, and the format is ' output target variable name: ' output target URL ';
a data input variable list for defining internal variables received from each data stream processing module in the format: "internal variable name: data stream processing module ID";
an output variable mapping arrangement for defining internal variables for storage or transmission to an external destination, in the format: the name of the internal variable is output as the name of the target variable.
In a second aspect, a plug-in heterogeneous Syslog access processing method is provided, including the following steps:
step 1, adding a data input script, a data processing script, a data output script and an external data management script through a plug-in script management module;
and 6, executing a data output script by the data stream output module, and sending the processed data to a Logstash server, a log server and a message queue or storing the processed data in a database.
With reference to the second aspect, in one implementation manner, the step 3 includes: starting a corresponding external data management module according to the configuration of a variable segment 'external data management module ID' in the external data management script; according to the configuration of the variable segment 'external data source address and user name and password', connecting an external database or opening a configuration file; mapping values in an external database or a configuration file to custom variables according to the configuration of the variable segment 'external variable name list and mapping configuration'; and turning to a code segment, and processing the custom variable.
With reference to the second aspect, in one implementation manner, the step 4 includes: starting a corresponding data stream input module according to the configuration of a variable segment 'data stream input module ID' of a data input script, wherein the data stream input module receives Syslog data; processing the Syslog data according to the configuration of the code segments; and sending the processed data to a data stream processing module according to the configuration of the variable segment 'data output variable list'.
With reference to the second aspect, in one implementation manner, the step 5 includes: starting a corresponding data stream processing module according to the configuration of a variable segment 'data stream processing module ID' of a data processing script; receiving data of a data stream input module according to the configuration of a variable segment 'data input variable list'; acquiring an external variable according to the configuration of the variable segment 'external data management module ID list'; according to the configuration of the code segment, carrying out washing, conversion, formatting and enriching processing operations on the data; and sending the processed data to a data stream output module according to the configuration of the variable segment 'data output variable list'.
With reference to the second aspect, in one implementation manner, the step 6 includes: starting a data stream output module according to the configuration of a variable segment 'data stream output module ID' of a data output script; receiving data of a data stream processing module according to the configuration of the input variable list; connecting a Logstash server, a log server, a message queue or a database according to the configuration of the output address and the user name and password configuration; processing the received data according to the configuration of the code segment; and sending the processed data to a Logstash server, a log server, a message queue or a database according to the configuration of the output variable mapping configuration.
Has the advantages that:
1. the system disclosed by the embodiment of the application is suitable for analyzing and processing Syslog input data in different formats;
2. the system of the embodiment of the application supports debugging, scheduling and monitoring management of different types of data processing scripts;
3. the system has the expansion capability, and can realize the expansion of functions by editing the script;
4. the system of the embodiment of the application supports interaction and correlation analysis among a plurality of processing flows.
The plug-in heterogeneous Syslog access processing system and method provided by the embodiment of the application support the flexible access, flexible processing and flexible output of multi-source heterogeneous Syslog data in a plug-in script mode, effectively filter useless alarm data, improve the readability of the alarm data, improve the working efficiency of an analyst and provide technical support for processing threat events more quickly.
Drawings
The foregoing and other advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings.
Fig. 1 is a schematic structural diagram of a plug-in heterogeneous Syslog access processing system according to an embodiment of the present application.
Fig. 2 is a schematic diagram of an interface of a plug-in script management module in the plug-in heterogeneous Syslog access processing system according to an embodiment of the present application.
Fig. 3 is a schematic diagram of an interface of a flow processing task management module in the plug-in heterogeneous Syslog access processing system according to an embodiment of the present application.
Fig. 4 is a schematic diagram of a typical flow processing task in the plug-in heterogeneous Syslog access processing method provided in the embodiment of the present application.
Detailed Description
The invention is further explained below with reference to the drawings and the embodiments.
The first embodiment of the invention discloses a plug-in heterogeneous Syslog access processing system which can be applied to the field of network security and provides technical support for processing threat events more quickly.
As shown in fig. 1, the plug-in heterogeneous Syslog access processing system supports plug-in streaming processing of multiple heterogeneous Syslog data sources, and includes:
the system comprises a data stream input module, a data stream processing module, a data stream output module, an external data management module, a plug-in script management module and a stream processing task management module;
the data stream input module is used for accessing the heterogeneous Syslog data and sending the accessed heterogeneous Syslog data to the data stream processing module; the heterogeneous Syslog data comprises a host log, a flow log, alarm data of various network security devices and the like;
the data stream processing module is used for carrying out data cleaning, conversion, formatting and enriching processing operations;
the data stream output module is used for outputting the processed data to a database, a Logstash server, a log server or a message queue;
the external data management module is used for introducing an external data source and supporting data processing related operations of the data stream processing module;
the plug-in script management module is used for managing a data input script to be executed by the data stream input module, a data processing script to be executed by the data stream processing module, a data output script to be executed by the data stream output module and an external data management script to be executed by the external data management module; the scripts managed by the plug-in script management module comprise variable sections and code sections;
the stream processing task management module is used for managing data stream processing tasks, and each data stream processing task consists of more than one data stream input module, more than one data stream processing module, more than one data stream output module and more than zero external data management modules; the stream processing task management module can monitor and manage the running state of the composition modules.
The second embodiment of the invention discloses a plug-in heterogeneous Syslog access processing method, the work flow of which is shown in fig. 1, and the method comprises the following steps:
step 1, adding a data input script, a data processing script, a data output script and an external data management script through a plug-in script management module;
as shown in fig. 2, the plug-in script management module supports the addition, deletion, modification and check of a script library, the script is composed of variable segments and code segments, and the contents of each type of script are shown in the following table:
TABLE 1 script types and content description
as shown in fig. 3, the stream processing task management module supports monitoring of the operating states of all external data management modules, data stream input modules, data stream processing modules, and data stream output modules in the system, including resource consumption, alarm data, and call times, and supports management operations of the modules, including start, pause, and stop operations.
As shown in fig. 4, a stream processing task may be composed of 4 kinds of modules, that is, a data stream input module, a data stream processing module, a data stream output module, and an external data management module, where the number of the external data management module may be 0 or more, and the number of other types of modules may be 1 or more. The data flow input module accesses various heterogeneous Syslog data information and sends the accessed data to 1 or more data flow processing modules, the data flow processing modules receive data from the 1 or more data flow input modules, perform data processing by using external variable data provided by 0 or more external data management modules and output the result to the 1 or more data flow output modules, and the data flow output modules can receive the data of the 1 or more data flow processing modules and store the result in an external database, a Logstash server, a log server or send the result to an external message queue.
each external data management module has an independent ID, and the ID is automatically generated by the system;
the external data source supports two types of a database and a configuration file, the database supports mysql, and the configuration file adopts a properties format.
The naming format of the database is as follows: database internal names, database URL;
the configuration file naming format is as follows: "internal name of configuration file: file URL";
the external data management module is connected with a database or reads a configuration file according to configuration, and then maps values in the database or the configuration file to custom variables according to an external variable name list and mapping configuration;
the mapping format of the database variables is as follows: "variable name: database internal name: SQL statement";
the configuration file variable mapping format is as follows: the variable name, the file internal name, the field name.
Examples of configurations are shown in the table:
table 2 external data management module example
As shown in the above table, the external data management module ID is EXTVAR _0XX002, after the module is started, the module connects to the database DB _1 according to the configuration data, opens the configuration FILE _1, searches for relevant data in the DB _1 according to the configured SQL statement and assigns the relevant data to the VAR _1, reads a field value in the configuration FILE _1, and assigns the field value to the VAR _ 2. And after the external data reading operation is finished, the external data is transferred into a code segment, the variable can be further processed according to the user-defined code, and the code adopts a Lua format.
the variable segments of the data input script include a data stream input module ID, a data stream processing module ID list, and a data output variable list.
Each data stream input module has an independent ID, and the ID is automatically generated by the system;
the data stream processing module ID list represents a destination module list to which received data are to be forwarded, and a plurality of IDs are separated by commas;
the data output variable list defines internal variables sent to each data stream processing module, and the format is as follows: the name of the internal variable is data stream processing module ID.
Examples of configurations are shown in the table:
table 3 data stream input module example
As shown in the above table, the ID of the data stream access module is INPUT _0XX001, the module starts to receive SYSLOG data after being started, processes and assigns data matching pattern1 to OUTPUT _1 according to the code segment configuration, processes and assigns data matching pattern2 to OUTPUT _2, sends OUTPUT _1 to the data stream processing module PROCESS _0XX001 according to the variable segment configuration, and sends OUTPUT _2 to the data stream processing module PROCESS _0XX 002.
the variable section of the data processing script includes a data stream processing module ID, a data stream input module ID list, a data stream output module ID list, a data input variable list, a data output variable list, and an external data management module ID list.
Each data stream processing module has an independent ID, and the ID is automatically generated by the system;
the data stream input module ID list indicates data to be received from a plurality of data stream input modules, the plurality of IDs being separated by commas;
the data stream output module ID list represents a destination module list to which the processed data are to be forwarded, and a plurality of IDs are separated by commas;
the data input variable list defines the internal variables received from each data stream input module in the format: "internal variable name: data stream input module ID";
the data output variable list defines internal variables sent to each data stream output module, and the format is as follows: "internal variable name: data stream output module ID";
the external data management module ID list defines variables in the external data management module to be referred to, and the variables are in the format of 'internal variable name:: external data management module ID:: external variable name'.
Examples of configurations are shown in the table:
table 4 data stream processing module example
As shown in the table, the data stream processing module ID is PROCESS _0XX001, after the module is started, the module starts to receive data from INPUT _0XX001 and INPUT _0XX002 modules, stores the received data into variables INPUT _1 and INPUT _2, simultaneously connects to the database DB _1 according to the configuration data, opens the FILE _1, assigns the VAR _1 variable of the external data management module EXTVAR _0XX001 to the internal variable VAR _1, assigns the VAR _2 variable of the external data management module EXTVAR _0XX002 to the internal variable VAR _2, performs an enrichment operation on INPUT _1 by using VAR _1 according to the configuration of the code segment, stores the result into OUTPUT _1, performs an enrichment operation on INPUT _2 by using VAR _2, stores the result into OUTPUT _2, and finally transmits OUTPUT _1 and OUTPUT _2 to the data stream OUTPUT modules OUTPUT _0XX001 and OUTPUT _ 0.
And 6, executing a data output script by the data stream output module, and sending the data to the Logstash server, the log server, the message queue or storing the data in a database.
The variable section of the data output script comprises a data stream output module ID, a data stream processing module ID list, an output address and user name password configuration, an input variable list and an output variable mapping configuration.
Each data stream output module has an independent ID, and the ID is automatically generated by the system;
the data stream processing module ID list indicates data to be received from a plurality of data stream processing modules, the plurality of IDs being separated by commas;
the configuration list of the output address and the user name and the password shows that the processed data is to be stored in a database or sent to a message queue, a Logstash server or a log server, and the format is' output target variable name: output target URL ";
the data input variable list defines internal variables received from each data stream processing module in the format: "internal variable name: data stream processing module ID";
the output variable mapping configuration defines internal variables that are stored or sent to an external destination in the format: the name of the internal variable is output as the name of the target variable.
Examples of configurations are shown in the table:
table 5 data flow output module example
As shown in the above table, the data stream OUTPUT module ID is OUTPUT _0XX001, after the module is started, the module starts to receive data from PROCESS _0XX001 and PROCESS _0XX002 modules, stores the received data into variables INPUT _1 and INPUT _2, connects the database DB _1, the log server SL _1, the message queue MQ _1 and the Logstash server LS _1 according to configuration data, calculates OUTPUT _1, OUTPUT _2, OUTPUT _3 and OUTPUT _4 according to the code segment configuration, and OUTPUTs OUTPUT _1 to DB _1, OUTPUT _2 to MQ _1, OUTPUT _3 to SL _1 and OUTPUT _4 to LS _1 according to the OUTPUT variable mapping configuration.
The present invention provides a plug-in heterogeneous Syslog accessing and processing system and method, and a plurality of methods and ways for implementing the technical solution are provided, and the above description is only a specific embodiment of the present invention, and it should be noted that, for those skilled in the art, a plurality of modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention. All the components not specified in the present embodiment can be realized by the prior art.
Claims (10)
1. A plug-in heterogeneous Syslog access processing system is characterized by comprising a data stream input module, a data stream processing module, a data stream output module, an external data management module, a plug-in script management module and a stream processing task management module;
the data stream input module is used for accessing the heterogeneous Syslog data, preprocessing the accessed heterogeneous Syslog data and then sending the preprocessed data to the data stream processing module;
the data stream processing module is used for carrying out data cleaning, conversion, formatting and enriching processing operations;
the data stream output module is used for outputting the processed data to a database, a Logstash server, a log server or a message queue;
the external data management module is used for introducing an external data source and supporting data processing related operations of the data stream processing module;
the plug-in script management module is used for managing a data input script to be executed by the data stream input module, a data processing script to be executed by the data stream processing module, a data output script to be executed by the data stream output module and an external data management script to be executed by the external data management module; the scripts managed by the plug-in script management module comprise variable sections and code sections;
the stream processing task management module is used for managing data stream processing tasks, and each data stream processing task consists of more than one data stream input module, more than one data stream processing module, more than one data stream output module and more than zero external data management modules; the stream processing task management module can monitor and manage the running state of the composition modules.
2. The plugin heterogeneous Syslog access processing system according to claim 1, wherein the external data source of the external data management module includes an external database and a configuration file, the external database supports mysql, and the configuration file is in a properties format;
the variable section of the external data management script comprises an external data management module ID, an external data source address, a user name and password, an external variable name list and a mapping configuration,
the external data management module ID is used for uniquely identifying one external data management module;
the external data source address and the username password are used for connecting the external data management module with an external database or reading a configuration file, and the format is as follows:
external data source address and username password of external database: the internal names in the database are database URL;
external data source address and username password of configuration file: the internal name of the configuration file is file URL;
the external variable name list and the mapping configuration are used for mapping the values in the external database or the configuration file to the user-defined variables by the external data management module according to the external variable name list and the mapping configuration, and the format is as follows;
external variable name list and mapping configuration of external database: the variable names comprise the internal names in the database and SQL sentences;
external variable name list and mapping configuration of configuration file: the variable names comprise the internal names of files and the field names.
3. The plug-in heterogeneous Syslog access processing system of claim 1, wherein the variable segments of the data input script comprise a data stream input module ID, a list of data stream processing module IDs, and a list of data output variables,
the data stream input module ID is used for uniquely identifying one data stream input module;
a data stream processing module ID list for representing a data stream processing module list to which the data stream input module transmits heterogeneous Syslog data, the IDs being separated by commas;
a data output variable list used for defining internal variables sent to each data stream processing module, and the format is as follows: and the internal variable name is the ID of the data stream processing module.
4. The plug-in heterogeneous Syslog access processing system according to claim 1, wherein the stream processing task management module is capable of monitoring and managing the operation status of the component modules, including monitoring resource consumption, alarm data and call times, and performing start, pause and stop management operations on the component modules;
the variable section of the data processing script includes a data stream processing module ID, a data stream input module ID list, a data stream output module ID list, a data input variable list, a data output variable list, and an external data management module ID list,
the data stream processing module ID is used for uniquely identifying one data stream processing module;
a data stream input module ID list for indicating data to be received from a plurality of data stream input modules, the plurality of IDs being separated by commas;
a data stream output module ID list for indicating a data stream output module list to which the processed data is to be forwarded, the plurality of IDs being separated by commas;
a data input variable list for defining internal variables received from each data stream input module in the format: the internal variable name is that the data stream inputs the module ID;
a data output variable list for defining internal variables sent to each data stream output module in the format: the internal variable name is data stream output module ID;
and the external data management module ID list is used for defining variables in the external data management module to be referred, and the format of the variables is internal variable name, external data management module ID, and external variable name.
5. The plug-in heterogeneous Syslog access processing system of claim 1, wherein the variable segments of the data output script comprise a data stream output module ID, a data stream processing module ID list, an output address and username password configuration, an input variable list, and an output variable mapping configuration,
the data stream output module ID is used for uniquely identifying one data stream output module;
a data stream processing module ID list for indicating data to be received from a plurality of data stream processing modules, the plurality of IDs being separated by commas;
the configuration of the output address and the user name and the password is used for indicating that the processed data is to be stored in a database or sent to a message queue, a Logstash server or a log server, and the format is as follows: outputting the target variable name, namely outputting a target URL;
a data input variable list for defining internal variables received from each data stream processing module in the format: the internal variable name is data stream processing module ID;
an output variable mapping arrangement for defining internal variables for storage or transmission to an external destination, in the format: and outputting the target variable name.
6. A plug-in heterogeneous Syslog access processing method is characterized in that the Syslog comprises the following steps:
step 1, adding a data input script, a data processing script, a data output script and an external data management script through a plug-in script management module;
step 2, adding a stream processing task through a stream processing task management module, calling an external data management module, a data stream input module, a data stream processing module and a data stream output module to execute the stream processing task, and monitoring the task state;
step 3, the external data management module executes the external data management script to obtain an external variable;
step 4, the data stream input module executes the data input script and transmits the data into the data stream processing module;
step 5, the data stream processing module executes the data processing script, receives the data output by the data stream input module and the external variable output by the external data management module, processes the data, and transmits the processed data to the data stream output module;
and 6, executing a data output script by the data stream output module, and transmitting the processed data to a Logstash server, a log server and a message queue or storing the processed data in a database.
7. The plug-in heterogeneous Syslog access processing method according to claim 6, wherein the step 3 comprises: starting a corresponding external data management module according to the configuration of the variable segment external data management module ID in the external data management script; connecting an external database or opening a configuration file according to the configuration of the source address of the external data of the variable segment and the user name and the password; mapping values in an external database or a configuration file to custom variables according to the configuration of the external variable name list and the mapping configuration of the variable segment; and turning to a code segment, and processing the custom variable.
8. The plugin heterogeneous Syslog access processing method according to claim 6, wherein the step 4 includes: starting a corresponding data stream input module according to the configuration of a variable segment data stream input module ID of a data input script, wherein the data stream input module receives Syslog data; processing the Syslog data according to the configuration of the code segments; and according to the configuration of the variable segment data output variable list, sending the processed data to a data stream processing module.
9. The plug-in heterogeneous Syslog access processing method according to claim 6, wherein the step 5 comprises: starting a corresponding data stream processing module according to the configuration of the variable segment data stream processing module ID of the data processing script; receiving data of a data stream input module according to the configuration of a variable section data input variable list; acquiring an external variable according to the configuration of the variable section external data management module ID list; according to the configuration of the code segment, carrying out washing, conversion, formatting and enriching processing operations on the data; and according to the configuration of the variable segment data output variable list, sending the processed data to a data stream output module.
10. The plugin heterogeneous Syslog access processing method according to claim 6, wherein the step 6 includes: starting a data stream output module according to the configuration of the variable segment data stream output module ID of the data output script; receiving data of a data stream processing module according to the configuration of an input variable list; connecting a Logstash server, a log server, a message queue or a database according to the configuration of the output address and the configuration of the user name and the password; processing the received data according to the configuration of the code segment; and sending the processed data to a Logstash server, a log server, a message queue or a database according to the configuration of the output variable mapping configuration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110969343.9A CN113704226B (en) | 2021-08-23 | 2021-08-23 | Plug-in heterogeneous Syslog access processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110969343.9A CN113704226B (en) | 2021-08-23 | 2021-08-23 | Plug-in heterogeneous Syslog access processing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113704226A true CN113704226A (en) | 2021-11-26 |
| CN113704226B CN113704226B (en) | 2023-01-31 |
Family
ID=78654142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110969343.9A Active CN113704226B (en) | 2021-08-23 | 2021-08-23 | Plug-in heterogeneous Syslog access processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113704226B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101083656A (en) * | 2007-07-05 | 2007-12-05 | 上海交通大学 | Data stream technique based multi-source heterogeneous data integrated system |
| CN104104738A (en) * | 2014-08-06 | 2014-10-15 | 江苏瑞中数据股份有限公司 | FTP-based (file transfer protocol-based) data exchange system |
| CN111698194A (en) * | 2019-03-14 | 2020-09-22 | 捷翊信息科技(上海)有限公司 | Multi-source heterogeneous data integration system and method |
-
2021
- 2021-08-23 CN CN202110969343.9A patent/CN113704226B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101083656A (en) * | 2007-07-05 | 2007-12-05 | 上海交通大学 | Data stream technique based multi-source heterogeneous data integrated system |
| CN104104738A (en) * | 2014-08-06 | 2014-10-15 | 江苏瑞中数据股份有限公司 | FTP-based (file transfer protocol-based) data exchange system |
| CN111698194A (en) * | 2019-03-14 | 2020-09-22 | 捷翊信息科技(上海)有限公司 | Multi-source heterogeneous data integration system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113704226B (en) | 2023-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109542011B (en) | Standardized acquisition system of multisource heterogeneous monitoring data | |
| CN106875156B (en) | Universal intelligent auditing platform and auditing method thereof | |
| CN108616419B (en) | Data packet acquisition and analysis system and method based on Docker | |
| CN106790718A (en) | Service call link analysis method and system | |
| CN103701783B (en) | Preprocessing unit, data processing system consisting of same, and processing method | |
| CN103618652A (en) | Audit and depth analysis system and audit and depth analysis method of business data | |
| CN103593613A (en) | Method, terminal, server and system for computer virus detection | |
| CN111930886A (en) | Log processing method, system, storage medium and computer equipment | |
| US20190166143A1 (en) | Method for collecting cyber threat intelligence data and system thereof | |
| CN108829505A (en) | A kind of distributed scheduling system and method | |
| CN110851234A (en) | Log processing method and device based on docker container | |
| US20210344703A1 (en) | Visualized Penetration Testing (VPEN) | |
| CN112732663A (en) | Log information processing method and device | |
| CN110334119A (en) | A kind of data correlation processing method, device, equipment and medium | |
| CN114465741A (en) | Anomaly detection method and device, computer equipment and storage medium | |
| CN110442582B (en) | Scene detection method, device, equipment and medium | |
| CN111787030A (en) | Network security inspection method, device, equipment and storage medium | |
| CN111984505A (en) | Operation and maintenance data acquisition engine and acquisition method | |
| CN113704226B (en) | Plug-in heterogeneous Syslog access processing system and method | |
| CN113382010B (en) | Large-scale network security defense system based on cooperative intrusion detection | |
| CN103997438A (en) | Method for automatically monitoring distributed network spiders in cloud computing | |
| CN113676354A (en) | Hybrid cloud operation and maintenance management method and system | |
| CN111177239B (en) | Unified log processing method and system based on HDP big data cluster | |
| CN117389825A (en) | Method, system and device for monitoring Flink job log in real time | |
| CN115168297A (en) | Bypassing log auditing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |