CN105589940A

CN105589940A - Safety management operation and maintenance service platform based on unstructured real-time database

Info

Publication number: CN105589940A
Application number: CN201510937594.3A
Authority: CN
Inventors: 凌飞; 李木金
Original assignee: Nanjing Liancheng Technology Development Co Ltd
Current assignee: Nanjing Liancheng Technology Development Co Ltd
Priority date: 2015-12-16
Filing date: 2015-12-16
Publication date: 2016-05-18

Abstract

The invention discloses a safety management operation and maintenance service platform based on an unstructured real-time database. The core concept of the safety management operation and maintenance service platform comprises the steps of establishing the unstructured real-time database, and storing log information of various kinds of hetero-structure equipment of different manufacturers through the database. By adoption of the safety management operation and maintenance service platform, on one hand, the log information can be stored based on time horizon or a fragmentation manner, and event association/or association analysis can be carried out among different equipment through the log fields; and on the other hand, the platform can adapt complex IT operation and maintenance environment.

Description

A kind of safety management O&M service platform based on destructuring real-time data base

Technical field

The present invention relates to information security applied technical field, relate in particular to the method and system of the safety management O&M platform architecture of destructuring real-time data base.

Background technology

The English comprising in the present invention is called for short as follows:

SSD:SolidStateDrives solid state hard disc

ID:identification mark

Language is processed in SPL:SearchProcessingLanguage search

ETL:Extract, TransformandLoad extraction, conversion and loading technique

SOC:SecurityOperationCenter security management center

IDS:IntrusionDetectionSystems intruding detection system

MIS:ManagementInformationSystem management information system

DMZ:demilitarizedzone isolated area or demilitarized zone

APP:Application application program

SNMP:SimpleNetworkManagementProtocol Simple Network Management Protocol

HDFS:HadoopDistributeFileSystemHadoop distributed file system

ODBC:OpenDatabaseConnectivity Open Database Connection

WMI:WindowsManagementInstrumentationWindows management regulation

The safe open platform of OPSEC:OpenPlatformforSecurity

NAS:NetworkAttachedStorage network attached storage

SAN:StorageAreaNetworkandSANProtocols storage area network and agreement thereof

IBM:InternationalBusinessMachinesCorporation International Business Machine Corporation (IBM)

MQ:MessageQueue message queue.

Safety in production is always the prerequisite that ensures that work in every is carried out in order, is also examination leading cadres at various levels' rejection index. Network and information security operation and maintenance system is the important component part of all kinds of enterprise safety operation work. Ensureing that network-efficient stably moves, is the basis of all market management activities of enterprise and normal operation.

Along with the construction of all kinds of enterprise information systems and perfect, effectively improve labor productivity, reduce operation cost. Once there is security incident or break down or forming property bottleneck in the each operation system of enterprise, can not find in time, process in time, recover in time, certainly will directly cause the carrying operation of all business thereon, affect the normal operation order of enterprise, business event can not normally be carried out. Therefore the safety guarantee of, implementing for Government and enterprise IT basis just seems especially important.

Along with the Government and enterprise level of informatization improves constantly. Between each operation system, contact is more and more closer, exchanges data is more and more frequent, each system has complex network or logic to connect, there is mass data exchange, even a fault can cause and become enterprise's the whole network fault, any or a kind of operation system start a leak and infect virus or under attack, will involve rapidly other operation system and network, even cause enterprise's the whole network paralysis.

Enterprise IT System has produced abundant daily record data, along with the reduction of memory device cost, have no reason to abandon these data, but, lacking corresponding analysis tool stores and the daily record data of associated these isomeries, make to carry out large data analysis more difficult, be mainly manifested in the following aspects:

1, the time response of daily record data and isomery characteristic, makes traditional SQL structured database be difficult to be applicable to.

Traditional database canned data, just represents certain state. Specifically, the real world on certain time point or virtual objects snapshot. For example, stock, plane ticket booking and company personnel's information. Moreover on the one hand, these data volumes are limited, for example, the impossible more than seven-thousand-million bar record of company personnel's information bank, because do not have the headcount more than seven-thousand-million of which company. In addition, the each record in database has one group of attribute (for example, name, address, wage, sex, date of birth). Although the attribute difference of each record,, the set of these record attributes can not exceed the upper limit of traditional database design. Therefore, such data are called as structural data. The analytical technology of structural data is considerably ripe, has many very professional analysis tools. Be applicable to very much structural data based on traditional SQL relational database.

By contrast, the daily record data that most of equipment produces, for example, syslog, webserver daily record, be all some non-structured texts. Although text may imply loose coupling,, the journal file that various equipment produces may be different, and can change continually, or even establish unimaginable unexpected variation. Conventionally, such data have represented the history of a complete event, instead of a time snapshot. Therefore, the data volume of the journal file that equipment generates, conventionally than the large several orders of magnitude of structural data. In addition, each record has a common attribute: Time To Event. Therefore, traditional analytical technology is no longer applicable to the data as daily record. In the past, although log packet contains very Useful Information,, seldom these daily records are analyzed, and often deleted to discharge memory space.

2, the time is the optimal relevance device of heterogeneous data source.

The thousands of component software (data source) that implement on the IT basis of enterprise, has produced ten hundreds of dissimilar daily record datas. Due to the destructuring character of these data, there is no usually said unique identification (or ID), to distinguish various daily record, carry out association analysis with the unique identification by daily record. But the time that each event occurs, this time is unique associated all kinds of destructuring daily record data that can be used for carrying out. Traditional relational database semanteme does not have the function of this association in time, because the identical daily record of the little generation time of distinct device.

3, the time is important characteristic.

Except the time is that the best conjunctive word of unstructured data, the time is also very important for data management and query optimization. Association analysis often relates to and is limited in the data within the scope of special time, instead of all data. In addition,, from the data in nearest time range, conventionally than the data before it, it is high that priority is wanted. Therefore, optimize nearest data very important. Although now carrying cost is declining, make to store TB DBMS (even PB DBMS) than wanting in the past economical,, Rapid storage technique, as SSD, or costliness. Therefore, the storage of a dynamic data should be paid the utmost attention to nearest data are stored in such as on SSD fast disc, and by than before data store on more cheap memory device. Traditional database is easy to memory time, and still, in the time of stored record, its time is not considered, makes to be like this difficult to analyze and optimize nearest data. If data can, by time cutting, be very efficient to the analysis of a certain section of time range.

For this reason, how to utilize information-based means to improve enterprise security manager O&M benefit, optimize the service of enterprise information security management O&M, make it for all kinds of enterprises provide specialty and high performance information safety operation and maintenance management service, to become an important topic that must solve in the design of information security management O&M especially.

Summary of the invention

The present invention, after having analyzed the defect and deficiency of above-mentioned all kinds of enterprise information security management O&M service platforms, has proposed a kind of safety management O&M service platform based on semi-structured real-time data base.

Core concept of the present invention is: build a destructuring real-time data base, the log information of storage text formatting.

Described destructuring real-time data base, the log information of storage, its field includes but not limited to: time (time), the time that event occurs; Unique identifier (ID), the i.e. mark relevant to event, for example, affairs ID(Transactionid), ID (userid), product (Productid) and message id (Messageid); Source IP(Source_IP); Source port number (Source_Port); Target IP (Dest_IP); Destination port number (Dest_Port); Event type (SourceType); Main frame (Host); Event description (EventText).

Described log information field, can realize the event correlation between distinct device.

Preferably, can be according to time range storing daily record information, or with a point sheet mode storing daily record information.

Preferably, the burst of index and storing daily record information, preserves multiple copies.

This patent provides a kind of method and device of high performance large data query for the subject matter existing in current technical scheme, have the feature that adapts to following IT operation:

(1) it is uncertain changing

Reaching the standard grade and rolling off the production line of equipment, is dynamic change, or even uncertain. Therefore, the log information of new equipment also can be different with the journal format of existing device, adopts the such daily record of traditional data library storage, need in advance Log data format to be analyzed, like this, and very time-consuming, effort, impossible only not.

(2) software and hardware upgrade is inevitable

Assembly needs regular upgrading and/or reconfigures, for device upgrade and System Expansion. For example, IPv4 upgrades to IPv6. Therefore, the log information of the equipment after upgrading also can be different with the journal format of updating apparatus not, adopts the such daily record of traditional data library storage, needs analyze Log data format in advance. Like this, be also very time-consuming, effort, also impossible only not.

(3) fault is inevitable

The length of service of hardware, the BUG of software, and maloperation and security attack, therefore, fault can not be utterly destroyed, but should detect as quickly as possible, and loss is reduced to minimum.

Brief description of the drawings

Fig. 1 is the MapReduce application scenarios schematic diagram of a kind of safety management O&M service platform based on destructuring real-time data base of the present invention;

Fig. 2 is the storing framework schematic diagram of a kind of safety management O&M service platform based on destructuring real-time data base of the present invention.

Detailed description of the invention

Be below with reference to the accompanying drawings with example to further description of the present invention:

The data of the destructuring form generating from various device, change into the structural data that traditional relational requires like that, and existing technology realizes (being ETL), a kind of extraction, conversion and loading technique by data pick-up. This instrument is in-problem for analyzing the data that produced by various device. Require to use the designers of ETL instrument extracted data must be familiar with total data form could correctly to complete the task of data pick-up. Pre-hypothesis is like this infeasible, and reason has following 2 points: the IT system of many enterprises is stored the full directory of data never to some extent, and the data that have never seen before extracting have just had difficulties. Secondly, the data volume in large scale system is very big, correctly extracts that they are unpractical often.

By adopting non-structured real-time data base, these problems are avoidable to a great extent, while finding new data, only need to add a new resolution rules; The method of possible resolution data has multiple.

When another common problem, definition " reservations " strategy while using the real time data that traditional relational database memory device produces, no matter this is angle, law and the compliance aspect from storing, and is very important; But this method still needs in advance equipment to be produced the understanding of data.

Based on the large Data Analysis Platform of Hadoop/Spark, ApacheHive proposed one general, the extensive interface of destructuring real-time data base, and provide ETL instrument, the mass data being stored in distributed file system (HDFS) is analyzed, adopt MapReduce model, and the job management engine of Hadoop; But Hive is designed to batch processing system, do not there is practical function, minimum operation has also needed a few minutes, and such performance can not meet information security management O&M at all locates the requirement of fault and security attack fast.

At present, unstructured data is processed the daily record data (in other words, being confined to limited data source) of the equipment generation that is only confined to our factory, is confined to understand in advance the form of daily record data.

Information security management O&M platform of the present invention, relates to the equipment of various manufacturer production. Its core concept is: build a destructuring real-time data base, the journal format of storage and compatible each producer's equipment.

Described destructuring real-time data base, its field includes but not limited to: time (time), the time that event occurs; Unique identifier (ID), the i.e. mark relevant to event, for example, affairs ID(Transactionid), ID (userid), product (Productid) and message id (Messageid); Source IP(Source_IP); Source port number (Source_Port); Target IP (Dest_IP); Destination port number (Dest_Port); Event type (SourceType); Main frame (Host); Event description (EventText).

Further, can be according to time range storing daily record information, or with a point sheet mode storing daily record information.

Fig. 1 is that one of the present invention is based on destructuring real-time data base search service MapReduce application scenarios schematic diagram.

Collection and the storage of large data, need to rely on data storage shelves structure highly reliably. As the basis of large data analysis, data store whether good relationship is very large.

Along with enterprise IT architecture is constantly expanded, the quantity of server, memory device is more and more, and it is more complicated that network becomes, if guarantee that business event is uninterrupted, operation is more efficient, the core missions of safety management Operation and Maintenance Center, also become O&M personnel must reply challenge.

For the data of these scale of construction super larges, original O&M thinking and O&M method have been difficult to meet the demand of the multiple functions such as its mass data calculating, storage, application and safety. Thousands of information technoloy equipments on the one hand, and various software systems; The service application of various complexity on the other hand, enterprise will realize large scale of construction system management by advanced automation operation management pattern.

By means of the large data platform of Hadoop/Spark, the core competitiveness of General Promotion information security management O&M platform. In Fig. 1, first, find out the burst (for example, adopting HDFS memory technology) that comprises described time range by index server; Then, find concurrently desired data; Finally, Fusion query result, and export to user.

Fig. 2 is a kind of data acquisition treatment method of information security management O&M service platform and functional framework schematic diagram of device based on large data of the present invention. In Fig. 2, there are many acquisition terminals, it is responsible for receiving the safety management O&M data of the various distinct devices that collect. In Fig. 2, also there are many index servers, it is responsible for storage and data query, and each index server is only responsible for inquiry and the write operation of jurisdiction stored data base. Described acquisition terminal, index server and storage were both established and can be made up of different computer systems, were made up of different software modules.

While storing the data that collect, acquisition terminal, according to " storage nearby " strategy, determine that one or more storage data server receives the data that collect, and the data retransmission that it is collected is to storage data server; On the other hand, acquisition terminal not only can be carried out some data correlation operations, for example, (to eliminate redundancy and irrelevant data, repeat alarm data), and all right current the collected data of buffer memory of acquisition terminal, to prevent that storing data server causes for various reasons not reception and cause loss of data. Determine that one or more storage data server receives the data that collect, can decide according to the place of image data and time, need to detect the timestamp in institute's image data, connect to which platform index server and storage server so that acquisition terminal determines to send image data.

Notice on the storage data server being distributed under different index servers, can realize with the ways of distribution that walks abreast, this framework is applicable to the MapReduce of Hadoop or the distributed computing platform of Spark very much. Contribute to promote the query performance of mass data by this computation model, contribute to promote the warehouse-in performance of magnanimity image data, contribute to promote user and experience. In the time storing data on the storage server under index server, can realize in concurrent mode, this is because storage server can be processed the data of being inputted in concurrent mode under multiple index servers. When this concurrent processing also can occur in data query, this is because multiple index servers also can be realized data query concomitantly. Its process is as follows:

First, index server receives the data from acquisition terminal, according to the position in acquisition terminal place (or IP) and timestamp, decides the index server and the storage data server that receive from acquisition terminal data; Secondly, index server checks that whether received data field is complete, for example, if lack time field, inserts the time (time that this time can be current data inserting). Notice collected data, may comprise many row, separate by carriage return, or line break, and an image data comprises a line or multirow. Before image data warehouse-in, need to eliminate the border of unnecessary character string and determination data etc. Can determine by heuristic rule (such as based on regular expression rule or based on separator rule) border of image data, also can determine image data border by predefined feature or feature string. These predefined features can comprise punctuation mark or other special character, for example, and carriage return, tab key, space or line break etc. In some cases, in order to make rule can adapt to the special demand of user, user can finely tune or configuration rule makes index server can utilize better it to determine the border of image data.

Secondly, index server is that each data that collect determine time of origin. As mentioned above, these times may directly be extracted from the data that collect, or insert time field, are similar to the time of origin of the data that collect. Under some situation, the time is exactly to receive the time of data or the time that data produce. Index server is undertaken associated by the data that the time collects with each. Therefore, safety management O&M platform can pass through the processing to image data, can remove the partial content (for example, irrelevant text, feature etc.) in image data, or remove the redundancy section in data. Notice that user also can be by utilizing regular expression or other possibility technology specify the part that will delete.

Secondly, can add selectively keyword index and contribute to search rapidly gathered data by keyword. In order to build keyword index, first index server will determine some keywords (for example, source IP, Target IP, event type). Then, index server comprises the keyword that some is definite, and each keyword and the image data that comprises this keyword are carried out associated or located the corresponding image data of this keyword. After, in the time that index server receives the inquiry based on keyword, index server just can be determined the image data that comprises keyword fast by access critical word indexing.

In certain embodiments, key word index can be included in " title-value " the right record that can find in image data, and one of them " title-value " is to comprising a pair of keyword being connected by symbol, as an equal symbol or colon. By this way, comprising the image data that these " title-values " are right can locate rapidly; For example, if character " destIP=10.0.1.2 " is found in image data, this " destIP " field can automatically create for this image data, and assign a value is " 10.0.1.2 ".

Finally, store the image data that data server is stored under index server, wherein, each image data will be stored a time with the easy-to-look-up event based on in certain time range. In some cases, stored image data is organized into multiple storage bursts. Wherein, the associated specific time range of image data that each storage burst is stored. This has not only improved the efficiency of time-based search/query, and it also allows nearest event to be stored in internal memory to facilitate the speed of retrieval/inquiry faster and access continually. For example, a storage burst that comprises up-to-date image data can be stored in flash memory, instead of on hard disk. Each index server is responsible for the storage of image data and the image data that retrieval is stored. By by image data multiple index servers and under storage data server carry out distributed storage, image data can be analyzed and inquire about to multiple index servers concurrently, for example, this framework can improve the performance of platform and promote user with ApacheHadoopMapReduce or Spark computation model to be experienced, wherein the Search Results of each index server returning part is to query engine, query engine remerges each subquery result that each index server feeds back, thereby inquires about answer for user produces one.

In addition, index and storage burst can have multiple copies, be stored in multiple different index servers and under storage data server on, to realize high availability and the disaster recovery functionality of safety management O&M platform.

The foregoing is only preferred embodiment of the present invention, be not used for limiting practical range of the present invention; Every equivalence of doing according to the present invention changes and amendment, is all regarded as the scope of the claims of the present invention and contains.

Claims

1. the invention provides a kind of safety management O&M service platform based on destructuring real-time data base, described destructuring real-time data base, comprises following one or more field:

Time (time), the time that event occurs;

Unique identifier (ID), the i.e. mark relevant to event, for example, affairs ID(Transactionid), ID (userid), product (Productid) and message id (Messageid);

Source IP(Source_IP);

Source port number (Source_Port);

Target IP (Dest_IP);

Destination port number (Dest_Port);

Event type (SourceType);

Main frame (Host);

Event description (EventText);

Distributed storage engine, can receive concurrently collected data, and store;

Distributed query, can inquire about image data the fast return of storing according to inquiry request;

Semi-structured real-time data base, the database of storage institute image data, adopts semi-structured real-time data base, or destructuring real-time data base.

2. destructuring real-time data base as claimed in claim 1, also comprises:

3. destructuring real-time data base as claimed in claim 1, also comprises:

Can be according to time range storing daily record information, or with a point sheet mode storing daily record information.

4. destructuring real-time data base as claimed in claim 1, also comprises:

The burst of index and storing daily record information, preserves multiple copies.