CN107766408A - The storage method of audit log - Google Patents
The storage method of audit log Download PDFInfo
- Publication number
- CN107766408A CN107766408A CN201710767773.6A CN201710767773A CN107766408A CN 107766408 A CN107766408 A CN 107766408A CN 201710767773 A CN201710767773 A CN 201710767773A CN 107766408 A CN107766408 A CN 107766408A
- Authority
- CN
- China
- Prior art keywords
- thread
- stored
- annular chain
- warehouse
- chain meter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to computer information technology field, more particularly to the storage method of audit log, according to N number of acquisition source of audit log, establish N number of caching thread and N number of warehouse-in thread, wherein each obtain source and correspond to a caching thread and a warehouse-in thread respectively, an annular chain meter is established between each caching thread and warehouse-in thread, log information to be stored is sequentially stored into annular chain meter by described caching thread, record the position A1 of first day will in each annular chain meter and the position B1 of tail daily record, the bar number that daily record is stored in annular chain meter is retrieved using spin mode, as B1 A1 ＞ M, warehouse-in thread is taken out in log information deposit database all in annular chain meter, and current B1 is set to A1.This method effectively solves the high efficiency storage of a large amount of log informations and the technical problem of inquiry.
Description
Technical field
The invention belongs to computer information technology field, and in particular to a kind of storage method of audit log.
Background technology
With becoming increasingly popular for Internet, in the communications field, network firewall, database audit system application increasingly
Extensively, increased therewith is occuring frequently for various attack meanses, and substantial amounts of log information is produced during information audit, in order to
Improve Data Audit performance it is necessary to improve the storage of log information, also for the mass data in database, it is necessary to
Efficient querying method, and the existing warehouse-in efficiency for log information is low, causes database thread to take excessively
CPU, so as to cause the hydraulic performance decline of whole network information audit system.
The content of the invention
In order to solve a large amount of log information warehouse-in efficiencies in the prior art it is low, inquiry spend the time length, cause information audit
The technical problem of the hydraulic performance decline of system, the present invention provides a kind of storage method of a large amount of audit logs, especially by following skill
Art scheme is achieved：
The storage method of audit log, according to N number of acquisition source of audit log, establish N number of caching thread and N number of storage line
Journey, wherein each obtain source corresponds to a caching thread and a warehouse-in thread respectively, in each caching thread and warehouse-in thread
Between establish an annular chain meter, log information to be stored is sequentially stored into annular chain meter by described caching thread；
The position A1 of first day will in each annular chain meter and the position B1 of tail daily record are recorded, annular is retrieved using spin mode
The bar number of daily record is stored in chained list, as B1-A1 ＞ M, warehouse-in thread takes out log information composition one all in annular chain meter
Bar SQL statement, perform the SQL statement and log information is stored in database, and current B1 is set to A1.
Wherein, it is described that log information to be stored is sequentially stored into caching thread in annular chain meter specifically, caching to
The time for each log information to be stored that thread record receives, daily record to be stored is stored in ring according to the sequencing of time
In shape chained list.
Wherein, described warehouse-in thread is taken out in log information deposit database all in annular chain meter, specifically includes,
N number of storage table is established in database, wherein the corresponding warehouse-in thread of each storage table, is carried out in storage table according to the period
Subregion, different periods are stored in subregion corresponding to the log information deposit of database.
Further, described M can use the arbitrary integer between 100-500.
Above technical scheme has following technique effect compared with prior art：
The method of the present invention sets multiple caching threads and warehouse-in thread, caching according to the cpu bus number of passes of database
The multiple log informations received are stored in annular chain meter by thread, when the daily record quantity in annular chain meter reaches setting value,
Once all log informations are stored in database by warehouse-in thread, improve the warehouse-in efficiency of log information, same to time
When will information stores in data, each thread, which is established, corresponding storage table, and storage table is carried out into subregion according to the period, different
Period log information is stored in different subregion, when carrying out log query, according to the period can between find corresponding subregion
Inquired about, improve log query efficiency.
Brief description of the drawings
Fig. 1 is flow chart of the method for the present invention；
Embodiment
To make the purpose, technical scheme and advantage of embodiments of the invention clearer, below in conjunction with accompanying drawing to this hair
It is bright to be described in further detail.
Present embodiments provide a kind of storage method of audit log, such as Fig. 1, for substantial amounts of audit log system, root
According to N number of acquisition source of audit log, N number of caching thread and N number of warehouse-in thread are established, wherein each obtain source corresponds to one respectively
Caching thread and a warehouse-in thread, establish an annular chain meter, wherein auditing between each caching thread and warehouse-in thread
The acquisition source of daily record can be that an audit log obtains source according to audit Protocol Plug division, an audit Protocol Plug.
Log information to be stored is sequentially stored into annular chain meter by caching thread, a cache lines corresponding to each log acquisition source
Journey, an annular chain meter and a warehouse-in thread；Record the position A1 of first day will in each annular chain meter and the position of tail daily record
B1, the bar number that daily record is stored in annular chain meter is retrieved using spin mode, as B1-A1 ＞ M.Wherein M configures according to database
Parameter be adjusted, M can be set to 100-500, and M is arranged to 100 in the present embodiment, you can storage 100 log informations, enter
Storehouse thread is taken out in log information deposit database all in annular chain meter, and all storage daily records between A1-B1 are spelt
Into a SQL statement, the SQL statement is then performed, by daily record storage into database in corresponding storage table and will be current
B1 is set to A1, and so annular chain meter is emptied, and completes in-stockroom operation.
Further, log information to be stored is sequentially stored into annular chain meter specifically, cache lines by caching thread
The time for each log information to be stored that Cheng Jilu is received, daily record to be stored is stored in annular according to the sequencing of time
In chained list.
Further, warehouse-in thread is taken out in log information deposit database all in annular chain meter, specifically includes,
N number of storage table is established in database, wherein the corresponding warehouse-in thread of each storage table, is divided in storage table according to the period
Area, different periods are stored in subregion corresponding to the log information deposit of database.It is any whole between the desirable 100-500 of described M
Number.
Claims (4)
1. the storage method of audit log, it is characterised in that according to N number of acquisition source of audit log, establish N number of caching thread and
N number of warehouse-in thread, wherein each obtain source corresponds to a caching thread and a warehouse-in thread respectively, in each caching thread and
An annular chain meter is established between warehouse-in thread, log information to be stored is sequentially stored into annular chain meter by described caching thread
In；
The position A1 of first day will in each annular chain meter and the position B1 of tail daily record are recorded, annular chain meter is retrieved using spin mode
The bar number of middle storage daily record, as B1-A1 ＞ M, warehouse-in thread takes out log information all in annular chain meter and forms a SQL
Sentence, perform the SQL statement and log information is stored in database, and current B1 is set to A1.
2. the method as described in claim 1, it is characterised in that it is described by caching thread by log information to be stored successively
Specifically, caching thread records the time of each log information to be stored received, according to the time in deposit annular chain meter
Daily record to be stored is stored in annular chain meter by sequencing.
3. the method as described in claim 1, it is characterised in that described warehouse-in thread takes out daily record all in annular chain meter
In information deposit database, specifically include, N number of storage table is established in database, wherein the corresponding storage of each storage table
Thread, subregion is carried out according to the period in storage table, different periods are stored in subregion corresponding to the log information deposit of database.
4. the method as described in claim 1, it is characterised in that described M values can between 100-500 arbitrary integer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710767773.6A CN107766408A (en) | 2017-08-31 | 2017-08-31 | The storage method of audit log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710767773.6A CN107766408A (en) | 2017-08-31 | 2017-08-31 | The storage method of audit log |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766408A true CN107766408A (en) | 2018-03-06 |
Family
ID=61264991
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710767773.6A Pending CN107766408A (en) | 2017-08-31 | 2017-08-31 | The storage method of audit log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766408A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457181A (en) * | 2019-08-02 | 2019-11-15 | 武汉达梦数据库有限公司 | A kind of the log method for optimization analysis and device of database |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| CN103914471A (en) * | 2012-12-31 | 2014-07-09 | 北京启明星辰信息技术股份有限公司 | Mass event processing method and device |
| CN103927338A (en) * | 2014-03-26 | 2014-07-16 | 网神信息技术（北京）股份有限公司 | Log information storage processing method and log information storage processing device |
| CN105302909A (en) * | 2015-11-06 | 2016-02-03 | 西安交大捷普网络科技有限公司 | Partition deviation calculation based big data retrieval method for network security log system |
| US9342544B2 (en) * | 2014-01-30 | 2016-05-17 | International Business Machines Corporation | Parallel load in a column-store database |
| CN105786917A (en) * | 2014-12-26 | 2016-07-20 | 亿阳信通股份有限公司 | Concurrent time series data loading method and device |
| CN108255884A (en) * | 2016-12-29 | 2018-07-06 | 大唐移动通信设备有限公司 | A kind of data processing method and device of the log information of OMC |
-
2017
- 2017-08-31 CN CN201710767773.6A patent/CN107766408A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| CN103914471A (en) * | 2012-12-31 | 2014-07-09 | 北京启明星辰信息技术股份有限公司 | Mass event processing method and device |
| US9342544B2 (en) * | 2014-01-30 | 2016-05-17 | International Business Machines Corporation | Parallel load in a column-store database |
| CN103927338A (en) * | 2014-03-26 | 2014-07-16 | 网神信息技术（北京）股份有限公司 | Log information storage processing method and log information storage processing device |
| CN105786917A (en) * | 2014-12-26 | 2016-07-20 | 亿阳信通股份有限公司 | Concurrent time series data loading method and device |
| CN105302909A (en) * | 2015-11-06 | 2016-02-03 | 西安交大捷普网络科技有限公司 | Partition deviation calculation based big data retrieval method for network security log system |
| CN108255884A (en) * | 2016-12-29 | 2018-07-06 | 大唐移动通信设备有限公司 | A kind of data processing method and device of the log information of OMC |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457181A (en) * | 2019-08-02 | 2019-11-15 | 武汉达梦数据库有限公司 | A kind of the log method for optimization analysis and device of database |
| CN110457181B (en) * | 2019-08-02 | 2023-05-16 | 武汉达梦数据库股份有限公司 | Log optimization analysis method and device for database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103488704B (en) | A kind of date storage method and device | |
| PH12019501499A1 (en) | Blockchain consensus method and device | |
| CN104407879B (en) | A kind of power network sequential big data loaded in parallel method | |
| CN102110132B (en) | Uniform resource locator matching and searching method, device and network equipment | |
| CN104462141B (en) | Method, system and the storage engines device of a kind of data storage and inquiry | |
| CN103902467B (en) | Compressed memory access control method, device and system | |
| CN104090889A (en) | Method and system for data processing | |
| CN108809854A (en) | A kind of restructural chip architecture for big flow network processes | |
| CN102682108B (en) | Row and line mixed database storage method | |
| CN102254024A (en) | Mass data processing system and method | |
| CN104618361B (en) | A kind of network flow data method for reordering | |
| CN104573065A (en) | Report display engine based on metadata | |
| CN104468752A (en) | Method and system for increasing utilization rate of cloud computing resources | |
| CN106777387A (en) | A kind of Internet of Things big data access method based on HBase | |
| CN110781184A (en) | Data table construction method, device, equipment and storage medium | |
| CN103685589A (en) | Binary coding-based domain name system (DNS) data compression and decompression methods and systems | |
| CN103885887A (en) | User data storage method, reading method and system | |
| CN108268529A (en) | It is a kind of that the data summarization method and system dispatched with multi engine are abstracted based on business | |
| CN105608126A (en) | Method and apparatus for establishing secondary indexes for massive databases | |
| CN105740410A (en) | Data statistics method based on Hbase secondary index | |
| CN104615684A (en) | Mass data communication concurrent processing method and system | |
| CN107491549A (en) | A kind of data processing method and system | |
| CN105515997A (en) | BF_TCAM (Bloom Filter-Ternary Content Addressable Memory)-based high-efficiency range matching method for realizing zero range expansion | |
| CN107766408A (en) | The storage method of audit log | |
| CN104714898B (en) | A kind of distribution method and device of Cache |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180306 |