CN109344137A - A kind of log storing method and system - Google Patents
A kind of log storing method and system Download PDFInfo
- Publication number
- CN109344137A CN109344137A CN201811173950.9A CN201811173950A CN109344137A CN 109344137 A CN109344137 A CN 109344137A CN 201811173950 A CN201811173950 A CN 201811173950A CN 109344137 A CN109344137 A CN 109344137A
- Authority
- CN
- China
- Prior art keywords
- data
- log
- storage
- storing method
- log storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of log storing method and system, the log storing method includes: after receiving daily record data, and program turn-on data receiving thread carries out data transmission；It receives the data of transmission and carries out queue processing；Queueization treated data are called, are stored in the form that multinode shunts.Log storing method of the present invention facilitates program to call and handle daily record data, reduces data packetloss, and enhancing service high availability realizes efficient data storage and extremely strong extension.
Description
Technical field
The present invention relates to data processing field, especially a kind of log storing method and system.
Background technique
Day aims at having very important effect in complete information system.It can recorde all rows caused by system
For, and expressed according to certain specification.It is that system carries out misarrangement, optimization that the information that log system is recorded, which can be used, in we
The performance of system, or the behavior according to these information adjustment system.In security fields, the critical role of log more so, can be with
Say it is one of most important tool of security audit aspect.So daily record data storage, which is recalled, to be particularly important.
Existing log memory technology, is generally divided into file system storage and database storage stores two parts.Log analysis
Daily record data packet is received by particular port, the data interactions such as XML, JSON format is then converted to and is called to foreground, it is this to belong to
The unmarked type storage of the plain text of file system；Database purchase is the storage of single type database, usually selection relationship type
Database reinforces recall precision by establishing multiple indexes in the retrieval of data.
The prior art has the following insufficient: log original text is not stored.In certain circumstances, daily record data contains easily quilt
Attacker utilizes, and by the forms such as distorting, forge, log server is caused to report by mistake, and attack record is erased, and allows the behaviour of attacker
Making behavior can seek without mark.Even if having the condition of log storage, in the case where possessing server permission, log original text can also be straight
It connects and is modified, cannot reach that data are credible, safely controllable purpose.
The daily record data being collected into is not stored with structuring, causes classification chaotic, text retrieval speed is slow.
Database type selecting is inappropriate.Database purchase is the storage of single type database, usually selection relational data
Library, and relevant database packet loss in the case where big data quantity is higher, it is slower for the retrieval rate of mass data, pass through
Multiple indexes are established to reinforce recall precision, the performance of server will be sacrificed.In addition, when being related to mass data association,
Need to use multilist conjunctive query, occupancy resource is more, and Data Sheet Design is complicated, and scalability is not strong.
Data packetloss is be easy to cause in the peak traffic phase.Due to the threaded design problem of program, when peak traffic period, number
According to a large amount of read-writes, the Caton of loss or reading data that data will be caused to be written.
Summary of the invention
The technical problem to be solved by the invention is to provide a kind of log storing method and systems.The log storage
Method and system facilitate the calling and data processing of program, reduce data packetloss, and enhancing service high availability realizes efficient
Data storage and extremely strong extension.
Above-mentioned technical problem to be solved by this invention, is resolved by following technical solution:
A kind of log storing method characterized by comprising
After receiving daily record data, program turn-on data receiving thread carries out data transmission；
It receives the data of transmission and carries out queue processing；
Queueization treated data are called, are stored in the form that multinode shunts.
Preferably, Message Queuingization processing, using daily record data source as the production source of message, persistence is extremely
Local disk, then called by Service controll.The serious forgiveness of service can be greatly improved in the method, the packet loss of data is reduced, in business
Peak period remain to ensure that user data is not lost, and due to the principle of message-oriented middleware, be not take up redundant system resource.
Preferably, after receiving daily record data, before program turn-on data receiving thread, program is open
Listening port monitors the inflow of daily record data.
Preferably, the turn-on data receiving thread, is carried out by thread pool management.When receiving daily record data, program
A data receiving thread is opened, by judging that the connection number of data channel increases Thread Count one by one.It is managed, is increased using thread pool
The communication of strong cross-thread, reduces the packet loss of data, and the full dose for reaching data receives.
Preferably, the queueization processing further includes that single issued transaction is decoupled into unified collect of message-oriented middleware to adjust
Match.When the peak traffic phase doing big data read-write interaction, message-oriented middleware service is quoted, data are done into queueization processing, are reduced
Single issued transaction is decoupled into unified collect of message-oriented middleware and deployed, then distinguished by concrete operations affairs by the coupling of business
It subscribes to and calls, enhancing service high availability.
Preferably, the log storage includes: that log is locally stored and log storage.
It is further preferred that the log is locally stored, it is to carry out this in the form of tree-shaped multinode
The storage of ground file.Using IP as root node, it is by date secondary nodes, finally using hourage as leaf node, is stored as txt
Text.
It is further preferred that the log storage, the method combined using NoSQL and cache database.Detailed process
Are as follows: the data received according to timestamps ordering are inserted into MongoDB, according to the type of assets and the type of log etc. into
Row is distinguished and is stored in Collection, and using the form of multinest Document, the data of same type are directly linked
To single table, the efficiency of data retrieval is improved, reduces the number of program pass tables of data, accelerates retrieval rate.
It is further preferred that the NoSQL method has used MongoDB Document image analysis.
It is further preferred that the MongoDB Document image analysis, provides efficient data storage and extremely strong expansion
Malleability is based on key-value, no coupling.
It is further preferred that the cache database method data buffer storage processing on, using the timing of Redis database
High frequency low frequency is accessed data separating by caching, and the data of reasonable distribution memory and CPU use, high frequency read-write do data
Pretreatment enhances user experience by dynamically dividing the technology of table.
Preferably, the tree-shaped multinode storage, is to be identified log flow according to separate sources, sectional lists of classifying
It deposits in system, is stored with formation " bibliographic structure tree ".With subsequent routine call is facilitated, be conducive to improve retrieval speed
Degree.
It is further preferred that the bibliographic structure tree improves the accurate of inquiry verification by the way of Multi-level traversal tree
Property and scalability.Can be as needed, define the mark of second level catalogue.
Preferably, the NoSQL can do data downloading and data sharing.Specifically, foreground sends message constructing request,
Resource id and device identification and timestamp are provided, mixed and disorderly data classification is downloaded into export, server provides multi-platform number
According to interface
The present invention also provides a kind of log storage systems comprising:
Data transmission unit, the data receipt unit are counted for opening a data receiving thread after receiving log
According to transmission；
Data processing unit, the data that the data processing unit is used to be collected into do queueization processing；
Data storage cell, the data storage cell are carried out for calling treated data in the form that multinode shunts
Storage.
The beneficial effects of the present invention are: storing using tree-shaped node type, be conducive to the calling and data processing of program, in full
According to aggregate statistics, data retrieval etc..Peak period data are handled using the mode of message queue, reduce data packetloss, enhancing clothes
Business High Availabitity.Non-relational database and data cached technology are selected, efficient data storage and extremely strong scalability are provided.In full
The combination of card technology is stored and deposited, ensures that data are credible, it is safely controllable.
Detailed description of the invention
Present invention will be further explained below with reference to the attached drawings and examples.
Fig. 1 is log Stored Procedure schematic diagram of the present invention；
Fig. 2 is log memory system architecture figure of the present invention.
Specific embodiment
In conjunction with the accompanying drawings, the present invention is further explained in detail.These attached drawings are simplified schematic diagram, only with
Illustration illustrates basic structure of the invention, therefore it only shows the composition relevant to the invention.
As shown in Figure 1, the present invention provides a kind of log storing method, the method specifically:
101. after receiving daily record data, program turn-on data receiving thread carries out data transmission；
102. receiving the data of transmission and carrying out queue processing；
103. calling queueization treated data, stored in the form that multinode shunts.
Step 101, specifically, program turn-on data receiving thread carries out data transmission after receiving daily record data.Data
The open listening port of program, monitors the inflow of daily record data before receiving
Step 102, the data received are subjected to queue processing, specifically, when big data read-write interaction is done in peak period, then
Message-oriented middleware service is quoted, the transaction of data sheet one is decoupled into the unified collection of message-oriented middleware by service message queue
Allotment, then calling is subscribed to respectively by specifically operating affairs.
Step 103, specifically, the storage of data is divided into, log is locally stored and log is put in storage two kinds of forms, and log is local
Storage is in such a way that tree-shaped multinode local file stores using IP as root node, is by date secondary nodes, finally with small
When number be leaf node, be stored as txt text such as: 172.16.1.1:2018-08-07:12.txt.
Log storage, the data received according to timestamps ordering are inserted into MongoDB, according to the type of assets and
The type etc. of log, which distinguishes, to be stored in Collection, using the form of multinest Document, same type
Data be directly linked to single table.
NoSQL can provide data downloading and data sharing, and foreground sends message constructing request, provides resource id and equipment mark
Mixed and disorderly data classification is downloaded export by knowledge and timestamp, and server provides multi-platform data-interface, as: cloud platform
Ssh, sftp, the message subscribing etc. of local host.
As shown in Fig. 2, the present invention provides a kind of log storage system, the system specifically:
201. data transmission units, the data receipt unit open a data receiving thread after being used to receive log, into
The transmission of row data；
202. data processing units, the data that the data processing unit is used to be collected into do queueization processing；
203. data storage cells, the data storage cell is for calling treated data, the form that is shunted with multinode
It is stored.
Taking the above-mentioned ideal embodiment according to the present invention as inspiration, through the above description, relevant staff completely may be used
Without departing from the scope of the technological thought of the present invention', to carry out various changes and amendments.The technical model of this invention
It encloses and is not limited to the contents of the specification, it is necessary to which the technical scope thereof is determined according to the scope of the claim.
Claims (10)
1. a kind of log storing method characterized by comprising
After receiving daily record data, program turn-on data receiving thread carries out data transmission；
It receives the data of transmission and carries out queue processing；
Queueization treated data are called, are stored in the form that multinode shunts.
2. log storing method according to claim 1, which is characterized in that after receiving daily record data, program turn-on data
Before receiving thread, the open listening port of program monitors the inflow of daily record data.
3. log storing method according to claim 1, which is characterized in that the turn-on data receiving thread passes through line
Cheng Chi management carries out.
4. log storing method according to claim 1, which is characterized in that the queueization processing includes by single thing
Business processing is decoupled into unified collect of message-oriented middleware and deploys.
5. log storing method according to claim 1, which is characterized in that the log storage includes: that log is locally deposited
Storage and log storage.
6. log storing method according to claim 5, which is characterized in that the log is locally stored, and is with tree-shaped more
Joint form carries out local file storage.
7. log storing method according to claim 5, which is characterized in that the log storage, using NoSQL and caching
The method that database combines.
8. log storing method according to claim 6, which is characterized in that the tree-shaped multinode storage, is by log
Flow is identified according to separate sources, and classification sectional lists is deposited in system, is stored with formation " bibliographic structure tree ".
9. log collection method according to claim 7, which is characterized in that the NoSQL can carry out data downloading and data
It is shared.
10. a kind of log storage system characterized by comprising
Data transmission unit, the data receipt unit is for opening a number after receiving log
According to receiving thread, carry out data transmission；
Data processing unit, the data processing unit are used to receive the data of transmission and progress
Queueization processing；
Data storage cell, the data storage cell treated for calling queueization number
According to, with multinode shunt form stored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811173950.9A CN109344137A (en) | 2018-10-09 | 2018-10-09 | A kind of log storing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811173950.9A CN109344137A (en) | 2018-10-09 | 2018-10-09 | A kind of log storing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109344137A true CN109344137A (en) | 2019-02-15 |
Family
ID=65308775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811173950.9A Pending CN109344137A (en) | 2018-10-09 | 2018-10-09 | A kind of log storing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344137A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222118A (en) * | 2019-05-23 | 2019-09-10 | 上海易点时空网络有限公司 | Asynchronous data processing method and processing device based on queue |
| CN110569112A (en) * | 2019-09-12 | 2019-12-13 | 华云超融合科技有限公司 | Log data writing method and object storage daemon device |
| CN110716908A (en) * | 2019-09-29 | 2020-01-21 | 捷开通讯(深圳)有限公司 | Log information writing method, system, storage medium and mobile terminal |
| WO2021258696A1 (en) * | 2020-06-24 | 2021-12-30 | 珠海格力电器股份有限公司 | Communication data processing method and apparatus, computer device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN106571960A (en) * | 2016-11-03 | 2017-04-19 | 北京农信互联科技有限公司 | Log collection and management system and log collection and management method |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
| CN108519942A (en) * | 2018-04-11 | 2018-09-11 | 车巴达（苏州）网络科技有限公司 | A kind of Log Analysis System |
-
2018
- 2018-10-09 CN CN201811173950.9A patent/CN109344137A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110040811A1 (en) * | 2009-08-17 | 2011-02-17 | International Business Machines Corporation | Distributed file system logging |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN106571960A (en) * | 2016-11-03 | 2017-04-19 | 北京农信互联科技有限公司 | Log collection and management system and log collection and management method |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
| CN108519942A (en) * | 2018-04-11 | 2018-09-11 | 车巴达（苏州）网络科技有限公司 | A kind of Log Analysis System |
Non-Patent Citations (1)
| Title |
|---|
| DREAM_LIXIANG: "自定义log4j日志文件命名规则", 《HTTPS://BLOG.CSDN.NET/DREAM_LIXIANG/ARTICLE/DETAILS/55264203》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222118A (en) * | 2019-05-23 | 2019-09-10 | 上海易点时空网络有限公司 | Asynchronous data processing method and processing device based on queue |
| CN110222118B (en) * | 2019-05-23 | 2022-04-05 | 上海易点时空网络有限公司 | Asynchronous data processing method and device based on queue |
| CN110569112A (en) * | 2019-09-12 | 2019-12-13 | 华云超融合科技有限公司 | Log data writing method and object storage daemon device |
| CN110569112B (en) * | 2019-09-12 | 2022-04-08 | 江苏安超云软件有限公司 | Log data writing method and object storage daemon device |
| CN110716908A (en) * | 2019-09-29 | 2020-01-21 | 捷开通讯(深圳)有限公司 | Log information writing method, system, storage medium and mobile terminal |
| WO2021258696A1 (en) * | 2020-06-24 | 2021-12-30 | 珠海格力电器股份有限公司 | Communication data processing method and apparatus, computer device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344137A (en) | A kind of log storing method and system | |
| CN111124679B (en) | Multi-source heterogeneous mass data-oriented time-limited automatic processing method | |
| WO2020001108A1 (en) | Block chain-based data processing method and device | |
| CN103559217B (en) | A kind of massive multicast data towards isomeric data storehouse warehouse-in implementation method | |
| CN107818150B (en) | Log auditing method and device | |
| US9621572B2 (en) | Storage appliance and threat indicator query framework | |
| CN103930887B (en) | The inquiry stored using raw column data collects generation | |
| CN108647361A (en) | A kind of date storage method, apparatus and system based on block chain | |
| CN112732811A (en) | Data open platform | |
| US20130191523A1 (en) | Real-time analytics for large data sets | |
| US11574242B1 (en) | Guided workflows for machine learning-based data analyses | |
| CN103064731A (en) | Device and method for improving message queue system performance | |
| US7734785B2 (en) | On demand message based financial network integration middleware | |
| CN103327124B (en) | A kind of method of file synchronization, device, client, server end and equipment | |
| CN104067281A (en) | Clustering event data by multiple time dimensions | |
| US20200401562A1 (en) | Parallel processing of filtered transaction logs | |
| CN103618652A (en) | Audit and depth analysis system and audit and depth analysis method of business data | |
| CN107800808A (en) | A kind of data-storage system based on Hadoop framework | |
| US12003595B2 (en) | Aggregated service status reporter | |
| US20160127380A1 (en) | System and method to verify predetermined actions by a computer on a network | |
| CN102332004B (en) | Data processing method and system for managing mass data | |
| US11450419B1 (en) | Medication security and healthcare privacy systems | |
| US11573971B1 (en) | Search and data analysis collaboration system | |
| CN108182263A (en) | A kind of date storage method of data center's total management system | |
| WO2017000592A1 (en) | Data processing method, apparatus and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190215 |