CN106254086A - Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform - Google Patents

Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform Download PDF

Info

Publication number
CN106254086A
CN106254086A CN201510308783.4A CN201510308783A CN106254086A CN 106254086 A CN106254086 A CN 106254086A CN 201510308783 A CN201510308783 A CN 201510308783A CN 106254086 A CN106254086 A CN 106254086A
Authority
CN
China
Prior art keywords
log
daily record
cloud
module
search engine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510308783.4A
Other languages
Chinese (zh)
Inventor
董昶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Dart Technology Co Ltd
Original Assignee
Chongqing Dart Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Dart Technology Co Ltd filed Critical Chongqing Dart Technology Co Ltd
Priority to CN201510308783.4A priority Critical patent/CN106254086A/en
Publication of CN106254086A publication Critical patent/CN106254086A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/02—Standardisation; Integration
    • H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425—Traffic logging, e.g. anomaly detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present invention provides a kind of cloud daily record centralized management, analyzes and monitoring method；Including: log acquisition module (log shipper), log analysis module (log parser), log query monitoring and alarm module (log virtualization).Log acquisition module can synchronize any operating system, the journal file of any application program, such as Linux, the system-level daily record of Unix, Windows, tomcat, apache, mysql, oracle, application program level logs.Log analysis module can resolve the journal file of any form, filtering useless information, retains and analyzes useful information.Log query monitoring module provides visualized graph interface, Real-time Alarm.

Description

Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform
Technical field
The present invention relates to cloud computing information security field, particularly relate to the method and system of a kind of cloud log analysis.
Background technology
One of challenge that the cloud epoch are maximum is massive logs file management, analyzes and monitors.Journal file dispersion is deposited, and quantity is many, can directly consult journal file retention cycle short, and pole is not easy to O&M；Journal format is inconsistent, can be readability too low；During enquiry fee or time-out, inefficiency；The relatedness of correlation log is low, it is impossible to clearly position；Substantial amounts of daily record cannot be added up, it is impossible to analyzes business accurately；
Journal file have recorded the critical event of system and application program, by daily record it is recognized that system and the running status of application program, access information and the error message to server end. carry out the journal file in each virtual machine resolving and can reach to monitor the most normal purpose of the operation conditions of server in virtual machine.Carry out the journal file of the application program in each virtual machine resolving and can reach to monitor the most normal purpose of the operation conditions of application program in virtual machine.
But so how the daily record data of magnanimity can statistical analysis be faster and better out the difficult problem that manager faces, at present industry have SME SOC SM the log management product such as SEM, wherein it is directed to log collection and analyzes module, the daily record major part bottleneck of these products is all on Analysis server IO, the slowest to the processing speed of massive logs, it is not the most the highest for causing collection analysis whole efficiency, this can affect the system (front end applications) of upstream usage log undoubtedly, and Consumer's Experience is less than promptness.Meanwhile, long massive logs processes, and is in IO peak value for a long time, even results in log collection and analyzes the crash of system, the deadlock of log collection Analysis server, has a strong impact on the use of system.
Thus, it is necessary to propose a kind of new technology, Analysis server can be made while quickly processing massive logs, to reduce the burden of Analysis server, promote overall log collection analytical performance, and ensure that in data base, analysis result upgrades in time.
Summary of the invention
The invention aims to overcome the defect of prior art, the method and system of a kind of cloud log analysis are provided, log analysis efficiency and real-time are all guaranteed, simultaneously scan for analysis result in engine and can present to user in visual mode, and system can be automatically to abnormal event alarming.
The method of a kind of cloud log analysis that the present invention provides, idiographic flow is: be first that all of cloud journal file is put together to daily record center by daily record center by encrypted transmission；Next to that log analysis is processed by logging host, and analysis result is sent to data center；It is finally to be stored in big data search engine after the log analysis result of each node is arranged merging by data center.
In addition, present invention also offers the system that a kind of large log is analyzed, this system includes three big modules, one is daily record scheduled transmission module, all of cloud journal file is put together to daily record center by encrypted transmission, two is log processing module, will put together Source log and is analyzed and processes, and transmit the result in big data search engine；Three are daily record visualization and report holds up console module, and analysis result visualization panel in big data search engine is presented to user, simultaneously to anomalous event automatic alarm.
The beneficial effect that technical solution of the present invention is brought:
It is not only able to alleviate daily record center resources consumption by the present invention, it is to avoid affect log collection, it is ensured that log collection quality, and log analysis efficiency and real-time can be improved, it is ensured that in data base, analysis result can upgrade in time, simultaneously to anomalous event automatic alarm.
Accompanying drawing explanation
Fig. 1 is the Organization Chart of this invention
Fig. 2 is visualized graph interface figure (one)
Fig. 3 is visualized graph interface figure (two).

Claims (5)

1. the system of a cloud log analysis, it is characterised in that this system includes three big modules, and is daily record scheduled transmission module, institute Some cloud journal files are put together to daily record center by encrypted transmission, and two is log processing module, will put together source day Will is analyzed and processes, and transmits the result in big data search engine: three is daily record visualization and alarm platform module, Analysis result visualization panel in big data search engine is presented to user, simultaneously to anomalous event automatic alarm.
System the most according to claim 2, it is characterised in that in the minds of in daily record, job scheduling module can determine whether implement plan, And it is transferred to daily record center according to ssl protocol to after log blocks encapsulation encryption.
System the most according to claim 2, it is characterised in that log analysis result arranges after merging and is stored in big data search engine In.
System the most according to claim 2, it is characterised in that daily record visualization and alarm platform module are by big data search engine Middle analysis result visualization panel presents to user, simultaneously to anomalous event automatic alarm.
5. the method for a cloud log analysis, it is characterised in that the idiographic flow of the method is: be first that daily record center is by all of cloud Journal file is put together to daily record center by encrypted transmission；Next to that log analysis is processed by logging host, and will analyze Result is sent to data center；It is finally to be stored in big data after the log analysis result of each node is arranged merging by data center In search engine.
CN201510308783.4A 2015-06-04 2015-06-04 Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform Pending CN106254086A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510308783.4A CN106254086A (en) 2015-06-04 2015-06-04 Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510308783.4A CN106254086A (en) 2015-06-04 2015-06-04 Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform

Publications (1)

Publication Number Publication Date
CN106254086A true CN106254086A (en) 2016-12-21

Family

ID=57626385

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510308783.4A Pending CN106254086A (en) 2015-06-04 2015-06-04 Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform

Country Status (1)

Country Link
CN (1) CN106254086A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107276996A (en) * 2017-06-06 2017-10-20 广州华多网络科技有限公司 The transmission method and system of a kind of journal file
CN109167672A (en) * 2018-07-13 2019-01-08 腾讯科技（深圳）有限公司 One kind returning source location of mistake method, apparatus, storage medium and system
CN109800140A (en) * 2018-12-27 2019-05-24 北京奇安信科技有限公司 Method, apparatus, equipment and the medium of service alarm event analysis of causes

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103227730A (en) * 2013-04-19 2013-07-31 蓝盾信息安全技术股份有限公司 Method and system for analyzing large log
CN103455636A (en) * 2013-09-27 2013-12-18 浪潮齐鲁软件产业有限公司 Automatic capturing and intelligent analyzing method based on Internet tax data
CN103838867A (en) * 2014-03-20 2014-06-04 网宿科技股份有限公司 Log processing method and device
CN104104734A (en) * 2014-08-04 2014-10-15 浪潮(北京)电子信息产业有限公司 Log analysis method and device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103227730A (en) * 2013-04-19 2013-07-31 蓝盾信息安全技术股份有限公司 Method and system for analyzing large log
CN103455636A (en) * 2013-09-27 2013-12-18 浪潮齐鲁软件产业有限公司 Automatic capturing and intelligent analyzing method based on Internet tax data
CN103838867A (en) * 2014-03-20 2014-06-04 网宿科技股份有限公司 Log processing method and device
CN104104734A (en) * 2014-08-04 2014-10-15 浪潮(北京)电子信息产业有限公司 Log analysis method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
顾君忠;陈民: "基于大数据分析的智能搜索引擎", 《软件产业与工程》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107276996A (en) * 2017-06-06 2017-10-20 广州华多网络科技有限公司 The transmission method and system of a kind of journal file
CN109167672A (en) * 2018-07-13 2019-01-08 腾讯科技（深圳）有限公司 One kind returning source location of mistake method, apparatus, storage medium and system
CN109167672B (en) * 2018-07-13 2020-07-10 腾讯科技（深圳）有限公司 Return source error positioning method, device, storage medium and system
CN109800140A (en) * 2018-12-27 2019-05-24 北京奇安信科技有限公司 Method, apparatus, equipment and the medium of service alarm event analysis of causes

Similar Documents

Publication Publication Date Title
CN104022904B (en) Distributed computer room information technoloy equipment management platform
Wu et al. A fog computing-based framework for process monitoring and prognosis in cyber-manufacturing
CN107689953B (en) Multi-tenant cloud computing-oriented container security monitoring method and system
US9485317B2 (en) Method and system for monitoring execution of user request in distributed system
CN105631026A (en) Security data analysis system
CN105740142B (en) SSR centralized management platform pressure test management system
CN111756801A (en) Method and system for processing intelligent manufacturing big data
CN112702190A (en) Regional alarm message pushing method and system based on message queue
CN102135932A (en) Monitoring system and monitoring method thereof
CN103475544A (en) Service monitoring method based on cloud resource monitoring platform
CN105162632A (en) Automatic processing system for server cluster failures
CN112529528B (en) Workflow monitoring and warning method, device and system based on big data flow calculation
CN105071954A (en) Resource pool fault diagnosis and positioning processing method based on probe technology
CN106254086A (en) Cloud daily record is managed concentratedly, analyzes, monitoring and alarm platform
Monajjemi et al. Drums: A middleware-aware distributed robot monitoring system
CN104750041B (en) Abnormality monitoring method and system in WinCE system
CN103401722A (en) System and method for monitoring software resources
CN103227730A (en) Method and system for analyzing large log
Liu et al. System anomaly detection in distributed systems through MapReduce-Based log analysis
CN104899078A (en) Auditing system and method in virtual machine environment
CN112650706A (en) Method for realizing high situation perception capability under big data technology system
CN116862194A (en) Analysis, operation and detection system based on power grid business data
CN105573885A (en) Method and device for monitoring and counting bottom hardware behaviours
CN103744765A (en) Disk access request monitoring system and method in virtual environment
CN110602153A (en) Software big data safe storage verification system

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20161221

WD01 Invention patent application deemed withdrawn after publication