CN111581060B - Prometaus-based log alarm system, method and related equipment - Google Patents
Prometaus-based log alarm system, method and related equipment Download PDFInfo
- Publication number
- CN111581060B CN111581060B CN202010392449.2A CN202010392449A CN111581060B CN 111581060 B CN111581060 B CN 111581060B CN 202010392449 A CN202010392449 A CN 202010392449A CN 111581060 B CN111581060 B CN 111581060B
- Authority
- CN
- China
- Prior art keywords
- log data
- log
- unit
- periodicity
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 32
- 238000012544 monitoring process Methods 0.000 claims abstract description 58
- 230000000737 periodic effect Effects 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 5
- 239000000523 sample Substances 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 12
- 229920001971 elastomer Polymers 0.000 description 6
- 239000000806 elastomer Substances 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application discloses a log alarm system based on Prometaheus, which comprises the following components; the system comprises a log module, a monitoring module and an elastic search module; the monitoring module comprises a data acquisition unit, a Prometaus service unit and an alarm unit; the log module is used for acquiring keywords and sending the keywords to the data acquisition unit; the data acquisition unit is used for periodically searching the elastic search module according to the keywords so as to obtain first log data corresponding to the keywords, and is also used for providing http service for transmitting the first log data; the Prometaus service unit is used for acquiring first log data through an http service and storing the first log data; the alarm unit is used for alarming if the quantity of the first log data in the Prometaus service unit is larger than a first threshold value.
Description
Technical Field
The present application relates to the field of automatic alarms, and in particular, to a system, method and related device for a Prometaus-based log alarm.
Background
The elastiscearch is a Lucene-based search server. It provides a distributed multi-user capable full text search engine based on RESTful web interface. The elastomer search was developed in Java and released as open source under Apache license terms, is a currently popular enterprise-class search engine. The cloud computing system is designed to be used in cloud computing, can achieve real-time searching, is stable, reliable and quick, and is convenient to install and use.
ElastAlert is a log alert framework based on the elastiscearch system. Its principle of operation is to combine the elastomer search with components of rule type and alert type, periodically query the elastomer search and pass the data to the rule type, which determines when a match is found. When a match occurs, elastAlert will be given one or more alarms that take action based on the match. Prometheus is a set of open source monitoring, alarm, time series database combinations. Prometheus may require very little external reliance, is ultra-simple to install and use, and has very many system integration such as: docker HAProxy Nginx JMX, etc. With the development, more and more companies and organizations accept the use of promethaus.
If a company has two alarm systems, namely ElastAlert and promethaus, the problems of repeated alarm, tedious management and the like, which are unavoidable, occur.
Disclosure of Invention
The application provides a log alarming system, a log alarming method and related equipment based on Prometheus, which can combine an elastic search with the Prometheus to reduce management burden.
The first aspect of the application provides a Prometaheus-based log alert system.
The system comprises; the system comprises a log module, a monitoring module and an elastic search module;
the monitoring module comprises a data acquisition unit, a Prometaus service unit and an alarm unit;
the log module is used for acquiring keywords and sending the keywords to the data acquisition unit;
the data acquisition unit is used for periodically searching the elastic search module according to the keywords so as to obtain first log data corresponding to the keywords, and is also used for providing http service for transmitting the first log data;
the Prometaus service unit is used for acquiring first log data through an http service and storing the first log data;
the alarm unit is used for alarming if the quantity of the first log data in the Prometaus service unit is larger than a first threshold value.
Optionally, the data acquisition unit comprises a data acquisition subunit, a data format conversion subunit and an http service transmission subunit;
the data acquisition unit is specifically used for searching the elastic search module according to the keywords through the data acquisition subunit in a first periodicity mode so as to obtain first log data in a first format corresponding to the keywords;
the data format conversion subunit is used for converting the format of the first log data to obtain the first log data adapting to the second format of the Prometaheus service unit;
the data acquisition unit is specifically configured to provide an http service for transmitting the first log data in the second format through the http service transmission subunit.
Optionally, the promethaus service unit is specifically configured to obtain and store a first amount of first log data through an http service in a second period;
the alarm unit is specifically configured to alarm if the sum of the first number is greater than a first threshold value within a duration n×m, where M is the duration of the first periodicity, and the duration of the second periodicity is n×m, where N is an integer greater than 1.
Optionally, the log module is further configured to obtain period information of a second periodicity, where the period information includes N, and send the period information to the data acquisition unit;
the data acquisition unit is used for providing http service for transmitting the first log data according to the period information and with N multiplied by M as a period and a second period.
The second aspect of the application provides a Prometaheus-based log alert method.
The method comprises the following steps: the monitoring module acquires keywords;
the monitoring module periodically searches the elastic search module according to the keywords to obtain first log data corresponding to the keywords;
the monitoring module provides an http service for transmitting the first log data;
the monitoring module acquires first log data through an http service and stores the first log data;
if the number of the first log data in the Prometheus service unit is larger than a first threshold value, the monitoring module alarms.
Optionally, the monitoring module retrieves the elastic search module according to the keyword, and the first periodically to obtain the first log data corresponding to the keyword includes:
the monitoring module is used for searching the elastic search module according to the keywords in a first periodicity to obtain first log data in a first format corresponding to the keywords;
the method further comprises the steps of:
the monitoring module converts the format of the first log data to obtain the first log data adapting to the second format of the Prometaus service unit;
the monitoring module obtaining the first log data through the http service comprises:
and acquiring the first log data in the second format through an http service.
Optionally, the monitoring module providing an http service for transmitting the first log data includes:
the monitoring module is used for providing http service for transmitting the first log data in the first quantity in a second periodic manner;
if the number of the first log data in the Prometheus service unit is greater than the first threshold, the monitoring module alarms including:
if the sum of the first number is greater than a first threshold value within the duration N×M, the monitoring module alarms, M is a first periodic period duration, and N is an integer greater than 1.
Optionally, the method further comprises: the monitoring module acquires second periodic period information, wherein the period information comprises N;
the monitoring module takes N multiplied by M as a period according to the period information, and the second period provides http service for transmitting the first log data.
A third aspect of the present application provides a promethaus-based log alert device.
The device comprises: a first acquisition unit configured to acquire a keyword;
a second acquisition unit configured to periodically search the elastic search module according to the keyword, so as to obtain first log data corresponding to the keyword;
a processing unit for providing an http service for transmitting the first log data;
the third acquisition unit is used for acquiring the first log data through an http service;
and the alarm unit is used for alarming if the number of the first log data in the third acquisition unit is larger than a first threshold value.
Optionally, the second obtaining unit is specifically configured to retrieve the elastic search module according to the keyword in a first periodicity, so as to obtain first log data in a first format corresponding to the keyword.
The apparatus further comprises: and the conversion unit is used for converting the format of the first log data to obtain the first log data adapting to the second format of the Prometaus service unit.
The third obtaining unit is specifically configured to obtain the first log data in the second format through an http service.
Optionally, the processing unit is specifically configured to provide the http service for transmitting the first amount of the first log data in the second periodicity.
The alarm unit is specifically configured to alarm by the monitoring module if the sum of the first number is greater than a first threshold value within a duration n×m, where M is a first periodic period duration, and N is an integer greater than 1.
Optionally, the first obtaining unit is further configured to obtain period information of the second periodicity;
the processing unit is specifically configured to provide an http service for transmitting the first log data with a period of nxm according to the period information and a second period.
A fourth aspect of the present application provides a promethaus-based log alert device.
The device comprises: a memory, a processor;
wherein the memory is used for storing programs; the processor is configured to execute a program in memory comprising performing the method according to any of the second aspects above.
A fifth aspect of the embodiments of the present application provides a computer storage medium, comprising:
the computer storage medium has instructions stored therein, which when executed on a computer, cause the computer to perform the method according to any of the second aspects above.
A sixth aspect of the embodiments of the present application provides a computer program product, characterized in that the computer program product, when executed on a computer, causes the computer to perform the method according to any one of the second aspects above.
From the above technical scheme, the application has the following advantages: the data acquisition unit may retrieve the elastic search module to obtain first log data corresponding to the keyword, and provide an http service for transmitting the first log data, and the promethaus service unit may obtain the first log data through the http service. Thus, the Prometheus-based log alert system has the advantages of both elastic search and Prometheus, and can be implemented under a Prometheus-based alert framework, thus reducing the management burden.
Drawings
FIG. 1A is a schematic diagram of a Prometaus-based log alert system according to an embodiment of the present application;
FIG. 1B is a schematic diagram of another framework of a Prometaus-based log alert system in an embodiment of the present application;
FIG. 2 is a schematic flow chart of a Prometaus-based log alert method according to an embodiment of the present application;
FIG. 3 is a schematic flow chart of a Prometaus-based log alert method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a structure of a Prometaus-based log alert device according to an embodiment of the present application;
FIG. 5 is a schematic diagram of another embodiment of a Prometaus-based log alert device according to the present application;
fig. 6 is a schematic structural diagram of a promethaus-based log alert device according to an embodiment of the present application.
Detailed Description
The application provides a log alarming system, a log alarming method and related equipment based on Prometheus, which can combine an elastic search with the Prometheus to reduce management burden.
Cloud protogenesis is a continuously developing technical direction, and definition of the cloud protogenesis is also continuously evolving in the future. In a narrow sense, cloud native technology including containers, service grids, micro services, and services is a new way to build applications. The method can well support Internet application, and also can deeply influence new computing architecture and new intelligent data application. The cloud protogenesis technology is beneficial to each organization to construct and operate the application which can be elastically expanded in novel dynamic environments such as public cloud, private cloud, mixed cloud and the like. Representative technologies for cloud proto include containers, service grids, micro-services, immutable infrastructure, and declarative APIs.
The elastiscearch is an open-source, distributed, RESTful interface full-text search engine built based on Lucene. The elastomer search is also a distributed document database in which each field is indexed data and can be searched, which can be extended to hundreds of servers storing and processing PB-level data, and can store, search and analyze a large amount of data in a short time.
Prometheus is a cloud native monitoring tool deployed on Kubernates for observing workloads. The method fills an important blank existing in the cloud primary world through comprehensive indexes and rich dashboards. Prometheus integrates mainstream data visualization tools like Grafana. The functions that Kubernates next put forward on expansion and monitoring will depend on promethaus, which makes promethaus a critical item in cloud native platform construction.
In the cloud primordial age, promethaus has almost become a de facto standard in the monitoring field. Prometaus is stored by a self-contained efficient time sequence database, so that a single Prometaus can efficiently process a large amount of data. There is also a friendly and powerful PromQL syntax that can be used to flexibly query various monitoring data and configure alarm rules. Meanwhile, the pull model index collection mode is widely adopted, and a great number of applications realize the metrics interface of Prometheus to expose each data index of Prometheus for collection. It is therefore desirable to combine the search and monitoring advantages of elastiscearch and Prometaus to create a monitoring framework system that includes both advantages. Because elastomer search does not adapt the metrics interface to expose its own data metrics for acquisition by Prometheus, a third party application is required to help elastomer search adapt Prometheus to obtain a Prometheus-based log alert system, method and related devices.
The Prometaus-based log alert system will be described below by way of specific embodiments.
Referring to fig. 1A, fig. 1A is a schematic diagram of a frame of a log alert system based on promethaus in an embodiment of the present application.
Fig. 1A includes a log module 101, a monitor module 102, and an elastic search module 103. The monitoring module 102 includes a data acquisition unit 104, a promethaus service unit 105, and an alarm unit 106.
The monitoring module 102 is specifically configured to monitor and alarm, and may include operating system layer monitoring, application layer monitoring, service layer monitoring, cloud container monitoring, log monitoring, etc., where in the embodiment of the present application, the main function of the monitoring module 102 is related to the content of log monitoring. Log monitoring can be understood as log alerting, and in practical applications, log data is typically saved in text form. While data saved in text form can be searched by means of an elastsearch.
The log module 101 is a piece of software used to collect and query log data. The log module 101 is configured to obtain a keyword, and send the keyword to the data acquisition unit 104.
The elastic search module 103 may be an elastic search server, and stores log data.
The data collection unit 104 is configured to first periodically search the elastic search module 103 according to the keyword to obtain the first log data, and provide an http service for transmitting the first log data, which may also be understood as creating a metrics interface for the promethaus service unit 105 to collect the first log data. The period duration of the first periodicity is preset. The data acquisition unit 104 may agree in advance with the elastic search module 103 with the period duration of the first periodicity, or the data acquisition unit 104 may use the period duration of the first periodicity as the period duration of the first periodicity according to the default period duration of the elastic search module 103.
The promethaus service unit 105 is a promethaus tool for obtaining first log data via an http service;
the alarm unit 106 is configured to alarm if the number of the first log data in the promethaus service unit 105 is greater than a first threshold.
The data acquisition unit 104 in the implementation of the present application may retrieve the elastic search module 103 and provide an http service for transmitting the first log data, and the promethaus service unit 105 may obtain the first log data through the http service. Thus, the Prometheus-based log alert system has the advantages of both elastic search and Prometheus, and can be implemented under a Prometheus-based alert framework, thus reducing the management burden.
Referring to fig. 1B, fig. 1B is another schematic diagram of a journal alarm system based on promethaus according to an embodiment of the present application.
Optionally, on the basis of fig. 1A, the data acquisition unit 104 comprises a data acquisition subunit 107, a data format conversion subunit 108 and an http service transmission subunit 109.
The data acquisition subunit 107 is configured to retrieve the elastic search module 103 according to the keyword, and to obtain a first number of first log data in a first format corresponding to the keyword. The format of the first log data can be adapted to the promethaus service unit 105 by performing format conversion on the first log data.
The data format conversion subunit 108 is configured to convert the format of the first log data to obtain the first log data in the second format.
Optionally, the first format is json format and the second format is meta format.
Optionally, the data acquisition unit 104 is specifically configured to provide the http service for transmitting the first log data periodically; the Prometaus service unit is specifically configured to obtain a first amount of first log data through an http service in a second periodic manner; the alarm unit is specifically configured to alarm if the sum of the first number is greater than a first threshold value within a duration n×m, where M is a first periodic period duration, and N is an integer greater than 1. Wherein, the alarm unit 106 alarms only when the first number of sums is greater than the first threshold value within the duration n×m. Therefore, by the second periodic acquisition of the first log data by the promethaus service unit 105, the number of times the first log data is acquired by the promethaus service unit 105 can be reduced, the efficiency of acquiring the first log data can be improved, and the energy consumption can be reduced.
Optionally, the log module 101 is further configured to obtain period information of the second periodicity, where the period information includes the N, and send the period information to the data acquisition unit. The data acquisition unit 104 is specifically configured to provide an http service for transmitting the first log data with a period of nxm according to the period information and a second period. The log module 101 faces to a user, and besides a keyword, the user can also add second periodic period information in the log module for controlling the period duration of the http service for transmitting the first log data provided by the data acquisition unit. Therefore, the periodic information and the keywords are put into the log module, so that the user can conveniently manage and modify the alarm system in the embodiment of the application, and the complexity of management and modification is reduced.
Optionally, the log module 101 is further configured to obtain an interval duration of a first periodicity and a third periodicity, where the duration of one period of the first periodicity is M, and the duration of one period of the third periodicity is also M; the log module 101 is further configured to send the interval duration to the data acquisition unit 104. The data acquisition unit 104 is specifically configured to retrieve the elastic search module 103 according to the keyword, and obtain the second amount of the first log data. The promethaus service unit 105 is further configured to obtain a second amount of the first log data via an http service; the alarm unit is also used for alarming if the second number is larger than the first threshold value.
The period duration of the third periodicity is the same as the period duration of the first periodicity. The data acquisition unit 104 starts at different time points, and the interval between the two start points is the interval duration. Using the keyword search elastic search module 103, two time-series log data can be generated. Even with the same key, the same period duration, the alarm results of log data of different timings may be different. For example, a first number of sums over 3 consecutive periods is greater than 50 and a second number of sums over 3 consecutive periods is less than 50, the alert unit 106 alerts the first number of sums, and the alert unit 106 does not alert the second number of sums. Therefore, the log module 101 acquires the interval duration, and the data acquisition unit 104 can acquire log data of two time sequences, so that the alarm probability under the same keyword is improved, and the alarm accuracy is improved.
Alternatively, the data acquisition unit 104 is obtained according to the SDK programming of promethaus. The data acquisition unit 104 needs to interface with the promethaus service unit 105, which is a promethaus tool. Thus by using the SDK of promethaus, the programming effort can be reduced.
The Prometaus-based log alert system in the embodiment of the present application is described above, and the Prometaus-based log alert method in the embodiment of the present application is described below.
Referring to fig. 2, fig. 2 is a flow chart of a log alert method based on promethaus in an embodiment of the present application.
In step 201, the device obtains a keyword.
The device has computing power, and is provided with a monitoring module, wherein the monitoring module can be software, and the device can be a notebook, a desktop computer, a server and the like. The device may obtain multiple keywords for log alerts for the multiple keywords, such as table one, which is a table of keywords obtained by the device. The embodiment of the application will be described by taking only one keyword as an example.
| Numbering device | Keyword(s) |
| 1 | Software X |
| 2 | Software M |
List one
In step 202, the device periodically retrieves the elastic search module based on the key to obtain a first amount of first log data.
Taking the above keyword software X as an example, please refer to table two, which is a schematic structural diagram of the first log data. It should be determined that the keywords, the time period and the log number in the second table are all embodied in specific numerical values for the purpose of detailed description, and in practical application, the keywords, the time period and the log number can be changed correspondingly according to practical needs. For example, the duration of the time period may be changed, the starting time of the time period may be changed, the number of logs may be changed, and the like. The first number is a certain number of logs in table two, e.g. 10.
| Keyword(s) | Time period | Log quantity |
| Software X | 15:00:15-15:00:30 | 10 |
| Software X | 15:00:30-15:00:45 | 15 |
| Software X | 15:00:45-15:00:60 | 5 |
| Software X | 15:00:60-15:01:15 | 6 |
Watch II
In step 203 the device provides an http service transmitting said first log data. The device is provided with a data acquisition unit which is developed based on the go language version sdk of promethaus, and standard http services can be provided to the outside by exposing one port.
In step 204, the device obtains first log data via an http service.
After the device provides an http service for transmitting the first log data, the device may acquire a first amount of the first log data through the http service to obtain time sequence data of the first log data. Referring to table three, the table three is the time sequence data of the first log data.
| Keyword(s) | Time period | Log quantity |
| Software X | 15:00:15 | 10 |
| Software X | 15:00:30 | 25 |
| Software X | 15:00:45 | 30 |
| Software X | 15:00:60 | 36 |
Watch III
In step 205, the device alerts if the sum of the first number of the plurality of time periods obtained through the http service is greater than a first threshold.
Taking the above table three as an example, if the sum of the first number of consecutive 2 time periods is greater than 20, an alarm is given. Table three, the first number of first time periods 15:00:15-15:00:30 is 10, the first number of second time periods 15:00:30-15:00:45 is 15, the sum of the first number of 2 time periods is 25, greater than 20, and thus the device alerts.
The device in the application can search the elastic search module and provide an http service for transmitting the first log data, and the first log data is acquired through the http service. Therefore, the Prometheus-based log alert method has the advantages of both the elastic search and the Prometheus, and can be implemented under an alert framework based on Prometheus, thus reducing the management burden.
Referring to fig. 3, fig. 3 is another flow chart of a log alert method based on promethaus in an embodiment of the present application.
In step 301, the device obtains a key, N, and an interval duration.
Wherein N is an integer greater than 1. The embodiment of the application is described with N being 2.
The interval duration is the difference between the starting time point of the first periodicity and the starting time point of the third periodicity. One period of the first periodicity is M, and one period of the second periodicity is nxm. The embodiment of the application is described with an interval duration of 10S.
In step 302, the device retrieves an elastesearc module for a first period based on the key, the first period being a period duration, to obtain a first amount of first log data. The device also retrieves the elastesearc module in a third periodicity based on the key to obtain a second amount of first log data. The period duration of the third period is the same as the period duration of the first period. The first number of first log data refers to the second table, the second number of first log data refers to the fourth table, and the number of logs in the fourth table is the second number.
| Keyword(s) | Time period | Log quantity |
| Software X | 15:00:05-15:00:20 | 8 |
| Software X | 15:00:20-15:00:35 | 9 |
| Software X | 15:00:35-15:00:50 | 10 |
| Software X | 15:00:50-15:01:05 | 9 |
Table four
In step 303, the device provides an http service transmitting the first log data. In the foregoing step 301, the device acquires an alarm period N, i.e. a value of 2. The device provides an http service for transmitting the first log data with 2 first cycles being 1 cycle. It should be determined that the device may provide an http service for transmitting the first amount of first log data and an http service for transmitting the second amount of first log data at the same time at one port, or may provide an http service at a different port. The device is provided with a data acquisition unit.
In step 304, the device obtains a first amount of first log data and a second amount of first log data via an http service.
Since the device provides the http service for transmitting the first log data with 2 first periods being 1 period, the device will also acquire the first amount of the first log data with the http service with 2 first periods being 1 period, so as to obtain the time sequence data of the first amount of the first log data. The device may further acquire a second number of first log data through the http service with 2 first periods being 1 period, so as to obtain time sequence data of the second number of first log data. Table five is the time series data of the first log data of the first quantity, and table six is the time series data of the first log data of the second quantity.
| Keyword(s) | Time period | Log quantity |
| Software X | 15:00:15 | 25 |
| Software X | 15:00:45 | 36 |
TABLE five
| Keyword(s) | Time period | Log quantity |
| Software X | 15:00:05 | 17 |
| Software X | 15:00:35 | 19 |
TABLE six
In step 305, if the sum of the first number acquired by the http service for N consecutive periods is greater than a first threshold, the device alarms; if the sum of the second number acquired by the http service for N consecutive periods is greater than the first threshold, the device alarms.
Because the device obtains the first log data from the http service with 2 first periods being 1 period, i.e. the device obtains the first log data from the http service with a second period, the device may not calculate the sum of the first line and the second line any more, but directly make a decision with the value 25 or 17. For example, in the present embodiment, the value 25 is greater than the value 20, so that the sum of the first number of consecutive 2 first periods acquired by the http service is greater than 20, the device alerts. The value 17 is smaller than the value 20, so that the sum of the second number of consecutive 2 first periods acquired by the http service is smaller than 20, the device does not alert.
Regarding the description of the beneficial effects of the promethaus-based log alert method in the embodiments of the present application, reference may be made to the foregoing description of the beneficial effects of the promethaus-based log alert system of fig. 1A and 1B.
The method for alarming the Prometaus-based log in the embodiment of the application is described above, and the device for alarming the Prometaus-based log in the embodiment of the application is described below.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a log alert device based on promethaus in an embodiment of the present application.
The device comprises
A first acquiring unit 401 for acquiring a keyword;
a second obtaining unit 402, configured to periodically retrieve the elastic search module according to the keyword, so as to obtain first log data corresponding to the keyword;
a processing unit 403, configured to provide an http service for transmitting the first log data;
a third obtaining unit 404, configured to obtain the first log data through an http service;
and an alarm unit 405, configured to alarm if the number of the first log data in the third acquisition unit is greater than the first threshold.
The second acquiring unit 402 in the implementation of the present application may retrieve an elastic search module, the processing unit 403 may provide an http service for transmitting the first log data, and the third acquiring unit 404 may acquire the first log data through the http service. Therefore, the Prometheus-based log alert device has the advantages of both the elastic search and the Prometheus, and can be implemented under a Prometheus-based alert framework, thus reducing the management burden.
In addition to the promethaus-based log alert device illustrated in fig. 4, the promethaus-based log alert device may further include the following:
referring to fig. 5, fig. 5 is another schematic structural diagram of a log alert device based on promethaus in an embodiment of the present application.
Alternatively, the second obtaining unit 402 is specifically configured to retrieve the elastic search module according to the keyword, so as to obtain first log data in a first format corresponding to the keyword;
the apparatus further comprises: a conversion unit 501 for converting the format of the first log data to obtain first log data adapted to the second format of the promethaus service unit;
the third obtaining unit 404 is specifically configured to obtain the first log data in the second format through an http service.
Optionally, the processing unit 403 is specifically configured to provide an http service for transmitting the first amount of the first log data in the second periodicity.
The alarm unit 405 is specifically configured to alarm by the monitoring module if the sum of the first number is greater than a first threshold value within a duration n×m, where M is a first periodic period duration, and N is an integer greater than 1. Optionally, the first obtaining unit is further configured to obtain period information of the second periodicity;
the processing unit 403 is specifically configured to provide an http service for transmitting the first log data with a period of nxm according to the period information.
The promethaus-based log alert device in the embodiment of the present application is described above, and the promethaus-based log alert device in the embodiment of the present application is described below.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a log alert device based on promethaus in an embodiment of the present application.
The promethaus-based log alert device may be a desktop computer, a notebook computer, a server, or the like, and fig. 6 is a schematic structural diagram of a promethaus-based log alert device provided in an embodiment of the present application, where the promethaus-based log alert device 600 may include one or more central processing units (central processing units, CPU) 601 and a memory 605, where one or more application programs or data are stored in the memory 605.
Wherein the memory 605 may be volatile storage or persistent storage. The program stored in the memory 605 may include one or more modules, each of which may include a series of instruction operations on the server. Still further, the central processor 601 may be configured to communicate with the memory 605 to execute a series of instruction operations in the memory 605 on the promethaus-based log alert device 600.
The Prometheus-based log alert device 600 may also include one or more power sources 602, one or more wired or wireless network interfaces 603, one or more input/output interfaces 604, and/or one or more operating systems, such as Windows ServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, etc.
In the embodiment of the present application, the processor 601 included in the server further has the following functions:
acquiring keywords; according to the keywords, a first periodic search module is searched for obtaining first log data corresponding to the keywords; providing an http service for transmitting the first log data; acquiring first log data through an http service; and if the number of the first log data acquired through the http service is larger than a first threshold value, alarming.
Optionally, in an embodiment of the present application, the processor 601 included in the promethaus-based log alert device further has the following functions:
optionally, according to the keyword, searching the elastic search module periodically to obtain first log data in a first format corresponding to the keyword;
converting the format of the first log data to obtain first log data in a second format adapted to Prometheus;
and acquiring the first log data in the second format through an http service.
Optionally, a second periodic http service for transmitting the first log data is provided;
if the number of the first log data acquired through the http service is greater than a first threshold value within the duration N×M, alarming, wherein M is the period duration of the first periodicity, the period duration of the second periodicity is N times of the period duration of the first periodicity, and N is an integer greater than 1.
Optionally, acquiring period information of a second periodicity, where the period information includes the N, and sending the period information to the data acquisition unit;
according to the period information, taking NxM as a period, and providing http service for transmitting the first log data in a second period.
In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random access memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Claims (10)
1. A promethaus-based log alert system comprising;
the system comprises a log module, a monitoring module and an elastic search module;
the monitoring module comprises a data acquisition unit, a Prometaus service unit and an alarm unit;
the log module is used for acquiring keywords and sending the keywords to the data acquisition unit;
the data acquisition unit is used for retrieving the elastic search module according to the keywords in a first periodicity to obtain a first quantity of first log data corresponding to the keywords, and is also used for providing http services for transmitting the first log data;
the log module is further configured to obtain an interval duration of a first periodicity and a third periodicity, where the interval duration is a difference value between a start time point of the first periodicity and a start time point of the third periodicity;
the log module is further used for sending the interval duration to the data acquisition unit;
the data acquisition unit is further used for retrieving the elastic search module according to the interval duration and the keywords in a third period to obtain first log data of a second quantity corresponding to the keywords;
the Prometaus service unit is used for acquiring the first log data through the http service and storing the first log data;
the alarm unit is used for alarming if the sum of the first number obtained by http service in N continuous periods in the Prometaus service unit is larger than a first threshold; if the sum of the second number of the N continuous periods acquired through the http service in the Prometaus service unit is larger than a first threshold value, alarming; and N is an integer greater than or equal to 1.
2. The system of claim 1, wherein the data acquisition unit comprises a data acquisition subunit, a data format conversion subunit, and an http service transmission subunit;
the data acquisition unit is specifically configured to retrieve, by the data acquisition subunit, the elastiscearch module according to the keyword, for a first periodicity, so as to obtain the first log data in a first format corresponding to the keyword;
the data format conversion subunit is configured to convert a format of the first log data to obtain the first log data adapted to a second format of the promethaus service unit;
the data acquisition unit is specifically configured to provide, by means of the http service transmission subunit, an http service for transmitting the first log data in the second format.
3. The system according to claim 1 or 2, wherein,
the Prometaus service unit is specifically configured to obtain and store a first amount of the first log data through the http service in a second periodic manner;
the alarm unit is specifically configured to alarm if the sum of the first number is greater than the first threshold within a duration n×m, where M is the first periodic period duration, and the second periodic period duration is the n×m, where N is an integer greater than 1.
4. The system of claim 3, wherein the system further comprises a controller configured to control the controller,
the log module is further configured to obtain period information of the second periodicity, where the period information includes the N, and send the period information to the data acquisition unit;
the data acquisition unit is specifically configured to provide, according to the period information, an http service for transmitting the first log data with n×m as a period and a second period.
5. A method for alarming a log based on promethaus, comprising;
the monitoring module acquires keywords;
the monitoring module searches the elastic search module according to the keywords in a first periodicity to obtain first log data of a first quantity corresponding to the keywords;
the monitoring module acquires interval duration of a first periodicity and a third periodicity, wherein the interval duration is a difference value between a starting time point of the first periodicity and a starting time point of the third periodicity;
the monitoring module retrieves the elastic search module according to the interval duration and the keywords in a third period to obtain a second number of first log data corresponding to the keywords;
the monitoring module provides an http service for transmitting the first log data;
the monitoring module acquires the first log data through the http service and stores the first log data;
if the sum of the first number obtained by http service in N continuous periods in the Prometaus service unit is larger than a first threshold value, the monitoring module alarms; if the sum of the second number of the N continuous periods obtained by http service in the Prometaus service unit is larger than a first threshold value, the monitoring module alarms; and N is an integer greater than or equal to 1.
6. The method of claim 5, wherein the step of determining the position of the probe is performed,
the monitoring module searches the elastic search module periodically according to the keyword to obtain first log data corresponding to the keyword, wherein the first log data comprises:
the monitoring module searches the elastic search module according to the keywords in a first periodicity to obtain first log data in a first format corresponding to the keywords;
the method further comprises the steps of:
the monitoring module converts the format of the first log data to obtain the first log data adapting to the second format of the Prometaus service unit;
the monitoring module obtaining the first log data through the http service includes:
and acquiring the first log data in the second format through the http service.
7. A method according to claim 5 or 6, wherein,
the monitoring module providing an http service for transmitting the first log data includes:
the monitoring module provides the http service for transmitting a first amount of the first log data in a second periodic manner;
if the number of the first log data in the promethaus service unit is greater than a first threshold, the monitoring module alarms including:
if the sum of the first number is greater than the first threshold value within the period n×m, the monitoring module alarms, where M is the period duration of the first periodicity, the period duration of the second periodicity is the period n×m, and N is an integer greater than 1.
8. A promethaus-based log alert device comprising:
a first acquisition unit configured to acquire a keyword;
a second obtaining unit, configured to search an elastic search module periodically according to the keyword, so as to obtain a first number of first log data corresponding to the keyword;
the first obtaining unit is further configured to obtain an interval duration of a first periodicity and a third periodicity, where the interval duration is a difference between a start time point of the first periodicity and a start time point of the third periodicity;
the first obtaining unit is further configured to send the interval duration to the second obtaining unit;
the second obtaining unit is further configured to retrieve the elastic search module according to the interval duration and the keyword in a third periodicity, so as to obtain a second number of first log data corresponding to the keyword;
a processing unit, configured to provide an http service for transmitting the first log data;
a third obtaining unit, configured to obtain the first log data through the http service;
an alarm unit, configured to alarm if a sum of a first number of N consecutive periods acquired through an http service in the third acquisition unit is greater than a first threshold; if the sum of the second numbers acquired by http service in the continuous N periods in the third acquisition unit is larger than a first threshold value, alarming; and N is an integer greater than or equal to 1.
9. A computer device, comprising: a memory, a processor;
wherein the memory is used for storing programs;
the processor being adapted to execute programs in the memory, including performing the method of any of the preceding claims 5 to 7.
10. A computer storage medium having instructions stored therein, which when executed on a computer, cause the computer to perform the method of any of claims 5 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010392449.2A CN111581060B (en) | 2020-05-11 | 2020-05-11 | Prometaus-based log alarm system, method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010392449.2A CN111581060B (en) | 2020-05-11 | 2020-05-11 | Prometaus-based log alarm system, method and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111581060A CN111581060A (en) | 2020-08-25 |
| CN111581060B true CN111581060B (en) | 2024-03-12 |
Family
ID=72126462
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010392449.2A Active CN111581060B (en) | 2020-05-11 | 2020-05-11 | Prometaus-based log alarm system, method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111581060B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112199249A (en) * | 2020-09-16 | 2021-01-08 | 中国建设银行股份有限公司 | Monitoring data processing method, device, equipment and medium |
| CN112596994A (en) * | 2020-12-25 | 2021-04-02 | 福州掌中云科技有限公司 | XHProf-based PHP program performance detection method and device |
| CN113282920B (en) * | 2021-05-28 | 2023-10-10 | 平安科技（深圳）有限公司 | Log abnormality detection method, device, computer equipment and storage medium |
| CN113381884B (en) * | 2021-06-02 | 2023-01-31 | 上海数禾信息科技有限公司 | Full link monitoring method and device for monitoring alarm system |
| CN113626300A (en) * | 2021-08-03 | 2021-11-09 | 上海上讯信息技术股份有限公司 | Log management method and device |
| CN115473783A (en) * | 2022-08-04 | 2022-12-13 | 浪潮软件集团有限公司 | Prometheus-based index alarm management system and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107565953A (en) * | 2017-10-18 | 2018-01-09 | 南京邮电大学南通研究院有限公司 | A kind of control circuit of transition detection device and clock frequency regulating system |
| CN108921551A (en) * | 2018-06-11 | 2018-11-30 | 西安纸贵互联网科技有限公司 | Alliance's block catenary system based on Kubernetes platform |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN109656784A (en) * | 2018-12-25 | 2019-04-19 | 新华三技术有限公司 | A kind of log processing method and device |
| CN110175152A (en) * | 2019-05-30 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of log inquiring method, transfer server cluster and log query system |
| CN110321371A (en) * | 2019-07-01 | 2019-10-11 | 腾讯科技（深圳）有限公司 | Daily record data method for detecting abnormality, device, terminal and medium |
| CN110968482A (en) * | 2019-12-18 | 2020-04-07 | 上海良鑫网络科技有限公司 | Enterprise service and application intelligent monitoring system |
-
2020
- 2020-05-11 CN CN202010392449.2A patent/CN111581060B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107565953A (en) * | 2017-10-18 | 2018-01-09 | 南京邮电大学南通研究院有限公司 | A kind of control circuit of transition detection device and clock frequency regulating system |
| CN108921551A (en) * | 2018-06-11 | 2018-11-30 | 西安纸贵互联网科技有限公司 | Alliance's block catenary system based on Kubernetes platform |
| CN109245931A (en) * | 2018-09-19 | 2019-01-18 | 四川长虹电器股份有限公司 | The log management of container cloud platform based on kubernetes and the implementation method of monitoring alarm |
| CN109656784A (en) * | 2018-12-25 | 2019-04-19 | 新华三技术有限公司 | A kind of log processing method and device |
| CN110175152A (en) * | 2019-05-30 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of log inquiring method, transfer server cluster and log query system |
| CN110321371A (en) * | 2019-07-01 | 2019-10-11 | 腾讯科技（深圳）有限公司 | Daily record data method for detecting abnormality, device, terminal and medium |
| CN110968482A (en) * | 2019-12-18 | 2020-04-07 | 上海良鑫网络科技有限公司 | Enterprise service and application intelligent monitoring system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111581060A (en) | 2020-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111581060B (en) | Prometaus-based log alarm system, method and related equipment | |
| CN110362544B (en) | Log processing system, log processing method, terminal and storage medium | |
| US8423638B2 (en) | Performance monitoring of a computer resource | |
| CN111930547A (en) | Fault positioning method and device and storage medium | |
| US20190034498A1 (en) | Determining a presentation format for search results based on a presentation recommendation machine learning model | |
| CN108073625B (en) | System and method for metadata information management | |
| US11687826B2 (en) | Artificial intelligence (AI) based innovation data processing system | |
| He et al. | Parallel implementation of classification algorithms based on MapReduce | |
| KR102067032B1 (en) | Method and system for data processing based on hybrid big data system | |
| CN111125344B (en) | Related word recommendation method and device | |
| US20200112475A1 (en) | Real-time adaptive infrastructure scenario identification using syntactic grouping at varied similarity | |
| JP2015135668A (en) | Computing devices and methods of connecting people based on content and relational distance | |
| CN111400130A (en) | Task monitoring method and device, electronic equipment and storage medium | |
| JP2010250450A (en) | Database message analysis support program, method and device | |
| CN112948486A (en) | Batch data synchronization method and system and electronic equipment | |
| CN112182025A (en) | Log analysis method, device, equipment and computer readable storage medium | |
| CN115827797A (en) | Environmental data analysis and integration method and system based on big data | |
| Kavitha et al. | Discovering public opinions by performing sentimental analysis on real time Twitter data | |
| CN111625410B (en) | Information processing method, apparatus, and computer storage medium | |
| US11568344B2 (en) | Systems and methods for automated pattern detection in service tickets | |
| US20200110815A1 (en) | Multi contextual clustering | |
| CN106557483B (en) | Data processing method, data query method, data processing equipment and data query equipment | |
| CN116521664A (en) | Data monitoring method and device for data warehouse, computing equipment and storage medium | |
| Chen et al. | Related technologies | |
| Ma | A survey of big data for IoT in cloud computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |