CN110377569B - Log monitoring method, device, computer equipment and storage medium - Google Patents

Log monitoring method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN110377569B
CN110377569B CN201910531261.9A CN201910531261A CN110377569B CN 110377569 B CN110377569 B CN 110377569B CN 201910531261 A CN201910531261 A CN 201910531261A CN 110377569 B CN110377569 B CN 110377569B
Authority
CN
China
Prior art keywords
monitoring
log
data
calculation
script
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910531261.9A
Other languages
Chinese (zh)
Other versions
CN110377569A (en
Inventor
田晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Life Insurance Company of China Ltd
Original Assignee
Ping An Life Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Life Insurance Company of China Ltd filed Critical Ping An Life Insurance Company of China Ltd
Priority to CN201910531261.9A priority Critical patent/CN110377569B/en
Publication of CN110377569A publication Critical patent/CN110377569A/en
Application granted granted Critical
Publication of CN110377569B publication Critical patent/CN110377569B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The application relates to a log monitoring method and device based on cloud technology, computer equipment and storage medium. The method comprises the following steps: the method comprises the steps of obtaining a log record of an application system server, generating original log data according to a preset format by the log record, obtaining a data calculation script, carrying out data calculation on the original log data according to the data calculation script, and generating calculation results corresponding to all to-be-monitored characteristics of the original log data. And comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, triggering a monitoring operation when the calculation result accords with the preset threshold value, acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log. By setting different monitoring trigger conditions for the original log data and simultaneously running a plurality of corresponding monitoring scripts, the method realizes comprehensive monitoring of the log recorded by the system and further improves the working efficiency.

Description

Log monitoring method, device, computer equipment and storage medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a log monitoring method, a log monitoring device, a computer device, and a storage medium.
Background
With the increasing development and advancement of computer technology, in order to query for transactions, operations, or the like performed by a computer system, log files for recording and storing operations, or the like, of the computer system have appeared. The log monitoring platform for monitoring and associating the log files can locate the position where the abnormality occurs by accessing the log files when the computer system is abnormal, and timely solve the problem of the abnormality, so that the computer system is timely restored to be normal.
The traditional log monitoring platform is mainly applied to the scene of small server quantity and data quantity scale, when the data quantity increases faster or more server platforms are involved, the traditional log monitoring platform cannot support multi-platform or multi-dimensional complex monitoring, and in a series of processes from abnormal to abnormal confirmation in the monitoring process, and the problem is solved, a large amount of manual query operations are required to be executed, a large amount of manpower resources are consumed, and the working efficiency is low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log monitoring method, apparatus, computer device, and storage medium capable of improving the working efficiency.
A log monitoring method, the method comprising:
acquiring a record log of an application system server;
generating original log data from the log record according to a preset format;
acquiring a data calculation script, and carrying out data calculation on the original log data according to the data calculation script to generate calculation results corresponding to all the features to be monitored of the original log data;
comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
and acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log.
In one embodiment, the obtaining the data calculation script, and performing data calculation on the original log data according to the data calculation script, to generate a calculation result corresponding to each feature to be monitored of the original log data, includes:
when the generation of the original log data is detected, classifying the original log data to obtain digital log data, byte log data and text log data;
Acquiring a first type of computation script corresponding to the digital log data, a second type of computation script corresponding to the byte type log data and a third type of computation script corresponding to the text type log data;
when a data calculation instruction is detected, executing each data calculation script, and acquiring execution attribute information corresponding to each data calculation script;
respectively counting the number of data pieces of the original log data by utilizing each data calculation script to obtain corresponding data statistics;
acquiring the data content of the corresponding original log data according to the execution attribute information;
according to the data statistics and the data content of each piece of original data, calculating to obtain a corresponding calculation result; the calculation results include a first calculation result corresponding to the digital type log data, a second calculation result corresponding to the byte type log data, and a third calculation result corresponding to the text type log data.
In one embodiment, the data statistics include a first statistic corresponding to the digital log data, a second statistic corresponding to the byte log data on drinks, and a third statistic corresponding to the text log data; comparing the calculation result with a preset threshold corresponding to a preset monitoring triggering condition, and before triggering a monitoring operation when the calculation result accords with the preset threshold, further comprising:
Respectively synthesizing each calculation result and data statistics to obtain corresponding historical data; the historical data are stored in a database, and when a calculation and analysis instruction is detected, the analysis result corresponding to each feature to be monitored of each original log data is obtained through calculation and analysis, wherein the analysis result comprises first historical data corresponding to the digital log data, second historical data corresponding to the byte log data and third historical data corresponding to the text log data;
acquiring a calculation analysis script corresponding to each historical data; the computational analysis script comprises a first computational analysis script corresponding to the first historical data, a second computational analysis script corresponding to the second historical data, and a third computational analysis script corresponding to the third historical data;
when a calculation analysis instruction is detected, executing each calculation analysis script, and acquiring a calculation analysis rule corresponding to each calculation analysis script;
calculating and analyzing each historical data by utilizing each calculation and analysis rule to obtain a corresponding analysis result; the analysis results correspond to preset thresholds of the preset monitoring trigger conditions and comprise first analysis results corresponding to the first historical data, second analysis results corresponding to the second historical data and third analysis results corresponding to the third historical data;
Obtaining a corresponding relation between each analysis result and a preset threshold value of each preset monitoring trigger condition from a first mapping relation table of the analysis result and the preset threshold value of the preset monitoring trigger condition;
and acquiring a preset threshold value of a preset monitoring trigger condition corresponding to each analysis result.
In one embodiment, the preset threshold value of the preset monitoring trigger condition includes an IP request number threshold value of each request interface, a system request total number threshold value in unit time, a user total number threshold value in unit time, a single IP request total number threshold value, a single user IP change frequency threshold value, and a single user request flow threshold value; and comparing the calculation result with a preset threshold progress corresponding to a preset monitoring triggering condition, and triggering a monitoring operation when the calculation result accords with the preset threshold, wherein the method comprises the following steps:
obtaining calculation results corresponding to all the features to be monitored from the calculation results and a second mapping relation table of all the features to be monitored; the characteristics to be monitored comprise IP request times of each request interface, total system request number in unit time, total user number in unit time, total single IP request times, single user IP change frequency and single user request flow;
Comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition;
and triggering monitoring operation when the calculation result of the feature to be monitored exceeds a preset threshold value of a corresponding preset monitoring triggering condition.
In one embodiment, the monitoring scripts include a user dimension monitoring script, an IP dimension monitoring script, and a system dimension monitoring script; the step of obtaining the monitoring script corresponding to the monitoring operation, and running the monitoring script to monitor each feature to be monitored of the log, comprising:
setting corresponding monitoring items for each feature to be monitored; the monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items;
and running the user dimension monitoring script, the IP dimension monitoring script and the system dimension monitoring script to monitor each feature to be monitored respectively.
In one embodiment, the running each monitoring script monitors each feature to be monitored, including:
running the user dimension monitoring script, and when the total user number threshold, the single IP request total times threshold, the single user IP change frequency threshold and the single user request flow threshold in the unit time exceed the corresponding preset thresholds, generating a first warning file according to the current running condition and sending the first warning file to a developer;
Running the IP dimension monitoring script, and when the total frequency threshold of the single IP request and the frequency threshold of the single user IP change exceed the corresponding preset thresholds, generating a second warning file according to the current running condition and sending the second warning file to a developer;
and running the system dimension monitoring script, and when the IP request times threshold value of each request interface and the total system request number threshold value in unit time exceed preset corresponding preset threshold values, generating a third warning file according to the current running condition and sending the third warning file to a developer.
In one embodiment, the generating the original log data from the log record according to a preset format includes:
acquiring a preset format from a local storage; the preset formats comprise a basic log format and a custom log format; the basic log format comprises information which each log should contain, including date, time, log level, code position, log content and error code; the user-defined log format can output information according to the user definition of the formatter, wherein the information comprises threads, class names, file names, method names, information levels, file levels and file line numbers of the date;
Formatting the log according to the preset format to obtain a formatted log; the formatting process comprises a basic formatting process corresponding to the basic log format and a custom formatting process corresponding to the custom log format;
and generating original log data according to the formatted log record.
A log monitoring device, the device comprising:
the log acquisition module is used for acquiring the log of the application system server;
the original log data generation module is used for generating original log data from the record log according to a preset format;
the calculation module is used for acquiring a data calculation script, carrying out data calculation on the original log data according to the data calculation script, and generating calculation results corresponding to all the features to be monitored of the original log data;
the monitoring operation triggering module is used for comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
and the log monitoring module is used for acquiring the monitoring script corresponding to the monitoring operation, running the monitoring script and monitoring each feature to be monitored of the log.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring a record log of an application system server;
generating original log data from the log record according to a preset format;
acquiring a data calculation script, and carrying out data calculation on the original log data according to the data calculation script to generate calculation results corresponding to all the features to be monitored of the original log data;
comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
and acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring a record log of an application system server;
generating original log data from the log record according to a preset format;
acquiring a data calculation script, and carrying out data calculation on the original log data according to the data calculation script to generate calculation results corresponding to all the features to be monitored of the original log data;
Comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
and acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log.
According to the log monitoring method, the log monitoring device, the computer equipment and the storage medium, the terminal generates the original log data according to the preset format through the log recorded by the application system server, the data calculation script is obtained, the data calculation is carried out on the original log data according to the data calculation script, and the calculation results corresponding to all the features to be monitored of the original log data are generated. And comparing the calculation result with a preset threshold corresponding to a preset monitoring triggering condition, triggering a monitoring operation when the calculation result accords with the preset threshold, acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log. By setting different monitoring trigger conditions for the original log data and simultaneously running a plurality of corresponding monitoring scripts, the comprehensive monitoring of the log recorded by the system is realized, and the working efficiency is further improved.
Drawings
FIG. 1 is a flow chart of a log monitoring method according to an embodiment;
FIG. 2 is a flow chart of generating calculation results corresponding to each feature to be monitored of original log data in one embodiment;
FIG. 3 is a flowchart of a log monitoring method according to another embodiment;
FIG. 4 is a block diagram of a log monitoring device according to one embodiment;
fig. 5 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The log monitoring method provided by the application can be applied to the terminal of the log monitoring process. The terminal generates original log data according to a preset format by acquiring a log of an application system server and generating the log according to the preset format. The terminal acquires a data calculation script, performs data calculation on the original log data according to the data calculation script, generates calculation results corresponding to all to-be-monitored characteristics of the original log data, compares the calculation results with a preset threshold value corresponding to a preset monitoring trigger condition, and triggers monitoring operation when the calculation results accord with the preset threshold value. And monitoring each feature to be monitored of the log by acquiring a monitoring script corresponding to the monitoring operation and running the monitoring script. The terminal may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
In one embodiment, as shown in fig. 1, a log method is provided, and the method is applied to a terminal for illustration, and includes the following steps:
s102, the terminal acquires a log recorded by the application system server.
S104, the terminal generates original log data according to the record log and the preset format.
Specifically, the terminal acquires a log of the application system server, acquires a preset format from the local storage, formats the log by using the preset format, acquires the formatted log, and generates original log data according to the formatted log. The preset format comprises a basic log format and a custom log format, wherein the basic log format comprises information which each log should contain, and the information comprises date, time, log level, code position, log content and error code. The custom log format can output information according to the formatter, including thread, class name, file name, method name, information level, file level and file line number where the date is located. A common formatter is a PatternLayout for formatting log information, and may append other information to output more information or layout display.
Further, the log record means a log file for recording and storing the transaction, operation, etc. executed by the application system server. After generating the original log data, the terminal stores the original data in a first repository. The first repository is an operation hive repository, and is used for storing formatted record logs, namely original data. hive is a data warehouse tool based on Hadoop, which can map a structured data file into a database table, provide a simple sql query function, and convert sql statements into execution tasks for execution.
S106, the terminal acquires the data calculation script, and performs data calculation on the original log data according to the data calculation script to generate calculation results corresponding to all the features to be monitored of the original log data.
Specifically, when the generation of the original log data is detected, the terminal classifies the original log data to obtain digital log data, byte log data and text log data, and obtains a first type of computation script corresponding to the digital log data, a second type of computation script corresponding to the byte log data and a third type of computation script corresponding to the text log data.
When a data calculation instruction is detected, the terminal executes each data calculation script, acquires execution attribute information corresponding to each data calculation script, respectively counts the number of data of each original log data by utilizing each data calculation script, acquires corresponding data statistics, and further acquires the data content of each corresponding original log data according to each execution attribute information.
And calculating according to the data statistics and the data content of each original data to obtain corresponding calculation results, wherein the calculation results comprise a first calculation result corresponding to the digital log data, a second calculation result corresponding to the byte log data and a third calculation result corresponding to the text log data.
The data calculation script is used for calculating data of the formatted log record, namely the original data in the first storage library, and the generated calculation result is stored in the second storage library. The preset period represents a time period for the terminal to execute the data calculation script, and in the scheme, the preset period can be set to be 1 hour, and the execution rule of the data calculation script is executed once per hour. The second repository table is a marketing hive repository for storing the calculation results corresponding to the original data.
S108, the terminal compares the calculation result with a preset threshold corresponding to a preset monitoring trigger condition, and when the calculation result accords with the preset threshold, the terminal triggers the monitoring operation.
Specifically, the terminal obtains a calculation result corresponding to each feature to be monitored from the calculation result and a second mapping relation table of each feature to be monitored, wherein the feature to be monitored comprises the IP request times of each request interface, the total number of system requests in unit time, the total number of users in unit time, the total number of single IP requests, the single user IP change frequency and the single user request flow. And comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition, and triggering monitoring operation when the calculation result of the feature to be monitored exceeds the preset threshold value of the corresponding preset monitoring trigger condition.
Further, the terminal obtains the calculation result of each feature to be monitored recorded in the log from the first calculation result, the second calculation result and the third calculation result. The features to be monitored comprise IP request times of the request interfaces, total system request numbers in unit time, total user numbers in unit time, total single IP request times, single user IP change frequency and single user request flow.
S110, the terminal acquires a monitoring script corresponding to the monitoring operation, runs the monitoring script, and monitors each feature to be monitored of the log.
Specifically, the terminal sets corresponding monitoring items for each feature to be monitored, and different features to be monitored are respectively provided with corresponding monitoring items. The monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items, and the features to be monitored corresponding to the user dimension monitoring items comprise: a total number of users per unit time threshold, a total number of single IP requests threshold, a single user IP change frequency threshold, and a single user request traffic threshold. The features to be monitored corresponding to the IP dimension monitoring items comprise: a single IP request total times threshold, a single user IP change frequency threshold. The features to be monitored corresponding to the system dimension monitoring items comprise: the number of IP requests of each request interface is threshold and the total number of system requests in unit time is threshold.
And when the calculation result corresponding to the monitoring feature exceeds the preset threshold value of the preset monitoring triggering condition, acquiring the monitoring script corresponding to each monitoring item. The monitoring scripts comprise user dimension monitoring scripts, IP dimension monitoring scripts and system dimension monitoring scripts. And running a user dimension monitoring script, an IP dimension monitoring script and a system dimension monitoring script to monitor each feature to be monitored.
Further, the terminal runs each monitoring script to monitor each feature to be monitored, including:
and the terminal runs a user dimension monitoring script, and when the total number of users threshold, the total number of single IP requests threshold, the single user IP change frequency threshold and the single user request flow threshold in unit time exceed corresponding preset thresholds, a first warning file is generated according to the current running condition and is sent to a developer. And running an IP dimension monitoring script, and when the total number of times of single IP requests and the single user IP change frequency threshold exceed corresponding preset thresholds, generating a second warning file according to the current running condition and sending the second warning file to a developer. And running a system dimension monitoring script, and when the IP request times threshold of each request interface and the total number threshold of system requests in unit time exceed preset corresponding preset thresholds, generating a third warning file according to the current running condition and sending the third warning file to a developer.
In the log monitoring method, the terminal generates the original log data according to the preset format by logging of the application system server, acquires the data calculation script, performs data calculation on the original log data according to the data calculation script, and generates calculation results corresponding to all to-be-monitored characteristics of the original log data. And comparing the calculation result with a preset threshold corresponding to a preset monitoring triggering condition, triggering a monitoring operation when the calculation result accords with the preset threshold, acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log. By setting different monitoring trigger conditions for the original log data and simultaneously running a plurality of corresponding monitoring scripts, the comprehensive monitoring of the log recorded by the system is realized, and the working efficiency is further improved.
In one embodiment, as shown in fig. 2, there is provided a step of generating a calculation result corresponding to each feature to be monitored of the original log data, that is, performing data calculation on the original log data according to a data calculation script, and generating a calculation result corresponding to each feature to be monitored of the original log data, which specifically includes the following steps S202 to S210:
s202, when the generation of the original log data is detected, the terminal classifies the original log data to obtain digital log data, byte log data and text log data.
Specifically, when the terminal detects that the original log data is generated, the generated original log data is obtained, and the data types of the original log data are obtained, wherein the data types corresponding to the original log data comprise a digital type, a byte type and a text type, and the corresponding digital type log data, byte type log data and text type log data can be obtained.
S204, the terminal acquires a first type of computation script corresponding to the digital log data, a second type of computation script corresponding to the byte log data and a third type of computation script corresponding to the text log data.
Specifically, a corresponding relation exists between original log data of different data types and data calculation scripts, wherein the data calculation scripts corresponding to the digital log data are first-class calculation scripts, the data calculation scripts corresponding to the byte log data are second-class calculation scripts, and the data calculation scripts corresponding to the text log data are third-class calculation scripts.
S206, when the data calculation instruction is detected, the terminal executes each data calculation script and acquires the execution attribute information corresponding to each data calculation script.
Specifically, when a data calculation instruction is detected at the terminal, the terminal executes each acquired data calculation script including a first type calculation script, a second type calculation script and a third type calculation script, executes each data calculation script, and acquires execution attribute information corresponding to each data calculation script. The execution attribute information corresponding to each data calculation script comprises an execution expression of the data calculation script and acceptable data types, namely, a first type of calculation script is used for calculating digital log data, a second type of calculation script is used for calculating byte log data, and a third type of calculation script is used for calculating text log data.
S208, calculating scripts by using the data, and respectively counting the number of data pieces of the original log data to obtain corresponding data statistics.
Specifically, the first class of calculation scripts are utilized to count the number of data of the digital log data, a first statistical result is obtained, the second class of calculation scripts are utilized to count the number of data of the byte type log data, a second statistical result is obtained, and the third class of calculation scripts are utilized to count the number of data of the text type log data, so that a third statistical result is obtained.
S210, according to the execution attribute information, acquiring the data content of the corresponding original log data.
Specifically, the acceptable data types in the execution attribute information of the first type of computing script are digital log data, and the data content of the digital log data is obtained according to the execution expression in the execution attribute information of the first type of computing script. Similarly, the acceptable data type in the execution attribute information of the second type of computing script is byte-type log data, and the data content of the byte-type log data is obtained according to the execution expression in the execution attribute information of the second type of computing script. The acceptable data types in the execution attribute information of the third type of computation script are text type log data,
S212, calculating to obtain corresponding calculation results according to the data statistics and the data content of each original data, wherein the calculation results comprise a first calculation result corresponding to the digital log data, a second calculation result corresponding to the byte log data and a third calculation result corresponding to the text log data.
Specifically, according to the first type of calculation script, the first statistics and the data content of the digital log data can be calculated to obtain a corresponding first calculation result, likewise, according to the second type of script, the second statistics and the data content of the byte log data can be calculated to obtain a corresponding second calculation result, and according to the third type of calculation script, the second statistics and the data content of the text log data can be calculated to obtain a corresponding third calculation result.
The calculation results obtained for the original log data of different data types comprise IP request times of different request interfaces, total system request numbers in unit time, total system user numbers in unit time, total single IP request times, single user IP change frequency and single user request flow.
In the above steps, when the terminal detects that the original log data is generated, the data calculation scripts corresponding to the different types of original log data are obtained by classifying the original log data, and when the data calculation instruction is detected, each data calculation script is executed, and execution attribute information corresponding to each data calculation script is obtained. Counting the number of data pieces of each original log data by utilizing each data calculation script to obtain corresponding data statistics, and obtaining the data content of each corresponding original log data according to each execution attribute information; and calculating according to the data statistics and the data content of each original data to obtain a corresponding calculation result. The method can avoid the problem that the accuracy of the calculation result is not high enough due to the fact that the same data calculation script is used for repeatedly carrying out data calculation on a plurality of original log data of different types, and improves the calculation efficiency of the calculation result for the original log data.
In one embodiment, as shown in fig. 3, a log monitoring method is provided, where when a calculation result is compared with a preset threshold corresponding to a preset monitoring trigger condition, and when the calculation result meets the preset threshold, before triggering a monitoring operation, the log monitoring method further includes:
S302, the terminal respectively synthesizes each calculation result and data statistics to obtain corresponding historical data, the historical data are stored in a database, and when a calculation and analysis instruction is detected, the terminal is used for calculating and analyzing to obtain analysis results corresponding to each original log data, wherein the analysis results comprise first historical data corresponding to digital log data, second historical data corresponding to byte log data and third historical data corresponding to text log data.
Specifically, the terminal may obtain the first historical data corresponding to the digital log data by integrating the first calculation result and the first statistics result, obtain the second historical data corresponding to the byte type log data by integrating the second calculation result and the second statistics result, and similarly, obtain the third historical data corresponding to the text type log data by integrating the third calculation result and the third statistics result.
S304, the terminal acquires a calculation analysis script corresponding to each historical data, wherein the calculation analysis script comprises a first calculation analysis script corresponding to the first historical data, a second calculation analysis script corresponding to the second historical data and a third calculation analysis script corresponding to the third historical data.
S306, when the calculation analysis instruction is detected, the terminal executes each calculation analysis script and acquires the calculation analysis rule corresponding to each calculation analysis script.
Specifically, when the terminal detects the calculation analysis instruction, the terminal executes the acquired first, second and third calculation analysis scripts respectively, and acquires calculation analysis rules corresponding to the calculation analysis scripts, including a first calculation analysis rule corresponding to the first calculation analysis script, a second calculation analysis rule corresponding to the second calculation analysis script, and a third calculation analysis script corresponding to the third calculation analysis script.
S308, the terminal calculates and analyzes each historical data by utilizing each calculation analysis rule to obtain corresponding analysis results, wherein the analysis results correspond to preset thresholds of preset monitoring trigger conditions and comprise a first analysis result corresponding to the first historical data, a second analysis result corresponding to the second historical data and a third analysis result corresponding to the third historical data.
Specifically, the terminal performs calculation and analysis on the first historical data by using a first calculation and analysis rule to obtain a first analysis result corresponding to the first historical data, performs calculation and analysis on the second historical data by using a second calculation and analysis rule to obtain a second analysis result corresponding to the second historical data, and similarly performs calculation and analysis on the third historical data by using a third calculation and analysis rule to obtain a third analysis result corresponding to the third historical data.
S310, the terminal obtains the corresponding relation between each analysis result and the preset threshold value of each preset monitoring trigger condition from a first mapping relation table of the analysis result and the preset threshold value of the preset monitoring trigger condition.
S312, the terminal obtains a preset threshold value of a preset monitoring trigger condition corresponding to each analysis result.
Specifically, by establishing a corresponding relation between each analysis result and a preset threshold value of a preset monitoring trigger condition, a corresponding first mapping relation table is obtained, and the preset threshold value of the preset monitoring trigger condition corresponding to each analysis result can be obtained from the established first mapping relation table.
According to the log monitoring method, the calculation results and the data statistics are respectively integrated to obtain corresponding historical data, calculation analysis scripts corresponding to the historical data are obtained, and when calculation analysis instructions are detected, the calculation analysis scripts are executed, and calculation analysis rules corresponding to the calculation analysis scripts are obtained. And calculating and analyzing each historical data by utilizing each calculation analysis rule to obtain a corresponding analysis result, and obtaining a preset threshold value of a preset monitoring trigger condition corresponding to each analysis result from a first mapping relation table of the analysis result and the preset threshold value of the preset monitoring trigger condition. The method can avoid the problem that the accuracy of analysis results is not high enough and needs to be recalculated because the same calculation analysis script is used for repeatedly carrying out calculation analysis on the historical data of a plurality of different types of original log data, and improves the accuracy of the obtained analysis results corresponding to the historical data.
In one embodiment, there is provided a step of triggering a monitoring operation, that is, comparing a calculation result with a preset threshold corresponding to a preset monitoring triggering condition, and when the calculation result meets the preset threshold, triggering the monitoring operation, including:
obtaining calculation results corresponding to the features to be monitored from the calculation results and the second mapping relation table of the features to be monitored; the characteristics to be monitored comprise the IP request times of each request interface, the total system request number in unit time, the total user number in unit time, the total single IP request times, the single user IP change frequency and the single user request flow;
comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition; and triggering monitoring operation when the calculation result of the feature to be monitored exceeds a preset threshold value of a corresponding preset monitoring triggering condition.
Specifically, in different unit time, the preset threshold value sizes of the single interface under different application conditions can be obtained from the preset single interface and the mapping relation table of the threshold value sizes under different conditions respectively. Under different application conditions, different thresholds are set for a single interface according to the preset threshold value. For example, when counting the number of single user transmission requests acceptable by a single interface, the maximum number of requests sent by the single user acceptable by the interface is obtained from the mapping relation table, and is set as a threshold value of the number of single user transmission requests acceptable by the interface, the number of requests sent by the single user acceptable by the interface is counted, and whether the number of requests reaches a preset threshold value is judged.
Similarly, when judging whether the total number of system users in unit time exceeds a threshold, presetting a threshold corresponding to the total number of system users in unit time, and judging whether the accumulated number of users exceeds the threshold of the preset total number of users by accumulating the number of users corresponding to the requests received by the interfaces.
The terminal generates a statistical result by acquiring a calculation result after executing the data calculation script and synchronizing the calculation result into the second storage library, so that the subsequent historical result inquiry and historical data analysis are facilitated, and the terminal can generate preset threshold values corresponding to different monitoring trigger conditions according to the historical data in the statistical result.
Further, for the monitoring triggering condition, comparing the IP request times of different request interfaces in the calculation result with corresponding preset thresholds, and when judging that the IP request times of the different request interfaces exceed the corresponding thresholds, indicating that the monitoring triggering condition is reached, and triggering the monitoring operation in real time is needed.
Similarly, when the total number of system requests exceeds a corresponding preset threshold value in unit time, the total number of system users exceeds a corresponding preset threshold value in unit time, the total number of single IP requests exceeds a corresponding preset threshold value, special characters exist in the IP requests, the changing frequency of the single user IP exceeds a corresponding preset threshold value, and the single user request flow exceeds a corresponding preset threshold value, the condition that the monitoring triggering condition is reached is indicated, and the monitoring operation is required to be triggered in real time.
And the step of comparing the calculation result of each feature to be monitored with the corresponding preset threshold value of the preset monitoring triggering condition by acquiring each feature to be monitored corresponding to the log from the calculation result and acquiring the corresponding preset threshold value of the preset monitoring triggering condition of each feature to be monitored, and triggering the monitoring operation when the calculation result of the feature to be monitored exceeds the corresponding preset threshold value of the preset monitoring triggering condition. The method and the system realize comprehensive monitoring of all the characteristics of the log, can find out abnormal problems of all the characteristics to be monitored in time, and improve the working efficiency.
In one embodiment, there is provided a step of monitoring a log of records, the monitoring script including a user dimension monitoring script, an IP dimension monitoring script, and a system dimension monitoring script; namely, acquiring a monitoring script corresponding to the monitoring operation, and running the monitoring script to monitor each feature to be monitored of the log, wherein the step comprises the following steps:
the terminal sets corresponding monitoring items aiming at each feature to be monitored; the monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items; and running a user dimension monitoring script, an IP dimension monitoring script and a system dimension monitoring script to monitor each feature to be monitored.
The terminal sets a user dimension monitoring script for the user dimension monitoring item, sets an IP dimension monitoring script for the IP dimension monitoring item, sets a system dimension monitoring script for the system dimension monitoring item, judges whether the corresponding monitoring script needs to be acquired or not by judging whether a calculation result corresponding to the monitoring feature exceeds a preset threshold value of a preset monitoring trigger condition or not, and runs the corresponding monitoring script.
Specifically, the user dimension monitoring script is used for monitoring a total user number threshold value, a total single IP request number threshold value, a single user IP change frequency threshold value and a single user request flow threshold value in unit time. And the IP dimension monitoring script is used for monitoring the total frequency threshold value of the single IP request and the single user IP change frequency threshold value. And the system dimension monitoring script is used for monitoring the IP request times threshold value of each request interface and the total number threshold value of the system requests in unit time.
And setting corresponding monitoring items for each feature to be monitored, and when the calculation result corresponding to the monitoring feature exceeds the preset threshold value of the preset monitoring trigger condition, acquiring a monitoring script corresponding to each monitoring item, and running the corresponding monitoring script to realize the monitoring of each feature to be monitored. By setting corresponding monitoring items aiming at the features to be monitored and running corresponding monitoring scripts, the comprehensive monitoring of all the features of the log can be realized, the abnormal problems of all the features to be monitored can be found in time, and the working efficiency is improved.
In one embodiment, there is provided a method for running each monitoring script to monitor each feature to be monitored, including:
running a user dimension monitoring script, and generating a first warning file according to the current running condition and sending the first warning file to a developer when a total user number threshold, a single IP request total times threshold, a single user IP change frequency threshold and a single user request flow threshold in unit time exceed corresponding preset thresholds;
running an IP dimension monitoring script, when a single IP request total frequency threshold and a single user IP change frequency threshold exceed corresponding preset thresholds, generating a second warning file according to the current running condition, and sending the second warning file to a developer;
and running a system dimension monitoring script, and when the IP request times threshold of each request interface and the total number threshold of system requests in unit time exceed preset corresponding preset thresholds, generating a third warning file according to the current running condition and sending the third warning file to a developer.
Specifically, comparing the number of single IP requests in unit time with a corresponding preset threshold value, wherein the terminal obtains that a certain user IP is in an hour, the number of system access times is 50, the corresponding threshold value is 30, the number of system access times exceeds the threshold value, the user behavior corresponding to the IP is abnormal, an alarm mail is generated according to the abnormal condition and then sent to an operator, the operator adds the user into a system blacklist, the user can not access the system any more, after password modification is carried out, the user actively reports the problem that the system can not be accessed, and applies again, the operator carries out auditing on the user information, and the system can be accessed again after the user information passes.
According to the steps, corresponding monitoring scripts are respectively arranged in multiple dimensions aiming at the user dimension, the IP dimension and the system dimension, when the monitoring scripts of the corresponding dimensions are operated, the current situation of each feature to be monitored in different dimensions can be timely obtained, the abnormal problem of each feature to be monitored can be timely obtained, the abnormal problem is timely sent to a developer, the problem is solved, and the working efficiency is further improved.
In one embodiment, there is provided a method of generating raw log data from a log record in a preset format, comprising:
the terminal acquires a preset format from the local storage; the preset formats comprise a basic log format and a custom log format; the basic log format comprises information which each log should contain, including date, time, log level, code position, log content and error code; the user-defined log format can output information according to the user definition of the formatter, wherein the information comprises threads, class names, file names, method names, information levels, file levels and file line numbers of the date;
formatting the log according to a preset format to obtain a formatted log; the formatting process comprises a basic formatting process corresponding to a basic log format and a custom formatting process corresponding to a custom log format; and generating original log data according to the formatted log record.
Specifically, the terminal includes a basic log format and a custom log format according to a preset format obtained from a local storage, wherein the preset format represents a log format which can be received by an application server, and the log is subjected to basic formatting according to the basic log format to obtain a formatted log. Similarly, the terminal may perform custom formatting processing on the log according to the custom log format to obtain a corresponding formatted log, and generate original log data.
According to the steps, the recorded logs are formatted according to the preset format, original log data are generated, unified and standard original log data are obtained, the problem that subsequent operations are abnormal due to inconsistent data formats is avoided, and the working efficiency is improved.
It should be understood that, although the steps in the flowcharts of fig. 1 to 3 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 1 to 3 may comprise a plurality of sub-steps or phases, which are not necessarily performed at the same time, but may be performed at different times, nor does the order of execution of the sub-steps or phases necessarily follow one another, but may be performed alternately or alternately with at least a portion of the sub-steps or phases of other steps or other steps.
In one embodiment, as shown in fig. 4, there is provided a log monitoring apparatus, including: a log acquisition module 402, an original log data generation module 404, a calculation module 406, a monitoring operation triggering module 408, and a log monitoring module 410, wherein:
the log obtaining module 402 is configured to obtain a log of an application server.
The original log data generating module 404 is configured to generate original log data from the log record according to a preset format.
The calculation module 406 is configured to obtain a data calculation script, perform data calculation on the original log data according to the data calculation script, and generate a calculation result corresponding to each feature to be monitored of the original log data.
The monitoring operation triggering module 408 is configured to compare the calculation result with a preset threshold corresponding to a preset monitoring triggering condition, and trigger a monitoring operation when the calculation result meets the preset threshold.
The log monitoring module 410 is configured to obtain a monitoring script corresponding to the monitoring operation, run the monitoring script, and monitor each feature to be monitored of the log.
According to the log monitoring device, different monitoring trigger conditions are set for the original log data, and a plurality of corresponding monitoring scripts can be run simultaneously, so that the overall monitoring of the log recorded by the system is realized, and the working efficiency is further improved.
In one embodiment, a computing module is provided that is further configured to:
when the generation of the original log data is detected, classifying the original log data to obtain digital log data, byte log data and text log data; acquiring a first type of computation script corresponding to the digital log data, a second type of computation script corresponding to the byte log data and a third type of computation script corresponding to the text log data; when a data calculation instruction is detected, executing each data calculation script, and acquiring execution attribute information corresponding to each data calculation script; respectively counting the number of data pieces of each original log data by utilizing each data calculation script to obtain corresponding data statistics; acquiring the data content of each corresponding original log data according to each execution attribute information; according to the data statistics and the data content of each original data, calculating to obtain a corresponding calculation result; the calculation results include a first calculation result corresponding to the digital-type log data, a second calculation result corresponding to the byte-type log data, and a third calculation result corresponding to the text-type log data.
The calculation module can avoid the problem that the accuracy of the calculation result is not high enough because the same data calculation script is generated and the data calculation is repeatedly performed on a plurality of original log data of different types, and improves the calculation efficiency of the calculation result for the original log data.
In one embodiment, a log monitoring device is provided, and the log monitoring device further includes a preset threshold acquiring module, configured to:
respectively synthesizing each calculation result and data statistics to obtain corresponding historical data; the historical data is stored in a database, and when a calculation and analysis instruction is detected, the historical data is used for calculating and analyzing to obtain analysis results corresponding to each original log data, wherein the analysis results comprise first historical data corresponding to digital log data, second historical data corresponding to byte log data and third historical data corresponding to text log data;
acquiring a calculation analysis script corresponding to each historical data; the computation analysis script comprises a first computation analysis script corresponding to the first historical data, a second computation analysis script corresponding to the second historical data and a third computation analysis script corresponding to the third historical data;
when a calculation analysis instruction is detected, executing each calculation analysis script, and acquiring a calculation analysis rule corresponding to each calculation analysis script; calculating and analyzing each historical data by utilizing each calculation and analysis rule to obtain a corresponding analysis result; the analysis results correspond to preset thresholds of preset monitoring trigger conditions and comprise first analysis results corresponding to the first historical data, second analysis results corresponding to the second historical data and third analysis results corresponding to the third historical data;
Obtaining the corresponding relation between each analysis result and the preset threshold value of each preset monitoring trigger condition from a first mapping relation table of the analysis result and the preset threshold value of the preset monitoring trigger condition; and acquiring a preset threshold value of a preset monitoring trigger condition corresponding to each analysis result.
According to the log monitoring device, the problem that the accuracy of analysis results is not high enough and recalculation is needed due to the fact that the same calculation analysis script is avoided and calculation analysis is repeatedly performed on historical data of a plurality of different types of original log data can be avoided, and the accuracy of the obtained analysis results corresponding to the historical data is improved.
In one embodiment, a monitoring operation triggering module is provided, further configured to:
obtaining calculation results corresponding to the features to be monitored from the calculation results and the second mapping relation table of the features to be monitored; the characteristics to be monitored comprise the IP request times of each request interface, the total system request number in unit time, the total user number in unit time, the total single IP request times, the single user IP change frequency and the single user request flow; comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition; and triggering monitoring operation when the calculation result of the feature to be monitored exceeds a preset threshold value of a corresponding preset monitoring triggering condition.
The monitoring operation triggering module realizes the comprehensive monitoring of all the characteristics of the recorded logs, can discover the abnormal problems of all the characteristics to be monitored in time, and improves the working efficiency.
In one embodiment, a log monitoring module is provided that is further configured to:
setting corresponding monitoring items for each feature to be monitored; the monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items; and running a user dimension monitoring script, an IP dimension monitoring script and a system dimension monitoring script to monitor each feature to be monitored.
According to the log monitoring module, the corresponding monitoring items are set for the features to be monitored, and the corresponding monitoring scripts are operated, so that the overall monitoring of all the features of the log can be realized, the abnormal problems of all the features to be monitored can be found in time, and the working efficiency is improved.
In one embodiment, a feature monitoring unit to be monitored is provided for:
running a user dimension monitoring script, and generating a first warning file according to the current running condition and sending the first warning file to a developer when a total user number threshold, a single IP request total times threshold, a single user IP change frequency threshold and a single user request flow threshold in unit time exceed corresponding preset thresholds; running an IP dimension monitoring script, when a single IP request total frequency threshold and a single user IP change frequency threshold exceed corresponding preset thresholds, generating a second warning file according to the current running condition, and sending the second warning file to a developer; and running a system dimension monitoring script, and when the IP request times threshold of each request interface and the total number threshold of system requests in unit time exceed preset corresponding preset thresholds, generating a third warning file according to the current running condition and sending the third warning file to a developer.
According to the feature monitoring unit to be monitored, corresponding monitoring scripts are respectively arranged in multiple dimensions aiming at the user dimension, the IP dimension and the system dimension, when the monitoring scripts of the corresponding dimensions are operated, the current situation of each feature to be monitored in different dimensions can be timely obtained, the abnormal problem of each feature to be monitored can be timely obtained, the abnormal problem is timely sent to a developer, the problem is solved, and the working efficiency is further improved.
In one embodiment, there is provided an original log data generation module further configured to:
acquiring a preset format from a local storage; the preset formats comprise a basic log format and a custom log format; the basic log format comprises information which each log should contain, including date, time, log level, code position, log content and error code; the user-defined log format can output information according to the user definition of the formatter, wherein the information comprises threads, class names, file names, method names, information levels, file levels and file line numbers of the date;
formatting the log according to a preset format to obtain a formatted log; the formatting process comprises a basic formatting process corresponding to a basic log format and a custom formatting process corresponding to a custom log format; and generating original log data according to the formatted log record.
The original log data generation module formats the recorded log according to the preset format to generate the original log data, so that the original log data with unified specification is obtained, the problem that subsequent operation is abnormal due to inconsistent data formats is avoided, and the working efficiency is improved.
The specific limitation of the log monitoring device can be referred to the limitation of the log monitoring method hereinabove, and will not be described herein. The various modules in the log monitoring device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a log monitoring method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 5 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided comprising a memory storing a computer program and a processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the respective method embodiments described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.

Claims (10)

1. A log monitoring method, the method comprising:
acquiring a record log of an application system server; generating original log data from the log record according to a preset format;
when the generation of the original log data is detected, classifying the original log data to obtain digital log data, byte log data and text log data, and obtaining a first type of calculation script corresponding to the digital log data, a second type of calculation script corresponding to the byte log data and a third type of calculation script corresponding to the text log data;
When a data calculation instruction is detected, executing each data calculation script, acquiring execution attribute information corresponding to each data calculation script, and respectively counting the number of data pieces of each original log data by using each data calculation script to obtain corresponding data statistics; the data statistics include a first statistics corresponding to the digital log data, a second statistics corresponding to the byte log data for a beverage, and a third statistics corresponding to the text log data;
acquiring the data content of the corresponding original log data according to the execution attribute information, and calculating to obtain a corresponding calculation result according to the data statistics and the data content of the original log data; the calculation results comprise a first calculation result corresponding to the digital log data, a second calculation result corresponding to the byte log data and a third calculation result corresponding to the text log data;
respectively synthesizing each calculation result and data statistics to obtain corresponding historical data; the historical data are stored in a database, and when a calculation and analysis instruction is detected, the analysis result corresponding to each feature to be monitored of each original log data is obtained through calculation and analysis, wherein the analysis result comprises first historical data corresponding to the digital log data, second historical data corresponding to the byte log data and third historical data corresponding to the text log data;
Acquiring a calculation analysis script corresponding to each historical data; the computational analysis script comprises a first computational analysis script corresponding to the first historical data, a second computational analysis script corresponding to the second historical data, and a third computational analysis script corresponding to the third historical data;
when a calculation analysis instruction is detected, executing each calculation analysis script to obtain a calculation analysis rule corresponding to each calculation analysis script, and respectively calculating and analyzing each historical data by utilizing each calculation analysis rule to obtain a corresponding analysis result; the analysis results correspond to preset thresholds of preset monitoring trigger conditions and comprise first analysis results corresponding to the first historical data, second analysis results corresponding to the second historical data and third analysis results corresponding to the third historical data;
obtaining a corresponding relation between each analysis result and a preset threshold value of each preset monitoring trigger condition from a first mapping relation table of the analysis result and the preset threshold value of the preset monitoring trigger condition;
acquiring a preset threshold value of a preset monitoring trigger condition corresponding to each analysis result, comparing the calculation result with the preset threshold value corresponding to the preset monitoring trigger condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
And acquiring a monitoring script corresponding to the monitoring operation, running the monitoring script, and monitoring each feature to be monitored of the log.
2. The method according to claim 1, wherein the preset thresholds of the preset monitoring trigger conditions include an IP request number threshold, a system request total number threshold per unit time, a user total number threshold per unit time, a single IP request total number threshold, a single user IP change frequency threshold, and a single user request traffic threshold of each request interface; and comparing the calculation result with a preset threshold progress corresponding to a preset monitoring triggering condition, and triggering a monitoring operation when the calculation result accords with the preset threshold, wherein the method comprises the following steps:
obtaining calculation results corresponding to all the features to be monitored from the calculation results and a second mapping relation table of all the features to be monitored; the characteristics to be monitored comprise IP request times of each request interface, total system request number in unit time, total user number in unit time, total single IP request times, single user IP change frequency and single user request flow;
comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition;
And triggering monitoring operation when the calculation result of the feature to be monitored exceeds a preset threshold value of a corresponding preset monitoring triggering condition.
3. The method of claim 2, wherein the monitoring scripts include user dimension monitoring scripts, IP dimension monitoring scripts, and system dimension monitoring scripts; the step of obtaining the monitoring script corresponding to the monitoring operation, and running the monitoring script to monitor each feature to be monitored of the log, comprising:
setting corresponding monitoring items for each feature to be monitored; the monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items;
and running the user dimension monitoring script, the IP dimension monitoring script and the system dimension monitoring script to monitor each feature to be monitored respectively.
4. A method according to claim 3, wherein said running the monitoring script monitors each of the features to be monitored, comprising:
running the user dimension monitoring script, and when the total user number threshold, the single IP request total times threshold, the single user IP change frequency threshold and the single user request flow threshold in the unit time exceed the corresponding preset thresholds, generating a first warning file according to the current running condition and sending the first warning file to a developer;
Running the IP dimension monitoring script, and when the total frequency threshold of the single IP request and the frequency threshold of the single user IP change exceed the corresponding preset thresholds, generating a second warning file according to the current running condition and sending the second warning file to a developer;
and running the system dimension monitoring script, and when the IP request times threshold value of each request interface and the total system request number threshold value in unit time exceed preset corresponding preset threshold values, generating a third warning file according to the current running condition and sending the third warning file to a developer.
5. The method of claim 1, wherein generating the log record into raw log data in a predetermined format comprises:
acquiring a preset format from a local storage; the preset formats comprise a basic log format and a custom log format; the basic log format comprises information which each log should contain, including date, time, log level, code position, log content and error code; the user-defined log format is used for outputting information according to the user definition of the formatter, wherein the information comprises threads, class names, file names, method names, information levels, file levels and file line numbers of the date;
Formatting the log according to the preset format to obtain a formatted log; the formatting process comprises a basic formatting process corresponding to the basic log format and a custom formatting process corresponding to the custom log format;
and generating original log data according to the formatted log record.
6. A log monitoring device for implementing the method of any one of claims 1 to 5, the device comprising:
the log acquisition module is used for acquiring the log of the application system server;
the original log data generation module is used for generating original log data from the record log according to a preset format;
the calculation module is used for acquiring a data calculation script, carrying out data calculation on the original log data according to the data calculation script, and generating calculation results corresponding to all the features to be monitored of the original log data;
the monitoring operation triggering module is used for comparing the calculation result with a preset threshold value corresponding to a preset monitoring triggering condition, and triggering monitoring operation when the calculation result accords with the preset threshold value;
And the log monitoring module is used for acquiring the monitoring script corresponding to the monitoring operation, running the monitoring script and monitoring each feature to be monitored of the log.
7. The apparatus of claim 6, wherein the monitoring operation triggering module is further configured to:
obtaining calculation results corresponding to all the features to be monitored from the calculation results and a second mapping relation table of all the features to be monitored; the characteristics to be monitored comprise IP request times of each request interface, total system request number in unit time, total user number in unit time, total single IP request times, single user IP change frequency and single user request flow; comparing the calculation result of each feature to be monitored with a preset threshold value of a corresponding preset monitoring trigger condition; and triggering monitoring operation when the calculation result of the feature to be monitored exceeds a preset threshold value of a corresponding preset monitoring triggering condition.
8. The apparatus of claim 6, wherein the log monitoring module is further configured to:
setting corresponding monitoring items for each feature to be monitored; the monitoring items comprise user dimension monitoring items, IP dimension monitoring items and system dimension monitoring items; and running a user dimension monitoring script, an IP dimension monitoring script and a system dimension monitoring script to monitor each feature to be monitored respectively.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
CN201910531261.9A 2019-06-19 2019-06-19 Log monitoring method, device, computer equipment and storage medium Active CN110377569B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910531261.9A CN110377569B (en) 2019-06-19 2019-06-19 Log monitoring method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910531261.9A CN110377569B (en) 2019-06-19 2019-06-19 Log monitoring method, device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN110377569A CN110377569A (en) 2019-10-25
CN110377569B true CN110377569B (en) 2023-07-28

Family

ID=68249306

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910531261.9A Active CN110377569B (en) 2019-06-19 2019-06-19 Log monitoring method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN110377569B (en)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111352975B (en) * 2020-03-04 2024-01-30 建信金融科技有限责任公司 Data quality management method, client, server and system
CN111800292B (en) * 2020-05-28 2023-08-22 中国平安财产保险股份有限公司 Early warning method and device based on historical flow, computer equipment and storage medium
CN112100229B (en) * 2020-06-10 2024-04-30 深圳市思迪信息技术股份有限公司 Data state multi-dimensional monitoring method, system, equipment and storage medium
CN111680010B (en) * 2020-08-14 2020-12-22 北京东方通软件有限公司 Log system design method suitable for JavaEE application server
CN112000644B (en) * 2020-08-21 2022-08-02 北京浪潮数据技术有限公司 Data processing method and system, electronic equipment and storage medium
CN112148686A (en) * 2020-09-25 2020-12-29 酒泉钢铁(集团)有限责任公司 Data monitoring processing method based on software and hardware integrated machine
CN112988772B (en) * 2021-02-08 2023-07-21 平安科技（深圳）有限公司 Behavior data monitoring method, device, equipment and medium
CN113656145A (en) * 2021-08-18 2021-11-16 南瑞集团有限公司 Hydropower station AGC control device script scheduling calculation method
CN115037656B (en) * 2022-05-19 2024-02-20 无线生活（杭州）信息科技有限公司 Alarm method and device
CN114915488A (en) * 2022-06-15 2022-08-16 中国联合网络通信集团有限公司 Flow calculation monitoring method and apparatus

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107632918A (en) * 2017-08-30 2018-01-26 中国工商银行股份有限公司 Calculate the monitoring system and method for storage device
CN108874644A (en) * 2018-06-06 2018-11-23 平安科技（深圳）有限公司 Data monitoring method, device, computer equipment and storage medium
CN109669848A (en) * 2017-10-16 2019-04-23 比亚迪股份有限公司 Log generation method, application server, log server and log generating system
KR101984714B1 (en) * 2018-08-23 2019-05-31 (주) 글루시스 Vertualized resource monitoring system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107632918A (en) * 2017-08-30 2018-01-26 中国工商银行股份有限公司 Calculate the monitoring system and method for storage device
CN109669848A (en) * 2017-10-16 2019-04-23 比亚迪股份有限公司 Log generation method, application server, log server and log generating system
CN108874644A (en) * 2018-06-06 2018-11-23 平安科技（深圳）有限公司 Data monitoring method, device, computer equipment and storage medium
KR101984714B1 (en) * 2018-08-23 2019-05-31 (주) 글루시스 Vertualized resource monitoring system

Also Published As

Publication number Publication date
CN110377569A (en) 2019-10-25

Similar Documents

Publication Publication Date Title
CN110377569B (en) Log monitoring method, device, computer equipment and storage medium
CN110445637B (en) Event monitoring method, system, computer device and storage medium
CN113489713B (en) Network attack detection method, device, equipment and storage medium
CN110347561B (en) Monitoring alarm method and terminal equipment
WO2021147559A1 (en) Service data quality measurement method, apparatus, computer device, and storage medium
CN109543891B (en) Method and apparatus for establishing capacity prediction model, and computer-readable storage medium
CN112636957A (en) Early warning method and device based on log, server and storage medium
US11693842B2 (en) Generating compact data structures for monitoring data processing performance across high scale network infrastructures
CN110399268A (en) A kind of method, device and equipment of anomaly data detection
CN111400361A (en) Data real-time storage method and device, computer equipment and storage medium
CN110457255B (en) Method, server and computer readable storage medium for archiving data
CN109697155B (en) IT system performance evaluation method, device, equipment and readable storage medium
CN106557483B (en) Data processing method, data query method, data processing equipment and data query equipment
CN108111328B (en) Exception handling method and device
CN112130944A (en) Page abnormity detection method, device, equipment and storage medium
CN110781232A (en) Data processing method, data processing device, computer equipment and storage medium
CN112085588A (en) Method and device for determining safety of rule model and data processing method
CN109542947B (en) Data statistical method, device, computer equipment and storage medium
CN114997750A (en) Risk information pushing method, system, equipment and medium
CN111274112B (en) Application program pressure measurement method, device, computer equipment and storage medium
CN113326064A (en) Method for dividing business logic module, electronic equipment and storage medium
CN113626387A (en) Task data export method and device, electronic equipment and storage medium
CN113761443A (en) Website page data acquisition and statistics method, storage medium and equipment
CN111045983A (en) Nuclear power station electronic file management method and device, terminal equipment and medium
CN117707653B (en) Parameter monitoring method, device, electronic equipment and computer readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant