CN117909892A - LOF algorithm and Prophet model-based data anomaly real-time early warning method and system - Google Patents

Abstract

The invention provides a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model, wherein the method comprises the following steps: step S101, data acquisition, namely acquiring and monitoring a data source; step S102, processing the data and performing relevant characteristic processing; step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point; step S104, performing outlier judgment and propset model training, and when outlier judgment reaches a judgment threshold, comparing the outlier with the propset model training result, and performing step S105; step S105, entering an abnormal early warning monitoring mode, judging an early warning threshold value, comparing the LOF calculation result with the propset model training result, and entering S106 when the result reaches the preset early warning threshold value; step S106, pushing the early warning notice. The LOF algorithm and the Prophet model are combined to provide a data abnormality real-time early warning method and system, so that the data abnormality can be better monitored and responded, a user or the system is helped to effectively manage and control potential risks, and early warning is timely carried out.

Description

LOF algorithm and Prophet model-based data anomaly real-time early warning method and system

Technical Field

The invention relates to the technical field of real-time early warning, in particular to a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model.

Background

In modern internet applications, with rapid development of technologies such as big data and internet of things, more and more real-time data needs to be monitored and analyzed in real time, so as to identify potential abnormal situations in time and take corresponding measures. However, the untimely detection of data anomalies may lead to serious consequences, including system failures, security vulnerabilities, service instability, and the like. LOF (local anomaly factor) algorithms and Prophet (time series prediction) models are two common and efficient data analysis methods.

The local abnormality factor algorithm is an algorithm for outlier detection, and the basic principle is that the abnormality degree of a data point is judged by calculating the density comparison between the data point and other data points in the neighborhood of the data point. The LOF algorithm can effectively identify data points with abnormal behaviors in time sequence data, and has important significance for abnormal detection in a real-time early warning system. The propset model is a model specially designed for time series prediction, and can automatically learn and capture regularity in time series data such as trend, seasonal pattern and the like. The Prophet model can predict future time and provide relevant information such as confidence interval and the like, and plays an important role in anomaly prediction in a real-time early warning system.

The method and the system for realizing real-time early warning based on the local anomaly factor algorithm and the Prophet model combine the advantages of the two methods to realize real-time anomaly detection and prediction of the time sequence data stream. By calculating local anomaly factors of data points in real time, possible anomaly points can be identified in time. Meanwhile, the Prophet model is used for predicting the future time in real time, and early warning information of abnormal conditions can be further provided.

Disclosure of Invention

The invention provides a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model, which aim to solve the problem of untimely detection of data flow, improve the sensitivity and accuracy of data and predict the capacity in real time based on trend and historical data.

In order to achieve the above purpose, the invention provides a real-time data anomaly early warning method based on an LOF algorithm and a Prophet model, which comprises the following steps:

step S101, data acquisition, namely acquiring and monitoring a data source;

Step S102, processing the data and performing relevant characteristic processing;

Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;

Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;

step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result judges that the result does not reach the set early warning threshold value;

step S106, pushing the early warning notice.

In one embodiment, in step S102, the data processing includes cleaning data, missing value, and outlier processing;

The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.

In one embodiment, in step S103, the LOF calculation formula includes a Z-score calculation formula, a Robust Z-score calculation formula.

In one embodiment, in step S103, feature vectors or data points in the data set are first determined, an appropriate neighborhood size is selected, the local density of each data point is determined by calculating its average distance from other data points in the neighborhood, and the relative reachable distance is calculated to obtain the LOF.

In one embodiment, in step S104, the calculated LOF is compared to a preset threshold, and data points exceeding the threshold are marked as outliers.

In one embodiment, in step S104, the propset model training includes historical data analysis, model parameter setting, and dynamically updating the model to ensure accuracy and adaptability of model prediction.

In one embodiment, step S105 compares the predicted result of the outlier and Prophet model training with a preset early warning threshold, and if the outlier mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.

In one embodiment, in step S106, the early warning notification manner includes one or more of a short message, a mail, and a push message, so as to notify the relevant personnel of the early warning information in time.

A real-time early warning system based on an LOF algorithm and a Prophet model is used for realizing a data abnormality real-time early warning method based on the LOF algorithm and the Prophet model, and comprises the following steps:

a monitoring data source module that monitors a plurality of data sources;

The data processing module is connected with the monitoring data source module and is used for carrying out real-time processing and preprocessing on data;

The LOF calculation module is connected with the data processing module and used for carrying out LOF calculation on the processed data so as to obtain abnormal points;

the Prophet model training module is connected with the data processing module and used for predicting data trend to obtain a prediction result;

and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.

In one embodiment, the data sources include a first data source, a second data source, a third data source, and a fourth data source;

the data processing module comprises a data cleaning unit, a data conversion unit and a data aggregation unit;

the Prophet model training module comprises a data acquisition unit, a data training unit and an anomaly prediction unit.

The invention has the following beneficial effects:

1. Detecting abnormality in real time: the local anomaly factor algorithm is used for detecting anomaly points in the data in real time and helping a system to quickly capture possible problems or anomalies. Through calculation and analysis of the anomaly factors, data points that are significantly different from the normal mode can be accurately identified.

2. Prediction based on historical data: prophet model prediction is based on historical data trend, seasonal and special event factors, and the like, and real-time prediction is carried out. The method can provide reliable prediction of future data points by using a time sequence mode, and help the early warning system respond timely.

3. The accuracy of real-time early warning is improved: the results of real-time anomaly detection and trend prediction can be comprehensively considered by combining a local anomaly factor algorithm and a Prophet model, and the accuracy of real-time early warning is improved. The sensitivity of the anomaly factor algorithm is combined with the prediction capability of the Prophet model, so that potential anomalies can be better captured, and early warning can be performed in time.

Drawings

FIG. 1 is a flow chart of a real-time early warning method for data anomalies based on LOF algorithm and Prophet model according to an embodiment of the invention;

fig. 2 is a block diagram of a real-time early warning system based on LOF algorithm and propset model according to an embodiment of the present invention.

Wherein 100 is a monitoring data source module; 110 is a first data source; 120 is a second data source; 130 is a third data source; 140 is a fourth data source; 200 is a data processing module; 210 is a data cleaning unit; 220 is a data conversion unit; 230 is a data aggregation unit; 300 is a LOF calculation module; 400 is a propset model training module; 410 is a data acquisition unit; 420 is a data training unit; 430 is an anomaly prediction unit.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application become more apparent, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The described embodiments are some, but not all, embodiments of the application. The embodiments described below by referring to the drawings are illustrative and intended to explain the present application and should not be construed as limiting the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

Fig. 1 is a flow chart of a method for implementing real-time early warning of data anomalies based on an LOF (local anomaly factor) algorithm and a propset (time series prediction) model according to an embodiment of the present invention, including:

step S101, data acquisition, namely acquiring and monitoring a data source;

Step S102, processing the data and performing relevant characteristic processing;

Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;

Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;

Step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result does not reach the set early warning threshold value;

step S106, pushing the early warning notice.

Specifically, the data acquisition is performed to determine the source and the acquisition mode of real-time data, so that the data can enter an early warning system in time; the data preprocessing includes cleaning the data, processing missing values and outliers, ensuring reliable quality of the data input into the algorithm and model. The feature extraction comprises the steps of extracting time, numerical value, trend and other features from real-time data, and is used for inputting a local LOF algorithm and a Prophet model.

In one embodiment, in step S102, the data processing includes cleaning data, missing value, and outlier processing;

The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.

In one embodiment, in step S103, the LOF calculation formula includes a Z-score calculation formula, a Robust Z-score calculation formula.

In one embodiment, in step S103, feature vectors or data points in the data set are first determined, an appropriate neighborhood size is selected, the local density of each data point is determined by calculating its average distance from other data points in the neighborhood, and the relative reachable distance is calculated to obtain the LOF.

Specifically, higher values represent more outlier data points, ordered by LOF value.

In one embodiment, in step S104, the calculated LOF is compared to a preset threshold, and data points exceeding the threshold are marked as outliers.

In one embodiment, in step S104, the propset model training includes historical data analysis, model parameter setting, and dynamically updating the model to ensure accuracy and adaptability of model prediction.

Specifically, historical data analysis is performed, a propset model is trained and constructed by using the historical data, and characteristics of data such as trend, seasonality and the like are analyzed. Model parameter setting, namely setting parameters of a Prophet model, such as window size, seasonal modes and the like according to data characteristics and requirements. And dynamically updating the model, namely dynamically updating the Prophet model according to the input of real-time data, and keeping the accuracy and adaptability of the model.

In one embodiment, step S105 compares the predicted result of the outlier and Prophet model training with a preset early warning threshold, and if the outlier mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.

In one embodiment, in step S106, the early warning notification manner includes one or more of a short message, a mail, and a push message, so as to notify the relevant personnel of the early warning information in time.

Specifically, the early warning trigger condition: the detection is based on the outlier identification result of the LOF algorithm and the prediction result of the Prophet model.

Fig. 2 is a block diagram of a real-time early warning system based on a LOF algorithm and a propset model according to an embodiment of the present invention, configured to implement a data anomaly real-time early warning method based on the LOF algorithm and the propset model, including:

A monitoring data source module 100, the monitoring data source module 100 monitoring a plurality of data sources;

the data processing module 200 is connected with the monitoring data source module 100 and is used for carrying out real-time processing and preprocessing on data;

The LOF calculation module 300 is connected with the data processing module 200, and performs LOF calculation on the processed data to obtain an abnormal point;

the propset model training module 400 is connected with the data processing module 200 and is used for predicting data trend to obtain a prediction result;

and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.

In one embodiment, the data sources include a first data source 110, a second data source 120, a third data source 130, and a fourth data source 140;

the data processing module 200 includes a data cleansing unit 210, a data conversion unit 220, and a data aggregation unit 230;

The Prophet model training module 400 includes a data acquisition unit 410, a data training unit 420, and an anomaly prediction unit 430.

An embodiment of the present invention is as follows:

Data acquisition and real-time processing: and data acquisition is carried out by monitoring the data source, and the data is transmitted to the real-time data stream processing platform. In one embodiment, multiple data sources may be monitored and data collected simultaneously, or designated data sources may be configured to be monitored and data collected. And carrying out real-time processing and preprocessing on the data on a real-time data stream processing platform. This includes data cleansing, data transformation, data aggregation, etc. preprocessing work. Ensuring data quality and accuracy.

Dynamic adjustment and anomaly detection of LOF algorithm: in the preprocessed data, the parameters of the LOF algorithm are dynamically adjusted using an adaptive sliding window algorithm. By monitoring the change of the data, the proper window size and neighborhood number are automatically selected. The LOF of each data point was calculated using the Z-score algorithm. Z-score measures the degree of deviation between the data point and the average value, thereby determining whether the data is abnormal. The calculated LOF is compared with a preset threshold. Data points that exceed a threshold are marked as outliers.

Training and prediction of propset model: time series data are collected and divided into two columns, date and target variable. The date column is named "ds" and the target variable column is named "y" to ensure that the data type is correct. The Prophet model is then trained using the historical time series data. The model automatically learns and captures trends and seasonal patterns of data, as well as other relevant patterns and influencing factors. And then the model is adjusted and optimized according to the parameter settings of the model, such as seasonal mode setting, trend flexibility and the like. And finally, predicting future time by using the trained Prophet model to obtain a prediction result.

Abnormality detection and early warning: combining the outlier mark of the LOF algorithm with the prediction result of the Prophet model. And comparing the abnormal point marks and the prediction results with preset early warning conditions. If the early warning condition is met, namely the abnormal point mark and the prediction result meet the early warning condition at the same time, the system triggers a corresponding early warning notice so as to take measures in time.

Through the steps, the method realizes the combined use of a local anomaly factor detection algorithm and a time sequence prediction model for real-time processing and dynamic adjustment of monitoring data so as to detect and predict anomalies

The invention has the following beneficial effects:

1. Detecting abnormality in real time: the local anomaly factor algorithm is used for detecting anomaly points in the data in real time and helping a system to quickly capture possible problems or anomalies. Through calculation and analysis of the anomaly factors, data points that are significantly different from the normal mode can be accurately identified.

2. Prediction based on historical data: prophet model prediction is based on historical data trend, seasonal and special event factors, and the like, and real-time prediction is carried out. The method can provide reliable prediction of future data points by using a time sequence mode, and help the early warning system respond timely.

3. The accuracy of real-time early warning is improved: the results of real-time anomaly detection and trend prediction can be comprehensively considered by combining a local anomaly factor algorithm and a Prophet model, and the accuracy of real-time early warning is improved. The sensitivity of the anomaly factor algorithm is combined with the prediction capability of the Prophet model, so that potential anomalies can be better captured, and early warning can be performed in time.

In the description of the present application, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments in accordance with the present application. For ease of description, the dimensions of the various features shown in the drawings are not drawn to actual scale. Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but should be considered part of the specification where appropriate. In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.

It should be noted that, in the present application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. It should also be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

It should be further noted that, unless explicitly stated or limited otherwise, the words "connected" and "brought about" and the like used in the description of the present application should be interpreted broadly, and may be directly or through intermediaries, or relationships between two elements, and their specific meaning in the present application may be understood by those skilled in the art according to circumstances.

The embodiments described above are intended to provide those skilled in the art with a full range of modifications and variations to the embodiments described above without departing from the spirit of the application, and therefore the scope of the application is not limited to the embodiments described above, but is to be accorded the broadest scope consistent with the novel features set forth in the claims.

Claims (10)

1. The real-time data abnormality early warning method based on the LOF algorithm and the Prophet model is characterized by comprising the following steps of:

step S101, data acquisition, namely acquiring and monitoring a data source;

Step S102, processing the data and performing relevant characteristic processing;

Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;

Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;

step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result judges that the result does not reach the set early warning threshold value;

step S106, pushing the early warning notice.

2. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 1, wherein in step S102, the data processing includes cleaning data, missing values and outlier processing;

The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.

3. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 1, wherein in step S103, the LOF calculation formula includes a Z-score calculation formula and a Robust Z-score calculation formula.

4. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 3, wherein in step S103, feature vectors or data points in the data set are determined first, a suitable neighborhood size is selected, the local density of each data point is determined by calculating the average distance from the data point to other data points in the neighborhood, and the relative reachable distance is calculated to obtain LOF.

5. The method for real-time warning of abnormal data based on LOF algorithm and propset model according to claim 4, wherein in step S104, the calculated LOF is compared with a preset threshold, and the data points exceeding the threshold are marked as abnormal points.

6. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 5, wherein in step S104, the propset model training includes historical data analysis, model parameter setting and dynamic updating of the model to ensure accuracy and adaptability of model prediction.

7. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 6, wherein step S105 compares the predicted result of training abnormal points and propset model with a preset early warning threshold, and if the abnormal point mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.

8. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 1, wherein in step S106, the early warning notification means includes one or more of a short message, a mail and a push message, so as to notify the relevant personnel of early warning information in time.

9. A real-time early warning system based on LOF algorithm and propset model for implementing the data anomaly real-time early warning method based on LOF algorithm and propset model according to any one of claims 1-8, comprising:

a monitoring data source module that monitors a plurality of data sources;

The data processing module is connected with the monitoring data source module and is used for carrying out real-time processing and preprocessing on data;

The LOF calculation module is connected with the data processing module and used for carrying out LOF calculation on the processed data so as to obtain abnormal points;

the Prophet model training module is connected with the data processing module and used for predicting data trend to obtain a prediction result;

and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.

10. The method for real-time early warning of data anomalies based on the LOF algorithm and the propset model according to claim 9, wherein the data sources comprise a first data source, a second data source, a third data source and a fourth data source;

the data processing module comprises a data cleaning unit, a data conversion unit and a data aggregation unit;

the Prophet model training module comprises a data acquisition unit, a data training unit and an anomaly prediction unit.