CN117909892A - LOF algorithm and Prophet model-based data anomaly real-time early warning method and system - Google Patents
LOF algorithm and Prophet model-based data anomaly real-time early warning method and system Download PDFInfo
- Publication number
- CN117909892A CN117909892A CN202311746080.0A CN202311746080A CN117909892A CN 117909892 A CN117909892 A CN 117909892A CN 202311746080 A CN202311746080 A CN 202311746080A CN 117909892 A CN117909892 A CN 117909892A
- Authority
- CN
- China
- Prior art keywords
- data
- early warning
- lof
- propset
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 30
- 230000002159 abnormal effect Effects 0.000 claims abstract description 27
- 238000004364 calculation method Methods 0.000 claims abstract description 25
- 238000012544 monitoring process Methods 0.000 claims abstract description 20
- 230000005856 abnormality Effects 0.000 claims abstract description 9
- 238000004140 cleaning Methods 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 5
- 238000004220 aggregation Methods 0.000 claims description 5
- 238000007405 data analysis Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 239000013598 vector Substances 0.000 claims description 3
- 238000001514 detection method Methods 0.000 description 10
- 230000001932 seasonal effect Effects 0.000 description 6
- 230000035945 sensitivity Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013450 outlier detection Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2123/00—Data types
- G06F2123/02—Data types in the time domain, e.g. time-series data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model, wherein the method comprises the following steps: step S101, data acquisition, namely acquiring and monitoring a data source; step S102, processing the data and performing relevant characteristic processing; step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point; step S104, performing outlier judgment and propset model training, and when outlier judgment reaches a judgment threshold, comparing the outlier with the propset model training result, and performing step S105; step S105, entering an abnormal early warning monitoring mode, judging an early warning threshold value, comparing the LOF calculation result with the propset model training result, and entering S106 when the result reaches the preset early warning threshold value; step S106, pushing the early warning notice. The LOF algorithm and the Prophet model are combined to provide a data abnormality real-time early warning method and system, so that the data abnormality can be better monitored and responded, a user or the system is helped to effectively manage and control potential risks, and early warning is timely carried out.
Description
Technical Field
The invention relates to the technical field of real-time early warning, in particular to a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model.
Background
In modern internet applications, with rapid development of technologies such as big data and internet of things, more and more real-time data needs to be monitored and analyzed in real time, so as to identify potential abnormal situations in time and take corresponding measures. However, the untimely detection of data anomalies may lead to serious consequences, including system failures, security vulnerabilities, service instability, and the like. LOF (local anomaly factor) algorithms and Prophet (time series prediction) models are two common and efficient data analysis methods.
The local abnormality factor algorithm is an algorithm for outlier detection, and the basic principle is that the abnormality degree of a data point is judged by calculating the density comparison between the data point and other data points in the neighborhood of the data point. The LOF algorithm can effectively identify data points with abnormal behaviors in time sequence data, and has important significance for abnormal detection in a real-time early warning system. The propset model is a model specially designed for time series prediction, and can automatically learn and capture regularity in time series data such as trend, seasonal pattern and the like. The Prophet model can predict future time and provide relevant information such as confidence interval and the like, and plays an important role in anomaly prediction in a real-time early warning system.
The method and the system for realizing real-time early warning based on the local anomaly factor algorithm and the Prophet model combine the advantages of the two methods to realize real-time anomaly detection and prediction of the time sequence data stream. By calculating local anomaly factors of data points in real time, possible anomaly points can be identified in time. Meanwhile, the Prophet model is used for predicting the future time in real time, and early warning information of abnormal conditions can be further provided.
Disclosure of Invention
The invention provides a data abnormity real-time early warning method and system based on an LOF algorithm and a Prophet model, which aim to solve the problem of untimely detection of data flow, improve the sensitivity and accuracy of data and predict the capacity in real time based on trend and historical data.
In order to achieve the above purpose, the invention provides a real-time data anomaly early warning method based on an LOF algorithm and a Prophet model, which comprises the following steps:
step S101, data acquisition, namely acquiring and monitoring a data source;
Step S102, processing the data and performing relevant characteristic processing;
Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;
Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;
step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result judges that the result does not reach the set early warning threshold value;
step S106, pushing the early warning notice.
In one embodiment, in step S102, the data processing includes cleaning data, missing value, and outlier processing;
The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.
In one embodiment, in step S103, the LOF calculation formula includes a Z-score calculation formula, a Robust Z-score calculation formula.
In one embodiment, in step S103, feature vectors or data points in the data set are first determined, an appropriate neighborhood size is selected, the local density of each data point is determined by calculating its average distance from other data points in the neighborhood, and the relative reachable distance is calculated to obtain the LOF.
In one embodiment, in step S104, the calculated LOF is compared to a preset threshold, and data points exceeding the threshold are marked as outliers.
In one embodiment, in step S104, the propset model training includes historical data analysis, model parameter setting, and dynamically updating the model to ensure accuracy and adaptability of model prediction.
In one embodiment, step S105 compares the predicted result of the outlier and Prophet model training with a preset early warning threshold, and if the outlier mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.
In one embodiment, in step S106, the early warning notification manner includes one or more of a short message, a mail, and a push message, so as to notify the relevant personnel of the early warning information in time.
A real-time early warning system based on an LOF algorithm and a Prophet model is used for realizing a data abnormality real-time early warning method based on the LOF algorithm and the Prophet model, and comprises the following steps:
a monitoring data source module that monitors a plurality of data sources;
The data processing module is connected with the monitoring data source module and is used for carrying out real-time processing and preprocessing on data;
The LOF calculation module is connected with the data processing module and used for carrying out LOF calculation on the processed data so as to obtain abnormal points;
the Prophet model training module is connected with the data processing module and used for predicting data trend to obtain a prediction result;
and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.
In one embodiment, the data sources include a first data source, a second data source, a third data source, and a fourth data source;
the data processing module comprises a data cleaning unit, a data conversion unit and a data aggregation unit;
the Prophet model training module comprises a data acquisition unit, a data training unit and an anomaly prediction unit.
The invention has the following beneficial effects:
1. Detecting abnormality in real time: the local anomaly factor algorithm is used for detecting anomaly points in the data in real time and helping a system to quickly capture possible problems or anomalies. Through calculation and analysis of the anomaly factors, data points that are significantly different from the normal mode can be accurately identified.
2. Prediction based on historical data: prophet model prediction is based on historical data trend, seasonal and special event factors, and the like, and real-time prediction is carried out. The method can provide reliable prediction of future data points by using a time sequence mode, and help the early warning system respond timely.
3. The accuracy of real-time early warning is improved: the results of real-time anomaly detection and trend prediction can be comprehensively considered by combining a local anomaly factor algorithm and a Prophet model, and the accuracy of real-time early warning is improved. The sensitivity of the anomaly factor algorithm is combined with the prediction capability of the Prophet model, so that potential anomalies can be better captured, and early warning can be performed in time.
Drawings
FIG. 1 is a flow chart of a real-time early warning method for data anomalies based on LOF algorithm and Prophet model according to an embodiment of the invention;
fig. 2 is a block diagram of a real-time early warning system based on LOF algorithm and propset model according to an embodiment of the present invention.
Wherein 100 is a monitoring data source module; 110 is a first data source; 120 is a second data source; 130 is a third data source; 140 is a fourth data source; 200 is a data processing module; 210 is a data cleaning unit; 220 is a data conversion unit; 230 is a data aggregation unit; 300 is a LOF calculation module; 400 is a propset model training module; 410 is a data acquisition unit; 420 is a data training unit; 430 is an anomaly prediction unit.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application become more apparent, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The described embodiments are some, but not all, embodiments of the application. The embodiments described below by referring to the drawings are illustrative and intended to explain the present application and should not be construed as limiting the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Fig. 1 is a flow chart of a method for implementing real-time early warning of data anomalies based on an LOF (local anomaly factor) algorithm and a propset (time series prediction) model according to an embodiment of the present invention, including:
step S101, data acquisition, namely acquiring and monitoring a data source;
Step S102, processing the data and performing relevant characteristic processing;
Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;
Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;
Step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result does not reach the set early warning threshold value;
step S106, pushing the early warning notice.
Specifically, the data acquisition is performed to determine the source and the acquisition mode of real-time data, so that the data can enter an early warning system in time; the data preprocessing includes cleaning the data, processing missing values and outliers, ensuring reliable quality of the data input into the algorithm and model. The feature extraction comprises the steps of extracting time, numerical value, trend and other features from real-time data, and is used for inputting a local LOF algorithm and a Prophet model.
In one embodiment, in step S102, the data processing includes cleaning data, missing value, and outlier processing;
The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.
In one embodiment, in step S103, the LOF calculation formula includes a Z-score calculation formula, a Robust Z-score calculation formula.
In one embodiment, in step S103, feature vectors or data points in the data set are first determined, an appropriate neighborhood size is selected, the local density of each data point is determined by calculating its average distance from other data points in the neighborhood, and the relative reachable distance is calculated to obtain the LOF.
Specifically, higher values represent more outlier data points, ordered by LOF value.
In one embodiment, in step S104, the calculated LOF is compared to a preset threshold, and data points exceeding the threshold are marked as outliers.
In one embodiment, in step S104, the propset model training includes historical data analysis, model parameter setting, and dynamically updating the model to ensure accuracy and adaptability of model prediction.
Specifically, historical data analysis is performed, a propset model is trained and constructed by using the historical data, and characteristics of data such as trend, seasonality and the like are analyzed. Model parameter setting, namely setting parameters of a Prophet model, such as window size, seasonal modes and the like according to data characteristics and requirements. And dynamically updating the model, namely dynamically updating the Prophet model according to the input of real-time data, and keeping the accuracy and adaptability of the model.
In one embodiment, step S105 compares the predicted result of the outlier and Prophet model training with a preset early warning threshold, and if the outlier mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.
In one embodiment, in step S106, the early warning notification manner includes one or more of a short message, a mail, and a push message, so as to notify the relevant personnel of the early warning information in time.
Specifically, the early warning trigger condition: the detection is based on the outlier identification result of the LOF algorithm and the prediction result of the Prophet model.
Fig. 2 is a block diagram of a real-time early warning system based on a LOF algorithm and a propset model according to an embodiment of the present invention, configured to implement a data anomaly real-time early warning method based on the LOF algorithm and the propset model, including:
A monitoring data source module 100, the monitoring data source module 100 monitoring a plurality of data sources;
the data processing module 200 is connected with the monitoring data source module 100 and is used for carrying out real-time processing and preprocessing on data;
The LOF calculation module 300 is connected with the data processing module 200, and performs LOF calculation on the processed data to obtain an abnormal point;
the propset model training module 400 is connected with the data processing module 200 and is used for predicting data trend to obtain a prediction result;
and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.
In one embodiment, the data sources include a first data source 110, a second data source 120, a third data source 130, and a fourth data source 140;
the data processing module 200 includes a data cleansing unit 210, a data conversion unit 220, and a data aggregation unit 230;
The Prophet model training module 400 includes a data acquisition unit 410, a data training unit 420, and an anomaly prediction unit 430.
An embodiment of the present invention is as follows:
Data acquisition and real-time processing: and data acquisition is carried out by monitoring the data source, and the data is transmitted to the real-time data stream processing platform. In one embodiment, multiple data sources may be monitored and data collected simultaneously, or designated data sources may be configured to be monitored and data collected. And carrying out real-time processing and preprocessing on the data on a real-time data stream processing platform. This includes data cleansing, data transformation, data aggregation, etc. preprocessing work. Ensuring data quality and accuracy.
Dynamic adjustment and anomaly detection of LOF algorithm: in the preprocessed data, the parameters of the LOF algorithm are dynamically adjusted using an adaptive sliding window algorithm. By monitoring the change of the data, the proper window size and neighborhood number are automatically selected. The LOF of each data point was calculated using the Z-score algorithm. Z-score measures the degree of deviation between the data point and the average value, thereby determining whether the data is abnormal. The calculated LOF is compared with a preset threshold. Data points that exceed a threshold are marked as outliers.
Training and prediction of propset model: time series data are collected and divided into two columns, date and target variable. The date column is named "ds" and the target variable column is named "y" to ensure that the data type is correct. The Prophet model is then trained using the historical time series data. The model automatically learns and captures trends and seasonal patterns of data, as well as other relevant patterns and influencing factors. And then the model is adjusted and optimized according to the parameter settings of the model, such as seasonal mode setting, trend flexibility and the like. And finally, predicting future time by using the trained Prophet model to obtain a prediction result.
Abnormality detection and early warning: combining the outlier mark of the LOF algorithm with the prediction result of the Prophet model. And comparing the abnormal point marks and the prediction results with preset early warning conditions. If the early warning condition is met, namely the abnormal point mark and the prediction result meet the early warning condition at the same time, the system triggers a corresponding early warning notice so as to take measures in time.
Through the steps, the method realizes the combined use of a local anomaly factor detection algorithm and a time sequence prediction model for real-time processing and dynamic adjustment of monitoring data so as to detect and predict anomalies
The invention has the following beneficial effects:
1. Detecting abnormality in real time: the local anomaly factor algorithm is used for detecting anomaly points in the data in real time and helping a system to quickly capture possible problems or anomalies. Through calculation and analysis of the anomaly factors, data points that are significantly different from the normal mode can be accurately identified.
2. Prediction based on historical data: prophet model prediction is based on historical data trend, seasonal and special event factors, and the like, and real-time prediction is carried out. The method can provide reliable prediction of future data points by using a time sequence mode, and help the early warning system respond timely.
3. The accuracy of real-time early warning is improved: the results of real-time anomaly detection and trend prediction can be comprehensively considered by combining a local anomaly factor algorithm and a Prophet model, and the accuracy of real-time early warning is improved. The sensitivity of the anomaly factor algorithm is combined with the prediction capability of the Prophet model, so that potential anomalies can be better captured, and early warning can be performed in time.
In the description of the present application, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments in accordance with the present application. For ease of description, the dimensions of the various features shown in the drawings are not drawn to actual scale. Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but should be considered part of the specification where appropriate. In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
It should be noted that, in the present application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. It should also be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
It should be further noted that, unless explicitly stated or limited otherwise, the words "connected" and "brought about" and the like used in the description of the present application should be interpreted broadly, and may be directly or through intermediaries, or relationships between two elements, and their specific meaning in the present application may be understood by those skilled in the art according to circumstances.
The embodiments described above are intended to provide those skilled in the art with a full range of modifications and variations to the embodiments described above without departing from the spirit of the application, and therefore the scope of the application is not limited to the embodiments described above, but is to be accorded the broadest scope consistent with the novel features set forth in the claims.
Claims (10)
1. The real-time data abnormality early warning method based on the LOF algorithm and the Prophet model is characterized by comprising the following steps of:
step S101, data acquisition, namely acquiring and monitoring a data source;
Step S102, processing the data and performing relevant characteristic processing;
Step S103, setting an LOF judgment threshold value, and calculating LOF to obtain an abnormal point;
Step S104, performing outlier judgment and propset model training, comparing outlier with propset model training results when outlier judgment reaches a judgment threshold, and performing step S105, and performing no feedback when outlier judgment does not reach the judgment threshold;
step S105, entering an abnormal early warning monitoring mode, setting an early warning threshold value, judging the early warning threshold value, comparing the LOF calculation result with the Prophet model training result, entering S106 when the result reaches the set early warning threshold value, and not feeding back when the result judges that the result does not reach the set early warning threshold value;
step S106, pushing the early warning notice.
2. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 1, wherein in step S102, the data processing includes cleaning data, missing values and outlier processing;
The relevant feature processing includes extraction of time, values and trends for LOF calculation and propset model prediction.
3. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 1, wherein in step S103, the LOF calculation formula includes a Z-score calculation formula and a Robust Z-score calculation formula.
4. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 3, wherein in step S103, feature vectors or data points in the data set are determined first, a suitable neighborhood size is selected, the local density of each data point is determined by calculating the average distance from the data point to other data points in the neighborhood, and the relative reachable distance is calculated to obtain LOF.
5. The method for real-time warning of abnormal data based on LOF algorithm and propset model according to claim 4, wherein in step S104, the calculated LOF is compared with a preset threshold, and the data points exceeding the threshold are marked as abnormal points.
6. The method for real-time early warning of data anomalies based on LOF algorithm and propset model according to claim 5, wherein in step S104, the propset model training includes historical data analysis, model parameter setting and dynamic updating of the model to ensure accuracy and adaptability of model prediction.
7. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 6, wherein step S105 compares the predicted result of training abnormal points and propset model with a preset early warning threshold, and if the abnormal point mark and the predicted result exceed the early warning threshold at the same time, the system triggers a corresponding early warning notification so as to take measures in time.
8. The method for real-time early warning of abnormal data based on LOF algorithm and propset model according to claim 1, wherein in step S106, the early warning notification means includes one or more of a short message, a mail and a push message, so as to notify the relevant personnel of early warning information in time.
9. A real-time early warning system based on LOF algorithm and propset model for implementing the data anomaly real-time early warning method based on LOF algorithm and propset model according to any one of claims 1-8, comprising:
a monitoring data source module that monitors a plurality of data sources;
The data processing module is connected with the monitoring data source module and is used for carrying out real-time processing and preprocessing on data;
The LOF calculation module is connected with the data processing module and used for carrying out LOF calculation on the processed data so as to obtain abnormal points;
the Prophet model training module is connected with the data processing module and used for predicting data trend to obtain a prediction result;
and comparing the combination of the abnormal points obtained by the LOF algorithm module and the prediction result of the Prophet model with a preset early warning condition to judge whether the system needs early warning notification or not.
10. The method for real-time early warning of data anomalies based on the LOF algorithm and the propset model according to claim 9, wherein the data sources comprise a first data source, a second data source, a third data source and a fourth data source;
the data processing module comprises a data cleaning unit, a data conversion unit and a data aggregation unit;
the Prophet model training module comprises a data acquisition unit, a data training unit and an anomaly prediction unit.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311746080.0A CN117909892A (en) | 2023-12-18 | 2023-12-18 | LOF algorithm and Prophet model-based data anomaly real-time early warning method and system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311746080.0A CN117909892A (en) | 2023-12-18 | 2023-12-18 | LOF algorithm and Prophet model-based data anomaly real-time early warning method and system |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117909892A true CN117909892A (en) | 2024-04-19 |
Family
ID=90680849
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311746080.0A Pending CN117909892A (en) | 2023-12-18 | 2023-12-18 | LOF algorithm and Prophet model-based data anomaly real-time early warning method and system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117909892A (en) |
-
2023
- 2023-12-18 CN CN202311746080.0A patent/CN117909892A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6890382B2 (en) | Production system | |
Caggiano et al. | Cloud manufacturing framework for smart monitoring of machining | |
Pacella et al. | Using recurrent neural networks to detect changes in autocorrelated processes for quality monitoring | |
CN110895526A (en) | Method for correcting data abnormity in atmosphere monitoring system | |
JP6141235B2 (en) | How to detect anomalies in time series data | |
CN111177505A (en) | Training method, recommendation method and device of index anomaly detection model | |
CN110995153B (en) | Abnormal data detection method and device for photovoltaic power station and electronic equipment | |
JP2022519228A (en) | Systems and methods for detecting and measuring signal anomalies generated by components used in industrial processes | |
CN113344133B (en) | Method and system for detecting abnormal fluctuation of time sequence behaviors | |
CN116244765A (en) | Equipment maintenance management method based on industrial Internet | |
US20230221684A1 (en) | Explaining Machine Learning Output in Industrial Applications | |
CN112906738A (en) | Water quality detection and treatment method | |
CN117086699B (en) | Stainless steel member cutting temperature measurement method and system | |
Trajkova et al. | Active learning for automated visual inspection of manufactured products | |
CN117909892A (en) | LOF algorithm and Prophet model-based data anomaly real-time early warning method and system | |
CN117435908A (en) | Multi-fault feature extraction method for rotary machine | |
Wang et al. | Broad learning system based visual fault diagnosis for electrical equipment thermography images | |
Salvador et al. | Online detection of shutdown periods in chemical plants: A case study | |
Hoelzl et al. | Detecting seasonal dependencies in production lines for forecast optimization | |
CN110770753A (en) | Device and method for real-time analysis of high-dimensional data | |
CN113377630B (en) | Universal KPI anomaly detection framework implementation method | |
JP6798968B2 (en) | Noise cause estimation device | |
CN103974028A (en) | Method for detecting fierce behavior of personnel | |
CN114095337A (en) | KPI (Key Performance indicator) anomaly detection method and device, computing equipment and computer storage medium | |
CN113203843A (en) | Water quality monitoring method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |