CN115828244A - Memory leak detection method and device and related equipment - Google Patents
Memory leak detection method and device and related equipment Download PDFInfo
- Publication number
- CN115828244A CN115828244A CN202211625088.7A CN202211625088A CN115828244A CN 115828244 A CN115828244 A CN 115828244A CN 202211625088 A CN202211625088 A CN 202211625088A CN 115828244 A CN115828244 A CN 115828244A
- Authority
- CN
- China
- Prior art keywords
- data
- historical
- subsequence
- duration
- historical data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Examining Or Testing Airtightness (AREA)
Abstract
The application discloses a memory leak detection method, which comprises the following steps: acquiring a historical data sequence of a preset data type in a target virtual machine, wherein the preset data type is a memory utilization rate; determining a historical change point in the historical data sequence; dividing the historical data sequence according to the historical change points to obtain each historical data subsequence; performing model training by using each historical data subsequence to obtain a detection model; and performing memory leak detection on the target virtual machine by using the detection model. By applying the technical scheme provided by the application, timely and effective memory leakage detection can be realized in a production environment, and the availability and reliability of the virtual machine are further ensured. The application also discloses a memory leak detection device, a system and a computer readable storage medium, which also have the beneficial effects.
Description
Technical Field
The present application relates to the field of network security technologies, and in particular, to a memory leak detection method, a memory leak detection apparatus, a memory leak detection system, and a computer-readable storage medium.
Background
Cloud computing is widely used in various industries because of its ability to provide inexpensive and on-demand access, computing, and storage resources, and users can deploy their applications on these virtual machines using only the needed resources, which allows for efficient use of physical hardware and reduces overall costs. However, memory leaks in applications deployed on the cloud may affect the availability and reliability of applications and even virtual machines, and therefore, it is important to quickly identify the virtual machine memory grow-out phenomenon.
Currently, in a development environment, memory leak problems can be easily detected by using static source code analysis tools or analyzing heap dumps, but in a production environment running on the cloud, memory leak detection is a very big challenge when knowledge on the details of the specific application or its internal object allocation is not known, and is only detected when an exception occurs in runtime, a system resource usage exception, an application crashes or a virtual machine restarts, and then such problems are solved at the cost of reducing the availability and reliability of the application or the virtual machine.
Therefore, how to implement timely and effective memory leak detection in a production environment, and further ensure availability and reliability of a virtual machine is a problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The memory leak detection method can achieve timely and effective memory leak detection in a production environment, and further guarantee availability and reliability of a virtual machine; another object of the present application is to provide a memory leak detection apparatus, system and computer readable storage medium, all having the above advantages.
In a first aspect, the present application provides a memory leak detection method, including:
acquiring a historical data sequence of a preset data type in a target virtual machine, wherein the preset data type is the memory utilization rate;
determining a historical change point in the historical data sequence;
dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
performing model training by using each historical data subsequence to obtain a detection model;
and carrying out memory leak detection on the target virtual machine by using the detection model.
Preferably, the determining the historical change point in the historical data sequence includes:
performing first-order difference calculation on the historical data sequence to obtain difference absolute values;
calculating to obtain a standard score corresponding to each difference absolute value;
and determining whether the historical data corresponding to the differential absolute value is the historical change point or not according to the standard score.
Preferably, the dividing the historical data sequence according to the historical change point to obtain each historical data subsequence includes:
for each other historical change point in the historical data sequence except for the last historical change point, combining the historical data between the other historical change point and the last historical change point into the historical data subsequence.
Preferably, the performing model training by using each of the historical data subsequences to obtain a detection model includes:
for each historical data subsequence, obtaining fitting data corresponding to the historical data subsequence through linear fitting, wherein the fitting data comprises a fitting curve, a fitting curve slope, fitting curve duration and abnormal duration;
and in all the fitting data, taking the fitting data with the fitting curve slope as the maximum fitting curve slope, the abnormality duration as the maximum abnormality duration and the fitting curve duration not longer than the preset duration as the detection model.
Preferably, the detecting memory leakage of the target virtual machine by using the detection model includes:
acquiring a to-be-detected data sequence of the preset data type in the target virtual machine;
determining a change point in the data sequence to be detected;
dividing the data sequence to be detected according to the change points to obtain each data subsequence to be detected;
processing each sub-sequence of the data to be detected by using the detection model to obtain a processing result;
and determining a memory leakage detection result according to the processing result.
Preferably, the determining the memory leak detection result according to the processing result includes:
in the case where the current fitting curve may last for no more than the preset duration and the current fitness is no less than a preset fitness,
if the slope of the current fitting curve is not smaller than the maximum slope of the fitting curve and the current abnormity duration is not smaller than the maximum abnormity duration, determining that the to-be-detected data subsequence corresponding to the processing result is an abnormal data subsequence;
if the slope of the current fitting curve is greater than the slope of the maximum fitting curve or the current abnormity duration is greater than the maximum abnormity duration, judging whether fitting data meeting preset conditions exist in all the fitting data, wherein the preset conditions are as follows: the fitted curve slope in the fitted data is less than the current fitted curve slope and the anomaly duration is less than the current anomaly duration;
and if so, determining the data subsequence to be detected corresponding to the processing result as an abnormal data subsequence.
Preferably, the memory leak detection method further includes:
determining the longest abnormal data subsequence in all abnormal data subsequences;
and outputting the longest abnormal data subsequence and the current fitting curve sustainable time corresponding to the longest abnormal data subsequence.
In a second aspect, the present application further discloses a memory leak detection apparatus, including:
the acquisition module is used for acquiring a historical data sequence of a preset data type in the target virtual machine, wherein the preset data type is the memory utilization rate;
a determining module, configured to determine a history change point in the history data sequence;
the dividing module is used for dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
the training module is used for performing model training by utilizing each historical data subsequence to obtain a detection model;
and the detection module is used for detecting the memory leakage of the target virtual machine by using the detection model.
In a third aspect, the present application further discloses a memory leak detection system, including:
a memory for storing a computer program;
a processor for implementing the steps of any of the memory leak detection methods described above when executing the computer program.
In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of any of the memory leak detection methods described above.
The memory leak detection method provided by the application comprises the following steps: acquiring a historical data sequence of a preset data type in a target virtual machine, wherein the preset data type is a memory utilization rate; determining a historical change point in the historical data sequence; dividing the historical data sequence according to the historical change points to obtain each historical data subsequence; performing model training by using each historical data subsequence to obtain a detection model; and carrying out memory leak detection on the target virtual machine by using the detection model.
Obviously, based on the implementation mode, timely and effective memory leak detection can be performed in a production environment, operation and maintenance personnel can be helped to locate the problem of memory leak as early as possible, and the usability and the reliability of the virtual machine are further ensured; in addition, according to the technical scheme, the memory usage rate single index of the virtual machine is only used as a main measurement, and the memory leakage problem can be quickly detected without depending on knowledge and experience of technicians.
The memory leak detection device, the memory leak detection system and the computer-readable storage medium provided by the application all have the beneficial effects, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings that are needed to be used in the description of the prior art and the embodiments of the present application will be briefly described below. Of course, the following description of the drawings related to the embodiments of the present application is only a part of the embodiments of the present application, and it will be obvious to those skilled in the art that other drawings can be obtained from the provided drawings without any creative effort, and the obtained other drawings also belong to the protection scope of the present application.
Fig. 1 is a schematic flow chart illustrating a memory leak detection method according to the present application;
FIG. 2 is a schematic flow chart of a training method for a detection model provided in the present application;
fig. 3 is a schematic flow chart of a method for implementing memory leak detection based on a detection model according to the present application;
fig. 4 is a schematic structural diagram of a memory leak detection apparatus provided in the present application;
fig. 5 is a schematic structural diagram of a memory leak detection system provided in the present application.
Detailed Description
The core of the application is to provide a memory leak detection method, which can realize timely and effective memory leak detection in a production environment, thereby ensuring the availability and reliability of a virtual machine; another core of the present application is to provide a memory leak detection apparatus, system and computer readable storage medium, which also have the above-mentioned advantages.
In order to more clearly and completely describe the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides a memory leak detection method.
Referring to fig. 1, fig. 1 is a schematic flow chart of a memory leak detection method provided in the present application, where the memory leak detection method may include the following steps S101 to S105.
S101: and acquiring a historical data sequence of a preset data type in the target virtual machine, wherein the preset data type is the memory utilization rate.
The method comprises the steps of obtaining historical data, wherein the historical data is data information generated by a target virtual machine in a historical operation process, and the target virtual machine is a virtual machine needing memory leak detection. When memory leak detection needs to be performed on a certain virtual machine, the virtual machine can be used as a target virtual machine, historical data of preset data types on the target virtual machine are collected and combined into a historical data sequence, wherein the preset data types can be memory usage rates, that is, the historical data in the historical data sequence is the historical memory usage rate of the target virtual machine.
When the historical data sequence is collected for the target virtual machine, a historical time period and a time interval for data collection can be preset, for example, the memory utilization rate within seven historical days can be collected, and the historical memory utilization rate is collected every 30 minutes in the collection process, so that all the historical memory utilization rates collected within seven historical days are combined into the historical data sequence of the target virtual machine.
Further, after the data acquisition is completed, the acquired data may be subjected to a preprocessing operation, such as linear interpolation processing, smoothing processing, etc., to obtain a more accurate historical data sequence. It can be understood that, in the data acquisition process, a situation that the acquired data has a missing value may occur, that is, the acquired data are not arranged according to a strict time interval or are different in step length, and at this time, the linear difference of mean values of the original acquired data may be processed; meanwhile, in order to avoid the situation of severe jitter in the sequence, the original collected data can be smoothed to eliminate noise data.
S102: historical change points in the historical data sequence are determined.
The step aims to determine a history change point in the history data sequence, and it can be understood that the change point refers to a point which changes suddenly in the time sequence, and the change point can divide the corresponding time sequence into a plurality of segments, wherein values in each segment have similar characteristics, such as a mean value, a standard deviation, a linear trend and the like, and therefore, the history change point is the point which changes suddenly in the history data sequence.
In an embodiment of the application, the determining the historical change point in the historical data sequence may include the following steps:
performing first-order difference calculation on the historical data sequence to obtain each difference absolute value;
calculating to obtain a standard score corresponding to each difference absolute value;
and determining whether the historical data corresponding to the difference absolute value is a historical change point or not according to the standard score.
The embodiment of the application provides an implementation method for determining a historical change point in a historical data sequence, which can be implemented based on first-order difference calculation and standard score calculation. Firstly, performing first-order difference calculation on a historical data sequence, namely performing difference calculation on any two adjacent historical data of the historical data sequence to obtain corresponding difference values, and then taking absolute values of the difference values to obtain each difference absolute value, wherein obviously, when the data quantity of the historical data in the historical data sequence is n, the number of the difference absolute values is n-1; further, for each difference absolute value, calculating a standard Score (standard Score), which is also called Z-Score (Z-Score), and is a process of dividing the difference between a number and a mean by a standard deviation; finally, whether the corresponding historical data is the historical change point or not can be determined according to the standard score, and specifically, when the standard score is greater than or equal to 3sigma (three sigma criterion), the corresponding historical data can be determined to be the historical change point, so that all the historical change points in the historical data sequence are obtained.
S103: and dividing the historical data sequence according to the historical change points to obtain each historical data subsequence.
The method comprises the steps of dividing a historical data sequence to obtain a plurality of historical data subsequences, so that model training can be conveniently carried out according to each historical data subsequence to obtain a detection model for detecting memory leakage. When the historical data subsequence is divided, the division can be realized according to each historical change point in the historical data sequence.
In an embodiment of the application, the dividing the historical data sequence according to the historical change point to obtain each historical data subsequence may include: for each other history change point except the last history change point in the history data sequence, combining the history data between the other history change point and the last history change point into a history data subsequence.
The embodiment of the application provides a method for dividing a historical data sequence according to historical change points. After determining each history change point in the history data sequence, firstly determining the last history change point in all the history change points, namely the history change point arranged at the last bit according to the arrangement sequence of each history change point in the history data sequence; then, each history change point except the last history change point in all the history change points is used as other history change points, and for each other history change point, the history data between the other history change point and the last history change point can be combined into a history data subsequence corresponding to the other history change point, so as to obtain all the history data subsequences.
For example, it is assumed that there are five history change points in the history data sequence, which are respectively a first history change point, a second history change point, a third history change point, a fourth history change point, and a fifth history change point, where the fifth history change point is an end history change point, and the remaining four history change points are other history change points, so as to obtain the following history data subsequences: the historical data subsequence is obtained by combining the historical data between the first historical change point and the fifth historical change point, the historical data subsequence is obtained by combining the historical data between the second historical change point and the fifth historical change point, the historical data subsequence is obtained by combining the historical data between the third historical change point and the fifth historical change point, and the historical data subsequence is obtained by combining the historical data between the fourth historical change point and the fifth historical change point.
S104: and performing model training by using each historical data subsequence to obtain a detection model.
The method comprises the steps of achieving model training, obtaining a detection model for detecting the memory leak, conducting model training based on machine learning according to each historical data subsequence after each historical data subsequence is obtained based on each historical change point, obtaining the detection model, and achieving the memory leak detection aiming at a target virtual machine based on the detection model.
It can be understood that, in order to ensure the accuracy of the memory leak detection result, the model training process of S101 to S104 may be performed once during each memory leak detection, so as to ensure that the detection model used during the memory leak detection is most suitable for the current state of the target virtual machine, thereby ensuring the accuracy of the detection result.
In an embodiment of the application, the performing model training by using each historical data subsequence to obtain a detection model may include the following steps:
for each historical data subsequence, obtaining fitting data corresponding to the historical data subsequence through linear fitting, wherein the fitting data comprises a fitting curve, a fitting curve slope, fitting curve duration and abnormal duration;
and in all fitting data, taking the fitting data as a detection model, wherein the fitting curve slope is the maximum fitting curve slope, the abnormality duration is the maximum abnormality duration, and the fitting curve duration does not exceed the preset duration.
The embodiment of the application provides a method for realizing training of a detection model. First, after obtaining each historical data subsequence, for each historical data subsequence, linear fitting (LR fitting) may be performed by using each historical data therein to obtain fitting data corresponding to the historical data subsequence, where the fitting data may include, but is not limited to, a fitting curve slope, a fitting curve sustainable time, and an abnormal duration time, where the fitting curve sustainable time is a time at which the memory usage rate of the target virtual machine reaches a minimum defined value based on a curve trend of the fitting curve, and the abnormal duration time is a time at which the curve sustainable trend of the fitting curve is reached. Thus, fitting data corresponding to each historical data subsequence is obtained.
Further, of all the fitting data, fitting data satisfying the following conditions may be searched as a detection model for performing memory leak detection: in the fitting data, the slope of the fitting curve is the maximum slope of the fitting curve, the anomaly duration time is the maximum anomaly duration time, and the fitting curve duration time does not exceed the preset duration time. That is, the slope of the fitting curve and the anomaly duration in the fitting data are both maximized and the duration of the fitting curve is equal to or less than a preset duration. The preset duration is a critical time point, and the specific value is a limit time which enables the memory utilization rate of the target virtual machine to reach a minimum limit value based on the curve trend of the fitting curve.
S105: and carrying out memory leak detection on the target virtual machine by using the detection model.
In this step, the memory leak detection of the target virtual machine is realized, and after the detection model is obtained based on the training of the above steps S101 to S104, the detection model can be directly called to perform the memory leak detection on the target virtual machine, so as to determine whether the memory leak problem occurs in the target virtual machine. It should be noted that the detection model and the virtual machine correspond to each other, that is, for different virtual machines, a detection model adapted to itself needs to be trained.
In an embodiment of the present application, the performing memory leak detection on the target virtual machine by using the detection model may include the following steps:
acquiring a to-be-detected data sequence of a preset data type in a target virtual machine;
determining a change point in a data sequence to be detected;
dividing the data sequence to be detected according to the change points to obtain each data subsequence to be detected;
processing each to-be-detected data subsequence by using a detection model to obtain a processing result;
and determining a memory leak detection result according to the processing result.
The embodiment of the application provides a method for detecting whether a target virtual machine has memory leakage or not based on a detection model. Firstly, data acquisition is carried out on a target virtual machine to obtain a data sequence to be detected, certainly, data to be detected in the data sequence to be detected is also the memory utilization rate, the memory utilization rate can be the memory utilization rate of the target virtual machine in the latest time period (such as the latest two days), and similarly, preprocessing operation can be carried out on the data sequence to be detected after the data acquisition is finished to obtain a more accurate data sequence to be detected; further, determining a change point in the data sequence to be detected, and dividing the change point based on the change point to obtain a plurality of data subsequences to be detected, wherein the process can refer to the model training process, which is not described herein again; and finally, processing each sub-sequence of the data to be detected by using the detection model, namely processing each sub-sequence of the data to be detected in sequence by using the detection model to obtain a processing result corresponding to each sub-sequence of the data to be detected, thereby determining a memory leakage detection result according to each processing result, namely determining whether the target virtual machine has memory leakage.
In an embodiment of the application, the processing result may include a slope of a current fitting curve, a duration of the current fitting curve, a duration of a current anomaly, and a current degree of fitting, and the determining the memory leak detection result according to the processing result may include the following steps:
in the case where the current fitting curve may last for no more than a preset duration and the current fitness is no less than the preset fitness,
if the slope of the current fitting curve is not less than the maximum slope of the fitting curve and the duration of the current abnormity is not less than the maximum abnormity duration, determining that the to-be-detected data subsequence corresponding to the processing result is an abnormal data subsequence;
if the slope of the current fitting curve is greater than the slope of the maximum fitting curve or the current abnormal duration is greater than the maximum abnormal duration, judging whether fitting data meeting preset conditions exist in all the fitting data, wherein the preset conditions are as follows: the slope of a fitting curve in the fitting data is smaller than the slope of the current fitting curve, and the anomaly duration is smaller than the current anomaly duration;
and if so, determining the data subsequence to be detected corresponding to the processing result as an abnormal data subsequence.
The embodiment of the application provides an implementation method for determining a memory leak detection result according to a processing result. Firstly, the processing result of the data subsequence to be detected based on the detection model can comprise the current fitting curve slope, the current fitting curve sustainable time, the current abnormal duration and the current fitting degree corresponding to the current data subsequence to be detected; further, it can be determined whether the corresponding data subsequence to be detected is an abnormal data subsequence based on the data information:
for all processing results, the processing results of which the current fitting curve duration is not greater than the preset duration and the current fitting degree is not less than the preset fitting degree can be searched first, and the screened processing results are further identified (of course, the to-be-detected data subsequence corresponding to the processing result which does not satisfy the conditions can be regarded as a normal data subsequence). On the basis, judging whether the slope of the current fitting curve and the current abnormal duration of each screened processing result meet the following conditions: if the current fitting curve slope is not less than the maximum fitting curve slope and the current abnormity duration is not less than the maximum abnormity duration, determining that the to-be-detected data subsequence corresponding to the processing result is an abnormal data subsequence; if the two can not satisfy the above conditions at the same time, a further determination may be made, that is, all fitting data generated during the model training process are reviewed, and whether fitting data satisfying a preset condition exists in the fitting data is determined, where the preset condition is: and if the fitting curve slope in the fitting data is smaller than the current fitting curve slope and the exception duration is smaller than the current exception duration, determining that the to-be-detected data subsequence corresponding to the processing result is an exception data subsequence, otherwise determining that the to-be-detected data subsequence corresponding to the processing result is a normal data subsequence. Thus, determination of the memory leak detection result is achieved based on the processing result.
In an embodiment of the present application, the memory leak detection method may further include the following steps:
determining the longest abnormal data subsequence in all abnormal data subsequences;
and outputting the longest abnormal data subsequence and the current fitting curve sustainable time corresponding to the longest abnormal data subsequence.
The memory leak detection method provided by the embodiment of the application can further realize exception prompting. After memory leak detection is completed on a target virtual machine, whether each data subsequence to be detected is an abnormal data subsequence or not is determined, all detected abnormal data subsequences can be screened out to obtain an abnormal data subsequence with the longest data length, namely the longest abnormal data subsequence, and the longest abnormal data subsequence and the corresponding current fitting curve sustainable time are output, wherein the current fitting curve sustainable time is used for informing operation and maintenance personnel of the time when the memory usage rate of the target virtual machine reaches the lowest limit value based on the curve trend of the current fitting curve, so that the operation and maintenance personnel can solve the memory leak problem in time in the time period, and normal operation of the target virtual machine is guaranteed.
Obviously, based on the implementation mode, the memory leak detection can be timely and effectively carried out in a production environment, operation and maintenance personnel can be helped to locate the memory leak problem as early as possible, and the usability and the reliability of the virtual machine are further ensured; in addition, the technical scheme only needs to use a single index of the memory utilization rate of the virtual machine as a main measurement, and the memory leakage problem can be quickly detected without depending on knowledge and experience of technicians.
On the basis of the foregoing embodiment, another memory leak detection method is provided in the embodiments of the present application, and an implementation flow thereof is as follows:
the memory leak detection of each virtual machine mainly comprises two stages: an online training phase and an online prediction phase. For each virtual machine, memory usage rate data of the virtual machine in the first seven days can be acquired for realizing memory leak detection, wherein the data of the first five days are used for online training, and the data of the last two days are used for online detection.
1. And (3) an online learning stage:
referring to fig. 2, fig. 2 is a schematic flow chart of a detection model training method provided in the present application, and the implementation flow mainly includes:
1. data acquisition and preprocessing:
the method comprises the steps of obtaining the memory utilization rate of the virtual machine from the first seven days to the first two days to obtain a historical data sequence, and carrying out preprocessing operation on the historical data sequence, wherein the preprocessing operation includes but is not limited to linear interpolation processing, smoothing processing and the like.
2. Determining a historical change point in the historical data sequence:
(1) And performing first-order difference on the historical data sequence, and taking absolute values to obtain absolute values of the differences.
(2) And calculating z-score of each difference absolute value, and if the z-score is more than or equal to 3sigma, considering the data point as a historical change point.
Thus, a historical change point set of the historical data sequence is obtained:
CP＝{CP 1 ,CP 2 ,...,CP k }；
wherein k is less than or equal to n-1,n is the sequence length of the historical data sequence.
3. Linear fitting:
(1) Initializing model parameters, setting an Rmin threshold (preset fitting degree), setting maximum abnormal duration Dmax =0, setting maximum fitting curve slope Smax =0, setting a value limit (the lowest limit value of the memory usage rate) with the highest memory usage rate, and setting a critical time point C (preset duration), wherein C represents that if the curve trend of the fitting curve enables the time of the memory usage rate reaching limit to be smaller than C, the sequence is considered to have the risk of memory leakage.
(2) Acquiring a historical change point set CP, acquiring historical data subsequences corresponding to the historical change points, and performing model training based on the historical data subsequences to obtain each training result, namely: and each historical data subsequence corresponds to fitting data.
(3) And (3) acquiring a training result through the training process in the step (2), wherein the training result comprises a fitting curve Trend with the maximum abnormal duration Dmax, the maximum fitting curve slope Smax and the fitting curve duration Tb less than or equal to C.
3. And (3) an online detection stage:
referring to fig. 3, fig. 3 is a schematic flow chart of the method for implementing memory leak detection based on the detection model provided in the present application, and the implementation flow mainly includes:
1. data acquisition and preprocessing
And acquiring the memory utilization rate of the virtual machine from the first two days to the current day to obtain a data sequence to be detected, and performing preprocessing operation on the data sequence to be detected, wherein the preprocessing operation includes but is not limited to linear interpolation processing, smoothing processing and the like.
2. Determining change points in the data sequence to be detected:
reference is made to the corresponding process of the on-line training phase.
3. And (3) detecting memory leakage:
for all the change points in the data sequence to be detected, except the last change point, fitting the data subsequence to be detected between other change points and the last change point in sequence (the sequence length can be in the order from small to large), and obtaining the processing result of each data subsequence to be detected, wherein the processing result comprises: the slope s of the current fitting curve, the sustainable time of the current fitting curve, the current abnormal duration d and the current fitting degree r2 of each processing result can be judged as follows:
(1) Time in the processing result is less than or equal to C, and r2 is more than or equal to Rmin:
a. and judging whether s and d reach the following conditions: s is not less than Smax and d is not less than Dmax; if so, marking the data subsequence to be detected corresponding to the processing result as an abnormal data subsequence;
b. if not, traversing the historical fitting curve ht belongs to Trends (obtained in an online training stage), and judging whether a fitting curve meeting the following conditions exists: s is not less than S ht And D is not less than D ht If the data subsequence corresponding to the processing result exists, the data subsequence to be detected corresponding to the processing result is marked as an abnormal data subsequence, and if the data subsequence does not exist, the data subsequence to be detected corresponding to the processing result is marked as a normal data subsequence.
(2) Time > C and r2 < Rmin in the treatment results: and marking the data subsequence to be detected corresponding to the processing result as a normal data subsequence.
(3) And finally, after traversing, acquiring the longest abnormal data subsequence and the time corresponding to the longest abnormal data subsequence, outputting and generating an alarm at the same time so as to inform operation and maintenance personnel to solve the problem of memory leakage of the virtual machine as soon as possible.
Obviously, based on the implementation mode, the memory leak detection can be timely and effectively carried out in a production environment, operation and maintenance personnel can be helped to locate the memory leak problem as early as possible, and the usability and the reliability of the virtual machine are further ensured; in addition, according to the technical scheme, the memory usage rate single index of the virtual machine is only used as a main measurement, and the memory leakage problem can be quickly detected without depending on knowledge and experience of technicians.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a memory leak detection apparatus provided in the present application, where the memory leak detection apparatus may include:
the system comprises an acquisition module 1, a storage module and a processing module, wherein the acquisition module 1 is used for acquiring a historical data sequence of a preset data type in a target virtual machine, and the preset data type is the memory utilization rate;
a determining module 2, configured to determine a historical change point in the historical data sequence;
the dividing module 3 is used for dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
the training module 4 is used for performing model training by utilizing each historical data subsequence to obtain a detection model;
and the detection module 5 is used for performing memory leak detection on the target virtual machine by using the detection model.
Obviously, the memory leak detection device provided by the embodiment of the application performs machine learning training by using historical data in the target virtual machine to obtain a detection model for realizing memory leak detection, and further realizes the memory leak detection of the target virtual machine by using the detection model, so that the online detection based on machine learning is realized, and obviously, based on the realization mode, the memory leak detection can be timely and effectively performed in a production environment, operation and maintenance personnel can be helped to locate the problem of memory leak as early as possible, and the usability and reliability of the virtual machine are further ensured; in addition, according to the technical scheme, the memory usage rate single index of the virtual machine is only used as a main measurement, and the memory leakage problem can be quickly detected without depending on knowledge and experience of technicians.
In an embodiment of the present application, the determining module 2 may include:
the first calculation unit is used for performing first-order difference calculation on the historical data sequence to obtain each difference absolute value;
the second calculating unit is used for calculating and obtaining a standard score corresponding to each difference absolute value;
and the judging unit is used for determining whether the historical data corresponding to the difference absolute value is a historical change point according to the standard score.
In an embodiment of the present application, the dividing module 3 may be specifically configured to, for each other history change point in the history data sequence except for the last history change point, combine the history data between the other history change point and the last history change point into the history data subsequence.
In an embodiment of the present application, the training module 4 may include:
the linear fitting unit is used for obtaining fitting data corresponding to the historical data subsequences through linear fitting for each historical data subsequence, and the fitting data comprise a fitting curve, a fitting curve slope, fitting curve duration and abnormal duration;
and the model generation unit is used for taking the fitting data of which the fitting curve slope is the maximum fitting curve slope, the abnormal duration is the maximum abnormal duration and the fitting curve duration does not exceed the preset duration as the detection model in all the fitting data.
In an embodiment of the present application, the detection module 5 may include:
the acquisition unit is used for acquiring a to-be-detected data sequence of a preset data type in the target virtual machine;
a determining unit for determining a change point in the data sequence to be examined;
the dividing unit is used for dividing the data sequence to be detected according to the change points to obtain each data subsequence to be detected;
the processing unit is used for processing each to-be-detected data subsequence by using the detection model to obtain a processing result;
and the judging unit is used for determining the memory leakage detection result according to the processing result.
In an embodiment of the application, the processing result may include a current fitting curve slope, a current fitting curve sustainable time, a current anomaly duration, and a current degree of fitting, and the determining unit may be specifically configured to determine, if the current fitting curve sustainable time is not greater than a preset duration and the current degree of fitting is not less than the preset degree of fitting, that the to-be-detected data subsequence corresponding to the processing result is an abnormal-state data subsequence if the current fitting curve slope is not less than the maximum fitting curve slope and the current anomaly duration is not less than the maximum anomaly duration; if the slope of the current fitting curve is greater than the slope of the maximum fitting curve or the current abnormal duration is greater than the maximum abnormal duration, judging whether fitting data meeting preset conditions exist in all the fitting data, wherein the preset conditions are as follows: the slope of a fitting curve in the fitting data is smaller than the slope of the current fitting curve, and the anomaly duration is smaller than the current anomaly duration; and if so, determining the data subsequence to be detected corresponding to the processing result as an abnormal data subsequence.
In an embodiment of the present application, the detecting module 5 may further include an output unit, configured to determine a longest abnormal data subsequence in all abnormal data subsequences; and outputting the longest abnormal data subsequence and the current fitting curve sustainable time corresponding to the longest abnormal data subsequence.
For the introduction of the apparatus provided in the present application, please refer to the above method embodiments, which are not described herein again.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a memory leak detection system provided in the present application, where the memory leak detection system may include:
a memory for storing a computer program;
the processor, when executing the computer program, may implement any of the steps of the memory leak detection methods described above.
As shown in fig. 5, which is a schematic diagram of a composition structure of a memory leak detection system, the memory leak detection system may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through a communication bus 13.
In the embodiment of the present application, the processor 10 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, etc.
The processor 10 may call a program stored in the storage 11, and in particular, the processor 10 may perform operations in an embodiment of the memory leak detection method.
The memory 11 is used for storing one or more programs, the program may include program codes, the program codes include computer operation instructions, in this embodiment, the memory 11 stores at least the program for implementing the following functions:
acquiring a historical data sequence of a preset data type in a target virtual machine, wherein the preset data type is a memory utilization rate;
determining a historical change point in the historical data sequence;
dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
performing model training by using each historical data subsequence to obtain a detection model;
and carrying out memory leak detection on the target virtual machine by using the detection model.
In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created during use.
Further, the memory 11 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device.
The communication interface 12 may be an interface of a communication module for connecting with other devices or systems.
Of course, it should be noted that the structure shown in fig. 5 does not constitute a limitation of the memory leak detection system in the embodiment of the present application, and in practical applications, the memory leak detection system may include more or less components than those shown in fig. 5, or some components in combination.
The present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the memory leak detection methods described above can be implemented.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of the computer-readable storage medium provided in the present application, please refer to the above method embodiments, which are not described herein again.
The embodiments are described in a progressive mode in the specification, the emphasis of each embodiment is on the difference from the other embodiments, and the same and similar parts among the embodiments can be referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The technical solutions provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principle of the present application, the present application can also make several improvements and modifications, and those improvements and modifications also fall into the protection scope of the present application.
Claims (10)
1. A memory leak detection method, comprising:
acquiring a historical data sequence of a preset data type in a target virtual machine, wherein the preset data type is a memory utilization rate;
determining a historical change point in the historical data sequence;
dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
performing model training by using each historical data subsequence to obtain a detection model;
and performing memory leak detection on the target virtual machine by using the detection model.
2. The memory leak detection method of claim 1, wherein the determining the historical change point in the historical data sequence comprises:
performing first-order difference calculation on the historical data sequence to obtain each difference absolute value;
calculating to obtain a standard score corresponding to each difference absolute value;
and determining whether the historical data corresponding to the differential absolute value is the historical change point or not according to the standard score.
3. The method according to claim 1, wherein the dividing the historical data sequence according to the historical change point to obtain each historical data subsequence comprises:
for each other historical change point except for the last historical change point in the historical data sequence, combining the historical data between the other historical change points and the last historical change point into the historical data subsequence.
4. The memory leak detection method according to claim 1, wherein the performing model training using each of the historical data subsequences to obtain a detection model comprises:
for each historical data subsequence, obtaining fitting data corresponding to the historical data subsequence through linear fitting, wherein the fitting data comprises a fitting curve, a fitting curve slope, fitting curve duration and abnormal duration;
and in all the fitting data, taking the fitting data with the fitting curve slope as the maximum fitting curve slope, the abnormality duration as the maximum abnormality duration and the fitting curve duration not longer than the preset duration as the detection model.
5. The memory leak detection method according to claim 4, wherein the performing memory leak detection on the target virtual machine by using the detection model includes:
acquiring a to-be-detected data sequence of the preset data type in the target virtual machine;
determining a change point in the data sequence to be detected;
dividing the data sequence to be detected according to the change points to obtain each data subsequence to be detected;
processing each sub-sequence of the data to be detected by using the detection model to obtain a processing result;
and determining a memory leakage detection result according to the processing result.
6. The method according to claim 5, wherein the processing result includes a current fitting curve slope, a current fitting curve duration, a current anomaly duration, and a current fitness, and the determining the memory leak detection result according to the processing result includes:
in the case where the current fitting curve may last for no more than the preset duration and the current fitness is no less than a preset fitness,
if the slope of the current fitting curve is not smaller than the slope of the maximum fitting curve and the current abnormal duration is not smaller than the maximum abnormal duration, determining that the to-be-detected data subsequence corresponding to the processing result is an abnormal data subsequence;
if the slope of the current fitting curve is greater than the slope of the maximum fitting curve or the current abnormal duration is greater than the maximum abnormal duration, judging whether fitting data meeting preset conditions exist in all the fitting data, wherein the preset conditions are as follows: the fitted curve slope in the fitted data is less than the current fitted curve slope and the anomaly duration is less than the current anomaly duration;
and if so, determining the to-be-detected data subsequence corresponding to the processing result as an abnormal data subsequence.
7. The memory leak detection method of claim 6, further comprising:
determining the longest abnormal data subsequence in all abnormal data subsequences;
and outputting the longest abnormal data subsequence and the current fitting curve sustainable time corresponding to the longest abnormal data subsequence.
8. A memory leak detection apparatus, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a historical data sequence of a preset data type in a target virtual machine, and the preset data type is the memory utilization rate;
a determining module, configured to determine a history change point in the history data sequence;
the dividing module is used for dividing the historical data sequence according to the historical change points to obtain each historical data subsequence;
the training module is used for performing model training by utilizing each historical data subsequence to obtain a detection model;
and the detection module is used for detecting the memory leakage of the target virtual machine by using the detection model.
9. A memory leak detection system, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the memory leak detection method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the memory leak detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211625088.7A CN115828244A (en) | 2022-12-16 | 2022-12-16 | Memory leak detection method and device and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211625088.7A CN115828244A (en) | 2022-12-16 | 2022-12-16 | Memory leak detection method and device and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115828244A true CN115828244A (en) | 2023-03-21 |
Family
ID=85516384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211625088.7A Pending CN115828244A (en) | 2022-12-16 | 2022-12-16 | Memory leak detection method and device and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115828244A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117687769A (en) * | 2023-06-02 | 2024-03-12 | 荣耀终端有限公司 | Memory repair and cleaning method and related equipment |
-
2022
- 2022-12-16 CN CN202211625088.7A patent/CN115828244A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117687769A (en) * | 2023-06-02 | 2024-03-12 | 荣耀终端有限公司 | Memory repair and cleaning method and related equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106156355B (en) | Log processing method and device | |
| CN110635962B (en) | Abnormity analysis method and device for distributed system | |
| CN109857618B (en) | Monitoring method, device and system | |
| CN105095912A (en) | Data clustering method and device | |
| CN112181430B (en) | Code change statistical method, device, electronic equipment and storage medium | |
| CN114327983A (en) | Log-based fault determination method, device, equipment and medium | |
| CN115828244A (en) | Memory leak detection method and device and related equipment | |
| JP2019215847A (en) | Reducing buffer overflow | |
| US9645873B2 (en) | Integrated configuration management and monitoring for computer systems | |
| CN109213476A (en) | A kind of generation method of installation kit, computer readable storage medium and terminal device | |
| CN113886237A (en) | Analysis report generation method and device, electronic equipment and storage medium | |
| US9734204B2 (en) | Managed runtime cache analysis | |
| CN116166967B (en) | Data processing method, equipment and storage medium based on meta learning and residual error network | |
| CN112463564A (en) | Method and device for determining correlation index influencing host state | |
| CN116108449B (en) | Software fuzzy test method, device, equipment and storage medium | |
| CN109254898B (en) | Software module execution sequence monitoring method and system | |
| CN115248783B (en) | Software testing method, system, readable storage medium and computer equipment | |
| CN108470242B (en) | Risk management and control method, device and server | |
| CN111752819A (en) | Abnormity monitoring method, device, system, equipment and storage medium | |
| CN115525455A (en) | Method, device, server and storage medium for repairing abnormality of multi-level system | |
| CN109284354B (en) | Script searching method and device, computer equipment and storage medium | |
| CN111104963B (en) | Target user determining method and device, storage medium and electronic equipment | |
| CN113934566A (en) | Exception handling method and device and electronic equipment | |
| CN112907257A (en) | Risk threshold determining method, abnormality detecting device and electronic equipment | |
| CN112799911A (en) | Node health state detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |