CN111190788B - Data monitoring method and device, electronic equipment and readable medium - Google Patents
Data monitoring method and device, electronic equipment and readable medium Download PDFInfo
- Publication number
- CN111190788B CN111190788B CN201911275685.XA CN201911275685A CN111190788B CN 111190788 B CN111190788 B CN 111190788B CN 201911275685 A CN201911275685 A CN 201911275685A CN 111190788 B CN111190788 B CN 111190788B
- Authority
- CN
- China
- Prior art keywords
- data
- output
- time period
- estimated
- resource
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a data monitoring method, a data monitoring device, an electronic device and a readable medium. This application is through acquireing the data resource of treating monitoring and with the first output time quantum of presupposing that data resource corresponds, judge whether data resource is in first output data of generating in the output time quantum of presupposing, if data resource is not in first output data of generating in the output time quantum of presupposing sends the unusual alarm of first output, reaches the unusual purpose of quick notice relevant personnel data output, through the production time of presupposing that sets up data resource, when not presupposing output in the output time quantum with data resource, generate the unusual alarm of output to relevant personnel can investigate the trouble as early as possible, promote maintenance efficiency.
Description
Technical Field
The present application relates to the field of data monitoring technologies, and in particular, to a data monitoring method and apparatus, an electronic device, and a readable medium.
Background
With the advent of the big data era, the types and the quantity of data resources are increasing, and the dependency relationship between the data resources becomes complicated, so that the management of the data center is difficult to some extent.
At present, a dependency relationship may exist between different data resources in a data center, and when a certain data resource does not generate first data, second data that can be generated only depending on the data cannot be produced, and at this time, a service person of the data center generally further checks to further determine a data resource that fails.
However, when the types and the number of the data resources are continuously increased, the difficulty of determining the failed data resource output channel by the service personnel is increased, and the related personnel can be notified to the failure point for maintenance after several hours, so that the efficiency is very low.
Disclosure of Invention
In order to solve the technical problems described above or at least partially solve the technical problems, the present application provides a data monitoring method, an apparatus, an electronic device, and a readable medium.
In a first aspect, the present application provides a data monitoring method, including:
acquiring a data resource to be monitored and a first estimated output time period corresponding to the data resource;
judging whether the data resource generates first output data in the first estimated output time period or not;
and if the data resource does not generate first output data in the first estimated output time period, sending a first output abnormity alarm.
Optionally, the method further comprises:
judging whether the data resource outputs the first output data or not;
if second yield data which is dependent exists when the data resource generates the first yield data, judging whether the second yield data is generated;
if the second output data is generated, acquiring a second estimated output time period and output moment of the second output data;
and if the output moment of the second output data is within the second estimated output time period, executing a step of sending a first output abnormity alarm.
Optionally, the method further comprises:
if the output moment of the second output data is not within the second estimated output time period, judging whether third output data which is dependent exists when the second output data is generated;
and if no dependent third output data exists when the second output data is generated, generating an output abnormity alarm of the second output data.
Optionally, the method further comprises:
acquiring an identifier of the first output data and an identifier of the second output data;
generating dependency relationship information according to the identifier of the first output data and the identifier of the second output data;
generating a notification of abnormal output according to the dependency relationship information;
and sending the output abnormity notification to a first alarm receiving end corresponding to the data resource for a maintainer to check.
Optionally, after the step of sending the yield anomaly alarm to a first alarm receiving end corresponding to the first yield data, the method further includes:
if the acknowledgement information which is sent by the first alarm receiving end and responds to the abnormal output alarm is not received, acquiring a data output period corresponding to the data resource;
calculating the sum of the data output period and the estimated output time period to obtain a third estimated output time period;
judging whether the newly produced data of the data resource is produced in the third estimated production time period after one production period;
and if the newly produced data of the data resource is not produced in the newly estimated production time period, sending an abnormal production alarm to a second alarm receiving terminal corresponding to the data resource, wherein the second alarm receiving terminal is different from the first alarm receiving terminal.
Optionally, the method further comprises:
recording the actual yield time of the first yield data;
if the actual output time of the first output data is behind the estimated output time period, calculating a difference value between the actual output time and the end moment of the estimated output time period;
and if the difference is larger than a preset threshold value, sending a prompt of abnormal output.
Optionally, the method further comprises:
acquiring actual output time of a plurality of first output data generated by the same data resource of a historical record;
calculating the difference value of the adjacent actual output time of every two random times;
calculating an average value of the difference values to obtain an adjustment value;
calculating the sum of the first estimated output time period and the adjustment value to obtain a new first estimated output time period;
and replacing the first estimated output time period with a new first estimated output time period.
In a second aspect, the present application provides a data resource monitoring apparatus, including:
the system comprises an acquisition module, a monitoring module and a judgment module, wherein the acquisition module is used for acquiring data resources to be monitored and a first estimated output time period corresponding to the data resources;
the judging module is used for judging whether the data resources generate first output data within the first estimated output time period;
and the sending module is used for sending a first output abnormity alarm if the data resource does not generate first output data in the first pre-estimated output time period.
In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and the processor implements the steps of the method according to any one of the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method of any of the first aspects.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages: this application is through obtaining the data resource that waits to monitor and with the first output time quantum of presupposing that data resource corresponds, judge whether data resource is in first output data of generating in the first output time quantum of presupposing, if data resource is not in first output data of generating in the first output time quantum of presupposing sends the unusual alarm of first output, reaches the unusual purpose of quick notice relevant personnel data output.
According to the embodiment of the invention, through setting the estimated production time of the data resources, when the output data of the data resources are not output in the estimated production time period, an output abnormity alarm is generated, so that related personnel can check the fault as soon as possible, and the overhaul efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a data monitoring method according to an embodiment of the present application;
fig. 2 is another schematic flow chart of a data monitoring method according to an embodiment of the present disclosure;
fig. 3 is a schematic flowchart of another data monitoring method according to an embodiment of the present disclosure;
fig. 4 is a schematic flowchart of another data monitoring method according to an embodiment of the present disclosure;
fig. 5 is a schematic flowchart of another data monitoring method according to an embodiment of the present application;
fig. 6 is a schematic flowchart of another data monitoring method according to an embodiment of the present disclosure;
fig. 7 is a block diagram of a data monitoring apparatus according to an embodiment of the present disclosure;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the current big data era, with increasing data demands, the types of data resources are increasing, and at the same time, the dependency relationship between the data resources is also becoming complicated, for example, in a data center of an e-commerce, there may be a dependency relationship between one data resource and dozens or hundreds of data resources, in this case, if the data resource does not produce data, the data resource having a dependency relationship with the data resource cannot produce data, and unpredictable loss is caused to a data application party (a recommendation system, an inspection system, a BI system, etc.). Based on this, an embodiment of the present invention provides a data monitoring method, where the data monitoring method may be applied to a data center, and the data monitoring method may be applied to a server, that is, steps in the following method embodiments are all performed by the server, as shown in fig. 1, where the method includes:
step S101, acquiring a data resource to be monitored and a first estimated output time period corresponding to the data resource;
in this step, the data resource refers to a specific data or data set, such as: a Hadoop distributed system infrastructure file (exposure log, search log, click log), a Hive data warehouse tool table, or a MySQL relational database management system table, etc.
The first estimated output time period corresponding to the data resource may be manually preset, for example: according to the historical experience value, it can be determined that the data resource a produces at 11 am 10 min each day in most cases, and the estimated production time period of the data resource a can be set from 10 am 50 min to 11 am 20 min each day, for example: under the condition that a large number of data resources a are determined according to historical experience values, and the data resources a are output at 10 minutes every Monday morning, the estimated output time period of the data resources a may be set to be 10 minutes every Monday morning to 20 minutes every 10 am, and the specific set monitoring period and the monitoring duration may be determined according to actual conditions, which is not specifically limited in the embodiment of the present invention.
In the embodiment of the present invention, the data resource to be monitored may be set manually, and specifically, the data resource may be acquired after the server identifies the identifier by inputting the identifier of the data resource, so as to monitor the data resource, where the manner of acquiring the data resource to be monitored may be determined according to an actual situation, specifically, may be acquired from a database for storing the data resource, or may be acquired from a terminal device for storing the data resource.
Step S102, judging whether the data resource generates first output data in the first estimated output time period;
in the implementation of the present invention, the data resource may correspond to first output data of a plurality of outputs, and the first output data may indicate specific contents of the data resource, for example: the data resource is a Hive table, and the first yield data produced by the data resource may be a partition corresponding to a certain day in the Hive table, for example: if the data resource is a MySQL file, the first output data generated by the data resource may be some data in the MySQL table.
In the embodiment of the invention, a time mode of recording output data when the data resources output data can be adopted to judge whether the output data of the data resources are output in the estimated output time period, so that the purpose of monitoring the output condition of each data resource is achieved, and the condition that the data resources do not output data can be rapidly found.
Step S103, if the data resource does not generate first output data in the first estimated output time period, sending a first output abnormity alarm;
in the embodiment of the present invention, the first output anomaly alarm is information for prompting a service worker responsible for the data resource to perform maintenance, and specific content may be determined according to an actual situation, which is not specifically limited in the embodiment of the present invention.
In the actual application process, the time point of program execution may be after server initialization, step S101 is executed, at this time, all data resources to be monitored do not start to produce data, that is, whether the data resources produce data is monitored is achieved, when the data resources do not generate first output data within a first estimated output time period set in advance by a human, the purpose of prompting relevant maintenance personnel is achieved by sending a first output abnormal alarm, and the time point of specific program execution may be determined according to an actual situation, for example: after the first output data is generated by the data resource N, step S101 is executed to list the data resource in a monitoring list, which is not limited in this embodiment of the present invention,
according to the invention, the estimated production time of each data resource is set, so that each data resource is monitored, and when the data resource does not produce data in the estimated production time period, an abnormal output alarm is generated, so that related personnel can check the fault as soon as possible, and the overhaul efficiency is improved.
Currently, there may be dependencies between different data resources in a data center, such as: the data resource a depends on the data resource B, and the data resource B depends on the data resource C, which means that the data resource B can be produced only when the data resource C is produced, and finally the data resource a can be produced, and when the data resource a is not produced, the data resource B may not be produced, or the data resource C may not be produced, at this time, a service person in the data center generally further checks to determine a data resource production channel with a fault, so based on this, the embodiment of the present invention further provides another data monitoring method, as shown in fig. 2, the method includes:
step S201, judging whether the data resource outputs the second output data depended on when the first output data is output;
in the embodiment of the present invention, the data resources corresponding to the second output data may be different from the data resources corresponding to the first output data, and when the first output data depends on the second output data, if the second output data is not generated, the data resources cannot generate the first output data.
If yes, step S202, determining whether the second output data is generated;
in the embodiment of the present invention, if there is second yield data that the data resource depends on when producing the first yield data, it is determined whether the second yield data is generated, and it is determined whether the first yield data is not generated because the second yield data is not generated.
If not, executing step S103, and sending a first output abnormity alarm;
in the embodiment of the present invention, if there is no second yield data that the data resource depends on when producing the first yield data, it may be determined that an abnormality occurs in the process of producing the data by the data resource, step S103 is executed, and a step of sending an alarm of the first yield abnormality is performed;
if yes, step S203, obtaining a second estimated output time period and output time of the second output data;
in the embodiment of the present invention, if the second output data is generated, a second estimated output time period and output time of the second output data are obtained, where the output time is a time recorded by the server when the second output data is generated, the second estimated output time period is an estimated output time period of the data resource corresponding to the second output data, and a setting of the specific estimated output time period may be determined according to an actual situation.
Step S204, if the output time of the second output data is within the second estimated output time period, executing step S103, and sending a first output abnormity alarm;
in the embodiment of the present invention, by determining whether the output time of the second output data is generated within the second estimated output time period, it may be determined whether the second output data is not generated within the second estimated output time period, so that the first output data having a dependency relationship with the second output data cannot be generated within the first estimated output time period.
If not, step S205 determines whether there is dependent third output data when generating the second output data;
in the embodiment of the present invention, if the second yield data is not generated within the second estimated yield time period, it may be that the data resource of the second yield data fails, or it may be that the data resource corresponding to the third yield data that the second yield data depends on fails, and similarly, with reference to the above-mentioned embodiment, it may be further determined whether there is dependent third yield data when the second yield data is generated.
If so, judging whether the third output data is generated, and similarly, referring to the steps in the subsequent specific working process;
if not, step S206, generating an abnormal output alarm of the second output data;
in the embodiment of the present invention, if there is no third output data that is dependent on the second output data when the second output data is generated, it may be determined that the data resource corresponding to the second output data has a fault, that is, the data resource corresponding to the second output data does not generate the second output data in the second estimated output time period, so that an output abnormality alarm of the second output data is generated to prompt a service person of the data resource corresponding to the second output data. The yield anomaly alarm of the second yield data may include a data resource identifier corresponding to the second yield data, or other preset content.
According to the embodiment of the invention, the technical effect of tracing the source is achieved by searching the output data which is not generated and on which the output data depends, so that the root cause of the data resources which do not generate the output data can be determined, the problem of quick maintenance of the responsible person of the failed data resources can be quickly prompted, and the maintenance efficiency is improved.
In an actual application scenario, a data resource a exists, an estimated output time period of the data resource a is 11 pm to 11 pm 30 min, if the data resource a does not generate first output data a1 from 11 pm to 11 pm 30 min, it is determined whether second output data B1 on which the first output data a1 depends is generated in a second estimated output time period, where the second output data B1 is generated by the data resource B, if so, it may be determined that the data resource a fails and the first output data a1 cannot be generated from 11 pm to 11 pm 30 min, if not, it may be that the data resource of the second output data B1 fails, or that the data resource corresponding to third output data C1 on which the second output data depends on B1 fails, so it is first determined whether third output data C1 depending on which exists when the second output data B1 is generated, if the output data exists, whether the output data depended on exists when the third output data C1 is generated is continuously judged, if the output data does not exist, the data resource B is determined to be in fault, and then an output abnormity alarm of the second output data is generated to prompt a maintainer in charge of managing the data resource B to overhaul as soon as possible, so that the generation of other output data is prevented from being influenced.
In the embodiment of the present invention, as shown in fig. 3, the method further includes:
step S301, acquiring an identifier of the first output data and an identifier of the second output data;
in the embodiment of the present invention, the identifier of the first yield data and the identifier of the second yield data may be determined according to actual situations, for example: the name of the data resource identified by the first yield data is uv _ log, and the generation cycle is one day, the identifier of the first yield data may be uv _ log.20191028, and the identifier of the first data generated in the next generation cycle may be uv _ log.20102919.
Step S302, generating dependency relationship information according to the identifier of the first output data and the identifier of the second output data;
in the embodiment of the invention, the dependency relationship information can comprise the identifier of the first output data, the identifier of the second output data and the dependency relationship between the first output data and the second output data.
Step S303, generating a yield abnormity notification according to the dependency relationship information;
in the embodiment of the present invention, the output anomaly notification may include dependency relationship information, and may further include preset prompt content, identification information of a failed data resource, and the like, and the specific setting condition may be determined according to an actual situation.
Step S304, sending the output abnormity notification to a first alarm receiving end corresponding to the data resource for a maintainer to check;
in the embodiment of the invention, one data resource can correspond to a plurality of alarm receiving terminals which are manually preset, wherein the first alarm receiving terminal can be the receiving terminal with the highest priority.
In addition, in an actual application process, the inventor finds that a worker responsible for data resource maintenance generally receives an output abnormal alarm through a mobile terminal, and may not be able to timely know a data resource fault message due to a fault of the mobile terminal or a human factor (no answer or check), so as to delay generation of output data of a next cycle of a data resource, based on this, an embodiment of the present invention further provides another data monitoring method, and in step S304, the output abnormal notification is sent to a first alarm receiving end corresponding to the data resource, so that after a maintainer checks the output abnormal notification, as shown in fig. 4, the method further includes:
step S401, if receiving no acknowledgement message sent by the first alarm receiving end and responding to the abnormal output alarm, acquiring a data output period corresponding to the data resource;
in the embodiment of the invention, the acknowledgement information can refer to response information fed back by a worker in charge of maintenance after receiving the output abnormity alarm through the first alarm receiving terminal.
In the embodiment of the present invention, the data output period refers to a duration of the data resource generating the output data, for example, if the data resource a generates one output data every seven days, the data output period of the data resource a is seven days.
Step S402, calculating the sum of the data output period and the estimated output time period to obtain a third estimated output time period;
in the embodiment of the present invention, the sum of the data output period and the estimated output time period is an estimated output time period for the next generation of output data by the data resource, that is, a third estimated output time period, for example: the estimated output time period is from 10 am to 10 am 30 min on the T day, the data output period is one day, the sum of the data output period and the estimated output time period is from 10 am to 10 am 30 min on the T +1 day, namely, the third estimated output time period is from 10 am to 10 am 30 min on the T +1 day.
Step S403, determining whether the newly produced data of the data resource is produced within the third estimated production time period after a production cycle;
in the embodiment of the present invention, by determining whether the newly produced data of the data resource is produced in the third estimated production time period after a production period, it may be determined whether the data resource normally produces the production data according to the production period.
If so, setting the third estimated output time period as the estimated output time period, and re-executing step S402 to calculate whether the newly generated data of the data resource is generated within the new estimated output time period after the next output cycle.
If not, step S404, sending an output anomaly alarm to a second alarm receiving end corresponding to the data resource, where the second alarm receiving end is different from the first alarm receiving end;
in the embodiment of the present invention, the second alarm receiving terminal may refer to a terminal with a lower priority than the first alarm receiving terminal, and when the first alarm receiving terminal does not return the acknowledgement information, in order to ensure that the maintenance personnel can perform maintenance as soon as possible, after the next production period, the production abnormal alarm is sent to the second alarm receiving terminal, and so on, a plurality of alarm receiving terminals (the third alarm receiving terminal and the fourth alarm receiving terminal … …) may be further provided to ensure that the maintenance work can be performed as soon as possible.
In addition, for the problem that the difference between the actual output time and the estimated output time of the output data is too large, the estimated output time may be set to be poor, or the actual output time may be changed due to a change in the execution process of generating the output data by the data resource, based on which, the embodiment of the present invention further provides another data monitoring method, as shown in fig. 5, the method further includes:
step S501, recording the actual output time of the first output data;
in the embodiment of the present invention, the actual yield time refers to a time when the first yield data is generated and recorded, for example: the actual output time is 15 pm and 20 min X month X day X year.
Step S502, if the actual output time of the first output data is behind the estimated output time period, calculating the difference between the actual output time and the end time of the estimated output time period;
in the embodiment of the invention, if the actual output time of the first output data is before the estimated output time period, the generation of the output data which has a dependency relationship with the actual output time is not influenced, so that related maintenance personnel are not required to be informed. However, when the actual output time of the first output data is located after the estimated output time period, the difference between the actual output time and the end time of the estimated output time period is calculated to determine whether there is a large deviation between the actual output time and the estimated output time period.
For example: the preset estimated output time period is from 10 o 'clock to 11 o' clock in the morning, and the actual output time is 2 o 'clock in the afternoon, and considering that the actual output time and the end value (11 o' clock) of the estimated output time period have a large deviation, the execution program of the data resource for generating the output data may be changed, or the setting of the estimated output time period is not good.
Step S503, if the difference value is larger than a preset threshold value, sending a yield abnormity prompt;
in the embodiment of the present invention, when the difference between the actual output time and the estimated output time period is greater than the preset threshold, the output abnormality prompt may be sent, where the content of the output abnormality prompt may be determined according to an actual situation, for example, the output abnormality prompt may include preset text prompt content, or may include the actual output time, the estimated output time period, and data resources corresponding to the first output data, and the like.
According to the embodiment of the invention, the output abnormity prompt is sent to prompt relevant maintenance personnel, the estimated output time period can be readjusted, or the generation logic of the data resources is adjusted, so that the data resources can be ensured to normally generate the output data, the normal generation of other output data with dependency relationship can be ensured, and the robustness of the algorithm is improved.
In the embodiment of the present invention, as shown in fig. 6, the method further includes:
step S601, acquiring actual output time of a plurality of first output data generated by the same data resource of a historical record;
in the embodiment of the invention, when the data resource generates the first output data in each production period, the generation time of the first output data is automatically recorded, and the actual output time is obtained. The actual output time of a plurality of first output data generated by the same data resource meets the condition that the difference value of any two adjacent actual output times is the output period of the data resource under an ideal state.
Step S602, calculating the difference value of the actual output time adjacent to each other in every arbitrary two times;
in the embodiment of the present invention, any two actual production times that are adjacent in time refer to any two actual production times that are adjacent in time, for example: three actual production time points h1, h2 and h3 are 13 points, 14 points and 15 points in the same day, so that h1 is adjacent to h2 in time, and h2 is adjacent to h3 in time.
Step S603, calculating an average value of a plurality of difference values to obtain an adjustment value;
in the embodiment of the invention, the adjustment value can be obtained by averaging the plurality of difference values, and in practical application, because the difference between any two adjacent actual output times on the time axis in the plurality of output actual output times is similar to the output period of the data resource, the average value of the plurality of difference values can be calculated by taking the difference between any two adjacent actual output times on the time axis, so that the purpose of finding the adjustment value most fitting the rule of the actual output times is achieved.
Step S604, calculating the sum of the first estimated output time period and the adjustment value to obtain a new first estimated output time period;
in the embodiment of the invention, the adjustment value can be positive or negative, and a new estimated output time period can be obtained by adding the estimated output time period and the adjustment value, wherein the specific calculation mode can adopt the way of adding the initial value of the estimated output time period and the adjustment value to obtain a new initial value, then adding the end value of the estimated output time period and the adjustment value to obtain a new end value, and finally determining the new estimated output time period according to the new initial value and the new end value.
Step S605, replacing the first estimated output time period with a new first estimated output time period;
in a practical application scenario, for example, the yield cycle of the data resource is 1 day, and ideally, the first yield data generated at 10 o ' clock and 10 min is recorded on day X, the second first yield data generated at 10 o ' clock and 10 min is recorded on day X +1, and the third first yield data … … generated at 10 o ' clock and 10 min is recorded on day X + 2. However, in the actual operation process, there may be a slight difference in the actual output time, for example, the second first output data is recorded at 10 o 'clock 09 on day X +1, the third first output data is recorded at 10 o' clock 11 on day X +2, and so on, so that the actual output time of multiple history records is obtained, and the purpose of automatically adjusting the estimated output time period when the cycle of generating output data by the data resource changes can be achieved, so that the program operation version can be iterated.
In still another embodiment of the present invention, there is also provided a data monitoring apparatus, as shown in fig. 7, including:
the acquisition module 01 is used for acquiring data resources to be monitored and a first estimated output time period corresponding to the data resources;
the judging module 02 is configured to judge whether the data resource generates first output data within the first estimated output time period;
the sending module 03 is configured to send a first yield anomaly alarm if the data resource does not generate first yield data within the first estimated yield time period.
It can be clearly understood by those skilled in the art in the embodiments of the present invention that, for convenience and brevity of description, the specific working processes of the systems, apparatuses and units described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In still another embodiment of the present invention, there is also provided an electronic apparatus including: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the information transmission method in the embodiment of the method when executing the program stored in the memory.
In the electronic device provided by the embodiment of the invention, the processor implements a mode of sending an information transmission request to the information provider through the information transmission channel in the forbidden state by executing the program stored in the memory, whether the information provider is online can be quickly judged according to the information transmission response returned by the information provider, if the information provider system is determined to be online, the state of the information transmission channel is automatically switched from the forbidden state to the enabled state, and further, the purpose of quickly opening the information transmission channel in the forbidden state after the information provider is switched from the offline state to the online state is achieved, the efficiency of opening the information transmission channel is improved, and the timeliness of opening the information transmission channel is ensured.
The communication bus 1140 mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 8, but this is not intended to represent only one bus or type of bus.
The communication interface 1120 is used for communication between the electronic device and other devices.
The memory 1130 may include a Random Access Memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The processor 1110 may be a general-purpose processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the integrated circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
In still another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a program of an information transmission method, which when executed by a processor, implements the steps of the information transmission method described in the method embodiment.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A method for monitoring data, the method comprising:
acquiring a data resource to be monitored and a first estimated output time period corresponding to the data resource;
judging whether the data resource generates first output data in the first estimated output time period or not;
if the data resource does not generate first output data in the first estimated output time period, sending a first output abnormity alarm;
the method further comprises the following steps:
judging whether second output data which is dependent exists when the data resource generates the first output data;
if second yield data which is dependent exists when the data resource generates the first yield data, judging whether the second yield data is generated;
if the second output data is generated, acquiring an identifier of the first output data and an identifier of the second output data;
generating dependency relationship information according to the identifier of the first output data and the identifier of the second output data;
generating a notification of abnormal output according to the dependency relationship information;
sending the output abnormity notification to a first alarm receiving end corresponding to the data resource for a maintainer to check;
after the step of sending the output anomaly notification to a first alarm receiving end corresponding to the data resource for a maintainer to check, the method further comprises the following steps:
if the acknowledgement information which is sent by the first alarm receiving end and responds to the output abnormity notification is not received, acquiring a data output period corresponding to the data resource;
calculating the sum of the data output period and the estimated output time period to obtain a third estimated output time period;
judging whether the newly produced data of the data resource is produced in the third estimated production time period after one production period;
and if the newly produced data of the data resource is not produced in the third estimated production time period, sending an abnormal production alarm to a second alarm receiving terminal corresponding to the data resource, wherein the second alarm receiving terminal is different from the first alarm receiving terminal.
2. The method of claim 1, wherein if the second yield data has been generated, the method further comprises:
acquiring a second estimated output time period and output moment of the second output data;
and if the output moment of the second output data is within the second estimated output time period, executing a step of sending a first output abnormity alarm.
3. The data monitoring method of claim 2, further comprising:
if the output moment of the second output data is not within the second estimated output time period, judging whether third output data which is dependent exists when the second output data is generated;
and if no dependent third output data exists when the second output data is generated, generating an output abnormity alarm of the second output data.
4. The data monitoring method of claim 1, further comprising:
recording the actual yield time of the first yield data;
if the actual output time of the first output data is behind the estimated output time period, calculating a difference value between the actual output time and the end moment of the estimated output time period;
and if the difference is larger than a preset threshold value, sending a prompt of abnormal output.
5. The data monitoring method of claim 4, further comprising:
acquiring actual output time of a plurality of first output data generated by the same data resource of a historical record;
calculating the difference value of the adjacent actual output time of every two random times;
calculating an average value of the difference values to obtain an adjustment value;
calculating the sum of the first estimated output time period and the adjustment value to obtain a new first estimated output time period;
and replacing the first estimated output time period with a new first estimated output time period.
6. A data monitoring device, comprising:
the system comprises an acquisition module, a monitoring module and a judgment module, wherein the acquisition module is used for acquiring data resources to be monitored and a first estimated output time period corresponding to the data resources;
the judging module is used for judging whether the data resources generate first output data within the first estimated output time period;
the sending module is used for sending a first output abnormity alarm if the data resource does not generate first output data in the first estimated output time period;
the apparatus is further configured to:
judging whether second output data which is dependent exists when the data resource generates the first output data;
if second yield data which is dependent exists when the data resource generates the first yield data, judging whether the second yield data is generated;
if the second output data is generated, acquiring an identifier of the first output data and an identifier of the second output data;
generating dependency relationship information according to the identifier of the first output data and the identifier of the second output data;
generating a notification of abnormal output according to the dependency relationship information;
sending the output abnormity notification to a first alarm receiving end corresponding to the data resource for a maintainer to check;
after the step of sending the output anomaly notification to a first alarm receiving end corresponding to the data resource for a maintainer to check, the method further comprises the following steps:
if the acknowledgement information which is sent by the first alarm receiving end and responds to the output abnormity notification is not received, acquiring a data output period corresponding to the data resource;
calculating the sum of the data output period and the estimated output time period to obtain a third estimated output time period;
judging whether the newly produced data of the data resource is produced in the third estimated production time period after one production period;
and if the newly produced data of the data resource is not produced in the third estimated production time period, sending an abnormal production alarm to a second alarm receiving terminal corresponding to the data resource, wherein the second alarm receiving terminal is different from the first alarm receiving terminal.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor implements the steps of the method of any of claims 1 to 5 when executing the computer program.
8. A computer-readable medium having non-volatile program code executable by a processor, wherein the program code causes the processor to perform the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275685.XA CN111190788B (en) | 2019-12-12 | 2019-12-12 | Data monitoring method and device, electronic equipment and readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275685.XA CN111190788B (en) | 2019-12-12 | 2019-12-12 | Data monitoring method and device, electronic equipment and readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111190788A CN111190788A (en) | 2020-05-22 |
| CN111190788B true CN111190788B (en) | 2020-12-22 |
Family
ID=70705774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911275685.XA Active CN111190788B (en) | 2019-12-12 | 2019-12-12 | Data monitoring method and device, electronic equipment and readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111190788B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103490949A (en) * | 2013-09-06 | 2014-01-01 | 京信通信***（中国）有限公司 | Method and device for collecting data information and detecting data information anomalies |
| CN110213071A (en) * | 2018-04-16 | 2019-09-06 | 腾讯科技（深圳）有限公司 | Verification of data method, apparatus, system, computer equipment and storage medium |
| CN110516971A (en) * | 2019-08-29 | 2019-11-29 | 杭州网易再顾科技有限公司 | Method, apparatus, medium and the calculating equipment of abnormality detection |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070286087A1 (en) * | 2006-06-13 | 2007-12-13 | International Business Machines Corporation | Distributed Network Enhanced Wellness Checking |
| CN108880838B (en) * | 2017-05-10 | 2021-11-09 | 阿里巴巴集团控股有限公司 | Service fault monitoring method and device, computer equipment and readable medium |
| JP6972735B2 (en) * | 2017-07-26 | 2021-11-24 | 富士通株式会社 | Display control program, display control method and display control device |
| CN108107292B (en) * | 2017-12-18 | 2020-02-07 | 国网湖南省电力有限公司 | Service data monitoring system and method of electric energy quality on-line monitoring system |
-
2019
- 2019-12-12 CN CN201911275685.XA patent/CN111190788B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103490949A (en) * | 2013-09-06 | 2014-01-01 | 京信通信***（中国）有限公司 | Method and device for collecting data information and detecting data information anomalies |
| CN110213071A (en) * | 2018-04-16 | 2019-09-06 | 腾讯科技（深圳）有限公司 | Verification of data method, apparatus, system, computer equipment and storage medium |
| CN110516971A (en) * | 2019-08-29 | 2019-11-29 | 杭州网易再顾科技有限公司 | Method, apparatus, medium and the calculating equipment of abnormality detection |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111190788A (en) | 2020-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109522287B (en) | Monitoring method, system, equipment and medium for distributed file storage cluster | |
| CN110213068B (en) | Message middleware monitoring method and related equipment | |
| CN111193609B (en) | Application abnormity feedback method and device and application abnormity monitoring system | |
| CN108737132B (en) | Alarm information processing method and device | |
| CN109426510B (en) | Software processing method and device, electronic equipment and computer readable storage medium | |
| CN108319538B (en) | The monitoring method and system of big data platform operating status | |
| CN111143163A (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN107577527B (en) | Task generation and scheduling method and device | |
| CN114461439A (en) | Fault diagnosis method, device, equipment and storage medium | |
| CN110072199B (en) | Method and system for monitoring short message sending abnormity | |
| CN111190788B (en) | Data monitoring method and device, electronic equipment and readable medium | |
| CN111949421B (en) | SDK calling method, device, electronic equipment and computer readable storage medium | |
| CN111552746A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN111601329A (en) | Method and device for processing port interrupt alarm | |
| CN110855003A (en) | Method and device for calling and comparing self-adaptive configuration of main station | |
| CN113157472A (en) | Service information processing method, device, equipment and computer storage medium | |
| CN115277479A (en) | Method and system for realizing system operation condition monitoring based on monitoring assistant | |
| CN110166528B (en) | Method and device for preventing node change notification from being lost and computer equipment | |
| CN115033927A (en) | Method, device, equipment and medium for detecting data integrity | |
| CN114036032A (en) | Real-time program monitoring method and device | |
| CN109508356B (en) | Data abnormality early warning method, device, computer equipment and storage medium | |
| CN113535449A (en) | Abnormal event repairing processing method and device, computer equipment and storage medium | |
| CN114691395A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN113283865A (en) | Industrial production information management and control method and device | |
| CN115150249A (en) | Storage system warning method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | "change of name, title or address" |
Address after: 519080 Building 8, No.1, Harbin Institute of technology, Tangjiawan Town, Zhuhai City, Guangdong Province Patentee after: Zhuhai necessary Industrial Technology Co.,Ltd. Address before: Building 8, No.1, hagongda Road, Tangjiawan Town, Zhuhai City, Guangdong Province Patentee before: ZHUHAI SUIBIAN TECHNOLOGY Co.,Ltd. |
|
| CP03 | "change of name, title or address" |