CN113760677A - Abnormal link analysis method, device, equipment and storage medium - Google Patents

Abnormal link analysis method, device, equipment and storage medium Download PDF

Info

Publication number
CN113760677A
CN113760677A CN202110142549.4A CN202110142549A CN113760677A CN 113760677 A CN113760677 A CN 113760677A CN 202110142549 A CN202110142549 A CN 202110142549A CN 113760677 A CN113760677 A CN 113760677A
Authority
CN
China
Prior art keywords
task
abnormal
processed
operation information
link analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110142549.4A
Other languages
Chinese (zh)
Inventor
马千里
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Wodong Tianjun Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Wodong Tianjun Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN202110142549.4A priority Critical patent/CN113760677A/en
Publication of CN113760677A publication Critical patent/CN113760677A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database

Abstract

The embodiment of the application provides an abnormal link analysis method, an abnormal link analysis device, an abnormal link analysis equipment and a storage medium, wherein an abnormal link analysis request of a user is received, current operation information and historical operation information of a task to be processed are obtained according to an identification of the task to be processed, whether the task to be processed is abnormal is determined according to the current operation information, the historical operation information and a service level protocol configured in advance, when the task to be processed is determined to be abnormal, whether a previous-level task of the task to be processed is abnormal is recursively analyzed based on task level information until a task level without the abnormal task is determined, and an abnormal link analysis result is obtained. According to the technical scheme, the abnormal link analysis can be executed based on the needs of the user, the reason of the abnormal link can be located, the accuracy of task monitoring is improved, and implementation conditions are provided for improving the quality of data output by a big data platform.

Description

Abnormal link analysis method, device, equipment and storage medium
Technical Field
The embodiment of the application relates to the technical field of internet, in particular to an abnormal link analysis method, an abnormal link analysis device, abnormal link analysis equipment and a storage medium.
Background
With the rapid development of internet technology, enterprises pay attention to the construction of a big data platform, and various types of data are processed and processed by using the big data platform, so that required data can be obtained. In order to meet the requirement that the data output by the big data platform meets the quality requirement, the task running in the big data platform needs to be monitored so as to ensure that the big data platform outputs punctual and accurate data.
In the related technology, task monitoring in a big data platform is mainly to set some alarm attributes for each task by performing operations such as grading, labeling, Service Level Agreement (SLA) setting and the like on all tasks, and then, in practical application, when a certain task is analyzed, the task can be analyzed for abnormality by combining the alarm attributes of the task and the historical operating condition of the task, and an alarm notification is sent when the task is determined to be abnormal. Specifically, an operating time threshold is set for each task, then the monitored tasks are periodically scanned, whether the operating time of the monitored tasks exceeds the threshold time or not is judged, and if yes, an alarm is triggered.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: when a certain monitored task triggers an alarm because the running time of the certain monitored task does not meet the time threshold, the task is probably not caused by the abnormality of the task, but is caused by the running abnormality of one or more parent tasks at the upper level of the current task, the real reason of the task abnormality alarm cannot be directly positioned by the conventional task monitoring method, and the problem of low task monitoring accuracy exists.
Disclosure of Invention
The embodiment of the application provides an abnormal link analysis method, an abnormal link analysis device, an abnormal link analysis equipment and a storage medium, which are used for solving the problem of low task monitoring accuracy in the existing task monitoring method.
According to a first aspect of the present application, an embodiment of the present application provides an abnormal link analysis method, including:
receiving an abnormal link analysis request of a user, wherein the abnormal link analysis request comprises: identification of the task to be processed;
acquiring current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;
determining whether the task to be processed is abnormal or not according to the current operation information, the historical operation information and a pre-configured service level protocol;
and when the task to be processed is determined to be abnormal, recursively analyzing whether the previous-level task of the task to be processed is abnormal or not based on task level information until the task level without the abnormal task is determined, and obtaining an abnormal link analysis result.
In a possible design of the first aspect, the obtaining current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed includes:
according to the identification of the task to be processed, current operation information of the task to be processed is obtained from a cache database, and historical operation information of the task to be processed is obtained from a data warehouse;
the information in the cache database is obtained by consuming metadata information of each task in a big data platform; and historical operation information of each task in the big data platform is stored in the data warehouse.
Optionally, before the obtaining of the current operation information and the historical operation information of the task to be processed according to the identifier of the task to be processed, the method further includes:
acquiring metadata information of each task from a business system;
cleaning the metadata information of each task, and determining the current operation information of each task;
and storing the current running information of each task into the cache database, wherein the running information of each task in the cache database has an expiration time.
Optionally, the method further includes:
determining the periodic processing time of the cache database;
when the current time reaches the periodic processing time, transmitting the running information of at least one task with the storage time exceeding the expiration time in the cache database to the data warehouse;
and updating historical operation information of each task in the data warehouse.
In another possible design of the first aspect, the determining whether the task to be processed is abnormal according to the current operation information, the historical operation information, and a pre-configured service level agreement includes:
determining whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement or not according to the current operation information and the service level agreement;
and when the completion time of the task to be processed exceeds the threshold time set in the service level agreement, determining whether the task to be processed is abnormal or not according to the current operation information and the historical operation information.
In yet another possible design of the first aspect, the recursively analyzing whether a previous-stage task of the to-be-processed task is abnormal based on task level information until a task level without an abnormal task is determined, and obtaining an abnormal link analysis result includes:
determining all the upper-level tasks of the tasks to be processed according to the task level information;
and according to the running state of each task in the previous-level task, performing exception analysis on the tasks meeting the preset conditions to obtain an exception link analysis result.
Optionally, the performing, according to the running state of each task in the previous-stage task, an abnormal analysis on the task meeting a preset condition to obtain an abnormal link analysis result includes:
when the tasks which do not run exist in the previous-level tasks, performing exception analysis on all the tasks which do not run to obtain an exception link analysis result;
when the tasks which are not operated do not exist in the upper-level task, but the tasks which are operated and completed, all the tasks which are operated and the tasks of which the operation ending time exceeds the threshold value time are subjected to abnormal analysis, and the abnormal link analysis result is obtained;
and when all the tasks in the previous-stage task are the tasks which are finished to run, performing exception analysis on the tasks of which the running end time exceeds the threshold time in the previous-stage task until determining the task level without exception, and obtaining an exception link analysis result.
In yet another possible design of the first aspect, the method further includes:
and outputting the abnormal link analysis result in the form of a link map.
According to a second aspect of the present application, an embodiment of the present application provides an abnormal link analyzing apparatus, including:
a receiving module, configured to receive an abnormal link analysis request of a user, where the abnormal link analysis request includes: identification of the task to be processed;
the acquisition module is used for acquiring the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed;
and the processing module is used for determining whether the task to be processed is abnormal or not according to the current operation information, the historical operation information and a pre-configured service level protocol, and recursively analyzing whether the previous-level task of the task to be processed is abnormal or not based on task level information when the task to be processed is determined to be abnormal until the task level without the abnormal task is determined, so as to obtain an abnormal link analysis result.
In a possible design of the second aspect, the obtaining module is specifically configured to obtain, according to the identifier of the task to be processed, current operation information of the task to be processed from a cache database, and obtain historical operation information of the task to be processed from a data warehouse;
the information in the cache database is obtained by consuming metadata information of each task in a big data platform; and historical operation information of each task in the big data platform is stored in the data warehouse.
Optionally, the obtaining module is further configured to obtain metadata information of each task from a business system before obtaining current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;
the processing module is further configured to clean the metadata information of each task, determine current operation information of each task, store the current operation information of each task in the cache database, where the operation information of each task in the cache database has an expiration time.
Optionally, the processing module is further configured to:
determining the periodic processing time of the cache database;
when the current time reaches the periodic processing time, transmitting the running information of at least one task with the storage time exceeding the expiration time in the cache database to the data warehouse;
and updating historical operation information of each task in the data warehouse.
In another possible design of the second aspect, the processing module is configured to determine whether the task to be processed is abnormal according to the current operation information, the historical operation information, and a preconfigured service level protocol, and specifically includes:
the processing module is specifically configured to:
determining whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement or not according to the current operation information and the service level agreement;
and when the completion time of the task to be processed exceeds the threshold time set in the service level agreement, determining whether the task to be processed is abnormal or not according to the current operation information and the historical operation information.
In another possible design of the second aspect, the processing module is configured to recursively analyze whether a previous-stage task of the to-be-processed task is abnormal based on task level information until a task level without an abnormal task is determined, and obtain an abnormal link analysis result, where the analyzing step specifically includes:
the processing module is specifically configured to:
determining all the upper-level tasks of the tasks to be processed according to the task level information;
and according to the running state of each task in the previous-level task, performing exception analysis on the tasks meeting the preset conditions to obtain an exception link analysis result.
Optionally, the processing module is configured to perform exception analysis on the tasks meeting the preset condition according to the running state of each task in the previous-stage task, so as to obtain an exception link analysis result, and specifically includes:
the processing module is specifically configured to:
when the tasks which do not run exist in the previous-level tasks, performing exception analysis on all the tasks which do not run to obtain an exception link analysis result;
when the tasks which are not operated do not exist in the upper-level task, but the tasks which are operated and completed, all the tasks which are operated and the tasks of which the operation ending time exceeds the threshold value time are subjected to abnormal analysis, and the abnormal link analysis result is obtained;
and when all the tasks in the previous-stage task are the tasks which are finished to run, performing exception analysis on the tasks of which the running end time exceeds the threshold time in the previous-stage task until determining the task level without exception, and obtaining an exception link analysis result.
In yet another possible design of the second aspect, the apparatus further includes:
and the output module is used for outputting the abnormal link analysis result in a link diagram form.
According to a third aspect of the present application, an electronic device is provided in an embodiment of the present application, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method according to the first aspect and possible designs.
According to a fourth aspect of the present application, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored, and the computer instructions, when executed by a processor, are configured to implement the method according to the first aspect and possible designs.
According to a fifth aspect of the present application, an embodiment of the present application provides a computer program product, including: a computer program, stored in a readable storage medium, from which at least one processor of an electronic device can read the computer program, execution of the computer program by the at least one processor causing the electronic device to perform the method of the first aspect.
According to the abnormal link analysis method, the abnormal link analysis device, the abnormal link analysis equipment and the abnormal link analysis storage medium, the abnormal link analysis request of a user is received, the current running information and the historical running information of the task to be processed are obtained according to the identification of the task to be processed, whether the task to be processed is abnormal is determined according to the current running information, the historical running information and a service level protocol configured in advance, when the task to be processed is determined to be abnormal, whether the previous-level task of the task to be processed is abnormal is recursively analyzed based on task level information until the task level without the abnormal task is determined, and an abnormal link analysis result is obtained. According to the technical scheme, the abnormal link analysis can be executed based on the needs of the user, the reason of the abnormal link can be located, the accuracy of task monitoring is improved, and implementation conditions are provided for improving the quality of data output by a big data platform.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a schematic diagram of a network architecture to which an abnormal link analysis method according to an embodiment of the present application is applied;
fig. 2 is a schematic flowchart of a first embodiment of an abnormal link analysis method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a second embodiment of an abnormal link analysis method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a third embodiment of an abnormal link analysis method according to an embodiment of the present application;
fig. 5 is a schematic flowchart of a fourth embodiment of an abnormal link analysis method according to the present application;
FIG. 6 is a schematic diagram illustrating a task link distribution of multiple levels of a task scheduling system according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an embodiment of an abnormal link analysis apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an embodiment of an electronic device provided in the embodiment of the present application.
With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
First, terms related to embodiments of the present application will be explained:
the dispatching system: a task scheduling system (Buffalo), also called workflow scheduling system, is a heavy product for offline calculation of a big data platform, and not only bears synchronous work between various databases and data sets, but also bears various offline data calculation work. The task scheduling system can conveniently and quickly manage the timed tasks, for example, newly-added data is imported from a database to a data platform at regular time, and the data processed by the data platform is exported to the database or a file system. In addition, the task scheduling system also supports the establishment of a dependency relationship among tasks, the quick completion and re-running of the tasks, a strong monitoring function and the like, thereby providing good job management service.
A data warehouse: data warehouse (data consumer, DW or DWH). The data warehouse mainly provides strategic sets of all types of data support for decision making processes of all levels of enterprises. It is a single data store created for analytical reporting and decision support purposes that can provide business-directed process improvement, monitoring time, cost, quality, and control for enterprises that need business intelligence.
Hive: hive is a data warehouse tool based on Hadoop, is used for data extraction, transformation and loading, and is a mechanism capable of storing, inquiring and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map the structured data file into a database table, provide SQL query function and convert SQL sentences into MapReduce tasks for execution. Hive has the advantages that the learning cost is low, rapid MapReduce statistics can be realized through similar SQL sentences, MapReduce is simpler, and a special MapReduce application program does not need to be developed. hive is well suited for statistical analysis of data warehouses.
Kafka: kafka is an open source stream processing platform, written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site.
Redis: remote dictionary service (Redis) is an open-source, log-based, Key-Value database written in ANSI C language, supporting network, based on memory and persistent, and provides API for multiple languages.
Flink: apache Flink is an open source stream processing framework, and the core of the Apache Flink is a distributed stream data stream engine written in Java and Scale. The Flink executes any stream data program in a data parallel and pipeline mode, and when the pipeline of the Flink runs, the system can execute batch processing and stream processing programs. In addition, the runtime of Flink itself supports the execution of iterative algorithms.
SLA: a Service Level Agreement (SLA) refers to an agreement or contract that is agreed between an enterprise providing a service and a client, in terms of quality, level, performance, and the like of the service. A typical service level agreement includes the following: the provision of the service provided and the expiration date of the protocol by the participating parties; time specification during service provisioning, including testing, maintenance, and upgrades; the number of users, the location and the services provided by the corresponding hardware; a description of the fault reporting process, including the conditions for the fault escalation to higher level support; specification of expected response time for fault reporting; the flow of the change request will be described. Optionally, the service level agreement may also include expected times to complete routine change requests; specification of service level objectives; a service-related charging provision; user responsibility specifications (user training, ensuring correct desktop configuration, no unnecessary software, no disruption to change management processes, etc.); a description of a procedure for solving different opinions about services, etc.
With the rapid development of network technology and the growth of business of each enterprise, a large amount of unstructured data related to processes and rules also grows explosively. The big data platform can provide services such as resource management, data calculation and analysis, data storage, service monitoring management and the like for mass data, integrates various services such as system installation, cluster configuration, safe access control, monitoring and early warning and the like on the basis of a distributed file system and distributed parallel computation, and provides reliable data support for enterprise big data core business, business intelligence, operation analysis, business decision and the like.
In order to meet the requirement of a user on the quality of data output by a big data platform, the prior art can carry out operation monitoring on tasks operated in the big data platform so as to ensure that the data can be accurately produced on time.
As can be seen from the above description in the background art, in the task monitoring scheme in the related art, if the running time of the monitored current task does not meet the set time threshold, an alarm is triggered, but actually, the task monitoring scheme may not be caused by an abnormality of the task itself, but may be caused by an abnormal running of one or more parent tasks at the upper level of the current task, which may not directly locate a true reason for the task abnormality alarm, and thus, the task monitoring accuracy is low.
Aiming at the technical problems, the conception process of the technical scheme of the application is as follows: the task scheduling system of the big data platform is monitored by a tracing mode, and the running information of each task in the scheduling system is processed periodically, so that the running information of each task in the current time period and the running information of each task in the historical time period are stored separately, a foundation is laid for subsequent abnormal analysis of the tasks, secondly, when an abnormal analysis request is received, the current task is analyzed in an abnormal mode, when the current task is determined to be abnormal, a parent task of the current task is analyzed in a recursion mode until the last level of the abnormal task or the level without the abnormal task is determined, the real reason of the abnormal task can be determined by the method, and the accuracy of task monitoring is improved.
Based on the technical conception process, the embodiment of the application provides an abnormal link analysis method, when an abnormal link analysis request of a user is received, current operation information and historical operation information of a task to be processed can be obtained according to an identifier of the task to be processed, which is included in the abnormal link analysis request, whether the task to be processed is abnormal is determined according to the current operation information, the historical operation information and a service level protocol which is configured in advance, when the task to be processed is determined to be abnormal, whether a previous-stage task of the task to be processed is abnormal is recursively analyzed based on task level information until a task level without the abnormal task is determined, and an abnormal link analysis result is obtained. According to the technical scheme, the abnormal link analysis can be executed based on the needs of the user, the reason of the abnormal link can be located, the accuracy of task monitoring is improved, and implementation conditions are provided for improving the quality of data output by a big data platform.
Fig. 1 is a schematic diagram of a network architecture to which the abnormal link analysis method provided in the embodiment of the present application is applied. As shown in fig. 1, in this embodiment, the network structure diagram may include: a data platform 11 and a link analysis device 12 connected to each other. The data platform 11 may include a data source 111, a data processing module 112, and a data storage module 113, among others. The data storage module 113 is connected with the link analysis device 12, and the two realize information interaction.
The data provided by the data source 111 may be application data, data stored in a database, data logs, data of other data sources, and the like, and the embodiment of the present application does not limit the source of the data in the data source 111, and may be determined according to an actual scene, which is not described herein again.
The data processing module 112 may perform preprocessing such as cleaning and sorting on the data provided by the data source 111, and store the processed data in the data storage module 113.
The data storage module 113 may include: a cache database 1131 and a data store 1132. The data in the cache database 1131 may be current operation information obtained by the data platform 11 executing each task in the current time period, and the data in the data warehouse 1132 may be outdated data in the cache database 1131, which may be used as historical operation information of each task in the cache database 1131.
In an embodiment of the present application, when the link analysis device 12 has a human-computer interaction interface, a user may send a link analysis request including an identifier of a to-be-processed task to the link analysis device 12 through the human-computer interaction interface, so that the link analysis device 12 may obtain current operation information and historical operation information of the to-be-processed task from the cache database 1131 and the data warehouse 1132 included in the data storage module 113, respectively, and accordingly, the link analysis device 12 may perform analysis and positioning of an abnormal task based on the current operation information, the historical operation information, and a preconfigured service level protocol of the to-be-processed task. Optionally, the link analysis device 12 may further output the abnormal link analysis result through the human-computer interaction interface.
In another embodiment of the present application, the network structure diagram may further include: at this time, the user may send an abnormal link analysis request to the link analysis device 12 through the user terminal 13, so that the current operation information and the historical operation information of the to-be-processed task are respectively obtained from the cache database 1131 and the data warehouse 1132 included in the data storage module 113 based on the identifier of the to-be-processed task in the abnormal link analysis request, and correspondingly, the link analysis device 12 may perform analysis and positioning of the abnormal task based on the current operation information, the historical operation information, and a preconfigured service level protocol of the to-be-processed task. For example, the link analysis device 12 may feed back the abnormal link analysis result to the user terminal 13, so that the user terminal 13 performs presentation of the abnormal link analysis result.
Optionally, in another embodiment of the present application, the network structure diagram may further include: and the display device 14, wherein the display device 14 can receive the abnormal link analysis result sent by the link analysis device 12 and display the abnormal link analysis result.
Optionally, in an embodiment of the present application, the link analysis device may be implemented by a server, or may be implemented by a terminal device, which is not limited herein.
It can be understood that, in the embodiment of the present application, the types and the number of the devices included in the network structure schematic diagram are not limited, and may be set according to the scene requirements, which is not described herein again.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 2 is a schematic flowchart of a first embodiment of an abnormal link analysis method according to an embodiment of the present application. The method is explained by taking the link analysis device in the network architecture diagram shown in fig. 1 as an execution subject. As shown in fig. 2, the method for analyzing an abnormal link may include the following steps:
s201, receiving an abnormal link analysis request of a user, wherein the abnormal link analysis request comprises: and identifying the task to be processed.
In the embodiment of the present application, the link analyzing apparatus may be abstracted to be a monitoring system, but it is different from the monitoring system in the related art in that the link analyzing apparatus has an interface capable of receiving an external request.
Illustratively, the link analysis device may start an http service by running a loaded program, and may receive an abnormal link analysis request for a task to be processed, which is submitted by a user, through the http service, so that the link analysis device may analyze the running condition of each task in the task scheduling system in the big data platform.
Illustratively, after receiving an abnormal link analysis request from a user, an http service of the link analysis device parses the abnormal link analysis request to obtain an identifier of a task to be processed.
In the embodiment of the present application, the abnormal link analysis request is mainly an analysis for an abnormal link, and in practical applications, the abnormal link analysis request may be replaced by a latest link analysis request, so that the link analysis device executes a link analysis based on the latest link analysis request. It is understood that the link analysis device may adopt different analysis logics for different link analysis requests, which is not described herein in detail.
S202, obtaining current running information and historical running information of the task to be processed according to the identification of the task to be processed.
In the embodiment of the application, after the link analysis device analyzes the abnormal link analysis request and obtains the identifier of the task to be processed, the current operation information and the historical operation information of the task to be processed can be obtained by querying the data storage module for storing the operation information of the task to be processed based on the identifier of the task to be processed.
For example, in an embodiment of the present application, the link analysis device may obtain current operation information of the task to be processed from the cache database and obtain historical operation information of the task to be processed from the data warehouse according to the identifier of the task to be processed.
The information in the cache database is obtained by consuming metadata information of each task in the big data platform; historical operation information of each task in the big data platform is stored in the data warehouse.
It can be understood that the cache database and the data warehouse may be components of a data storage module in a big data platform, or may be components of a link analysis device, and specific locations of the cache database and the data warehouse may be set according to actual scenarios, which is not limited in this embodiment.
In this step, the current running information of each task can be obtained by consuming the metadata information of each task in the big data platform, and is stored in the cache database for subsequent use. Optionally, the data of each task in the cache database has an expiration time, and when the storage time of the data of each task in the cache database exceeds the expiration time, the historical operation information of each task is obtained. Optionally, the historical operating information of each task is periodically updated to the data warehouse for subsequent acquisition.
In practical applications, since the implementation of each service request usually needs to involve a plurality of tasks that are executed in sequence, the execution of each task is related to each other, and the information of the tasks is concatenated to form a link, the link analysis described in this application is to analyze the link where the task to be processed is located.
S203, determining whether the task to be processed is abnormal according to the current operation information, the historical operation information and a preset service level protocol.
In an embodiment of the present application, a service level protocol pre-configured for each task is stored in the link analysis device, and the service level protocol includes a running time threshold value of the corresponding task and the like. In general, the service level agreement includes a running time threshold of the corresponding task as an important criterion for determining whether the task is abnormal. Therefore, when a user requests to check an abnormal link, whether a task to be processed is abnormal or not can be compared according to parameter information transmitted by the user and by combining with the historical condition of task operation, specifically, after current operation information and historical operation information of the task to be processed are obtained, a time threshold of the task to be processed can be determined according to a pre-configured service level protocol, and then whether the task to be processed is abnormal or not can be jointly judged by combining the current operation information and the historical operation information, so that the judgment accuracy is improved.
S204, when the task to be processed is determined to be abnormal, recursively analyzing whether the previous-level task of the task to be processed is abnormal or not based on task level information until the task level without the abnormal task is determined, and obtaining an abnormal link analysis result.
In this embodiment, each task may involve multiple links, and each link may include multiple hierarchical levels. When the to-be-processed task is determined to be abnormal, in order to determine the true cause of the to-be-processed task, upward analysis can be performed in a recursive analysis mode based on task level information, that is, whether the previous task of the to-be-processed task is abnormal is firstly analyzed, and then upward analysis is performed when the previous task is also abnormal until a task level without an abnormal task is analyzed, that is, the root of the to-be-processed task is determined, so that an abnormal link analysis result corresponding to the abnormal link analysis request is obtained.
According to the abnormal link analysis method provided by the embodiment of the application, the abnormal link analysis request of a user is received, the current running information and the historical running information of the task to be processed are obtained according to the identification of the task to be processed, which is included by the abnormal link analysis request, and then whether the task to be processed is abnormal is determined according to the current running information, the historical running information and a pre-configured service level protocol.
On the basis of the foregoing embodiments, fig. 3 is a schematic flow chart of a second embodiment of an abnormal link analysis method provided in the embodiments of the present application. As shown in fig. 3, in this embodiment, before the step S202, the method for analyzing an abnormal link may further include the following steps:
s301, acquiring metadata information of each task from the business system.
For example, in the embodiment of the present application, the task scheduling system may generate metadata information of each task in real time during the process of scheduling the task to run. Optionally, in the big data platform, a task has idle (i.e., success), failure, running and other states in the running process, and when the state of the task is changed, metadata information of the task is generated and can be written into the service system of the big data platform. Correspondingly, in the embodiment of the application, the link analysis device may consume the metadata information of each task in the business system through the real-time computing engine, so as to obtain the metadata information of each task, so as to analyze the metadata information.
Alternatively, the business system may be a distributed publish-subscribe messaging system, such as Kafka, which may handle all the action flow data of a user in a website. The real-time computing engine may be a flink, and in practical application, the real-time computing engine may also be a storm computing engine, spark computing engine, or the like.
Optionally, the metadata information of each task may include a task name, a task Id, a running state of the task, a starting running time of the task, a running end time of the task, and the like. The present embodiment does not limit the specific content included in each metadata information, and may be determined according to an actual scene.
S302, cleaning the metadata information of each task, and determining the current operation information of each task.
For example, in this embodiment, in order to avoid storing too much data that is useless for link analysis in the cache database, the link analysis device may first perform cleaning and filtering on metadata information of each task, and remove some data that is useless for abnormal analysis, to obtain current operation information of each task.
S303, storing the current running information of each task into a cache database, wherein the running information of each task in the cache database has an expiration time.
Optionally, the link analysis device may store the obtained current operation information of each task in the cache database, and set an expiration time for the operation information of each task. For example, the link analysis device may set an expiration time of 24 hours for the running information of each task, i.e., cache the running information of each task for one day.
Illustratively, the cache database may be any one of caches such as Redis, Hbase, clickwause, and the like, and the present embodiment does not limit the cache database.
It can be understood that, when performing the abnormal link analysis on the task to be processed, the running condition of the superior task (parent task) needs to be utilized, and therefore, the link analysis device needs to cache the running information of the parent task in advance into the cache database, and the storage mode of the link analysis device is similar to that of the current task, which is not described herein again.
And S304, determining the cycle processing time of the cache database.
In an embodiment of the present application, each of the cache databases has a period processing time, so that the link analysis device may periodically process data in the cache databases based on the period processing time.
For example, if the processing cycle of the cache database is 1 day, the cache database may be updated at the same time every day according to the scheduled date of the task, for example, at 11 o' clock 55 minutes at night every day, so as to be utilized when performing real-time analysis on the next day. It is understood that the embodiment of the present application does not limit the specific periodic processing time, and may be determined according to an actual scenario.
S305, when the current time reaches the cycle processing time, transmitting the running information of at least one task with the storage time exceeding the expiration time in the cache database to a data warehouse.
In an embodiment of the present application, the link analysis device may compare the current time with the above cycle processing time, and process the operation of each task in the cache database at the cycle processing time of each cycle, for example, transmit the operation information of at least one task whose storage time in the cache database exceeds the expiration time to the data warehouse, so that the link analysis device utilizes the operation information in the data warehouse in the next cycle, that is, the historical operation information of the task.
And S306, updating historical operation information of each task in the data warehouse.
Optionally, after receiving the operation information of at least one task transmitted from the cache database, the link analysis device may periodically update the historical operation information of each task.
For example, the historical running information of each task in the data warehouse may include, for example, an average start running time, an average end running time, an average execution time, previous and subsequent level information (task consanguinity information), tag information of the task, and the like. The embodiment of the application does not limit the concrete embodiment of the historical operation information, and the historical operation information can be determined according to an actual scene.
According to the abnormal link analysis method provided by the embodiment of the application, the metadata information of each task is obtained from a business system, the metadata information of each task is cleaned, the current operation information of each task is determined, the current operation information of each task is stored in a cache database, and the operation information of each task in the cache database has an expiration time; correspondingly, when the current time reaches the periodic processing time of the cache database, the running information of at least one task with the storage time exceeding the expiration time in the cache database can be transmitted to the data warehouse, and the historical running information of each task in the data warehouse can be updated. According to the technical scheme, the current operation information of each task is stored in the cache database, and the historical operation information is updated to the data warehouse, so that a foundation is laid for real-time link analysis of subsequent tasks, and the efficiency and accuracy of abnormal positioning are improved.
On the basis of the foregoing embodiments, fig. 4 is a schematic flowchart of a third embodiment of an abnormal link analysis method provided in the embodiment of the present application. As shown in fig. 4, in this embodiment, the step S203 may be implemented by:
s401, determining whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement or not according to the current operation information and the service level agreement.
Illustratively, pre-configured service level protocols are stored in the link analysis device, and each service level protocol comprises threshold time for running each task, so that the threshold time set in the service level protocol can be obtained by inquiring the service level protocol of the task to be processed based on the identifier of the task to be processed, then the completion time of the figure to be processed is obtained according to the current running information of the task of the person to be processed, and the completion time of the figure to be processed is compared with the completion time of the task to be processed, so as to judge whether the completion time of the task to be processed exceeds the threshold time set in the service level protocol.
As an example, when the completion time of the to-be-processed task does not exceed the threshold time set in the service level agreement, it indicates that the to-be-processed task is normal, and no upward recursive analysis is required.
As another example, when the completion time of the task to be processed exceeds the threshold time set in the service level agreement, it needs to be determined whether the task to be processed is abnormal.
S402, when the completion time of the task to be processed exceeds the threshold time set in the service level protocol, determining whether the task to be processed is abnormal or not according to the current operation information and the historical operation information.
Optionally, when the completion time of the task to be processed exceeds the threshold time set in the service level protocol, the current running information of the task to be processed may be compared with the historical running information of the task to be processed, for example, the starting running time in the current running information is compared with the starting running time in the historical running information, or the ending running time in the current running information is compared with the ending running time in the historical running information, or the relationship between the execution time length of the task to be processed and the average execution time length is directly determined, so as to determine whether the task to be processed is abnormal.
For example, when the error between the execution time length of the to-be-processed task and the average execution time length is smaller than the preset error, the to-be-processed task may be considered to be normal, and at this time, the upward recursive analysis is not required. And when the error between the execution time length of the task to be processed and the average execution time length is greater than or equal to the preset error, the task to be processed is considered to be abnormal, and at the moment, upward recursive analysis is needed to determine the true reason of the abnormal task to be processed.
According to the abnormal link analysis method provided by the embodiment of the application, whether the completion time of the task to be processed exceeds the threshold time set in the service level protocol or not is determined according to the current operation information and the service level protocol, and when the completion time of the task to be processed exceeds the threshold time set in the service level protocol, whether the task to be processed is abnormal or not is determined according to the current operation information and the historical operation information. According to the technical scheme, whether the task to be processed is abnormal or not is judged according to the current operation information, the historical operation information and the service level protocol, so that the accuracy of abnormal analysis is improved, and a foundation is laid for improving the quality of output data of a big data platform.
On the basis of the foregoing embodiments, fig. 5 is a schematic flowchart of a fourth embodiment of an abnormal link analysis method provided in the embodiment of the present application. As shown in fig. 5, in this embodiment, the step S204 may be implemented by:
s501, determining all the upper-level tasks of the tasks to be processed according to the task level information.
In the embodiment of the application, the tasks in the task scheduling system in the big data platform usually have multiple levels, each task can be on the same link with the tasks in different levels, and in order to accurately analyze the reason of the abnormality of the task to be processed, all the previous-level tasks of the task to be processed need to be determined based on the task level information in the task scheduling system.
Fig. 6 is a schematic diagram illustrating a distribution of task links of multiple hierarchies included in the task scheduling system in the embodiment of the present application. As shown in fig. 6, in the present embodiment, the task scheduling system has 8 levels, which are respectively from the 1 st level to the 8 th level from top to bottom. Each level may be distributed with the same or different number of tasks, e.g. level 1 with 3 tasks, respectively task 11, task 12 and task 13, and level 2 with 5 tasks, respectively task 21 to task 25.
Alternatively, in the schematic diagram shown in fig. 6, there may be a plurality of upper-level tasks (i.e., parent tasks) of each hierarchy, and may be located in different hierarchies. Illustratively, in the schematic diagram shown in fig. 6, the 8 th level task 81 has 4 upper level tasks, one of which is at the 7 th level and is task 71, and the other three upper level tasks are at the 2 nd level and are task 23, task 24 and task 25, respectively.
S502, according to the running state of each task in the previous-level task, performing exception analysis on the tasks meeting the preset conditions to obtain exception link analysis results.
In the embodiment of the application, the link analysis device can determine the previous-level task needing recursive analysis according to the running state of each task in the previous-level task, and determine the real reason of the exception of the task to be processed in the recursive analysis process to obtain the exception link analysis result.
Illustratively, this step may be specifically implemented by the following scheme:
as an example, when there is an un-run task in the upper-level task, all the un-run tasks are subjected to exception analysis to obtain an exception link analysis result.
If the task which is not operated exists in the previous-stage task, the fact that the previous-stage task is not operated possibly due to the abnormality is shown, all the tasks which are not operated need to be analyzed, the true reason of the abnormality of the task to be processed is determined by performing abnormality analysis on the previous-stage task which is not operated, and therefore an abnormal link analysis result is obtained.
As another example, when there is no task that is not running in the previous-level task, but there are running tasks and running tasks that are completed, all running tasks and tasks whose running end time exceeds the threshold time are subjected to exception analysis, so as to obtain an exception link analysis result.
Optionally, if there is no task that is not running in the previous-stage task, it indicates that the previous-stage task can be executed normally, but because there are running tasks and running tasks that are completed, all running tasks and tasks whose running end time exceeds the threshold time may be subjected to exception analysis, and a real reason that the previous-stage task is running and a reason that the running end time exceeds the threshold time are analyzed, so as to obtain an exception link analysis result.
As another example, when all tasks in the previous-stage task are tasks that are completed in operation, performing exception analysis on the tasks whose operation end time exceeds the threshold time in the previous-stage task until determining the task level without exception, and obtaining an exception link analysis result.
Optionally, if all the tasks in the previous-stage task are tasks that are completed in running, it is indicated that the previous-stage task can be completed in running, at this time, the task whose running end time exceeds the threshold time may be subjected to exception analysis, whether the previous-stage task is abnormal is analyzed, if the abnormal task exists in the previous-stage task whose running end time exceeds the threshold time, the exception condition of the parent task of the previous-stage task is recursively analyzed upwards until the task level without the abnormal task is determined, and an exception link analysis result is obtained.
According to the abnormal link analysis method, all the previous-level tasks of the tasks to be processed are determined according to the task level information, and the tasks meeting the preset conditions are subjected to abnormal analysis according to the running state of each task in the previous-level tasks, so that an abnormal link analysis result is obtained. According to the technical scheme, the real reason of the link abnormity can be determined through a recursive analysis method, and the task monitoring efficiency and the monitoring effect in a big data platform are improved.
Further, in an embodiment of the present application, the method for analyzing an abnormal link may further include the following steps:
and outputting the abnormal link analysis result in the form of a link diagram.
In this embodiment, after obtaining the abnormality analysis result, the link analysis device may represent the abnormality analysis result in the form of a link diagram, and output the abnormality analysis result in the form of a link diagram at the final output.
As an example, when the link analysis device has a display function, the link analysis device may output an abnormal link analysis result in the form of a link map through the display function itself has.
As another example, when the link analysis device does not have a display function, the link analysis device may transmit the abnormal link analysis result in the form of a link map to a display device connected thereto so that the display device can present it.
It can be understood that, in both of the above two examples, the abnormal link analysis result can be output to the user in a link diagram manner, that is, whenever the user queries the link abnormality according to the task to be processed, a real-time abnormal link analysis result is obtained and displayed to the user, so that the user can obtain an intuitive analysis result conveniently.
Illustratively, as shown with reference to fig. 6 described above, there may be abnormal tasks in the tasks of the respective hierarchies, such as the task 11 in the 1 st hierarchy, the task 23 in the 2 nd hierarchy, the task 31 in the 3 rd hierarchy, the task 43 in the 4 th hierarchy, the tasks 53 and 54 in the 5 th hierarchy, and the task 61 in the 6 th hierarchy, and the like. It is to be understood that the link diagram shown in fig. 6 is a result of performing an instantaneous analysis according to each abnormal link analysis request. And each link for connecting the next subtask by the abnormal task is an abnormal link. The thin solid lines in fig. 6 are normal link lines, and the thick broken lines are abnormal link lines. The arrow of each link line represents the task from the upper level to the next level. The present embodiment is not limited thereto.
As can be seen from the above analysis, in the embodiment of the present application, current operation information of a task to be processed, for example, information such as a task state, an operation start time, an operation end time, an operation duration, and the like, is obtained, and then a real-time analysis of an abnormal link is performed by combining a historical operation condition of the task and a direct blood relationship (hierarchical relationship) of the task, so that an abnormal link analysis result can be obtained and displayed in a link diagram manner, so that a user can locate a real condition of an abnormal reason when performing the abnormal link analysis, and judge whether an abnormality of the task is caused by a parent task or an execution abnormality of the task itself by visually observing an operation condition of an upstream task, thereby improving accuracy of task monitoring.
The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.
Fig. 7 is a schematic structural diagram of an embodiment of an abnormal link analysis device according to an embodiment of the present application. Referring to fig. 7, the abnormal link analyzing apparatus may include:
a receiving module 701, configured to receive an abnormal link analysis request of a user, where the abnormal link analysis request includes: identification of the task to be processed;
an obtaining module 702, configured to obtain current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;
the processing module 703 is configured to determine whether the task to be processed is abnormal according to the current operation information, the historical operation information, and a preconfigured service level protocol, and when it is determined that the task to be processed is abnormal, recursively analyze whether a previous-level task of the task to be processed is abnormal based on task level information until a task level without an abnormal task is determined, so as to obtain an abnormal link analysis result.
In a possible design of the embodiment of the present application, the obtaining module 702 is specifically configured to obtain, according to the identifier of the task to be processed, current operation information of the task to be processed from a cache database, and obtain historical operation information of the task to be processed from a data warehouse;
the information in the cache database is obtained by consuming metadata information of each task in a big data platform; and historical operation information of each task in the big data platform is stored in the data warehouse.
Optionally, the obtaining module 702 is further configured to obtain metadata information of each task from a business system before obtaining the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed;
the processing module 703 is further configured to clean the metadata information of each task, determine current operation information of each task, store the current operation information of each task in the cache database, where the operation information of each task in the cache database has an expiration time.
Optionally, the processing module 703 is further configured to:
determining the periodic processing time of the cache database;
when the current time reaches the periodic processing time, transmitting the running information of at least one task with the storage time exceeding the expiration time in the cache database to the data warehouse;
and updating historical operation information of each task in the data warehouse.
In another possible design of the embodiment of the present application, the processing module 703 is configured to determine whether the task to be processed is abnormal according to the current operation information, the historical operation information, and a preconfigured service level protocol, and specifically includes:
the processing module 703 is specifically configured to:
determining whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement or not according to the current operation information and the service level agreement;
and when the completion time of the task to be processed exceeds the threshold time set in the service level agreement, determining whether the task to be processed is abnormal or not according to the current operation information and the historical operation information.
In another possible design of the embodiment of the present application, the processing module 703 is configured to recursively analyze whether a previous-stage task of the to-be-processed task is abnormal based on task level information until a task level without an abnormal task is determined, and obtain an abnormal link analysis result, and specifically includes:
the processing module 703 is specifically configured to:
determining all the upper-level tasks of the tasks to be processed according to the task level information;
and according to the running state of each task in the previous-level task, performing exception analysis on the tasks meeting the preset conditions to obtain an exception link analysis result.
Optionally, the processing module 703 is configured to perform exception analysis on the task that meets the preset condition according to the running state of each task in the previous-stage task, so as to obtain an exception link analysis result, and specifically includes:
the processing module 703 is specifically configured to:
when the tasks which do not run exist in the previous-level tasks, performing exception analysis on all the tasks which do not run to obtain an exception link analysis result;
when the tasks which are not operated do not exist in the upper-level task, but the tasks which are operated and completed, all the tasks which are operated and the tasks of which the operation ending time exceeds the threshold value time are subjected to abnormal analysis, and the abnormal link analysis result is obtained;
and when all the tasks in the previous-stage task are the tasks which are finished to run, performing exception analysis on the tasks of which the running end time exceeds the threshold time in the previous-stage task until determining the task level without exception, and obtaining an exception link analysis result.
In another possible design of the embodiment of the present application, the apparatus further includes:
and the output module 704 is used for outputting the abnormal link analysis result in the form of a link map.
The apparatus provided in the embodiment of the present application may be used to implement the technical solution described in the embodiment of the method, and the implementation principle and the technical effect are similar, which are not described herein again.
It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the processing module may be a processing element separately set up, or may be implemented by being integrated in a chip of the apparatus, or may be stored in a memory of the apparatus in the form of program code, and a function of the processing module may be called and executed by a processing element of the apparatus. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Fig. 8 is a schematic structural diagram of an embodiment of an electronic device provided in the embodiment of the present application. As shown in fig. 8, the electronic device may include: the system comprises a processor 801, a memory 802, a communication interface 803 and a system bus 804, wherein the memory 802 and the communication interface 803 are connected with the processor 801 through the system bus 804 and complete mutual communication, the memory 802 is used for storing computer instructions, the communication interface 803 is used for communicating with other equipment, and the technical scheme of the method embodiment is realized when the processor 801 executes the computer instructions.
Optionally, in an embodiment of the present application, the electronic device may further include a user operation interface 805, where the user operation interface 805 may be used for an abnormal link analysis request of a user.
The receiving module 701 and the output module 704 in the abnormal link analyzing apparatus shown in fig. 7 may be implemented by a communication interface 803, and the processing module 703 may be implemented by the processor 801.
In fig. 8, the processor 801 may be a general-purpose processor, including a central processing unit CPU, a Network Processor (NP), and the like; but also a digital signal processor DSP, an application specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components.
The memory 802 may include a Random Access Memory (RAM), a read-only memory (RAM), and a non-volatile memory (non-volatile memory), such as at least one disk memory.
The communication interface 803 is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries).
The system bus 804 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
Optionally, an embodiment of the present application further provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are run on a computer, the computer is caused to execute the technical solution described in the foregoing method embodiment.
Optionally, an embodiment of the present application further provides a chip for executing the instruction, where the chip is configured to execute the technical solution described in the foregoing method embodiment.
An embodiment of the present application further provides a computer program product, including: a computer program, stored in a readable storage medium, from which at least one processor of the electronic device can read the computer program, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any of the embodiments described above.
Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (12)

1. An abnormal link analysis method, comprising:
receiving an abnormal link analysis request of a user, wherein the abnormal link analysis request comprises: identification of the task to be processed;
acquiring current operation information and historical operation information of the task to be processed according to the identifier of the task to be processed;
determining whether the task to be processed is abnormal or not according to the current operation information, the historical operation information and a pre-configured service level protocol;
and when the task to be processed is determined to be abnormal, recursively analyzing whether the previous-level task of the task to be processed is abnormal or not based on task level information until the task level without the abnormal task is determined, and obtaining an abnormal link analysis result.
2. The method according to claim 1, wherein the obtaining current operation information and historical operation information of the to-be-processed task according to the identifier of the to-be-processed task includes:
according to the identification of the task to be processed, current operation information of the task to be processed is obtained from a cache database, and historical operation information of the task to be processed is obtained from a data warehouse;
the information in the cache database is obtained by consuming metadata information of each task in a big data platform; and historical operation information of each task in the big data platform is stored in the data warehouse.
3. The method according to claim 2, wherein before the obtaining of the current running information and the historical running information of the to-be-processed task according to the identifier of the to-be-processed task, the method further comprises:
acquiring metadata information of each task from a business system;
cleaning the metadata information of each task, and determining the current operation information of each task;
and storing the current running information of each task into the cache database, wherein the running information of each task in the cache database has an expiration time.
4. The method of claim 3, further comprising:
determining the periodic processing time of the cache database;
when the current time reaches the periodic processing time, transmitting the running information of at least one task with the storage time exceeding the expiration time in the cache database to the data warehouse;
and updating historical operation information of each task in the data warehouse.
5. The method of claim 1, wherein determining whether the task to be processed is abnormal according to the current operation information, the historical operation information and a pre-configured service level agreement comprises:
determining whether the completion time of the task to be processed exceeds the threshold time set in the service level agreement or not according to the current operation information and the service level agreement;
and when the completion time of the task to be processed exceeds the threshold time set in the service level agreement, determining whether the task to be processed is abnormal or not according to the current operation information and the historical operation information.
6. The method according to any one of claims 1 to 5, wherein the recursively analyzing whether a previous task of the to-be-processed task is abnormal based on task level information until a task level without an abnormal task is determined to obtain an abnormal link analysis result includes:
determining all the upper-level tasks of the tasks to be processed according to the task level information;
and according to the running state of each task in the previous-level task, performing exception analysis on the tasks meeting the preset conditions to obtain an exception link analysis result.
7. The method according to claim 6, wherein the performing an exception analysis on the task meeting a preset condition according to the running state of each task in the previous-stage task to obtain an exception link analysis result comprises:
when the tasks which do not run exist in the previous-level tasks, performing exception analysis on all the tasks which do not run to obtain an exception link analysis result;
when the tasks which are not operated do not exist in the upper-level task, but the tasks which are operated and completed, all the tasks which are operated and the tasks of which the operation ending time exceeds the threshold value time are subjected to abnormal analysis, and the abnormal link analysis result is obtained;
and when all the tasks in the previous-stage task are the tasks which are finished to run, performing exception analysis on the tasks of which the running end time exceeds the threshold time in the previous-stage task until determining the task level without exception, and obtaining an exception link analysis result.
8. The method of claim 1, further comprising:
and outputting the abnormal link analysis result in the form of a link map.
9. An abnormal link analyzing apparatus, comprising:
a receiving module, configured to receive an abnormal link analysis request of a user, where the abnormal link analysis request includes: identification of the task to be processed;
the acquisition module is used for acquiring the current running information and the historical running information of the task to be processed according to the identifier of the task to be processed;
and the processing module is used for determining whether the task to be processed is abnormal or not according to the current operation information, the historical operation information and a pre-configured service level protocol, and recursively analyzing whether the previous-level task of the task to be processed is abnormal or not based on task level information when the task to be processed is determined to be abnormal until the task level without the abnormal task is determined, so as to obtain an abnormal link analysis result.
10. An electronic device comprising a processor, a memory and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method according to any of the claims 1-8.
11. A computer-readable storage medium having stored thereon computer instructions for implementing the method of any one of claims 1-8 when executed by a processor.
12. A computer program product, comprising: computer program, characterized in that the computer program is adapted to carry out the method of any of claims 1-8 when executed by a processor.
CN202110142549.4A 2021-02-02 2021-02-02 Abnormal link analysis method, device, equipment and storage medium Pending CN113760677A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110142549.4A CN113760677A (en) 2021-02-02 2021-02-02 Abnormal link analysis method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110142549.4A CN113760677A (en) 2021-02-02 2021-02-02 Abnormal link analysis method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113760677A true CN113760677A (en) 2021-12-07

Family

ID=78786584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110142549.4A Pending CN113760677A (en) 2021-02-02 2021-02-02 Abnormal link analysis method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113760677A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114510329A (en) * 2022-01-21 2022-05-17 北京火山引擎科技有限公司 Method, device and equipment for determining predicted output time of task node
CN114756469A (en) * 2022-04-24 2022-07-15 阿里巴巴（中国）有限公司 Data relation analysis method and device and electronic equipment
CN115842860A (en) * 2023-02-28 2023-03-24 江苏金恒信息科技股份有限公司 Monitoring method, device and system for data link

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114510329A (en) * 2022-01-21 2022-05-17 北京火山引擎科技有限公司 Method, device and equipment for determining predicted output time of task node
CN114510329B (en) * 2022-01-21 2023-08-08 北京火山引擎科技有限公司 Method, device and equipment for determining estimated output time of task node
CN114756469A (en) * 2022-04-24 2022-07-15 阿里巴巴（中国）有限公司 Data relation analysis method and device and electronic equipment
CN115842860A (en) * 2023-02-28 2023-03-24 江苏金恒信息科技股份有限公司 Monitoring method, device and system for data link

Similar Documents

Publication Publication Date Title
US10129168B2 (en) Methods and systems providing a scalable process for anomaly identification and information technology infrastructure resource optimization
CN113760677A (en) Abnormal link analysis method, device, equipment and storage medium
US10116534B2 (en) Systems and methods for WebSphere MQ performance metrics analysis
US20210133622A1 (en) Ml-based event handling
WO2022151668A1 (en) Data task scheduling method and apparatus, storage medium, and scheduling tool
US10372572B1 (en) Prediction model testing framework
CN112905323B (en) Data processing method, device, electronic equipment and storage medium
CN111400288A (en) Data quality inspection method and system
US9922116B2 (en) Managing big data for services
US11897527B2 (en) Automated positive train control event data extraction and analysis engine and method therefor
WO2015187001A2 (en) System and method for managing resources failure using fast cause and effect analysis in a cloud computing system
US11182386B2 (en) Offloading statistics collection
CN113360581A (en) Data processing method, device and storage medium
US20210224102A1 (en) Characterizing operation of software applications having large number of components
CN110011845B (en) Log collection method and system
CN109324892B (en) Distributed management method, distributed management system and device
CN114090268B (en) Container management method and container management system
CN113220530B (en) Data quality monitoring method and platform
US11838171B2 (en) Proactive network application problem log analyzer
CN113722141A (en) Method and device for determining delay reason of data task, electronic equipment and medium
CN113779017A (en) Method and apparatus for data asset management
US10733070B2 (en) Executing test scripts with respect to a server stack
CN117389841B (en) Method and device for monitoring accelerator resources, cluster equipment and storage medium
Curtis A Comparison of Real Time Stream Processing Frameworks
CN115202979A (en) SQL real-time monitoring method, system, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination