CN114385438A - Service operation risk early warning method, system and storage medium - Google Patents
Service operation risk early warning method, system and storage medium Download PDFInfo
- Publication number
- CN114385438A CN114385438A CN202111512566.9A CN202111512566A CN114385438A CN 114385438 A CN114385438 A CN 114385438A CN 202111512566 A CN202111512566 A CN 202111512566A CN 114385438 A CN114385438 A CN 114385438A
- Authority
- CN
- China
- Prior art keywords
- service
- health score
- index information
- early warning
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3471—Address tracing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3495—Performance evaluation by tracing or monitoring for systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/875—Monitoring of systems including the internet
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a business operation risk early warning method, a system and a storage medium, wherein the method comprises the following steps: s1: burying a point agent probe in each service of the distributed system; s2: when the service is called, the interceptor collects index information before calling and index information transferred to the next service after calling, calculates the health score of the service according to the index information and the scoring rule of the corresponding service in the health score database, and stores the health score in the database; s3: calculating a health score of the system according to the weight and the health score of each service in the health score database; s4: collecting and analyzing calling link information to form a network topological graph of the system; s5: and displaying the index information and the health score of each service on the corresponding node of the network topological graph. The method can clearly show the health state of each service, and realize the comprehensive monitoring of the system. Operation and maintenance personnel and testing personnel can find the problem quickly when the fault occurs, and the working efficiency is improved.
Description
Technical Field
The invention relates to the technical field of business risk early warning, in particular to a business operation risk early warning method, a business operation risk early warning system and a storage medium.
Background
Risk early warning based on big data is a control means for realizing advanced accident prevention by depending on big data and related technologies thereof. The physical hardware equipment of the power system is highly dependent on the information space of the system, so that the safety of the power system is directly influenced by the information system. The propagation of the risk of the power system has strong coupling, for example, when the information system fails, the normal operation of the measuring equipment and the control terminal is also affected, so that the normal operation of the whole system is further affected. With the continuous expansion of the scale of the whole system, the amount of log information accumulated in the information system is huge, the types of the information are various, a large amount of information related to the operation state of the system can be hidden, the logs are shown to have typical big data characteristics, and index data and the like acquired by the system in real time in the same way also have big data characteristics. If a log collection and processing means of big data is adopted to deeply mine log information and system index data collected in real time, possible problems in the operation of the system can be found in time, fault early warning of the system is realized, and unnecessary loss is avoided.
The existing protection means can not identify the abnormity of a service layer, a data flow panoramic intelligent monitoring view of the service needs to be established, and the whole state of the full node state of the data flow needs to be intelligently analyzed.
Disclosure of Invention
The purpose of the invention is as follows: the invention aims to provide a business operation risk early warning method which can analyze the panoramic health state of a business data stream and clearly know the health condition and the non-health reasons of each service.
Another objective of the present invention is to provide a system capable of implementing the business operation risk early warning method, and a storage medium storing a computer program instantiated by the method.
The technical scheme is as follows: the business operation risk early warning method comprises the following steps:
s1: burying a point agent probe in each service of the distributed system;
s2: when the service is called, the interceptor collects index information before calling and index information transferred to the next service after calling, calculates the health score of the service according to the index information and the scoring rule of the corresponding service in the health score database, and stores the health score in the database;
s3: calculating a health score of the system according to the weight and the health score of each service in the health score database;
s4: collecting and analyzing calling link information to form a network topological graph of the system;
s5: and displaying the index information and the health score of each service on the corresponding node of the network topological graph.
Further, the agent probe in the step S1 is a plug-in java agent.
Further, a sampling frequency is set in the agent probe in the step S1.
Further, in step S2, if the interceptor collects information that carries Context, the interceptor forcibly collects index information of the service.
Further, the interceptor in step S2 is an abstract method interceptor.
Further, after the step S5, the method further includes:
s6: a log is formed from the call link information and the health service score.
The business operation risk early warning system of the invention comprises: an Agent end: the device is used for loading agent probes, burying points in each service of the distributed system and collecting link information; an interceptor: the system is used for collecting index information before and after service calling when the service is called; the OAP server: the system is used for analyzing the link information and the index information, calculating the health score of the service according to the health score database and storing the health score in the database; and the UI end: and the system is used for forming a network topology map of the system by calling the link information and displaying the index information and the health score of each service and the health score of the system on corresponding nodes of the network topology map.
The storage medium of the present invention stores a computer program, and the computer program is configured to implement the service operation risk early warning method when running.
Has the advantages that: compared with the prior art, the invention has the following advantages: 1. acquiring index information of the service by burying a point agent probe in the service, calculating a health score, and finally displaying the index information and the index information on corresponding nodes of a network topological graph to realize monitoring and understanding of the health condition of each service in the system; 2. the agent probe adopts a plug-in JavaAgent form, so that the integration is convenient; 3. the agent is internally provided with a sampling period, and whether forced sampling is carried out is judged according to whether context exists in the interception information or not, so that the completeness of a link is ensured, and meanwhile, huge data volume is avoided.
Drawings
Fig. 1 is a flowchart of a service operation risk early warning method according to an embodiment of the present invention;
fig. 2 is a system block diagram of a service operation risk early warning system according to an embodiment of the present invention.
Detailed Description
The technical scheme of the invention is further explained by combining the attached drawings.
Referring to fig. 1, the service operation risk early warning method according to the embodiment of the present invention includes the following steps:
s1: burying a point agent probe in each service of the distributed system;
s2: when the service is called, the interceptor collects index information before calling and index information transferred to the next service after calling, calculates the health score of the service according to the index information and the scoring rule of the corresponding service in the health score database, and stores the health score in the database;
s3: calculating a health score of the system according to the weight and the health score of each service in the health score database;
s4: collecting and analyzing calling link information to form a network topological graph of the system;
s5: and displaying the index information and the health score of each service on the corresponding node of the network topological graph.
According to the business operation risk early warning method of the technical scheme, the agent probe is intercepted by the interceptor to obtain index information before and after service calling, the health score of the service is calculated according to the scoring rule pre-stored in the health score database, the health score of the system is calculated according to the weight of each service and the health score, and finally the health score is displayed on the corresponding node of the system network topological graph, so that operation and maintenance personnel can clearly know the health condition and the reasons of non-health phenomena of each service while obtaining the associated information between services in the whole system, and the system can be comprehensively monitored. The problem can be found fast when breaking down to fortune dimension personnel and the tester of being convenient for, and the timely processing problem practices thrift manpower and materials, promotes fortune dimension and tester's work efficiency.
In order to facilitate integration and facilitate custom integration for projects, in this embodiment, the agent probe of the buried point is in a JavaAgent plus plug-in form, and different technologies can be applied to integrate various plug-ins for different projects. Meanwhile, in order to avoid overlarge data volume, each agent probe is provided with sampling frequency, and the sampling frequency is acquired once every 1 second by default. In practice, each service call is unlikely to be at a uniform point in time, and there is network delay, etc., which may cause some calls to be sampled on service a and not on services B and C, thus failing to analyze the performance of the call chain. Therefore, in this embodiment, if there is a carry Context in the upstream before the call, which indicates that the upstream is sampled, the downstream service enforces sampling, so as to avoid a huge data volume while ensuring that the link is complete.
In this embodiment, the health score database stores whether a service is a critical service, a weight of the service, and an index threshold and a deduction value of an index of index information of each service. And calculating the health score of the service by adopting a deduction mode, wherein the full score is 100, the deducted score is determined according to the importance of the index, if the service downtime is serious, the deduction value is 100, the influence of the CPU utilization rate exceeding a set threshold value is small, and the deduction value is 10. The overall system health score is the sum of the product of the score of each service and its weight in the overall system. If any of the health scores marked as critical services is 0, the health score of the entire system is 0 regardless of its weight.
Example 1: the system to be monitored contains A, B, C three services. The service A is a critical service, the weight is 0.8, the cpu utilization threshold is 30% -85%, and the deduction fraction required by the index exceeding the threshold is set to be 20 in advance; the service B is a noncritical service, the weight is 0.1, and the deduction fraction required by service offline is set to be 100 in advance; service C is a noncritical service, the weight is 0.1, the service response time index threshold is 100ms, and the preset score is 30. When the service Acpu usage rate is 90%, the service B is down, the service C response time is 200ms, and the overall health score of the system is 71.
Example 2: the system to be monitored comprises A, B, C, file service and other four services. Therefore, when the weight preset value of a certain service as a critical service, such as a file service, is 0.3, and no matter the weight is any value, a downtime (deduction of 100 points) occurs in the operation process, the file service is 0 points, and the health score of the whole system is 0 points. When the overall score of the project is low, the reason that the overall score is low can be visually analyzed, for example, the CPU utilization rate exceeds 90%, the memory is about to overflow, and the like, and alarm information is sent to the emergency contact person so as to be processed in time.
In practice, in addition to displaying the health score and index information of each service at the corresponding node in the network topology, the health score and index information of each service can also be displayed, the call information, the response time, the circulation process and the like of each service can be displayed, and the services can be ranked according to the call chain response time, so that the services with longer response time can be adjusted and optimized conveniently. Meanwhile, a log is formed according to the collected call link information, and the rule for forming the log can be configured by technicians in a user-defined mode according to project requirements.
Referring to fig. 2, the business operation risk early warning system according to the embodiment of the present invention includes an Agent end, an interceptor, an OAP server, and a UI end. The Agent end is used for loading the Agent probe, is buried in each service of the distributed system and collects link information; the interceptor is used for collecting index information before and after the service is called when the service is called; the OAP server is used for analyzing the link information and the index information, calculating the health score of the service according to the health score database and storing the health score in the database; and the UI end is used for forming a network topological graph of the system by calling the link information and displaying the index information and the health score of each service on a corresponding node of the network topological graph.
In this embodiment, the system is built on the basis of the SkyWalking tool. The Agent end adds Agent probes (-javaagent: corresponding path/Agent name. jar) in each service of the project to be monitored, so that logs corresponding to the system are collected without invasion, the number of micro-services passed by one request in the system is monitored, and middleware such as a database, redis, mq and the like is called midway.
When loading the agent probe, firstly loading and configuring the agent probe from the agent.config into the Properties, loading the file of the agent.config into a file stream, and then reading the file by using the Properties; analyzing the environment variable and covering a corresponding static field in the config; the JAVAagent parameter is parsed and the corresponding static field in config is overwritten. Then finding and parsing the skywalk-plug-in. Initializing a default class loader, loading plugin. def resources by using class loader. getresources (), scanning out the resources, putting the resources into a PluginCfg. PluginClassList, scanning the loaded list, initializing by reflecting class. forName, and then loading into a plug-in; traversing instantiated Abstract ClassClassEnhancePluginDefine and classifying according to ClassMatcher type returned by enhanceClass () method, putting the type into different places according to plug-in types, filtering some and enhancing some, if the return value type is a Match type, recording the corresponding Abstract ClassEnhancePluginDefine into a nameMatchDefine set, if the return value is not a Match type, recording Abstract ClassEnhancePluginDefine into a signatureMatchDefine set, traversing nameMatchDefine and signatureMatchDefine sets, and determining all matched plug-ins by using class Matisch () method. And (3) creating Agentbuild by using a ByteBuddy library, dynamically enhancing the target class according to the loaded plug-in, and inserting the embedded point logic. And then, using a JDK SPI loading mode and starting BootService service, starting all services, such as JVM condition monitoring service JVMservice, and then executing prefix, startup and onComplete. And finally, adding a JVM hook, and closing all BootService services when the JVM exits.
When a service is called, parameters passed in the service SpringMVC from the agent probe site are intercepted by an abstract methodinterface interceptor. After the data integration is finished, the TraceSegmentServiceClient is responsible for serializing trace data and sending the trace data to the OAP cluster, a production consumer is maintained in an agent internal cache, the produced data is placed into the cache according to partitions when the data is collected, the consumer consumes the data by multiple threads, the cached data is packaged into GRPC data objects and sent to the collector, all spans are completed, and then the given span is stopped. To track the kernel must ensure that the span must match in the stack module; this context is completed and the owner is notified.
In the OAP server, a TraceSegmentReportServerHandler (distributed Link trace processor) receives data sent by an agent through a GRPC, then sends the data to a receiver through a Segment ParaService.send (Segment), and sends the data to a TraceAnalyzer to analyze the transmitted Segment data.
When the TraceAnalyzer # doAnalyzer method receives Segment data, analyzing, firstly judging that no Span data directly returns, then creating a Span listener, notifying the Segment listener, and monitoring the whole Segment process; the parsing core sends the data to the persistence layer by notifying the portal listener notifyEntryListener (spanObject, segmentObject) after the parsing point when it traverses the segment. And scheduling the segment data, and analyzing the segment data into a persistent layer analyzable mode record segment entity. Finally, the given metrics are transferred to the executable insert statement by the recordpersystem worker # in method, which is driven by the stream processing, and the data is pushed into the database in an asynchronous mode.
And the UI terminal transmits the query condition to a GraphQLQueryHandler # doPost method through GraphQL. In the doPost method, firstly, the buffer reader provides a general buffer mode for text reading, a readLine method is provided, a text line is read, the text is read from a character input stream, and each character is buffered, so that efficient reading of characters, arrays and lines is provided. After reading the byte stream information of the req, converting the byte stream information into text information and adding the text information into the request; and then converting the json object into a java entity, finally analyzing the java entity and constructing an inquiry statement, inquiring by the graph QL to obtain an execution result, and returning the execution result to the front-end UI display interface.
The storage medium according to the embodiment of the present invention stores the computer program instantiated by the service operation risk early warning method.
Claims (8)
1. A business operation risk early warning method is characterized by comprising the following steps:
s1: burying a point agent probe in each service of the distributed system;
s2: when the service is called, the interceptor collects index information before calling and index information transferred to the next service after calling, calculates the health score of the service according to the index information and the scoring rule of the corresponding service in the health score database, and stores the health score in the database;
s3: calculating a health score of the system according to the weight and the health score of each service in the health score database;
s4: collecting and analyzing calling link information to form a network topological graph of the system;
s5: and displaying the index information and the health score of each service on the corresponding node of the network topological graph.
2. The business operation risk early warning method according to claim 1, wherein the agent probe in the step S1 is a plug-in java agent.
3. The business operation risk early warning method according to claim 1, wherein a sampling frequency is set in the agent probe in the step S1.
4. The business operation risk early warning method according to claim 3, wherein in step S2, if the information collected by the interceptor carries Context, the interceptor forcibly collects index information of the service.
5. The business operation risk early warning method according to claim 1, wherein the interceptor in step S2 is an abstract method interceptor.
6. The business operation risk early warning method according to claim 1, wherein the step S5 is followed by further comprising:
s6: a log is formed from the call link information and the health service score.
7. A service operation early warning system, comprising:
an Agent end: the device is used for loading agent probes, burying points in each service of the distributed system and collecting link information;
an interceptor: the system is used for collecting index information before and after service calling when the service is called;
the OAP server: the system is used for analyzing the link information and the index information, calculating the health score of the service according to the health score database and storing the health score in the database;
and the UI end: and the system is used for forming a network topology map of the system by calling the link information and displaying the index information and the health score of each service and the health score of the system on corresponding nodes of the network topology map.
8. A storage medium storing a computer program, wherein the computer program is configured to implement the service operation warning method according to any one of claims 1 to 6 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111512566.9A CN114385438A (en) | 2021-12-08 | 2021-12-08 | Service operation risk early warning method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111512566.9A CN114385438A (en) | 2021-12-08 | 2021-12-08 | Service operation risk early warning method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114385438A true CN114385438A (en) | 2022-04-22 |
Family
ID=81196910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111512566.9A Pending CN114385438A (en) | 2021-12-08 | 2021-12-08 | Service operation risk early warning method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114385438A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115967649A (en) * | 2022-11-09 | 2023-04-14 | 北京白龙马云行科技有限公司 | Service health degree checking method and system based on service topological relation |
-
2021
- 2021-12-08 CN CN202111512566.9A patent/CN114385438A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115967649A (en) * | 2022-11-09 | 2023-04-14 | 北京白龙马云行科技有限公司 | Service health degree checking method and system based on service topological relation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111092852B (en) | Network security monitoring method, device, equipment and storage medium based on big data | |
| CN110888783B (en) | Method and device for monitoring micro-service system and electronic equipment | |
| CN111404909B (en) | Safety detection system and method based on log analysis | |
| CN108259268B (en) | Network monitoring data processing method and device, computer equipment and storage medium | |
| CN103220173B (en) | A kind of alarm monitoring method and supervisory control system | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| CN111614483A (en) | Link monitoring method and device, storage medium and computer equipment | |
| CN112069049A (en) | Data monitoring management method and device, server and readable storage medium | |
| CN112699007A (en) | Method, system, network device and storage medium for monitoring machine performance | |
| CN114356499A (en) | Kubernetes cluster alarm root cause analysis method and device | |
| CN102056200A (en) | Service processing flow monitoring method and system | |
| US7617313B1 (en) | Metric transport and database load | |
| CN111130944B (en) | System monitoring method and system | |
| CN114385438A (en) | Service operation risk early warning method, system and storage medium | |
| US8554908B2 (en) | Device, method, and storage medium for detecting multiplexed relation of applications | |
| CN107968727A (en) | A kind of detection method, device and the medium of CIFS services | |
| CN113282920B (en) | Log abnormality detection method, device, computer equipment and storage medium | |
| CN113656252A (en) | Fault positioning method and device, electronic equipment and storage medium | |
| US9645877B2 (en) | Monitoring apparatus, monitoring method, and recording medium | |
| CN110109803B (en) | User behavior reporting method and system | |
| CN110633165B (en) | Fault processing method, device, system server and computer readable storage medium | |
| CN115686381B (en) | Prediction method and device for storage cluster running state | |
| CN116662127A (en) | Method, system, equipment and medium for classifying and early warning equipment alarm information | |
| CN105978749A (en) | Monitoring method of computer hardware information in local area network and system thereof | |
| CN108289035B (en) | Method and system for visually displaying running states of network and business system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |