CN113157545A - Method, device and equipment for processing service log and storage medium - Google Patents
Method, device and equipment for processing service log and storage medium Download PDFInfo
- Publication number
- CN113157545A CN113157545A CN202110551959.4A CN202110551959A CN113157545A CN 113157545 A CN113157545 A CN 113157545A CN 202110551959 A CN202110551959 A CN 202110551959A CN 113157545 A CN113157545 A CN 113157545A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- service
- log data
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method, a device, equipment and a storage medium for processing a service log, wherein the method comprises the following steps: collecting log data of a service system and transmitting the log data to a real-time stream computing component through message distribution middleware, wherein the log data comprises at least one service level; preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level and storing the log streams to a search engine; analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result; and executing corresponding processing operation based on the alarm pushing result. According to the technical scheme, the service problems existing in the log stream can be automatically found through the preset alarm rule, so that the possibility of fault occurrence can be pre-judged in advance, the alarm push result can be accurately generated, the corresponding processing operation is executed based on the alarm push result, the fault problems can be timely processed, and the service processing efficiency is improved.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a service log.
Background
With the upgrade and expansion of company services, the service types are more abundant, more and more application systems are introduced and deployed, various application systems are deployed on numerous servers, and in the operation process of the systems, various service logs can be output to reflect the system state, feedback of service execution conditions and the like. However, in the service operation process, some service modules may have a fault, which affects the service operation, and therefore, it is very important to monitor and process the service modules.
At present, in the related art, a corresponding service monitoring system is established to monitor a service module, but in an environment with a large service volume, a large number of service logs can be generated, and a traditional service monitoring system cannot accurately position the large number of service logs, so that a fault cannot be timely processed, and thus the service processing efficiency is reduced.
Disclosure of Invention
In view of the foregoing defects or shortcomings in the prior art, it is desirable to provide a method, an apparatus, a device and a storage medium for processing a service log.
In a first aspect, the present application provides a method for processing a service log, where the method includes:
collecting log data of a service system and transmitting the log data to a real-time stream computing component through message distribution middleware, wherein the log data comprises at least one service level;
preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level and storing the log streams to a search engine;
analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result;
and executing corresponding processing operation based on the alarm pushing result.
In a second aspect, the present application provides an apparatus for processing a service log, where the apparatus includes:
the data transmission module is used for collecting log data of a service system and transmitting the log data to the real-time stream computing component through the message distribution middleware, wherein the log data comprises at least one service level;
the storage module is used for preprocessing the log data through the real-time stream computing component, generating log streams corresponding to all the service levels and storing the log streams to a search engine;
the analysis module is used for analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result;
and the execution module is used for executing corresponding processing operation based on the alarm pushing result.
In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the first aspect when executing the program.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, the computer program being configured to implement the method according to the first aspect.
The method, the device, the equipment and the storage medium for processing the service logs, provided by the embodiment of the application, are used for acquiring log data of a service system, transmitting the log data to the real-time stream computing component through the message distribution middleware, preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level, storing the log streams to a search engine, analyzing the log streams on the search engine according to preset alarm rules, generating alarm pushing results, and executing corresponding processing operations based on the alarm pushing results. According to the technical scheme, the service problems existing in the log stream can be automatically found through the preset alarm rule, so that the possibility of fault occurrence can be pre-judged in advance, the alarm push result can be accurately generated, the corresponding processing operation is executed based on the alarm push result, the fault problems can be timely processed, and the service processing efficiency is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1 is a schematic structural diagram of a service log processing system according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for processing a service log according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a method for processing a service log according to an embodiment of the present application;
fig. 4 is a schematic interface diagram of an alarm pushing result provided in the embodiment of the present application;
fig. 5 is a schematic interface diagram for executing corresponding operations according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a method for processing a service log according to an embodiment of the present application;
FIG. 7 is a system architecture diagram of an operation and maintenance platform according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a device for processing a service log according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a device for processing a service log according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
It can be understood that with the development of network information technology, a great deal of convenience is provided for users by using large data service systems, which may generally include a plurality of servers, and the completion of each service requires the cooperation of the plurality of servers. In the execution process of a certain service flow, each service server generates a service log, and in the service operation process, a fault of a service module cannot be found in time, which leads to continuous propagation and upgrading, even affects all services. At present, in the related art, a service monitoring system is established to monitor a service module, but in an environment with a large service volume, a large number of service logs are generated, and a traditional service monitoring system cannot accurately position the large number of service logs, so that a fault cannot be processed in time, and thus the service processing efficiency is reduced.
Based on the above defects, the present application provides a method, an apparatus, a device, and a storage medium for processing a service log, and compared with the related art, the scheme can automatically find a service problem existing in a log stream through a preset alarm rule, so that the possibility of a fault occurrence can be pre-determined in advance, an alarm push result can be accurately generated, and a corresponding processing operation is executed based on the alarm push result, so that the fault problem can be processed in time, and further the processing efficiency of the service is improved.
Fig. 1 is a diagram of an implementation environment structure of a service log processing method according to an embodiment of the present application. As shown in fig. 1, the implementation environment structure includes: a terminal 100 and a server 200.
Optionally, the terminal 100 may be a smart phone, a notebook computer, a tablet computer, or the like, and the application does not limit the type of the operating system, and for example, the operating system may be an Android operating system, an apple (ios) operating system, a Windows (Windows) operating system, or the like. The terminal 100 has a client running thereon, and a user can perform any service operation on the client of the terminal 100, thereby generating log data and storing the service data.
The server 200 may be a server, or may be a server cluster composed of several servers, or the server 200 may include one or more virtualization platforms, or the server 200 may be a cloud computing service center.
The server 200 may be a server device that provides a background service for the application system installed in the terminal 100. The server 200 has a data processing function.
The terminal 100 and the server 200 establish a communication connection therebetween through a wired or wireless network.
Optionally, the wireless network or wired network described above uses standard communication techniques and/or protocols. The Network is typically the Internet, but may be any Network including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wireline or wireless Network, a private Network, or any combination of virtual private networks.
For convenience of understanding and explanation, a method, an apparatus, a device, and a storage medium for processing a service log according to an embodiment of the present application are described in detail below with reference to fig. 2 to 10.
Fig. 2 is a schematic flowchart of a method for processing a service log according to an embodiment of the present application, where the method is applied to a device for processing a service log, and as shown in fig. 2, the method includes:
s101, collecting log data of a service system and transmitting the log data to a real-time stream computing component through message distribution middleware, wherein the log data comprises at least one service level.
Specifically, the service exists in the service network in a micro-service form, the service system generates log data during the operation process, the log data can be collected through Filebeat and Logstash, the log data can also be collected through a log API interface, the collected log data can be transmitted to the message distribution middleware in a UDP transmission manner, and the message distribution middleware can be kafka.
It should be noted that filebed is a lightweight log collector implemented by using gold, is used for reporting and collecting logs, is also a member of the elastic search stack, and can read logs at corresponding positions according to configuration and report the logs into the kafka data pipeline. Logstash is a server-side data processing pipeline that can dynamically collect data from multiple sources, transform the data, and then send the data into a kafka data pipeline.
The log data may be a binary log file, and is used to record sql statement information of a user operating on data of the service system, for example, sql statements that change a database table and change contents are recorded in the log data.
Optionally, the log data of the service system may be collected first, where the collected log data may be in a row log format or a json string log format, then the data format of the log data is unified, when the log data is collected, a corresponding service level is marked for each log data to determine the service level of the log data in the unified data format, and according to the service level, the corresponding log data is sent to the message distribution middleware kafka, and then the log data is transmitted to the real-time stream computing component through the kafka, where the real-time stream computing component may be, for example, Flink.
It should be noted that Kafka is a high-throughput distributed publish-subscribe messaging system, which can process all action flow data of consumers in websites, and these data are usually solved by processing logs and log aggregation due to throughput requirements, and the purpose of Kafka is to unify online and offline message processing through a parallel loading mechanism of Hadoop, in order to provide real-time messages through clustering. Multithreading concurrency can be performed by utilizing the kafka open source component, and high-performance processing is realized. The collected log data can comprise a plurality of pieces of message data, each piece of message data has a category, the category is called topic, and users can respectively store messages of different service categories to different topics according to own service forms, and the messages of different topics are stored separately in physics. Kafka manages messages with topics, each of which contains one or more message partitions.
The Flink is a distributed processing engine for streaming data and batch data, can execute any data program in a data parallel and pipeline mode, and can execute batch processing and streaming processing programs by a pipeline runtime system of the Flink.
S102, preprocessing the log data through a real-time stream computing component, generating log streams corresponding to each service level and storing the log streams to a search engine.
In this step, after obtaining log data of different service levels, the log data corresponding to each service level may be analyzed and filtered through the real-time stream computing component to obtain filtered log data, then desensitize the filtered log data to generate log streams corresponding to each service level, and store the log streams corresponding to each service level to a search engine according to a preset storage manner, where the search engine may be, for example, an elastic search engine.
Specifically, when the log data is analyzed and filtered, a data format corresponding to the log data may be determined, and the keyword in the data format may be analyzed to obtain log information, filter out other interference data, and then perform desensitization processing on the filtered log data. After desensitization processing is performed on the filtered log data and a log stream corresponding to each service level is generated, the log stream corresponding to each service level can be stored in an elastic search according to different service levels and a preset storage mode. The preset storage mode may be that log streams of the same service level are stored in the same storage database. For example, the log data may be divided into cold data, hot data, and warm data according to the service level, where the cold data refers to offline log data, the hot data refers to log data with high real-time performance, and the warm data refers to data other than the cold data and the hot data. The cold data, the hot data and the warm data may be stored in corresponding storage databases in the Elasticsearch log cluster, for example, the cold data may be stored in an HDFS file Database, the warm data may be stored in a Database, and the hot data may be stored in a Cache.
It can be understood that the above described elasticsearch engine, as a powerful distributed search engine, supports near real-time data storage and search, and the data storage amount can reach 1 hundred million documents, and supports log query, and the daily average query amount can reach 5 ten thousand. With the rapid development of services, the ES erection scheme of the log center is continuously evolved, and the read-write stability of the ES cluster can be guaranteed in real time.
The data desensitization refers to data deformation of certain sensitive information through desensitization rules, reliable protection of sensitive private data is achieved, under the condition that customer safety data or some commercial sensitive data are related, real data are modified and test use is provided under the condition that system rules are not violated, and data desensitization is needed to be carried out on personal information such as user identity numbers, mobile phone numbers, bank card numbers, customer numbers and the like. Desensitization processing may be performed on the filtered log data using a desensitization algorithm, which may be, for example, preserving beginning and ending characters, with middle portions represented by special characters such as asterisks as masks.
For example, the desensitization treatment of the identification number 120115201406180712 may be: 120115 × 0712; the desensitization process of the bank card number 9558820200019833888 may be: 955882*********3888.
In the embodiment, the collected log data of the service system can be uniformly stored, the service levels of the log data are graded when the log data are collected, and the log data are graded and stored in the elastic search, so that the log data can be uniformly managed, the service data can be displayed on a monitoring platform, and members in a team can observe the service operation condition in real time.
S103, analyzing the log stream on the search engine according to a preset alarm rule, and generating an alarm pushing result.
In this step, the alarm rule is set by the developer in advance based on the alarm service according to different service levels of the log data, and the alarm rule can be set by writing a BOE _ ALERT program. The ALERT service may be, for example, a BOE _ ALERT service, where the BOE _ ALERT service implements an elastic search and an Instant Message (IM) http restful, and a business party may directly access the BOE _ ALERT service through api, and the BOE _ ALERT service operates in an ALERT platform, and the business party may directly access the ALERT platform using a page in an interface display manner. The BOE _ ALERT service also provides functions of access recording, alarm rule setting, current limiting, degradation and the like, and provides functions of the same access, management and control of application authority and the like.
On the basis of the foregoing embodiment, as an implementable manner of S103, fig. 3 is a schematic flow diagram of a method for performing alarm analysis on a log stream according to an embodiment of the present application, and as shown in fig. 3, the method includes:
s201, sending a data query request to a search engine, wherein the data query request comprises a log identifier.
S202, receiving query result information corresponding to the log identification sent by the search engine, wherein the query result information is used for indicating the number and the service level of the log streams corresponding to the log identification received in a first preset time.
S203, generating an alarm pushing result according to a preset alarm rule based on the number and the service level of the log streams corresponding to the received log identification in the first preset time.
Specifically, taking a search engine as an Elasticsearch as an example, in the process of performing alarm analysis on a log stream in the Elasticsearch, a data query request may be sent to the Elasticsearch through instant messaging software, where the data query request includes a log identifier, and the Elasticsearch receives and responds to the data query request, obtains and sends query result information based on the log identifier, so that the instant messaging software receives the query result information corresponding to the log identifier, and the query result information is used to indicate the number of log streams and the service level corresponding to the log identifier received in a first preset event.
Optionally, the count value may be determined based on the number of log streams and the service level corresponding to the log identifier received within the first preset time, and the count value is accumulated within the second preset time to obtain a sum of the count values, where the second preset time is greater than the first preset time. Then, comparing the count values with a preset threshold value, and generating an alarm pushing result when the sum of the count values is greater than the preset threshold value; and when the sum of the count values is not greater than a preset threshold value, not performing alarm pushing, and indicating that the log stream is in a normal state, wherein the first preset time and the second preset time can be experience values set in advance according to requirements.
In this step, in the process of determining the count value, a predefined data dimension weight value corresponding to each service level may be obtained first, then the number of log streams corresponding to the service level and the corresponding data dimension weight value are subjected to weighting summation, and the count value is determined, where the data dimension weight value is predefined, and different data dimension weight values are defined according to the difference in the service level. The first data dimension weight value may be a data dimension weight value corresponding to a log stream of a first level, and the first level may be a level corresponding to hot data; the second data dimension weight value may be a data dimension weight value corresponding to a log stream of a second level, and the second level may be a level corresponding to the temperature data; the third data dimension weight value may be a data dimension weight value corresponding to a third level of the log stream, which may be a level corresponding to cold data. Can be expressed by the following formula:
wherein x is1Number of log streams, x, corresponding to the first level2Number of log streams, x, corresponding to the second levelkFor the number of log streams corresponding to the k-th level, f1Is a first data dimension weight value, f2Is a second data dimension weight value, fkIs the k-th data dimension weight value, n is the number,
is a count value.
For example, the first preset time may be 1 minute, the second preset time may be 5 minutes, the number and the number of log streams and the service level corresponding to different service levels within the first preset time may be obtained first, and for example, the first level, the second level, and the third level may be included, the number of the log streams corresponding to the first level is 0, the number of the log streams corresponding to the second level is 1, and the number of the log streams corresponding to the third level is 1, then a data dimension weight value corresponding to each service level is obtained, for example, the first data dimension weight value corresponding to the first level is 80%, the second data dimension weight value corresponding to the second level is 15%, and the third data dimension weight value corresponding to the third level is 5%, and then the number 0 of the first level and the first data dimension weight value are weighted, and the number 1 and the second data dimension weight value of the second level are respectively 15%, (for example, And performing weighted summation on the number 1 of the third level and a third data dimension weight value of 5% to obtain a count value of 0.2, then continuing to obtain count values in the next 1 minute to obtain five count values in five minutes, wherein the obtained five count values are 0.2, 0.3, 0.2, 0.2 and 0.2 respectively, adding and accumulating the five count values to obtain a sum of count values of 1.1, and when the sum of count values of 1.1 is greater than a preset threshold value of 1, generating an alarm pushing result.
The alarm pushing result may be, for example, triggering an alarm message to the instant messaging software, for example, displaying the alarm message in the form of a check box on an instant messaging software interface, so as to remind the user to process the problem fault in time. As shown in fig. 4, for example, an alarm time, an alarm source, and alarm content may be displayed on the interface, and the alarm time may be, for example, 0: 31PM, the source of the alarm may be, for example, from mars, and the content of the alarm may be, for example, the following: "the applet reports errors, now you choose # current error log for a total of 242".
In the embodiment, the possibility of fault occurrence can be pre-judged in advance by setting the alarm rule, so that a user can timely process problem faults.
And S104, executing corresponding processing operation based on the alarm pushing result.
Specifically, after the alarm pushing result is pushed to the user, the user may operate on the interface according to the alarm pushing result, so that the service processing device receives a processing operation instruction of the user, where the processing operation instruction carries a processing type determined based on the alarm pushing result, and then, in response to the processing operation instruction, executes a processing operation corresponding to the processing type. For example, the processing type may be a restart system, may also be checking error details, or may be other processing operations, as shown in fig. 5, when a user clicks a check box of an alarm push result, three processing types may appear, which are "click restart service", "check error details", "do nothing", or see.
In this step, the user can make a corresponding selection in the instant communication software to trigger the human-computer interaction interface, thereby completing the fault processing of the service problem.
For more clearly describing the present application, fig. 6 is a schematic structural diagram of a processing method of a service log provided in an embodiment of the present application, please refer to fig. 6, which can acquire log data through Filebeat and logstation, or acquire log data through a log API interface, where the acquired log data can be transmitted to Kafka for data aggregation through a UDP transmission method, the log data can include at least one service level, Kafka performs message management on log data of different service levels with topic, and transmits the log data to a real-time stream calculation engine fin for data preprocessing, for example, desensitization processing can be included, log streams corresponding to the service levels are generated, and the log streams corresponding to each service level are stored in an Elasticsearch according to a preset storage method.
After the log data is stored in the elastic search, log application can be carried out, including visual display of the log data on a monitoring platform, so that feasible management is realized; the man-machine interaction processing can also be realized by instant messaging software, for example, a data query request can be sent to the Elasticsearch by the instant messaging software, the data query request comprises a log identifier, then, according to a preset alarm rule, a log stream on the Elasticsearch and the log identifier is analyzed, an alarm push result is generated, and a user executes corresponding processing operation based on the alarm push result, wherein the alarm rule is realized by writing a BOE _ ALERT alarm program, so that the alarm push is performed; or the log data can be queried through an API interface.
Fig. 7 is a system architecture diagram of an operation and maintenance platform provided in an embodiment of the present application, and as shown in fig. 7, the system architecture may include a presentation layer, a load balancing layer, a gateway layer, a service layer, a public technology layer, and an infrastructure layer.
Wherein, the infrastructure layer can comprise MySql cluster, Redis cluster, elastic search cluster and RabbitMQ cluster; the public technology layer can communicate with the infrastructure layer and the service layer and can be used for performing gateway service, authentication service, task scheduling, cache service, report service, workflow engine, log service, notification service and the like; the service layer can be communicated with the public technology layer and the gateway layer and can comprise a service microservice, the service microservice can comprise a data terminal and a home Internet of things, and the data terminal can be communicated with the home Internet of things through an Istio Client/http communication protocol; the gateway layer can communicate with the load balancing layer and the service layer, can comprise a plurality of gateways, and can authenticate and authorize the gateways through the authentication center cluster; the load balancing layer is respectively communicated with the display layer and the gateway layer, and can comprise at least one NGINX, wherein the NGINX is used for receiving data sent by the gateway layer, each NGINX is communicated through keepalive, and then the data are transmitted to the display layer for display; the display layer may include an operation and maintenance background, a monitoring background, a management background, and the like running on the terminal device, so that a developer or an operation and maintenance person can view log data in real time in the operation and maintenance background and the monitoring background, where the terminal device may be, for example, a mobile terminal, a tablet computer, or other processing devices; a client or an operator can use the terminal equipment to execute corresponding operation through a client such as a website running on the terminal equipment; and the operator can check, monitor and manage the log data in real time in the management background.
In the framework, log data can be monitored through a Prometous monitoring and (Elasticisch, Logstash and Kiabana, ELK for short) log center, and the operation and maintenance functions of the system can be realized through Maven, SVN/Git, Jenkins, Docker and the like.
The method, the device, the equipment and the storage medium for processing the service logs, provided by the embodiment of the application, are used for acquiring log data of a service system, transmitting the log data to the real-time stream computing component through the message distribution middleware, preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level, storing the log streams to a search engine, analyzing the log streams on the search engine according to preset alarm rules, generating alarm pushing results, and executing corresponding processing operations based on the alarm pushing results. According to the technical scheme, the service problems existing in the log stream can be automatically found through the preset alarm rule, so that the possibility of fault occurrence can be pre-judged in advance, the alarm push result can be accurately generated, the corresponding processing operation is executed based on the alarm push result, the fault problems can be timely processed, and the service processing efficiency is improved.
On the other hand, fig. 8 is a schematic structural diagram of a device for processing a service log according to an embodiment of the present application. The apparatus may be an apparatus in a terminal, as shown in fig. 8, and the apparatus 600 includes:
the data transmission module 610 is used for acquiring log data of the service system and transmitting the log data to the real-time stream computing component through the message distribution middleware, wherein the log data comprises at least one service level;
the storage module 620 is configured to preprocess the log data through the real-time stream calculation component, generate a log stream corresponding to each service level, and store the log stream to the search engine;
an analysis module 630, configured to analyze a log stream on the search engine according to a preset alarm rule, and generate an alarm pushing result;
and the executing module 640 is configured to execute a corresponding processing operation based on the alarm pushing result.
Optionally, the storage module 620 is configured to:
analyzing and filtering the log data corresponding to each service level through a real-time stream computing component to obtain filtered log data;
desensitizing the filtered log data to generate log streams corresponding to each service level;
storing the log stream corresponding to each service level to the search according to a preset storage modeCable engine。
Optionally, as shown in fig. 9, the analyzing module 630 includes:
a transmitting unit 631 for transmitting a data query request to the search engine, the data query request including a log identifier;
a first receiving unit 632, configured to receive query result information corresponding to a log identifier sent by a search engine, where the query result information is used to indicate the number of log streams and the service level of the log streams corresponding to the log identifier received within a first preset time;
the generating unit 633 is configured to generate an alarm pushing result according to a preset alarm rule based on the number of log streams and the service level corresponding to the received log identifier within the first preset time.
Optionally, the generating unit 633 is specifically configured to:
determining a count value based on the number of log streams and the service level corresponding to the log identification received in a first preset time;
accumulating the count values within a second preset time to obtain the sum of the count values, wherein the second preset time is longer than the first preset time;
and when the sum of the count values is greater than a preset threshold value, generating an alarm pushing result.
Optionally, the generating unit 633 is further configured to:
acquiring a data dimension weight value corresponding to each service level;
and carrying out weighted summation processing on the number of the log streams corresponding to the service level and the corresponding data dimension weight value, and determining a count value.
Optionally, the executing module 640 includes:
the second receiving unit 641 is configured to receive a processing operation instruction, where the processing operation instruction carries a processing type determined based on the alarm push result;
and an execution unit 642, configured to execute, in response to the processing operation instruction, a processing operation corresponding to the processing type.
Optionally, the data transmission module 610 is specifically configured to:
collecting log data of a service system and unifying the data format of the log data;
determining the service level of the log data in the uniform data format;
according to the service level, sending the corresponding log data to the message distribution middleware;
the log data is transmitted to the real-time stream computation component by the message distribution middleware.
According to the processing device of the service log, the data transmission module collects log data of a service system and transmits the log data to the real-time stream computing assembly through the message distribution middleware, the storage module is used for preprocessing the log data through the real-time stream computing assembly, log streams corresponding to all service levels are generated and stored to the search engine, then the analysis module analyzes the log streams on the search engine according to preset alarm rules to generate alarm pushing results, and the execution module executes corresponding processing operations based on the alarm pushing results. According to the technical scheme, the service problems existing in the log stream can be automatically found through the preset alarm rule, so that the possibility of fault occurrence can be pre-judged in advance, the alarm push result can be accurately generated, the corresponding processing operation is executed based on the alarm push result, the fault problems can be timely processed, and the service processing efficiency is improved.
On the other hand, the device provided by the embodiment of the present application includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the method for processing the service log as described above is implemented.
Referring to fig. 10, fig. 10 is a schematic structural diagram of a computer system of a terminal device according to an embodiment of the present application.
As shown in fig. 10, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 703 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM702, and the RAM703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 703 and/or installed from the removable medium 711. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present application may be implemented by software or hardware. The described units or modules may also be provided in a processor, and may be described as: a processor, comprising: the device comprises a data transmission module, a storage module, an analysis module and an execution module. Where the names of such units or modules do not in some cases constitute a limitation of the units or modules themselves, for example, the data transmission module may also be described as "for collecting log data of the business system, the log data including at least one business level, and transmitting to the real-time flow computation component through the message distribution middleware".
As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may be separate and not incorporated into the electronic device. The computer-readable storage medium stores one or more programs, and when the programs are used by one or more processors to execute the method for processing the service log described in the present application:
collecting log data of a service system and transmitting the log data to a real-time stream computing component through message distribution middleware, wherein the log data comprises at least one service level;
preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level and storing the log streams to a search engine;
analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result;
and executing corresponding processing operation based on the alarm pushing result.
To sum up, the method, the apparatus, the device and the storage medium for processing the service log provided in the embodiments of the present application acquire log data of a service system and transmit the log data to the real-time stream computing component through the message distribution middleware, preprocess the log data through the real-time stream computing component, generate a log stream corresponding to each service level and store the log stream to the search engine, analyze the log stream on the search engine according to a preset alarm rule, generate an alarm push result, and execute a corresponding processing operation based on the alarm push result. According to the technical scheme, the service problems existing in the log stream can be automatically found through the preset alarm rule, so that the possibility of fault occurrence can be pre-judged in advance, the alarm push result can be accurately generated, the corresponding processing operation is executed based on the alarm push result, the fault problems can be timely processed, and the service processing efficiency is improved.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by a person skilled in the art that the scope of the invention as referred to in the present application is not limited to the embodiments with a specific combination of the above-mentioned features, but also covers other embodiments with any combination of the above-mentioned features or their equivalents without departing from the inventive concept. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (10)
1. A method for processing a service log, the method comprising:
collecting log data of a service system and transmitting the log data to a real-time stream computing component through message distribution middleware, wherein the log data comprises at least one service level;
preprocessing the log data through the real-time stream computing component, generating log streams corresponding to each service level and storing the log streams to a search engine;
analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result;
and executing corresponding processing operation based on the alarm pushing result.
2. The method of claim 1, wherein preprocessing the log data by the real-time stream computation component to generate a log stream corresponding to each of the service levels and store the log stream to a search engine comprises:
analyzing and filtering the log data corresponding to each service level through the real-time stream computing component to obtain filtered log data;
desensitizing the filtered log data to generate log streams corresponding to each service level;
and storing the log stream corresponding to each service level to the search engine according to a preset storage mode.
3. The method of claim 1, wherein analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result comprises:
sending a data query request to the search engine, wherein the data query request comprises a log identifier;
receiving query result information corresponding to the log identification sent by the search engine, wherein the query result information is used for indicating the number and the service level of the log streams corresponding to the log identification received in a first preset time;
and generating an alarm pushing result according to a preset alarm rule based on the number and the service level of the log streams corresponding to the log identification received in the first preset time.
4. The method of claim 3, wherein generating an alarm pushing result according to a preset alarm rule based on the number of log streams and the service level corresponding to the service level received within a first preset time comprises:
determining a count value based on the number and the service level of the log streams corresponding to the log identification received in a first preset time;
accumulating the count values within a second preset time to obtain the sum of the count values, wherein the second preset time is longer than the first preset time;
and when the sum of the count values is greater than a preset threshold value, generating an alarm pushing result.
5. The method of claim 4, wherein determining a count value based on the number of log streams and the service level corresponding to the log identifier received within a first preset time comprises:
acquiring a data dimension weight value corresponding to each service level;
and carrying out weighted summation processing on the number of the log streams corresponding to the service level and the corresponding data dimension weight value, and determining a count value.
6. The method of claim 1, wherein performing a corresponding processing operation based on the alert push result comprises:
receiving a processing operation instruction, wherein the processing operation instruction carries a processing type determined based on the alarm pushing result;
and responding to the processing operation instruction, and executing the processing operation corresponding to the processing type.
7. The method of claim 1, wherein collecting log data of a business system and transmitting the log data to a real-time stream computing component through message distribution middleware comprises:
collecting log data of a service system and unifying the data format of the log data;
determining the service level of the log data in the uniform data format;
according to the service level, sending the corresponding log data to a message distribution middleware;
transmitting, by the message distribution middleware, the log data to a real-time stream computation component.
8. An apparatus for processing a service log, the apparatus comprising:
the data transmission module is used for collecting log data of a service system and transmitting the log data to the real-time stream computing component through the message distribution middleware, wherein the log data comprises at least one service level;
the storage module is used for preprocessing the log data through the real-time stream computing component, generating log streams corresponding to all the service levels and storing the log streams to a search engine;
the analysis module is used for analyzing the log stream on the search engine according to a preset alarm rule to generate an alarm pushing result;
and the execution module is used for executing corresponding processing operation based on the alarm pushing result.
9. A computer arrangement, characterized in that the computer arrangement comprises a memory, a processor and a computer program stored in the memory and executable on the processor, the processor being adapted to implement the method of processing a traffic log according to any of claims 1-7 when executing the program.
10. A computer-readable storage medium, characterized in that a computer program is stored thereon for implementing the method of processing a traffic log according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110551959.4A CN113157545A (en) | 2021-05-20 | 2021-05-20 | Method, device and equipment for processing service log and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110551959.4A CN113157545A (en) | 2021-05-20 | 2021-05-20 | Method, device and equipment for processing service log and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113157545A true CN113157545A (en) | 2021-07-23 |
Family
ID=76876881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110551959.4A Pending CN113157545A (en) | 2021-05-20 | 2021-05-20 | Method, device and equipment for processing service log and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113157545A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113608969A (en) * | 2021-08-25 | 2021-11-05 | 中国银行股份有限公司 | Service system operation monitoring system and method |
| CN113687946A (en) * | 2021-08-19 | 2021-11-23 | 海尔数字科技(青岛)有限公司 | Task management method, device, server and storage medium |
| CN113824554A (en) * | 2021-08-30 | 2021-12-21 | 山东健康医疗大数据有限公司 | Dynamic authentication method and device for data transmission between middleware and computer medium |
| CN113904913A (en) * | 2021-08-19 | 2022-01-07 | 济南浪潮数据技术有限公司 | Alarm processing method, device, equipment and storage medium based on pipeline |
| CN113938429A (en) * | 2021-09-07 | 2022-01-14 | 南京星云数字技术有限公司 | Flow control method, flow control device and computer readable storage medium |
| CN114969083A (en) * | 2022-06-24 | 2022-08-30 | 在线途游（北京）科技有限公司 | Real-time data analysis method and system |
| CN115134260A (en) * | 2022-07-12 | 2022-09-30 | 北京东土拓明科技有限公司 | User perception improving method and device, computing equipment and storage medium |
| CN115361274A (en) * | 2022-08-30 | 2022-11-18 | 中国银行股份有限公司 | Alarm message processing method and device |
| CN115460214A (en) * | 2022-11-10 | 2022-12-09 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN115714718A (en) * | 2022-09-23 | 2023-02-24 | 上海芯赛云计算科技有限公司 | Log early warning method and system based on memory, computer equipment and storage medium |
| CN116991661A (en) * | 2023-07-20 | 2023-11-03 | 北京直客通科技有限公司 | Problem alarm system and method for software system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080168308A1 (en) * | 2007-01-06 | 2008-07-10 | International Business Machines | Adjusting Sliding Window Parameters in Intelligent Event Archiving and Failure Analysis |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据***有限公司 | A kind of big data log intelligent analysis system and method |
| CN110109841A (en) * | 2019-05-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | Localization method, device, equipment and the computer readable storage medium of abnormal problem |
| CN111581054A (en) * | 2020-04-30 | 2020-08-25 | 重庆富民银行股份有限公司 | ELK-based log point-burying service analysis and alarm system and method |
| CN112068979A (en) * | 2020-09-11 | 2020-12-11 | 重庆紫光华山智安科技有限公司 | Service fault determination method and device |
| CN112115114A (en) * | 2020-09-25 | 2020-12-22 | 北京百度网讯科技有限公司 | Log processing method, device, equipment and storage medium |
| CN112286757A (en) * | 2020-10-12 | 2021-01-29 | 浙江深大智能科技有限公司 | Data synchronization monitoring method and device, electronic equipment and storage medium |
-
2021
- 2021-05-20 CN CN202110551959.4A patent/CN113157545A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080168308A1 (en) * | 2007-01-06 | 2008-07-10 | International Business Machines | Adjusting Sliding Window Parameters in Intelligent Event Archiving and Failure Analysis |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据***有限公司 | A kind of big data log intelligent analysis system and method |
| CN110109841A (en) * | 2019-05-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | Localization method, device, equipment and the computer readable storage medium of abnormal problem |
| CN111581054A (en) * | 2020-04-30 | 2020-08-25 | 重庆富民银行股份有限公司 | ELK-based log point-burying service analysis and alarm system and method |
| CN112068979A (en) * | 2020-09-11 | 2020-12-11 | 重庆紫光华山智安科技有限公司 | Service fault determination method and device |
| CN112115114A (en) * | 2020-09-25 | 2020-12-22 | 北京百度网讯科技有限公司 | Log processing method, device, equipment and storage medium |
| CN112286757A (en) * | 2020-10-12 | 2021-01-29 | 浙江深大智能科技有限公司 | Data synchronization monitoring method and device, electronic equipment and storage medium |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687946A (en) * | 2021-08-19 | 2021-11-23 | 海尔数字科技(青岛)有限公司 | Task management method, device, server and storage medium |
| CN113904913A (en) * | 2021-08-19 | 2022-01-07 | 济南浪潮数据技术有限公司 | Alarm processing method, device, equipment and storage medium based on pipeline |
| CN113608969A (en) * | 2021-08-25 | 2021-11-05 | 中国银行股份有限公司 | Service system operation monitoring system and method |
| CN113824554B (en) * | 2021-08-30 | 2024-02-13 | 山东浪潮智慧医疗科技有限公司 | Dynamic authentication method, device and computer medium for data transmission between middleware |
| CN113824554A (en) * | 2021-08-30 | 2021-12-21 | 山东健康医疗大数据有限公司 | Dynamic authentication method and device for data transmission between middleware and computer medium |
| CN113938429A (en) * | 2021-09-07 | 2022-01-14 | 南京星云数字技术有限公司 | Flow control method, flow control device and computer readable storage medium |
| CN114969083A (en) * | 2022-06-24 | 2022-08-30 | 在线途游（北京）科技有限公司 | Real-time data analysis method and system |
| CN115134260A (en) * | 2022-07-12 | 2022-09-30 | 北京东土拓明科技有限公司 | User perception improving method and device, computing equipment and storage medium |
| CN115361274A (en) * | 2022-08-30 | 2022-11-18 | 中国银行股份有限公司 | Alarm message processing method and device |
| CN115361274B (en) * | 2022-08-30 | 2024-05-24 | 中国银行股份有限公司 | Alarm message processing method and device |
| CN115714718A (en) * | 2022-09-23 | 2023-02-24 | 上海芯赛云计算科技有限公司 | Log early warning method and system based on memory, computer equipment and storage medium |
| CN115460214A (en) * | 2022-11-10 | 2022-12-09 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN115460214B (en) * | 2022-11-10 | 2023-02-07 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN116991661A (en) * | 2023-07-20 | 2023-11-03 | 北京直客通科技有限公司 | Problem alarm system and method for software system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113157545A (en) | Method, device and equipment for processing service log and storage medium | |
| US11586972B2 (en) | Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs | |
| EP3752921B1 (en) | Multi-variant anomaly detection from application telemetry | |
| US11983639B2 (en) | Systems and methods for identifying process flows from log files and visualizing the flow | |
| US11138168B2 (en) | Data analysis and support engine | |
| US20190278700A1 (en) | System and method for automated service layer testing and regression | |
| JP2021530798A (en) | Systems and methods for real-time processing of data streams | |
| US9355007B1 (en) | Identifying abnormal hosts using cluster processing | |
| US11847480B2 (en) | System for detecting impairment issues of distributed hosts | |
| US10353799B2 (en) | Testing and improving performance of mobile application portfolios | |
| JP7373611B2 (en) | Log auditing methods, equipment, electronic equipment, media and computer programs | |
| US20210092160A1 (en) | Data set creation with crowd-based reinforcement | |
| US11449798B2 (en) | Automated problem detection for machine learning models | |
| US11546380B2 (en) | System and method for creation and implementation of data processing workflows using a distributed computational graph | |
| US10484257B1 (en) | Network event automatic remediation service | |
| US11570214B2 (en) | Crowdsourced innovation laboratory and process implementation system | |
| US11074652B2 (en) | System and method for model-based prediction using a distributed computational graph workflow | |
| US10372572B1 (en) | Prediction model testing framework | |
| US20220058745A1 (en) | System and method for crowdsensing-based insurance premiums | |
| US11212162B2 (en) | Bayesian-based event grouping | |
| CN108768742B (en) | Network construction method and device, electronic equipment and storage medium | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN114064757A (en) | Application program optimization method, device, equipment and medium | |
| CN113052509A (en) | Model evaluation method, model evaluation apparatus, electronic device, and storage medium | |
| US11366660B1 (en) | Interface latency estimation based on platform subcomponent parameters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |