CN116185771A

CN116185771A - Data processing method, device, electronic equipment and storage medium

Info

Publication number: CN116185771A
Application number: CN202310102970.1A
Authority: CN
Inventors: 李亚楠
Original assignee: Beijing Dajia Internet Information Technology Co Ltd
Current assignee: Beijing Dajia Internet Information Technology Co Ltd
Priority date: 2023-01-29
Filing date: 2023-01-29
Publication date: 2023-05-30

Abstract

The disclosure relates to a data processing method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: acquiring log management configuration information; acquiring target log data from the log data set based on the attribute information; and displaying the log management information according to the target log data. According to the technical scheme provided by the embodiment of the disclosure, the target log data can be automatically acquired from the target log data set by acquiring the attribute information corresponding to the target log data configured in the log management configuration information, so that the labor cost is reduced, and the acquisition efficiency of the log data is effectively improved; according to the target log data obtained automatically, the log management information can be displayed automatically in a visualized mode, the display efficiency of the log data is effectively improved, and the management efficiency of the log data is improved as a whole.

Description

Data processing method, device, electronic equipment and storage medium

Technical Field

The disclosure relates to the field of computer technology, and in particular, to a data processing method, a data processing device, electronic equipment and a storage medium.

Background

With the development of computer technology and internet technology, various applications are widely used, and log management of applications is also increasingly important.

In the related art, a developer needs to write a large amount of codes to perform data embedding, so as to collect log data. The collected log data can be visually displayed. Each time a visual presentation is made, a developer is required to manually retrieve the log data required for the visual presentation.

In the related art, the labor cost for managing the log data is high, and the management efficiency of the log data is low.

Disclosure of Invention

The disclosure provides a data processing method, a device, an electronic device and a storage medium, so as to at least solve the problems of high labor cost and low log data management efficiency in the related technology. The technical scheme of the present disclosure is as follows:

according to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

acquiring log management configuration information, wherein the log management configuration information comprises attribute information corresponding to target log data;

acquiring the target log data from a log data set based on the attribute information;

and displaying log management information according to the target log data.

In some possible designs, the obtaining log management configuration information includes:

displaying a log management configuration page;

Responding to configuration operation triggered by the log management configuration page, and acquiring the attribute information;

and generating the log management configuration information based on the attribute information.

In some possible designs, the obtaining the target log data from a log data set based on the attribute information includes:

generating a data query instruction corresponding to the attribute information;

and executing the data query instruction on the log data set to obtain the target log data.

In some possible designs, the method further comprises:

acquiring log data corresponding to the data log record;

and carrying out data processing on the log data according to a preset data processing rule to obtain the log data set.

In some possible designs, the data log record includes a service log record, and the acquiring log data corresponding to the data log record includes:

responding to an uploading instruction corresponding to the business log record triggered by the main thread, and creating a sub thread;

determining a judging result between the service log record and a space occupation threshold value based on the sub-thread, and sending the judging result to a main thread;

cutting and uploading business log data corresponding to the business log records under the condition that the judgment result indicates that the space occupation amount corresponding to the business log records exceeds the space occupation threshold value, wherein the log data comprises the business log data;

And uploading the service log data under the condition that the judging result indicates that the space occupation amount does not exceed the first space occupation threshold value.

In some possible designs, the data log records include at least two types of service log records, and the acquiring log data corresponding to the data log records includes:

collecting the at least two types of service log records;

and formatting the data in the at least two types of service log records to obtain service log data in a target format, wherein the service log data comprises the service log data.

In some possible designs, the data processing rule includes a data length threshold and a preset character type, and the data processing is performed on the log data according to the preset data processing rule to obtain the log data set, including:

inquiring a target data item with the data length exceeding the data length threshold value in the log data, and a target character string corresponding to the preset character type;

under the condition that the target data item is inquired, carrying out segmentation cutting processing on the target data item to obtain segmented data;

Under the condition that the target character string is inquired, cleaning the target character string to obtain a cleaned character string;

and storing the segmented data and the cleaned character string in the log data set.

In some possible designs, the presenting log management information according to the target log data includes:

analyzing and processing the target log data to obtain at least one index data, wherein the at least one index data represents the running condition of the target application service;

the at least one index data is displayed, and the log management information includes the at least one index data.

According to a second aspect of embodiments of the present disclosure, there is provided a data processing apparatus comprising:

the configuration information acquisition module is configured to acquire log management configuration information, wherein the log management configuration information comprises attribute information corresponding to target log data;

a log data acquisition module configured to perform acquisition of the target log data from a log data set based on the attribute information;

and the management information display module is configured to display log management information according to the target log data.

In some possible designs, the configuration information acquisition module includes:

a page display unit configured to execute a display log management configuration page;

an attribute information acquisition unit configured to perform a configuration operation triggered in response to the log management configuration page, to acquire the attribute information;

and a configuration information generation unit configured to perform generation of the log management configuration information based on the attribute information.

In some possible designs, the log data acquisition module includes:

a query instruction generation unit configured to execute a data query instruction corresponding to the attribute information;

and the log data query unit is configured to execute the data query instruction on the log data set to obtain the target log data.

In some possible designs, the apparatus further comprises:

the log data acquisition module is configured to execute the log data corresponding to the acquired data log record;

and the log data sorting module is configured to execute data processing on the log data according to a preset data processing rule to obtain the log data set.

In some possible designs, the data log record includes a traffic log record, and the log data collection module includes:

A sub-thread creation unit configured to execute an upload instruction corresponding to the service log record triggered by the main thread, and create a sub-thread;

a judgment result determining unit configured to perform a judgment result between the service log record and a space occupation threshold value based on the sub-thread determination, and send the judgment result to a main thread;

a log data uploading unit configured to execute cutting and uploading of service log data corresponding to the service log record, where the judging result indicates that the space occupation amount corresponding to the service log record exceeds the space occupation threshold, and the log data includes the service log data;

the log data uploading unit is further configured to perform uploading of the service log data if the judgment result indicates that the space occupation amount does not exceed the first space occupation threshold value.

In some possible designs, the data log records include at least two types of service log records, and the log data collection module further includes:

a service log acquisition unit configured to perform acquisition of the at least two types of service log records;

And the service log formatting unit is configured to perform formatting processing on the data in the at least two types of service log records to obtain service log data in a target format, wherein the service log data comprises the service log data.

In some possible designs, the data processing rule includes a data length threshold and a preset character type, and the log data sorting module includes:

the data query unit is configured to perform query on target data items with data length exceeding the data length threshold and target character strings corresponding to the preset character types in the log data;

the data segmentation unit is configured to execute segmentation cutting processing on the target data item under the condition that the target data item is queried, so as to obtain segmented data;

the data cleaning unit is configured to perform cleaning processing on the target character string under the condition that the target character string is inquired, so as to obtain a cleaned character string;

and a data storage unit configured to perform storing of the segmented data and the cleaned character string in the log data set.

In some possible designs, the management information presentation module includes:

The index data determining unit is configured to perform analysis processing on the target log data to obtain at least one index data, and the at least one index data represents the running condition of the target application service;

an index data presentation unit configured to perform presentation of the at least one index data, the log management information including the at least one index data.

According to a third aspect of embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data processing method according to any of the first aspects above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the data processing method of any one of the first aspects of embodiments of the present disclosure.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method of any one of the first aspects of embodiments of the present disclosure.

The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:

the target log data can be automatically acquired from the target log data set by acquiring the attribute information corresponding to the target log data configured in the log management configuration information, so that the labor cost is reduced, and the acquisition efficiency of the log data is effectively improved; according to the target log data obtained automatically, the log management information can be displayed automatically in a visualized mode, the display efficiency of the log data is effectively improved, and the management efficiency of the log data is improved as a whole.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.

FIG. 1 is a schematic diagram of an application environment shown in accordance with an exemplary embodiment;

FIG. 2 is a flowchart illustrating a method of data processing according to an exemplary embodiment;

FIG. 3 is a flow chart II illustrating a method of data processing according to an exemplary embodiment;

FIG. 4 illustrates a schematic diagram of a log management configuration page;

FIG. 5 illustrates a second schematic diagram of a log management configuration page;

FIG. 6 is a block diagram of a data processing apparatus according to an exemplary embodiment;

FIG. 7 is a block diagram of an electronic device for data processing, according to an example embodiment.

Detailed Description

In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by the user or sufficiently authorized by each party.

Referring to fig. 1, fig. 1 is a schematic diagram illustrating an application environment, which may include a terminal 100 and a server 200, as shown in fig. 1, according to an exemplary embodiment.

The terminal 100 may be used to provide log data management services to any user. Specifically, the terminal 100 may include, but is not limited to, a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an augmented reality (augmented reality, AR)/Virtual Reality (VR) device, a smart wearable device, or other type of electronic device, or may be software running on the electronic device, such as an application program, etc. Alternatively, the operating system running on the electronic device may include, but is not limited to, an android system, an IOS system, linux, windows, and the like.

In an alternative embodiment, the server 200 may provide background services for the terminal 100. Specifically, the server 200 may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Network, content delivery networks), and basic cloud computing services such as big data and artificial intelligence platforms.

In addition, it should be noted that, fig. 1 is only an application environment provided by the present disclosure, and in practical application, other application environments may also be included, for example, may include more terminals.

In the embodiment of the present disclosure, the terminal 100 and the server 200 may be directly or indirectly connected through a wired or wireless communication manner, which is not limited herein.

FIG. 2 is a flowchart illustrating a method of data processing according to an exemplary embodiment. Optionally, the data processing method is used in an electronic device. As shown in fig. 2, the method may include the following steps (210-230).

Step 210, obtaining log management configuration information.

Optionally, the log management configuration information includes attribute information corresponding to the target log data. Specifically, the log management configuration information is data configured by the target user account in the log management configuration page. The method comprises attribute information corresponding to target log data to be acquired.

The attribute information includes, but is not limited to, a service identifier corresponding to a service to which the target log data belongs, domain name information corresponding to the service, an item identifier corresponding to the target log data in the service, service environment information, and the like.

Optionally, the log management configuration information further includes a data query instruction corresponding to the target log data.

In an exemplary embodiment, as shown in fig. 3, the step 210 may include the following steps (211 to 213), and fig. 3 is a second flowchart of a data processing method according to an exemplary embodiment.

Step 211, displaying a log management configuration page.

Alternatively, the log management configuration page may be a configuration page provided by a log management application. Specifically, the above-described log management application is a visual log management application.

And step 212, responding to the configuration operation triggered by the log management configuration page to acquire attribute information.

The log management configuration page is used for receiving configuration operation of attribute information, and a user can select preset attribute information in the log management configuration page or input custom attribute information; the log management configuration page can detect the set attribute information and acquire the configured attribute information after detecting the confirmation operation.

In one example, as shown in FIG. 4, a schematic diagram of a log management configuration page is shown. In the log management configuration page 40, a variable name 41 of a variable to be acquired and definition information corresponding to the variable name 41 may be configured. For definition information, a domain name or a data query instruction corresponding to the corresponding variable may be defined.

In one example, as shown in FIG. 5, a schematic diagram II of a log management configuration page is illustratively shown. A setting box of variable contents is set in the log management configuration page 50. Such as a general settings box 51 and a text options box 52. The user can set name information, type information, tag information, hidden information, and the like corresponding to the variables in the general setting frame 51. For domain names to which variables correspond may also be set in textbox 52.

Thus, when the contents of the variables are queried, corresponding table data can be searched out from the target data set according to the set fixed characters or the data query instruction.

According to the technical scheme provided by the embodiment of the disclosure, the attribute information corresponding to the target log data to be acquired can be configured through the configuration page, so that subsequent inquiry is facilitated, the data embedding is not needed manually, the log data required by visual display is not needed to be manually called each time of visual display, and the configuration efficiency of the visual display of the data is effectively improved.

In step 213, log management configuration information is generated based on the attribute information.

Step 220, acquiring target log data from the log data set based on the attribute information.

In one possible implementation manner, for unified log data management, the cluster- > database- > table- > fields of unified reporting of the data are specified, reporting of the data source is closed, and when the data is consumed, the data is fetched from the unified data source, namely the target data set, so that the cost of the server can be saved, and the maintainability is enhanced.

Meanwhile, as the data sources are uniformly closed, the format of a data query instruction for querying the data is basically fixed, and the target log data can be queried at minimum cost only by modifying a few parameters (such as a unique service identifier, a corresponding service domain name and the like). Specifically, the capability of configuring custom variables (variables) in the log management application can store mapping relations of attribute information such as domain names, item identifications, environment information and the like of different service directions into a data table through the variables of custom (domain names, item ids and the like) dimensions, and inquire data to be selected from the table through a data inquiry instruction in the variables. The variable is used in the data query instruction, so that a new data query instruction can be automatically generated, and the labor cost is saved.

In an exemplary embodiment, the above step 220 may include the following steps (221 to 222).

Step 221, generating a data query instruction corresponding to the attribute information.

Optionally, the data query instruction may be a data query instruction automatically generated according to the attribute information, and is used for querying target log data corresponding to the attribute information. Of course, the data query instruction may be configured on the log management configuration page. Alternatively, the above

Optionally, the data query instruction is an SQL (Structured Query Language ) instruction. The structured query language is a database query and programming language for accessing data and querying, updating and managing relational database systems.

Step 222, executing a data query instruction on the log data set to obtain target log data.

According to the technical scheme provided by the embodiment of the disclosure, the data query instruction can be automatically generated according to the attribute information to query and acquire the log data, so that the automatic acquisition of the log data is realized, and the acquisition efficiency of the log data is effectively improved.

In an exemplary embodiment, as shown in FIG. 3, the above method further includes the following steps (240-250).

Step 240, obtaining log data corresponding to the data log record.

In one possible embodiment, the data log records include service log records. The traffic log records may be collected multi-threaded asynchronously, non-blocking, high-speed, and fully using WebWorker threads.

Specifically, the process of acquiring the service log data in the service log record is as follows:

and responding to an uploading instruction corresponding to the business log record triggered by the main thread, and creating a sub thread. The traffic log record may be a record in the target application. The main thread is the main thread corresponding to the target application.

The programming language to which the target application corresponds may be JavaScript language. The JavaScript language employs a single-threaded model, that is, all tasks can only be completed on one thread (e.g., the main thread described above) and can only do one thing at a time. The former task is not done and the latter task can only be done. With the enhancement of the computing power of a computer, particularly the appearance of a multi-core CPU, a single thread brings great inconvenience, and the computing power of the computer cannot be fully exerted.

The WebWorker role may be to create a multi-threaded environment for JavaScript, allowing the main thread Cheng Chuang to build the above-mentioned sub-threads (Worker threads), with some tasks assigned to the latter run. While the main thread runs, the workbench thread runs in the background, and the workbench thread are not interfered with each other. Waiting until the Worker thread completes the calculation task, and returning the result to the main thread. This has the advantage that some computationally intensive or highly delayed tasks, which are burdened by the Worker thread, are fluent and not blocked or slowed down by the main thread (which is typically responsible for user interface UI interactions).

Once the new creation is successful, the child thread will run without interruption by activity on the main thread (e.g., user clicking a button, submitting a form). This facilitates communication in response to the main thread at any time. If the sub-thread is running all the time, the comparison consumes resources, so that once the sub-thread is used up, it is turned off.

After the sub-thread is created, a judging result between the business log record and the space occupation threshold value can be determined based on the sub-thread, and the judging result is sent to the main thread.

Optionally, the main thread adopts a new command, invokes an objective function corresponding to the sub-thread, such as a workbench () construction function, and drives the sub-thread to calculate the size relationship between the space occupation amount corresponding to the service log record and the space occupation threshold, thereby generating the judgment result. After the sub thread calculates the judging result, the judging result can be sent to the main thread, so that whether the segmentation uploading is needed or not is determined according to the judging result.

Specifically, the sub-thread determines whether the byte count occupied by the service log record, i.e., the space occupation amount, exceeds the limit of the space occupation threshold. And calculating according to the maximum 4 bytes occupied by one character in the service log record, and determining the judgment result so as to return to the main thread.

And cutting the service log data corresponding to the uploaded service log record when the judging result indicates that the space occupation amount corresponding to the service log record exceeds the space occupation threshold value, wherein the log data comprises the service log data. At this time, the main thread may segment the service log data and upload each segment of the segmented service log data to the target data set. The space occupation amount corresponding to each service log data after segmentation is smaller than or equal to the upper limit value of single data transmission amount, so that data loss during data transmission is avoided.

And uploading the service log data under the condition that the judging result indicates that the space occupation amount does not exceed the first space occupation threshold value. The main thread can directly upload the business data log with small occupied space to the target data set, so that the data transmission efficiency is improved.

In another possible embodiment, the data log records further comprise at least two types of service log records. Optionally, the at least two types of service log records include a Node service log record and an nminux service log record. And aiming at the Node service log record and the Nginx service log record, the cross-service type log can be collected uniformly by using a preset script.

Nginx service: nginx is a lightweight Web server/reverse proxy server and email (IMAP/POP 3) proxy server that issues under the BSD-like protocol. The method is characterized by small occupied memory and strong concurrency capability, and in fact, the concurrency capability of Nginx is better represented in the same type of web servers.

Node service: js is JavaScript running on the server. Js is a platform built based on browser JavaScript runtime. Js is an event-driven I/O server JavaScript environment, and the speed of executing JavaScript based on a browser engine is high, and the performance is good.

The process of acquiring service log data in the service log record is specifically as follows:

at least two types of service log records are collected. Specifically, the Rollup is used as a module packer to write a log unified reporting tool package. And issuing the written unified log reporting toolkit to the NPM, introducing the toolkit by a developer, deploying the toolkit in equipment, acquiring and running the unified log reporting toolkit by the equipment, and acquiring the at least two types of service log records and carrying out subsequent formatting processing.

The Rollup is a JavaScript module packager, which can compile small blocks of codes into large blocks of miscellaneous codes, such as library or application programs. NPM is a software registry that can share and borrow code.

Optionally, the log unified reporting tool package includes a log formatting method library corresponding to the Node service, where the log formatting method library includes a preset method for obtaining a log record of the Node service and a method for formatting the log record of the Node service.

Optionally, the log unified reporting kit further includes a log collection module corresponding to the nmginx service. Alternatively, the log collection module may be injected into the nmginx service profile through a provided command number tool. The log acquisition module comprises a preset method for acquiring the log record of the Nginx service and a method for formatting the Nginx service.

After the at least two types of service log records are collected, formatting processing can be carried out on data in the at least two types of service log records to obtain service log data in a target format, wherein the service log data comprises the service log data. The data format employed will also vary for different types of service log records. Therefore, the service log data recorded by each of the at least two types of service log records can be converted into the service log data in a unified format, namely, a target format according to a preset format conversion rule. The embodiment of the disclosure does not limit the types corresponding to the target format and the service log record.

According to the technical scheme provided by the embodiment of the disclosure, the service logs of the cross-service type can be automatically collected, the service log data of the uniform format are stored through formatting, the acquisition efficiency of the service logs of the cross-service type is effectively improved, the data normalization of the log data set is improved, configuration and query are facilitated, and automatic generation of data query instructions is facilitated.

And 250, carrying out data processing on the log data according to a preset data processing rule to obtain a log data set.

For the collected log data, unified data arrangement processing is needed, so that the collected log data is saved to a log data set, and the normalization of the data is improved. For example, the ultralong data in the log data is identified, and the ultralong data is subjected to the segmentation cutting process. For example, the cleaning process is performed on special characters and multi-bit characters (characters such as emoji characters that do not affect the validity of the log) in the log data.

In one possible implementation, the log data is serialized to obtain serialized log data in a target format, such as JSON format, and the serialized log data is stored to a target dataset.

Optionally, the data processing rule includes a data length threshold and a preset character type.

Inquiring target data items with data length exceeding a data length threshold value in log data and target character strings corresponding to preset character types. The target data item may be a data item having a data length greater than a data length threshold in the log data. The target character string may be a character string corresponding to the above-described preset character type.

Under the condition that the target data item is inquired, carrying out segmentation cutting processing on the target data item to obtain segmented data. For data in the object (object) format in log data, it is necessary to convert to JSON format at the time of data sorting, thereby storing the above-described target data set. When the data length exceeds the data length threshold, the JSON serialization is likely to fail, so that the target data item is segmented and cut to obtain segmented data.

And under the condition that the target character string is inquired, cleaning the target character string to obtain the cleaned character string. Also, when the target character string appears in the data in the object format, JSON serialization may fail, so that special characters or multi-bit characters in the target character string are subjected to cleaning processing, such as deletion, to obtain cleaned character strings.

The segmented data and the cleaned character string are stored in a log dataset.

According to the technical scheme provided by the embodiment of the disclosure, the ultra-long data are identified, the special character strings are cut in a segmented mode and cleaned, so that the integrity and the effectiveness of the data are guaranteed, and the interference data are removed.

Step 230, according to the target log data, the log management information is displayed.

Specifically, the log management information may be presented on a log management interface. The target management interface is an information display interface corresponding to the data visualization application program. Alternatively, the log management information may be a data curve or other statistical chart, which is not limited in the embodiments of the present application.

In one possible implementation, the log management information described above may be presented based on grafana. grafana is a program for visualizing large-scale measurement data, providing a powerful and elegant way to create, share, browse data, is a visualization panel (Dashboard) capable of chart and layout presentation, has a fully functional metric Dashboard and graphic editor, and can be used as a visualization of time-series data and application analysis of an infrastructure.

In an exemplary embodiment, the above step 230 may include the following steps (231 to 232).

And 231, analyzing and processing the target log data to obtain at least one index data.

The at least one index data characterizes the behavior of the target application service.

Specifically, the source data corresponding to the at least one index data may be determined from the target log data, so that according to an index data calculation formula corresponding to each of the at least one index data, calculation processing is performed on the source data corresponding to each of the at least one index data, so as to obtain the at least one index data, that is, a calculation result, thereby scientifically measuring the effect of the target application running service according to the calculation result and providing a data basis for continuous integration and optimization of the target application.

According to the technical scheme provided by the embodiment of the disclosure, through determining the at least one index data, the running condition of the target application service can be evaluated scientifically, reasonably and quantitatively, and a data basis is provided for continuous integration and optimization of the target application.

At step 232, at least one index data is displayed.

The log management information includes at least one index data.

In an exemplary embodiment, the at least one index data includes, but is not limited to, at least one of the following index data.

(1) Alarm delay (a): the difference between the actual time of actually generating the abnormality and the actual time of sending the alarm is used for measuring the index of the alarm efficiency, if the expected value approaches to 0, the fact that the actually generating abnormality is consistent with the time of sending the alarm is indicated, and the alarm is sent in real time. The target log data may include the actual time of the actual occurrence of the abnormality and the actual time of the alarm.

(2) Alarm viewing rate (B): in the period (generally, single or double weeks), the developer actually looks at the ratio of the total number of the alarm messages concerned to the total number of the alarm emissions, and the index is used for measuring the importance of the developer on the alarm, and is expected to approach 100%. The target log data can include the number of the actually checked alarm messages and the number of the alarm sent out, and the alarm check rate can be calculated based on the two data.

(3) Alarm viewing delay (C): the time counting starts when the alarm message is sent out in a period (generally single or double weeks), and the time interval from when the developer views the message is used for measuring the retry degree of the alarm and another dimension index of the developer, and the index is expected to be <5 minutes.

(4) Alarm index: and the alarm number generated by a single page in a period is not larger or smaller, namely the alarm number is better, the value is a reasonable interval obtained by comprehensively evaluating the coverage area, the traffic volume, the alarm condition and the like of the service, the index is different in different services and the same direction, and the local condition is required.

(5) On-line abnormal fishing rate (E): the service code is abnormal, so that the probability of active repair is also an important index for measuring the effective degree of the alarm, and the more reasonable the alarm is, the more the recovery rate can be improved.

(6) Alarm effective rate (F): the method is characterized in that the ratio of the total number of alarms and the total number of alarms, which are abnormal caused by a program or other subjective factors, to the total number of the alarms is determined from the alarm sending to the problem investigation, the index is used for measuring the effective degree of the alarms, the higher the alarm effective rate is, the better the alarm accuracy condition is, and the index is expected to be close to 100%.

(7) Alarm coverage (G): whether the alarm range covers all core business processes is used for measuring the coverage condition of the alarm, the index is expected to approach 100%, and if the alarm coverage is at a reduced level, the alarm coverage indicates that a part of core business processes lack an active discovery mechanism and means when abnormal.

(8) Alarm comprehensive index (Q): the index is approximately 100% which indicates that the service is better in alarm setting and alarm processing aspects, and conversely, worse; the specific calculation formula is as follows:

Q1＝(A-100)*(1/6)；

Q2＝B*(1/6)；

Q3＝(C-100)*(1/6)；

Q4＝E*(1/6)；

Q5＝F*(1/6)；

Q6＝G*(1/6)；

Q＝Q1+Q2+Q3+Q4+Q5+Q6。

Wherein A is alarm time delay time length (unit: minutes), and the time delay time length exceeds 100 minutes and is calculated as 100 minutes; b is the alarm checking rate; c is alarm viewing delay (unit: minutes), more than 100 minutes in 00 minutes; e is the abnormal on-line fishing rate; f is the effective alarm rate; g is the alarm coverage rate. Q1 is an alarm time delay mass fraction, Q2 is an alarm checking rate mass fraction, Q3 is an alarm checking time delay mass fraction, Q4 is an on-line abnormal fishing rate mass fraction, Q5 is an alarm effective rate mass fraction, Q6 is an alarm coverage rate mass fraction, and Q is an alarm comprehensive fraction.

(9) Alarm rating:

a:80< q < = 100; if 80< q < = 100, the alarm rating is class a.

B:60< q < = 80; if 60< q < = 80, the alarm rating is class B.

C:40< q < = 60; if 40< q < = 60, the alarm rating is class C.

D:20< q < = 40; if 20< q < = 40, the alarm rating is class D.

E:0< = Q <20; if 0< = Q <20, the alarm rating is class E.

Wherein A, B, C, D, E scale is sequentially reduced.

In summary, according to the technical scheme provided by the embodiment of the disclosure, by acquiring the attribute information corresponding to the target log data configured in the log management configuration information, the target log data can be automatically acquired from the target log data set, so that the labor cost is reduced, and the log data acquisition efficiency is effectively improved; according to the target log data obtained automatically, the log management information can be displayed automatically in a visualized mode, the display efficiency of the log data is effectively improved, and the management efficiency of the log data is improved as a whole.

In addition, in the technical scheme provided by the embodiment of the disclosure, on the log collection side, the service log uses the sub-threads to carry out multi-thread asynchronous, non-blocking, high-speed and complete collection. And performing cross-service type unified collection on Node and Nginx service logs by using a preset tool kit, so that automatic and efficient collection of the logs is completed, and the collection efficiency of log data is improved.

On the log data sorting side, segmentation cutting processing is carried out by identifying the ultra-long data, cleaning processing is carried out on special characters and multi-code characters in the log, the integrity of the data is ensured, and interference is eliminated.

On the log data acquisition side, related target log data can be queried through configuring each item of attribute information configured in a page, and data visualization processing is presented to complete automatic configuration of log data visualization.

And each evaluation index data can be calculated according to the target log data, an omnibearing and multidimensional alarm measurement index is specified, the health condition of each measurement system is subjected to weighted operation, an alarm comprehensive index is obtained, an alarm comprehensive rating is defined according to the alarm comprehensive index score, and the running condition of the target application is comprehensively reflected.

FIG. 6 is a block diagram of a data processing apparatus according to an example embodiment. Referring to fig. 6, the apparatus 600 includes:

a configuration information obtaining module 610 configured to perform obtaining log management configuration information, where the log management configuration information includes attribute information corresponding to target log data;

a log data acquisition module 620 configured to perform acquisition of the target log data from a log data set based on the attribute information;

and a management information presentation module 630 configured to perform presentation of log management information according to the target log data.

In some possible designs, the log data acquisition module includes:

In some possible designs, the apparatus further comprises:

It should be noted that, in the apparatus provided in the foregoing embodiment, when implementing the functions thereof, only the division of the foregoing functional modules is used as an example, in practical application, the foregoing functional allocation may be implemented by different functional modules, that is, the internal structure of the device is divided into different functional modules, so as to implement all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein.

Fig. 7 is a block diagram illustrating an electronic device for data processing, which may be a terminal, according to an exemplary embodiment, and an internal structure diagram thereof may be as shown in fig. 7. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the electronic device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method. The display screen of the electronic equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the electronic equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of a portion of the structure associated with the disclosed aspects and is not limiting of the electronic device to which the disclosed aspects apply, and that a particular electronic device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In an exemplary embodiment, there is also provided an electronic device including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement a data processing method as in the embodiments of the present disclosure.

In an exemplary embodiment, a computer readable storage medium is also provided, which when executed by a processor of an electronic device, enables the electronic device to perform the data processing method in the embodiments of the present disclosure.

In an exemplary embodiment, a computer program product containing instructions is also provided, which when run on a computer, cause the computer to perform the data processing method in the embodiments of the present disclosure.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of data processing, comprising:

and displaying log management information according to the target log data.

2. The method of claim 1, wherein the obtaining log management configuration information comprises:

Displaying a log management configuration page;

3. The method of claim 1, wherein the obtaining the target log data from a log data set based on the attribute information comprises:

generating a data query instruction corresponding to the attribute information;

4. The method according to claim 1, wherein the method further comprises:

acquiring log data corresponding to the data log record;

5. The method of claim 4, wherein the data log record comprises a traffic log record, and the obtaining log data corresponding to the data log record comprises:

6. The method of claim 4, wherein the data log records comprise at least two types of service log records, and the obtaining log data corresponding to the data log records comprises:

collecting the at least two types of service log records;

7. The method according to claim 4, wherein the data processing rule includes a data length threshold and a preset character type, and the performing data processing on the log data according to the preset data processing rule to obtain the log data set includes:

8. The method according to any one of claims 1 to 7, wherein said presenting log management information according to said target log data comprises:

9. A data processing apparatus, comprising:

10. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of any one of claims 1 to 8.