CN115391292A

CN115391292A - Log data processing method, device, equipment and storage medium

Info

Publication number: CN115391292A
Application number: CN202211113966.7A
Authority: CN
Inventors: 毋与伦; 谢俊; 宋绍磊; 张子奇
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2022-09-14
Filing date: 2022-09-14
Publication date: 2022-11-25

Abstract

The invention discloses a log data processing method, a log data processing device, log data processing equipment and a log data processing storage medium. The method comprises the following steps: acquiring log data generated by the micro server in the process of processing at least one request task; wherein, the log data has an execution state identifier; storing and executing log data under different behavior categories through a preset message queue, and acquiring the log data from the message queue as data to be processed; processing the data to be processed by adopting a scheduling thread corresponding to the behavior category of the data to be processed to obtain log data to be stored; and storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored. The embodiment of the invention realizes the real-time processing of the log data and improves the processing efficiency of the log data.

Description

Log data processing method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a log data processing method, apparatus, device, and storage medium.

Background

The system of the middle platform system is complex, the service quantity is large, the system for recording and analyzing fine-grained log data is not established and is not perfect, and the analysis speed of the log data is slow, if the data can not be operated quickly, the real-time calculation of the request and the response of each service can not be realized.

Disclosure of Invention

The invention provides a log data processing method, a device, equipment and a storage medium, which are used for realizing real-time processing of log data and improving the log data processing efficiency.

According to an aspect of the present invention, there is provided a log data processing method, the method including:

acquiring log data generated by the micro server in the process of processing at least one request task; the log data is provided with an execution state identifier;

storing and executing log data under different behavior categories through a preset message queue, and acquiring the log data from the message queue as data to be processed;

processing the data to be processed by adopting a scheduling thread corresponding to the behavior category of the data to be processed to obtain log data to be stored;

and storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored.

According to another aspect of the present invention, there is provided a log data processing apparatus including:

the log data acquisition module is used for acquiring log data generated by the micro server in the process of processing at least one request task; wherein, the log data is provided with an execution state identifier;

the device comprises a to-be-processed data acquisition module, a to-be-processed data acquisition module and a processing module, wherein the to-be-processed data acquisition module is used for storing and executing log data under different behavior types through a preset message queue and acquiring the log data from the message queue as to-be-processed data;

the data processing module to be processed is used for processing the data to be processed by adopting a scheduling thread corresponding to the behavior category of the data to be processed to obtain log data to be stored;

and the storage data processing module is used for storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log data processing method according to any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the log data processing method according to any one of the embodiments of the present invention when the computer instructions are executed.

The embodiment of the invention processes the data to be processed by adopting the scheduling thread corresponding to the behavior category of the data to be processed to obtain the log data to be stored; and storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored, so that the real-time processing of the log data is realized, and the processing efficiency of the log data is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1A is a flowchart of a log data processing method according to an embodiment of the present invention;

FIG. 1B is a block diagram of a log data processing system according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a log data processing apparatus according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device implementing the log data processing method according to the embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1A is a flowchart of a log data processing method according to an embodiment of the present invention, where this embodiment is applicable to a case of processing real-time log data under a large number of microservice architectures, and the method may be executed by a log data processing apparatus, where the log data processing apparatus may be implemented in a form of hardware and/or software, and the log data processing apparatus may be configured in an electronic device. As shown in fig. 1A, the method includes:

s110, acquiring log data generated in the process of processing at least one request task by the micro server; the log data is provided with an execution state identifier.

The micro server can be a novel server system for processing service requests and responses. In general, during the process of deploying micro servers, different micro servers may execute different services, and therefore, a micro service cluster composed of at least one micro server may be used to process a large number of service requests.

Wherein a large amount of request data and response data may be generated during the execution of the requested task. For example, the requesting task may be an order task, or the like. The log data may include request data and response data.

The log data is provided with an execution state identifier, and the execution state identifier may be a state of a complete illness cycle of log data execution, for example, the execution state identifier includes an acquired identifier, a consumed identifier, and an unconsumed identifier. In different execution processes, the execution state identifier of the log data may be changed. For example, when the log data completes execution in the subsequent step, its corresponding execution status identifier may be updated to the consumed identifier.

Illustratively, log data generated in the execution process of each micro server to the request task in the micro server cluster is obtained. Different micro servers can execute different request tasks and generate different log data.

It should be noted that in some individual scenarios, some sensitive data related to personal information may exist in the log data center acquired by the micro server. To ensure the security of such log data, such log data may also be subjected to encryption processing.

In an optional embodiment, after obtaining the log data generated by the micro server during processing of the at least one requested task, the method further includes: identifying sensitive information in log data; and encrypting the sensitive information to obtain encrypted log data.

And encrypting the sensitive information related in the log data to obtain the encrypted log data. Optionally, the log data may be formatted and desensitized to perform a filtering process on the log data.

And S120, storing and executing the log data under different behavior types through a preset message queue, and acquiring the log data from the message queue as to-be-processed data.

The message queue can be preset by the related technical personnel according to the actual requirement. For example, the Message Queue may be kafka (distributed subscription-publish schema based Message Queue) or RabbitMQ (rabbitmessage Queue).

Wherein the different behavior categories include storage behavior and operation behavior. It should be noted that some of the obtained log data are used for participating in subsequent operations, and some data are only used for data storage. Accordingly, the log data includes arithmetic log data for performing the arithmetic behavior and storage log data for performing the storage behavior.

For example, the obtained log data may be stored in a preset message queue, and operation log data for performing an operation behavior and storage log data for performing a storage behavior may be obtained from the message queue based on a corresponding scheduling thread. And taking the operation log data and the storage log data as data to be processed.

And S130, processing the data to be processed by adopting a scheduling thread corresponding to the behavior category of the data to be processed to obtain log data to be stored.

For example, a scheduling thread for performing an operation behavior may be preset, and a scheduling thread for performing a storage behavior may be preset. And processing the acquired data to be processed based on two different scheduling threads, so that the processed data is used as log data to be stored.

In an alternative embodiment, the behavior categories include storage behaviors and calculation behaviors; correspondingly, the method for processing the data to be processed by adopting the scheduling thread corresponding to the behavior category of the data to be processed to obtain the log data to be stored comprises the following steps: processing data to be processed for executing a storage behavior by adopting a preset first scheduling thread to obtain first log data to be stored; processing the data to be processed of the operation behavior by adopting a preset second scheduling thread to obtain second log data to be stored; generating to-be-stored log data including first to-be-stored log data and second to-be-stored log data.

The first scheduling thread and the second scheduling thread may be preset by a relevant technician according to actual requirements. The first scheduling thread is used for processing data to be processed of executing the storage behaviors; the second scheduling thread is specifically configured to process data to be processed for executing the operation behavior.

For example, a first scheduling thread may be used to perform storage processing on data to be processed for executing a storage behavior, so as to obtain first log data to be stored. And performing operation processing on the data to be processed for executing the operation behavior by adopting a second scheduling thread to obtain second log data to be stored.

It should be noted that, in the process of processing the to-be-processed data for executing the storage behavior by the first scheduling thread, a library writing operation may be performed based on a preset thread pool.

In an optional embodiment, the processing, by using a preset first scheduling thread, data to be processed for executing a storage behavior to obtain first log data to be stored includes: constructing a thread pool for processing to-be-processed data for executing the storage behavior; and transmitting the data to be processed for executing the storage behavior to a thread pool by adopting a first scheduling thread, and processing the data to be processed for executing the storage behavior in the thread pool to obtain first log data to be stored.

The thread pool is in a multi-thread processing mode, tasks can be added into the queue in the processing process, and then the tasks are automatically started after the threads are created. The purpose of using the thread pool is to flexibly control the number of threads according to system requirements and hardware environment, and to uniformly manage and control all threads, thereby improving the operating efficiency of the system and reducing the operating pressure of the system.

Illustratively, a thread pool for processing pending data for performing the store action may be pre-constructed by one of ordinary skill in the art. And transmitting the data to be processed for executing the storage behavior to the thread, and processing the data to be processed for executing the storage behavior in the thread pool, thereby obtaining first log data to be stored.

It can be understood that, in order to accurately construct a thread pool to be used for processing data to be processed for executing the storage behavior, and avoid waste of space, the thread pool may also be constructed in the following manner.

In an alternative embodiment, building a thread pool for processing pending data for performing a storage action includes: and constructing a thread pool for processing the data to be processed of the execution storage behavior according to the task amount and the task time of each request task.

The thread pool may be constructed by determining a core thread count, a blocking queue size, and a maximum thread count for the thread pool.

For example, the core thread number threadcount may be determined as follows:

where tasks represents the number of requested tasks per second. For example, 100 business tasks are performed per second. the taskcost represents the elapsed time for each requested task. For example, it takes 0.1 second to perform the query task.

Illustratively, the blocking queue size queueCapacity may be determined as follows:

where coreSizePool represents the kernel thread pool size. For example, it can be understood how many threads are to process the requested task; responsetime represents the maximum response time that the system is allowed to tolerate, e.g., it takes up to 1 second to perform the query task.

For example, the maximum number of lines maxPoolSize may be determined as follows:

where max (tasks) represents the maximum number of tasks.

In an optional embodiment, the processing, by using a preset second scheduling thread, the to-be-processed data for performing the operation behavior to obtain second to-be-stored log data includes: adopting a second scheduling thread, and performing stream processing on the data to be processed for executing the operation behavior based on a preset model to obtain second log data to be stored; and updating the execution state identifier of the second log data to be stored according to a processing result obtained by processing the data to be processed for executing the operation behavior.

The preset model may be a model for performing operation processing on the data to be processed. For example, the pre-set model may be a Flink distributed processing engine and framework. Flink has a high throughput and low latency, and can process large amounts of data quickly. Also, flink supports different restart strategies, so it can control how to restart when a failure occurs.

For example, the Flink framework may perform stream processing or batch processing on the to-be-processed data for executing the operation behavior, so as to obtain the second to-be-stored log data. And updating the execution state identifier of the second log data to be stored according to a processing result obtained by processing the data to be processed for executing the operation behavior. For example, if the to-be-processed data completes execution and is successfully executed, the execution status identifier of the to-be-processed data may be updated to the consumed status; if the pending data is not executed or is not executed successfully, the execution status flag of the pending data may be updated to an unconsumed status.

And S140, storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored.

Illustratively, if the execution state identifier corresponding to the log data to be stored is consumed, storing the log data to be stored; and if the execution state identifier corresponding to the log data to be stored is not consumed, reloading the log data to be stored.

It is understood that the processed data to be processed in the thread pool disappears after being processed in the thread pool, and at this time, the disappeared data to be processed may be determined as the log data that has been processed.

In an optional embodiment, after obtaining the first log data to be stored, the method further includes; updating the execution state identifier of the first log data to be stored according to the processing result of the thread pool on the data to be processed of the execution storage behavior; the execution state identifier comprises an unconsumed identifier and a consumed identifier; correspondingly, according to the execution state identifier corresponding to the log data to be stored, the processing of storing and/or reloading the log data to be stored includes: if the execution state identifier corresponding to the first log data to be stored is the consumed identifier, storing the first log data to be stored; and if the execution state identifier corresponding to the first log data to be stored is the unconsumed identifier, reloading the first log data to be stored.

For example, if the processing result of the thread pool on the to-be-processed data for executing the storage behavior is data complete processing or data successful processing, the execution state identifier of the first to-be-stored log data is updated to be consumed; and if the processing result of the thread pool on the to-be-processed data for executing the storage behavior is data incomplete processing or data processing failure, updating the execution state identifier of the first to-be-stored log data to be unconsumed. If the execution state identifier corresponding to the first log data to be stored is the consumed identifier, storing the first log data to be stored; and if the execution state identifier corresponding to the first log data to be stored is the unconsumed identifier, reloading the first log data to be stored.

The embodiment of the invention processes the data to be processed by adopting the scheduling thread corresponding to the behavior category of the data to be processed to obtain the log data to be stored; according to the execution state identifier corresponding to the log data to be stored, the log data to be stored is stored and/or reloaded, so that the real-time processing of the log data is realized, and the processing efficiency of the log data is improved.

In an optional embodiment, an embodiment of the present invention further provides a log data processing system, configured to execute the log data processing method described in any embodiment of the present invention. FIG. 1B is a schematic diagram of a log data processing system.

The log data processing system can comprise a control layer for data filtering, an access layer for data storage and transmission, a calculation layer for data calculation, a storage layer for data storage, a service layer for data integration service and an application layer for data application.

Illustratively, micro server A, micro server B, \8230 \ 8230;, and micro server N in the micro service cluster collectively perform input and output processing on request data and response data of a request task. And taking the request data and the response data as log data, and performing filtering operations such as formatting, desensitization and the like on sensitive information in the log data through a regular expression by a control layer. And the real-time data bus is used for reading and writing the log data to prevent blockage. The real-time data bus can be selected from kafka and rabbitmq. And for the log data which only needs to be stored, one scheduling thread is used for reading, so that dirty reading and repeated reading are avoided. The library is written using a thread pool. Determining the core thread number, the size of the blocking queue and the maximum thread number of the thread pool, and constructing the thread pool based on the core thread number, the size of the blocking queue and the maximum thread number. And setting a task state recording table, marking the execution state of each data read from the real-time data bus, and preventing data loss caused by the fact that the tasks in the thread pool queue are not executed when the machine is down. And calculating the log information needing to be subjected to complex operation by using the flink. flink supports different restart strategies so it is possible to control how a restart occurs when a failure occurs. For the persistent data and the real-time data, various query services can be provided through a service layer, so that the effects of quickly positioning problems and finding the problems in advance are achieved. Optionally, the log data may be applied to at least one application service in the application layer.

Example two

Fig. 2 is a schematic structural diagram of a log data processing apparatus according to a second embodiment of the present invention. The log data processing apparatus provided in an embodiment of the present invention is applicable to a situation where real-time log data under a large number of microservice architectures is processed, and the log data processing apparatus may be implemented in a form of hardware and/or software, as shown in fig. 2, and specifically includes: a log data acquisition module 201, a to-be-processed data acquisition module 202, a to-be-processed data processing module 403, and a stored data processing module 204. Wherein the content of the first and second substances,

a log data obtaining module 201, configured to obtain log data generated by the micro server in a process of processing at least one request task; the log data is provided with an execution state identifier;

a to-be-processed data obtaining module 202, configured to store and execute log data in different behavior categories through a preset message queue, and obtain the log data from the message queue as to-be-processed data;

the to-be-processed data processing module 203 is configured to process the to-be-processed data by using a scheduling thread corresponding to a behavior category of the to-be-processed data, so as to obtain to-be-stored log data;

and the storage data processing module 204 is configured to perform storage and/or reloading processing on the log data to be stored according to the execution state identifier corresponding to the log data to be stored.

Optionally, the behavior category includes a storage behavior and an operation behavior;

correspondingly, the to-be-processed data processing module 203 includes:

the first storage data determining unit is used for processing the data to be processed for executing the storage behavior by adopting a preset first scheduling thread to obtain first log data to be stored; and (c) a second step of,

the second storage data determining unit is used for processing the data to be processed for executing the operation behavior by adopting a preset second scheduling thread to obtain second log data to be stored;

and the to-be-processed data processing unit is used for generating to-be-stored log data comprising the first to-be-stored log data and the second to-be-stored log data.

Optionally, the first stored data determining unit includes:

a thread pool constructing subunit, configured to construct a thread pool for processing the to-be-processed data of the execution storage behavior;

and the first storage data determining subunit is configured to transmit the to-be-processed data for performing the storage behavior to the thread pool by using the first scheduling thread, and process the to-be-processed data for performing the storage behavior in the thread pool to obtain first to-be-stored log data.

Optionally, the thread pool building subunit is specifically configured to:

and constructing a thread pool for processing the data to be processed of the execution storage behavior according to the task amount and the task time of each request task.

Optionally, the first stored data determining unit further includes:

the identification updating subunit is configured to update the execution state identification of the first log data to be stored according to the processing result of the thread pool on the data to be processed of the execution storage behavior; the execution state identification comprises an unconsumed identification and a consumed identification;

correspondingly, the storage data processing module comprises:

the storage processing unit is used for storing the first log data to be stored if the execution state identifier corresponding to the first log data to be stored is the consumed identifier;

and the reloading unit is used for reloading the first log data to be stored if the execution state identifier corresponding to the first log data to be stored is the unconsumed identifier.

Optionally, the second storage data determining unit includes:

the second storage data determining subunit is used for performing stream processing on the to-be-processed data for executing the operation behavior based on a preset model by using the second scheduling thread to obtain second to-be-stored log data;

and the second storage data determining subunit is used for updating the execution state identifier of the second log data to be stored according to a processing result obtained by processing the data to be processed for executing the operation behavior.

Optionally, the apparatus further comprises:

the sensitive information identification module is used for identifying sensitive information in log data after the log data generated by the micro server in the process of processing at least one request task is acquired;

and the information encryption module is used for encrypting the sensitive information to obtain encrypted log data.

The log data processing device provided by the embodiment of the invention can execute the log data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

EXAMPLE III

FIG. 3 shows a schematic block diagram of an electronic device 30 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 3, the electronic device 30 includes at least one processor 31, and a memory communicatively connected to the at least one processor 31, such as a Read Only Memory (ROM) 32, a Random Access Memory (RAM) 33, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 31 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 32 or the computer program loaded from the storage unit 38 into the Random Access Memory (RAM) 33. In the RAM 33, various programs and data necessary for the operation of the electronic apparatus 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other via a bus 34. An input/output (I/O) interface 35 is also connected to bus 34.

A plurality of components in the electronic device 30 are connected to the I/O interface 35, including: an input unit 36 such as a keyboard, a mouse, etc.; an output unit 37 such as various types of displays, speakers, and the like; a storage unit 38 such as a magnetic disk, optical disk, or the like; and a communication unit 39 such as a network card, modem, wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 31 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 31 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 31 performs the various methods and processes described above, such as a log data processing method.

In some embodiments, the log data processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 30 via the ROM 32 and/or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the log data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log data processing method, comprising:

acquiring log data generated by the micro server in the process of processing at least one request task; wherein, the log data is provided with an execution state identifier;

2. The method of claim 1, wherein the behavior categories include storage behaviors and calculation behaviors;

correspondingly, the processing the data to be processed by using the scheduling thread corresponding to the behavior category of the data to be processed to obtain the log data to be stored includes:

processing the data to be processed for executing the storage behavior by adopting a preset first scheduling thread to obtain first log data to be stored; and the number of the first and second groups,

processing the data to be processed for executing the operation behavior by adopting a preset second scheduling thread to obtain second log data to be stored;

and generating to-be-stored log data comprising the first to-be-stored log data and the second to-be-stored log data.

3. The method according to claim 2, wherein the processing the to-be-processed data that performs the storing action by using a preset first scheduling thread to obtain first to-be-stored log data includes:

constructing a thread pool for processing the data to be processed of the execution storage behavior;

and transmitting the data to be processed for executing the storage behavior to the thread pool by adopting the first scheduling thread, and processing the data to be processed for executing the storage behavior in the thread pool to obtain first log data to be stored.

4. The method of claim 3, wherein building a thread pool for processing pending data for the execution of the storage activity comprises:

5. The method according to claim 3, further comprising, after the obtaining the first log data to be stored;

updating the execution state identifier of the first log data to be stored according to the processing result of the thread pool on the data to be processed of the execution storage behavior; the execution state identification comprises an unconsumed identification and a consumed identification;

correspondingly, the storing and/or reloading the log data to be stored according to the execution state identifier corresponding to the log data to be stored includes:

if the execution state identifier corresponding to the first log data to be stored is the consumed identifier, storing the first log data to be stored;

and if the execution state identifier corresponding to the first log data to be stored is the unconsumed identifier, reloading the first log data to be stored.

6. The method according to claim 2, wherein the processing the to-be-processed data that performs the operation by using a preset second scheduling thread to obtain second to-be-stored log data includes:

performing stream processing on the data to be processed for executing the operation behavior by adopting the second scheduling thread based on a preset model to obtain second log data to be stored;

and updating the execution state identifier of the second log data to be stored according to a processing result obtained by processing the data to be processed for executing the operation behavior.

7. The method according to any one of claims 1-6, further comprising, after the obtaining log data generated by the micro server during processing of the at least one requested task:

identifying sensitive information in the log data;

and encrypting the sensitive information to obtain encrypted log data.

8. A log data processing apparatus characterized by comprising:

the to-be-processed data acquisition module is used for storing and executing log data under different behavior categories through a preset message queue and acquiring the log data from the message queue as to-be-processed data;

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log data processing method of any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions for causing a processor to implement the log data processing method of any one of claims 1 to 7 when executed.