CN111045782B - Log processing method, device, electronic equipment and computer readable storage medium - Google Patents
Log processing method, device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111045782B CN111045782B CN201911142773.2A CN201911142773A CN111045782B CN 111045782 B CN111045782 B CN 111045782B CN 201911142773 A CN201911142773 A CN 201911142773A CN 111045782 B CN111045782 B CN 111045782B
- Authority
- CN
- China
- Prior art keywords
- log
- data
- processing thread
- log data
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 243
- 238000000034 method Methods 0.000 claims abstract description 57
- 230000008569 process Effects 0.000 claims abstract description 37
- 238000004891 communication Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 8
- 238000011161 development Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 239000000758 substrate Substances 0.000 claims 1
- 230000002159 abnormal effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log processing method, a log processing device, electronic equipment and a computer readable storage medium. The method comprises the following steps: creating a business processing thread and a log processing thread; the log data generated in the running process of the application program is written into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file. The invention introduces a log processing thread, adopts an asynchronous processing mode, obtains log data generated in the running process of an application program by the service processing thread, sends the log data to the log processing thread through a queue, and outputs the log data to a log file by the log processing thread, so that a large-overhead log processing task and a data processing task are effectively isolated, the log processing pressure of the service processing thread is reduced, the data processing capacity is improved, and the data processing efficiency of the service processing thread is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log processing method, a log processing device, an electronic device, and a computer readable storage medium.
Background
With the rapid development of the internet and the continuous improvement of traffic, the data access flow based on the network is rapidly increased, and especially the access flow of data centers, large enterprises, portal sites and the like is even up to the level of 10 Gb/s. Compared with the development of network technology, the increase of network bandwidth and application service is far higher than the increase of server processing speed and memory access speed, and the number of users is continuously increased while the network bandwidth is increased, so that the resources of the server are seriously consumed, the server becomes a network bottleneck and also often becomes a network failure point. Therefore, constructing server clusters, and balancing traffic among server clusters using load balancing techniques becomes a major solution to these problems.
Currently, in order to improve load balancing performance, many new techniques are applied to load balancers. For example: the DPVS (Data Plane Development Kit-Virtual Server, data plane development kit) load equalizer is a Virtual Server (Virtual Server) based on DPDK (Data Plane Development Kit ), so that a DPDK is used to realize a high-performance four-layer load equalizer, and the data processing performance and throughput can be improved by avoiding interrupt, lockless, CPU (central processing unit ) affinity, large page memory and other technologies, and the working efficiency of the data plane application program is improved.
However, when an anomaly occurs in the DPVS load balancer, for example: when the memory resources are insufficient or other resources are insufficient, the DPVS load balancer can cause low data processing efficiency due to processing of log data corresponding to abnormal information. Specifically, the DPVS load balancer prints log data in a multi-core multithreaded polling mode. That is, in the DPVS load balancer, the data cores and threads are in one-to-one correspondence, when polling among a plurality of data cores, the thread corresponding to the polled data core prints log data generated by the polled data core through an IO (Input/Output) interface operation, but because the time of the IO interface operation is long, the next data core needs to be polled after waiting for the completion of the IO interface operation, so that the polling process time is too long. When an abnormality occurs in the running process of the application program, the data core can print a large amount of log data of the abnormality information, and the data processing efficiency is reduced.
Disclosure of Invention
An embodiment of the invention aims to provide a log processing method, a log processing device, electronic equipment and a computer readable storage medium, so as to solve the problem that when a DPVS load balancer is abnormal, the DPVS load balancer can cause low data processing efficiency due to processing log data corresponding to abnormal information. The specific technical scheme is as follows:
in a first aspect of the present invention, there is provided a log processing method, including: creating a business processing thread and a log processing thread; the log data generated in the running process of the application program is written into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
The creating a business processing thread and a log processing thread comprises the following steps: creating a plurality of business processing threads and a log processing thread in a DPVS load balancer; each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core.
The outputting the log data to a preset log file includes: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core.
The process of writing the log data generated in the running process of the application program into the queue of the preset type through the service processing thread further comprises the following steps: if the queue of the preset type is blocked, the log data generated in the running process of the application program is temporarily written into the queue of the preset type through the service processing thread, or the log data only keeping the information of the preset field is written into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following: function name and line number corresponding to the exception code.
Before the log data is output to a preset log file, the method further comprises the following steps: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; performing deduplication processing on the plurality of log data according to hash values respectively corresponding to the plurality of log data to obtain a plurality of log data without repetition; the outputting the log data to a preset log file comprises the following steps: and outputting the plurality of log data without repetition to a preset log file through an input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
In a second aspect of the present invention, there is also provided a log processing apparatus, including: the creation module is used for creating a business processing thread and a log processing thread; the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file.
In yet another aspect of the present invention, there is also provided an electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory perform communication with each other through the communication bus; a memory for storing a computer program; and a processor for implementing any of the above-described method steps when executing a program stored on the memory.
In yet another aspect of the present invention, there is also provided a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform any of the methods described above.
In yet another aspect of the invention there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the methods described above.
The embodiment of the invention provides a log processing method, a device, electronic equipment and a computer readable storage medium, which introduce a log processing thread, acquire log data generated in the running process of an application program by a business processing thread in an asynchronous processing mode, send the log data to the log processing thread through a queue, output the log data to a log file by the log processing thread, effectively isolate a large-cost log processing task from a data processing task, reduce the log processing pressure of the business processing thread, improve the data processing capacity and improve the data processing efficiency of the business processing thread.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow chart of a log processing method according to an embodiment of the invention;
FIG. 2 is a flowchart of a log processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a log processing method according to an embodiment of the invention;
FIG. 4 is a block diagram of a log processing apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.
The embodiment of the invention provides a log processing method. The execution body of the embodiment is a device that needs to process log data. For example: the execution body is a server, a load balancer and the like. The load balancer may be a DPVS load balancer.
Fig. 1 is a flowchart of a log processing method according to an embodiment of the invention.
In step S110, a business processing thread and a log processing thread are created.
And the business processing thread is used for transmitting the log data generated in the running process of the application program to the log processing thread. The application program runs in the execution body of the present embodiment.
And the log processing thread is used for outputting the log data transmitted by the service processing thread to the log file.
In this embodiment, with the CPU affinity of the DPVS load balancer, a plurality of traffic processing threads and one log processing thread can be created at the DPVS load balancer. Each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core.
And step S120, writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread.
In this embodiment, the service processing thread may transmit log data to the log processing thread through a preset type of queue. The type of the queue includes: lock-free queues and/or message queues.
The log data includes at least the following field information: function name and line number corresponding to the exception code. Of course, in the log data, the following field information may be further included: information of a data core and custom field information of the log data are generated. The information of the data core is, for example, the ID of the data core. The custom field information may be set according to requirements, for example: the code module name where the exception code is located.
Specifically, during the running process of the application program, the data core generates log data, and the service processing thread acquires the log data and writes the log data into a queue of a preset type. The data cores and the service processing threads have corresponding relations, and the service processing threads acquire log data generated by the corresponding data cores, so that the service processing threads acquiring the log data can be determined according to the information of the data cores in the log data.
In the process of writing the log data generated in the running process of the application program into the preset type of queue through the service processing thread, the preset type of queue may have a blocking problem, if the preset type of queue is blocked, the writing of the log data generated in the running process of the application program into the preset type of queue through the service processing thread is suspended, or the writing of the log data only keeping the preset field information into the preset type of queue through the service processing thread is suspended. The preset field information may be important information in log data, for example: the preset field information includes at least one of: function name and line number corresponding to the exception code. Only keeping important field information in the log data can shorten the length of the log data and reduce the processing pressure of the log processing thread.
And step S130, acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
And acquiring and caching the log data from the queue through a log processing thread, and outputting the cached log data to a preset log file by adopting the same or different output modes according to the difference of the number of the cached log data every preset time period. For example:
if the number of the log data is one, the log data is directly output to a preset log file through IO interface operation.
If the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; performing deduplication processing on the plurality of log data according to hash values respectively corresponding to the plurality of log data to obtain a plurality of log data without repetition; and outputting the plurality of log data without repetition to a preset log file through IO (Input/Output) interface operation. Further, the deduplication process includes: inquiring a plurality of log data with the same hash value, and reserving one log data in the plurality of log data with the same hash value.
The above example adopts different output modes to output the log data to a preset log file. When the log data are more, particularly for the abnormal log data with strong repeatability, a large number of repeated log data can be filtered out by carrying out de-duplication processing on the log data, so that a large number of nonsensical overheads are reduced.
In this embodiment, the number of service processing threads may be multiple, and before the log data is output to a preset log file, a log file may be set for each service processing thread correspondingly, or a log file that is common to multiple service processing threads may be set. If a log file is correspondingly set for each service processing thread, determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core. For example: the field information of the log data may be the ID of the data core.
In this embodiment, a log processing thread is introduced, an asynchronous processing mode is adopted, log data generated in the running process of an application program is obtained by a service processing thread, the log data is sent to the log processing thread through a queue, the log processing thread outputs the log data to a log file, a log processing task with high overhead and a data processing task are effectively isolated, the log processing pressure of the service processing thread is reduced, the data processing capability is improved, and the data processing efficiency of the service processing thread is improved.
The log processing method of the present invention will be further described below taking as an example log processing performed at the DPVS load balancer.
Fig. 2 is a flowchart of a log processing method according to an embodiment of the invention.
Step S210, a plurality of business processing threads and a log processing thread are created, and the business processing threads and the log processing thread are respectively bound with corresponding data cores.
The DPVS load equalizer comprises a plurality of data cores, and the created threads and the data cores are in one-to-one correspondence.
The log processing thread is correspondingly bound with a log processing data core. The log processing data core is used for providing operation resources and storage resources for log processing threads.
Each service processing thread is correspondingly bound with one service processing data core. Each business process data core is configured to provide computing resources and storage resources for a business process thread, and each business process data core runs a portion of code of an application program and generates log data during the running of the portion of code.
In step S220, the plurality of service processing threads respectively write the log data generated by the corresponding data core (service processing data core) into a preset lock-free queue.
The lock-free queue can be correspondingly arranged for each service processing thread, and a common lock-free queue can be correspondingly arranged for a plurality of service processing threads. And the service processing thread writes the log data generated by the corresponding service processing data core into the lock-free queue corresponding to the service processing thread.
Fig. 3 is a schematic diagram of a log processing method according to an embodiment of the invention. The DPVS load equalizer comprises n data cores (n is greater than 1), each data core is correspondingly bound with a service processing thread, and each service processing thread is correspondingly provided with a lock-free queue, so that n lock-free queues are totally arranged, and the service processing threads write log data generated by the corresponding data core into the corresponding lock-free queues, for example: the service processing thread 1 writes the log generated by the data core 1 into the lock-free queue 1.
In the process of writing the log data into the lock-free queue, if the lock-free queue is full (blocked), the log data written into the lock-free queue can be suspended, and the log data which needs to be written into the lock-free queue is directly discarded; important field information in the log data can be selected, the length of the log data is shortened, and the log data with the shortened length is written into the lock-free queue, so that the log data is processed, the data processing performance is not affected, and the balance between the log processing efficiency and the data processing performance is achieved.
In step S230, the log processing thread acquires log data from the lock-free queue, and performs deduplication processing on the acquired log data every a preset time period.
After the log processing thread acquires the log data, the log processing thread does not directly output the log data to the log file, but caches the log data. As shown in fig. 3, the log processing thread acquires log data from the lock-free queue 1 to the lock-free queue n, and caches the acquired log data in the corresponding log processing data core so as to integrate the acquired log data to obtain log data without repetition, and then outputs the log data without repetition to the log file.
Step S240, the log processing thread outputs log data obtained after the duplicate removal processing to a preset log file.
In this embodiment, by means of asynchronous processing, the multiple service processing threads are respectively responsible for obtaining log data generated by the corresponding service processing data cores, and the log processing threads uniformly output the log data generated by the multiple service processing data cores to the log file, so that the large-overhead log processing task and the data processing task are effectively isolated, the log printing pressure of the service processing data cores is relieved, and each service processing data core can still execute the data processing task with high efficiency under the abnormal condition. The problem of the decline of the data core processing capability under the abnormal condition is solved through the embodiment, and the stability of the DPVS load equalizer is effectively improved, thereby providing a stable network environment for high-concurrency and large-flow business.
The embodiment of the invention also provides a log processing device. As shown in fig. 4, a log processing apparatus according to an embodiment of the present invention is a structural diagram.
The log processing device comprises: a creation module 410, a writing module 420 and an output module 430.
A creation module 410, configured to create a business processing thread and a log processing thread.
And the writing module 420 is configured to write log data generated in the running process of the application program into a queue of a preset type through the service processing thread.
And an output module 430, configured to obtain the log data from the queue through the log processing thread, and output the log data to a preset log file.
Wherein, the creation module 410 is configured to: creating a plurality of business processing threads and a log processing thread in a DPVS load balancer; each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core.
Wherein, the output module 430 is further configured to: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core.
The writing module 420 is further configured to, in a process of writing, by the service processing thread, log data generated in an application running process into a preset type of queue, if the preset type of queue is blocked, suspend writing, by the service processing thread, log data generated in the application running process into the preset type of queue, or write, by the service processing thread, log data that only retains preset field information into the preset type of queue.
Wherein the preset field information at least comprises one of the following: function name and line number corresponding to the exception code.
Wherein, the output module 430 is further configured to: before the log data are output to a preset log file, if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; performing deduplication processing on the plurality of log data according to hash values respectively corresponding to the plurality of log data to obtain a plurality of log data without repetition; and outputting the plurality of log data without repetition to a preset log file through an input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
The embodiment of the present invention further provides an electronic device, as shown in fig. 5, including a processor 510, a communication interface 520, a memory 530, and a communication bus 540, where the processor 510, the communication interface 520, and the memory 530 complete communication with each other through the communication bus 540.
A memory 530 for storing a computer program.
The processor 510 is configured to execute the program stored in the memory 530, and implement the following steps: creating a business processing thread and a log processing thread; the log data generated in the running process of the application program is written into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
The creating a business processing thread and a log processing thread comprises the following steps: creating a plurality of business processing threads and a log processing thread in a DPVS load balancer; each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core.
The outputting the log data to a preset log file includes: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core.
The process of writing the log data generated in the running process of the application program into the queue of the preset type through the service processing thread further comprises the following steps: if the queue of the preset type is blocked, the log data generated in the running process of the application program is temporarily written into the queue of the preset type through the service processing thread, or the log data only keeping the information of the preset field is written into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following: function name and line number corresponding to the exception code.
Before the log data is output to a preset log file, the method further comprises the following steps: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; performing deduplication processing on the plurality of log data according to hash values respectively corresponding to the plurality of log data to obtain a plurality of log data without repetition; the outputting the log data to a preset log file comprises the following steps: and outputting the plurality of log data without repetition to a preset log file through an input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer readable storage medium is provided, in which instructions are stored, which when run on a computer, cause the computer to perform the log processing method according to any one of the above embodiments.
In yet another embodiment of the present invention, a computer program product containing instructions that, when run on a computer, cause the computer to perform the log processing method of any of the above embodiments is also provided.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (8)
1. A log processing method, comprising:
creating a business processing thread and a log processing thread;
the creating a business processing thread and a log processing thread comprises the following steps:
creating a plurality of business processing threads and a log processing thread in a DPVS load balancer of a data plane development suite virtual server; wherein, each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core;
the log data generated in the running process of the application program is written into a queue of a preset type through the service processing thread;
acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file; the outputting the log data to a preset log file comprises the following steps: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core.
2. The method of claim 1, wherein during writing log data generated during application running into a pre-set type of queue by the service processing thread, further comprising:
if the queue of the preset type is blocked, the log data generated in the running process of the application program is temporarily written into the queue of the preset type through the service processing thread, or the log data only keeping the information of the preset field is written into the queue of the preset type through the service processing thread.
3. The method of claim 2, wherein the preset field information includes at least one of: function name and line number corresponding to the exception code.
4. The method of claim 1, wherein the step of determining the position of the substrate comprises,
before the log data is output to a preset log file, the method further comprises the following steps:
if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data;
performing deduplication processing on the plurality of log data according to hash values respectively corresponding to the plurality of log data to obtain a plurality of log data without repetition;
the outputting the log data to a preset log file comprises the following steps:
and outputting the plurality of log data without repetition to a preset log file through an input/output (IO) interface operation.
5. The method of any of claims 1-4, wherein the type of queue comprises: lock-free queues and/or message queues.
6. A log processing apparatus, comprising:
the creation module is used for creating a business processing thread and a log processing thread; the creating a business processing thread and a log processing thread comprises the following steps: creating a plurality of business processing threads and a log processing thread in a DPVS load balancer of a data plane development suite virtual server; wherein, each business processing thread is correspondingly bound with one business processing data core, and each log processing thread is correspondingly bound with one log processing data core;
the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread;
the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file; the outputting the log data to a preset log file comprises the following steps: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread according to the service processing thread corresponding to the service processing data core.
7. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1-5 when executing a program stored on a memory.
8. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method steps of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142773.2A CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142773.2A CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111045782A CN111045782A (en) | 2020-04-21 |
| CN111045782B true CN111045782B (en) | 2024-01-12 |
Family
ID=70232504
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142773.2A Active CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111045782B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113810337B (en) * | 2020-06-12 | 2023-08-08 | 天翼云科技有限公司 | Method, device and storage medium for network message deduplication |
| CN111858077A (en) * | 2020-07-15 | 2020-10-30 | 济南浪潮数据技术有限公司 | Recording method, device and equipment for IO request log in storage system |
| CN112306980A (en) * | 2020-10-30 | 2021-02-02 | 深圳市元征科技股份有限公司 | Log processing method and terminal equipment |
| CN112380001A (en) * | 2020-10-30 | 2021-02-19 | 网宿科技股份有限公司 | Log output method, load balancing device and computer readable storage medium |
| CN112948136A (en) * | 2021-02-02 | 2021-06-11 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Method for implementing asynchronous log record of embedded operating system |
| CN113342632A (en) * | 2021-07-07 | 2021-09-03 | 北京物芯科技有限责任公司 | Simulation data automatic processing method and device, electronic equipment and storage medium |
| CN113420032A (en) * | 2021-07-20 | 2021-09-21 | 奇安信科技集团股份有限公司 | Classification storage method and device for logs |
| CN114265637A (en) * | 2021-12-24 | 2022-04-01 | ***数智科技有限公司 | Archive management system configuration method, system, electronic device and storage medium |
| CN117891695B (en) * | 2024-03-14 | 2024-06-21 | 新华三技术有限公司 | Log recording method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9069790B2 (en) * | 2013-03-14 | 2015-06-30 | Stephen P. LORD | Multi-threaded message passing journal |
-
2019
- 2019-11-20 CN CN201911142773.2A patent/CN111045782B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111045782A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045782B (en) | Log processing method, device, electronic equipment and computer readable storage medium | |
| KR102563165B1 (en) | Memory access request scheduling method, device, apparatus, and storage medium | |
| US10613992B2 (en) | Systems and methods for remote procedure call | |
| CN110389905B (en) | Resource release method, resource allocation method, device and computer program product | |
| CN110753112A (en) | Elastic expansion method and device of cloud service | |
| US11201836B2 (en) | Method and device for managing stateful application on server | |
| US20180006970A1 (en) | Technologies for scalable packet reception and transmission | |
| US9424091B2 (en) | Deploying software in a multi-instance node | |
| EP2829972B1 (en) | Method and apparatus for allocating stream processing unit | |
| CN114595043A (en) | IO (input/output) scheduling method and device | |
| US20210117263A1 (en) | Using Real-Time Analytics To Manage Application Features | |
| US11108698B2 (en) | Systems and methods for client-side throttling after server handling in a trusted client component | |
| US10616317B2 (en) | Method and system for affinity load balancing | |
| US10664191B2 (en) | System and method for providing input/output determinism based on required execution time | |
| US10459776B2 (en) | Transmission of large messages in computer systems | |
| CN112579282A (en) | Data processing method, device, system and computer readable storage medium | |
| CN110968370B (en) | Data processing method and device | |
| US12039378B2 (en) | In-band modification of event notification preferences for server events | |
| JP2019179487A (en) | Information processing device, control method, and program | |
| CN115297169B (en) | Data processing method, device, electronic equipment and medium | |
| CN114185682B (en) | Log output method and device, electronic equipment and storage medium | |
| CN111459649B (en) | Method, apparatus and computer readable medium for managing memory of computing resources | |
| CN114063891B (en) | Method, apparatus and computer program product for managing data transmission | |
| CN116233039A (en) | Message sending method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |