CN108989101B - Log output system and method and electronic equipment - Google Patents
Log output system and method and electronic equipment Download PDFInfo
- Publication number
- CN108989101B CN108989101B CN201810725475.5A CN201810725475A CN108989101B CN 108989101 B CN108989101 B CN 108989101B CN 201810725475 A CN201810725475 A CN 201810725475A CN 108989101 B CN108989101 B CN 108989101B
- Authority
- CN
- China
- Prior art keywords
- log
- flow
- server
- log data
- switch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention provides a log output system, a log output method and electronic equipment, wherein the system comprises: the system comprises a switch, a load balancing server and a log acquisition server; the exchanger is used for receiving the service flow, carrying out mirror image operation on the received service flow to obtain mirror image flow and sending the mirror image flow to the log collection server; the service flow comprises the following steps: the switch is used for sending to the load balancing server and/or receiving from the load balancing server; and the log collection server is used for collecting log data from the received mirror flow and sending the collected log data to the ELK cluster. The log processing work which is originally taken charge of by the load balancing server is transferred to the log collection server by additionally arranging the log collection server, so that the bearing pressure of the load balancing server is reduced, the problems of packet loss or delay, loss of key logs and the like can be prevented, and the working performance of the log output system is improved.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a log output system, a log output method and electronic equipment.
Background
As shown in fig. 1, the log output system in the current network generally includes: a switch 11 and a load balancing server 12, the log output system belongs to a subsystem of a service processing system, and the service processing system generally comprises: a switch 11, a load balancing server 12 and a traffic processing server 13. The main functions of the log output system comprise: service distribution and log collection, in which a service is distributed to a service processing server for processing, and collected log data is imported to an ELK (elastic search, logstack, Kibana, a log analysis system) cluster 14 for analysis, storage and display. Specifically, the load balancing server 12 receives the service traffic, and sends the service traffic to the switch 11 and/or to the service processing server 13, and the load balancing server 13 performs log collection on the received service traffic, and sends collected log data to the ELK cluster 14 for analysis, storage, and display.
However, the inventor finds that the prior art has at least the following problems in the process of implementing the invention: the load balancing server needs to distribute the service and collect the log, so that the load pressure of the load balancing server is too large. And the excessive load pressure of the load balancing server often causes the loss of key logs or causes the phenomenon of packet loss delay, thereby causing the reduction of the working performance of a log output system.
Disclosure of Invention
An embodiment of the present invention provides a log output system, a log output method, an electronic device, and a storage medium, so as to improve the working performance of the log output system. The specific technical scheme is as follows:
in a first aspect, to achieve the above object, an embodiment of the present invention discloses a log output system, where the system includes: the system comprises a switch, a load balancing server and a log acquisition server;
the switch is used for receiving the service flow, carrying out mirror image operation on the received service flow to obtain mirror image flow and sending the mirror image flow to the log collection server; wherein the service traffic comprises: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
and the log collection server is used for collecting log data from the received mirror flow and sending the collected log data to the ELK cluster.
Optionally, the log collection server is specifically configured to record log data from the received mirror traffic, compress the recorded log data, and send the compressed log data to the ELK cluster.
Optionally, the log collection server is specifically configured to filter and record log data according to a preset rule from the received mirror traffic, compress the recorded log data, and send the compressed log data to the ELK cluster.
Optionally, the preset rule is set according to at least one of the following elements:
source IP, destination IP, source port, destination port, transport layer protocol.
In another aspect of the present invention, an embodiment of the present invention further provides a method for log output, where the method is applied to a switch in the log output system, and the log output system includes: the system comprises a switch, a load balancing server and a log acquisition server;
the method comprises the following steps:
receiving a service flow, wherein the service flow comprises: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
performing mirror image operation on the service flow to obtain mirror image flow;
and sending the mirror image flow to the log collection server so that the log collection server collects log data from the received mirror image flow and sends the collected log data to the ELK cluster.
Optionally, the step of performing a mirroring operation on the service traffic to obtain a mirrored traffic includes:
and carrying out full-volume mirror image operation on the service flow to obtain mirror image flow.
In another aspect of the present invention, an embodiment of the present invention further provides a log output method, where the method is applied to a log collection server in the log output system, and the log output system includes: the system comprises a switch, a load balancing server and a log acquisition server;
the method comprises the following steps:
receiving mirror image flow sent by the switch; the mirror flow is obtained by the switch performing mirror operation on the service flow received by the switch, and the service flow includes: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
and collecting log data from the received mirror image flow, and sending the collected log data to the ELK cluster.
Optionally, the step of collecting log data from the received mirror image traffic and sending the collected log data to the ELK cluster includes:
and recording log data from the received mirror image flow, compressing the recorded log data, and sending the compressed log data to the ELK cluster.
Optionally, the step of collecting log data from the received mirror image traffic and sending the collected log data to the ELK cluster includes:
and filtering and recording log data according to a preset rule from the received mirror image flow, compressing the recorded log data, and sending the compressed log data to the ELK cluster.
Optionally, the preset rule is set according to at least one of the following elements:
source IP, destination IP, source port, destination port, transport layer protocol.
In another aspect of the present invention, an embodiment of the present invention further provides an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
the memory is used for storing a computer program;
the processor is configured to implement the method steps of any log output method when executing the program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which, when executed on a computer, cause the computer to perform the method steps of any of the log output methods described above.
In another aspect of the present invention, the present invention also provides a computer program product containing instructions, which when executed on a computer, causes the computer to perform the method steps of any of the log output methods described above.
In the log output system, the log output method, the electronic device and the storage medium provided by the embodiment of the invention, the switch receives the service traffic, performs mirroring operation on the received service traffic to obtain the mirror traffic, and sends the mirror traffic to the log collection server; the service flow comprises the following steps: the switch is used for sending to the load balancing server and/or receiving from the load balancing server; and the log collection server collects log data from the received mirror flow and sends the collected log data to the ELK cluster.
By additionally arranging the log acquisition server, the log processing work which is originally taken charge of by the load balancing server is transferred to the log acquisition server, so that the bearing pressure of the load balancing server is reduced, and the problems of packet loss or delay, loss of key logs and the like can be effectively prevented. Therefore, the operation performance of the log output system can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a schematic structural diagram of a business processing system (with an ELK cluster connected) in the prior art;
fig. 2 is a schematic structural diagram of a log output system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a service processing system (connected with an ELK cluster) according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a log output method according to an embodiment of the present invention;
fig. 5 is another schematic flowchart of a log output method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
In the prior art, the load balancing server needs to distribute the service and collect the logs, so that the load balancing server has overlarge load pressure. And the excessive load pressure often causes the loss of key logs or causes the phenomenon of packet loss delay, thereby causing the reduction of the working performance of a log output system. In order to solve the technical problem, embodiments of the present invention provide a log output system, a log output method, an electronic device, and a storage medium.
Referring to fig. 2, the log output system includes: the switch 21, the load balancing server 22 and the log collection server 23;
the switch 21 is configured to receive the service traffic, perform mirroring operation on the received service traffic to obtain mirror traffic, and send the mirror traffic to the log acquisition server 23; the service flow comprises the following steps: the switch 21 is used for sending to the load balancing server 22, and/or the switch 21 receives from the load balancing server 22;
in the traffic received by the switch 21, the part for sending to the load balancing server 22 may be sent to the load balancing server 22 by the switch 21, and then distributed to each traffic processing server by the load balancing server 22, so as to implement service distribution. The traffic received by the switch 21 may be forwarded by the switch 21 in portions received from the load balancing server 22.
The traffic flow may include: the switch 21 may send the request traffic to the load balancing server 22, and the load balancing server 22 sends the request traffic to the service processing server to process the service. The response service traffic is a response made by the service processing server to the service request, and may be sent to the switch 21 through the load balancing server 22, and then returned to the source device of the service request traffic by the switch 21.
The service traffic received by the switch 21 may be only extranet traffic, or only intranet traffic, or may be part of intranet traffic and another part of extranet traffic.
When the switch 21 performs the mirroring operation on the received service traffic, the switch may perform a full mirroring operation on the received service traffic or may perform a partial mirroring operation on the received service traffic. The mirroring operation refers to copying the traffic from a source port to a destination port, where the destination port refers to a port for sending the obtained mirroring traffic to other devices, and the mirroring operation may be performed to copy the traffic.
Specifically, the total mirroring operation or the partial mirroring operation is set according to the size of the received service traffic and the maximum traffic limit that can be carried by the switch 21, and the sum of the size of the service traffic and the size of the mirroring traffic cannot be greater than the maximum traffic limit that can be carried by the switch 21. By performing the full-volume mirroring operation on the traffic sent to the load balancing server 22 and the traffic from the load balancing server 22, the obtained mirrored traffic includes the full-volume log information, which facilitates the collection of the full-volume log data.
And the log collection server 23 is configured to collect log data from the received mirror traffic, and send the collected log data to the ELK cluster.
The traffic flow may include first information used for generating log data, such as a time point, a source IP, a destination IP, a source port, a destination port, a transport layer protocol, a minimum delay, a maximum delay, and the like, and the mirror traffic obtained through the mirror operation may include not only the first information used for generating log data included in the traffic flow, but also second information used for determining whether the traffic flow is rejected or allowed, and the log data may be collected by using the first information and the second information.
The ELK cluster is hung with the storage disk, so that distributed storage of log data can be realized, and meanwhile, analysis and display of the log data can be realized.
By applying the embodiment shown in fig. 2, the log acquisition server 23 is additionally arranged, and the log processing work originally handled by the load balancing server 22 is transferred to the log acquisition server 23, so that the bearing pressure of the load balancing server 22 is reduced, and the problems of packet loss or delay, loss of key logs and the like can be effectively prevented. Therefore, the operation performance of the log output system can be improved.
In an implementation manner of the embodiment of the present invention, the log collection server 23 is specifically configured to record log data from the received mirror traffic, compress the recorded log data, and send the compressed log data to the ELK cluster. The log collection server 23 completes the recording and compressing actions, i.e. completes the collection of the log, so that the storage space can be saved.
In another implementation manner of the embodiment of the present invention, the log collection server 23 is specifically configured to filter and record log data according to a preset rule from the received mirror traffic, compress the recorded log data, and send the compressed log data to the ELK cluster. The log collection server completes the actions of recording, filtering and compressing, namely, completes the collection of the log. Recording and filtering may be performed simultaneously, with recording being selected after filtering. The setting not only saves the storage space, but also can realize better and more targeted display of the ELK cluster through the filtering operation, thereby being convenient for keeping the real-time performance of log data and quickly responding to the query requirement of the service.
Further, the preset rule is set according to at least one of the following elements:
source IP, destination IP, source port, destination port, transport layer protocol. Specifically, for example, the preset rule may be a preset rule that only log information of the source IP, the destination IP, the source port, the destination port, the transport layer protocol and the like of the traffic flow is determined according to the source IP, the destination IP, the source port, the destination port and the transport layer protocol.
In practical application, the following information can be obtained through log collection: outbound and inbound traffic, the NIC (network card) to which the traffic is adapted, preset rules, and whether the traffic is allowed or denied. Illustratively, the log collection server refers to table one from the log data collected in the mirror traffic.
It is understood that the preset rule may be set according to the element in the column of type in table one, that is, it may be considered which log data is collected according to the actual requirement.
Watch 1
It should be noted that, in the foregoing embodiment, the number of the switches, the load balancing server, and the log collecting server in the log outputting system is not specifically limited.
Thus, it will be appreciated that in other embodiments, traffic for a switch to send to a load balancing server may be sent directly or indirectly, e.g., traffic from the switch to another switch to the load balancing server. Similarly, traffic received by a switch from a load balancing server may be sent directly or indirectly, e.g., from the load balancing server to another switch and then to the switch.
In addition, one log collection server may only receive the mirror traffic sent by one switch, or may also receive the mirror traffic sent by multiple switches, and the load balancing server may only receive the service traffic sent by one switch, or may also receive the service traffic sent by multiple switches, or of course, the load balancing server may also send the service traffic to one or multiple switches. Under the condition that the log collection server comprises a plurality of log collection servers, the same preset rules can be set for different log collection servers, and different preset rules can also be set through flexible rule change. Through setting up different rules of predetermineeing, can realize the high-efficient screening of log data.
Illustratively, referring to fig. 3, a business process system includes: a plurality of switches 31 (e.g., two), a plurality of load balancing servers 32 (e.g., four), a plurality of log collection servers 33 (e.g., two), and a plurality of service processing servers 34 (e.g., two); the switches 31 may include an intranet switch and an extranet switch, the service traffic received by the extranet switch may be extranet traffic, and the service traffic received by the intranet switch 31 may be intranet traffic. The workflow of the system may include: the switch 31 receives the service traffic from the intranet or the extranet, performs mirroring operation on the received service traffic to obtain mirrored traffic, and sends the service traffic to one or more load balancing servers 32, and the corresponding load balancing server 32 distributes the service traffic to one or more service processing servers 34 for service processing. The exchanger 31 sends the mirror image traffic to one or more log collection servers 33 for log collection, and the log collection servers 33 send collected log data to the ELK cluster 35 for analysis, storage and display.
Of course, the workflow of the system may also include: the switch 31 receives the service traffic from the intranet, performs mirroring operation on the received service traffic to obtain mirrored traffic, and forwards the service traffic to the external device of the system. The exchanger 31 sends the mirror image traffic to one or more log collection servers 33 for log collection, and the log collection servers 33 send collected log data to the ELK cluster 35 for analysis, storage and display.
Based on the log output system, the embodiment of the invention also provides a log output method.
Referring to fig. 4, the journaling output method is applied to a switch in a journaling output system, the journaling output system including: the system comprises a switch, a load balancing server and a log acquisition server; the method may comprise the following three steps:
s401, receiving service flow, wherein the service flow comprises: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
s402, performing mirror image operation on the service flow to obtain mirror image flow;
s403, the mirror image flow is sent to a log collection server, so that the log collection server collects log data from the received mirror image flow, and sends the collected log data to an ELK cluster.
In the method, the steps performed by the switch are consistent with the actions to be performed by the switch in the log output system, and the relevant explanation can refer to the introduction of relevant parts in the log output system embodiment.
In the technical scheme, the log processing work which is originally taken charge of by the load balancing server is transferred to the log collecting server, so that the bearing pressure of the load balancing server is reduced, and the problems of packet loss or delay, loss of key logs and the like can be effectively prevented. Therefore, the operation performance of the log output system can be improved.
Further, the step of performing a mirror image operation on the service traffic to obtain a mirror image traffic includes:
and carrying out full-volume mirror image operation on the service flow to obtain mirror image flow. Of course, in other embodiments, the traffic may also be partially mirrored to obtain mirrored traffic. For the related explanation of the full-volume mirroring operation and the partial mirroring operation, reference may be made to the description of the related parts in the above-described log output system embodiment.
Based on the log output system, the embodiment of the invention also provides a log output method.
Referring to fig. 5, the log output method is applied to a log collection server in a log output system, and the log output system includes: the system comprises a switch, a load balancing server and a log acquisition server; the method may comprise the following two steps:
s501, receiving mirror image flow sent by a switch; the mirror flow is obtained by the switch performing mirror operation on the service flow received by the switch, and the service flow includes: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
it should be noted that, when the switch performs mirror operation on the received service traffic, the switch may perform full mirror operation on the received service traffic or may perform partial mirror operation. Specifically, what kind of method is adopted for mirroring is set according to the size of the received service traffic and the maximum traffic limit that can be carried by the switch, and the sum of the size of the service traffic and the size of the mirrored traffic cannot be greater than the maximum traffic limit that can be carried by the switch.
S502, collecting log data from the received mirror image flow, and sending the collected log data to an ELK cluster.
In the method, the steps performed by the log collection server are consistent with the actions to be executed by the log collection server in the log output system, and the related explanation can refer to the introduction of related contents in the log output system embodiment.
In the technical scheme, the log collection server is additionally arranged, log processing work which is originally taken charge of by the load balancing server is transferred to the log collection server, the bearing pressure of the load balancing server is reduced, and the problems of packet loss or delay, loss of key logs and the like can be effectively prevented. Therefore, the operation performance of the log output system can be improved.
In an implementation manner of the embodiment of the present invention, the step of collecting log data from the received mirror flow and sending the collected log data to the ELK cluster includes:
and recording log data from the received mirror image flow, compressing the recorded log data, and sending the compressed log data to the ELK cluster. The log collection server finishes recording and compressing actions, namely, the collection of logs is finished, and the storage space can be saved by the arrangement.
In another implementation manner of the embodiment of the present invention, the step of collecting log data from the received mirror flow and sending the collected log data to the ELK cluster includes:
and filtering and recording log data according to a preset rule from the received mirror image flow, compressing the recorded log data, and sending the compressed log data to the ELK cluster. The log collection server completes the actions of recording, filtering and compressing, namely, the collection of the log is completed. Recording and filtering may be performed simultaneously, with recording being selected after filtering. The setting not only saves the storage space, but also can realize better and more targeted display of the ELK cluster through the filtering operation, thereby being convenient for keeping the real-time performance of log data and quickly responding to the query requirement of the service.
The preset rule is set according to at least one of the following elements:
source IP, destination IP, source port, destination port, transport layer protocol.
The setting of the preset rule may refer to the related contents in the above-described log output system embodiment.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 61, a communication interface 62, a memory 63, and a communication bus 64, where the processor 61, the communication interface 62, and the memory 63 complete mutual communication through the communication bus 64,
a memory 63 for storing a computer program;
the processor 61 is configured to implement the following steps when executing the program stored in the memory 63:
(1) receiving service flow, wherein the service flow comprises: for sending to, and/or receiving from, a load balancing server;
(2) carrying out mirror image operation on the service flow to obtain mirror image flow;
(3) and sending the mirror image flow to a log collection server so that the log collection server collects log data from the received mirror image flow and sends the collected log data to the ELK cluster.
Or the following steps are realized:
(1) receiving mirror image flow sent by the switch; the mirror flow is obtained by the switch performing mirror operation on the service flow received by the switch, and the service flow includes: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
(2) and collecting log data from the received mirror image flow, and sending the collected log data to the ELK cluster.
In the technical scheme, the log collection server is additionally arranged, log processing work which is originally taken charge of by the load balancing server is transferred to the log collection server, the bearing pressure of the load balancing server is reduced, and the problems of packet loss or delay, loss of key logs and the like can be effectively prevented. Therefore, the operation performance of the log output system can be improved.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment provided by the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, which when run on a computer, cause the computer to perform the log output method of any one of the above embodiments.
In yet another embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the log output method of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium, and the computer program product embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
Claims (6)
1. A log output system, the system comprising: the system comprises a switch, a load balancing server and a log acquisition server;
the switch is used for receiving the service flow, carrying out mirror image operation on the received service flow to obtain mirror image flow and sending the mirror image flow to the log collection server; wherein the service traffic comprises: the switch is used for sending to the load balancing server and/or receiving from the load balancing server; the mirror image operation is a full mirror image operation; the traffic flow further includes first information for generating log data and second information for determining whether the traffic flow is denied or allowed; the log collection server is used for collecting log data from the received mirror flow and sending the collected log data to the ELK cluster;
the log collection server is specifically configured to filter and record log data according to a preset rule from the received mirror flow, compress the recorded log data, and send the compressed log data to the ELK cluster.
2. The system of claim 1, wherein the preset rule is set according to at least one of the following elements: source IP, destination IP, source port, destination port, transport layer protocol.
3. A log output method is applied to a switch in a log output system, and the log output system comprises: the system comprises a switch, a load balancing server and a log acquisition server; the method comprises the following steps:
receiving a service flow, wherein the service flow comprises: the switch is used for sending to the load balancing server and/or receiving from the load balancing server;
performing mirror image operation on the service flow to obtain mirror image flow; the mirror image operation is a full mirror image operation; the traffic flow further includes first information for generating log data and second information for determining whether the traffic flow is denied or allowed;
sending the mirror image flow to the log collection server so that the log collection server collects log data from the received mirror image flow and sends the collected log data to the ELK cluster; the log collection server is specifically configured to filter and record log data according to a preset rule from the received mirror flow, compress the recorded log data, and send the compressed log data to the ELK cluster.
4. A log output method is applied to a log collection server in a log output system, and the log output system comprises: the system comprises a switch, a load balancing server and a log acquisition server;
the method comprises the following steps:
receiving mirror image flow sent by the switch; the mirror flow is obtained by the switch performing mirror operation on the service flow received by the switch, and the service flow includes: the switch is used for sending to the load balancing server and/or receiving from the load balancing server; the traffic flow further includes first information for generating log data and second information for determining whether the traffic flow is denied or allowed;
collecting log data from the received mirror image flow, and sending the collected log data to an ELK cluster;
the step of collecting log data from the received mirror image traffic and sending the collected log data to the ELK cluster includes:
and filtering and recording log data according to a preset rule from the received mirror image flow, compressing the recorded log data, and sending the compressed log data to the ELK cluster.
5. The method of claim 4, wherein the preset rule is set according to at least one of the following elements: source IP, destination IP, source port, destination port, transport layer protocol.
6. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, implementing the method steps of any of claims 3-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810725475.5A CN108989101B (en) | 2018-07-04 | 2018-07-04 | Log output system and method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810725475.5A CN108989101B (en) | 2018-07-04 | 2018-07-04 | Log output system and method and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108989101A CN108989101A (en) | 2018-12-11 |
| CN108989101B true CN108989101B (en) | 2022-01-21 |
Family
ID=64536788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810725475.5A Active CN108989101B (en) | 2018-07-04 | 2018-07-04 | Log output system and method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108989101B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110336811A (en) * | 2019-06-29 | 2019-10-15 | 上海淇馥信息技术有限公司 | A kind of Cyberthreat analysis method, device and electronic equipment based on honey pot system |
| CN111343037B (en) * | 2019-08-19 | 2022-05-31 | 海通证券股份有限公司 | Flow monitoring method and device for cloud platform load according to application, and computer equipment |
| CN111176951A (en) * | 2019-12-31 | 2020-05-19 | 上海擎感智能科技有限公司 | Log output configuration/log processing method/system, medium, and server |
| CN114157458A (en) * | 2021-11-18 | 2022-03-08 | 深圳依时货拉拉科技有限公司 | Flow detection method, device, equipment and medium for hybrid cloud environment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102664789B (en) * | 2012-04-09 | 2016-08-17 | 北京百度网讯科技有限公司 | The processing method of a kind of large-scale data and system |
| CN104699592B (en) * | 2012-09-25 | 2018-09-04 | 北京奇虎科技有限公司 | A kind of method and system of daily record data transmission |
| CN104537120A (en) * | 2015-01-26 | 2015-04-22 | 浪潮通信信息***有限公司 | User behavior analysis based DNS data mining system and method |
| US10951489B2 (en) * | 2015-12-29 | 2021-03-16 | Digital River, Inc. | SLA compliance determination with real user monitoring |
| CN105786683B (en) * | 2016-03-03 | 2019-02-12 | 四川长虹电器股份有限公司 | Customed result collection system and method |
-
2018
- 2018-07-04 CN CN201810725475.5A patent/CN108989101B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108989101A (en) | 2018-12-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108989101B (en) | Log output system and method and electronic equipment | |
| WO2019227689A1 (en) | Data monitoring method and apparatus, and computer device and storage medium | |
| CN107943841A (en) | Stream data processing method, system and computer-readable recording medium | |
| CN110225104B (en) | Data acquisition method and device and terminal equipment | |
| CN105592123B (en) | Storage management system, management device and method | |
| CN105607986A (en) | Acquisition method and device of user behavior log data | |
| CN106027595A (en) | Access log processing method and system for CDN node | |
| JP2015508543A (en) | Processing store visit data | |
| CN107181821A (en) | A kind of information push method and device based on SSE specifications | |
| US20230017300A1 (en) | Query method and device suitable for olap query engine | |
| CN110928934A (en) | Data processing method and device for business analysis | |
| US11038803B2 (en) | Correlating network level and application level traffic | |
| CN107977418A (en) | A kind of screenshot picture management method, screenshot picture managing device and mobile terminal | |
| US8914517B1 (en) | Method and system for predictive load balancing | |
| CN111198885A (en) | Data processing method and device | |
| EP3481099B1 (en) | Load balancing method and associated device | |
| CN109981697B (en) | File unloading method, system, server and storage medium | |
| CN114285786B (en) | Construction method and device of network link library | |
| KR20150139546A (en) | Removable storage device identity and configuration information | |
| US20220303360A1 (en) | Reduction of data transmissions based on end-user context | |
| CN110311868B (en) | Service processing method, device, member equipment and machine-readable storage medium | |
| KR102093764B1 (en) | Managment server for managing the server and storage | |
| CN111124365A (en) | RPA demand collection method and device | |
| CN106664223A (en) | Detection method and detection device for the number of shared access hosts | |
| CN112527787B (en) | Safe and reliable multiparty data deduplication system, method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |