CN112231191A - Log collection method and device - Google Patents
Log collection method and device Download PDFInfo
- Publication number
- CN112231191A CN112231191A CN202011147788.0A CN202011147788A CN112231191A CN 112231191 A CN112231191 A CN 112231191A CN 202011147788 A CN202011147788 A CN 202011147788A CN 112231191 A CN112231191 A CN 112231191A
- Authority
- CN
- China
- Prior art keywords
- log
- node
- target
- identifier
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present application relates to the field of network security technologies, and in particular, to a log collection method and apparatus. The log collection method is applied to log collection equipment comprising a log accepting and rejecting model, wherein the log accepting and rejecting model comprises an annular linked list consisting of a plurality of nodes, each node is provided with a corresponding state identifier, if the state identifier of one node is a first identifier, a log received based on the one node is reserved, and if the state identifier of one node is a second identifier, the log received based on the one node is discarded, and the method comprises the following steps: updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment; receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model; if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool; otherwise, the target log is discarded.
Description
Technical Field
The present application relates to the field of network security technologies, and in particular, to a log collection method and apparatus.
Background
With the increasingly deep informatization, the internet is becoming a key information infrastructure of the country, and the internet security is related to the fundamental interests of the country and the society. The network security situation perception technology can synthesize security factors of all aspects, dynamically reflect network security conditions on the whole, predict and early warn development trend of the security conditions, and provide reliable reference basis for enhancing network security. Therefore, security situation awareness research on networks has become a research hotspot in the field of network security at present.
The acquisition and normalization processing of various device logs are the basis of network security situation awareness, and the log acquisition mode is generally divided into two modes, namely passive log acquisition and active log acquisition, wherein the passive log acquisition refers to that a network security situation awareness platform serves as a server to passively receive logs sent by a device. In passive log collection, the network security situation awareness platform cannot control log sending of the device, and a scene that the device continuously sends a large amount of logs may exist. If the sending rate of the logs is higher than the log analysis and processing performance of the situation awareness platform, the abnormality of the situation awareness platform can be caused. Therefore, for receiving the passive log, the network security situation awareness platform needs to make a log receiving speed-limiting strategy to ensure the stable operation of the network security situation awareness platform.
Under the premise, the log collection equipment can pre-cache the received logs into a cache pool, and after the number of the logs in the cache pool reaches a set threshold value, the logs are sent to the next link in batches for processing. And combining the log speed limit requirement, when the cache pool is full, the log is not received any more until the batch of logs is processed and written into a batch of logs, and the logs collected during the process are directly discarded. Then, the discarded logs may be continuous, and the analysis result of the subsequent network security situation awareness platform will be affected.
Disclosure of Invention
The application provides a log collection method and device, which are used for solving the problem that in the prior art, the analysis result of a network security situation perception platform is inaccurate due to the fact that continuous logs are discarded.
In a first aspect, the present application provides a log collection method applied to a log collection device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and a corresponding state identifier is set for each node in the annular linked list, where if a state identifier of a node is a first identifier, a log received based on the node is retained, and if a state identifier of a node is a second identifier, a log received based on the node is discarded, and the method includes:
updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool;
and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, the step of updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device includes:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, the step of determining, based on the updated log accepting/rejecting model, a state identifier of a target node corresponding to the target log includes:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
In a second aspect, the present application provides a log collecting device, which is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and is directed to each node in the annular linked list is provided with a corresponding state identifier, where if the state identifier of a node is a first identifier, the log received by the node is retained, and if the state identifier of a node is a second identifier, the log received by the node is discarded, and the log collecting device includes:
the updating unit is used for updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
the judging unit is used for receiving a target log sent by network equipment in the current period and judging the state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
the cache unit caches the target log to a cache pool if the judging unit judges that the state identifier of the target node is a first identifier;
and the discarding unit discards the target log if the judging unit judges that the state identifier of the target node is the second identifier.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, when the log accepting and rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, when the state identifier of the target node corresponding to the target log is determined based on the updated log accepting and rejecting model, the determining unit is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
In a third aspect, an embodiment of the present application provides a log collecting device, where the log collecting device includes:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the steps of the method according to any one of the above first aspects in accordance with the obtained program instructions.
In a fourth aspect, the present application further provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps of the method according to any one of the above first aspects.
To sum up, the log collection method provided by the embodiment of the present application is applied to a log collection device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and is directed to each node in the annular linked list is provided with a corresponding state identifier, where if the state identifier of a node is a first identifier, the log received by the node is retained, and if the state identifier of a node is a second identifier, the log received by the node is discarded, and the method includes: updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment; receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model; if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool; and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
By adopting the log collection method provided by the embodiment of the application, the log discarding strategy of the next time period is determined according to the log quantity of the previous time period, the dynamic speed limit of the log is realized, the scene requirements that the actual log quantity does not discard the log in the performance specification of the network security situation awareness platform, more logs are discarded when the actual log quantity exceeds the performance specification of the network security situation awareness platform are met, and the like are met, the log is directly filtered at the log receiving end, and the data impact damage during the operation of the network security situation awareness platform in the environment exceeding the performance specification of the network security situation awareness platform is effectively avoided. The logs of corresponding entries are discarded by adopting the state identifiers of all the nodes in the annular linked list, the condition of continuously discarding continuous logs for a long time is avoided, and the accuracy of the analysis result of the network security situation perception platform is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.
Fig. 1 is a detailed flowchart of a log collection method according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of an annular linked list provided in the embodiment of the present application;
fig. 3 is a detailed flowchart of another log collection method according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a log collection device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another log collection device according to an embodiment of the present application.
Detailed Description
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
Exemplarily, referring to fig. 1, a detailed flowchart of a log collecting method provided in an embodiment of the present application is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes a ring-shaped linked list composed of a plurality of nodes, and a corresponding state identifier is set for each node in the ring-shaped linked list, where if a state identifier of one node is a first identifier, a log received based on the one node is retained, and if a state identifier of one node is a second identifier, a log received based on the one node is discarded, where the method includes the following steps:
step 100: and updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log acquisition equipment.
In practical application, the log collection device receives logs reported by each network device in the network, maintains a log accepting and rejecting model at a log receiving inlet, confirms that each log reported by each network device is discarded or reserved when passing through the log accepting and rejecting model, does not perform any processing on the log when the log is discarded, and stores the log into a cache to wait for the processing of the next link when the log is reserved.
Correspondingly, when the log collection device is started, parameters of a log selection model included in the log collection device can be pre-configured based on a specific application scenario and/or user requirements. Specifically, the log processing rate of the log processing device, the period of updating the log accepting model, and the like may be included, and in this embodiment, the log processing rate may be understood as the maximum log processing capacity of the log processing device.
In this embodiment of the present application, when a cycle is finished, an operation of updating a log accepting and rejecting model needs to be started, specifically, in a first case, if the current cycle is a first cycle, state identifiers of nodes in the ring-shaped chain table in the log accepting and rejecting model are all set as first identifiers.
That is to say, when the log selection and deletion model is initialized, since there is no previous cycle, in the embodiment of the present application, the state identifiers of the nodes in the above-mentioned circular linked list in the log selection and deletion model are all set as the first identifiers, that is, all logs received in the current cycle are cached in the cache pool.
In the second case, if the current period is not the first period, when the log accepting/rejecting model is updated based on the counted log reporting rate of the previous period and the log processing rate of the log processing device, a preferable implementation manner is to calculate the log reporting rate of the network device in the previous period based on the total number of logs received in the previous period and the period duration; determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment; calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs; and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
That is, based on the total number of the log pieces received in the previous period and the number of the log pieces that can be processed in the previous period by the log processing device, the ratio between the number of the log pieces expected to be kept and the number of the log pieces expected to be discarded in the next period is determined, and the state identifier of each node in the circular linked list in the log selection and deletion model is adjusted and updated according to the ratio.
Preferably, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
For example, assuming that the duration of the preset period is 2 seconds (S), the log processing rate of the preset log processing device is 4000 pieces/second, and the total number of the log pieces received in the previous period is 20000 pieces, then the ratio of the number of the log expected to be kept and the number of the log discarded in the previous period can be calculated according to the above parameters, where the ratio is: in this embodiment, the state identifier of each node in the updated circular linked list may be calculated according to the ratio of the log expected to be kept to the number of discarded logs calculated in the previous period and the total node data (10) of the circular linked list in the log selection model. Illustratively, referring to fig. 2, for the structural diagram of a circular linked list provided in the embodiment of the present application, assuming that the total number of nodes in the circular linked list is 10, the next node of node 1 is node 2, the next node of node 2 is node 3, … …, the next node of node 9 is node 10, and the next node of node 10 is node 1, if the calculated ratio of the number of logs expected to be kept and the number of discarded logs in the previous cycle is (2:3), when updating the log selection model, a preferred implementation manner is to set the state identifiers of node 1 and node 2 as a first identifier (e.g., true), set the state identifiers of node 3 to node 5 as a second identifier (e.g., false), set the state identifiers of node 6 and node 7 as the first identifier, set the state identifiers of node 8 to node 10 as the second identifier, namely, the node which comprises 4 state identifiers and 6 state identifiers and is the first identifier comprises the node which comprises the first identifier and the second identifier.
It should be noted that the preset period may be set in a self-defined manner according to different application scenarios and/or different user requirements, for example, set to be 1 second, 2 seconds, 5 seconds, and the like.
In this embodiment of the application, it is assumed that the circular linked list of the log selection and deletion model includes n nodes, and one node corresponds to one log, for example, when receiving log 1, it determines to keep log 1 or discard log 1 based on the state identifier of node 1 corresponding to log 1, when receiving log 2, it determines to keep log 2 or discard log 2 based on the state identifier of node 2 corresponding to log 2, … …, when receiving log n, it determines to keep log n or discard log n based on the state identifier of node n corresponding to log n, when receiving log n +1, because the next node of node n is node 1, it determines to keep log n +1 or discard log n +1, … … based on the state identifier of node 1 corresponding to log n + 1.
Step 110: and receiving a target log sent by the network equipment in the current period, and judging the state identifier of a target node corresponding to the target log based on the updated log selection model.
In this embodiment of the application, when determining the state identifier of the target node corresponding to the target log based on the updated log selection model, a preferred implementation manner is to determine the state identifier of the target node pointed by the pointer after receiving the target log, process the target log based on the state identifier of the target node, and point the pointer to a node next to the target node in the circular linked list.
It should be noted that, in the embodiment of the present application, the purpose of using the circular linked list is that after each log comes, it can be determined whether the log is reserved only according to the state identifier of the node to which the pointer currently points, and pointer operation in the computer greatly reduces the decision performance loss. When a log is received, if the state identifier of the node currently pointed by the pointer is a first identifier, the log is cached to a cache for subsequent processing, if the state identifier of the node currently pointed by the pointer is a second identifier, the log is directly discarded, and then the pointer is moved to the next node.
Of course, in this embodiment of the present application, when a log is received, the pointer may be moved first, and then the node state identifier is determined, that is, when a log is received, the pointer is moved to a next node of the currently-pointed node first, then the log is determined to be cached/discarded according to the state identifier of the next node, after the log is processed, the pointer is not moved for the moment, and only when the next log is received, the pointer is moved again, and then the node state identifier is determined.
Step 120: and if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool.
Step 130: and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
The following describes the log collection method provided in the embodiment of the present application in detail with reference to a specific application scenario. Illustratively, referring to fig. 3, a detailed flowchart of another log collection method provided in an embodiment of the present application is shown, where the method includes the following steps: the method comprises the steps that a network device/a safety device/a host and the like in a networking actively report logs to a log collection device, when the log collection device receives any log, whether the log is reserved or not is judged based on a log accepting and rejecting model, if so, the log is cached in a cache pool, and if not, the log is discarded; further, the log accepting and rejecting model updates the log accepting and rejecting model based on a preset period and the total number of logs reported by the network device, the security device, the host and the like received in the previous period, for example, when the current period is determined to be finished, the total number of logs received in the previous period is obtained, the log accepting and rejecting model is updated based on the total number of logs and the log processing capacity of the log processing device, and in the next period, whether each received log is reserved is determined by using the updated log accepting and rejecting model.
Exemplarily, referring to fig. 4, a schematic structural diagram of a log collecting device provided in an embodiment of the present application is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and a corresponding state identifier is set for each node in the annular linked list, where if a state identifier of one node is a first identifier, a log received based on the one node is retained, and if a state identifier of one node is a second identifier, a log received based on the one node is discarded, where the log collecting device includes:
the updating unit 40 is configured to update the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device;
a determining unit 41, configured to receive a target log sent by a network device in a current period, and determine, based on the updated log accepting/rejecting model, a state identifier of a target node corresponding to the target log;
a caching unit 42, if the determining unit 41 determines that the status identifier of the target node is the first identifier, the caching unit 42 caches the target log in a cache pool;
a discarding unit 43, wherein if the determining unit 41 determines that the status flag of the target node is the second flag, the discarding unit 43 discards the target log.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, when the log accepting and rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit 40 is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, when the state identifier of the target node corresponding to the target log is determined based on the updated log accepting/rejecting model, the determining unit 41 is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
The above units may be one or more integrated circuits configured to implement the above methods, for example: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above units is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Further, in the log collecting device provided in the embodiment of the present application, from a hardware aspect, a schematic diagram of a hardware architecture of the log collecting device may be shown in fig. 5, where the log collecting device may include: a memory 50 and a processor 51, which,
the memory 50 is used for storing program instructions; the processor 51 calls the program instructions stored in the memory 50 and executes the above-described method embodiments according to the obtained program instructions. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present application further provides a log collecting device, which includes at least one processing element (or chip) for executing the above method embodiments.
Optionally, the present application also provides a program product, such as a computer-readable storage medium, having stored thereon computer-executable instructions for causing the computer to perform the above-described method embodiments.
Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and so forth. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (10)
1. A log collection method is applied to log collection equipment comprising a log accepting and rejecting model, wherein the log accepting and rejecting model comprises an annular linked list consisting of a plurality of nodes, and a corresponding state identifier is set for each node in the annular linked list, if the state identifier of one node is a first identifier, a log received based on the one node is reserved, and if the state identifier of one node is a second identifier, the log received based on the one node is discarded, and the method comprises the following steps:
updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool;
and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
2. The method of claim 1, wherein if the current cycle is a first cycle, the state flags of the nodes in the circular linked list in the log rounding model are all set to a first flag.
3. The method of claim 1, wherein if the current period is not the first period, the step of updating the log rounding model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device comprises:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
4. The method of claim 3, wherein nodes with state identities of a first identity and nodes with state identities of a second identity are evenly distributed in the updated circular linked list.
5. The method of claim 1, wherein the step of determining the status identifier of the target node corresponding to the target log based on the updated log pruning model comprises:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
6. The utility model provides a log collection system which characterized in that is applied to the log collection equipment who includes log accepting model, log accepting model includes the cyclic annular linked list of constituteing by a plurality of nodes, is provided with corresponding state sign to each node in the cyclic annular linked list, wherein, if the state sign of a node is first sign, then keep the log that is received based on a node, if the state sign of a node is the second sign, then abandon the log that is received based on a node, the device includes:
the updating unit is used for updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
the judging unit is used for receiving a target log sent by network equipment in the current period and judging the state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
the cache unit caches the target log to a cache pool if the judging unit judges that the state identifier of the target node is a first identifier;
and the discarding unit discards the target log if the judging unit judges that the state identifier of the target node is the second identifier.
7. The apparatus of claim 6, wherein the state flags of the nodes in the circular linked list in the log rounding model are all set to a first flag if the current cycle is a first cycle.
8. The apparatus of claim 6, wherein if the current period is not the first period, when the log accepting and rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
9. The apparatus of claim 8, wherein nodes with state identities of a first identity and nodes with state identities of a second identity are evenly distributed in the updated circular linked list.
10. The apparatus according to claim 6, wherein when determining the status identifier of the target node corresponding to the target log based on the updated log-culling model, the determining unit is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011147788.0A CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011147788.0A CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112231191A true CN112231191A (en) | 2021-01-15 |
CN112231191B CN112231191B (en) | 2023-03-31 |
Family
ID=74110771
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011147788.0A Active CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112231191B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113282920A (en) * | 2021-05-28 | 2021-08-20 | 平安科技(深圳)有限公司 | Log abnormity detection method and device, computer equipment and storage medium |
CN116346729A (en) * | 2023-02-24 | 2023-06-27 | 安芯网盾(北京)科技有限公司 | Data log reporting current limiting method and system |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101557291A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Method for log aggregation and device thereof |
CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
CN103795577A (en) * | 2014-03-03 | 2014-05-14 | 网神信息技术(北京)股份有限公司 | Log processing method and device of log server |
US20140160952A1 (en) * | 2012-12-11 | 2014-06-12 | International Business Machines Corporation | Detecting and isolating dropped or out-of-order packets in communication networks |
CN105138606A (en) * | 2015-08-03 | 2015-12-09 | 上海斐讯数据通信技术有限公司 | Server log management method and system |
CN108471387A (en) * | 2018-03-27 | 2018-08-31 | 中国农业银行股份有限公司 | A kind of daily record flow decentralized control method and system |
CN108616556A (en) * | 2016-12-13 | 2018-10-02 | 阿里巴巴集团控股有限公司 | Data processing method, device and system |
CN109344034A (en) * | 2018-09-29 | 2019-02-15 | 郑州云海信息技术有限公司 | A kind of method and apparatus for managing log |
CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
-
2020
- 2020-10-23 CN CN202011147788.0A patent/CN112231191B/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101557291A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Method for log aggregation and device thereof |
CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
US20140160952A1 (en) * | 2012-12-11 | 2014-06-12 | International Business Machines Corporation | Detecting and isolating dropped or out-of-order packets in communication networks |
CN103795577A (en) * | 2014-03-03 | 2014-05-14 | 网神信息技术(北京)股份有限公司 | Log processing method and device of log server |
CN105138606A (en) * | 2015-08-03 | 2015-12-09 | 上海斐讯数据通信技术有限公司 | Server log management method and system |
CN108616556A (en) * | 2016-12-13 | 2018-10-02 | 阿里巴巴集团控股有限公司 | Data processing method, device and system |
CN108471387A (en) * | 2018-03-27 | 2018-08-31 | 中国农业银行股份有限公司 | A kind of daily record flow decentralized control method and system |
CN109344034A (en) * | 2018-09-29 | 2019-02-15 | 郑州云海信息技术有限公司 | A kind of method and apparatus for managing log |
CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113282920A (en) * | 2021-05-28 | 2021-08-20 | 平安科技(深圳)有限公司 | Log abnormity detection method and device, computer equipment and storage medium |
CN113282920B (en) * | 2021-05-28 | 2023-10-10 | 平安科技(深圳)有限公司 | Log abnormality detection method, device, computer equipment and storage medium |
CN116346729A (en) * | 2023-02-24 | 2023-06-27 | 安芯网盾(北京)科技有限公司 | Data log reporting current limiting method and system |
CN116346729B (en) * | 2023-02-24 | 2024-02-09 | 安芯网盾(北京)科技有限公司 | Data log reporting current limiting method and system |
Also Published As
Publication number | Publication date |
---|---|
CN112231191B (en) | 2023-03-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110109953B (en) | Data query method, device and equipment | |
CN112231191B (en) | Log collection method and device | |
CN108429800A (en) | A kind of mobile device | |
CN112910802B (en) | Message processing method and device | |
CN113328944B (en) | Flow table management method and device | |
CN109815298B (en) | Method and device for determining character relationship network and storage medium | |
CN110889132A (en) | Distributed application permission verification method and device | |
CN113486037A (en) | Cache data updating method, manager and cache server | |
CN112162912A (en) | Cloud resource monitoring method and system | |
CN108399175B (en) | Data storage and query method and device | |
CN105262634A (en) | Monitoring threshold generation method, device and system | |
EP3785414B1 (en) | Detecting unauthorized cloud access by detecting malicious velocity incidents | |
CN116303418A (en) | Mass rule processing method, system and storage medium based on Internet of things platform | |
CN114585035B (en) | Voice call method, device and computer readable storage medium | |
CN112612804B (en) | Service management parameter updating method and device | |
CN116074332A (en) | Mirror image management method, distributed system, electronic equipment and medium | |
CN114039893A (en) | API gateway speed limiting method and device | |
CN113608856A (en) | Node scheduling method, system and device | |
CN112615971A (en) | Image transmission method and device | |
CN112988463B (en) | Fault node isolation method and device | |
CN116132528B (en) | Flight management message pushing method and device and electronic equipment | |
CN113055419B (en) | Information sending method and device | |
CN113810864B (en) | Regional short message targeted group sending method and system | |
CN116346729B (en) | Data log reporting current limiting method and system | |
CN114006731B (en) | Network attack processing method, device, equipment and machine-readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |