CN113179176A - Log processing method, device and equipment and machine readable storage medium - Google Patents
Log processing method, device and equipment and machine readable storage medium Download PDFInfo
- Publication number
- CN113179176A CN113179176A CN202110346638.0A CN202110346638A CN113179176A CN 113179176 A CN113179176 A CN 113179176A CN 202110346638 A CN202110346638 A CN 202110346638A CN 113179176 A CN113179176 A CN 113179176A
- Authority
- CN
- China
- Prior art keywords
- information
- log information
- matching module
- log
- matching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present disclosure provides a log processing method, apparatus, device and machine-readable storage medium, the method comprising: receiving log information, and acquiring special adaptive flag bit information of the log information; selecting an applicable matching module according to the special adaptation zone bit information of the log information; matching the regular expression associated with the log information by using the selected matching module; and carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information. According to the technical scheme, different matching modules are customized in advance, corresponding matching modules are selected according to different types shown by the received log information by using the special adaptation marks, and each piece of log information does not need to be matched with the regular expression in a traversing collision mode, so that the performance overhead of the computer is reduced.
Description
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a log processing method, apparatus, device, and machine-readable storage medium.
Background
With the rapid development of big data technology, mass log data are often used for service analysis and display in various service scenes. The massive log data analysis model mainly comprises the following three parts: log collection, heterogeneous log normalization (log analysis) and specific service analysis. Unstructured data has great difficulty in analyzing and displaying services, so that heterogeneous log normalization is an essential link for solving the problem.
The most common log analysis scheme at present is a model based on regular collision, and the basic idea is that a piece of heterogeneous data is bound to a certain data source, then a regular expression bound with the data source is used for traversing collision matching, if the data source is matched, the regular expression is used for analyzing, the extracted value is searched in full, and the extracted value is mapped to a specified value. If not, the data is discarded or stored separately.
The above scheme consumes a large amount of computer performance.
Disclosure of Invention
In view of the above, the present disclosure provides a log processing method, a log processing apparatus, an electronic device, and a machine-readable storage medium to solve the problem of high performance consumption of the computer.
The specific technical scheme is as follows:
the present disclosure provides a log processing method applied to a computer device, the method including: receiving log information, and acquiring special adaptive flag bit information of the log information; selecting an applicable matching module according to the special adaptation zone bit information of the log information; matching the regular expression associated with the log information by using the selected matching module; and carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a general regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value as a first preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: matching the log heads of the log information by using a head regular pattern of a regular expression included in a rule base of a universal regular matching module, and matching the log content of the log information by using a body regular pattern associated with the matched head regular pattern of the log information with the matched head regular pattern; the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a collision regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value of a second preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a customized matching module as an applicable matching module according to the special adaptation zone bit information of the log information with the value of a third preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
The present disclosure also provides a log processing apparatus applied to a computer device, the apparatus including: the log source adding module is used for receiving log information and acquiring special adaptive flag bit information of the log information; the matching path selection module is used for selecting an applicable matching module according to the special adaptive flag bit information of the log information; the matching module is used for matching the regular expression associated with the log information by using the selected matching module; and the value normalization module is used for carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a general regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value as a first preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: matching the log heads of the log information by using a head regular pattern of a regular expression included in a rule base of a universal regular matching module, and matching the log content of the log information by using a body regular pattern associated with the matched head regular pattern of the log information with the matched head regular pattern; the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a collision regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value of a second preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
As a technical solution, selecting an applicable matching module according to the special adaptation flag bit information of the log information, including: selecting a customized matching module as an applicable matching module according to the special adaptation zone bit information of the log information with the value of a third preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
The present disclosure also provides an electronic device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions capable of being executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned log processing method.
The present disclosure also provides a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned log processing method.
The technical scheme provided by the disclosure at least brings the following beneficial effects:
different matching modules are customized in advance, corresponding matching modules are selected according to different types shown by the received log information by using special adaptation marks, and each piece of log information does not need to be matched with a regular expression in a traversal collision mode, so that the performance overhead of the computer is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present disclosure or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present disclosure, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present disclosure.
FIG. 1 is a flow diagram of a log processing method in one embodiment of the present disclosure;
FIG. 2 is a block diagram of a log processing apparatus according to an embodiment of the present disclosure;
fig. 3 is a hardware configuration diagram of an electronic device in an embodiment of the present disclosure.
Detailed Description
The terminology used in the embodiments of the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information in the embodiments of the present disclosure, such information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
The most common log analysis scheme at present is a model based on regular collision, and the basic idea is that a piece of heterogeneous data is bound to a certain data source, then a regular expression bound with the data source is used for traversing collision matching, if the data source is matched, the regular expression is used for analyzing, the extracted value is searched in full, and the extracted value is mapped to a specified value. If not, the data is discarded or stored separately.
The log regular bound by each log source grows in a thread mode along with the types of the logs concerned, and if the logs concerned have N types, each log needs to collide with N regular expressions during matching, wherein the best condition is O (1), and the worst condition is O (N). Generally, the amount of streamed log data M will be relatively large, and theoretically, the overall matching speed of a log source is preferably O (M), and the worst speed is O (M × N). Similarly, the value extracted after the regular matching needs to be searched in full amount for mapping, and the consumption performance is high.
In view of the above, the present disclosure provides a log processing method, a log processing apparatus, an electronic device, and a machine-readable storage medium to solve the problem of high performance consumption of the computer.
Specifically, the technical scheme is as follows.
In one embodiment, the present disclosure provides a log processing method applied to a computer device, the method including: receiving log information, and acquiring special adaptive flag bit information of the log information; selecting an applicable matching module according to the special adaptation zone bit information of the log information; matching the regular expression associated with the log information by using the selected matching module; and carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
Specifically, as shown in fig. 1, the method comprises the following steps:
and step S11, receiving the log information, and acquiring the special adaptation zone bit information of the log information.
And step S12, selecting an applicable matching module according to the special adaptive flag bit information of the log information.
And step S13, matching the regular expression associated with the log information by using the selected matching module.
And step S14, carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
Different matching modules are customized in advance, corresponding matching modules are selected according to different types shown by received log information by special adaptation marks, and each piece of log information does not need to be matched with a regular expression in a traversal collision mode, so that the performance overhead of a computer is reduced
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting a general regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value as a first preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: matching the log heads of the log information by using a head regular pattern of a regular expression included in a rule base of a universal regular matching module, and matching the log content of the log information by using a body regular pattern associated with the matched head regular pattern of the log information with the matched head regular pattern; the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting a collision regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value of a second preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting a customized matching module as an applicable matching module according to the special adaptation zone bit information of the log information with the value of a third preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
In this step, the unique log source information in the system can be generated through the log source adding module. The log source adding module forms unique log source information by a log source IP and a log source sending log PORT (PORT), informs a system collector to collect logs reported by the corresponding IP and the PORT, and simultaneously stores the reported logs in a cache with the IP: PORT as a key value. The unique log source binds manufacturer, type, model, software version number and special adaptation flag bit information, and the information can accurately and quickly determine which set of rules the log source will use for next adaptation in a plurality of sets of adaptation rules built in the system.
If the same log source reports two completely different logs, different unique log sources are formed through different port numbers when the log source is added, and then the log sources are bound with corresponding equipment information, so that another independent set of adaptation regular patterns is found. For example: the Hua-three prevention firewall sends logs to a collector through 182.9.0.1 and a port 514, and when a log source is added, 182.9.0.1:514 and device information built in the system are as follows: H3C/Firewall/M9000/comwareV 7/0 binding, so that the rules for resolving adaptations used by the logs reported by 182.9.0.1:514 this log source can be validated.
In one embodiment, the information of the special adaptation flag bit is normally empty, and when the flag bit is assigned, the corresponding matching module can be directly accessed.
In this step, an applicable matching module can be selected through the matching path selection module according to the special adaptation flag bit information of the log information, for example, the adaptation model has three types of collision regularization, universal regularization and customized matching, and the selection is made through the matching path module.
The log source information can determine which set of adaptation rules to use, and can also determine the adaptation module according to the special adaptation flag bit. In one embodiment, the default condition is set to indicate that the flag is empty or the rule is specified as 0 when the rule is predefined, in which case the adaptation rule uses a general regular matching module; the rule of 1 is assigned using the collision regular matching module and the other strings represent using the custom adaptation module entries.
In one embodiment, this may be accomplished using a collision-regularized adaptation module. When a log enters the adaptation module, the log traverses regular expressions in the adaptation rule in sequence and tries, once the trial is hit, the regular expressions are used for carrying out information extraction on the log and storing the information in corresponding fields, and then all mapping relations are searched for carrying out value mapping, which is consistent with the existing scheme. If the log reported by the equipment is log, the equipment has three regular patterns of reg1, reg2 and reg3, three regular matching logs are respectively used during analysis, and once the regular matching logs are matched, other regular patterns are stopped from being tried.
In one embodiment, a generic canonical matching module may be used for implementation. The module is established on the basis of a universal regular matching module adaptation rule base. The regular expression in the rule base corresponds to the log header and the log content respectively according to a header regular part and a body regular part, only one header regular part of a set of rules is needed, at least one body regular part is needed, at least one identification name information is bound to each body regular part, and two identification name information bits can be defaulted or dynamically expanded according to services. If the log content is matched with a certain log content, the first identification name and the second identification name can be bound when the body is regular, and simultaneously, the log type of each body is also bound, for example, the type is a community audit log. The rule base also contains a field mapping table and a value mapping table. The field mapping table mainly comprises four types of information of a pre-mapping field, a post-mapping field, an affiliated rule id and a log type, and the value mapping table mainly comprises five types of information of a mapping field, a pre-mapping value, a post-mapping value, an affiliated rule id and a log type.
The specific matching is as follows.
And performing header regular matching on the logs and extracting identification name fields, and performing body regular matching on the log contents if the matching is successful, and performing log classification and field extraction. The corresponding body regular pattern needs to be quickly positioned in the multiple body regular patterns, according to the identification name field extracted by the first step head regular pattern, when the body regular pattern is searched, the corresponding body regular pattern in the rule base is directly positioned by using the combination of the first identification name and the second identification name extracted from the log head, if the corresponding body regular pattern is not found, the first identification name is used for direct positioning search, and if the corresponding body regular pattern is found, the field extraction mapping is carried out. Specifically, fields needing mapping are searched according to the rule id, log classification and the fields before mapping, and the contents of the fields before mapping are assigned to the fields after mapping.
In one embodiment, a customized matching module implementation may be used. The first two adaptation modules can be covered in most cases, but in some specific service scenarios, the adaptation must be performed using customized codes. The module adopts a form of registering plug-ins, realizes that customized plug-ins are registered in the resolver, records the special adaptation identifiers as corresponding plug-in names when the log source information is appointed, dynamically loads and enters a plug-in resolving process when the resolving process reads the special adaptation identifiers, and performs customized resolving according to specific services. For example, a certain device ABC must be implemented by using a customized parser, then a plug-in XXX is written in a non-interpreted language and registered in the parser plug-in when customizing, and the log source is designated as a deep belief ABC _ XXX when designating the log source, wherein XXX is used as a customized identifier, and the XXX plug-in is automatically called to parse when the plug-in detects.
In this step, induction processing may be performed by the value normalization module, for example, the extracted field names are normalized, and part of the extracted values are normalized, so that subsequent service processing is facilitated after dimensions are unified. And classifying the logs which cannot be matched with the regular expression into system log storage, not performing field extraction, and replacing the logs with discarded logs according to a specific scene. And extracting fields and carrying out value normalization on the log information matched with the regular expression. The step is to normalize the values in the designated fields to a uniform standard, which facilitates subsequent business analysis. The field normalization is based on a value normalization table, which proceeds as follows: and searching a value to be mapped according to the field name, the rule id, the log classification and the value before mapping, and assigning the mapped content to the field.
In an embodiment, the present disclosure also provides a log processing apparatus, as shown in fig. 2, applied to a computer device, the apparatus including: the log source adding module 21 is configured to receive log information and obtain special adaptation flag bit information of the log information; a matching path selection module 22, configured to select an applicable matching module according to the special adaptation flag bit information of the log information; a matching module 23, configured to match the regular expression associated with the log information by using a selected matching module; and the value normalization module 24 is configured to perform value induction processing on the log information according to a result of matching the regular expression associated with the log information.
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting the universal regular matching module 232 as an applicable matching module according to the special adaptation zone bit information of the log information with the value of the first preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: matching the log heads of the log information by using a head regular pattern of a regular expression included in a rule base of a universal regular matching module, and matching the log content of the log information by using a body regular pattern associated with the matched head regular pattern of the log information with the matched head regular pattern; the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting a collision regular matching module 231 as an applicable matching module according to the special adaptation flag bit information of the log information with the value of the second preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
In one embodiment, selecting the applicable matching module according to the special adaptation flag bit information of the log information includes: selecting the customized matching module 233 as an applicable matching module according to the special adaptation flag bit information of the log information whose value is the third preset target value; the matching the regular expression associated with the log information by using the selected matching module comprises: analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
The device embodiments are the same or similar to the corresponding method embodiments and are not described herein again.
In an embodiment, the present disclosure provides an electronic device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions capable of being executed by the processor, and the processor executes the machine-executable instructions to implement the foregoing log processing method, and from a hardware level, a hardware architecture diagram may be as shown in fig. 3.
In one embodiment, the present disclosure provides a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the aforementioned log processing method.
Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and so forth. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The systems, devices, modules or units described in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the various elements may be implemented in the same one or more software and/or hardware implementations in practicing the disclosure.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (which may include, but is not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an embodiment of the present disclosure, and is not intended to limit the present disclosure. Various modifications and variations of this disclosure will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure should be included in the scope of the claims of the present disclosure.
Claims (10)
1. A log processing method applied to a computer device, the method comprising:
receiving log information, and acquiring special adaptive flag bit information of the log information;
selecting an applicable matching module according to the special adaptation zone bit information of the log information;
matching the regular expression associated with the log information by using the selected matching module;
and carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
2. The method of claim 1,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a general regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value as a first preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
matching the log header of the log information using a header regular of a regular expression included in a rule base of a generic regular matching module,
for the log information with the matched head regular pattern, matching the log content of the log information by using the body regular pattern associated with the matched head regular pattern;
the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
3. The method of claim 1,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a collision regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value of a second preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
4. The method of claim 1,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a customized matching module as an applicable matching module according to the special adaptation zone bit information of the log information with the value of a third preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
5. A log processing apparatus applied to a computer device, the apparatus comprising:
the log source adding module is used for receiving log information and acquiring special adaptive flag bit information of the log information;
the matching path selection module is used for selecting an applicable matching module according to the special adaptive flag bit information of the log information;
the matching module is used for matching the regular expression associated with the log information by using the selected matching module;
and the value normalization module is used for carrying out value induction processing on the log information according to the result of matching the regular expression associated with the log information.
6. The apparatus of claim 5,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a general regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value as a first preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
matching the log header of the log information using a header regular of a regular expression included in a rule base of a generic regular matching module,
for the log information with the matched head regular pattern, matching the log content of the log information by using the body regular pattern associated with the matched head regular pattern;
the regular expression included in the rule base of the universal regular matching module comprises a head regular expression and at least one body regular expression, wherein the body regular expression comprises at least one identification name information, and the identification name information is used for matching the body regular expression with log content.
7. The apparatus of claim 5,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a collision regular matching module as an applicable matching module according to the special adaptive zone bit information of the log information with the value of a second preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
and matching the regular expressions associated with the log information by using traversal operation according to the regular expressions included in the rule base of the collision regular matching module.
8. The apparatus of claim 5,
selecting an applicable matching module according to the special adaptation zone bit information of the log information, wherein the applicable matching module comprises:
selecting a customized matching module as an applicable matching module according to the special adaptation zone bit information of the log information with the value of a third preset target value;
the matching the regular expression associated with the log information by using the selected matching module comprises:
analyzing the characteristic information included in the log information, and dynamically loading a preset plug-in related to the characteristic information according to the characteristic information to match the log information.
9. An electronic device, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor to perform the method of any one of claims 1 to 4.
10. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110346638.0A CN113179176B (en) | 2021-03-31 | 2021-03-31 | Log processing method, device and equipment and machine readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110346638.0A CN113179176B (en) | 2021-03-31 | 2021-03-31 | Log processing method, device and equipment and machine readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113179176A true CN113179176A (en) | 2021-07-27 |
| CN113179176B CN113179176B (en) | 2022-05-27 |
Family
ID=76922816
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110346638.0A Active CN113179176B (en) | 2021-03-31 | 2021-03-31 | Log processing method, device and equipment and machine readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113179176B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090119307A1 (en) * | 2007-10-22 | 2009-05-07 | Check Point Software Technologies Ltd. | Syslog parser |
| WO2012031259A1 (en) * | 2010-09-03 | 2012-03-08 | Loglogic, Inc. | Dynamic parsing rules |
| US20130282739A1 (en) * | 2012-04-18 | 2013-10-24 | International Business Machines Corporation | Generating a log parser by automatically identifying regular expressions matching a sample log |
| CN105138593A (en) * | 2015-07-31 | 2015-12-09 | 山东蚁巡网络科技有限公司 | Method for extracting log key information in user-defined way by using regular expressions |
| CN111737091A (en) * | 2020-08-27 | 2020-10-02 | 北京安帝科技有限公司 | Log processing method and device and readable medium |
| CN112350989A (en) * | 2020-09-21 | 2021-02-09 | 西安交大捷普网络科技有限公司 | Log data analysis method |
-
2021
- 2021-03-31 CN CN202110346638.0A patent/CN113179176B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090119307A1 (en) * | 2007-10-22 | 2009-05-07 | Check Point Software Technologies Ltd. | Syslog parser |
| WO2012031259A1 (en) * | 2010-09-03 | 2012-03-08 | Loglogic, Inc. | Dynamic parsing rules |
| US20120197914A1 (en) * | 2010-09-03 | 2012-08-02 | Tim Harnett | Dynamic Parsing Rules |
| US20130282739A1 (en) * | 2012-04-18 | 2013-10-24 | International Business Machines Corporation | Generating a log parser by automatically identifying regular expressions matching a sample log |
| CN105138593A (en) * | 2015-07-31 | 2015-12-09 | 山东蚁巡网络科技有限公司 | Method for extracting log key information in user-defined way by using regular expressions |
| CN111737091A (en) * | 2020-08-27 | 2020-10-02 | 北京安帝科技有限公司 | Log processing method and device and readable medium |
| CN112350989A (en) * | 2020-09-21 | 2021-02-09 | 西安交大捷普网络科技有限公司 | Log data analysis method |
Non-Patent Citations (2)
| Title |
|---|
| ZHE FU等: "High Speed Regular Expression Matching Engine with Fast Pre-Processing", 《CHINA COMMUNICATION》 * |
| 张军等: "基于正则表达式的日志解析***构建研究", 《无线互联科技》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113179176B (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8543528B2 (en) | Exploitation of transition rule sharing based on short state tags to improve the storage efficiency | |
| CN111988231B (en) | Mask quintuple rule matching method and device | |
| CN110007906B (en) | Script file processing method and device and server | |
| CN111368289A (en) | Malicious software detection method and device | |
| CN112380401A (en) | Service data checking method and device | |
| CN113419960A (en) | Seed generation method and system for kernel fuzzy test of trusted operating system | |
| CN108399175B (en) | Data storage and query method and device | |
| CN115237805A (en) | Test case data preparation method and device | |
| CN110532773B (en) | Malicious access behavior identification method, data processing method, device and equipment | |
| CN113179176B (en) | Log processing method, device and equipment and machine readable storage medium | |
| CN111966673B (en) | Big data based data auditing method and device and storage medium | |
| CN111950000B (en) | Access control method and device | |
| CN116541887B (en) | Data security protection method for big data platform | |
| CN111026736B (en) | Data blood margin management method and device and data blood margin analysis method and device | |
| CN116975865A (en) | Malicious Office document detection method, device, equipment and storage medium | |
| CN114995880B (en) | Binary code similarity comparison method based on SimHash | |
| CN107547382B (en) | Neighbor relation discovery method and device | |
| CN112148782B (en) | Market data access method and device | |
| US11720614B2 (en) | Method and system for generating a response to an unstructured natural language (NL) query | |
| CN113282609A (en) | Intelligent data analysis method based on big data technology | |
| CN110808972A (en) | Data stream identification method and device | |
| CN110399403A (en) | Data processing method and device, storage medium, electronic device | |
| CN111563123A (en) | Live warehouse metadata real-time synchronization method | |
| CN109885739B (en) | Data processing method, system and storage medium | |
| CN111144086B (en) | Log formatting method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |