CN112800006A - Log storage method and device for network equipment - Google Patents

Log storage method and device for network equipment Download PDF

Info

Publication number
CN112800006A
CN112800006A CN202110109244.3A CN202110109244A CN112800006A CN 112800006 A CN112800006 A CN 112800006A CN 202110109244 A CN202110109244 A CN 202110109244A CN 112800006 A CN112800006 A CN 112800006A
Authority
CN
China
Prior art keywords
field
log
file
index file
hash value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110109244.3A
Other languages
Chinese (zh)
Other versions
CN112800006B (en
Inventor
余忠益
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou DPTech Technologies Co Ltd
Original Assignee
Hangzhou DPTech Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou DPTech Technologies Co Ltd filed Critical Hangzhou DPTech Technologies Co Ltd
Priority to CN202110109244.3A priority Critical patent/CN112800006B/en
Publication of CN112800006A publication Critical patent/CN112800006A/en
Application granted granted Critical
Publication of CN112800006B publication Critical patent/CN112800006B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/13File access structures, e.g. distributed indices
    • G06F16/137Hash-based
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/069Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/06Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols the encryption apparatus using shift registers or memories for block-wise or stream coding, e.g. DES systems or RC4; Hash functions; Pseudorandom sequence generators
    • H04L9/0643Hash functions, e.g. MD5, SHA, HMAC or f9 MAC

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Power Engineering (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The disclosure relates to a log storage method and device for a network device, an electronic device and a computer readable medium. The method comprises the following steps: acquiring a log generated by network equipment; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; and when the matching is unsuccessful, writing the field into the log file according to a pre-second rule. The log storage method and device for the network equipment, the electronic equipment and the computer readable medium can greatly reduce the occupation amount of the storage space on the network equipment under the condition that the log storage information is complete and effective.

Description

Log storage method and device for network equipment
Technical Field
The present disclosure relates to the field of computer information processing, and in particular, to a log storage method and apparatus for a network device, an electronic device, and a computer-readable medium.
Background
In general, the network device needs to store some logs. These logs may help the user diagnose network problems. One common log format is: i time i field 1 i field 2 i field 3. Wherein, the fields may be character strings or numbers with practical significance such as IP, MAC, log content and the like. In a network device, a log in the above format may be stored in a persistent memory in the form of a file or database. However, due to cost constraints, some network devices have relatively small permanent memory capacities, with typical capacities being 8MB, 16MB, 32MB, and so on. Because the network device firmware also takes up a portion of the space, the space left for log storage is actually more strained.
The existing technical scheme mainly realizes the storage of the log by writing in a file or a database. Logically, each log is sequentially arranged to form the following structure: "first log | second log | … …", when the logs need to be viewed, each log is read. Because of the large amount of logs generated in an actual network environment, the network device can only store a certain number of log pieces. When the maximum number of storable logs is exceeded, the system may delete some logs that were oldest in time, freeing up space for storing newly generated logs. In the prior art, the method is limited by storage space, the number of logs which can be stored is small, and long-time log backtracking cannot be performed. Therefore, a new log storage method, apparatus, electronic device, and computer readable medium for a network device are needed.
The above information disclosed in this background section is only for enhancement of understanding of the background of the disclosure and therefore it may contain information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
In view of this, the present disclosure provides a log storage method and apparatus for a network device, an electronic device, and a computer readable medium, which can greatly reduce the occupied amount of a storage space on the network device under the condition that log storage information is complete and effective.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the present disclosure, a log storage method for a network device is provided, the method including: acquiring a log generated by network equipment; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; and when the matching is unsuccessful, writing the field into the log file according to a pre-second rule.
In an exemplary embodiment of the present disclosure, calculating a hash value of a field in the log includes: and calculating the hash value of the field in the log based on a first secure hash algorithm.
In an exemplary embodiment of the present disclosure, before matching the hash value with a file name of an index file existing in the network device, the method further includes: and generating the index file according to the hash value of the field of the history log.
In an exemplary embodiment of the present disclosure, writing the field into the log file according to a pre-first rule includes: acquiring the index file successfully matched; acquiring a field number; storing the field and the field number in the log file.
In an exemplary embodiment of the present disclosure, obtaining the index file successfully matched includes: opening the index file; decompressing the index file; and dividing the file content in the index file according to separators.
In an exemplary embodiment of the present disclosure, obtaining the field number includes: when the field is successfully matched with the existing field in the index file, extracting the field number; and when the field is not successfully matched with the existing field in the index file, generating the file content and the field number.
In an exemplary embodiment of the present disclosure, further comprising: and when the field and the existing field in the index file are not matched successfully, compressing and updating the content and the separator in the field and the field number into the index file.
In an exemplary embodiment of the present disclosure, generating the file content and the field number includes: extracting the current maximum field number; and taking the next sequence number of the maximum field number as the field number.
In an exemplary embodiment of the present disclosure, writing the field into the log file according to a pre-second rule includes: creating an index file; writing the hash value of the field and the initial field number into the index file; storing the hash value of the field and the initial field number in the log file.
In an exemplary embodiment of the present disclosure, further comprising: extracting the log file; self-call index file path according to the hash value of the field; acquiring an index file based on the index file path; and acquiring and displaying the field value based on the index file.
According to an aspect of the present disclosure, a log storage apparatus for a network device is provided, the apparatus including: the log module is used for acquiring logs generated by the network equipment; the hash value module is used for calculating the hash value of the field in the log; the matching module is used for matching the hash value with the file name of the existing index file in the network equipment; the first rule module is used for writing the fields into the log file according to a first rule when the matching is successful; and the second rule module is used for writing the fields into the log file according to a second rule when the matching is unsuccessful.
According to an aspect of the present disclosure, an electronic device is provided, the electronic device including: one or more processors; storage means for storing one or more programs; when executed by one or more processors, cause the one or more processors to implement a method as above.
According to an aspect of the disclosure, a computer-readable medium is proposed, on which a computer program is stored, which program, when being executed by a processor, carries out the method as above.
According to the log storage method and device for the network equipment, the electronic equipment and the computer readable medium, logs generated by the network equipment are obtained; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; when the matching is unsuccessful, the field is written into the log file according to the second rule, so that the occupation amount of the storage space on the network equipment can be greatly reduced under the condition that the log storage information is complete and effective.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings. The drawings described below are merely some embodiments of the present disclosure, and other drawings may be derived from those drawings by those of ordinary skill in the art without inventive effort.
Fig. 1 is a system block diagram illustrating a log storage method and apparatus for a network device according to an example embodiment.
FIG. 2 is a flow chart illustrating a method for log storage for a network device in accordance with an exemplary embodiment.
Fig. 3 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment.
Fig. 4 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment.
Fig. 5 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment.
FIG. 6 is a block diagram illustrating a log storage for a network device in accordance with an example embodiment.
FIG. 7 is a block diagram illustrating an electronic device in accordance with an example embodiment.
FIG. 8 is a block diagram illustrating a computer-readable medium in accordance with an example embodiment.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one element from another. Thus, a first component discussed below may be termed a second component without departing from the teachings of the disclosed concept. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It is to be understood by those skilled in the art that the drawings are merely schematic representations of exemplary embodiments, and that the blocks or processes shown in the drawings are not necessarily required to practice the present disclosure and are, therefore, not intended to limit the scope of the present disclosure.
The technical abbreviations involved in this disclosure are explained as follows:
SHA-1: which may be referred to as the first secure hash algorithm or secure hash algorithm-1, is a cryptographic hash function designed by the united states security agency and promulgated by the National Institute of Standards and Technology (NIST) as the federal data processing standard (FIPS). SHA-1 may generate a 160-bit (20-byte) hash value, called a message digest, which is typically presented in the form of 40 hexadecimal numbers.
Linux: the system is a Unix-like operating system which is free to use and spread, and is a multi-user, multi-task, multi-thread and multi-CPU supporting operating system based on POSIX and Unix. It can run major Unix tool software, applications, and network protocols. It supports 32-bit and 64-bit hardware. Linux inherits the design idea of Unix with network as core, and is a multi-user network operating system with stable performance.
zlib; zlib is a library of functions that provide data compression.
After carefully analyzing the daily log generated by the network device, the inventor of the present application found that: over a period of time, the device generates a log of many field repetitions. Such as "2020-2-1108: 00|1.2.3.4| access network", "2020-2-1108: 01|1.2.3.4| disconnect network", "2020-2-1108: 02|1.2.3.4| access network", "2020-2-1108: 03|1.2.3.4| disconnect network". Obviously, the IP fields in all 4 logs are duplicated. However, the existing solution records 4 logs, including 4 identical IPs, 2 identical "access networks", and 2 identical "disconnect networks". Therefore, the logs generated by the network device may have repeated fields, occupy space and are not economical enough.
In view of the foregoing analysis and technical bottlenecks in the prior art, the present disclosure provides a log storage method and apparatus for a network device, which can solve the problem of storing duplicate fields in multiple logs.
Fig. 1 is a system block diagram illustrating a log storage method and apparatus for a network device according to an example embodiment.
As shown in fig. 1, system architecture 10 may include terminal devices 101, 102, 103, a network 104, and a network device 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the network device 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the network device 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The network device 105 may monitor the network data of the terminal devices 101, 102, 103 and generate log information. The network device 105 may, for example, obtain a log; network device 105 may, for example, calculate a hash value for a field in the log; network device 105 may, for example, match the hash value with a file name of an index file already in the network device; the network device 105 may, for example, upon successful matching, write the field into the log file according to a pre-first rule; network device 105 may write the fields to the log file according to the pre-second rule, for example, when the matching is unsuccessful.
It should be noted that the log storage method for the network device provided by the embodiment of the present disclosure may be executed by the network device 105, and accordingly, the log storage apparatus for the network device may be disposed in the network device 105.
FIG. 2 is a flow chart illustrating a method for log storage for a network device in accordance with an exemplary embodiment. The log storing method 20 for a network device includes at least steps S202 to S210.
As shown in fig. 2, in S202, a log generated by the network device is acquired. Network devices and components are physical entities connected into a network. The network device may include: hubs, switches, bridges, routers, gateways, Network Interface Cards (NICs), Wireless Access Points (WAPs), printers and modems, fiber optic transceivers, fiber optic cables, and the like. The log system is a very important functional component in the network device. Problems can be quickly known and diagnosed by looking at logs of switches, routers, and other network devices.
In S204, a hash value of a field in the log is calculated. The method comprises the following steps: and calculating the hash value of the field in the log based on a first secure hash algorithm. The first secure hash algorithm may be SHA-1. The hash value obtained after calculation is 40 hexadecimal numbers.
In S206, the hash value is matched with a file name of an index file existing in the network device.
Before matching the hash value with the file name of the existing index file in the network device, the method further includes: and generating the index file according to the hash value of the field of the history log.
In S208, when the matching is successful, the fields are written into the log file according to the pre-first rule. The index file matching successfully is obtained, for example; acquiring a field number; storing the field and the field number in the log file.
The details will be described in the embodiment corresponding to fig. 3.
In S210, when the matching is unsuccessful, the field is written into the log file according to a pre-second rule. An index file may be created, for example; writing the hash value of the field and the initial field number into the index file; storing the hash value of the field and the initial field number in the log file.
The details will be described in the embodiment corresponding to fig. 4.
According to the log storage method for the network equipment, the log generated by the network equipment is obtained; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; when the matching is unsuccessful, the field is written into the log file according to the second rule, so that the occupation amount of the storage space on the network equipment can be greatly reduced under the condition that the log storage information is complete and effective.
In the log storage method for the network device of the present disclosure, the field stored in the log file is "SHA-1 value-id" of the field value; saving field values through an additional index file; the fields are spliced and compressed according to the form of 'field 1+ separator + field 2' and then are stored in an index file; the first 2 hexadecimals of the value of field SHA-1 are used as the directory name; the remaining 38 hexadecimal values of the value of field SHA-1 are used as file names; the directory name and the file name form an index file path; the log stored by the above setting mode can greatly save the storage space,
it should be clearly understood that this disclosure describes how to make and use particular examples, but the principles of this disclosure are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.
In one embodiment, the step of reading the log stored by way of the present disclosure may include: extracting the log file; self-call index file path according to the hash value of the field; acquiring an index file based on the index file path; and acquiring and displaying the field value based on the index file.
In this disclosure, id means the number of the field stored in the index file. Since SHA-1 is a hash function, there is a small probability of "collision" with the hash function. In other words, there is a possibility that SHA-1 (field 1) ═ SHA-2 (field 2). Then field 1 and field 2 may hash to the same index file. For this case, when reading the log file, the problem of hash collision can be avoided by reading in the following way.
1) The field recorded in the log file is "SHA 1 (field) value-id", and the number of the field in the index file is indicated by id.
2) The fields are spliced according to the format of 'field 1, separator, field 2, separator, field 3, … …'.
3) And then compressing the spliced fields by using zlib, and finally storing the compressed fields into an index file.
4) id is the number of the field after splicing, for example, id of field 1 is 1, id of field 2 is 2, and so on.
And opening a log file to read the fields of the log when the log of the network equipment is checked subsequently. Since the field is filled with SHA-1 value-id before, first, the first 2 hexadecimals and the remaining 38 hexadecimals are extracted according to the SHA-1 value to form the index file path. The index file is then looked up in the file system. And after finding the index file, opening the index file, and decompressing the content of the index file by using zlib. Extracting the decompressed field content according to the separators, and acquiring the corresponding field value according to the id value. And finally displaying the field value.
Fig. 3 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment. The flow 30 shown in fig. 3 is a detailed description of "write the field in the log file according to the pre-first rule" in S208 in the flow shown in fig. 2.
As shown in fig. 3, in S302, the index file matching successfully is obtained. The index file may be opened, for example; decompressing the index file; and dividing the file content in the index file according to separators.
In S304, the field number is acquired. The field number can be extracted, for example, when the field is successfully matched with the existing field in the index file; and when the field is not successfully matched with the existing field in the index file, generating the file content and the field number.
Wherein, generating the file content and the field number comprises: extracting the current maximum field number; and taking the next sequence number of the maximum field number as the field number.
In one embodiment, the method further comprises compressing and updating the content and the separator in the field and the field number into the index file when the field and the existing field in the index file are not matched successfully.
In S306, the field and the field number are stored in the log file. If the field value is already found in the index file (while getting id), then the SHA-1 value-id of the field is written into the log file (assuming the log file is saved in/tmp/log/2020-2-10. log, where the file path is exemplified by the Linux operating system).
Fig. 4 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment. The flow 40 shown in fig. 4 is a detailed description of "write the field in the log file according to the pre-second rule" of S210 in the flow shown in fig. 2.
As shown in fig. 4, in S402, an index file is created. If the field values do not find the same matching information in the index file, an index file is created. The file name of the index file may be: "/tmp/index/6 d/da9b434e3d40bd18ee92181a0652cbd8f56 fba".
In S404, the hash value of the field, the initial field number, is written into the index file.
In S406, the hash value of the field and the initial field number are stored in the log file. The file path here takes the Linux operating system as an example, and the "SHA-1 value-1" of the field can be written into a log file/tmp/log/2020-2-10. log.
Fig. 5 is a flowchart illustrating a method of log storage for a network device, according to another example embodiment. The flow 50 shown in fig. 5 is a detailed description of the overall process of the present application.
As shown in fig. 5, in S501, the network device generates a log.
In S502, the SHA-1 value of the log field is calculated. First, the SHA-1 value of the field, i.e. 40 hexadecimal numbers, is calculated, and if the field value is "1.2.3.4", then SHA-1(1.2.3.4) ═ 6dda9b434e3d40bd18ee92181a0652cbd8f56fba ".
In S503, whether an index file corresponding to the SHA-1 value exists.
In S504, the index file is opened and decompressed; the file content is divided by delimiters.
In S505, a field is looked up for presence.
In S506, id is the number found.
In S507, an index file is created.
In S508, the compression field value id 1 is written into the index file.
In S509, id is the current maximum number + 1.
In S510, a new file is generated.
In S511, the field SHA-1 value and id are written to the log file. If the field value is already found in the index file (while getting id), then the SHA-1 value-id of the field is written to the log file. If a field value is not found in the index file, the "SHA-1 value-1" of the field is written to the log file. At the same time, an index file is created in the file system of the device. Such as: "/tmp/index/6 d/da9b434e3d40bd18ee92181a0652cbd8f56 fba". It is worth mentioning that the directory name 6d is the first two hexadecimals of SHA-1(1.2.3.4), and the file name da9b434e3d40bd18ee92181a0652cbd8f56fba is the remaining 38 hexadecimals of SHA-1 (1.2.3.4). Finally, after compressing the field value 1.2.3.4 using zlib tool, it is stored in the index file da9b434e3d40bd18ee92181a0652cbd8f56 fba.
Originally, a plurality of logs repeatedly store fields, in the method disclosed by the invention, only one field value is reserved, and only 'SHA-1 value-id' is recorded in the logs. If the length of the field is very long (for example, more than 100 characters), the method of the patent can effectively reduce the occupied storage space. In addition, the zlib compressed field value is adopted, and the space occupation is further reduced.
Example of index file:
file path:
/tmp/index/6d/da9b434e3d40bd18ee92181a0652cbd8f56fba
the file content is as follows:
+++++++++++
field value (id 1)
+++++++++++
File path:
/tmp/index/eb/3cf4c0f840058d986d19a9fa2a9ed48589c81e
the file content is as follows:
++++++++++++++++++++++++++
field value (id 1) delimiter field value (id 2)
++++++++++++++++++++++++++
Example of a log file:
file path: tmp/log/2020-2-10.log
The file content is as follows:
+++++++++++++++++++++++++++++++++++++++++++++
| time |6dda9b434e3d40bd18ee92181a0652cbd8f56fba-1|
+++++++++++++++++++++++++++++++++++++++++++++
| time | eb3cf4c0f840058d986d19a9fa2a9ed48589c81e-1|
+++++++++++++++++++++++++++++++++++++++++++++
| time | eb3cf4c0f840058d986d19a9fa2a9ed48589c81e-2|
+++++++++++++++++++++++++++++++++++++++++++++
As can be seen from the file example above, the log file stores the SHA-1 value-id. Meanwhile, the index file stores the compressed field values separately
It is worth mentioning that the method can also be implemented by newly creating a plurality of database tables and using the fields in table 1 to refer to the primary keys in table 2 and table 3. For example, table 1 is "id primary key | refer to table 2 primary key | refer to table 3 primary key"; table 2 is "id Primary Key | field 1"; table 3 "id primary key | field 1". That is, the log is stored as "1 |2| 3" in table 1, "2 | 1.2.3.4" in table 2, and "3 | access network" in table 3. Tables 2 and 3 correspond to the function of the index file. However, this solution requires the network device to support the database, and the small-capacity network device has limited performance and cannot support the database. While file systems are typically supported. This alternative has certain limitations.
Those skilled in the art will appreciate that all or part of the steps implementing the above embodiments are implemented as computer programs executed by a CPU. When executed by the CPU, performs the functions defined by the above-described methods provided by the present disclosure. The program may be stored in a computer readable storage medium, which may be a read-only memory, a magnetic or optical disk, or the like.
Furthermore, it should be noted that the above-mentioned figures are only schematic illustrations of the processes involved in the methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods. For details not disclosed in the embodiments of the apparatus of the present disclosure, refer to the embodiments of the method of the present disclosure.
Fig. 6 is a block diagram illustrating a log storage for a network device in accordance with another example embodiment. As shown in fig. 6, the log storage 60 for a network device includes: a log module 602, a hash value module 604, a matching module 606, a first rule module 608, and a second rule module 610.
The log module 602 is configured to obtain a log generated by a network device;
the hash value module 604 is configured to calculate a hash value of a field in the log;
the matching module 606 is configured to match the hash value with a file name of an existing index file in the network device;
the first rule module 608 is configured to, when matching is successful, write the field into the log file according to a first rule;
the second rule module 610 is configured to, when the matching is unsuccessful, write the field into the log file according to a pre-second rule.
According to the log storage device for the network equipment, the log generated by the network equipment is obtained; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; when the matching is unsuccessful, the field is written into the log file according to the second rule, so that the occupation amount of the storage space on the network equipment can be greatly reduced under the condition that the log storage information is complete and effective.
FIG. 7 is a block diagram illustrating an electronic device in accordance with an example embodiment.
An electronic device 700 according to this embodiment of the disclosure is described below with reference to fig. 7. The electronic device 700 shown in fig. 7 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 is embodied in the form of a general purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one memory unit 720, a bus 730 that connects the various system components (including the memory unit 720 and the processing unit 710), a display unit 740, and the like.
Wherein the storage unit stores program code that can be executed by the processing unit 710 to cause the processing unit 710 to perform the steps according to various exemplary embodiments of the present disclosure described in this specification. For example, the processing unit 710 may perform the steps as shown in fig. 2, 3, 4, 5.
The memory unit 720 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)7201 and/or a cache memory unit 7202, and may further include a read only memory unit (ROM) 7203.
The memory unit 720 may also include a program/utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Bus 730 may be any representation of one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 700 may also communicate with one or more external devices 700' (e.g., keyboard, pointing device, bluetooth device, etc.), such that a user can communicate with devices with which the electronic device 700 interacts, and/or any devices (e.g., router, modem, etc.) with which the electronic device 700 can communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 750. Also, the electronic device 700 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 760. The network adapter 760 may communicate with other modules of the electronic device 700 via the bus 730. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, as shown in fig. 8, the technical solution according to the embodiment of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present disclosure.
The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The computer readable medium carries one or more programs which, when executed by a device, cause the computer readable medium to perform the functions of: acquiring a log generated by network equipment; calculating to obtain a hash value of a field in the log; matching the hash value with the file name of the existing index file in the network equipment; when the matching is successful, writing the fields into a log file according to a first rule; and when the matching is unsuccessful, writing the field into the log file according to a pre-second rule.
Those skilled in the art will appreciate that the modules described above may be distributed in the apparatus according to the description of the embodiments, or may be modified accordingly in one or more apparatuses unique from the embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Exemplary embodiments of the present disclosure are specifically illustrated and described above. It is to be understood that the present disclosure is not limited to the precise arrangements, instrumentalities, or instrumentalities described herein; on the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (11)

1. A log storage method for a network device, comprising:
acquiring a log generated by network equipment;
calculating to obtain a hash value of a field in the log;
matching the hash value with the file name of the existing index file in the network equipment;
when the matching is successful, writing the fields into a log file according to a first rule;
and when the matching is unsuccessful, writing the field into the log file according to a pre-second rule.
2. The method of claim 1, wherein calculating a hash value for a field in the log comprises:
and calculating the hash value of the field in the log based on a first secure hash algorithm.
3. The method of claim 1, wherein prior to matching the hash value to a filename of an index file already in the network device, further comprising:
and generating the index file according to the hash value of the field of the history log.
4. The method of claim 1, wherein writing the field to a log file according to a pre-first rule comprises:
acquiring the index file successfully matched;
acquiring a field number;
storing the field and the field number in the log file.
5. The method of claim 4, wherein obtaining the index file with a successful match comprises:
opening the index file;
decompressing the index file;
and dividing the file content in the index file according to separators.
6. The method of claim 4, wherein obtaining a field number comprises:
when the field is successfully matched with the existing field in the index file, extracting the field number;
and when the field is not successfully matched with the existing field in the index file, generating the file content and the field number.
7. The method of claim 6, further comprising:
and when the field and the existing field in the index file are not matched successfully, compressing and updating the content and the separator in the field and the field number into the index file.
8. The method of claim 5, wherein generating file content and the field number comprises:
extracting the current maximum field number;
and taking the next sequence number of the maximum field number as the field number.
9. The method of claim 1, wherein writing the field to a log file according to a pre-second rule comprises:
creating an index file;
writing the hash value of the field and the initial field number into the index file;
storing the hash value of the field and the initial field number in the log file.
10. The method of claim 1, further comprising:
extracting the log file;
self-call index file path according to the hash value of the field;
acquiring an index file based on the index file path;
and acquiring and displaying the field value based on the index file.
11. A log storage apparatus for a network device, comprising:
the log module is used for acquiring logs generated by the network equipment;
the hash value module is used for calculating the hash value of the field in the log;
the matching module is used for matching the hash value with the file name of the existing index file in the network equipment;
the first rule module is used for writing the fields into the log file according to a first rule when the matching is successful;
and the second rule module is used for writing the fields into the log file according to a second rule when the matching is unsuccessful.
CN202110109244.3A 2021-01-27 2021-01-27 Log storage method and device for network equipment Active CN112800006B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110109244.3A CN112800006B (en) 2021-01-27 2021-01-27 Log storage method and device for network equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110109244.3A CN112800006B (en) 2021-01-27 2021-01-27 Log storage method and device for network equipment

Publications (2)

Publication Number Publication Date
CN112800006A true CN112800006A (en) 2021-05-14
CN112800006B CN112800006B (en) 2023-05-26

Family

ID=75812057

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110109244.3A Active CN112800006B (en) 2021-01-27 2021-01-27 Log storage method and device for network equipment

Country Status (1)

Country Link
CN (1) CN112800006B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113535654A (en) * 2021-06-11 2021-10-22 安徽安恒数智信息技术有限公司 Log processing method, system, electronic device and storage medium

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040199535A1 (en) * 2003-04-04 2004-10-07 Nir Zuk Attack database structure
US20120221293A1 (en) * 2011-02-28 2012-08-30 Apple Inc. Performance logging framework
CN102929936A (en) * 2012-09-26 2013-02-13 东软集团股份有限公司 Log recording method, log inquiring method and system
US20130103982A1 (en) * 2011-10-25 2013-04-25 International Business Machines Corporation Log file compression
CN107515901A (en) * 2017-07-24 2017-12-26 中国科学院信息工程研究所 A kind of chain type daily record storage organization and its Hash Index Structure, data manipulation method and server, medium
CN107688624A (en) * 2017-08-18 2018-02-13 杭州迪普科技股份有限公司 A kind of daily record index structuring method and device
US20180157700A1 (en) * 2016-12-06 2018-06-07 International Business Machines Corporation Storing and verifying event logs in a blockchain
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN109857714A (en) * 2018-12-24 2019-06-07 浪潮电子信息产业股份有限公司 Journal obtaining method, device, electronic equipment and computer readable storage medium
CN109885549A (en) * 2019-03-04 2019-06-14 安克创新科技股份有限公司 A kind of log collecting method, device, system and computer storage medium
US20190245919A1 (en) * 2018-02-05 2019-08-08 Beijing Elex Technology Co., Ltd. Method and apparatus for information processing, server and computer readable medium
CN110569214A (en) * 2019-08-02 2019-12-13 杭州云纪网络科技有限公司 Index construction method and device for log file and electronic equipment
CN111008182A (en) * 2019-11-07 2020-04-14 中国电信集团工会上海市委员会 Variable field log type data query method and system
US20200134046A1 (en) * 2018-10-29 2020-04-30 EMC IP Holding Company LLC Compression of Log Data Using Field Types

Patent Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040199535A1 (en) * 2003-04-04 2004-10-07 Nir Zuk Attack database structure
US20120221293A1 (en) * 2011-02-28 2012-08-30 Apple Inc. Performance logging framework
US20130103982A1 (en) * 2011-10-25 2013-04-25 International Business Machines Corporation Log file compression
CN102929936A (en) * 2012-09-26 2013-02-13 东软集团股份有限公司 Log recording method, log inquiring method and system
US20180157700A1 (en) * 2016-12-06 2018-06-07 International Business Machines Corporation Storing and verifying event logs in a blockchain
CN107515901A (en) * 2017-07-24 2017-12-26 中国科学院信息工程研究所 A kind of chain type daily record storage organization and its Hash Index Structure, data manipulation method and server, medium
CN107688624A (en) * 2017-08-18 2018-02-13 杭州迪普科技股份有限公司 A kind of daily record index structuring method and device
US20190245919A1 (en) * 2018-02-05 2019-08-08 Beijing Elex Technology Co., Ltd. Method and apparatus for information processing, server and computer readable medium
US20200134046A1 (en) * 2018-10-29 2020-04-30 EMC IP Holding Company LLC Compression of Log Data Using Field Types
CN109684290A (en) * 2018-12-20 2019-04-26 东软集团股份有限公司 Log storing method, device, equipment and computer readable storage medium
CN109857714A (en) * 2018-12-24 2019-06-07 浪潮电子信息产业股份有限公司 Journal obtaining method, device, electronic equipment and computer readable storage medium
CN109885549A (en) * 2019-03-04 2019-06-14 安克创新科技股份有限公司 A kind of log collecting method, device, system and computer storage medium
CN110569214A (en) * 2019-08-02 2019-12-13 杭州云纪网络科技有限公司 Index construction method and device for log file and electronic equipment
CN111008182A (en) * 2019-11-07 2020-04-14 中国电信集团工会上海市委员会 Variable field log type data query method and system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113535654A (en) * 2021-06-11 2021-10-22 安徽安恒数智信息技术有限公司 Log processing method, system, electronic device and storage medium
CN113535654B (en) * 2021-06-11 2023-10-31 安徽安恒数智信息技术有限公司 Log processing method, system, electronic device and storage medium

Also Published As

Publication number Publication date
CN112800006B (en) 2023-05-26

Similar Documents

Publication Publication Date Title
JP6521403B2 (en) Efficient data compression and analysis as a service
CN111949710B (en) Data storage method, device, server and storage medium
CN107506256B (en) Method and device for monitoring crash data
JP7157141B2 (en) A Context-Aware Differencing Algorithm for Genome Files
CN111314388B (en) Method and apparatus for detecting SQL injection
CN110618999A (en) Data query method and device, computer storage medium and electronic equipment
WO2023092580A1 (en) Page display method and apparatus, storage medium, and electronic device
CN110674084A (en) Method, apparatus, and computer-readable storage medium for data protection
CN113268453A (en) Log information compression storage method and device
CN112800006B (en) Log storage method and device for network equipment
CN111046010A (en) Log storage method, device, system, electronic equipment and computer readable medium
CN112436943B (en) Request deduplication method, device, equipment and storage medium based on big data
CN111104259B (en) Database recovery method and device and storage medium
CN110750388B (en) Backup analysis method, device, equipment and medium
CN113760894A (en) Data calling method and device, electronic equipment and storage medium
CN112347383A (en) Sharing link generation method and device and electronic equipment
CN116069725A (en) File migration method, device, apparatus, medium and program product
CN116028917A (en) Authority detection method and device, storage medium and electronic equipment
US10162934B2 (en) Data de-duplication system using genome formats conversion
CN104011718A (en) Method, computer program, and computer for detecting trends in social medium
CN111988405A (en) Message rewriting method of load balancing device and load balancing device
CN111277581A (en) Certificate early warning management method and device, electronic equipment and storage medium
CN115061878B (en) Data transmission method and device, electronic equipment and storage medium
US20100023479A1 (en) Hexadecimal file fast decompression method
CN112732471B (en) Error correction method and error correction device for interface return data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant