Disclosure of Invention
In order to solve the defects of the prior art, the present invention provides a method and an apparatus for storing a log, and a computer system, so as to solve the above problems.
In order to achieve the above object, in a first aspect, the present invention provides a method for storing a log, where the method includes:
acquiring a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, wherein the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operating records of the preset service platform, and the configuration data comprises meanings corresponding to each data key, replacement rules corresponding to the preset data key and index library addresses corresponding to each preset identification;
analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed;
when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule;
generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
and storing the historical operating record to an index library address corresponding to the preset identification.
In some embodiments, the storing the historical operating record to the index repository address corresponding to the preset identifier includes:
sending a data storage request to a preset system corresponding to the index base address so that the preset system can return data access permission according to the data storage request;
and when receiving the returned data access right, storing the historical operating record to the index library address.
In some embodiments, the method comprises:
receiving a log reading request sent by a user;
generating a visual log according to a historical operating record corresponding to the log reading request;
and returning the visual log to the user.
In some embodiments, the pending log is in Json format.
In some embodiments, the obtaining the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service platform includes:
acquiring logs to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform by using a flash log acquisition system;
the analyzing the log to be processed and identifying the data key and the corresponding data value included in the log to be analyzed comprises:
and analyzing the log to be processed by using a Flink streaming computation engine, and identifying the data key and the corresponding data value included in the log to be analyzed.
In a second aspect, the present application provides a method for storing a log, where the method includes:
sending the generated to-be-processed log to a preset processing end so that the preset processing end can receive the to-be-processed log and obtain corresponding configuration data, wherein the to-be-processed log comprises key value pairs and preset identifications, the key value pairs are used for storing historical operation records of the preset service platform, and the configuration data comprise meanings corresponding to each data key, replacement rules corresponding to the preset data keys and index library addresses corresponding to each preset identification; analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed; when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule; and storing the historical operating record to an index library address corresponding to the preset identification.
In a third aspect, the present application provides a storage apparatus for a log, the apparatus comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operating records of the preset service platform, and the configuration data comprises meanings corresponding to each data key, replacement rules corresponding to the preset data keys and index library addresses corresponding to each preset identification;
the analysis module is used for analyzing the log to be processed and identifying the data key and the corresponding data value included in the log to be analyzed; when the included data key pair comprises a preset data key, modifying a data value corresponding to the included preset data key according to a corresponding replacement rule; generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
and the storage module is used for storing the historical operating record to the index library address corresponding to the preset identifier.
In a fourth aspect, the present application provides a computer system comprising:
one or more processors;
and memory associated with the one or more processors for storing program instructions that, when read and executed by the one or more processors, perform operations comprising:
acquiring a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, wherein the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operating records of the preset service platform, and the configuration data comprises meanings corresponding to each data key, replacement rules corresponding to the preset data key and index library addresses corresponding to each preset identification;
analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed;
when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule;
generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
and storing the historical operating record to an index library address corresponding to the preset identification.
The invention has the following beneficial effects:
the application provides a log storage method, which comprises the steps of obtaining a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, wherein the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operation records of the preset service platform, and the configuration data comprise meanings corresponding to each data key, replacement rules corresponding to the preset data keys and index base addresses corresponding to each preset identification; analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed; when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule; generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key; the historical operation records are stored to the index base address corresponding to the preset identification, the format of the log to be processed generated by the service platform is preset, the processing efficiency of the log is improved, the stored analyzed log can be directly obtained when the log is required to be consulted subsequently, the log does not need to be analyzed again, the analyzed log can be understood by non-operation and maintenance personnel, the problem generated by the system can be conveniently processed by the non-operation and maintenance personnel, and the solution efficiency of the system when the problem occurs is improved.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As described in the background art, in order to solve the above technical problems, the present application provides a log storage system, which includes a service platform for generating logs and a server for storing logs. The system can comprise a plurality of service platforms, and each service platform is preset with the same log format. Preferably, the log may be in Json format.
Example one
As shown in fig. 1, the process of performing log storage and query by the storage system includes:
firstly, a service platform generates a log with a preset format and sends the log to a server;
the generated log may include a prefix and a Json string. Each prefix uniquely corresponds to an index store. The Json string stores historical operating records of the platform in a key-value pair format. The historical operating record may include operating process parameters of the service platform, data operations executed, and the like, which are used to record the operating condition and the service condition of the platform, and this is not limited in the present application.
Secondly, the server receives the logs by using a flash log acquisition system to acquire configuration data corresponding to the service platform;
the method is characterized by comprising the following steps that (1) the Flume is a high-availability, high-reliability and distributed system for acquiring, aggregating and transmitting mass logs, and supports various data senders to be customized in a log system and used for collecting data; at the same time, flash provides the ability to simply process data and write to various data recipients.
Each service platform has corresponding configuration data, and the configuration data comprises data such as the meaning of each data key, the field type of each key value pair, whether each data key is displayed, the enumeration value corresponding to the preset data key, the meaning corresponding to the enumeration value and the like contained in a log generated by the service platform.
Wherein the field type includes time, value, text and other types. And the data value corresponding to the data key which is configured to be required to be displayed in the configuration data is displayed to the non-operation and maintenance personnel inquiring the log.
The meaning of each data key is used to indicate the contribution of the data value to which each data key corresponds. For example, the data value corresponding to a certain data key may be specified to represent the shopping amount of the user on the service platform, and the like, which is not limited in the present application.
The configuration data may include the meaning of the enumerated value corresponding to the preset data key. For example, the configuration data may specify that a value corresponding to a certain data key may be 1 or 0, and when the value corresponding to the data key is 0, the value represents that the purchase has failed, and the value 1 represents that the purchase has succeeded, and the like, which is not limited in the present application. The configuration data may further include a filtering rule, and the data value corresponding to the data key included in the filtering rule is not stored.
And step three, the server side obtains the data value corresponding to each data key contained in the log by using Flink streaming calculation, and determines the function of the data value corresponding to each data key according to the meaning of each data key.
Flink is an open source streaming framework, and the core of the Flink is a distributed streaming data streaming engine written in Java and Scale. Flink executes arbitrary stream data programs in a data parallel and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs.
And according to the action of the data value corresponding to each data key, the server side can obtain the historical operating record of the service platform contained in the log to be stored. For example, the data value corresponding to a certain data key is used to indicate the purchase amount of the user on the service platform, the server may determine the purchase amount of the user according to the data value corresponding to the data key in the log, and generate a purchase record, where the historical operating record includes the purchase record.
Preferably, the historical operating records may be generated in any format, which is not limited in this application.
Step four, the server side determines an index library corresponding to the log according to the prefix;
because each prefix uniquely corresponds to one index library, the server can determine the corresponding index library to be stored according to the prefix. The index library can be deployed on any storage system or device such as an elastic search, which is not limited in the present application.
And fifthly, the server requests data access authority from the index database, and when the data access authority is acquired, the historical operating records are stored in the corresponding index database.
Step six, the server receives a log query request of a user;
and the server determines the log to be read according to the log query request of the user and positions the index library where the log to be read is located.
And step seven, the server side generates a corresponding visual log according to the acquired corresponding historical operating record and a preset template, and returns the visual log to the user.
And the server side requests the index database for data access authority, and when the data access authority is acquired, the corresponding historical operating record is acquired from the index database.
Example two
Corresponding to the foregoing embodiment, the present application provides a log storage method, as shown in fig. 2, where the method includes:
210. acquiring a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, wherein the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operating records of the preset service platform, and the configuration data comprises meanings corresponding to each data key, replacement rules corresponding to the preset data key and index library addresses corresponding to each preset identification;
220. analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed;
preferably, the acquiring the to-be-processed log in the preset format generated by the preset service platform and the configuration data corresponding to the preset service platform includes:
221. acquiring logs to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform by using a flash log acquisition system;
the analyzing the log to be processed and identifying the data key and the corresponding data value included in the log to be analyzed comprises:
222. and analyzing the log to be processed by using a Flink streaming computation engine, and identifying the data key and the corresponding data value included in the log to be analyzed.
230. When the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule;
240. generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
250. and storing the historical operating record to an index library address corresponding to the preset identification.
Preferably, the storing the historical operating record to the index library address corresponding to the preset identifier includes:
251. sending a data storage request to a preset system corresponding to the index base address so that the preset system can return data access permission according to the data storage request;
252. and when receiving the returned data access right, storing the historical operating record to the index library address.
Preferably, the log to be processed is in a Json format.
EXAMPLE III
In correspondence to the above-described embodiments, as shown in fig. 3, the present application provides a storage apparatus for a log, the apparatus including:
an obtaining module 310, configured to obtain a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, where the to-be-processed log includes a key value pair and a preset identifier, the key value pair is used to store a historical operating record of the preset service platform, and the configuration data includes a meaning corresponding to each data key, a replacement rule corresponding to the preset data key, and an index library address corresponding to each preset identifier;
the analysis module 320 is configured to analyze the log to be processed, and identify a data key and a corresponding data value included in the log to be analyzed; when the included data key pair comprises a preset data key, modifying a data value corresponding to the included preset data key according to a corresponding replacement rule; generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
the storage module 330 is configured to store the historical operating record to an index library address corresponding to the preset identifier.
Preferably, the storage module 330 is further configured to send a data storage request to a preset system corresponding to the index repository address, so that the preset system returns a data access permission according to the data storage request; and when receiving the returned data access right, storing the historical operating record to the index library address.
Preferably, the apparatus further includes a receiving module 340, configured to receive a log reading request sent by a user; the analysis module 320 may further be configured to generate a visual log according to the historical operating record corresponding to the log reading request; and returning the visual log to the user.
Preferably, the log to be processed is in a Json format.
Preferably, the obtaining module 310 may be further configured to obtain, by using a Flume log collecting system, a to-be-processed log in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, and the parsing module 320 may be further configured to parse, by using a Flink stream computing engine, the to-be-processed log, and identify a data key and a corresponding data value included in the to-be-parsed log.
Example four
Corresponding to the method, the application provides a log storage method, which comprises the following steps:
sending the generated to-be-processed log to a preset processing end so that the preset processing end can receive the to-be-processed log and obtain corresponding configuration data, wherein the to-be-processed log comprises key value pairs and preset identifications, the key value pairs are used for storing historical operation records of the preset service platform, and the configuration data comprise meanings corresponding to each data key, replacement rules corresponding to the preset data keys and index library addresses corresponding to each preset identification; analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed; when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule; and storing the historical operating record to an index library address corresponding to the preset identification.
EXAMPLE five
An embodiment of the present application provides a computer system, including: one or more processors; and memory associated with the one or more processors for storing program instructions that, when read and executed by the one or more processors, perform operations comprising:
acquiring a log to be processed in a preset format generated by a preset service platform and configuration data corresponding to the preset service platform, wherein the log to be processed comprises key value pairs and preset identifications, the key value pairs are used for storing historical operating records of the preset service platform, and the configuration data comprises meanings corresponding to each data key, replacement rules corresponding to the preset data key and index library addresses corresponding to each preset identification;
analyzing the log to be processed, and identifying a data key and a corresponding data value included in the log to be analyzed;
when the included data key pair contains the preset data key, modifying the data value corresponding to the included preset data key according to the corresponding replacement rule;
generating a historical operating record according to the corresponding meaning and the corresponding data value of each data key;
and storing the historical operating record to an index library address corresponding to the preset identification. Fig. 4 illustrates an architecture of a computer system, which may include, in particular, a processor 1510, a video display adapter 1511, a disk drive 1512, an input/output interface 1513, a network interface 1514, and a memory 1520. The processor 1510, video display adapter 1511, disk drive 1512, input/output interface 1513, network interface 1514, and memory 1520 may be communicatively coupled via a communication bus 1530.
The processor 1510 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solution provided by the present Application.
The Memory 1520 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1520 may store an operating system 1521 for controlling the operation of the computer system 1500, a Basic Input Output System (BIOS)1522 for controlling low-level operations of the computer system 1500. In addition, a web browser 1523, a data storage management 1524, an icon font processing system 1525, and the like may also be stored. The icon font processing system 1525 may be an application program that implements the operations of the foregoing steps in this embodiment of the application. In summary, when the technical solution provided by the present application is implemented by software or firmware, the relevant program codes are stored in the memory 1520 and called for execution by the processor 1510. The input/output interface 1513 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The network interface 1514 is used to connect a communication module (not shown) to enable the device to communicatively interact with other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
The bus 1530 includes a path to transfer information between the various components of the device, such as the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, and the memory 1520.
In addition, the computer system 1500 may also obtain information of specific extraction conditions from the virtual resource object extraction condition information database 1541 for performing condition judgment, and the like.
It should be noted that although the above devices only show the processor 1510, the video display adapter 1511, the disk drive 1512, the input/output interface 1513, the network interface 1514, the memory 1520, the bus 1530, etc., in a specific implementation, the devices may also include other components necessary for proper operation. Furthermore, it will be understood by those skilled in the art that the apparatus described above may also include only the components necessary to implement the solution of the present application, and not necessarily all of the components shown in the figures.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a cloud server, or a network device) to execute the method according to the embodiments or some parts of the embodiments of the present application.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.