CN113760845A - Log processing method, system, device, client and storage medium - Google Patents
Log processing method, system, device, client and storage medium Download PDFInfo
- Publication number
- CN113760845A CN113760845A CN202010834695.9A CN202010834695A CN113760845A CN 113760845 A CN113760845 A CN 113760845A CN 202010834695 A CN202010834695 A CN 202010834695A CN 113760845 A CN113760845 A CN 113760845A
- Authority
- CN
- China
- Prior art keywords
- log
- change operation
- processing
- data
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 230000008859 change Effects 0.000 claims abstract description 170
- 238000012545 processing Methods 0.000 claims abstract description 168
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000012216 screening Methods 0.000 claims abstract description 14
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000012544 monitoring process Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log processing method, a system, a device, a client and a storage medium. The method is applied to a log processing client, and comprises the following steps: acquiring a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by a log acquisition server when a preset binary log of a relational database is monitored based on the middleware to change the log; and screening target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before the change operation and data after the change operation. According to the technical scheme of the embodiment of the invention, the effect of simultaneously acquiring the data before the change operation and the data after the change operation is achieved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer application, in particular to a log processing method, a system, a device, a client and a storage medium.
Background
Currently, most of the large systems adopt MySQL to manage data, for example, the MySQL corresponding to the commodity system stores billions of data, which includes commodity information, category information, and the like. In the face of such a large amount of data, the commodity log processing system is developed in order to clearly record the operation process of the user on the data.
The architecture of the existing commodity log system is an ELK (Elasticsearch + logstack + Kibana) architecture, where logstack is used as a pipeline to read log data from MySQL and store the read result in Elasticsearch, and Kibana is a Web-based graphical interface capable of visualizing the log data stored in Elasticsearch.
In the process of implementing the invention, the inventor finds that the following technical problems exist in the prior art: the existing commodity log processing system cannot acquire data before changing operation, and user experience is poor.
Disclosure of Invention
The embodiment of the invention provides a log processing method, a log processing system, a log processing device, a log processing client and a log processing storage medium, and solves the problem that a log processing system cannot acquire data before change operation.
In a first aspect, an embodiment of the present invention provides a log processing method, which is applied to a log processing client, where the method may include:
acquiring a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by a log acquisition server when a preset binary log of a relational database is monitored based on the middleware to change the log;
and screening target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before the change operation and data after the change operation.
In a second aspect, an embodiment of the present invention further provides a log processing system, which may include: the system comprises a log database, a log processing client and a log obtaining server, wherein the log processing client is configured with middleware and is used for executing the log processing method provided by any embodiment of the invention.
In a third aspect, an embodiment of the present invention further provides a log processing apparatus, configured at a log processing client, where the apparatus may include:
the log obtaining module is used for obtaining a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by the log obtaining server when the log obtaining server monitors a preset binary log of the relational database to change the log based on the middleware;
and the log processing module is used for screening out target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies and storing processing results in a log database, wherein the processing results comprise data before change operation and data after change operation.
In a fourth aspect, an embodiment of the present invention further provides a log processing client, where the log processing client may include:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method provided by any embodiment of the present invention.
In a fifth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the log processing method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, after the change operation log is obtained through the middleware, the target message body belonging to the same transaction can be screened from the change operation log according to the type of each message body in the change operation log, and the target message body is processed, so that the data before the change operation and the data after the change operation can be respectively obtained, wherein the change operation log can be a corresponding log which is obtained and sent by a log obtaining service terminal when the log is changed by monitoring a preset binary log of a relational database based on the middleware. According to the technical scheme, the change operation log with log change is acquired from the binary log based on the middleware, and the target message bodies which are screened from the change operation log and belong to the same transaction are processed, so that the effect of acquiring data before and after the change operation is achieved, and the use experience of a user is improved.
Drawings
Fig. 1 is a flowchart of a log processing method according to a first embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a log processing system according to a second embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a log processing system according to a third embodiment of the present invention;
FIG. 4 is a schematic diagram of an application of a log processing system in a third embodiment of the present invention;
fig. 5 is a block diagram showing a configuration of a log processing apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a log processing client in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a log processing method applicable to a log processing client according to an embodiment of the present invention. The present embodiment is applicable to a case where the acquired change operation log is processed to obtain data before and after the change operation. The method can be executed by the log processing device provided by the embodiment of the invention, and the device can be realized by software and/or hardware.
Referring to fig. 1, the method of the embodiment of the present invention specifically includes the following steps:
s110, acquiring a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by the log acquisition server when the log acquisition server monitors a preset binary log of the relational database to change the log based on the middleware.
The relational database is a database established on the basis of a relational model, such as MySQL, SQL Server, oracle, Access, interbase, and the like. Related information related to a change operation performed on data stored in a relational database may be recorded in a binary log (binlog), where the binlog is a log file in a binary format, and the related information may include data before the change operation, data after the change operation, an operation library name, an operation table name, a table type, a table main key, an operation type, an operation time, an operation source, an operator, an operation ip, and the like, where the operation library name is a name of a database corresponding to the change operation, the operation table name is a name of a table corresponding to the change operation, the table type is a type of a table, such as a commodity table, a category (sku) table, and the like, the table main key is a service main key, i.e., an ID, and the operation type may be update (update), delete (delete), insert (insert), and the like. On the basis, optionally, the operator and the operation ip can be recorded in a binlog or an op _ log table, and when the recording paths of the operator and the operation ip are different, the log processing system can read the operator and the operation ip from the corresponding recording paths.
The log obtaining server can be configured with a database which needs to record the change operation log and the address of the corresponding table, so as to monitor the corresponding binary log according to the address. Therefore, when monitoring a binary log of a preset relational database based on the middleware to change the log, the log obtaining server can obtain a change operation log of the log from the binary log and send the change operation log to the log processing client configured with the middleware. Optionally, when the log obtaining server needs to record a new addition and/or deletion of a database and/or a table of a change operation log, the log obtaining server may perform corresponding configuration, so that the log obtaining server monitors the binlog corresponding to the new database and/or the table and/or does not monitor the binlog corresponding to the deleted database and/or table.
For example, taking the example that the relational database is MySQL and the middleware is canal, the reading process of the log obtaining server is exemplarily described. The operation principle of MySQL is that a master slave relationship exists in MySQL, and the master records change operation logs into binlogs, which may be called binary log events (binary log events), and can be viewed through show binlog events; the slave copies the register log events of the master into a relay log (relay log) of the slave; the slave redos the events in the relay log to reflect the change oplog with its own data. Therefore, canal can simulate the interaction protocol of the slave of MySQL and pretend itself to be the slave to send the dump request to the master of MySQL; and after receiving the dump request, the master pushes the change operation log in the binlog to the canal so that the log acquisition server configured with the canal acquires the change operation log.
And S120, screening target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before the change operation and data after the change operation.
In order to ensure that the subsequent log processing process is performed for a complete transaction, after receiving the change operation log sent by the log acquisition server, the log processing client may screen out target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, for example, the type may be transaction start, data before and after the change operation, and transaction end, that is, the message body of the transaction start, the message body of the data before and after the change operation, and the message body of the transaction end may form a complete transaction, and the message bodies are the target message bodies belonging to the same transaction.
Further, the target message body is processed, namely, all message bodies belonging to the same transaction are processed together, so that the situation that half change operation is recorded is avoided. The processing procedure can be selected according to an actual business scenario, for example, if the change operation log includes a plurality of fields, the fields independent of the log processing system in the plurality of fields can be filtered, for example, the fields independent of the log processing system can be data in which no change operation occurs, such as a plurality of fields of keyVlue ^ …, each keyVlue includes data before change operation and data after change operation, so that analysis can be performed inside each keyVlue to filter out fields in which data in which no change operation occurs, that is, only fields in which data in which change operation occurs are reserved; for another example, a special field in the plurality of fields may be specially processed and a general field may be commonly processed, where the special processing may be a processing logic customized according to a specific service scenario; and so on. On this basis, the target message body (i.e., the processing result) processed through the above-described processing procedure may contain information of data before the change operation, data after the change operation, an operation library name, an operation table name, a table type, a table home key, an operation type, an operation time, an operation source, an operator, an operation ip, and the like.
Still further, the processing results may be sent to a log database to store them in the log database, which may be an Elasticsearch, Hbase, etc., wherein Hbase is a database that supports mass data storage. Taking Hbase as an example, the above-mentioned storage procedure may be to package the processing result into an object containing information such as data before the change operation, data after the change operation, an operation library name, an operation table name, a table type, a table home key, an operation type, an operation time, an operation source, an operator, and an operation ip, and store the object into Hbase. Optionally, assuming that the number of fields in which data of a change operation occurs in a certain target message body is at least two, the fields may be packaged as different objects to be stored in Hbase respectively, thereby improving the clarity of data storage.
According to the technical scheme of the embodiment of the invention, after the change operation log is obtained through the middleware, the target message body belonging to the same transaction can be screened from the change operation log according to the type of each message body in the change operation log, and the target message body is processed, so that the data before the change operation and the data after the change operation can be respectively obtained, wherein the change operation log can be a corresponding log which is obtained and sent by a log obtaining service terminal when the log is changed by monitoring a preset binary log of a relational database based on the middleware. According to the technical scheme, the change operation log with log change is acquired from the binary log based on the middleware, and the target message bodies which are screened from the change operation log and belong to the same transaction are processed, so that the effect of acquiring data before and after the change operation is achieved, and the use experience of a user is improved.
In an optional technical solution, table information of each table in the relational database is encapsulated in a common packet, and the common packet is configured in the log processing client, where the table information includes a table name, and the processing of the target message body may specifically include: and screening out a target table name corresponding to the target message body from the table names, and processing the target message body based on a processing strategy corresponding to the target table name recorded in the common packet.
In order to enable the relational database and the tables therein to have flexible expandability, the related information of the tables can be packaged in an independent common packet, for example, the related information of various relational databases and various tables in each relational database is packaged in one common packet, and the common packet is directly introduced into the log processing client, namely the common packet is configured on the log processing client, at this time, the log processing client does not need to redefine the related information of the tables, and the related information of the tables can be processed by the common packet, namely, when the related information of tables needs to be added and/or deleted, or newly added, changed and/or deleted, the common packet can be directly changed in the common packet, and the common packet can be reintroduced into the log processing client without code modification on the log processing client. The common packet corresponds to a tool packet, which may contain information such as table name (i.e., table type), table number, table primary key, table field, and description.
On this basis, when the target message body is processed, the target table name corresponding to the target message body may be first screened from the table names in the common packet, for example, the target message body may include the corresponding table name, but it cannot be determined what the corresponding table is according to the table name of the target message body, and at this time, it may be determined what the corresponding table is according to the table names in the common packet that has been introduced, that is, what the corresponding table is according to the target table name in the common packet corresponding to the table name of the target message body; therefore, the target message body can be processed according to the processing strategy corresponding to the table, so that the tables needing special processing can be specially processed according to the custom logic.
In an optional technical solution, the log database includes Hbase; storing the processing result in a log database, which may specifically include: and determining a row key calculation factor according to the processing result, calculating a row key according to the row key calculation factor, and storing the processing result in a storage interval corresponding to the row key in Hbase. The Hbase is a nosql (not only sql) database, and during the process of adding, deleting, modifying and checking, a row key (rowkey) serves as a main key, which can uniquely identify a row of data. The row key calculation factor is a factor for calculating the rowkey, and both the specific selection thereof and the specific process of calculating the rowkey according to the row key calculation factor may be determined according to an actual service scenario, for example, the row key calculation factor may include at least one of a table type, a table main key, an operation timestamp, and an operation version number of the target message body, and a row key rule may be preset according to the row key calculation factor to calculate the rowkey. This makes it possible to store the processing result in the storage section corresponding to the row key in Hbase. Illustratively, taking the row key calculation factors including the table type, the inverted table primary key, the 64-bit signed maximum value, the operation timestamp and the operation version number as examples, the 64-bit signed maximum value is a preset value, the operation timestamp is a timestamp corresponding to the operation time for updating the operation log, the operation version number is a version number corresponding to an object when a processing result is packaged into the object, and a rowkey generated based on a preset rowkey rule is composed of a binary array according to the table type + the inverted service primary key + (64-bit signed maximum value-operation timestamp-operation version number). On this basis, optionally, in order to disperse different processing results into different storage intervals of the Hbase, after rowkey is obtained through calculation, the hash value of the processing result can be recalculated, and the processing result is stored in the storage interval corresponding to the hash value in the Hbase, so that the occurrence of hot spot data is avoided.
An optional technical solution, where the log processing method further includes: and reading the operation time from the distributed service, comparing the operation time with the current time, and feeding back a comparison result, wherein the operation time is the time when a log acquisition service end accessed to the distributed service acquires from the change operation log and writes into the distributed service. The distributed service can be a distributed application program coordination service with distributed and open source codes, can package complex and error-prone key services, and provides a simple and easy-to-use interface for user application; the operation time is the time when the change operation log appears, and the current time is the time when the log processing client executes the above steps. Optionally, the distributed service may be zookeeper.
In consideration of application scenarios that may be involved in embodiments of the present invention, the distributed service may be used to record an operation site of the relational database, where the operation site is an operation time for changing the operation log. Therefore, when addresses of the distributed service are configured on the log obtaining service end and the log processing client to access the registration center of the distributed service, the log obtaining service end can obtain operation time from the change operation log and write the operation time into the distributed service; further, the log processing client may obtain the operation time from the distributed service, and execute the rest of operations according to a comparison result between the operation time and the current time, for example, determine whether some technologies need to be optimized according to the comparison result, monitor whether the current working state of the log processing system is abnormal, and the like.
For example, when the log processing client processes a certain change operation log, the log processing client may compare the operation time with the current time, and if the difference between the operation time and the current time is larger, it indicates that the change operation log is not processed in time, which may be caused by that the log obtaining server does not obtain the change operation log in time, and the log processing client fails to process the change operation log currently being processed in time due to more change operation logs to be processed, and so on; for another example, when the log processing client stores the processing result in the Hbase, the log processing client may compare the operation time with the current time, and if the difference between the operation time and the current time is large, it indicates that the change operation log is not stored in time, which may be caused by that the log obtaining server does not obtain the change operation log in time, the log processing client fails to process the change operation log currently being processed in time due to a large number of change operation logs to be processed, and the log processing client fails to store the processing result currently being stored in time due to a large number of processing results to be stored. In other words, when it is determined that the delay phenomenon exists according to the comparison result, a reason check may be performed to perform technical optimization, determine whether an abnormality exists in the current operating state of the log processing system, and the like.
Example two
Before the second embodiment is introduced, detailed analysis is performed again on the ELK architecture in the prior art: firstly, the Logstash occupies a CPU and a memory and has higher resource consumption when running; in addition, the log data is read from the MySQL table, the reading process generates a large pressure on MySQL, and the reading result is only data after operation update (i.e. incremental update), and the reading result cannot be directly operated. Secondly, the elastic search can provide three functions of collecting, analyzing and storing log data, and is more suitable for being applied in an application scene with more reading operations, while most log processing systems involve writing operations, and most reading operations occur in a small amount of scenes such as troubleshooting problems. Again, the Kibana visualization function is not particularly relevant in log processing systems where logging of log data is the main issue.
Correspondingly, fig. 2 is a schematic structural diagram of a log processing system provided in the second embodiment of the present invention. The present embodiment is applicable to a case where a change operation in a relational database is recorded. Referring to fig. 2, the log processing system according to the embodiment of the present invention may include:
the system comprises a log obtaining server 20 configured with middleware, a log processing server 21 configured with middleware and a log database 22, wherein the log obtaining server 20 is used for obtaining a corresponding change operation log when monitoring a binary log of a preset relational database based on the middleware to change the log, and sending the change operation log to the log processing server 21; the log processing server 21 is configured to screen a target message body belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store a processing result in the log database 22, where the processing result includes data before the change operation and data after the change operation.
It should be noted that, in the log processing system according to the embodiment of the present invention, the change operation log is read from the binlog based on the middleware, which can achieve synchronization of the change operation log in a near real-time manner, and even if the system faces massive data, the system does not exert pressure on the relational database and affect on the online service, and the system can directly operate the read result to obtain data before and after the change operation, where the data before and after the change operation is essential key data in the log processing system. Therefore, when facing mass data, the log processing system can still accurately record the change operation related to the mass data in real time, and the performance and the stability of the log processing system are improved, and meanwhile, the resource consumption cost is reduced.
According to the technical scheme of the embodiment of the invention, when monitoring the binary log of the preset relational database to change the log based on the middleware through the log obtaining server, the corresponding change operation log is obtained and sent to the log processing client configured with the middleware; the log processing client can screen out target message bodies belonging to the same transaction from the log processing client according to the types of the message bodies in the change operation log, process the target message bodies, and store processing results in a log database, wherein the processing results can comprise data before change operation and data after change operation. According to the technical scheme, the change operation log with log change is acquired from the binary log based on the middleware, and the target message bodies which are screened from the change operation log and belong to the same transaction are processed, so that the effect of acquiring data before and after the change operation at the same time is achieved, and the use experience of a user is improved.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a log processing system provided in the third embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the log processing system may further include a log search server, configured to search in a log database according to a received search condition input by the user, and return the searched target data that matches the search condition. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
Referring to fig. 3, the log processing system of the present embodiment may include: the system comprises a log obtaining server 30 configured with middleware, a log processing client 31 configured with the middleware, a log database 32 and a log searching server 33, wherein the log obtaining server 30 is used for obtaining a corresponding change operation log when monitoring a preset binary log of a relational database to change the log based on the middleware, and sending the change operation log to the log processing client 31; the log processing client 31 is configured to screen a target message body belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store a processing result in the log database 32, where the processing result includes data before the change operation and data after the change operation; and the log search server 33 is configured to search in the log database 32 according to the received search condition input by the user, and return the searched target data matched with the search condition.
The log search service 33 may provide a front page for the user to input a search condition in the front page, so that after receiving the search condition input by the user, the log search service may search the log database 32 according to the search condition to search out target data matching the search condition, and return the target data to the front page for presentation to the user. For example, in consideration of application scenarios that may be involved in embodiments of the present invention, the front-end page (i.e., web page) may be implemented based on vm and/or js, and may implement a function of selecting and/or inputting contents such as table types, table main keys, time periods of operation time, keywords, and the like, so that the log search service 33 may construct a search condition according to the selected and/or input contents of the user, and search the log database 32 according to the search condition to return target data matching the search condition.
It should be noted that the search result searched by the search function provided by the Elasticsearch is a string of character strings including various types of data, and such a search result is poor in readability for the user, and it is difficult for the user to find the relevance between the data from the stacked various data. Accordingly, the target data searched by the log search server according to the embodiment of the invention can separately present various types of data, for example, the data before the change operation is placed on the left side of the page and the data after the change operation is placed on the right side of the page, so that the readability of the search result is improved.
According to the technical scheme of the embodiment of the invention, the log search server receives the search condition input by the user, and searches out the matched target data in the log database according to the search condition and returns the target data, so that the effect of quickly and accurately searching the data stored in the log database is achieved.
On this basis, optionally, the log database includes Hbase, and the log search server is specifically configured to calculate a line key according to a received search condition input by the user, and return target data in a storage interval corresponding to the line key in the Hbase. Wherein, as mentioned above, when the log database is Hbase, the specific storage interval in which it is stored is related to the calculated rowkey. Therefore, when a search is performed on Hbase in accordance with the search condition, rowkey may be calculated in accordance with the search condition, and data stored in a storage section corresponding to rowkey in Hbase may be set as target data. If the operation time in the search condition is a time period, one rowkey may be calculated from the start time and the end time of the time period, and the data in the storage interval in which the two rowkeys are located and the storage interval located between the two storage intervals may be used as the target data.
Further, optionally, the search condition includes a keyword, table information of each table in the relational database is encapsulated in a common packet, and the common packet is configured in the log search server, where the table information includes a mapping relationship between attribute information of the tables; and the log search server is also used for screening the target data according to the keywords and the mapping relation and updating the target data according to the screening result. The data in the storage interval where the rowkey is located is not necessarily the target data required by the user, and at this time, the data can be filtered again through the keywords input by the user. It should be noted that each data may be composed of a plurality of fields, and the field in which the keyword is located may be only one of the fields, so to determine the remaining corresponding fields according to the one field of the keyword, the mapping relationship between the attribute information of each table may be encapsulated in a common packet, and the common packet is configured at the log search server, so that the log search server determines the remaining attribute information matching the keyword according to the mapping relationship in the common packet, and each attribute information may occupy at least one field. For example, the table type and table number may be mapped to a HashMap, the table type and table primary key may be mapped to a HashMap, the table number and table primary key may be mapped to a HashMap, and the table type and table description information may be mapped to a HashMap. Taking the example that the keyword is a table number, the rest of attribute information corresponding to the table number can be determined according to the HashMap, and the search result can be screened again according to the attribute information.
In order to better understand the specific working process of the log processing system, the log processing system of the present embodiment is exemplarily described below with reference to specific examples. For example, as shown in fig. 4, the Log processing system may include a canal server (i.e., a Log acquisition server configured with a middleware canal), a canal client (i.e., a Log processing client configured with a middleware canal), a Log Search Web (i.e., a Log Search server providing a front page), and an Hbase, wherein the canal server fetches a change operation Log from MySQL based on the canal and sends the fetched result to the canal client; further, the canal client performs a series of processing such as analysis and filtering on the received change operation log, and stores the processing result in the Hbase; therefore, the subsequent Log Search Web can find matched target data in the Hbase according to the Search condition input by the user, and therefore the effects of real-time acquisition and accurate Search of the change operation Log are achieved.
Example four
Fig. 5 is a block diagram of a log processing apparatus according to a fourth embodiment of the present invention, where the apparatus is configured to a log processing client and is configured to execute the log processing method according to the first embodiment. The device and the log processing method of the first embodiment belong to the same inventive concept, and details which are not described in detail in the embodiment of the log processing device can refer to the embodiment of the log processing method. Referring to fig. 5, the apparatus may specifically include: a log obtaining module 410 and a log processing module 420.
The log obtaining module 410 is configured to obtain a change operation log based on the middleware, where the change operation log is a corresponding log sent by the log obtaining server when monitoring a binary log of a preset relational database based on the middleware to change the log;
and the log processing module 420 is configured to screen out a target message body belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store a processing result in the log database, where the processing result includes data before the change operation and data after the change operation.
Optionally, table information of each table in the relational database is encapsulated in a common packet, and the common packet is configured in the log processing client, where the table information includes a table name;
correspondingly, the log processing module 420 may specifically include:
and the log processing unit is used for screening out a target table name corresponding to the target message body from the table names and processing the target message body based on a processing strategy corresponding to the target table name recorded in the common packet.
Optionally, the log database includes Hbase, and the log processing module 420 may specifically include:
the log storage unit is used for determining a row key calculation factor according to the processing result, calculating a row key according to the row key calculation factor, and storing the processing result in a storage interval corresponding to the row key in Hbase;
wherein the row key calculation factor includes at least one of a table type, a table primary key, an operation time stamp, and an operation version number of the target message body.
Optionally, the log storage unit may be further configured to: and recalculating the hash value of the row key, and updating the row key according to the calculation result.
Optionally, on the basis of the above apparatus, the apparatus may further include:
and the time comparison module is used for reading the operation time from the distributed service, comparing the operation time with the current time and feeding back a comparison result, wherein the operation time is the time of writing the operation time into the distributed service after the operation time is acquired from the change operation log by the log acquisition service terminal accessed into the distributed service.
In the log processing apparatus provided in the fourth embodiment of the present invention, the log obtaining module and the log processing module are mutually matched, and after the change operation log is obtained through the middleware, a target message body belonging to the same transaction may be screened from the change operation log according to the type of each message body in the change operation log, and the target message body is processed, so that data before the change operation and data after the change operation may be obtained respectively, where the change operation log may be a corresponding log that is obtained and sent by the log obtaining server when the log is changed based on a binary log that is monitored by the middleware to a preset relational database. According to the device, the change operation log with log change is acquired from the binary log based on the middleware, and the target message bodies which are screened from the change operation log and belong to the same transaction are processed, so that the effect of acquiring data before and after the change operation is achieved, and the use experience of a user is improved.
The log processing device provided by the embodiment of the invention can execute the log processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, in the embodiment of the log processing apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE five
Fig. 6 is a schematic structural diagram of a log processing client according to a fifth embodiment of the present invention, as shown in fig. 6, the log processing client includes a memory 510, a processor 520, an input device 530, and an output device 540. The number of processors 520 in the log processing client may be one or more, and one processor 520 is taken as an example in fig. 6; the memory 510, processor 520, input device 530, and output device 540 in the log processing client may be connected by a bus or other means, such as by bus 550 in fig. 6.
The memory 510 is used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the log processing method in the embodiment of the present invention (for example, the log obtaining module 410 and the log processing module 420 in the log processing apparatus). The processor 520 executes various functional applications of the log processing client and data processing by executing software programs, instructions, and modules stored in the memory 510, that is, implements the log processing method described above.
The memory 510 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the log processing client, and the like. Further, the memory 510 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 510 may further include memory located remotely from processor 520, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.
EXAMPLE six
An embodiment of the present invention provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a log processing method when executed by a computer processor, and the method is applied to a log processing client, and the method may include:
acquiring a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by a log acquisition server when a preset binary log of a relational database is monitored based on the middleware to change the log;
and screening target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before the change operation and data after the change operation.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the log processing method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. With this understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (12)
1. A log processing method is applied to a log processing client and comprises the following steps:
acquiring a change operation log based on a middleware, wherein the change operation log is a corresponding log which is sent by a log acquisition server when a preset binary log of a relational database is monitored based on the middleware to perform log change;
and screening target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before change operation and data after change operation.
2. The method of claim 1, wherein table information of each table in the relational database is encapsulated in a common packet and the common packet is configured in the log processing client, wherein the table information comprises a table name, and wherein the processing the target message body comprises:
and screening out a target table name corresponding to the target message body from each table name, and processing the target message body based on a processing strategy corresponding to the target table name recorded in the common packet.
3. The method of claim 1, wherein the log database includes Hbase, and wherein storing the processing results in the log database includes:
determining a row key calculation factor according to a processing result, calculating a row key according to the row key calculation factor, and storing the processing result in a storage interval corresponding to the row key in Hbase;
wherein the row key calculation factor includes at least one of a table type, a table primary key, an operation timestamp, and an operation version number of the target message body.
4. The method of claim 3, further comprising, after said calculating a row key based on said row key calculation factor:
and recalculating the hash value of the row key, and updating the row key according to the calculation result.
5. The method of claim 1, further comprising:
reading operation time from distributed service, comparing the operation time with the current time, and feeding back a comparison result, wherein the operation time is the time written in the distributed service after the log obtaining service terminal accessed to the distributed service obtains from the change operation log.
6. A log processing system, comprising: the system comprises a log database, a log processing client side and a log obtaining server side, wherein the log processing client side is configured with middleware and is used for executing the log processing method of any one of claims 1-5.
7. The system of claim 6, further comprising:
and the log search server is used for searching in the log database according to the received search condition input by the user and returning the searched target data matched with the search condition.
8. The system of claim 7, wherein the log database comprises Hbase;
the log search server is specifically configured to calculate a line key according to a received search condition input by a user, and return target data in a storage interval corresponding to the line key in the Hbase.
9. The system according to claim 8, wherein the search condition includes a key, table information of each table in the relational database is packaged in a common packet and the common packet is configured in the log search server, and the table information includes a mapping relationship between attribute information of the table;
and the log search server is further used for screening the target data according to the keywords and the mapping relation and updating the target data according to a screening result.
10. A log processing apparatus, provided at a log processing client, comprising:
the log obtaining module is used for obtaining a change operation log based on the middleware, wherein the change operation log is a corresponding log which is sent by a log obtaining server when monitoring a binary log of a preset relational database based on the middleware to change the log;
and the log processing module is used for screening out target message bodies belonging to the same transaction from the change operation log according to the types of the message bodies in the change operation log, processing the target message bodies and storing processing results in a log database, wherein the processing results comprise data before change operation and data after change operation.
11. A log processing client, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log processing method of any of claims 1-5.
12. A computer-readable storage medium, on which a computer program is stored, the computer program, when being executed by a processor, implementing a log processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010834695.9A CN113760845A (en) | 2020-08-17 | 2020-08-17 | Log processing method, system, device, client and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010834695.9A CN113760845A (en) | 2020-08-17 | 2020-08-17 | Log processing method, system, device, client and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760845A true CN113760845A (en) | 2021-12-07 |
Family
ID=78785600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010834695.9A Pending CN113760845A (en) | 2020-08-17 | 2020-08-17 | Log processing method, system, device, client and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760845A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117648432A (en) * | 2023-10-16 | 2024-03-05 | 深圳市酷开网络科技股份有限公司 | Monitoring and early warning system and method based on log acquisition |
| CN117648432B (en) * | 2023-10-16 | 2024-07-26 | 深圳市酷开网络科技股份有限公司 | Monitoring and early warning system and method based on log acquisition |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101212481A (en) * | 2007-12-21 | 2008-07-02 | 华为技术有限公司 | Transaction control method, system, and device |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| US20190361913A1 (en) * | 2016-11-16 | 2019-11-28 | Silcroad Soft, Inc. | Data replication technique in database management system |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
| CN110837535A (en) * | 2018-08-16 | 2020-02-25 | ***通信集团江西有限公司 | Data synchronization method, device, equipment and medium |
-
2020
- 2020-08-17 CN CN202010834695.9A patent/CN113760845A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101212481A (en) * | 2007-12-21 | 2008-07-02 | 华为技术有限公司 | Transaction control method, system, and device |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| US20190361913A1 (en) * | 2016-11-16 | 2019-11-28 | Silcroad Soft, Inc. | Data replication technique in database management system |
| CN110837535A (en) * | 2018-08-16 | 2020-02-25 | ***通信集团江西有限公司 | Data synchronization method, device, equipment and medium |
| CN110597914A (en) * | 2019-09-18 | 2019-12-20 | 北京思维造物信息科技股份有限公司 | Data transmission system, method, device and equipment |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117648432A (en) * | 2023-10-16 | 2024-03-05 | 深圳市酷开网络科技股份有限公司 | Monitoring and early warning system and method based on log acquisition |
| CN117648432B (en) * | 2023-10-16 | 2024-07-26 | 深圳市酷开网络科技股份有限公司 | Monitoring and early warning system and method based on log acquisition |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110990473B (en) | Tag data processing system and method | |
| CN110162512B (en) | Log retrieval method, device and storage medium | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| CN112860777B (en) | Data processing method, device and equipment | |
| CN113204598B (en) | Data synchronization method, system and storage medium | |
| CN114356921A (en) | Data processing method, device, server and storage medium | |
| CN111078695B (en) | Method and device for calculating association relation of metadata in enterprise | |
| WO2021179488A1 (en) | Monitoring data storage method and device, server and storage medium | |
| CN114722092A (en) | Time series data processing method, system, storage medium and electronic equipment | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| US10678789B2 (en) | Batch data query method and apparatus | |
| CN115757626A (en) | Data quality detection method and device, electronic equipment and storage medium | |
| CN111047434A (en) | Operation record generation method and device, computer equipment and storage medium | |
| CN114238085A (en) | Interface testing method and device, computer equipment and storage medium | |
| CN113672668A (en) | Log real-time processing method and device in big data scene | |
| CN107590160B (en) | Method and device for monitoring internal structure of radix tree to realize test | |
| CN110362590A (en) | Data managing method, device, system, electronic equipment and computer-readable medium | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN113760845A (en) | Log processing method, system, device, client and storage medium | |
| CN110851437A (en) | Storage method, device and equipment | |
| CN114860432A (en) | Method and device for determining information of memory fault | |
| CN112199426B (en) | Interface call management method, device, server and medium under micro-service architecture | |
| CN115237889A (en) | Database switching method and device, storage medium and computer equipment | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN114547206A (en) | Data synchronization method and data synchronization system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |