CN104714878A - Method and device for collecting log data - Google Patents
Method and device for collecting log data Download PDFInfo
- Publication number
- CN104714878A CN104714878A CN201310676525.2A CN201310676525A CN104714878A CN 104714878 A CN104714878 A CN 104714878A CN 201310676525 A CN201310676525 A CN 201310676525A CN 104714878 A CN104714878 A CN 104714878A
- Authority
- CN
- China
- Prior art keywords
- file
- journal
- journal file
- described appointment
- appointment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and device for collecting log data and belongs to the field of computer technology. The method includes the following steps that an appointed log file system is monitored through a file change notification interface; the event that an appointed log file is written into log data is obtained; whether a current file fingerprint of the appointed log file is identical to a former file fingerprint of the appointed log file is judged; if the current file fingerprint of the appointed log file is not identical to the former file fingerprint of the appointed log file, a new filename, an original filename and original file offset corresponding to the appointed log file are obtained, and the log data are collected from a log file corresponding to the original filename and a log file corresponding to the new filename according to the new filename, the original filename and the original file offset corresponding to the appointed log file. The device comprises a monitoring module, a first obtaining module, a first judgment module and a first collecting module. Through the method and device, the log data recorded in the renamed log file can be collected, and the complete log data can be collected.
Description
Technical field
The application relates to field of computer technology, is specifically related to a kind of method and device of collector journal data.
Background technology
Along with the development of computer technology, the function of computing machine also gets more and more, and computing machine has become requisite instrument.Computing machine needs dependence program to run, for the ease of safeguarding program, ensure that computing machine normally runs, usually the logdata records such as the running status produced in program operation process are being specified in journal file, just can obtain the relevant information of program by query log files.In a computer cluster, often to need the collection of log data of some pre-set programs in each computing machine, on server, to carry out unified supervision and process to daily record data.
The method of existing collector journal data is as follows: according to the filename of specifying journal file, inquire appointment journal file, collector journal data from appointment journal file.
But, in order to prevent journal file excessive, log rollback can be carried out according to preset rules (time interval or file size etc.), that is: when specifying journal file to meet preset rules, appointment journal file is renamed (being namely revised as other filename), the daily record data then after a newly-built blank journal file record rollback the same with specifying the old file name of journal file.The method of existing collector journal data, be the filename according to specifying journal file, collector journal data from appointment journal file, when after log rollback, be recorded in daily record data in the journal file after renaming just collect less than, make the daily record data collected incomplete.
Summary of the invention
Technical problems to be solved in this application are the method and the device that provide a kind of collector journal data, can collect the daily record data be recorded in the journal file after renaming, and make it possible to collect complete complete daily record data.
In order to solve the problem, this application discloses a kind of method of collector journal information, described method comprises:
The Journaling File System of specifying is monitored by file change notification interface;
Obtain the event of specifying journal file to be written into daily record data;
Judge the file fingerprint that described appointment journal file is current, whether consistent with the file fingerprint before described appointment journal file;
If inconsistent, then obtain and new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename.
Further, described method also comprises:
Described monitored the Journaling File System of specifying by file change notification interface after, also comprise:
Obtain described appointment journal file by the event of rename;
Record new filename corresponding to described appointment journal file and old file name.
Further, described file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event that described appointment journal file is written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, described appointment journal file is comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
Further, described acquisition new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename, comprising:
Document misregistration amount current for described appointment journal file and filename are recorded as described original side-play amount and described old file name;
Document misregistration amount current for described appointment journal file is reset to 0, collects the daily record data in described appointment journal file, regenerate and record the file fingerprint of described appointment journal file;
When it is determined that the presence of the record of new filename corresponding to described appointed date will file and old file name, according to described original side-play amount, collector journal data from the journal file that described new filename is corresponding.
Further, described acquisition appointment journal file judges before whether consistent with the file fingerprint before described appointment journal file, also to comprise the file fingerprint that described appointment journal file is current after being written into the event of daily record data:
Judge whether first time is collected daily record data to described appointment journal file;
If so, described appointment journal file is joined and collects in file set, the document misregistration amount of described appointment journal file is set to 0, collects the daily record data in described appointment journal file, generate and record the file fingerprint of described appointment journal file;
If not, then perform the file fingerprint that described judgement described appointment journal file is current, with the step whether file fingerprint before described appointment journal file is consistent.
Further, whether the file fingerprint that described judgement described appointment journal file is current, after consistent with the file fingerprint before described appointment journal file, also comprise:
If consistent, then current according to described appointment journal file document misregistration amount, collector journal data from described appointment journal file, and the document misregistration amount resetting described appointment journal file after collector journal data.
In order to solve the problem, this application discloses a kind of device of collector journal data, described device comprises:
Monitoring modular, for monitoring the Journaling File System of specifying by file change notification interface;
First acquisition module, for obtaining the event of specifying journal file to be written into daily record data;
First judge module, for judging the file fingerprint that described appointment journal file is current, whether consistent with the file fingerprint before described appointment journal file;
First collection module, if be inconsistent for the judged result of described first judge module, then obtain and new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename.
Further, described device also comprises:
Second acquisition module, for obtaining described appointment journal file by the event of rename;
Logging modle, for recording new filename corresponding to described appointment journal file and old file name.
Further, described file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event that described appointment journal file is written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, described appointment journal file is comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
Further, described first collection module comprises:
Record cell, if for inconsistent, records document misregistration amount current for described appointment journal file and filename as described original side-play amount and described old file name;
Processing unit, for document misregistration amount current for described appointment journal file is reset to 0, collects the daily record data in described appointment journal file, regenerates and records the file fingerprint of described appointment journal file;
Collector unit, for when it is determined that the presence of the record of new filename corresponding to described appointed date will file and old file name, according to described original side-play amount, collector journal data from the journal file that described new filename is corresponding.
Further, described device also comprises:
Second judge module, specify after journal file is written into the event of daily record data for obtaining, judge the file fingerprint that described appointment journal file is current, before whether consistent with the file fingerprint before described appointment journal file, judge whether first time is collected daily record data to described appointment journal file;
Processing module, if for the judged result of described second judge module be, described appointment journal file is joined and collects in file set, the document misregistration amount of described appointment journal file is set to 0, collect the daily record data in described appointment journal file, generate and record the file fingerprint of described appointment journal file;
Notification module, if for the judged result of described second judge module be, then notify that described first judge module performs the current file fingerprint of described judgement described appointment journal file, with the step whether file fingerprint before described appointment journal file is consistent.
Further, described device also comprises:
Second collection module, if be consistent for the judged result of described first judge module, then current according to described appointment journal file document misregistration amount, collector journal data from described appointment journal file, and the document misregistration amount resetting described appointment journal file after collector journal data.
Compared with prior art, the application can obtain and comprise following technique effect:
The Journaling File System of specifying is monitored by file change notification interface, when getting the event of specifying journal file to be written into daily record data, judge the file fingerprint of specifying journal file current, during with the file fingerprint of the appointment journal file of local record inconsistent (proof there occurs log rollback), obtain and the new filename corresponding according to appointment journal file, old file name and original side-play amount, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename, make it possible to collect the daily record data be recorded in the journal file after renaming, make it possible to collect complete complete daily record data.
Certainly, the arbitrary product implementing the application must not necessarily need to reach above-described all technique effects simultaneously.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide further understanding of the present application, and form a application's part, the schematic description and description of the application, for explaining the application, does not form the improper restriction to the application.In the accompanying drawings:
Fig. 1 is the method flow diagram of the first collector journal data of the embodiment of the present application;
Fig. 2 is the method flow diagram of the second collector journal data of the embodiment of the present application;
Fig. 3 is the apparatus structure schematic diagram of the first collector journal data of the embodiment of the present application;
Fig. 4 is the apparatus structure schematic diagram of the second collector journal data of the embodiment of the present application;
Fig. 5 is the apparatus structure schematic diagram of the third collector journal data of the embodiment of the present application;
Fig. 6 is the apparatus structure schematic diagram of the 4th kind of collector journal data of the embodiment of the present application.
Embodiment
Drawings and Examples will be coordinated below to describe the embodiment of the application in detail, by this to the application how application technology means solve technical matters and the implementation procedure reaching technology effect can fully understand and implement according to this.
In one typically configuration, computing equipment comprises one or more processor (CPU), input/output interface, network interface and internal memory.
Internal memory may comprise the volatile memory in computer-readable medium, and the forms such as random access memory (RAM) and/or Nonvolatile memory, as ROM (read-only memory) (ROM) or flash memory (flash RAM).Internal memory is the example of computer-readable medium.
Computer-readable medium comprises permanent and impermanency, removable and non-removable media can be stored to realize information by any method or technology.Information can be computer-readable instruction, data structure, the module of program or other data.The example of the storage medium of computing machine comprises, but be not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic RAM (DRAM), the random access memory (RAM) of other types, ROM (read-only memory) (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc ROM (read-only memory) (CD-ROM), digital versatile disc (DVD) or other optical memory, magnetic magnetic tape cassette, tape magnetic rigid disk stores or other magnetic storage apparatus or any other non-transmitting medium, can be used for storing the information can accessed by computing equipment.According to defining herein, computer-readable medium does not comprise non-temporary computer readable media (transitory media), as data-signal and the carrier wave of modulation.
embodiment describes
Be described further with the realization of an embodiment to the application's method below.As shown in Figure 1, be the method flow diagram of a kind of collector journal data of the embodiment of the present application, the method comprises:
S101: monitored the Journaling File System of specifying by file change notification interface.
S102: obtain the event of specifying journal file to be written into daily record data.
S103: judge the file fingerprint of specifying journal file current, whether consistent with the file fingerprint of specifying before journal file.
S104: if inconsistent, then obtain and according to specifying corresponding new filename, old file name and the original side-play amount of journal file, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename.
Preferably, after monitoring the Journaling File System of specifying by file change notification interface, also comprise:
Obtain and specify journal file by the event of rename;
The new filename and old file name that journal file is corresponding specified in record.
Preferably, file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event of specifying journal file to be written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, journal file is specified to be comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
Preferably, to obtain and according to specifying corresponding new filename, old file name and the original side-play amount of journal file, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename, comprising:
Record specifying the current document misregistration amount of journal file and filename as original side-play amount and old file name;
By specifying the current document misregistration amount of journal file to reset to 0, collecting the daily record data of specifying in journal file, regenerating and recording the file fingerprint of specifying journal file;
When it is determined that the presence of the record of new filename corresponding to appointment journal file and old file name, according to original side-play amount, collector journal data from the journal file that new filename is corresponding.
Preferably, whether acquisition appointment journal file judges the file fingerprint of specifying journal file current, before consistent with the file fingerprint of specifying before journal file, also comprises after being written into the event of daily record data:
Whether first time is collected daily record data to judge to specify journal file;
If so, appointment journal file being joined and collects in file set, by specifying the document misregistration amount of journal file to be set to 0, collecting the daily record data of specifying in journal file, generate and record the file fingerprint of specifying journal file;
If not, then perform the file fingerprint judging to specify journal file current, with the step whether file fingerprint of specifying before journal file is consistent.
Preferably, judge the file fingerprint of specifying journal file current, after whether consistent with the file fingerprint of specifying before journal file, also comprise:
If consistent, then according to the document misregistration amount of specifying journal file current, collector journal data from appointment journal file, and after collector journal data, reset the document misregistration amount of specifying journal file.
The method of the collector journal data described in the present embodiment, the Journaling File System of specifying is monitored by file change notification interface, when getting the event of specifying journal file to be written into daily record data, judge the file fingerprint of specifying journal file current, during with the file fingerprint of the appointment journal file of local record inconsistent (proof there occurs log rollback), obtain and the new filename corresponding according to appointment journal file, old file name and original side-play amount, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename, make it possible to collect the daily record data be recorded in the journal file after renaming, make it possible to collect complete complete daily record data.
The method of the collector journal data described in the embodiment of the present application can collect the daily record data in the various system such as linux system, windows system, for the ease of understanding, be described further for the daily record data collected in linux system below, as shown in Figure 2, for the method flow diagram of a kind of collector journal data of the embodiment of the present application, the method comprises:
S201: monitored the Journaling File System of specifying by inotify.
Wherein, inotify monitoring to as if catalogue.After monitoring catalogue changes, inotify can produce corresponding sequence of events in real time.
S202: judging whether to get in_modify(file has new write) immigration of event, in_move_from event (file shifts out) or in_move_to(file) event, if get in_modify event, then perform S203; If get in_move_from event, then perform S211; If get in_move_to event, then perform S213.
S203: whether first time is collected daily record data to judge to specify journal file, if so, then performs S204, otherwise, perform S205.
S204: joined by appointment journal file and collect in file set, by specifying the document misregistration amount of journal file to be set to 0, collects the daily record data of specifying in journal file, generates and records the file fingerprint of specifying journal file, then terminating.
Particularly, the word of predetermined word joint number can be collected as file fingerprint from appointment journal file, judge that whether the content of specifying journal file to comprise is the same by file fingerprint.As front 100 bytes can be collected as file fingerprint from appointment journal file.
S205: judge the file fingerprint of specifying journal file current, whether consistent with the file fingerprint of specifying before journal file, if inconsistent, then perform S206; Otherwise, perform S210.
Particularly, if inconsistent, then illustrate and specify journal file to there occurs log rollback.
S206: mark specifies journal file generation rollback, using specifying the current document misregistration amount of journal file and filename to record as original side-play amount and old file name, obtains side-play amount record (old file name, original side-play amount).
Wherein, document misregistration amount represents that the daily record data before being positioned at this file side-play amount has all been collected.
Such as: when the access.log occurrence log rollback of apache2, access.log is by RNTO access.log.1, then newly-built go out an access.log log data, the current document misregistration amount of access.log is 3033, then side-play amount is recorded as: (old file name: access.log, original side-play amount: 3033).
S207: by specifying the current document misregistration amount of journal file to reset to 0, collecting the daily record data of specifying in journal file, regenerating and recording the file fingerprint of specifying journal file.
S208: judge whether the record having new filename corresponding to appointment journal file and old file name, if had, then perform S209; Otherwise, terminate.
Particularly, if before in_modify event occurs, first there occurs in_move_from event and in_move_to event, then can have the record of new filename corresponding to appointment journal file and old file name.As having the new filename record (cookie:123, old file name: access.log, new filename: access.log.1) in S215.
S209: according to original side-play amount, from the journal file that new filename is corresponding, collector journal data, then terminate.
Such as: according to side-play amount record (old file name: access.log, original side-play amount: the original side-play amount 3033 3033), from new filename record (cookie:123, old file name: access.log, new filename: access.log.1) in journal file corresponding to new filename access.log.1 in collector journal data.
S210: according to the document misregistration amount of specifying journal file current, collector journal data from appointment journal file, and after collector journal data, reset the document misregistration amount of specifying journal file.Then terminate.
S211: judge whether the file be moved out of needs the appointment journal file of collector journal data, if so, then performs S212; Otherwise, terminate.
S212: using the cookie of event and specify the current filename of journal file to record as old file name, obtains old file name record (cookie, old file name), then terminates.
Wherein, cookie is the numbering for connecting two events, and the numbering of two relevant events is the same.
Such as: when the access.log occurrence log rollback of apache2, access.log is by RNTO access.log.1, then newly-built go out an access.log log data, and the cookie of correspondence is 123, then now, old file name is recorded as: (cookie:123, old file name: access.log).
S213: obtain and move into cookie corresponding to file.
S214: by cookie, judges that moving into file is mobile under same catalogue coming, and if so, then performs S215; Otherwise, terminate.
Particularly, whether can there is identical cookie by the record of looking into in_move_from event, judge to move into that file and the file be moved out of be whether mobile under same catalogue comes.
S215: the filename moving into file being recorded as specifying the new filename of journal file, obtaining new filename record (cookie, old file name, new filename).
Such as: when the access.log occurrence log rollback of apache2, access.log is by RNTO access.log.1, then newly-built go out an access.log log data, and the cookie of correspondence is 123, then now, new filename is recorded as: (cookie:123, old file name: access.log, new filename: access.log.1).
S216: judge whether to have original side-play amount and old file name record, if had, then perform S209; Otherwise, terminate.
Particularly, during as performed this step, executed S206, namely has side-play amount record (old file name, original side-play amount), then can perform S209.
The method of the collector journal data described in the present embodiment, the Journaling File System of specifying is monitored by file change notification interface, when getting the event of specifying journal file to be written into daily record data, judge the file fingerprint of specifying journal file current, during with the file fingerprint of the appointment journal file of local record inconsistent (proof there occurs log rollback), obtain and the new filename corresponding according to appointment journal file, old file name and original side-play amount, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename, make it possible to collect the daily record data be recorded in the journal file after renaming, make it possible to collect complete complete daily record data.
As shown in Figure 3, be the structure drawing of device of the embodiment of the present application.A device for collector journal data, comprising:
Monitoring modular 301, for monitoring the Journaling File System of specifying by file change notification interface;
First acquisition module 302, for obtaining the event of specifying journal file to be written into daily record data;
First judge module 303, for judging the file fingerprint of specifying journal file current, whether consistent with the file fingerprint of specifying before journal file;
First collection module 304, if be inconsistent for the judged result of the first judge module 303, then to obtain and according to specifying corresponding new filename, old file name and the original side-play amount of journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename.
Preferably, see Fig. 4, this device also comprises:
Second acquisition module 305, specifies journal file by the event of rename for obtaining;
Logging modle 306, specifies for recording the new filename and old file name that journal file is corresponding.
Preferably, file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event of specifying journal file to be written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, journal file is specified to be comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
Preferably, the first collection module 304 comprises:
Record cell, if for inconsistent, records specifying the current document misregistration amount of journal file and filename as original side-play amount and old file name;
Processing unit, for by specifying the current document misregistration amount of journal file to reset to 0, collecting the daily record data of specifying in journal file, regenerating and recording the file fingerprint of specifying journal file;
Collector unit, for when it is determined that the presence of the record of new filename corresponding to appointment journal file and old file name, according to original side-play amount, collector journal data from the journal file that described new filename is corresponding.
Preferably, see Fig. 5, this device also comprises:
Second judge module 307, specify after journal file is written into the event of daily record data for obtaining, judge the file fingerprint of specifying journal file current, whether before consistent with the file fingerprint of specifying before journal file, whether first time is collected daily record data to judge to specify journal file;
Processing module 308, if for the judged result of the second judge module 307 be, appointment journal file is joined and collects in file set, 0 is set to by specifying the document misregistration amount of journal file, collect the daily record data of specifying in journal file, generate and record the file fingerprint of specifying journal file;
Notification module 309, if for the judged result of the second judge module 307 be, then notifies that the first judgement mould 303 pieces performs the described file fingerprint judging to specify journal file current, with the step whether file fingerprint of specifying before journal file is consistent.
Preferably, see Fig. 6, this device also comprises:
Second collection module 310, if be consistent for the judged result of the first judge module 303, then according to the document misregistration amount of specifying journal file current, collector journal data from appointment journal file, and after collector journal data, reset the document misregistration amount of specifying journal file.
The device of the collector journal data described in the present embodiment, the Journaling File System of specifying is monitored by file change notification interface, when getting the event of specifying journal file to be written into daily record data, judge the file fingerprint of specifying journal file current, during with the file fingerprint of the appointment journal file of local record inconsistent (proof there occurs log rollback), obtain and the new filename corresponding according to appointment journal file, old file name and original side-play amount, collector journal data the journal file corresponding from old file name and journal file corresponding to new filename, make it possible to collect the daily record data be recorded in the journal file after renaming, make it possible to collect complete complete daily record data.
Above-mentioned explanation illustrate and describes some preferred embodiments of the application, but as previously mentioned, be to be understood that the application is not limited to the form disclosed by this paper, should not regard the eliminating to other embodiments as, and can be used for other combinations various, amendment and environment, and can in invention contemplated scope described herein, changed by the technology of above-mentioned instruction or association area or knowledge.And the change that those skilled in the art carry out and change do not depart from the spirit and scope of the application, then all should in the protection domain of the application's claims.
Claims (12)
1. a method for collector journal data, is characterized in that, described method comprises:
The Journaling File System of specifying is monitored by file change notification interface;
Obtain the event of specifying journal file to be written into daily record data;
Judge the file fingerprint that described appointment journal file is current, whether consistent with the file fingerprint before described appointment journal file;
If inconsistent, then obtain and new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename.
2. the method for claim 1, is characterized in that, described monitored the Journaling File System of specifying by file change notification interface after, also comprise:
Obtain described appointment journal file by the event of rename;
Record new filename corresponding to described appointment journal file and old file name.
3. method as claimed in claim 2, it is characterized in that, described file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event that described appointment journal file is written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, described appointment journal file is comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
4. method as claimed in claim 2, it is characterized in that, described acquisition new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename, comprising:
Document misregistration amount current for described appointment journal file and filename are recorded as described original side-play amount and described old file name;
Document misregistration amount current for described appointment journal file is reset to 0, collects the daily record data in described appointment journal file, regenerate and record the file fingerprint of described appointment journal file;
When it is determined that the presence of the record of new filename corresponding to described appointed date will file and old file name, according to described original side-play amount, collector journal data from the journal file that described new filename is corresponding.
5. the method as described in claim as arbitrary in claim 1-4, it is characterized in that, after described acquisition appointment journal file is written into the event of daily record data, judge the file fingerprint that described appointment journal file is current, before whether consistent with the file fingerprint before described appointment journal file, also comprise:
Judge whether first time is collected daily record data to described appointment journal file;
If so, described appointment journal file is joined and collects in file set, the document misregistration amount of described appointment journal file is set to 0, collects the daily record data in described appointment journal file, generate and record the file fingerprint of described appointment journal file;
If not, then perform the file fingerprint that described judgement described appointment journal file is current, with the step whether file fingerprint before described appointment journal file is consistent.
6. method as claimed in claim 5, is characterized in that the file fingerprint that described judgement described appointment journal file is current, after whether consistent with the file fingerprint before described appointment journal file, also comprises:
If consistent, then current according to described appointment journal file document misregistration amount, collector journal data from described appointment journal file, and the document misregistration amount resetting described appointment journal file after collector journal data.
7. a device for collector journal data, is characterized in that, described device comprises:
Monitoring modular, for monitoring the Journaling File System of specifying by file change notification interface;
First acquisition module, for obtaining the event of specifying journal file to be written into daily record data;
First judge module, for judging the file fingerprint that described appointment journal file is current, whether consistent with the file fingerprint before described appointment journal file;
First collection module, if be inconsistent for the judged result of described first judge module, then obtain and new filename, old file name and the original side-play amount corresponding according to described appointment journal file, collector journal data the journal file corresponding from described old file name and journal file corresponding to described new filename.
8. device as claimed in claim 7, it is characterized in that, described device also comprises:
Second acquisition module, for obtaining described appointment journal file by the event of rename;
Logging modle, for recording new filename corresponding to described appointment journal file and old file name.
9. device as claimed in claim 8, it is characterized in that, described file change notification interface comprises: the file system monitoring FileSystemWatcher in the inotify in linux system, windows system;
Correspondingly, the event that described appointment journal file is written into daily record data comprises: the file in inotify has new write in_modify event, the change Changed event in FileSystemWatcher;
Correspondingly, described appointment journal file is comprised by the event of rename: the file in inotify shifts out in_move_from event and file moves into in_move_to event, the rename Renamed event in FileSystemWatcher.
10. device as claimed in claim 8, it is characterized in that, described first collection module comprises:
Record cell, if for inconsistent, records document misregistration amount current for described appointment journal file and filename as described original side-play amount and described old file name;
Processing unit, for document misregistration amount current for described appointment journal file is reset to 0, collects the daily record data in described appointment journal file, regenerates and records the file fingerprint of described appointment journal file;
Collector unit, for when it is determined that the presence of the record of new filename corresponding to described appointed date will file and old file name, according to described original side-play amount, collector journal data from the journal file that described new filename is corresponding.
Device as described in 11. claims as arbitrary in claim 7-10, it is characterized in that, described device also comprises:
Second judge module, specify after journal file is written into the event of daily record data for obtaining, judge the file fingerprint that described appointment journal file is current, before whether consistent with the file fingerprint before described appointment journal file, judge whether first time is collected daily record data to described appointment journal file;
Processing module, if for the judged result of described second judge module be, described appointment journal file is joined and collects in file set, the document misregistration amount of described appointment journal file is set to 0, collect the daily record data in described appointment journal file, generate and record the file fingerprint of described appointment journal file;
Notification module, if for the judged result of described second judge module be, then notify that described first judge module performs the current file fingerprint of described judgement described appointment journal file, with the step whether file fingerprint before described appointment journal file is consistent.
12. devices as claimed in claim 11, it is characterized in that, described device also comprises:
Second collection module, if be consistent for the judged result of described first judge module, then current according to described appointment journal file document misregistration amount, collector journal data from described appointment journal file, and the document misregistration amount resetting described appointment journal file after collector journal data.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310676525.2A CN104714878B (en) | 2013-12-11 | 2013-12-11 | A kind of method and device of collector journal data |
| HK15107341.4A HK1206838A1 (en) | 2013-12-11 | 2015-07-31 | Method for collecting log data and device thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310676525.2A CN104714878B (en) | 2013-12-11 | 2013-12-11 | A kind of method and device of collector journal data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104714878A true CN104714878A (en) | 2015-06-17 |
| CN104714878B CN104714878B (en) | 2017-11-21 |
Family
ID=53414237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310676525.2A Active CN104714878B (en) | 2013-12-11 | 2013-12-11 | A kind of method and device of collector journal data |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104714878B (en) |
| HK (1) | HK1206838A1 (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105955876A (en) * | 2016-05-13 | 2016-09-21 | 腾讯科技（深圳）有限公司 | Data monitoring processing method and apparatus |
| CN106250496A (en) * | 2016-08-02 | 2016-12-21 | 北京集奥聚合科技有限公司 | A kind of method and system of the data collection in journal file |
| CN108038253A (en) * | 2018-01-23 | 2018-05-15 | 北京奇安信科技有限公司 | A kind of processing method and processing device of log query |
| CN108062323A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | A kind of log read method and device |
| CN108733546A (en) * | 2018-04-02 | 2018-11-02 | 阿里巴巴集团控股有限公司 | A kind of log collection method, device and equipment |
| CN110378154A (en) * | 2019-07-24 | 2019-10-25 | 秒针信息技术有限公司 | A kind of method and device of file set completeness check |
| CN111856249A (en) * | 2020-07-24 | 2020-10-30 | 安测半导体技术(江苏)有限公司 | Chip test monitoring method, client and system |
| CN112506884A (en) * | 2020-12-10 | 2021-03-16 | 杭州安恒信息技术股份有限公司 | Log checking method, device, equipment and storage medium |
| CN112905106A (en) * | 2019-12-04 | 2021-06-04 | 贵州白山云科技股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
| CN117056288A (en) * | 2023-08-17 | 2023-11-14 | 齐鲁空天信息研究院 | Method and system for searching and downloading server file |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1361489A (en) * | 2000-12-28 | 2002-07-31 | 深圳市中兴通讯股份有限公司 | Automatic circular daily record information storing method and system |
| JP2008198039A (en) * | 2007-02-15 | 2008-08-28 | Hitachi Ltd | Log collection and management method |
| CN101561807A (en) * | 2008-04-18 | 2009-10-21 | 北京闻言科技有限公司 | Method for recording logs with file by flexibly planning catalog |
| CN102880718A (en) * | 2012-10-12 | 2013-01-16 | 互动在线（北京）科技有限公司 | Flexible log storage and acquisition method |
| CN102929998A (en) * | 2012-10-25 | 2013-02-13 | 浪潮(北京)电子信息产业有限公司 | File increment intercepting method and device |
| JP2013191012A (en) * | 2012-03-14 | 2013-09-26 | Hitachi Ltd | Computer log collection system and attachment/detachment type log storage device |
-
2013
- 2013-12-11 CN CN201310676525.2A patent/CN104714878B/en active Active
-
2015
- 2015-07-31 HK HK15107341.4A patent/HK1206838A1/en unknown
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1361489A (en) * | 2000-12-28 | 2002-07-31 | 深圳市中兴通讯股份有限公司 | Automatic circular daily record information storing method and system |
| JP2008198039A (en) * | 2007-02-15 | 2008-08-28 | Hitachi Ltd | Log collection and management method |
| CN101561807A (en) * | 2008-04-18 | 2009-10-21 | 北京闻言科技有限公司 | Method for recording logs with file by flexibly planning catalog |
| JP2013191012A (en) * | 2012-03-14 | 2013-09-26 | Hitachi Ltd | Computer log collection system and attachment/detachment type log storage device |
| CN102880718A (en) * | 2012-10-12 | 2013-01-16 | 互动在线（北京）科技有限公司 | Flexible log storage and acquisition method |
| CN102929998A (en) * | 2012-10-25 | 2013-02-13 | 浪潮(北京)电子信息产业有限公司 | File increment intercepting method and device |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105955876A (en) * | 2016-05-13 | 2016-09-21 | 腾讯科技（深圳）有限公司 | Data monitoring processing method and apparatus |
| CN105955876B (en) * | 2016-05-13 | 2020-08-14 | 腾讯科技（深圳）有限公司 | Data monitoring processing method and device |
| CN106250496A (en) * | 2016-08-02 | 2016-12-21 | 北京集奥聚合科技有限公司 | A kind of method and system of the data collection in journal file |
| CN108062323A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | A kind of log read method and device |
| CN108038253A (en) * | 2018-01-23 | 2018-05-15 | 北京奇安信科技有限公司 | A kind of processing method and processing device of log query |
| CN108038253B (en) * | 2018-01-23 | 2021-03-12 | 奇安信科技集团股份有限公司 | Log query processing method and device |
| CN108733546A (en) * | 2018-04-02 | 2018-11-02 | 阿里巴巴集团控股有限公司 | A kind of log collection method, device and equipment |
| CN110378154A (en) * | 2019-07-24 | 2019-10-25 | 秒针信息技术有限公司 | A kind of method and device of file set completeness check |
| CN110378154B (en) * | 2019-07-24 | 2021-04-02 | 秒针信息技术有限公司 | File set integrity checking method and device |
| CN112905106A (en) * | 2019-12-04 | 2021-06-04 | 贵州白山云科技股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN111856249A (en) * | 2020-07-24 | 2020-10-30 | 安测半导体技术(江苏)有限公司 | Chip test monitoring method, client and system |
| CN112506884A (en) * | 2020-12-10 | 2021-03-16 | 杭州安恒信息技术股份有限公司 | Log checking method, device, equipment and storage medium |
| CN114996335A (en) * | 2022-08-03 | 2022-09-02 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
| CN114996335B (en) * | 2022-08-03 | 2022-11-01 | 海看网络科技(山东)股份有限公司 | IPTV log real-time clustering analysis method |
| CN117056288A (en) * | 2023-08-17 | 2023-11-14 | 齐鲁空天信息研究院 | Method and system for searching and downloading server file |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104714878B (en) | 2017-11-21 |
| HK1206838A1 (en) | 2016-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104714878A (en) | Method and device for collecting log data | |
| US9934104B2 (en) | Metadata generation for incremental backup | |
| WO2023050705A1 (en) | Monitoring data management method and apparatus, electronic device and storage medium | |
| US8762944B2 (en) | Build process management system | |
| US8140573B2 (en) | Exporting and importing business objects based on metadata | |
| CN106992876A (en) | Cloud platform blog management method and system | |
| WO2015167603A1 (en) | Maintaining files in a retained file system | |
| CN110224874A (en) | A kind of processing method and processing device of equipment fault | |
| CN104636242A (en) | Method for automatically deleting repeated content in system logs on basis of Linux operating system | |
| CN105446825A (en) | Database test method and device | |
| CN110515795A (en) | A kind of monitoring method of big data component, device, electronic equipment | |
| CN105635301A (en) | Access log merging method and log processing server and system | |
| CN106354587A (en) | Mirror image server and method for exporting mirror image files of virtual machine | |
| CN109471901B (en) | Data synchronization method and device | |
| CN112579383A (en) | Method and device for simulating system fault | |
| CN116028469A (en) | Data security link tracking method and device | |
| CN115658391A (en) | Backup recovery method of WAL mechanism based on QianBase MPP database | |
| CN104317675A (en) | Disaster tolerance treatment method and device of applications | |
| CN111209138A (en) | Operation and maintenance method and device of data storage system | |
| CN114978963A (en) | Network system monitoring analysis method and device, electronic equipment and storage medium | |
| CN107608869A (en) | log recording method and device | |
| CN115129664B (en) | Data recording device, data file management method and apparatus | |
| CN115658626B (en) | Distributed network small file storage management method | |
| CN107590286A (en) | The management method and device of transaction information in cluster file system | |
| US20240103751A1 (en) | Garbage collection for object-based storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1206838 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: GR Ref document number: 1206838 Country of ref document: HK |