CN114584464B - Cloud platform full-automatic management log acquisition method and terminal - Google Patents
Cloud platform full-automatic management log acquisition method and terminal Download PDFInfo
- Publication number
- CN114584464B CN114584464B CN202210221852.8A CN202210221852A CN114584464B CN 114584464 B CN114584464 B CN 114584464B CN 202210221852 A CN202210221852 A CN 202210221852A CN 114584464 B CN114584464 B CN 114584464B
- Authority
- CN
- China
- Prior art keywords
- filebeat
- configuration file
- configuration
- server
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000008569 process Effects 0.000 claims abstract description 24
- 125000004122 cyclic group Chemical group 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 abstract description 3
- 206010048669 Terminal state Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
- H04L41/082—Configuration setting characterised by the conditions triggering a change of settings the condition being updates or upgrades of network functionality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a cloud platform full-automatic management log acquisition method, which comprises the following steps: judging Filebeat whether the configuration file is started, reading SideCar configuration files, updating log acquisition configuration according to heartbeat interval time during operation, adding the configuration files to be newly added to a filebeat configuration file directory, and closing the module. According to the cloud platform full-automatic management log acquisition method and the terminal, the acquisition configuration is automatically pulled from the server, the latest configuration file is dynamically loaded, and restarting is not needed in the period. Meanwhile, the log acquisition terminal sends the running state of the log acquisition terminal to the server, and whether the log acquisition terminal runs normally can be found through a monitoring state mode. And automatically detecting the new version of the acquisition terminal, automatically downloading, updating and restarting the new version, and the process does not need manual intervention of a user.
Description
Technical Field
The invention relates to the technical field of log acquisition terminal systems, in particular to a cloud platform full-automatic management log acquisition method and a terminal.
Background
With the development of cloud platforms, more and more service products provide software services for users through the cloud platforms to media, including databases, fort machines and the like which are commonly used.
In the prior art, a large number of various logs are generated when the service product runs, the running state and the behavior of the service product are recorded by the logs, and the cloud platform needs to collect, store and analyze the logs to know whether the service product has the problems of stability, performance and the like.
However, in the evolution process of service products, some log files which are not required to be collected originally need to be collected, one log is originally changed into a plurality of lines for printing, the original problem that a plurality of lines of matching rules are not applicable any more will affect the log collection terminal, the collection configuration of the log collection terminal needs to be reset, and then mirror images are made to provide service for the collection of the service product logs. Meanwhile, in the operation process of the acquisition terminal, the problem that the log cannot be normally acquired or cannot be normally sent to a back-end storage system after the log is acquired due to the fact that the operation is stopped or the problem occurs caused by some non-pit-capable factors is avoided. The problem found in the operation of the log acquisition component is repaired, the update cannot be uniformly performed after the new version is pushed out, and the update can be performed only by the user self-update or the user virtual machine invasion.
Disclosure of Invention
The invention aims to provide a cloud platform full-automatic management log acquisition method and a terminal, which are used for solving the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: a cloud platform full-automatic management log acquisition method and a terminal thereof comprise the following steps:
Judging Filebeat whether the operation is started or not, if not, starting Filebeat;
Reading SideCar a configuration file, obtaining a server URL, the interval time between the terminal nodeLabel and the heartbeat, and if the server URL is not configured, using a default URL; acquiring a current Filebeat version number;
During operation, updating log acquisition configuration is carried out according to heartbeat interval time, filebeat component states are obtained and reported to a server, and Filebeat version is automatically updated and restarted;
Adding the configuration file to be newly added to a filebeat configuration file directory, updating the content of the configuration file to be edited to the configuration file under the configuration file directory, deleting the configuration file to be deleted under the configuration file directory, and updating the local cache file to provide metadata for reading SideCar configuration file;
The module is closed, filebeat is closed by the hooks when the SideCar assembly is closed, and then itself is closed again.
Preferably, the procedure for updating the acquisition configuration is as follows:
a. Reading a local cache file at intervals of 10 times of heartbeat interval time to obtain the current file names and content md5 of all input files, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, the cyclic query result is compared with the profile name in the entry, and the profile which does not exist in the entry is placed in the newly added profile list.
Preferably, the process of acquiring Filebeat the component status and reporting to the server includes the following steps:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. And starting Filebeat the component, and reporting nodeLabel, virtual machine ip and running state information to the server after the starting is successful.
Preferably, the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. And comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window. Otherwise, entering a step 3;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. The old version Filebeat component is deleted from the local disk.
A cloud platform full-automatic management log acquisition terminal comprises a starting module, a data acquisition module, an updating acquisition module, a file configuration module and a closing module;
The data acquisition module acquires the URL of the server, the self terminal nodeLabel and the heartbeat interval time, the update acquisition module performs update log acquisition configuration according to the heartbeat interval time, the file configuration module adds the configuration file to be newly added to the filebeat configuration file directory, and the closing module is used for closing the state.
Preferably, the starting module is configured to determine Filebeat whether it is started, if not, then start Filebea.
Preferably, the data obtaining module is configured to read SideCar the configuration file, obtain the URL of the server, the interval time between the terminal nodeLabel and the heartbeat, and use a default URL if the URL of the server is not configured; the current Filebeat version number is obtained.
Preferably, the update collection module is used for carrying out update log collection configuration according to heartbeat interval time during operation, obtaining Filebeat component states, reporting the component states to a server, automatically updating Filebeat version and restarting;
The procedure for updating the acquisition configuration is as follows:
a. Reading a local cache file at intervals of 10 times of heartbeat interval time to obtain the current file names and content md5 of all input files, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, comparing the cyclic query result with the configuration file name in the entering parameter, and placing the configuration file which does not exist in the entering parameter into a newly added configuration file list;
The process steps of obtaining Filebeat the component state and reporting to the server are as follows:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. Starting Filebeat the component, and reporting nodeLabel, virtual machine ip and running state information to a server after the starting is successful;
the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. And comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window. Otherwise, entering a step 3;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. The old version Filebeat component is deleted from the local disk.
Preferably, the file configuration module is configured to add the configuration file to be newly added to the filebeat configuration file directory, update the content of the configuration file to be edited to the configuration file under the configuration file directory, delete the configuration file to be deleted under the configuration file directory, and update the local cache file to provide metadata for reading the SideCar configuration file.
Preferably, the closing module is used to close SideCar the assembly by hooking Filebeat and then closing itself.
Compared with the prior art, the invention has the beneficial effects that:
according to the cloud platform full-automatic management log acquisition method and the terminal, the acquisition configuration is automatically pulled from the server, the latest configuration file is dynamically loaded, and restarting is not needed in the period. Meanwhile, the log acquisition terminal sends the running state of the log acquisition terminal to the server, and whether the log acquisition terminal runs normally can be found through a monitoring state mode. Automatically detecting a new version of the acquisition terminal, automatically downloading, updating and restarting the new version, and avoiding manual intervention of a user in the process;
According to the invention, after the log file is changed or the acquisition file is changed, each service product does not need to be specially changed for log acquisition, and for the historical service product instance, the configuration change can be carried out under the condition of not invading a virtual machine of a user so as to adapt to new business requirements. Meanwhile, a powerful reference basis is provided for the problem of locating log acquisition through inquiring the terminal state history table, and data support is also provided for subsequent monitoring and alarming of the terminal running state. The automatic updating and downloading function of the collector component provides convenience for repairing the collector problem, and the updated version is smoothly updated without user participation and perception in the whole process.
Drawings
FIG. 1 is a diagram showing a SideCar send heartbeat, pull acquisition configuration, and auto-update timing.
Detailed Description
In order to make the objects, technical solutions, and advantages of the present invention more apparent, the embodiments of the present invention will be further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are some, but not all, embodiments of the present invention, are intended to be illustrative only and not limiting of the embodiments of the present invention, and that all other embodiments obtained by persons of ordinary skill in the art without making any inventive effort are within the scope of the present invention.
In the description of the present invention, it should be noted that the terms "center," "middle," "upper," "lower," "left," "right," "inner," "outer," "top," "bottom," "side," "vertical," "horizontal," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate description of the present invention and simplify the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "a," an, "" the first, "" the second, "" the third, "" the fourth, "" the fifth, "and the sixth" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
For purposes of brevity and description, the principles of the embodiments are described primarily by reference to examples. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one of ordinary skill in the art that the embodiments may be practiced without limitation to these specific details. In some instances, well-known methods and structures have not been described in detail so as not to unnecessarily obscure the embodiments. In addition, all embodiments may be used in combination with each other.
Referring to fig. 1, the present invention provides a technical solution: a cloud platform full-automatic management log acquisition method and a terminal are characterized by comprising the following steps:
Judging Filebeat whether the operation is started or not, if not, starting Filebeat;
Reading SideCar a configuration file, obtaining a server URL, the interval time between the terminal nodeLabel and the heartbeat, and if the server URL is not configured, using a default URL; acquiring a current Filebeat version number;
During operation, updating log acquisition configuration is carried out according to heartbeat interval time, filebeat component states are obtained and reported to a server, and Filebeat version is automatically updated and restarted;
The procedure for updating the acquisition configuration is as follows:
a. Reading a local cache file at intervals of 10 times of heartbeat interval time to obtain the current file names and content md5 of all input files, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, comparing the cyclic query result with the configuration file name in the entering parameter, and placing the configuration file which does not exist in the entering parameter into a newly added configuration file list;
The process steps of obtaining Filebeat the component state and reporting to the server are as follows:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. Starting Filebeat the component, and reporting nodeLabel, virtual machine ip and running state information to a server after the starting is successful;
the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. And comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window. Otherwise, entering a step 3;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. deleting the old version Filebeat component from the local disk;
Adding the configuration file to be newly added to a filebeat configuration file directory, updating the content of the configuration file to be edited to the configuration file under the configuration file directory, deleting the configuration file to be deleted under the configuration file directory, and updating the local cache file to provide metadata for reading SideCar configuration file;
The module is closed, filebeat is closed by the hooks when the SideCar assembly is closed, and then itself is closed again.
A cloud platform full-automatic management log acquisition terminal comprises a starting module, a data acquisition module, an updating acquisition module, a file configuration module and a closing module;
The data acquisition module acquires the URL of the server, the self terminal nodeLabel and the heartbeat interval time, the update acquisition module performs update log acquisition configuration according to the heartbeat interval time, the file configuration module adds the configuration file to be newly added to the filebeat configuration file directory, and the closing module is used for closing the state.
The startup module is configured to determine Filebeat whether startup has occurred, and if not, start Filebeat.
The data acquisition module is used for reading SideCar the configuration file, acquiring the URL of the server side, the interval time between the terminal nodeLabel and the heartbeat, and if the URL of the server side is not configured, using a default URL; the current Filebeat version number is obtained.
The update acquisition module is used for carrying out update log acquisition configuration according to the heartbeat interval time during operation, acquiring Filebeat component states, reporting the states to the server, automatically updating Filebeat version and restarting;
The procedure for updating the acquisition configuration is as follows:
a. Reading a local cache file at intervals of 10 times of heartbeat interval time to obtain the current file names and content md5 of all input files, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, comparing the cyclic query result with the configuration file name in the entering parameter, and placing the configuration file which does not exist in the entering parameter into a newly added configuration file list;
The process steps of obtaining Filebeat the component state and reporting to the server are as follows:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. Starting Filebeat the component, and reporting nodeLabel, virtual machine ip and running state information to a server after the starting is successful;
the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. And comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window. Otherwise, entering a step 3;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. The old version Filebeat component is deleted from the local disk.
The file configuration module is used for adding the configuration file to be newly added to the filebeat configuration file catalog, updating the content of the configuration file to be edited to the configuration file under the configuration file catalog, deleting the configuration file to be deleted under the configuration file catalog, and updating the local cache file to provide metadata for reading the SideCar configuration file.
The closing module is used to close SideCar the assembly Filebeat by means of a hook and then close itself.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (2)
1. The full-automatic management log acquisition method for the cloud platform is characterized by comprising the following steps of:
Judging Filebeat whether the operation is started or not, if not, starting Filebeat;
reading SideCar a configuration file, obtaining a server URL, and the time between the terminal nodeLabel and the heartbeat, and if the server URL is not configured, using a default URL; acquiring a current Filebeat version number;
During operation, updating log acquisition configuration is carried out according to heartbeat interval time, filebeat component states are obtained and reported to a server, and Filebeat version is automatically updated and restarted;
Adding the configuration file to be newly added to a filebeat configuration file directory, updating the content of the configuration file to be edited to the configuration file under the configuration file directory, deleting the configuration file to be deleted under the configuration file directory, and updating the local cache file to provide metadata for reading SideCar configuration file;
A closing module, closing Filebeat by hooks when closing SideCar the assembly, and then closing itself again;
The procedure for updating the acquisition configuration is as follows:
a. reading a local cache file to obtain the current file names and contents md5 of all input files at intervals of 10 times of heartbeat intervals, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, comparing the cyclic query result with the configuration file name in the entering parameter, and placing the configuration file which does not exist in the entering parameter into a newly added configuration file list;
The process steps of obtaining Filebeat the component state and reporting to the server are as follows:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, the virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. starting Filebeat the component, and reporting nodeLabel, the virtual machine ip and the running state information to a server after the starting is successful;
the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window; otherwise, entering a step c;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. The old version Filebeat component is deleted from the local disk.
2. A cloud platform full-automatic management log acquisition terminal, its characterized in that: the system comprises a starting module, a data acquisition module, an updating acquisition module, a file configuration module and a closing module;
The data acquisition module acquires the URL of the service end, the interval time between the terminal nodeLabel and the heartbeat, the update acquisition module performs update log acquisition configuration according to the interval time of the heartbeat, the file configuration module adds the configuration file to be newly added to the filebeat configuration file directory, and the closing module is used for closing the state;
The starting module is used for judging whether Filebeat is started or not, if yes, starting Filebeat;
The data acquisition module is used for reading SideCar the configuration file, acquiring the URL of the server side, the interval time between the terminal nodeLabel and the heartbeat, and if the URL of the server side is not configured, using a default URL; acquiring a current Filebeat version number;
The update acquisition module is used for carrying out update log acquisition configuration according to heartbeat interval time during operation, acquiring Filebeat component states, reporting the states to a server, automatically updating Filebeat version and restarting;
The procedure for updating the acquisition configuration is as follows:
a. reading a local cache file to obtain the current file names and contents md5 of all input files at intervals of 10 times of heartbeat intervals, and accessing a server to obtain the latest input configuration of the virtual machine, wherein the latest input configuration comprises newly added configuration, modified configuration and deleted configuration;
b. Inquiring all configurations of the starting state and md5 values according to the virtual machine label; if the query is empty, putting all the configuration file names in the input parameters into a deletion configuration file list; if the query is not empty, comparing all the configuration file names in the circularly entered parameters with the query result, and placing the configuration file names which are not present in the query result in a deleted configuration file list; the existing md5 values are not consistent and put into a modified configuration file list; if the query is not empty, comparing the cyclic query result with the configuration file name in the entering parameter, and placing the configuration file which does not exist in the entering parameter into a newly added configuration file list;
The process steps of obtaining Filebeat the component state and reporting to the server are as follows:
a. acquiring Filebeat current state by calling Filebeat interfaces at intervals of heartbeat intervals;
b. if the step a calls the interface to fail, the Filebeat is determined to be a death state, and if the call is successful, the state information is used;
c. sending nodeLabel, the virtual machine ip and state information to a server;
d. The server side receives the report information of the SideCar assembly and then inserts the report information into an operation state history table of the acquisition terminal;
e. If Filebeat is the dead state in the step b, the process number of the Filebeat component is obtained from the current operating system, and if so, the process is kill;
f. starting Filebeat the component, and reporting nodeLabel, the virtual machine ip and the running state information to a server after the starting is successful;
the process of automatically updating Filebeat version and restarting is as follows:
a. the interval is 20 times of the heartbeat interval time, and the service interface is called to obtain the latest Filebeat version;
b. comparing with the Filebeat version number of the current running state, if the version number of the server is the same as the current version number, ending the update task and waiting for the next execution of the update window; otherwise, entering a step c;
c. taking the latest version number obtained in the step a as a parameter request server side downloading interface to download Filebeat components matched with the version number;
d. closing Filebeat components in the current running state, and running the downloaded new version Filebeat components;
e. deleting the old version Filebeat component from the local disk;
The file configuration module is used for adding the configuration file to be newly added to the filebeat configuration file catalog, updating the content of the configuration file to be edited to the configuration file under the configuration file catalog, deleting the configuration file to be deleted under the configuration file catalog, and updating the local cache file to provide metadata for reading the SideCar configuration file;
The closing module is used to close SideCar the assembly Filebeat by a hook and then close itself.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221852.8A CN114584464B (en) | 2022-03-07 | 2022-03-07 | Cloud platform full-automatic management log acquisition method and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221852.8A CN114584464B (en) | 2022-03-07 | 2022-03-07 | Cloud platform full-automatic management log acquisition method and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114584464A CN114584464A (en) | 2022-06-03 |
| CN114584464B true CN114584464B (en) | 2024-06-18 |
Family
ID=81773858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210221852.8A Active CN114584464B (en) | 2022-03-07 | 2022-03-07 | Cloud platform full-automatic management log acquisition method and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114584464B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115576792A (en) * | 2022-11-24 | 2023-01-06 | 北京宝兰德软件股份有限公司 | Log collection system and method |
| CN116719702B (en) * | 2023-08-10 | 2024-03-26 | 深圳开源互联网安全技术有限公司 | Method and device for collecting open source information, electronic equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4652321B2 (en) * | 2006-12-25 | 2011-03-16 | Ｎｅｃフィールディング株式会社 | Log information collection system and method for maintenance target apparatus, and program |
| CN106959866B (en) * | 2016-01-08 | 2020-12-01 | 阿里巴巴集团控股有限公司 | Log collection client and upgrading method thereof |
| CN110727557A (en) * | 2019-09-23 | 2020-01-24 | 北京百度网讯科技有限公司 | Log collector control method, platform, electronic device and storage medium |
| CN110968478B (en) * | 2019-11-21 | 2023-04-25 | 掌阅科技股份有限公司 | Log acquisition method, server and computer storage medium |
| CN111639098A (en) * | 2020-05-11 | 2020-09-08 | 紫光云技术有限公司 | Cloud log management method |
| CN113157658B (en) * | 2021-05-13 | 2021-11-09 | 心动互动娱乐有限公司 | Client log collecting and distributing method and device and computer equipment |
| CN113760878A (en) * | 2021-08-03 | 2021-12-07 | 浪潮软件集团有限公司 | Micro-service architecture log analysis method and system based on domestic CPU and operating system |
-
2022
- 2022-03-07 CN CN202210221852.8A patent/CN114584464B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114584464A (en) | 2022-06-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114584464B (en) | Cloud platform full-automatic management log acquisition method and terminal | |
| CN101616018A (en) | The method of upgrading network management software and device | |
| US20060288056A1 (en) | File version management device, method, and program | |
| CN111190962B (en) | File synchronization method and device and local terminal | |
| CN108197270B (en) | Distributed file system data recovery method | |
| CN1459207A (en) | Method for over air mobile station management | |
| CN101317381A (en) | Ability management object maintenance, ability management method, system and terminal | |
| CN112463450B (en) | Incremental backup management method, system, electronic equipment and storage medium | |
| CN111654532A (en) | Centralized management system, method and device for configuration files | |
| CN111163340A (en) | IVI system remote LOG reporting method and device based on Internet of vehicles | |
| CN107682219B (en) | Android system-based dial testing method and device for testing number | |
| CN113407555A (en) | Operation data updating method and system of equipment handheld terminal | |
| CN110362548B (en) | Log extraction method and related equipment | |
| CN116010388A (en) | Data verification method, data acquisition server and data verification system | |
| CN115470192A (en) | Method for collecting and managing metadata of database | |
| CN111416842B (en) | Automatic resource cluster distribution and hot update system and method | |
| CN114928610A (en) | Data update notification method, data processing method, device, equipment and medium | |
| CN115002507A (en) | Video data updating method, device, equipment and readable storage medium | |
| CN113132790B (en) | Video dumping method and device | |
| CN111399929A (en) | Test application starting method and device | |
| CN115686583B (en) | Mass Internet of things equipment upgrading method and device, electronic equipment and storage medium | |
| JP2003085007A (en) | Digital information degradation method, device and program | |
| US20060069757A1 (en) | Automated PCN downloads | |
| CN115686942A (en) | PostgreSQL database misoperation recovery method, device, equipment and medium | |
| CN114911496A (en) | Update control method and device of cloud platform and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |