CN111835562B - Log uploading method, log inquiring method and system - Google Patents
Log uploading method, log inquiring method and system Download PDFInfo
- Publication number
- CN111835562B CN111835562B CN202010617963.1A CN202010617963A CN111835562B CN 111835562 B CN111835562 B CN 111835562B CN 202010617963 A CN202010617963 A CN 202010617963A CN 111835562 B CN111835562 B CN 111835562B
- Authority
- CN
- China
- Prior art keywords
- log
- initial
- uploading
- level
- uploaded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a log uploading method, a log inquiring method, a system, computer equipment and a storage medium. The method is applied to the edge server and comprises the following steps: responding to a log uploading request, and acquiring an initial log generated by terminal equipment, wherein the terminal equipment and an edge server are in the same communication cluster; if the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded; uploading the log to be uploaded to the cloud server. It can be understood that the initial log generated by the terminal equipment in the same communication cluster with the edge server is obtained through the edge server, and the corresponding screening strategy is selected to screen the initial log according to the type of the initial log, so that the log to be uploaded is obtained and uploaded, the deletion and filtration of the initial log are realized, the volume of uploading the log is reduced, the occupation of network bandwidth is reduced, and the stability and efficiency of uploading the log are improved.
Description
Technical Field
The present application relates to the field of digital information transmission technologies, and in particular, to a log uploading method, a log query method, and a system.
Background
With the development of internet technology, logs (logs) play an increasingly important role. The log is used to record specific information (including prompts, warnings, and error messages, etc.) in the computer program, which can be used to analyze the running process of the computer program.
In the edge computing scenario, when the application service is deployed at the edge, the application service continuously processes the data collected by the machine node, thereby generating a large amount of logs. However, since the network of the edge computing scenario is generally unstable, when a conventional log uploading method is used to upload a substantial amount of log data, a problem of log loss occurs, and the stability of log uploading is poor.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log uploading method, a log querying method, a system, a computer device, and a storage medium that can improve log uploading stability.
A log uploading method, the method being applied to an edge server, comprising:
responding to a log uploading request, and acquiring an initial log generated by terminal equipment, wherein the terminal equipment and the edge server are in the same communication cluster;
If the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded;
uploading the log to be uploaded to a cloud server.
In one embodiment, the type of the initial log includes a level of the initial log;
if the type of the initial log is a screening type, selecting a log screening policy corresponding to the screening type to screen the initial log to obtain a log to be uploaded, including:
if the initial log comprises a log with the level of error, acquiring the above log of the log with the level of error;
taking the log with the level of error and the log above as the log to be uploaded;
and/or the number of the groups of groups,
if the initial log comprises a log with a debug level, discarding the log with the debug level to obtain the log to be uploaded;
and/or the number of the groups of groups,
and if the initial log comprises the log with the level of the upload, taking the log with the level of the upload as the log to be uploaded.
In one embodiment, if the initial log includes a log with an error level, obtaining the above log of the log with the error level includes:
If the initial log comprises a log with the level of error, acquiring information of terminal equipment associated with the log with the level of error, wherein the log with the level of error is generated by the terminal equipment;
and acquiring the upper log of the log with the level error according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file, wherein the log file contains the log with the level error and the upper log thereof.
In one embodiment, the obtaining the above log of the log with the level error according to the information of the terminal device and the correspondence between the terminal device and the log file includes:
and if the log with the level error is the log after the round robin, acquiring the above log of the log with the level error from the round robin log file according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file.
In one embodiment, the obtaining, in response to the log upload request, an initial log generated by the terminal device includes:
responding to a log uploading request, and storing the received initial log sent by the terminal equipment in a message queue through a production thread of the message queue;
And consuming and obtaining the initial log from the message queue through a consumption thread of the message queue.
In one embodiment, the uploading the log to be uploaded to a cloud server includes:
acquiring network transmission parameters between the edge server and the cloud server;
determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm;
and uploading the log to be uploaded to a cloud server at the uploading speed after the current limiting.
In one embodiment, the network transmission parameters include a data transmission speed; the determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm comprises the following steps:
determining the adding speed of the tokens in a token bucket according to the data transmission speed and a preset token bucket algorithm, and adding the tokens into the token bucket according to the adding speed of the tokens;
when the log uploading request is received, the log uploading request is processed according to the number of added tokens in the token bucket, and the uploading rate of the log to be uploaded after the log to be uploaded is limited is determined.
In one embodiment, the method further comprises:
receiving a log query request for target service sent by the cloud server, wherein the log query request carries the current time;
acquiring a service list according to the log query request;
determining a terminal equipment identifier corresponding to the target service according to the service list;
and obtaining the log data of the target service from the log file with the generation time closest to the current time interval corresponding to the terminal equipment identifier, and uploading the log data to the cloud server.
A log query method, the method being applied to a cloud server, comprising:
sending a log query request aiming at a target service to an edge server, wherein the log query request carries the current time;
receiving log data of a target service uploaded by the edge server in response to the log query request, wherein the log data is obtained by the edge server according to the log query request, obtaining a resource service list, determining a terminal equipment identifier corresponding to the target service according to the resource service list, and obtaining a log file with the generation time corresponding to the terminal equipment identifier nearest to the current time interval; and when the type of the initial log generated by the terminal equipment is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log.
A log upload system comprising an edge server, the edge server comprising:
the uploading module is used for responding to a log uploading request, acquiring an initial log generated by the terminal equipment, selecting a log screening strategy corresponding to the screening type to screen the initial log if the type of the initial log is the screening type, obtaining a log to be uploaded, and uploading the log to be uploaded to a cloud server, wherein the terminal equipment and the edge server are in the same communication cluster.
In one embodiment, the edge server further comprises a storage management module;
the uploading module is further configured to send the received initial log sent by the terminal device to a storage management module of the edge server;
the storage management module is used for storing the initial log sent by the terminal equipment into a message queue through a production thread of the message queue;
the uploading module is further configured to obtain the initial log from the message queue through consumption threads of the message queue;
the edge server also comprises a log inquiry module and an index management module; the type of the initial log comprises the level of the initial log;
The uploading module is specifically configured to, if the initial log includes a log with an error level, obtain, by using the log query module, an uplink log of the log with the error level, and use the log with the error level and the uplink log thereof as the log to be uploaded;
the log query module is configured to obtain information of a terminal device associated with the log with the level error, and obtain the above log of the log with the level error according to the information of the terminal device and a correspondence between the terminal device and a log file obtained by the index management module, where the log with the level error is generated by the terminal device.
A computer device comprising a memory storing a computer program and a processor which when executing the computer program performs the steps of:
responding to a log uploading request, and acquiring an initial log generated by terminal equipment, wherein the terminal equipment and the edge server are in the same communication cluster;
if the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded;
Uploading the log to be uploaded to a cloud server.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
responding to a log uploading request, and acquiring an initial log generated by terminal equipment, wherein the terminal equipment and the edge server are in the same communication cluster;
if the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded;
uploading the log to be uploaded to a cloud server.
According to the log uploading method, the log inquiring system, the computer equipment and the storage medium, the initial log generated by the terminal equipment in the same communication cluster with the edge server is obtained through the edge server in response to the log uploading request, and then when the type of the initial log is a screening type, a corresponding log screening strategy is selected to screen the initial log, so that the log to be uploaded is obtained and uploaded. It can be understood that the application realizes the deletion and filtration of the initial log, reduces the volume of log uploading, reduces the occupation of network bandwidth, and improves the stability and efficiency of log uploading.
Drawings
FIG. 1 is an application environment diagram of a log upload method in one embodiment;
FIG. 2 is a flow diagram of a log uploading method in one embodiment;
FIG. 3 is a flow chart of a complementary scheme for selecting a log filtering policy corresponding to a filtering type to filter an initial log if the level type of the initial log is the filtering type in one embodiment;
FIG. 4 is a flow diagram of asynchronous transfer of logs using message queues in one embodiment;
FIG. 5 is a flow diagram of a real-time query log in one embodiment;
FIG. 6 is a flowchart of a log query method according to an embodiment;
FIG. 7 is an application framework diagram of a log upload method in one embodiment;
FIG. 8 is a block diagram of an edge server in one embodiment;
FIG. 9 is a block diagram of a log uploading device in one embodiment;
FIG. 10 is a block diagram illustrating a structure of a log query device according to an embodiment;
FIG. 11 is an internal block diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The log uploading method and the log inquiring method provided by the application can be applied to an application environment shown in figure 1. The terminal device 102 communicates with the edge server 104 through a network, and the edge server 104 communicates with the cloud server 106 through the network. The log data generated by the terminal device 102 is sent to the edge server 104, the edge server 104 gathers the log data to obtain an initial log, and filters the initial log according to the type of the initial log to obtain a log to be uploaded and uploads the log to the cloud server 106. The terminal device 102 may be, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices, and the edge server 104 and the cloud server 106 may be implemented by separate servers or a server cluster composed of multiple servers.
In an exemplary embodiment, as shown in fig. 2, a log uploading method is provided, and the method is applied to the edge server in fig. 1 for illustration, and includes the following steps:
step S202, an initial log generated by the terminal equipment is obtained in response to the log uploading request.
The terminal equipment and the edge server are in the same communication cluster. In one embodiment, the construction of the communication cluster of the edge server and the terminal equipment is realized by embedding a Kubernetes engine in the edge server and the terminal equipment.
In one embodiment, the log upload request may be an upload request of log data generated for the terminal device sent to the edge server. The edge server receives the log uploading request and responds to the terminal equipment to acquire an initial log generated by the terminal equipment. In another embodiment, the log uploading request may be an uploading request automatically generated by the edge server after receiving log data sent by the terminal device. In this case, the edge server responds to the log upload request by storing the initial log through the memory of the edge server and then reading or consuming the initial log therefrom.
In one embodiment, the initial log may be a log generated by each terminal device managed by the Kubernetes engine. For example, the initial log may be a log generated by each terminal device, and sent to the edge server by the terminal device. Further, a log collector can be embedded in a Kubernetes engine in the terminal device, and the log collector is used for collecting logs generated by each terminal device and sending the collected logs to the edge server. By adopting the log collector, log data generated by each terminal device can be managed in a centralized way, the times of sending the log to the edge server are reduced, and the IO pressure of the edge server is reduced.
Step S204, if the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded.
In the process of processing and transmitting the log, the log is generally classified for achieving efficient and accurate processing and transmission due to the large volume of the log. Different log classification strategies may be employed for different needs. In this step, a screen type is set for screening the initial log. The filter type is used to identify and filter the log of the type from among a plurality of log data. And, to different screening types, can select different screening strategies to reach diversified screening's purpose.
Specifically, after determining the type of the initial log, the edge server judges whether the type is a preset screening type, if so, selects a log screening strategy corresponding to the screening type to screen the initial log, and obtains the log to be uploaded. The log filtering policy includes deleting the log, obtaining the above log of the log, directly uploading the log without processing, and the like.
Step S206, uploading the log to be uploaded to the cloud server.
Specifically, the edge server uploads the log to be uploaded to the cloud server. Optionally, the edge server uploads the log to be uploaded to the cloud server through an HTTP interface exposed by the cloud server.
In the log uploading method, in response to a log uploading request, an initial log generated by a terminal device in the same communication cluster as the edge server is obtained through the edge server, and then when the type of the initial log is a screening type, a corresponding log screening strategy is selected to screen the initial log, so that a log to be uploaded is obtained and uploaded. It can be understood that the method realizes the deletion and filtration of the initial log, reduces the volume of log uploading, reduces the occupation of network bandwidth, and improves the stability and efficiency of log uploading.
To enhance the performance of the Kubernetes engine in a communication cluster, in an exemplary embodiment, the edge server is run in a depoyment manner by the Kubernetes engine. In the embodiment of the application, the edge server operates in a depoyment mode, so that only one edge server is deployed on the edge cluster (the communication cluster formed by the edge server and the terminal equipment) to provide service, and meanwhile, the service is not limited to operate on a certain terminal equipment node, but only needs to operate on the edge cluster.
In an exemplary embodiment, one possible implementation of obtaining an initial log generated by a terminal device is involved. On the basis of the above embodiment, step S202 may be specifically implemented by the following steps:
step S2022, receiving an initial log sent by the log collector.
The initial log is obtained by collecting log data sent by each terminal device by a log collector. The terminal device has a Kubernetes engine embedded therein. The log collector is embedded in the Kubernetes engine in the terminal device.
Specifically, firstly, a Kubernetes engine is deployed in the terminal device, so that the terminal device and the edge server together form an edge terminal relative to a cloud end where the cloud server is located. The method comprises the steps that a cluster is built at the edge end through a Kubernetes engine, a log collector is deployed at each terminal equipment node in the Kubernetes engine, log data generated by each terminal equipment node can be collected through the log collector, the collected log data are collected to obtain an initial log, the initial log is sent to an edge server, the edge server can receive the initial log conveniently, and screening, storage and uploading of subsequent initial logs are achieved.
In the embodiment of the application, the Kubernetess engine is adopted to manage the edge cluster, so that deployment implementation and high availability can be realized more conveniently.
In an exemplary embodiment, the log collector comprises a Fluent-bit collector. The Fluent-bit collector is embedded in a Kubernetes engine in the terminal equipment and operates in a DaemonSet mode, wherein the DaemonSet mode can enable each terminal equipment node on the Kubernetes cluster to operate an application service Pod. Specifically, the Fluent-bit collector uses the command mode of tail to acquire the log managed by the Kubernetes engine under/var/log/containers, wherein/var/log/containers/is the file path of the log stored on the terminal equipment node, and the path is uniformly managed by the Kubernetes engine. The Fluent-bit collector uses kubernetes filter to filter and format package the obtained log to obtain an initial log. The format package is to package related information of the Kubernetes engine, for example, after packaging a log, the log may contain label data such as k8s_container_name, k8s_pod, k8s_node_name, k8s_pod_name, and the like. And then, the Fluent-bit collector sends the packaged initial log to the edge server through an http output interface. The edge server may receive the initial log over an HTTP interface.
In the embodiment of the application, the Fluent-bit collector is adopted to collect the log data of each terminal equipment node, so that the resource consumption is lower, the strength of using the resource is lighter, and the method is more suitable for computing at the edge end.
In another exemplary embodiment, the log collector comprises a Fluentd collector. The Fluentd collector is embedded in the Kubernetes engine in the terminal device. In the embodiment of the application, the fluent d collector is adopted to collect the log data of each terminal equipment node, so that the stability is higher.
In an exemplary embodiment, the type of the initial log includes a level of the initial log. The level of the initial log may be set according to the actual functional requirement. In one embodiment, the level of the initial log includes one or more of error, info, debug, warn or upload. In one embodiment, the level of the initial log may be determined based on the particular fields contained in the initial log. Alternatively, the specific field may be a level name, for example, an "error" field is included in the log of the error level, an "info" field is included in the log of the info level, and so on. Based on this, in an exemplary embodiment, if the type of the initial log is a filtering type, a log filtering policy corresponding to the filtering type is selected to filter the initial log, so as to obtain a possible implementation manner of the log to be uploaded. On the basis of the above embodiment, step S204 may be specifically implemented by the following steps:
Step S212, if the initial log contains a log with the level of error, acquiring an uplink log of the log with the level of error;
step S214, taking the log with the level of error and the log above the log as the log to be uploaded.
Error indicates that a system error and abnormality occur, and the target operation cannot be completed normally.
Specifically, for logs of the level error, the edge server obtains the above log of the level error. Where the above log refers to the above content of the log. In one embodiment, assuming that a log with a level error is located on line 100 of the log file, the above log is all the logs from line 1 to line 99 in the log file. In another embodiment, the size of the above log may also be preconfigured, for example, the above log of the log with the level error is configured as 20 rows, and then the above log is the log of the first 20 rows of the log with the level error. Then, the edge server takes the log with the level error and the upper log thereof as logs to be uploaded and uploads the logs to the cloud server.
Optionally, if the initial log includes a log of error fields, the log is determined to be an error-level log. Of course, other specific fields may be set as the identification of the log with the identification level error, for example, the err field.
In the embodiment of the application, the log with the level of error is adopted, and the upper log matched with the log with the level of error is uploaded to the cloud server together under the condition that the log with the level of error records error contents, so that the log back-checking function is realized, and the positions of errors and anomalies of the system can be accurately positioned.
In an exemplary embodiment, one possible implementation of the above log of the log with the level error is obtained if the initial log includes the log with the level error. On the basis of the above embodiment, step S212 may be specifically implemented by the following steps:
step S2122, if the initial log contains a log with the level of error, acquiring information of a terminal device associated with the log with the level of error;
step S2124, obtaining the upper log of the log with the error level according to the information of the terminal device and the correspondence between the terminal device and the log file.
The log with the level error is generated by the associated terminal equipment. The log file contains logs with the level error and the above log.
Specifically, if the initial log includes a log with an error level, the edge server queries information of a terminal device associated with the log with the error level, for example, a terminal device ID, and establishes communication connection with the terminal device through ssh network protocol, and queries the above log of the log with the error level according to the corresponding relationship between the terminal device and the log file under/var/log/containers.
In an exemplary embodiment, one possible implementation manner of obtaining the above log of the log with the level error according to the information of the terminal device and the corresponding relation between the terminal device and the log file is related. On the basis of the above embodiment, step S2122 may be specifically implemented by the following steps:
in step S212a, if the log with the level error is a log after rotation, the context log of the log with the level error is obtained from the log file of rotation according to the information of the terminal device and the correspondence between the terminal device and the log file.
The log rotation is used for controlling the log file to be in a size. And if the data volume of the log in a certain log file reaches a preset threshold value, the log is rotated once so that a new log is written into the new log file.
Specifically, if the log with the level error is a log after rotation, the edge server acquires the above log of the log with the level error from the log file of rotation according to the information of the terminal device and the corresponding relation between the terminal device and the log file. Optionally, the edge server obtains the position of the log file of the round trip where the log with the level error is located according to the information of the terminal device and the hierarchical mapping relation of Namespace, node, pod, container, and queries and obtains the above log of the log with the level error from the log file of the round trip.
In the embodiment of the application, after the logs to be uploaded are rotated, the logs above the logs can still be accurately found, and the accuracy of log inquiry is improved. Further, the hierarchical mapping relation of Namespace, node, pod, container is used as an index to realize the inquiry of the log, so that the inquiry use efficiency of the edge server when uploading the log is improved.
In an exemplary embodiment, if the type of the initial log is a filtering type, a log filtering policy corresponding to the filtering type is selected to filter the initial log, so as to obtain another possible implementation manner of the log to be uploaded. On the basis of the above embodiment, step S204 may be specifically implemented by the following steps:
in step S222, if the initial log includes a log with a debug level, the log with a debug level is discarded to obtain a log to be uploaded.
The debug level is the lowest, and is used for knowing the running state of the system in more detail during debugging.
Specifically, if the initial log includes a log with a debug level, the edge server discards the log with the debug level to obtain a log to be uploaded. For the logs with the level of debug, a large number of debug logs are generated under the scene of edge calculation, but the logs with the level of debug are not required to be uploaded to the cloud server, so that the logs with the level of debug are required to be discarded, the deletion and filtration of the initial logs are realized, the transmission quantity of the logs is reduced, the occupation of network bandwidth is reduced, and the uploading efficiency of the logs is improved.
Alternatively, if the initial log includes a log of debug fields, the log is determined as a log of debug level. Of course, other specific fields may be set as the identification of the log with the identification level debug, for example, the deb field.
In the embodiment of the application, the logs which do not need to be uploaded and are in the debug level are discarded, so that the deletion and the filtration of the initial logs are realized, and the uploading efficiency of the logs can be effectively improved.
In an exemplary embodiment, if the type of the initial log is a filtering type, a log filtering policy corresponding to the filtering type is selected to filter the initial log, so as to obtain another possible implementation manner of the log to be uploaded. On the basis of the above embodiment, step S204 may be specifically implemented by the following steps:
in step S232, if the initial log includes the log with the level of upload, the log with the level of upload is used as the log to be uploaded.
Specifically, if the initial log includes a log with an uplink level, the edge server does not process the log with the uplink level, directly uses the log with the uplink level as a log to be uploaded, and uploads the log to the cloud server.
Optionally, if the initial log includes a log of the upload field, the log is determined to be a log of the level upload. Of course, other specific fields may be set as the identification of the log with the identification level of the upload, for example, upl field.
In the embodiment of the application, the log with the level of upload to be uploaded is uploaded to the cloud server, so that operation and maintenance personnel can call the log at any time, and the running state of the system can be known in time.
In an exemplary embodiment, step S202 may be specifically implemented by the following steps:
step S242, in response to the log uploading request, storing the received initial log sent by the terminal equipment in the message queue through the production thread of the message queue;
in step S244, the initial log is obtained from the consumption of the message queue by the consumption thread of the message queue.
The message queue is provided with a production thread and a consumption thread, so that asynchronous transmission of the log can be realized.
Specifically, when the edge server receives the log data sent by the terminal device and automatically generates a log uploading request, the edge server responds to the log uploading request and firstly stores an initial log, specifically, the edge server stores the initial log in a message queue through a production thread of the message queue. Optionally, if the stored data amount reaches the data amount threshold, deleting the stored history log. By regularly sorting the history log data, the high availability of storage is improved. And then, the edge server consumes and obtains an initial log from the message queue through a consumption thread of the message queue. Optionally, the consumption thread reads the log with the preset size according to the configuration information, and merges the log with the preset size into a network message to upload the network message to the cloud server, so that the network load pressure can be reduced.
In the embodiment of the application, the asynchronous uploading of the log is realized by adopting the message queue, particularly when the network condition is poor or the network bandwidth is insufficient, the asynchronous uploading can reduce the waiting time generated when the log writing speed is higher than the uploading speed, the stability and the reliability of the edge calculation can be improved, and the high-availability effect of the edge calculation can be realized by balancing the storage and the network resource.
In an exemplary embodiment, step S206 may be specifically implemented by the following steps:
step S2062, obtaining network transmission parameters between the edge server and the cloud server;
step S2064, determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm;
in step S2066, the log to be uploaded is uploaded to the cloud server at the uploading speed after the current limiting.
Wherein the network transmission parameters include at least one of data transmission speed, network bandwidth, timeout retransmission time, or round trip delay. The preset throttling algorithm includes at least one of a token bucket algorithm or a leaky bucket algorithm.
Specifically, the edge server firstly acquires network transmission parameters between the edge server and the cloud server, then determines the uploading speed of the log to be uploaded after the current limiting according to the network transmission parameters and a preset current limiting algorithm, and further uploads the log to be uploaded to the cloud server at the uploading speed after the current limiting.
Alternatively, in one embodiment, step S2064 may be implemented specifically by:
step S206a, determining the adding speed of the tokens in the token bucket according to the data transmission speed and a preset token bucket algorithm, and adding the tokens into the token bucket according to the adding speed of the tokens;
step S206b, when receiving the log uploading request, processing the log uploading request according to the number of added tokens in the token bucket, and determining the uploading rate after the log to be uploaded is limited.
Specifically, when uploading logs to be uploaded, the edge server can limit the uploading rate of the logs to be uploaded by adopting a token bucket algorithm, so that the influence of excessive network occupation on other edge computing services is prevented. Further, the edge server performs network fault tolerance processing when uploading the log, and achieves the effect of high availability of network processing.
To enable real-time querying of logs, in an exemplary embodiment, the method further comprises the steps of:
step S252, receiving a log query request for target service sent by a cloud server;
step S254, obtaining a service list according to the log query request;
step S256, determining a terminal equipment identifier corresponding to the target service according to the service list;
In step S258, log data of the target service is obtained from the log file with the latest generation time and current time interval corresponding to the terminal device identifier, and is uploaded to the cloud server.
The log inquiry request carries the current time.
Specifically, the edge server receives a log query request for target service sent by the cloud server, acquires a service list capable of being queried, and determines a terminal equipment identifier corresponding to the target service according to the service list, so that a log file closest to the current time interval is queried in the terminal equipment, and log data of the target service contained in the log file is obtained and uploaded to the cloud server. In one embodiment, the edge server determines the terminal equipment identifier corresponding to the target service according to the hierarchical mapping relationship of Namespace, node, pod, container, and queries the log file closest to the current time interval in the terminal equipment through ssh tail/var/log/containers.
In an exemplary embodiment, as shown in fig. 6, a log query method is provided, and the method is applied to the cloud server in fig. 1 for illustration, and includes the following steps:
step S302, a log query request for the target service is sent to the edge server.
Step S304, receiving the log data of the target service uploaded by the edge server in response to the log query request.
The log query request carries the current time. The log data is obtained by the edge server obtaining a resource service list according to the log query request, determining a terminal equipment identifier corresponding to the target service according to the resource service list, obtaining the log data from a log file with the generation time corresponding to the terminal equipment identifier nearest to the current time interval, and selecting a log screening strategy corresponding to the screening type to screen the initial log when the type of the initial log generated by the terminal equipment is the screening type.
Specifically, the cloud server sends a log query request for the target service to the edge server. The edge server receives a log query request for target service sent by the cloud server, acquires a service list capable of being queried, and determines a terminal equipment identifier corresponding to the target service according to the service list, so that a log file closest to the current time interval is queried in the terminal equipment, and log data of the target service contained in the log file is obtained and uploaded to the cloud server. In one embodiment, the edge server determines the terminal equipment identifier corresponding to the target service according to the hierarchical mapping relationship of Namespace, node, pod, container, and queries the log file closest to the current time interval in the terminal equipment through sshtail/var/log/containers.
According to the log query method, the log query request is sent to the edge server, so that the edge server queries the latest log data of the target service according to the configuration information and uploads the latest log data, and the cloud server can query the log to the edge server in real time.
An embodiment of the present disclosure is described below in conjunction with a specific application scenario, with particular reference to fig. 7 and 8.
In an exemplary embodiment, a log uploading system is provided, including an edge server, the edge server comprising:
and the uploading module is used for responding to the log uploading request, acquiring an initial log generated by the terminal equipment, selecting a log screening strategy corresponding to the screening type to screen the initial log if the type of the initial log is the screening type, obtaining a log to be uploaded, and uploading the log to be uploaded to the cloud server, wherein the terminal equipment and the edge server are in the same communication cluster.
Further, in an exemplary embodiment, the edge server further comprises a storage management module; the uploading module is also used for sending the received initial log sent by the terminal equipment to the storage management module of the edge server;
The storage management module is used for storing the initial log sent by the terminal equipment into the message queue through the production thread of the message queue;
the uploading module is further used for obtaining an initial log from the message queue through consumption threads of the message queue, wherein the type of the initial log comprises the level of the initial log;
the edge server also comprises a log inquiry module and an index management module;
the uploading module is specifically configured to obtain, if the initial log includes a log with an error level, an upper log of the log with the error level through the log query module, and take the log with the error level and the upper log thereof as a log to be uploaded;
the log query module is used for acquiring information of the terminal equipment associated with the log with the level error, and acquiring the above log of the log with the level error according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file acquired by the index management module, wherein the log with the level error is generated by the terminal equipment.
Specifically, as shown in fig. 7 and 8, on the edge cluster side:
after the Fluent-bit module is deployed successfully, a log generated by each terminal equipment node (including an application program for log generation) is collected. The Fluent-bit module is configured to acquire the log data managed by Kubernetes under/var/log/containers in a tail mode, package the relevant information of Kubernetes on the acquired initial log by using Kubernetes filter, and then send the acquired initial log to an HTTP interface of the edge server through the configured HTTP output.
The edge server deploys and runs a Kubernetes engine, and receives an initial log sent by the Fluent-bit module through an HTTP interface. The edge server comprises the following modules:
and the uploading module (Uploader) receives the initial log through the HTTP interface, sends the initial log to the storage management module for storage, and starts the multithreading to call the storage management module to consume the initial log and upload the initial log to the cloud. If the consumed initial log is matched with a preset screening condition, for example, the initial log comprises a log with the level of error, according to the consumption state of the log with the level of error, acquiring the above log of the log by using a log query management module, and uploading the log and the above log to a cloud server. Optionally, when the uploading module uploads the log to be uploaded to the cloud server, the uploading is performed asynchronously and in batches. Optionally, the token bucket algorithm is used for limiting the current, so that the influence of excessive network occupation on other edge computing services is prevented. Optionally, network fault tolerance processing is performed during log uploading, so that high availability of network processing is realized.
And a Storage management module (Storage-manager) which constructs a local message queue and temporary Storage, and is used for reading and writing the stored log and providing a read-write interface. And meanwhile, storage management is supported, the upper limit of capacity is configured, and if the use amount of storage reaches the upper limit or reaches configuration, the historical data is regularly tidied. Considering that the edge network may be poor, the speed of writing to the upload module may be greater than the speed of the upload module consuming the upload to the cloud server, and thus the problem needs to be solved by asynchronous message queues.
And the Log query management module (Log-query-manager) is used for carrying out Log query on the Log files. When the uploading module acquires the above log of a certain log by using the log query management module, the uploading module is connected to the terminal equipment node through ssh according to the information of the terminal equipment node of the log, queries/var/log/containers logs, supports the processing of the log rotation configured by Kubernetes, and if the log is processed by rotation, the log query management module calls the index management module to acquire the position of the rotated log file, and queries the content of the rotated log file to acquire the above log.
An Index-manager module (Index-manager) for providing a list of resource relationships at the Kubernetes cluster level. The index management module obtains Namespace, node, pod, container the mapping (map) by invoking Kubernetes API Server, generating cached data for use by other modules of the edge server or the HTTP interface.
Interface modules (APIs): the edge server includes three interfaces: log-receive interface: receiving a log data interface sent by a fluent-bitHTTP output; the/closed-query-list interface: the cloud service system comprises a service interface provided for a cloud, and a resource service list which can be currently queried in real time is acquired through the interface; close-query-log interface: a service interface provided to the cloud server through which a service is specified, and a log of the service (Pod/Container) is queried.
On the cloud cluster side:
cloud gateway (Cloud gateway) receives logs uploaded by edge servers through HTTP interfaces. The cloud gateway is constructed for better expansibility and stability when the edge computing devices are more, so as to support better traffic management, monitoring, degradation and other services. The cloud gateway directly asynchronously rewrites the log to Kafka. The edge server can be responded more quickly in an asynchronous mode, and the edge network is prevented from being blocked.
Kafka, for reducing the load of the log storage service ElasticSearch (ES) at high concurrency.
Kafka Consumer, writes the log data of Kafka to the elastic search while configuring the ES index based on the log, currently the ES index is configured per service, day.
Elastic search: the method is used for storing log data, constructing an ES index and conveniently inquiring historical log data.
Kibana: after Kibana configures the ES, the log can be filtered by selecting the ES index and the Kubernetesfilter of the log.
When the Cloud server queries logs in real time, the Cloud server selects an edge cluster through Cloud WEBUI, and queries an interface exposed by an edge server deployed in the edge cluster. The service of the edge server on the Kubernetes is queried in the modes of VPN IP and KuberneteNodePort. Specifically, firstly, a service list capable of being queried is obtained through a/closed-query-list interface. And then, calling an index management module to acquire the mapping relation of the whole resource and returning the mapping relation to the cloud server as a response. And then inquiring a real-time log of a certain service through a/closed-query-log interface, inquiring the log of the service through a log inquiry management module after the edge server receives the request, acquiring a terminal equipment node where the service is located through all resource relations provided by an index management module by the log inquiry management module, and acquiring the latest log data in a ssh tail/var/log/containers/lower log file mode.
It should be understood that, although the steps in the flowcharts of fig. 2-6 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2-6 may include multiple steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of the steps or stages in other steps or other steps.
In an exemplary embodiment, as shown in fig. 9, there is provided a log uploading apparatus for an edge server, including: a log acquisition module 402, a log screening module 404, and a log upload module 406, wherein:
the log obtaining module 402 is configured to obtain an initial log generated by a terminal device in response to a log uploading request, where the terminal device and an edge server are in the same communication cluster.
The log filtering module 404 is configured to select a log filtering policy corresponding to the filtering type to filter the initial log if the type of the initial log is the filtering type, so as to obtain a log to be uploaded.
The log uploading module 406 is configured to upload a log to be uploaded to a cloud server.
In the log uploading device, in response to a log uploading request, an initial log generated by a terminal device in the same communication cluster as the edge server is obtained through the edge server, and then when the type of the initial log is a screening type, a corresponding log screening strategy is selected to screen the initial log, so that a log to be uploaded is obtained and uploaded. It can be understood that the device realizes the deletion and filtration of the initial log, reduces the volume of log uploading, reduces the occupation of network bandwidth, and improves the stability and efficiency of log uploading.
In an exemplary embodiment, the log filtering module 404 is specifically configured to obtain the above log of the log with the level error if the initial log includes the log with the level error; taking the log with the level error and the upper log as the log to be uploaded; and/or if the initial log comprises a log with the level of debug, discarding the log with the level of debug to obtain a log to be uploaded; and/or if the initial log contains the log with the level of the upload, taking the log with the level of the upload as the log to be uploaded.
In an exemplary embodiment, the log filtering module 404 is specifically configured to obtain, if the initial log includes a log with an error level, information of a terminal device associated with the log with the error level, where the log with the error level is generated by the terminal device; and acquiring the upper log of the log with the level error according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file, wherein the log file comprises the log with the level error and the upper log thereof.
In an exemplary embodiment, the log filtering module 404 is specifically configured to obtain, if the log with the level error is a log after rotation, the above log of the log with the level error from the log file of rotation according to the information of the terminal device and the correspondence between the terminal device and the log file.
In an exemplary embodiment, the log obtaining module 402 is specifically configured to store, in a message queue, the received initial log sent by the terminal device in response to a log upload request, through a production thread of the message queue; and consuming and obtaining an initial log from the message queue through a consumption thread of the message queue.
In an exemplary embodiment, the log uploading module 406 is specifically configured to obtain network transmission parameters between the edge server and the cloud server; determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm; uploading the log to be uploaded to the cloud server at the uploading speed after the current limiting.
In an exemplary embodiment, the log uploading module 406 is specifically configured to determine an adding speed of the token in the token bucket according to the data transmission speed and a preset token bucket algorithm, and add the token to the token bucket according to the adding speed of the token; when a log uploading request is received, the log uploading request is processed according to the number of added tokens in the token bucket, and the uploading rate of the log to be uploaded after the log is limited is determined.
For specific limitation of the log uploading device, reference may be made to the limitation of the log uploading method hereinabove, and the description thereof will not be repeated here. The various modules in the log uploading device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In an exemplary embodiment, as shown in fig. 10, there is provided a log query apparatus for a cloud server, including: a request sending module 502, a log receiving module 504, wherein:
the request sending module 502 is configured to send a log query request for the target service to the edge server, where the log query request carries the current time.
The log receiving module 504 is configured to receive log data of a target service uploaded by an edge server in response to a log query request, where the log data is obtained by the edge server according to the log query request, obtaining a resource service list, determining a terminal device identifier corresponding to the target service according to the resource service list, and obtaining a log file with a generation time corresponding to the terminal device identifier and closest to a current time interval; and when the type of the initial log generated by the terminal equipment is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log.
In the log query device, the log query request is sent to the edge server, so that the edge server queries the latest log data of the target service according to the configuration information and uploads the latest log data, and the cloud server can query the log to the edge server in real time.
The specific limitation of the log query device can be referred to the limitation of the log query method hereinabove, and will not be described herein. The various modules in the log querying device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In an exemplary embodiment, a computer device is provided, which may be a server, and an internal structure thereof may be as shown in fig. 11. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a log uploading method.
It will be appreciated by those skilled in the art that the structure shown in FIG. 11 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above. Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (11)
1. A log uploading method, wherein the method is applied to an edge server, and comprises the following steps:
responding to a log uploading request, acquiring an initial log generated by terminal equipment, wherein the terminal equipment and the edge server are in the same communication cluster, and the initial log is obtained by collecting log data sent by each terminal equipment by a log collector;
If the type of the initial log is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log to obtain a log to be uploaded, wherein the log screening strategy comprises deleting the log, acquiring the above log of the log and directly uploading the log without processing;
uploading the log to be uploaded to a cloud server.
2. The method of claim 1, wherein the type of initial log comprises a level of initial log;
if the type of the initial log is a screening type, selecting a log screening policy corresponding to the screening type to screen the initial log to obtain a log to be uploaded, including:
if the initial log comprises a log with the level of error, acquiring the above log of the log with the level of error;
taking the log with the level of error and the log above as the log to be uploaded;
and/or the number of the groups of groups,
if the initial log comprises a log with a debug level, discarding the log with the debug level to obtain the log to be uploaded;
and/or the number of the groups of groups,
and if the initial log comprises the log with the level of the upload, taking the log with the level of the upload as the log to be uploaded.
3. The method according to claim 2, wherein if the initial log includes a log with an error level, obtaining the upper log of the log with the error level includes:
if the initial log comprises a log with the level of error, acquiring information of terminal equipment associated with the log with the level of error, wherein the log with the level of error is generated by the terminal equipment;
and acquiring the upper log of the log with the level error according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file, wherein the log file contains the log with the level error and the upper log thereof.
4. The method according to claim 3, wherein the obtaining the context log of the log with the level error according to the information of the terminal device and the correspondence between the terminal device and the log file includes:
and if the log with the level error is the log after the round robin, acquiring the above log of the log with the level error from the round robin log file according to the information of the terminal equipment and the corresponding relation between the terminal equipment and the log file.
5. The method according to claim 1, wherein the obtaining, in response to the log upload request, an initial log generated by the terminal device includes:
responding to a log uploading request, and storing the received initial log sent by the terminal equipment in a message queue through a production thread of the message queue;
and consuming and obtaining the initial log from the message queue through a consumption thread of the message queue.
6. The method of claim 1, wherein the uploading the log to be uploaded to a cloud server comprises:
acquiring network transmission parameters between the edge server and the cloud server;
determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm;
and uploading the log to be uploaded to a cloud server at the uploading speed after the current limiting.
7. The method of claim 6, wherein the network transmission parameters include a data transmission speed; the determining the uploading speed of the log to be uploaded after limiting the current according to the network transmission parameters and a preset current limiting algorithm comprises the following steps:
Determining the adding speed of the tokens in a token bucket according to the data transmission speed and a preset token bucket algorithm, and adding the tokens into the token bucket according to the adding speed of the tokens;
when the log uploading request is received, the log uploading request is processed according to the number of added tokens in the token bucket, and the uploading rate of the log to be uploaded after the log to be uploaded is limited is determined.
8. The method according to claim 1, wherein the method further comprises:
receiving a log query request for target service sent by the cloud server, wherein the log query request carries the current time;
acquiring a service list according to the log query request;
determining a terminal equipment identifier corresponding to the target service according to the service list;
and obtaining the log data of the target service from the log file with the generation time closest to the current time interval corresponding to the terminal equipment identifier, and uploading the log data to the cloud server.
9. The log query method is characterized by being applied to a cloud server and comprising the following steps:
sending a log query request aiming at a target service to an edge server, wherein the log query request carries the current time;
Receiving log data of a target service uploaded by the edge server in response to the log query request, wherein the log data is obtained by the edge server according to the log query request, obtaining a resource service list, determining a terminal equipment identifier corresponding to the target service according to the resource service list, and obtaining a log file with the generation time corresponding to the terminal equipment identifier nearest to the current time interval; when the type of the initial log generated by the terminal equipment is a screening type, selecting a log screening strategy corresponding to the screening type to screen the initial log, wherein the initial log is obtained by collecting log data sent by the terminal equipment by a log collector, and the log screening strategy comprises deleting the log, obtaining the above log of the log and directly uploading the log without processing.
10. A log upload system comprising an edge server, the edge server comprising:
the system comprises an uploading module, a log acquisition module and a cloud server, wherein the uploading module is used for responding to a log uploading request to acquire an initial log generated by a terminal device, if the type of the initial log is a screening type, a log screening strategy corresponding to the screening type is selected to screen the initial log to obtain a log to be uploaded, and the log to be uploaded is uploaded to the cloud server, wherein the terminal device and the edge server are in the same communication cluster, the initial log is obtained by summarizing log data sent by each terminal device and received by a log collector, and the log screening strategy comprises log deletion, log acquisition and direct uploading without processing of the log.
11. The system of claim 10, wherein the edge server further comprises a storage management module;
the uploading module is further configured to send the received initial log sent by the terminal device to a storage management module of the edge server;
the storage management module is used for storing the initial log sent by the terminal equipment into a message queue through a production thread of the message queue;
the uploading module is further configured to obtain the initial log from the message queue through consumption threads of the message queue;
the edge server also comprises a log inquiry module and an index management module; the type of the initial log comprises the level of the initial log;
the uploading module is specifically configured to, if the initial log includes a log with an error level, obtain, by using the log query module, an uplink log of the log with the error level, and use the log with the error level and the uplink log thereof as the log to be uploaded;
the log query module is configured to obtain information of a terminal device associated with the log with the level error, and obtain the above log of the log with the level error according to the information of the terminal device and a correspondence between the terminal device and a log file obtained by the index management module, where the log with the level error is generated by the terminal device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010617963.1A CN111835562B (en) | 2020-07-01 | 2020-07-01 | Log uploading method, log inquiring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010617963.1A CN111835562B (en) | 2020-07-01 | 2020-07-01 | Log uploading method, log inquiring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111835562A CN111835562A (en) | 2020-10-27 |
| CN111835562B true CN111835562B (en) | 2023-10-17 |
Family
ID=72899924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010617963.1A Active CN111835562B (en) | 2020-07-01 | 2020-07-01 | Log uploading method, log inquiring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111835562B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745291A (en) * | 2020-12-23 | 2022-07-12 | 北京国双科技有限公司 | Abnormal data cloud method and device, electronic equipment and storage medium |
| CN113360468A (en) * | 2021-05-12 | 2021-09-07 | 武汉联影医疗科技有限公司 | Log data uploading method and device, computer equipment and readable storage medium |
| CN113742031B (en) * | 2021-08-27 | 2022-11-04 | 北京百度网讯科技有限公司 | Node state information acquisition method and device, electronic equipment and readable storage medium |
| CN114826774B (en) * | 2022-06-02 | 2024-01-30 | 天翼数字生活科技有限公司 | Log request control method, server side and system for terminal equipment |
| CN115378804B (en) * | 2022-08-19 | 2024-06-14 | 深圳前海环融联易信息科技服务有限公司 | Log acquisition method, device, equipment and medium based on cloud edge cooperation |
| CN116401275B (en) * | 2023-03-14 | 2023-11-14 | 宁波浙鼎教育科技有限公司 | Cloud data processing method, cloud data processing system, storage medium and intelligent terminal |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7155514B1 (en) * | 2002-09-12 | 2006-12-26 | Dorian Software Creations, Inc. | Apparatus for event log management |
| CN101969386A (en) * | 2010-11-09 | 2011-02-09 | 道有道(北京)科技有限公司 | Log acquisition device and log acquisition method |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN105389314A (en) * | 2014-09-04 | 2016-03-09 | 中芯国际集成电路制造(上海)有限公司 | Log file query system and query method |
| CN105703932A (en) * | 2014-11-28 | 2016-06-22 | 广州华多网络科技有限公司 | Log recording and uploading method, log recording and reception method, terminal, and server |
| JP2016177594A (en) * | 2015-03-20 | 2016-10-06 | 株式会社リコー | Communication terminal device, communication system, communication method and program |
| CN106033322A (en) * | 2015-03-17 | 2016-10-19 | 北京元心科技有限公司 | Method and device for data storage |
| WO2016169145A1 (en) * | 2015-04-21 | 2016-10-27 | 中兴通讯股份有限公司 | Cloud terminal log processing method, device and system |
| CN107809337A (en) * | 2017-11-17 | 2018-03-16 | 深圳泉眼体育运营管理有限公司 | A kind of daily record method for uploading and device |
| CN109617726A (en) * | 2018-12-14 | 2019-04-12 | 深圳壹账通智能科技有限公司 | Error log acquisition method and server |
| CN110034610A (en) * | 2019-05-10 | 2019-07-19 | 国家电网有限公司 | A kind of transformer station measurement and control device redundancy configuration method and system |
| CN110198275A (en) * | 2018-03-28 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of flow control methods, system, server and storage medium |
| CN110995611A (en) * | 2019-12-20 | 2020-04-10 | 创盛视联数码科技（北京）有限公司 | Distributed current limiting method for high concurrency request |
| CN111190871A (en) * | 2019-12-18 | 2020-05-22 | 中移（杭州）信息技术有限公司 | Log generation method and device, computer equipment and storage medium |
| CN111352902A (en) * | 2020-02-28 | 2020-06-30 | 中国平安人寿保险股份有限公司 | Log processing method and device, terminal equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8868601B2 (en) * | 2009-08-17 | 2014-10-21 | International Business Machines Corporation | Distributed file system logging |
-
2020
- 2020-07-01 CN CN202010617963.1A patent/CN111835562B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7155514B1 (en) * | 2002-09-12 | 2006-12-26 | Dorian Software Creations, Inc. | Apparatus for event log management |
| CN101969386A (en) * | 2010-11-09 | 2011-02-09 | 道有道(北京)科技有限公司 | Log acquisition device and log acquisition method |
| CN105389314A (en) * | 2014-09-04 | 2016-03-09 | 中芯国际集成电路制造(上海)有限公司 | Log file query system and query method |
| CN105703932A (en) * | 2014-11-28 | 2016-06-22 | 广州华多网络科技有限公司 | Log recording and uploading method, log recording and reception method, terminal, and server |
| CN106033322A (en) * | 2015-03-17 | 2016-10-19 | 北京元心科技有限公司 | Method and device for data storage |
| JP2016177594A (en) * | 2015-03-20 | 2016-10-06 | 株式会社リコー | Communication terminal device, communication system, communication method and program |
| WO2016169145A1 (en) * | 2015-04-21 | 2016-10-27 | 中兴通讯股份有限公司 | Cloud terminal log processing method, device and system |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN107809337A (en) * | 2017-11-17 | 2018-03-16 | 深圳泉眼体育运营管理有限公司 | A kind of daily record method for uploading and device |
| CN110198275A (en) * | 2018-03-28 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of flow control methods, system, server and storage medium |
| CN109617726A (en) * | 2018-12-14 | 2019-04-12 | 深圳壹账通智能科技有限公司 | Error log acquisition method and server |
| CN110034610A (en) * | 2019-05-10 | 2019-07-19 | 国家电网有限公司 | A kind of transformer station measurement and control device redundancy configuration method and system |
| CN111190871A (en) * | 2019-12-18 | 2020-05-22 | 中移（杭州）信息技术有限公司 | Log generation method and device, computer equipment and storage medium |
| CN110995611A (en) * | 2019-12-20 | 2020-04-10 | 创盛视联数码科技（北京）有限公司 | Distributed current limiting method for high concurrency request |
| CN111352902A (en) * | 2020-02-28 | 2020-06-30 | 中国平安人寿保险股份有限公司 | Log processing method and device, terminal equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111835562A (en) | 2020-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111835562B (en) | Log uploading method, log inquiring method and system | |
| CN111131379B (en) | Distributed flow acquisition system and edge calculation method | |
| KR102337092B1 (en) | Traffic measurement method, device, and system | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| US11573725B2 (en) | Object migration method, device, and system | |
| CN108287905B (en) | Method for extracting and storing network flow characteristics | |
| WO2018072687A1 (en) | Resource scheduling method and apparatus, and filtered scheduler | |
| CN111966289B (en) | Partition optimization method and system based on Kafka cluster | |
| US20180295029A1 (en) | Managing groups of servers | |
| US20100229182A1 (en) | Log information issuing device, log information issuing method, and program | |
| US11888744B2 (en) | Spin-leaf network congestion control method, node, system, and storage medium | |
| CN106713378B (en) | Method and system for providing service by multiple application servers | |
| CN113596150B (en) | Message pushing method, device, computer equipment and storage medium | |
| US20240039995A1 (en) | Data access system and method, device, and network adapter | |
| CN106959820A (en) | A kind of data extraction method and system | |
| CN111064591B (en) | Data aggregation method, device, equipment, storage medium and system | |
| CN109982034A (en) | Access request processing method and processing device in video monitoring system | |
| CN116360954A (en) | Industrial Internet of things management and control method and system based on cloud edge cooperative technology | |
| CN112019604B (en) | Edge data transmission method and system | |
| CN112737980B (en) | Time-based network slice resource dynamic partitioning method and device | |
| CN115883330B (en) | Alarm event processing method, system, equipment and storage medium | |
| CN117240734A (en) | Cloud edge cooperation method, cloud edge cooperation system, computer equipment and storage medium | |
| CN114466031B (en) | CDN system node configuration method, device, equipment and storage medium | |
| CN115499514A (en) | Data storage service access method, computing device and computer storage medium | |
| CN111756798B (en) | Service scheduling method, device, equipment and storage medium based on gateway cascade |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |