WO2021096346A1

WO2021096346A1 - A computer-implemented system for management of container logs and its method thereof

Info

Publication number: WO2021096346A1
Application number: PCT/MY2020/050099
Authority: WO
Inventors: Rajendar KANDAR; Mohammad Fairus BIN KHALID; Bukhary Ikhwan BIN ISMAIL; Hong Hoe ONG
Original assignee: Mimos Berhad
Priority date: 2019-11-15
Filing date: 2020-10-08
Publication date: 2021-05-20

Abstract

The present invention discloses a computer-implemented system and method for managing container logs. The system comprises an orchestration module (1) linked to at least one node (2) having a plurality of containers (3) stored therewithin, in which the orchestration module (1) manages the container logs which are stored in the plurality of containers (3). The system further comprises a request handler (5), a log status monitoring module (6), a log management module (7) and a log volume prediction module (8) for managing the container logs. The system executes the steps as follows: identifying log dependencies among the container logs, obtaining log patterns based on the log dependencies, detecting duplication of container logs based on the log patterns, verifying log policy of the different container logs, initiating resource control on the container logs based on their respective log policy and estimating log volume of each container (3).

Description

A COMPUTER-IMPLEMENTED SYSTEM FOR MANAGEMENT OF CONTAINER

LOGS AND ITS METHOD THEREOF

FIELD OF INVENTION

The invention relates to a computer-implemented system for managing container logs. More particularly, the invention relates to a system for detecting container log dependency and predicting its log volume and its method thereof.

BACKGROUND OF THE INVENTION

In term of computer industries, a container is a standard unit of application that packages up container logs and all its dependencies so the application runs quickly and reliably from one computing environment to another. The use of container-based application has become popular due to a higher demand for faster development and deployment of applications. A container also includes everything necessary to run the application, such as files and libraries.

To manage the containers of the container-based application, several processes such as scheduling, application security, storage and monitoring of the containers are required. These processes are known as container management, which is the process of organizing, adding or replacing large numbers of containers. Due to the complexity of the container-based application, an orchestrator is introduced to automate the deployment, management, scaling, networking, and availability of container-based applications. Such management system is required by users for proper management of resources and troubleshooting.

However, the currently available container management system is not comprehensive to manage the containers effectively. The existing system is unable to analyse container logs based on dependencies of the container logs, lack of resource control based on log policy and unable to predict log volumes of the containers. The inadequacy of such management system causes the container logs captured at its running host to occupy a definite volume of storage and the volume grows over the period of time. Subsequently, this may cause the container- based application to experience slowdown. There are a few patented technologies over the prior art relating to the management system. US20140280961A1 discloses a virtualization environment adapted for development and deployment of at least one software workload, in which the virtualization environment is suitable for managing cloud services, applications, platforms and infrastructures. Most of the enterprises are currently unable to use cloud infrastructure due to a lack of security, control, and manageability of the computing capacity rented from the cloud infrastructure providers. To solve the above-mentioned problem, the virtualization environment provides a metamodel framework that allows the association of a policy to the software workload upon development of the workload. One advantage of such a metamodel framework is to enable a federated constituency of internal and external service providers that can be selected to provide best fit and value. For example, development projects which contain highly confidential information may be deployed on a cloud that has low cost but with very specific security requirements, while commercial services which are non-confidential in nature may be deployed on very fast, highly scalable cloud infrastructure to ensure high quality of service. Therefore, the metamodel framework lowers costs by maximizing the utility of the cloud infrastructure and aligning workload placement to provider environments which best fit to run those types of workloads.

Another method and apparatus for managing container logs in an application is disclosed in US8099396B1. This invention relates to a system and method for enhancing performance of a log adapted for use with a storage system. Due to the need to maintain transaction consistency, updates in such a storage system are stored in a temporary storage space of the system, such as a “log”. Particularly, the log is organized into a plurality of regions, wherein each region comprises one or more entries and wherein each region is allocated to a consistency collection. Therefore, the consistency collection maintains the atomicity of transaction updates to the collection of containers.

The above-mentioned prior arts do not disclose any feature to reduce storage volume of the containers within the application. Therefore, the container-based application suffers from slowdown after a prolonged period of time.

Accordingly, it would be desirable to provide a system and method to identify log impact among the containers, prevent log duplication by log policy verification and resource management, and forecast log volumes of the containers so that a measure is suggested automatically to mitigate the problem highlighted above.

SUMMARY OF INVENTION

One object of the invention is to provide users a system and method to identify log impact among the containers. Another object is to prevent log duplication through the process of log policy verification and resource management. In addition, the system also seeks to forecast log volumes of the containers so that a measure is suggested automatically to reduce log volume of the application.

The invention provides a computer-implemented system for management of container logs, the system comprising an orchestration module linked to at least one node having a plurality of containers stored therewithin, whereby the orchestration module manages the container logs which are stored in the plurality of containers, characterised in that the system further comprising a request handler for identifying log dependencies among the container logs; a log status monitoring module for obtaining log patterns based on the log dependencies; a log management module for detecting duplication of container log based on the log patterns, verifying log policy of the different container logs and initiating resource control on the container logs based on their respective log policy; and a log volume prediction module for estimating log volume of each container.

Preferably, the request handler identifies the log dependencies among the container logs by parsing a deployment file received by the orchestration module.

Preferably, the log patterns include a network communication rate among the containers and log impact rate.

Preferably, the log volume prediction module estimates the log volume based on a historical reference data and recorded log volume.

Preferably, the detected duplication and estimated log volume are communicated to the orchestration module for identification of log risk and thereby generating a suitable measure to troubleshoot the log risk. In a further aspect of this invention, there is provided a computer-implemented method for managing container logs, the method comprising the steps of identifying, by a request handler, log dependencies among the container logs; obtaining, by a log status monitoring module, log patterns based on the log dependencies; detecting, by a log management module, duplication of container logs based on the log patterns; verifying, by the log management module, log policy of the different container logs; initiating, by the log management module, resource control on the container logs based on their respective log policy; and estimating, by a log volume prediction module, log volume of each container.

Preferably, the identification of the log dependencies among the container logs comprises the steps of parsing, by the request handler, a deployment file received by the orchestration module; identifying, by the request handler, a container dependency list from the deployment file; verifying, by the request handler, dependency check across the container logs; and updating, by the request handler, the container logs in a database.

Preferably, the detected duplication and estimated log volume are communicated to the orchestration module or identification of log risk and thereby generating a suitable measure to troubleshoot the log risk.

One skilled in the art will readily appreciate that the invention is well adapted to carry out the objects and obtain the ends and advantages mentioned, as well as those inherent therein. The embodiments described herein are not intended as limitations on the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For the purpose of facilitating an understanding of the invention, there is illustrated in the accompanying drawing the preferred embodiments from an inspection of which when considered in connection with the following description, the invention, its construction and operation and many of its advantages would be readily understood and appreciated.

Fig. 1 is a diagram illustrating an exemplary embodiment of the container logs management system;

Fig. 2 is a flow chart illustrating an exemplary embodiment of the method for managing container logs;

Fig. 3 is a flow chart illustrating a preferred embodiment to identify container logs dependency;

Fig. 4 is a flow chart illustrating a preferred embodiment to initiate log monitoring;

Fig. 5 is a flow chart illustrating a preferred embodiment to detect duplication of container logs and initiate resource control; and

Fig. 6 is a flow chart illustrating a preferred embodiment to predict log volume for each container.

DETAILED DESCRIPTION OF THE INVENTION

It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, that execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

For the purpose of description, the term “container” is defined as a type of software that can virtually package and isolate applications for deployment.

The term “container log” is defined as the automatically produced and time-stamped documentation of events relevant to a particular system. The container log is stored within the container.

The term “nodes” is defined as a worker machine, a VM or a physical machine which contains services to run containers. It is controlled by a orchestration module which coordinates between all the nodes.

The term “container management” is defined as the collective processes and policies used to administer and facilitate the generation, scheduling, storage, monitoring and ultimate disposal of the large volumes of container log created within the container.

The term “host” is defined as a type of server that hosts or houses websites and/or related data, applications and services. The host is remotely accessible via Internet server. By way of example, the hosts may be a virtual machine which is a software computer used as emulation of an actual physical computer, or a physical server which is a single-tenant computer server designated to a single user.

The term “orchestration module” or orchestrator is a computer module for automating management, coordination and organization of complicated computer systems, services and middleware. The invention will now be described in greater detail, by way of example, with reference to the drawings. Referring to Fig. 1, there is provided a computer-implemented system for management of container logs. In one preferred embodiment, the system comprising an orchestration module 1 linked to at least one node 2 having a plurality of containers 3 stored therewithin. The orchestration module 1 manages the container logs which are stored in the plurality of containers 3.

In one particular embodiment, the computer-implemented system further comprises a plurality of modules, such as a request handler 5, a log management module 7, a log volume prediction module 8 and a log status monitoring module 6. The modules can be in the form of software or hardware-based computer-implementable instructions. Preferably, the log status monitoring module 6 is provided at each node, where each log status monitoring module 6 communicates with the request handler 5, log management module 7 and log volume prediction module 8 when the orchestration module 1 receives a deployment file 4.

In one preferred embodiment, the request handler 5 is used for identifying log dependencies among the container logs. The request handler 5 interreacts with the orchestration module 1 through a secured communication. By way of example, the secured communication can be in the form of valid credentials, command line or any valid API. The request handler 5 communicates with the orchestration module 1 to fetch a container request information and then parses the deployment file 4 to retrieve a container logs dependency list. Preferably, the container logs dependency list defines the dependency relationship between each container logs within the containers 3. The request handler 5 may be configured to verify the dependency check across the container logs which are defined and selected by the container log dependency list. Upon executing dependency check, the request handler 5 updates the container log dependency list across any database, such as aNoSQL database 9.

In one preferred embodiment, the log status monitoring module 6 provides a configurable file for monitoring the host where the computer-implemented system is running on, such as a virtual machine or a physical server. The log status monitoring module 6 is configured to obtain log patterns based on the log dependencies identified by the request handler 5. In this particular embodiment, the log patterns include a network communication rate among the containers 3 and log impact rate. The network communication rate refers to an in-and- outbound rate across the container logs which are defined and selected by the container log dependency list retrieved from the deployment file 4. The log status monitoring module 6 also records the log patterns, such as the log impact rate among the container logs.

In one particular embodiment, the log management module 7 is configured to detect duplication of container logs based on the log patterns, verify log policy of the different container logs and initiate resource control on the container logs based on their respective log policy. Preferably, the log management module 7 detects the duplication of container logs based on nature of the container logs and analysis of the log impact. In this particular embodiment, duplication of the container logs is identified by reading log entries of the container logs. In one example, the log management module 7 identifies repeated log entries at regular intervals as duplicated container logs. The log management module 7 may be configured to identify duplicated container logs based on number of repetitions for the container logs based on their respective container. By way of example, if the log entries are repeated for more than 10 times, the management module 7 may identify the repetitions as duplicated container logs. The log management module 7 also identifies if there are any historic references of similar type of container logs deployed. Upon detection of duplicated container logs, the log management module 7 records the status of affected container logs and reports the status to the orchestration module 1. The log management module 7 is configured to create and manage resources based on a log policy, which the log policy allows the user to specify the user, account, service, or other entity to receive permissions over the system. The log management module 7 applies the log policy over the resource based on log condition and monitor the resource status.

In this particular embodiment, the orchestration module 1 is configured to identify and calculate log risk. By way of example, the orchestration module 1 may be coupled with artificial intelligence or machine learning technique to achieve those functions. Preferably, the detected duplication and estimated log volume are communicated to the orchestration module 1 for identification of log risk and thereby generating a suitable measure to troubleshoot the log risk.

In this particular embodiment, the log volume prediction module 8 is used for estimating log volume of each container 3. The log volume prediction module 8 estimates the log volume based on a historical reference data and recorded log volume. By way of example, the log volume is predicted over a certain period of time such as day, week, month and year before the log volume is reported to orchestration module 1.

Fig. 2 illustrates an exemplary embodiment for a method for managing container logs based on the above-mentioned system. At step 100, the request handler 5 identifies the log dependencies among the container logs. At step 200, the log status monitoring module 6 initiates monitoring of the container logs. At step 300, the log management module 7 detects duplication of container logs and then initiates resource control on the container logs based on their respective log policy. At step 400, the log volume prediction module 8 predicts log volumes of each container 3. The flow in Fig. 2 is further elaborated in Fig. 3 to Fig. 6.

Fig. 3 illustrates a preferred embodiment to identify the log dependencies among the container logs. At step 101, the request handler 5 performs checking of log dependencies based on the deployment file 4 received by the orchestration module 1. The result of the log dependencies is then stored in any database in step 103. At step 104, the result is updated to the database. However, if the request handler 5 is unable to identify log dependencies among the container logs at step 101, the request handler 5 shall identify the log dependencies based on runtime at step 102 and proceeds to step 103 and 104.

Preferably, the request handler 5 parses the deployment file 4 received by the orchestration module 1 to identify the container dependency list from the deployment file 4. The request handler 5 then verifies dependency check across the container logs and updates the container logs in a database.

Fig. 4 illustrates a preferred embodiment to initiate log monitoring by log status monitoring module 6. At step 201 and 202, the log status monitoring module 6 identifies the nodes 2 which groups the containers 3 and then initiates monitoring of container logs. A step 203, the log status monitoring module 6 obtains log patterns based on the log dependencies. The log patterns include a network communication rate among the containers 3 and log impact rate. At step 204, the log pattern and log impact are stored in the database.

Fig. 5 illustrates a preferred embodiment to detect duplication of container logs initiate resource control. At step 301, the log management module 7 detects to analyse duplication of container logs based on the log patterns. At step 302, the log management module 7 identifies the container log dependency status. At step 303 and 304, the log management module 7 verifies the log policy of the different container logs and initiaties resource control on the container logs based on their respective log policy.

Fig. 6 illustrates a preferred embodiment to predict log volumes of each container 3. At step 401, the log volume prediction module 8 identifies a historic log spikes of the container logs which are dependent on one another. At step 402, the log volume prediction module 8 forecasts the log volumes of each container 3 over a certain time period based on a historical reference data and recorded log volume. At step 403, the detected duplication and estimated log volume are communicated to the orchestration module 1 for identification of log risk and thereby generating a suitable measure to troubleshoot the log risk.

The present disclosure includes as contained in the appended claims, as well as that of the foregoing description. Although this invention has been described in its preferred form with a degree of particularity, it is understood that the present disclosure of the preferred form has been made only by way of example and that numerous changes in the details of construction and the combination and arrangements of parts may be resorted to without departing from the scope of the invention.

Claims

1. A computer-implemented system for management of container logs, the system comprising: an orchestration module (1) linked to at least one node (2) having a plurality of containers (3) stored therewithin, whereby the orchestration module (1) manages the container logs which are stored in the plurality of containers (3), characterised in that the system further comprising: a request handler (5) for identifying log dependencies among the container logs; a log status monitoring module (6) for obtaining log patterns based on the log dependencies; a log management module (7) for detecting duplication of container logs based on the log patterns, verifying log policy of the different container logs and initiating resource control on the container logs based on their respective log policy; and a log volume prediction module (8) for estimating log volume of each container (3).

2. The system according to claim 1, wherein the request handler (5) identifies the log dependencies among the container logs by parsing a deployment file (4) received by the orchestration module (1).

3. The system according to claim 1, wherein the log patterns include a network communication rate among the containers (3) and log impact rate.

4. The system according to claim 1, wherein the log volume prediction module (8) estimates the log volume based on a historical reference data and recorded log volume.

5. The system according to claim 1, wherein the detected duplication and estimated log volume are communicated to the orchestration module (1) for identification of log risk and thereby generating a suitable measure to troubleshoot the log risk.

6. A computer-implemented method for managing container logs, the method comprising the steps of: identifying, by a request handler (5), log dependencies among the container logs; obtaining, by a log status monitoring module (6), log patterns based on the log dependencies; detecting, by a log management module (7), duplication of container logs based on the log patterns; verifying, by the log management module (7), log policy of the different container logs; initiating, by the log management module (7), resource control on the container logs based on their respective log policy; and estimating, by a log volume prediction module (8), log volume of each container (3).

7. The method according to claim 6, wherein the identification of the log dependencies among the container logs comprises the steps of: parsing, by the request handler (5), a deployment file (4) received by the orchestration module (1); identifying, by the request handler (5), a container dependency list from the deployment file

(4); verifying, by the request handler (5), dependency check across the container logs; and updating, by the request handler (5), the container logs in a database.

8. The method according to claim 6, wherein the log patterns include a network communication rate among the containers (3) and log impact rate.

9. The method according to claim 6, wherein the log volume prediction module (8) estimates the log volume based on a historical reference data and recorded log volume.

10. The method according to claim 6, wherein the detected duplication and estimated log volume are communicated to the orchestration module (1) for identification of log risk and thereby generating a suitable measure to troubleshoot the log risk.