WO2023123397A1 - Multi-cloud log processing method, storage method, and computer storage medium - Google Patents

Multi-cloud log processing method, storage method, and computer storage medium Download PDF

Info

Publication number
WO2023123397A1
WO2023123397A1 PCT/CN2021/143803 CN2021143803W WO2023123397A1 WO 2023123397 A1 WO2023123397 A1 WO 2023123397A1 CN 2021143803 W CN2021143803 W CN 2021143803W WO 2023123397 A1 WO2023123397 A1 WO 2023123397A1
Authority
WO
WIPO (PCT)
Prior art keywords
computing unit
log
target
target computing
cluster
Prior art date
Application number
PCT/CN2021/143803
Other languages
French (fr)
Chinese (zh)
Inventor
梅文鹏
林帅康
刘阳
Original Assignee
深圳晶泰科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳晶泰科技有限公司 filed Critical 深圳晶泰科技有限公司
Priority to PCT/CN2021/143803 priority Critical patent/WO2023123397A1/en
Publication of WO2023123397A1 publication Critical patent/WO2023123397A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9532—Query formulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries

Definitions

  • the present application belongs to the technical field of storage, and in particular relates to a multi-cloud log processing method, a storage method and a computer storage medium.
  • Logs are event records generated during the operation of the information system. Therefore, logs play an extremely important role in the later audit, fault location, and analysis of the information system. Now the country implements a mandatory requirement that logs must be stored for 180 days in the hierarchical protection of information systems, which shows that logs are extremely important.
  • a centralized log storage method is usually used at present, that is, logs in different clusters are stored in one storage server in a centralized manner.
  • the logs in the cluster are centrally stored in one storage server, which will lead to loss of public network traffic; when extracting the target logs, since there are a large number of logs stored in the storage server, it is necessary to filter the target logs from the storage server through a high-performance retrieval terminal , The construction of a high-performance retrieval terminal will also increase the cost.
  • the embodiment of the present application provides a multi-cloud log processing method, storage method, and computer storage medium, which can solve the problem that when extracting the target log, since there are a large number of logs stored in the storage server, it is necessary to use a high-performance retrieval terminal to extract the target log from the storage server. Screening out the target logs and building a high-performance retrieval terminal will also increase the cost.
  • the embodiment of the present application provides a multi-cloud log processing method, which is applied to the log retrieval terminal, including:
  • the log of the target computing unit is displayed to the user.
  • the obtaining the log of the target computing unit from the target cluster according to the target computing unit identifier includes:
  • the basic information includes the running date of the target computing unit, the running start time, the running end time, and the identity of the target cluster, and the generating the request address based on the basic information includes:
  • the target computing unit pointed to by the identification of the target cluster is generated every day during the operation date from the The request address corresponding to the log from the running start time to the running end time.
  • the method further includes:
  • the first gateway When the target cluster is a cluster in a foreign region, the first gateway sends the request address to a second gateway, and the second gateway is used to obtain the log of the target computing unit from the target cluster in the foreign region , the first gateway is connected to the second gateway through a private network.
  • the acquiring basic information of the target computing unit according to the target computing unit identifier includes:
  • the basic information of the target computing unit is obtained from an information system, and the information system is used to record the basic information of the target computing unit in real time.
  • the embodiment of the present application provides a multi-cloud log storage method, which is applied to a cluster, and the cluster is connected to a storage device through an intranet, and at least one computing unit is provided in the cluster, including:
  • the log generated by the target computing unit is used to receive the target computing unit identifier input by the user through the log retrieval terminal when the user inputs the target computing unit identifier; obtain the target computing unit from the target cluster according to the target computing unit identifier logs, the target computing unit is a computing unit in the target cluster; and the log of the target computing unit is displayed to the user.
  • the log generated by the target computing unit After the log generated by the target computing unit is parsed and formatted, it is stored in the storage device according to the generation date of the log.
  • an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned multi-cloud log processing method is implemented. And a multi-cloud log storage method.
  • the present application receives the target computing unit ID input by the user, and according to the target computing unit ID, obtains the log of the target computing unit from the target cluster, and the target computing unit
  • the unit is the computing unit in the target cluster, and the log of the target computing unit is displayed to the user, so that the log of the target computing unit can be obtained from the target cluster without obtaining the log of the computing unit from the centralized storage server, because
  • the logs in the target cluster are far smaller than the logs stored in the centralized storage server in the prior art. Therefore, there is no need to filter the target logs through a high-performance retrieval terminal, that is, there is no need to build a high-performance retrieval terminal, which avoids waste of cost.
  • FIG. 1 is a schematic flowchart of a multi-cloud log processing method provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a multi-cloud high-performance computing log provided by an embodiment of the present application
  • FIG. 3 is a schematic flowchart of a multi-cloud log storage method provided by an embodiment of the present application.
  • Fig. 4 is a schematic diagram of log processing of a single cluster provided by the embodiment of the present application.
  • Fig. 5 is a schematic diagram of centralized log service provided by the embodiment of the present application.
  • FIG. 6 is a schematic diagram of a hardware structure of a computer storage medium provided by an embodiment of the present application.
  • Logs are event records generated during the operation of the information system. These records are extremely important for later audits, fault location, and analysis. If the logs are lost, many audits and fault backtracking will not be possible. Now the country implements a mandatory requirement that logs must be stored for 180 days in the hierarchical protection of information systems, which shows that logs are extremely important.
  • the centralized cluster can bear the bottleneck of the maximum scale, and it is not easy to expand
  • embodiments of the present application provide a multi-cloud log processing method, storage method, and computer storage medium.
  • the following firstly introduces a multi-cloud log processing method provided by the embodiment of the present application.
  • FIG. 1 shows a schematic flowchart of a multi-cloud log processing method of the present application, which is applied to a log retrieval terminal. As shown in Figure 1, the method may include the following steps:
  • Step 101 receiving the target computing unit identifier input by the user
  • the computing unit identifier is an ID that uniquely identifies a computing task in high-performance computing (that is, a computing unit)
  • ID is a string of character codes with global uniqueness, and identifies a computing task in high-performance computing.
  • the running time of a computing unit depends on the specific situation, and the calculation time is different. The calculation can be completed within a few seconds for a short period of time, and the calculation unit for a long period of time can take several days to complete the calculation. All states (starting, running, abnormal, ending, etc.) of a computing unit during the entire life cycle will be recorded in a computing unit information system.
  • Step 102 according to the identification of the target computing unit, obtain the log of the target computing unit from the target cluster, where the target computing unit is a computing unit in the target cluster;
  • the log retrieval terminal After receiving the identification of the target computing unit input by the user, the log retrieval terminal directly obtains the log of the target computing unit from the target cluster, where the target computing unit is a computing unit in the target cluster.
  • step 102 includes the following sub-steps:
  • Sub-step S11 according to the identification of the target computing unit, obtain the basic information of the target computing unit;
  • the log retrieval terminal After receiving the target computing unit identifier input by the user, the log retrieval terminal obtains the corresponding basic information from the external information system.
  • Sub-step S12 generating a request address based on the basic information
  • the log retrieval terminal After obtaining the corresponding basic information, the log retrieval terminal automatically assembles and generates a request address according to the target computing unit identification through a preset program, such as http://log-gateway.test.com/cloustername/20211028/data.json .
  • the basic information includes the running date of the target computing unit, the running start time, the running end time, and the identification of the target cluster
  • the substep S12 includes:
  • Sub-step S121 based on the running date, running start time, running end time, and the target cluster ID of the target computing unit, generate the target computing unit pointed to by the target cluster ID every time during the running date The request address corresponding to the log from the start time of the operation to the end time of the operation in a day.
  • the log retrieval end can assemble and generate the target computing unit pointed to by the target cluster’s identifier based on the target computing unit’s running date, running start time, running end time, and the target cluster’s mark.
  • the request address corresponding to the log of the unit from the start time of the operation to the end time of the operation every day during the operation date.
  • Sub-step S13 sending the request address to a first gateway, wherein the first gateway is configured to obtain the log of the target computing unit from the target cluster according to the request address.
  • the log retrieval end The request address is sent to the first gateway, and the first gateway obtains the log of the target computing unit from the target cluster according to the request address.
  • the name of the target cluster is CloudACluster1
  • the log service entry of the target cluster can be accurately reached by accessing the request address http://log-gateway.test.com/CloudACluster1/.
  • Sub-step S131 when the target cluster is a cluster in a foreign region, the first gateway sends the request address to a second gateway, and the second gateway is used to obtain the target cluster from the target cluster in the foreign region
  • the first gateway is connected to the second gateway through a private network.
  • the first gateway and the second gateway are connected through a private network, that is, a high-quality private network is used in advance to build a connection between domestic and foreign countries.
  • the connection between the first gateway and the second gateway, the first gateway is deployed at the domestic end of the private network, and the second gateway is deployed at the foreign end.
  • the request address for accessing foreign clusters will be routed through the first gateway to the second gateway, and then the second gateway will proxy the request address to the foreign target cluster. This enables network acceleration for accessing foreign high-performance computing clusters.
  • step 102 after the step 102, it also includes:
  • Step 1021 receiving the log of each day during the operation date of the target computing unit
  • Step 1022 Eliminate empty logs of each day's log of the target computing unit during the running date, generate a display log result, and display the display log result to the user.
  • Step 103 displaying the log of the target computing unit to the user.
  • the log of the target computing unit may be displayed to the user through a web page.
  • the browser displays log issues corresponding to different dates. The user can choose which day's log to view, click on the corresponding time, and then accurately route to the target cluster through the first gateway, obtain the corresponding log file and return it to the browser.
  • the logs of a single cluster will only be stored in the log system of the cluster, and will not be transmitted to the unified storage server through the public network. This is a distributed storage solution.
  • Centralized storage of logs to the storage server will generate a large amount of public network traffic.
  • the current average traffic of the public network is 0.5 yuan/GB. If 1TB is transmitted a day, it will generate 15,000 yuan a month.
  • a distributed, highly scalable, and high-real-time ES search and data analysis engine needs to be equipped.
  • ES search and data analysis engine is a distributed, highly scalable, high real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed and explored.
  • the search and data analysis engine makes full use of the horizontal scalability of Elasticsearch, which can make data more valuable in the production environment.
  • the implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then uses the word segmentation controller to segment the corresponding sentence, and stores its weight and word segmentation results into the data. When the user searches for data At that time, the results will be ranked and scored according to the weight, and then the returned results will be presented to the user.
  • the log retrieval module (i.e., the log retrieval terminal) obtains the start time, The end time, and which cluster of which cloud it is running on, and in which days the entire handle has been calculated through calculation.
  • the log retrieval module has the URL (that is, the request address) of the log of which cloud and which cluster is saved, and is automatically assembled into a request URL through the program, such as http://log-gateway.test.com/cloustername/20211028/data.json. If the handle runs for multiple days, a multi-day URL will be assembled.
  • the log retrieval module will request these URLs concurrently, and then reach the desired log server through the log gateway (namely the first gateway) and the dedicated line acceleration module (namely the second gateway), and trigger the download function of the log static server to obtain the corresponding data , through fuzzy configuration handle ID, if there is in data.json, the URL will be recorded, if not, it means that no log was generated on this day.
  • the format of http://log-gateway.test.com/clustername/ is used because it has the function of access acceleration. This acceleration function is completed by the log gateway and the dedicated line acceleration module together.
  • the access address of the log gateway (that is, the first gateway) is: http://log-gateway.test.com/, which can be routed according to the clustername, such as the name of the Cluster1 cluster of CloudA cloud is CloudACluster1 , by visiting http://log-gateway.test.com/CloudACluster1/, you can accurately reach the log service entrance of the Cluster1 cluster of CloudA cloud.
  • the log gateway mainly uses the core functions of the open source Nginx, and can be routed according to the path.
  • the log gateway plus dedicated line acceleration module also solves the problem of slow access to multinational networks. We use high-quality private lines to build a bridge between China and foreign countries in advance.
  • the log gateway is deployed on a section of the dedicated line, and the dedicated line acceleration module is deployed on the foreign end.
  • the request to access the foreign log server will first be routed to the dedicated line gateway through the log gateway, and then the dedicated line gateway will proxy the web page request to a certain cluster of the foreign cloud.
  • This enables network acceleration for accessing foreign high-performance computing clusters. So far, the above has obtained the dates in which a handle has log files and returned them to the browser on the client side.
  • the browser displays the log problem corresponding to the date. The user can choose which day the log needs to be viewed, and click the corresponding time to trigger step 4 in Figure 2.
  • the log gateway is accurately routed to the real high-performance computing cluster. Get the corresponding log file and return it to the browser.
  • the application obtains the log of the target computing unit from the target cluster according to the target computing unit ID input by the user, and the target computing unit is the target computing unit in the target cluster.
  • the computing unit displays the log of the target computing unit to the user, and realizes obtaining the log of the target computing unit from the target cluster without obtaining the log of the computing unit from the centralized storage server, because the log in the target cluster is much smaller than the current.
  • the logs stored in the centralized storage server in the technology therefore, there is no need to filter the target logs through a high-performance retrieval terminal, that is, there is no need to build a high-performance retrieval terminal, which avoids waste of cost.
  • Fig. 3 shows a schematic flow diagram of a multi-cloud log storage method of the present application, which is applied to a cluster, and the cluster is connected to a storage device through an intranet. As shown in Fig. 3, the method may include the following steps:
  • Step 301 collecting logs generated by the target computing unit
  • the log generated by the target computing unit is used to receive the target computing unit identifier input by the user through the log retrieval terminal when the user inputs the target computing unit identifier; obtain the target computing unit from the target cluster according to the target computing unit identifier logs, the target computing unit is a computing unit in the target cluster; and the log of the target computing unit is displayed to the user.
  • the log of the target computing unit can be kept in a certain location of the target cluster, specifically, it can be a storage device connected to the intranet of the target cluster, and then According to a log file of each target computing unit, the directory is cut according to the date, and written into the directory of a specific storage device. For example, /data/20211027/handle1-xxx.log.
  • open source log collection tools can be used to obtain the update records of these files in real time, such as Fluentbit and other open source tools; Fluentbit is an open source log processor and forwarder, which can collect any data from different sources, such as metrics and logs, then enrich them with filters and send them to multiple destinations. Fluentbit sends newly added log records to the message queue for temporary storage, and the message queue can also be used as a local storage queue data to prevent log loss due to exceptions. In addition, Fluentbit can also detect the running status of the message queue. If there is any abnormality (for example, the message queue service is abnormal, resulting in the inability to receive logs; or the network is abnormally disconnected, etc.), it will temporarily stop sending, and reissue it after it is normal.
  • Fluentbit is an open source log processor and forwarder, which can collect any data from different sources, such as metrics and logs, then enrich them with filters and send them to multiple destinations. Fluentbit sends newly added log records to the message queue for temporary storage, and the message queue can also be used as
  • the present application may further include an open source log parser, such as Logstash.
  • Logstash has a log analysis function, which can split each log record, and then get which target computing unit and date this log belongs to, and then form a file according to the target computing unit and date, such as /data/20211027/handle1-xxx .log, which ensures that all the logs belonging to the target computing unit in the queue are recorded in the corresponding log files, so that the log files of the target computing unit are all concentrated on the disk of one server.
  • the tree structure storage is realized through the two dimensions of date and handle.
  • the present application can further compress the logs generated by the target computing unit and store them in the storage device.
  • Logstash has the function of compressed storage, and automatically compresses and writes the logs when storing them.
  • Logstash is a free and open server-side data processing pipeline that can collect data from multiple sources, transform the data, and then send the data to the target repository (ie, storage device).
  • Message queue is an important component in a distributed system. It mainly solves problems such as application coupling, asynchronous messages, and traffic cutting. It can be used to achieve high performance, high availability, scalability, and eventual consistency architecture. It is indispensable for large-scale distributed systems. middleware. Currently in the production environment, ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc. are widely used message queues.
  • the above embodiment can further implement a download function.
  • the user does not know which computing unit log files exist in the /data/20211027 directory, and it will be very slow to obtain the directory list by requesting the address when a large number of files are stored in the current directory, so this application pre-creates an index for the log files in this directory And save it in the /data/ directory, and agree to use the data.json file to store the index in this directory.
  • After writing the script put the script on the disk for persistence, and then use the timing trigger function that comes with the operating system to trigger.
  • the scheduled task can be set to execute once every minute, and the executed file will be written to the specified directory. Users can obtain this file to quickly obtain the log files corresponding to all computing units in the directory.
  • Nginx In order to deploy an open source static service on the same machine (that is, the server storing log files), you can choose Nginx, which has extremely low resource usage and can provide high-performance website services. Nginx has a directory listing function and configures the /data/ directory as a static site. In this way, the log file of the target cluster can be downloaded through the URL.
  • FIG. 6 shows a schematic diagram of a hardware structure of a computer storage medium provided by an embodiment of the present application.
  • the computer storage medium may include a processor 601 and a memory 602 storing computer program instructions.
  • the above-mentioned processor 601 may include a central processing unit (Central Processing Unit, CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application .
  • CPU Central Processing Unit
  • ASIC Application Specific Integrated Circuit
  • Memory 602 may include mass storage for data or instructions.
  • memory 602 may include a hard disk drive (Hard Disk Drive, HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (Universal Serial Bus, USB) drive or two or more Combinations of multiple of the above.
  • memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be a non-volatile solid-state memory.
  • the storage 602 can be inside or outside the comprehensive gateway disaster recovery device.
  • memory 602 may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices.
  • ROM read only memory
  • RAM random access memory
  • magnetic disk storage media devices e.g., magnetic disks
  • optical storage media devices e.g., magnetic disks
  • flash memory devices e.g., electrical, optical, or other physical/tangible memory storage devices.
  • memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions, and when the software is executed (e.g., by a or a plurality of processors) operable to perform the operations described with reference to the method according to an aspect of the present application.
  • the processor 301 reads and executes the computer program instructions stored in the memory 602 to implement the methods in the embodiments shown in FIG. 1 and FIG. 3 , which will not be repeated here for brevity.
  • a computer storage medium may also include a communication interface 603 and a bus 610 .
  • a processor 601 a memory 602 , and a communication interface 603 are connected through a bus 610 to complete mutual communication.
  • the communication interface 603 is mainly used to realize the communication between modules, devices, computing units and/or devices in the embodiments of the present application.
  • the bus 610 includes hardware, software or both, and couples the components of the online data traffic charging device to each other.
  • a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (Hyper Transport, HT) interconnect, Industry Standard Architecture (Industry Standard Architecture, ISA) bus, InfiniBand interconnect, Low Pin Count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, peripheral component interconnect PCI bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or other suitable bus or a combination of two or more of these combination.
  • Bus 610 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.
  • the embodiment of the present application may provide a computer storage medium for implementation.
  • Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the multi-cloud log processing method and the multi-cloud log storage method in the foregoing embodiments are implemented.
  • the functional blocks shown in the structural block diagrams described above may be implemented as hardware, software, firmware, or a combination thereof.
  • hardware When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a function card, and the like.
  • ASIC Application Specific Integrated Circuit
  • the elements of the present application are the programs or code segments employed to perform the required tasks.
  • Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves.
  • "Machine-readable medium" may include any medium that can store or transmit information.
  • machine-readable media examples include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, etc. wait.
  • Code segments may be downloaded via a computer network such as the Internet, an Intranet, or the like.
  • processors may be, but are not limited to, general purpose processors, special purpose processors, application specific processors, or field programmable logic circuits. It can also be understood that each block in the block diagrams and/or flowcharts and combinations of blocks in the block diagrams and/or flowcharts can also be realized by dedicated hardware for performing specified functions or actions, or can be implemented by dedicated hardware and combination of computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

Provided by the present invention are a multi-cloud log processing method, storage method, and computer storage medium. The log processing method comprises: receiving a target calculation unit identifier inputted by a user, and obtaining a log of a target calculation unit from a target cluster according to the target calculation unit identifier; the target calculation unit is a calculation unit in the target cluster and displays the log of the target calculation unit to a user such that the log of the target calculation unit is obtained from the target cluster and it is unnecessary to obtain the log of the calculation unit from the target cluster; the log in the target cluster is far smaller than the log stored in a centralized storage server in the prior art, therefore the target log does not need to be screened by means of a high-performance retrieval terminal, i.e. the high-performance retrieval terminal does not need to be built, and costs and waste are avoided.

Description

一种多云的日志处理方法、存储方法及计算机存储介质A multi-cloud log processing method, storage method and computer storage medium 【技术领域】【Technical field】
本申请属于存储技术领域，尤其涉及一种多云的日志处理方法、存储方法及计算机存储介质。The present application belongs to the technical field of storage, and in particular relates to a multi-cloud log processing method, a storage method and a computer storage medium.
【背景技术】【Background technique】
日志是信息***在运行过程中会产生的事件记录，因此，日志对信息***后期的审计、故障定位、分析有极其重要的作用。现在国家推行的信息***等级保护中强制要求日志必须存储180天，可见日志是极其重要的。Logs are event records generated during the operation of the information system. Therefore, logs play an extremely important role in the later audit, fault location, and analysis of the information system. Now the country implements a mandatory requirement that logs must be stored for 180 days in the hierarchical protection of information systems, which shows that logs are extremely important.
然而，本申请的发明人在长期研发中发现，目前通常会采用集中式的日志存储方式，即，将不同集群中的日志集中存储在一个存储服务器中，在存储日志时，需要通过公网将集群中的日志集中存储在一个存储服务器，会导致公网流量损耗；在提取目标日志时，由于存储服务器中存储有大量的日志，需要通过高性能的检索端，从存储服务器中筛选出目标日志，高性能的检索端的搭建也会增加成本。However, the inventors of the present application have discovered during long-term research and development that a centralized log storage method is usually used at present, that is, logs in different clusters are stored in one storage server in a centralized manner. The logs in the cluster are centrally stored in one storage server, which will lead to loss of public network traffic; when extracting the target logs, since there are a large number of logs stored in the storage server, it is necessary to filter the target logs from the storage server through a high-performance retrieval terminal , The construction of a high-performance retrieval terminal will also increase the cost.
【发明内容】【Content of invention】
本申请实施例提供一种多云的日志处理方法、存储方法及计算机存储介质，能够解决在提取目标日志时，由于存储服务器中存储有大量的日志，需要通过高性能的检索端，从存储服务器中筛选出目标日志，高性能的检索端的搭建也会增加成本的问题。The embodiment of the present application provides a multi-cloud log processing method, storage method, and computer storage medium, which can solve the problem that when extracting the target log, since there are a large number of logs stored in the storage server, it is necessary to use a high-performance retrieval terminal to extract the target log from the storage server. Screening out the target logs and building a high-performance retrieval terminal will also increase the cost.
第一方面，本申请实施例提供一种多云的日志处理方法，应用于日志检索端，包括：In the first aspect, the embodiment of the present application provides a multi-cloud log processing method, which is applied to the log retrieval terminal, including:
接收用户输入的目标计算单元标识；receiving the target computing unit identifier input by the user;
根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元；Obtain a log of the target computing unit from the target cluster according to the target computing unit identifier, where the target computing unit is a computing unit in the target cluster;
向用户展示所述目标计算单元的日志。The log of the target computing unit is displayed to the user.
可选地，所述根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，包括：Optionally, the obtaining the log of the target computing unit from the target cluster according to the target computing unit identifier includes:
根据所述目标计算单元标识，获取所述目标计算单元的基本信息；Obtain basic information of the target computing unit according to the target computing unit identifier;
基于所述基本信息，生成请求地址；Generate a request address based on the basic information;
将所述请求地址发送至第一网关，其中，所述第一网关用于根据所述请求地址从所述目标集群中获取目标计算单元的日志。Sending the request address to a first gateway, wherein the first gateway is configured to acquire the log of the target computing unit from the target cluster according to the request address.
可选地，所述基本信息包括所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，所述基于所述基本信息，生成请求地址，包括：Optionally, the basic information includes the running date of the target computing unit, the running start time, the running end time, and the identity of the target cluster, and the generating the request address based on the basic information includes:
基于所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，生成所述目标集群的标识指向的所述目标计算单元在所述运行日期期间每一天从所述运行开始时间开始、到所述运行结束时间结束的日志对应的请求地址。Based on the operation date, operation start time, operation end time, and the identification of the target cluster of the target computing unit, the target computing unit pointed to by the identification of the target cluster is generated every day during the operation date from the The request address corresponding to the log from the running start time to the running end time.
可选地，所述根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志后，还包括：Optionally, after obtaining the log of the target computing unit from the target cluster according to the target computing unit identifier, the method further includes:
接收所述目标计算单元在运行日期期间每一天的日志；receiving logs for each day during the run dates of the target computing unit;
将所述目标计算单元在运行日期期间每一天的日志的空日志剔除，并生成展示日志结果， 向用户展示所述展示日志结果。Empty logs of the logs of each day during the running date of the target computing unit are removed, and display log results are generated, and the display log results are displayed to the user.
可选地，所述将所述请求地址发送至第一网关后，还包括：Optionally, after sending the request address to the first gateway, it further includes:
在所述目标集群为国外区域的集群时，所述第一网关将所述请求地址发送至第二网关，所述第二网关用于从所述国外区域的目标集群中获取目标计算单元的日志，所述第一网关与第二网关之间通过专网连接。When the target cluster is a cluster in a foreign region, the first gateway sends the request address to a second gateway, and the second gateway is used to obtain the log of the target computing unit from the target cluster in the foreign region , the first gateway is connected to the second gateway through a private network.
可选地，所述根据所述目标计算单元标识，获取所述目标计算单元的基本信息，包括：Optionally, the acquiring basic information of the target computing unit according to the target computing unit identifier includes:
根据所述目标计算单元标识，从信息***中获取所述目标计算单元的基本信息，所述信息***用于实时记录所述目标计算单元的基本信息。According to the identification of the target computing unit, the basic information of the target computing unit is obtained from an information system, and the information system is used to record the basic information of the target computing unit in real time.
第二方面，本申请实施例提供了一种多云的日志的存储方法，应用于集群，所述集群通过内网连接存储装置，所述集群中设有至少一个计算单元，包括：In the second aspect, the embodiment of the present application provides a multi-cloud log storage method, which is applied to a cluster, and the cluster is connected to a storage device through an intranet, and at least one computing unit is provided in the cluster, including:
采集目标计算单元生成的日志；Collect logs generated by the target computing unit;
将所述目标计算单元生成的日志通过解析、格式化后存储在所述存储装置中；storing the log generated by the target computing unit in the storage device after parsing and formatting;
其中，所述目标计算单元生成的日志用于在用户输入目标计算单元标识时，通过日志检索端接收用户输入的目标计算单元标识；根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元；向用户展示所述目标计算单元的日志。Wherein, the log generated by the target computing unit is used to receive the target computing unit identifier input by the user through the log retrieval terminal when the user inputs the target computing unit identifier; obtain the target computing unit from the target cluster according to the target computing unit identifier logs, the target computing unit is a computing unit in the target cluster; and the log of the target computing unit is displayed to the user.
可选地，所述将所述目标计算单元生成的日志通过解析、格式化后存储在所述存储装置中，包括：Optionally, storing the log generated by the target computing unit in the storage device after parsing and formatting includes:
将所述目标计算单元生成的日志通过解析、格式化后，按照日志的生成日期存储在所述存储装置中。After the log generated by the target computing unit is parsed and formatted, it is stored in the storage device according to the generation date of the log.
可选地，所述存储装置通过磁盘存储日志信息。Optionally, the storage device stores log information through a disk.
第三方面，本申请实施例提供了一种计算机存储介质，所述计算机存储介质上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现如上所述的一种多云的日志处理方法以及一种多云的日志的存储方法。In a third aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned multi-cloud log processing method is implemented. And a multi-cloud log storage method.
本申请的有益效果是：区别于现有技术的情况，本申请通过接收用户输入的目标计算单元标识，根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元，向用户展示所述目标计算单元的日志，实现了从目标集群中获取标计算单元的日志，无需从集中式的存储服务器中获取计算单元的日志，由于目标集群中的日志远小于现有技术中集中式的存储服务器中存储的日志，因此，无需通过高性能的检索端筛选目标日志，即无需搭建高性能的检索端，避免了成本的浪费。The beneficial effects of the present application are: different from the situation in the prior art, the present application receives the target computing unit ID input by the user, and according to the target computing unit ID, obtains the log of the target computing unit from the target cluster, and the target computing unit The unit is the computing unit in the target cluster, and the log of the target computing unit is displayed to the user, so that the log of the target computing unit can be obtained from the target cluster without obtaining the log of the computing unit from the centralized storage server, because The logs in the target cluster are far smaller than the logs stored in the centralized storage server in the prior art. Therefore, there is no need to filter the target logs through a high-performance retrieval terminal, that is, there is no need to build a high-performance retrieval terminal, which avoids waste of cost.
【附图说明】【Description of drawings】
本申请将结合附图对实施方式进行说明。本申请的附图仅用于描述实施例，以展示为目的。在不偏离本申请原理的条件下，本领域技术人员能够轻松地通过以下描述根据所述步骤做出其他实施例。The present application will describe the implementation manners with reference to the accompanying drawings. The drawings of the present application are only used to describe the embodiments for the purpose of illustration. Without departing from the principles of the present application, those skilled in the art can easily make other embodiments according to the steps described below.
图1是本申请实施例提供的一种多云的日志处理方法的流程示意图；FIG. 1 is a schematic flowchart of a multi-cloud log processing method provided by an embodiment of the present application;
图2是本申请实施例提供的一种多云高性能计算日志示意图；FIG. 2 is a schematic diagram of a multi-cloud high-performance computing log provided by an embodiment of the present application;
图3是本申请实施例提供一种多云的日志的存储方法的流程示意图；FIG. 3 is a schematic flowchart of a multi-cloud log storage method provided by an embodiment of the present application;
图4是本申请实施例提供单集群的日志处理示意图；Fig. 4 is a schematic diagram of log processing of a single cluster provided by the embodiment of the present application;
图5是本申请实施例提供集中日志服务化示意图；Fig. 5 is a schematic diagram of centralized log service provided by the embodiment of the present application;
图6是本申请实施例提供的一种计算机存储介质的硬件结构示意图。FIG. 6 is a schematic diagram of a hardware structure of a computer storage medium provided by an embodiment of the present application.
【具体实施方式】【Detailed ways】
下面将详细描述本申请的各个方面的特征和示例性实施例，为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及具体实施例，对本申请进行进一步详细描述。应理解，此处所描述的具体实施例仅意在解释本申请，而不是限定本申请。对于本领域技术人员来说，本申请可以在不需要这些具体细节中的一些细节的情况下实施。下面对实施例的描述仅仅是为了通过示出本申请的示例来提供对本申请更好的理解。The characteristics and exemplary embodiments of various aspects of the application will be described in detail below. In order to make the purpose, technical solution and advantages of the application clearer, the application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only intended to explain the present application rather than limit the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the statement "comprising..." does not exclude the presence of additional same elements in the process, method, article or device comprising said element.
日志是信息***在运行过程中产生的事件记录，这些记录对后期的审计、故障定位、分析有及其重要的作用，如果失去了日志对于很多审计、故障回溯将无法进行。现在国家推行的信息***等级保护中强制要求日志必须存储180天，可见日志是极其重要的。Logs are event records generated during the operation of the information system. These records are extremely important for later audits, fault location, and analysis. If the logs are lost, many audits and fault backtracking will not be possible. Now the country implements a mandatory requirement that logs must be stored for 180 days in the hierarchical protection of information systems, which shows that logs are extremely important.
当前主流日志解决方案：Current mainstream log solutions:
(1)、采用集中式的日志存储(1), using centralized log storage
(2)、采用高配置的ES集群做日志检索服务(2), using highly configured ES clusters for log retrieval services
然而针对多云的高性能计算场景以上方案会带来新的问题：However, the above solutions for multi-cloud high-performance computing scenarios will bring new problems:
(1)、集中式存储会带来跨公有云的公网流量(1) Centralized storage will bring public network traffic across public clouds
(2)、跨国的公网极其不稳定，集中式存储会出现日志丢失的情况(2) The transnational public network is extremely unstable, and logs will be lost in centralized storage
(3)、搭配高配置的ES集群会对企业产生极大的额外成本(3), with a high-configuration ES cluster will cause a huge additional cost to the enterprise
(4)、集中式有集群能最大承受规模的瓶颈，不易扩展(4), the centralized cluster can bear the bottleneck of the maximum scale, and it is not easy to expand
综上，现有的技术方案并不能满足多云高性能计算的场景的需求，会带来极大的基础投产成本和扩展性差的问题。To sum up, the existing technical solutions cannot meet the needs of multi-cloud high-performance computing scenarios, which will bring about huge basic production costs and poor scalability.
为了解决现有技术问题，本申请实施例提供了一种多云的日志处理方法、存储方法及计算机存储介质。下面首先对本申请实施例所提供的一种多云的日志处理方法进行介绍。In order to solve the problems in the prior art, embodiments of the present application provide a multi-cloud log processing method, storage method, and computer storage medium. The following firstly introduces a multi-cloud log processing method provided by the embodiment of the present application.
图1示出了本申请一种多云的日志处理方法的流程示意图，应用于日志检索端。如图1所示，该方法可以包括以下步骤：FIG. 1 shows a schematic flowchart of a multi-cloud log processing method of the present application, which is applied to a log retrieval terminal. As shown in Figure 1, the method may include the following steps:
步骤101，接收用户输入的目标计算单元标识； Step 101, receiving the target computing unit identifier input by the user;
其中，所述计算单元标识是高性能计算中(即计算单元)唯一标识计算任务的ID(ID是一串字符编码，具有全局唯一性，在高性能计算中标识一个计算任务)。一个计算单元的运行时间根据具体情况的不同，计算的时间不一样。时间短的几秒内运算完，时间长的计算单元需要几天才可以运算完。一个计算单元在整个生命周期内的所有状态(开始、运行、异常、结束等状态)都会记录到一个计算单元信息***中。Wherein, the computing unit identifier is an ID that uniquely identifies a computing task in high-performance computing (that is, a computing unit) (ID is a string of character codes with global uniqueness, and identifies a computing task in high-performance computing). The running time of a computing unit depends on the specific situation, and the calculation time is different. The calculation can be completed within a few seconds for a short period of time, and the calculation unit for a long period of time can take several days to complete the calculation. All states (starting, running, abnormal, ending, etc.) of a computing unit during the entire life cycle will be recorded in a computing unit information system.
用户在得知目标计算单元运行异常后，需要快速获取到目标计算单元运行的时间周期以及对应的日期的日志下载链接，用户可以通过浏览器连接日志检索端，进而由日志检索端接收用户输入的目标计算单元标识。After the user knows that the target computing unit is running abnormally, he needs to quickly obtain the time period of the target computing unit’s operation and the log download link of the corresponding date. The target compute unit ID.
步骤102，根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元； Step 102, according to the identification of the target computing unit, obtain the log of the target computing unit from the target cluster, where the target computing unit is a computing unit in the target cluster;
在接收到用户输入的目标计算单元标识后，日志检索端定向的从目标集群中获取目标计算单元的日志，目标计算单元为所述目标集群中的计算单元。After receiving the identification of the target computing unit input by the user, the log retrieval terminal directly obtains the log of the target computing unit from the target cluster, where the target computing unit is a computing unit in the target cluster.
在本申请一实施例中，步骤102，包括如下子步骤：In an embodiment of the present application, step 102 includes the following sub-steps:
子步骤S11，根据所述目标计算单元标识，获取所述目标计算单元的基本信息；Sub-step S11, according to the identification of the target computing unit, obtain the basic information of the target computing unit;
在接收到用户输入的目标计算单元标识后，日志检索端向外部的信息***中获取对应的基本信息。After receiving the target computing unit identifier input by the user, the log retrieval terminal obtains the corresponding basic information from the external information system.
子步骤S12，基于所述基本信息，生成请求地址；Sub-step S12, generating a request address based on the basic information;
在获取对应的基本信息后，日志检索端通过预先设定的程序，根据目标计算单元标识，自动组装生成请求地址，如http://log-gateway.test.com/cloustername/20211028/data.json。After obtaining the corresponding basic information, the log retrieval terminal automatically assembles and generates a request address according to the target computing unit identification through a preset program, such as http://log-gateway.test.com/cloustername/20211028/data.json .
在本申请一实施例中，所述基本信息包括所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，所述子步骤S12，包括：In an embodiment of the present application, the basic information includes the running date of the target computing unit, the running start time, the running end time, and the identification of the target cluster, and the substep S12 includes:
子步骤S121，基于所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，生成所述目标集群的标识指向的所述目标计算单元在所述运行日期期间每一天从所述运行开始时间开始、到所述运行结束时间结束的日志对应的请求地址。Sub-step S121, based on the running date, running start time, running end time, and the target cluster ID of the target computing unit, generate the target computing unit pointed to by the target cluster ID every time during the running date The request address corresponding to the log from the start time of the operation to the end time of the operation in a day.
如果目标计算单元运行了多天，日志检索端可以基于目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，组装生成所述目标集群的标识指向的所述目标计算单元在所述运行日期期间每一天从所述运行开始时间开始、到所述运行结束时间结束的日志对应的请求地址。If the target computing unit has been running for multiple days, the log retrieval end can assemble and generate the target computing unit pointed to by the target cluster’s identifier based on the target computing unit’s running date, running start time, running end time, and the target cluster’s mark. The request address corresponding to the log of the unit from the start time of the operation to the end time of the operation every day during the operation date.
子步骤S13，将所述请求地址发送至第一网关，其中，所述第一网关用于根据所述请求地址从所述目标集群中获取目标计算单元的日志。Sub-step S13, sending the request address to a first gateway, wherein the first gateway is configured to obtain the log of the target computing unit from the target cluster according to the request address.
在一具体实施例中，第一网关使用了开源Nginx的核心功能，根据路径可以做路由，ginx(engine x)是一个高性能的HTTP和反向代理web服务器，同时也提供了IMAP/POP3/SMTP服务。In a specific embodiment, the first gateway has used the core function of open source Nginx, can do routing according to path, ginx (engine x) is a high-performance HTTP and reverse proxy web server, also provides IMAP/POP3/ SMTP service.
在组装生成所述目标集群的标识指向的所述目标计算单元在所述运行日期期间每一天从所述运行开始时间开始、到所述运行结束时间结束的日志对应的请求地址后，日志检索端将请求地址发送至第一网关，并由第一网关根据请求地址从所述目标集群中获取目标计算单元的日志。如，目标集群的名称是CloudACluster1，通过访问请求地址http://log-gateway.test.com/CloudACluster1/可以准确到达到目标集群的日志服务入口。After assembling and generating the request address corresponding to the log of the target computing unit pointed to by the identifier of the target cluster during the running date from the start time of the operation to the end time of the operation every day, the log retrieval end The request address is sent to the first gateway, and the first gateway obtains the log of the target computing unit from the target cluster according to the request address. For example, the name of the target cluster is CloudACluster1, and the log service entry of the target cluster can be accurately reached by accessing the request address http://log-gateway.test.com/CloudACluster1/.
在本申请一实施例中，所述子步骤S13后，还包括：In an embodiment of the present application, after the sub-step S13, it also includes:
子步骤S131，在所述目标集群为国外区域的集群时，所述第一网关将所述请求地址发送至第二网关，所述第二网关用于从所述国外区域的目标集群中获取目标计算单元的日志，所述第一网关与第二网关之间通过专网连接。Sub-step S131, when the target cluster is a cluster in a foreign region, the first gateway sends the request address to a second gateway, and the second gateway is used to obtain the target cluster from the target cluster in the foreign region For the log of the computing unit, the first gateway is connected to the second gateway through a private network.
在本申请实施例中，由于国外区域与国内之间通过第一网关与第二网关连接，第一网关与第二网关之间通过专网连接，即预先使用优质的专网搭建起国内到国外的第一网关与第二网关的连接，第一网关部署在专网的国内这一端，第二网关部署在国外的这一端。访问国外集群的 请求地址会先通过第一网关路由到第二网关，再由第二网关将请求地址代理到国外的目标集群。这就实现了访问国外高性能计算集群的网络加速。In the embodiment of this application, because the foreign region and the domestic are connected through the first gateway and the second gateway, and the first gateway and the second gateway are connected through a private network, that is, a high-quality private network is used in advance to build a connection between domestic and foreign countries. The connection between the first gateway and the second gateway, the first gateway is deployed at the domestic end of the private network, and the second gateway is deployed at the foreign end. The request address for accessing foreign clusters will be routed through the first gateway to the second gateway, and then the second gateway will proxy the request address to the foreign target cluster. This enables network acceleration for accessing foreign high-performance computing clusters.
在本申请一实施例中，所述步骤102后，还包括：In an embodiment of the present application, after the step 102, it also includes:
步骤1021，接收所述目标计算单元在运行日期期间每一天的日志；Step 1021, receiving the log of each day during the operation date of the target computing unit;
步骤1022，将所述目标计算单元在运行日期期间每一天的日志的空日志剔除，并生成展示日志结果，向用户展示所述展示日志结果。Step 1022 : Eliminate empty logs of each day's log of the target computing unit during the running date, generate a display log result, and display the display log result to the user.
步骤103，向用户展示所述目标计算单元的日志。 Step 103, displaying the log of the target computing unit to the user.
在获取目标计算单元的日志后，可以通过网页向用户展示目标计算单元的日志。浏览器显示不同日期对应的日志问题，用户可以选择需要查看哪天的日志，点击对应时间，然后经过第一网关准确路由到目标集群，获取到对应的日志文件返回给浏览器。After the log of the target computing unit is acquired, the log of the target computing unit may be displayed to the user through a web page. The browser displays log issues corresponding to different dates. The user can choose which day's log to view, click on the corresponding time, and then accurately route to the target cluster through the first gateway, obtain the corresponding log file and return it to the browser.
在整个过程中单个集群的日志只会存放在本集群的日志***中，不会公网传输到统一的存储服务器，这是一种分布式存储方案。During the whole process, the logs of a single cluster will only be stored in the log system of the cluster, and will not be transmitted to the unified storage server through the public network. This is a distributed storage solution.
在集中式的将日志存储到存储服务器，会产生大量的公网流量，如，现在公网的流量平均是0.5元/GB，假如一天传输1TB会产生，一个月会产生1.5万元。进一步的，集中式的将日志存储到存储服务器后，需要配备是一个分布式、高扩展、高实时的ES搜索与数据分析引擎，本申请仅需分别在目标集群中配备1个2核CPU、8GB内存的机器(260元/月)，而ES搜索与数据分析引擎在每个集群中的保守配置是8核CPU、32GB内存(1042元/月),按照20个集群的场景计算，当前集中式的将日志存储到存储服务器的方案在ES搜索与数据分析引擎上的花费，会比在每一个集群中配备2核CPU、8GB内存的机器多花费1.5万元/月。即，在前述条件下，本申请的技术方案在高性能计算场景中能减少3万/月的基础实施成本费用。另外一方面，由于本申请的技术方案采用的是分布式存储，具有天然的可扩展性，可以容纳国内外各种公有云，可以支撑大规模多云高性能计算场景。Centralized storage of logs to the storage server will generate a large amount of public network traffic. For example, the current average traffic of the public network is 0.5 yuan/GB. If 1TB is transmitted a day, it will generate 15,000 yuan a month. Furthermore, after the logs are stored in the storage server in a centralized manner, a distributed, highly scalable, and high-real-time ES search and data analysis engine needs to be equipped. This application only needs to be equipped with a 2-core CPU, A machine with 8GB memory (260 yuan/month), and the conservative configuration of the ES search and data analysis engine in each cluster is 8-core CPU, 32GB memory (1042 yuan/month), calculated according to the scenario of 20 clusters, the current centralized The traditional solution of storing logs to the storage server will cost 15,000 yuan/month more on the ES search and data analysis engine than a machine equipped with 2-core CPU and 8GB memory in each cluster. That is, under the aforementioned conditions, the technical solution of the present application can reduce the basic implementation cost of 30,000/month in high-performance computing scenarios. On the other hand, since the technical solution of this application uses distributed storage, it has natural scalability, can accommodate various public clouds at home and abroad, and can support large-scale multi-cloud high-performance computing scenarios.
ES搜索与数据分析引擎是一个分布式、高扩展、高实时的搜索与数据分析引擎。它能很方便的使大量数据具有搜索、分析和探索的能力。搜索与数据分析引擎充分利用Elasticsearch的水平伸缩性，能使数据在生产环境变得更有价值。Elasticsearch的实现原理主要分为以下几个步骤，首先用户将数据提交到Elasticsearch数据库中，再通过分词控制器去将对应的语句分词，将其权重和分词结果一并存入数据，当用户搜索数据时候，再根据权重将结果排名，打分，再将返回结果呈现给用户。ES search and data analysis engine is a distributed, highly scalable, high real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed and explored. The search and data analysis engine makes full use of the horizontal scalability of Elasticsearch, which can make data more valuable in the production environment. The implementation principle of Elasticsearch is mainly divided into the following steps. First, the user submits the data to the Elasticsearch database, and then uses the word segmentation controller to segment the corresponding sentence, and stores its weight and word segmentation results into the data. When the user searches for data At that time, the results will be ranked and scored according to the weight, and then the returned results will be presented to the user.
为了便于上述实施例的理解，以下将通过一具体实例进行举例说明，需要说明的是，本申请不仅限于以下实例：In order to facilitate the understanding of the above embodiments, a specific example will be used for illustration below. It should be noted that the present application is not limited to the following examples:
如图2，日志检索模块(即日志检索端)根据用户输入的handle ID(即目标计算单元标识)从外部handle信息***(即信息***)中获取对应handle(即目标计算单元)的开始时间、结束时间、以及在哪个云的哪个集群运行，另外通过计算获取到整个handle在哪几天内运算过。日志检索模块有保存哪个云哪个集群的日志的URL(即请求地址)，通过程序自动组装成请求URL，如http://log-gateway.test.com/cloustername/20211028/data.json。如果handle运行多天，将组装成多天的URL。日志检索模块会并发请求这些URL，然后通过日志网关(即第一网关)和专线加速模块(即第二网关)到达期望的日志服务器，并触发日志静态服务器的下载功能，从而获取到对应的数据，通过模糊配置handle ID，如果data.json中有则将URL记录下来，没有就说明在这天没有产生日志。使用http://log-gateway.test.com/cloustername/这种格式 是因为这种格式具有访问加速的功能。这个加速的功能是由日志网关和专线加速模块一起完成的。As shown in Figure 2, the log retrieval module (i.e., the log retrieval terminal) obtains the start time, The end time, and which cluster of which cloud it is running on, and in which days the entire handle has been calculated through calculation. The log retrieval module has the URL (that is, the request address) of the log of which cloud and which cluster is saved, and is automatically assembled into a request URL through the program, such as http://log-gateway.test.com/cloustername/20211028/data.json. If the handle runs for multiple days, a multi-day URL will be assembled. The log retrieval module will request these URLs concurrently, and then reach the desired log server through the log gateway (namely the first gateway) and the dedicated line acceleration module (namely the second gateway), and trigger the download function of the log static server to obtain the corresponding data , through fuzzy configuration handle ID, if there is in data.json, the URL will be recorded, if not, it means that no log was generated on this day. The format of http://log-gateway.test.com/clustername/ is used because it has the function of access acceleration. This acceleration function is completed by the log gateway and the dedicated line acceleration module together.
在一具体实施例中，日志网关(即第一网关)的访问地址是：http://log-gateway.test.com/，它可以根据cloustername做路由，比如CloudA云的Cluster1集群的名称是CloudACluster1,通过访问http://log-gateway.test.com/CloudACluster1/可以准确到达到CloudA云的Cluster1集群的日志服务入口。日志网关主要是使用了开源Nginx的核心功能，根据路径可以做路由。日志网关再加专线加速模块还解决了一个访问跨国网络慢的难题。我们预先使用优质的专线搭建起国内到国外的桥梁，日志网关部署在专线的一段，专线加速模块部署在国外端。访问国外日志服务器的请求会先通过日志网关路由到专线网关，再由专线网关将网页请求代理到国外云的某个集群。这就实现了访问国外高性能计算集群的网络加速。至此，以上已经获取到了一个handle在哪些日期内有日志文件并返回给用户端的浏览器。浏览器显示日期对应的日志问题，用户可以选择需要查看哪天的日志，点击对应时间，就会触发图2中的第4步，到达日志网关后经过日志网关准确路由到真实高性能计算集群，获取到对应的日志文件返回给浏览器。In a specific embodiment, the access address of the log gateway (that is, the first gateway) is: http://log-gateway.test.com/, which can be routed according to the clustername, such as the name of the Cluster1 cluster of CloudA cloud is CloudACluster1 , by visiting http://log-gateway.test.com/CloudACluster1/, you can accurately reach the log service entrance of the Cluster1 cluster of CloudA cloud. The log gateway mainly uses the core functions of the open source Nginx, and can be routed according to the path. The log gateway plus dedicated line acceleration module also solves the problem of slow access to multinational networks. We use high-quality private lines to build a bridge between China and foreign countries in advance. The log gateway is deployed on a section of the dedicated line, and the dedicated line acceleration module is deployed on the foreign end. The request to access the foreign log server will first be routed to the dedicated line gateway through the log gateway, and then the dedicated line gateway will proxy the web page request to a certain cluster of the foreign cloud. This enables network acceleration for accessing foreign high-performance computing clusters. So far, the above has obtained the dates in which a handle has log files and returned them to the browser on the client side. The browser displays the log problem corresponding to the date. The user can choose which day the log needs to be viewed, and click the corresponding time to trigger step 4 in Figure 2. After arriving at the log gateway, the log gateway is accurately routed to the real high-performance computing cluster. Get the corresponding log file and return it to the browser.
在本申请实施例中，本申请通过接收用户输入的目标计算单元标识，根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元，向用户展示所述目标计算单元的日志，实现了从目标集群中获取标计算单元的日志，无需从集中式的存储服务器中获取计算单元的日志，由于目标集群中的日志远小于现有技术中集中式的存储服务器中存储的日志，因此，无需通过高性能的检索端筛选目标日志，即无需搭建高性能的检索端，避免了成本的浪费。In this embodiment of the application, the application obtains the log of the target computing unit from the target cluster according to the target computing unit ID input by the user, and the target computing unit is the target computing unit in the target cluster. The computing unit displays the log of the target computing unit to the user, and realizes obtaining the log of the target computing unit from the target cluster without obtaining the log of the computing unit from the centralized storage server, because the log in the target cluster is much smaller than the current The logs stored in the centralized storage server in the technology, therefore, there is no need to filter the target logs through a high-performance retrieval terminal, that is, there is no need to build a high-performance retrieval terminal, which avoids waste of cost.
图3示出了本申请一种多云的日志的存储方法的流程示意图，应用于集群，所述集群通过内网连接存储装置，如图3所示，该方法可以包括以下步骤：Fig. 3 shows a schematic flow diagram of a multi-cloud log storage method of the present application, which is applied to a cluster, and the cluster is connected to a storage device through an intranet. As shown in Fig. 3, the method may include the following steps:
步骤301，采集目标计算单元生成的日志； Step 301, collecting logs generated by the target computing unit;
步骤302，将所述目标计算单元生成的日志通过解析、格式化后存储在所述存储装置中； Step 302, storing the log generated by the target computing unit in the storage device after parsing and formatting;
其中，所述目标计算单元生成的日志用于在用户输入目标计算单元标识时，通过日志检索端接收用户输入的目标计算单元标识；根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元；向用户展示所述目标计算单元的日志。Wherein, the log generated by the target computing unit is used to receive the target computing unit identifier input by the user through the log retrieval terminal when the user inputs the target computing unit identifier; obtain the target computing unit from the target cluster according to the target computing unit identifier logs, the target computing unit is a computing unit in the target cluster; and the log of the target computing unit is displayed to the user.
为了确保日志能被正确的记录到特定的位置，如图4，可以将目标计算单元的日志会保留在目标集群的某一个位置，具体的，可以是与目标集群内网连接的存储装置，然后按照每个目标计算单元的日志一个文件，按照日期来做目录切割，写入到特定的存储装置的目录里。比如，/data/20211027/handle1-xxx.log。日志记录好后可以采用开源的日志采集工具来实时获取这些文件的更新记录，如Fluentbit等其他开源工具；Fluentbit是一个开源的日志处理器和转发器，其可以从不同的来源收集任何数据，如指标和日志，然后用过滤器丰富它们，并将它们发送到多个目的地。Fluentbit将新添加的日志记录发送到消息队列临时存储，消息队列也可以作为本地保存队列的数据以防异常导致日志丢失。另外Fluentbit也可以检测消息队列的运行情况，如有异常(例如消息队列服务异常，导致不能接收日志；或者出现网络异常断开等)就会临时停止发送，待正常后再补发。本申请还可以进一步包括一个开源的日志解析器，例如Logstash。Logstash有日志解析功能，可以将每条日志记录进行拆分，然后得到这条日志属于 哪个目标计算单元和哪个日期，然后根据目标计算单元和日期组成一个文件，比如/data/20211027/handle1-xxx.log，这就保障队列里属于这个目标计算单元的所有日志都记录到对应的日志文件中，这样就实现了目标计算单元的日志文件都被集中到一台服务器的磁盘上。通过日期和handle这2个维度来实现树型结构存储。本申请还可以进一步将目标计算单元生成的日志压缩后存储在所述存储装置中，Logstash具有压缩存储的功能，存储日志时自动压缩写入。Logstash是免费且开放的服务器端数据处理管道，能够从多个来源采集数据，转换数据，然后将数据发送到目标存储库(即存储装置)中。In order to ensure that the log can be correctly recorded to a specific location, as shown in Figure 4, the log of the target computing unit can be kept in a certain location of the target cluster, specifically, it can be a storage device connected to the intranet of the target cluster, and then According to a log file of each target computing unit, the directory is cut according to the date, and written into the directory of a specific storage device. For example, /data/20211027/handle1-xxx.log. After the logs are recorded, open source log collection tools can be used to obtain the update records of these files in real time, such as Fluentbit and other open source tools; Fluentbit is an open source log processor and forwarder, which can collect any data from different sources, such as metrics and logs, then enrich them with filters and send them to multiple destinations. Fluentbit sends newly added log records to the message queue for temporary storage, and the message queue can also be used as a local storage queue data to prevent log loss due to exceptions. In addition, Fluentbit can also detect the running status of the message queue. If there is any abnormality (for example, the message queue service is abnormal, resulting in the inability to receive logs; or the network is abnormally disconnected, etc.), it will temporarily stop sending, and reissue it after it is normal. The present application may further include an open source log parser, such as Logstash. Logstash has a log analysis function, which can split each log record, and then get which target computing unit and date this log belongs to, and then form a file according to the target computing unit and date, such as /data/20211027/handle1-xxx .log, which ensures that all the logs belonging to the target computing unit in the queue are recorded in the corresponding log files, so that the log files of the target computing unit are all concentrated on the disk of one server. The tree structure storage is realized through the two dimensions of date and handle. The present application can further compress the logs generated by the target computing unit and store them in the storage device. Logstash has the function of compressed storage, and automatically compresses and writes the logs when storing them. Logstash is a free and open server-side data processing pipeline that can collect data from multiple sources, transform the data, and then send the data to the target repository (ie, storage device).
消息队列是分布式***中重要的组件，主要解决应用耦合、异步消息、流量削锋等问题，可用于实现高性能、高可用、可伸缩和最终一致性架构，是大型分布式***不可缺少的中间件。目前在生产环境，使用较多的消息队列有ActiveMQ、RabbitMQ、ZeroMQ、Kafka、MetaMQ、RocketMQ等。Message queue is an important component in a distributed system. It mainly solves problems such as application coupling, asynchronous messages, and traffic cutting. It can be used to achieve high performance, high availability, scalability, and eventual consistency architecture. It is indispensable for large-scale distributed systems. middleware. Currently in the production environment, ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc. are widely used message queues.
进一步地，如图5，上述实施例可进一步实现下载功能。用户并不知道/data/20211027目录下具有哪些计算单元的日志文件，并且在当前目录下大量存放文件时通过请求地址获取目录列表会非常慢，所以本申请预先将这个目录下的日志文件创建索引并保存在/data/目录，并约定用data.json文件存储这个目录下的索引。编写好脚本后将脚本放在磁盘持久化，然后利用操作***自带了定时触发功能触发。定时任务可以设置为每一分钟执行一次，将执行完的文件写入到指定目录下。用户可以获取这个文件来快速获取到目录下所有计算单元对应的日志文件。Further, as shown in FIG. 5 , the above embodiment can further implement a download function. The user does not know which computing unit log files exist in the /data/20211027 directory, and it will be very slow to obtain the directory list by requesting the address when a large number of files are stored in the current directory, so this application pre-creates an index for the log files in this directory And save it in the /data/ directory, and agree to use the data.json file to store the index in this directory. After writing the script, put the script on the disk for persistence, and then use the timing trigger function that comes with the operating system to trigger. The scheduled task can be set to execute once every minute, and the executed file will be written to the specified directory. Users can obtain this file to quickly obtain the log files corresponding to all computing units in the directory.
为了在同样的机器(即存储日志文件的服务器)上部署一个开源的静态服务，可以选择Nginx，Nginx具有极低的资源使用，可以提供高性能的网站服务。Nginx具有目录列表功能，将/data/目录配置为静态站点。这样就可以通过URL下载目标集群的日志文件了。In order to deploy an open source static service on the same machine (that is, the server storing log files), you can choose Nginx, which has extremely low resource usage and can provide high-performance website services. Nginx has a directory listing function and configures the /data/ directory as a static site. In this way, the log file of the target cluster can be downloaded through the URL.
图6示出了本申请实施例提供的一种计算机存储介质的硬件结构示意图。FIG. 6 shows a schematic diagram of a hardware structure of a computer storage medium provided by an embodiment of the present application.
计算机存储介质可以包括处理器601以及存储有计算机程序指令的存储器602。The computer storage medium may include a processor 601 and a memory 602 storing computer program instructions.
具体地，上述处理器601可以包括中央处理器(Central Processing Unit，CPU)，或者特定集成电路(Application Specific Integrated Circuit，ASIC)，或者可以被配置成实施本申请实施例的一个或多个集成电路。Specifically, the above-mentioned processor 601 may include a central processing unit (Central Processing Unit, CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application .
存储器602可以包括用于数据或指令的大容量存储器。举例来说而非限制，存储器602可包括硬盘驱动器(Hard Disk Drive，HDD)、软盘驱动器、闪存、光盘、磁光盘、磁带或通用串行总线(Universal Serial Bus，USB)驱动器或者两个或更多个以上这些的组合。在一个实例中，存储器602可以包括可移除或不可移除(或固定)的介质，或者存储器602是非易失性固态存储器。存储器602可在综合网关容灾设备的内部或外部。 Memory 602 may include mass storage for data or instructions. By way of example and not limitation, memory 602 may include a hard disk drive (Hard Disk Drive, HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (Universal Serial Bus, USB) drive or two or more Combinations of multiple of the above. In one example, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be a non-volatile solid-state memory. The storage 602 can be inside or outside the comprehensive gateway disaster recovery device.
在一个实例中，存储器602可包括只读存储器(ROM)，随机存取存储器(RAM)，磁盘存储介质设备，光存储介质设备，闪存设备，电气、光学或其他物理/有形的存储器存储设备。因此，通常，存储器602包括一个或多个编码有包括计算机可执行指令的软件的有形(非暂态)计算机可读存储介质(例如，存储器设备)，并且当该软件被执行(例如，由一个或多个处理器)时，其可操作来执行参考根据本申请的一方面的方法所描述的操作。In one example, memory 602 may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions, and when the software is executed (e.g., by a or a plurality of processors) operable to perform the operations described with reference to the method according to an aspect of the present application.
处理器301通过读取并执行存储器602中存储的计算机程序指令，以实现图1以及图3所示实施例中的方法，为简洁描述在此不再赘述。The processor 301 reads and executes the computer program instructions stored in the memory 602 to implement the methods in the embodiments shown in FIG. 1 and FIG. 3 , which will not be repeated here for brevity.
在一个示例中，一种计算机存储介质还可包括通信接口603和总线610。其中，如图6所示，处理器601、存储器602、通信接口603通过总线610连接并完成相互间的通信。In one example, a computer storage medium may also include a communication interface 603 and a bus 610 . Wherein, as shown in FIG. 6 , a processor 601 , a memory 602 , and a communication interface 603 are connected through a bus 610 to complete mutual communication.
通信接口603，主要用于实现本申请实施例中各模块、装置、计算单元和/或设备之间的通信。The communication interface 603 is mainly used to realize the communication between modules, devices, computing units and/or devices in the embodiments of the present application.
总线610包括硬件、软件或两者，将在线数据流量计费设备的部件彼此耦接在一起。举例来说而非限制，总线可包括加速图形端口(Accelerated Graphics Port，AGP)或其他图形总线、增强工业标准架构(Extended Industry Standard Architecture，EISA)总线、前端总线(Front Side Bus，FSB)、超传输(Hyper Transport，HT)互连、工业标准架构(Industry Standard Architecture，ISA)总线、无限带宽互连、低引脚数(LPC)总线、存储器总线、微信道架构(MCA)总线、***组件互连(PCI)总线、PCI-Express(PCI-X)总线、串行高级技术附件(SATA)总线、视频电子标准协会局部(VLB)总线或其他合适的总线或者两个或更多个以上这些的组合。在合适的情况下，总线610可包括一个或多个总线。尽管本申请实施例描述和示出了特定的总线，但本申请考虑任何合适的总线或互连。The bus 610 includes hardware, software or both, and couples the components of the online data traffic charging device to each other. By way of example and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (Hyper Transport, HT) interconnect, Industry Standard Architecture (Industry Standard Architecture, ISA) bus, InfiniBand interconnect, Low Pin Count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, peripheral component interconnect PCI bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or other suitable bus or a combination of two or more of these combination. Bus 610 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.
另外，结合上述实施例中的一种多云的日志处理方法以及一种多云的日志的存储方法，本申请实施例可提供一种计算机存储介质来实现。该计算机存储介质上存储有计算机程序指令；该计算机程序指令被处理器执行时实现上述实施例中的任意一种多云的日志处理方法以及一种多云的日志的存储方法。In addition, in combination with the multi-cloud log processing method and the multi-cloud log storage method in the foregoing embodiments, the embodiment of the present application may provide a computer storage medium for implementation. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the multi-cloud log processing method and the multi-cloud log storage method in the foregoing embodiments are implemented.
需要明确的是，本申请并不局限于上文所描述并在图中示出的特定配置和处理。为了简明起见，这里省略了对已知方法的详细描述。在上述实施例中，描述和示出了若干具体的步骤作为示例。但是，本申请的方法过程并不限于所描述和示出的具体步骤，本领域的技术人员可以在领会本申请的精神后，作出各种改变、修改和添加，或者改变步骤之间的顺序。It is to be understood that the application is not limited to the specific configurations and processes described above and shown in the figures. For conciseness, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art may make various changes, modifications and additions, or change the order of the steps after understanding the spirit of the present application.
以上所述的结构框图中所示的功能块可以实现为硬件、软件、固件或者它们的组合。当以硬件方式实现时，其可以例如是电子电路、专用集成电路(Application Specific Integrated Circuit，ASIC)、适当的固件、插件、功能卡等等。当以软件方式实现时，本申请的元素是被用于执行所需任务的程序或者代码段。程序或者代码段可以存储在机器可读介质中，或者通过载波中携带的数据信号在传输介质或者通信链路上传送。“机器可读介质”可以包括能够存储或传输信息的任何介质。机器可读介质的例子包括电子电路、半导体存储器设备、ROM、闪存、可擦除ROM(EROM)、软盘、CD-ROM、光盘、硬盘、光纤介质、射频(Radio Frequency，RF)链路，等等。代码段可以经由诸如因特网、内联网等的计算机网络被下载。The functional blocks shown in the structural block diagrams described above may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a function card, and the like. When implemented in software, the elements of the present application are the programs or code segments employed to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves. "Machine-readable medium" may include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, etc. wait. Code segments may be downloaded via a computer network such as the Internet, an Intranet, or the like.
还需要说明的是，本申请中提及的示例性实施例，基于一系列的步骤或者装置描述一些方法或***。但是，本申请不局限于上述步骤的顺序，也就是说，可以按照实施例中提及的顺序执行步骤，也可以不同于实施例中的顺序，或者若干步骤同时执行。It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiment, or may be different from the order in the embodiment, or several steps may be performed simultaneously.
上面参考根据本申请的实施例的方法、装置(***)和计算机程序产品的流程图和/或框图描述了本申请的各方面。应当理解，流程图和/或框图中的每个方框以及流程图和/或框图中各方框的组合可以由计算机程序指令实现。这些计算机程序指令可被提供给通用计算机、专用计算机、或其它可编程数据处理装置的处理器，以产生一种机器，使得经由计算机或其它可编程数据处理装置的处理器执行的这些指令使能对流程图和/或框图的一个或多个方框中指定的功能/动作的实现。这种处理器可以是但不限于是通用处理器、专用处理器、特殊应用处理器或者现场可编程逻辑电路。还可理解，框图和/或流程图中的每个方框以及框图和/或流程图中的方框的组合，也可以由执行指定的功能或动作的专用硬件来实现，或可由专用硬件和计算机指令的组合来实现。Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart and/or block diagrams, and combinations of blocks in the flowchart and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that execution of these instructions via the processor of the computer or other programmable data processing apparatus enables Implementation of the functions/actions specified in one or more blocks of the flowchart and/or block diagrams. Such processors may be, but are not limited to, general purpose processors, special purpose processors, application specific processors, or field programmable logic circuits. It can also be understood that each block in the block diagrams and/or flowcharts and combinations of blocks in the block diagrams and/or flowcharts can also be realized by dedicated hardware for performing specified functions or actions, or can be implemented by dedicated hardware and combination of computer instructions.
以上所述，仅为本申请的具体实施方式，所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的***、模块和计算单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。应理解，本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system, modules and computing units can be implemented by referring to the aforementioned methods The corresponding process in the example will not be repeated here. It should be understood that the protection scope of the present application is not limited thereto, and any person familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the application, and these modifications or replacements should cover all Within the protection scope of this application.

Claims (10)

  1. 一种多云的日志处理方法，其特征在于，应用于日志检索端，包括：A multi-cloud log processing method is characterized in that it is applied to the log retrieval end, including:
    接收用户输入的目标计算单元标识；receiving the target computing unit identifier input by the user;
    根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元；Obtain a log of the target computing unit from the target cluster according to the target computing unit identifier, where the target computing unit is a computing unit in the target cluster;
    向用户展示所述目标计算单元的日志。The log of the target computing unit is displayed to the user.
  2. 根据权利要求1所述的多云的日志处理方法，其特征在于，所述根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，包括：The multi-cloud log processing method according to claim 1, wherein the obtaining the log of the target computing unit from the target cluster according to the target computing unit identifier includes:
    根据所述目标计算单元标识，获取所述目标计算单元的基本信息；Obtain basic information of the target computing unit according to the target computing unit identifier;
    基于所述基本信息，生成请求地址；Generate a request address based on the basic information;
    将所述请求地址发送至第一网关，所述第一网关用于根据所述请求地址从所述目标集群中获取目标计算单元的日志。The request address is sent to a first gateway, and the first gateway is configured to acquire the log of the target computing unit from the target cluster according to the request address.
  3. 根据权利要求2所述的多云的日志处理方法，其特征在于，所述基本信息包括所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，所述基于所述基本信息，生成请求地址，包括：The multi-cloud log processing method according to claim 2, wherein the basic information includes the target computing unit's running date, running start time, running end time, and the identity of the target cluster, and the based on the The above basic information to generate a request address, including:
    基于所述目标计算单元的运行日期、运行开始时间、运行结束时间以及所述目标集群的标识，生成所述目标集群的标识指向的所述目标计算单元在所述运行日期期间每一天从所述运行开始时间开始、到所述运行结束时间结束的日志对应的请求地址。Based on the operation date, operation start time, operation end time, and the identification of the target cluster of the target computing unit, the target computing unit pointed to by the identification of the target cluster is generated every day during the operation date from the The request address corresponding to the log from the running start time to the running end time.
  4. 根据权利要求3所述的多云的日志处理方法，其特征在于，所述根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志后，还包括：The multi-cloud log processing method according to claim 3, wherein after obtaining the log of the target computing unit from the target cluster according to the target computing unit identifier, further comprising:
    接收所述目标计算单元在运行日期期间每一天的日志；receiving logs for each day during the run dates of the target computing unit;
    将所述目标计算单元在运行日期期间每一天的日志的空日志剔除，并生成展示日志结果，向用户展示所述展示日志结果。Empty logs of the logs of each day during the running date of the target computing unit are removed, and display log results are generated, and the display log results are displayed to the user.
  5. 根据权利要求2所述的多云的日志处理方法，其特征在于，所述将所述请求地址发送至第一网关后，还包括：The multi-cloud log processing method according to claim 2, wherein after sending the request address to the first gateway, further comprising:
    在所述目标集群为国外区域的集群时，所述第一网关将所述请求地址发送至第二网关，所述第二网关用于从所述国外区域的目标集群中获取目标计算单元的日志，所述第一网关与第二网关之间通过专网连接。When the target cluster is a cluster in a foreign region, the first gateway sends the request address to a second gateway, and the second gateway is used to obtain the log of the target computing unit from the target cluster in the foreign region , the first gateway is connected to the second gateway through a dedicated network.
  6. 根据权利要求2所述的多云的日志处理方法，其特征在于，所述根据所述目标计算单元标识，获取所述目标计算单元的基本信息，包括：The multi-cloud log processing method according to claim 2, wherein said acquiring basic information of the target computing unit according to the target computing unit identifier includes:
    根据所述目标计算单元标识，从信息***中获取所述目标计算单元的基本信息，所述信息***用于实时记录所述目标计算单元的基本信息。According to the identification of the target computing unit, the basic information of the target computing unit is obtained from an information system, and the information system is used to record the basic information of the target computing unit in real time.
  7. 一种多云的日志的存储方法，其特征在于，应用于集群，所述集群通过内网连接存储装置，所述集群中设有至少一个计算单元，包括：A multi-cloud log storage method is characterized in that it is applied to a cluster, the cluster is connected to a storage device through an intranet, and at least one computing unit is provided in the cluster, including:
    采集目标计算单元生成的日志；Collect logs generated by the target computing unit;
    将所述目标计算单元生成的日志通过解析、格式化后存储在所述存储装置中；Parse and format the log generated by the target computing unit and store it in the storage device;
    其中，所述目标计算单元生成的日志用于在用户输入目标计算单元标识时，通过日志检索端接收用户输入的目标计算单元标识；根据所述目标计算单元标识，从目标集群中获取目标计算单元的日志，所述目标计算单元为所述目标集群中的计算单元；向用户展示所述目标计算单元的日志。Wherein, the log generated by the target computing unit is used to receive the target computing unit identifier input by the user through the log retrieval terminal when the user inputs the target computing unit identifier; obtain the target computing unit from the target cluster according to the target computing unit identifier logs, the target computing unit is a computing unit in the target cluster; and the log of the target computing unit is displayed to the user.
  8. 根据权利要求7所述的多云的日志的存储方法，其特征在于，所述将所述目标计算单元生成的日志通过解析、格式化后存储在所述存储装置中，包括：The method for storing multi-cloud logs according to claim 7, wherein the parsing and formatting the logs generated by the target computing unit and storing them in the storage device includes:
    将所述目标计算单元生成的日志通过解析、格式化后，按照日志的生成日期存储在所述存储装置中。After parsing and formatting the log generated by the target computing unit, it is stored in the storage device according to the generation date of the log.
  9. 根据权利要求7所述的多云的日志的存储方法，其特征在于，所述存储装置通过磁盘存储日志信息。The method for storing multi-cloud logs according to claim 7, wherein the storage device stores log information through a disk.
  10. 一种计算机存储介质，其特征在于，所述计算机存储介质上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现如权利要求1-9任意一项所述的一种多云的日志处理方法以及一种多云的日志的存储方法。A computer storage medium, characterized in that computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, a multi-cloud log according to any one of claims 1-9 is realized A processing method and a multi-cloud log storage method.
PCT/CN2021/143803 2021-12-31 2021-12-31 Multi-cloud log processing method, storage method, and computer storage medium WO2023123397A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/143803 WO2023123397A1 (en) 2021-12-31 2021-12-31 Multi-cloud log processing method, storage method, and computer storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/143803 WO2023123397A1 (en) 2021-12-31 2021-12-31 Multi-cloud log processing method, storage method, and computer storage medium

Publications (1)

Publication Number Publication Date
WO2023123397A1 true WO2023123397A1 (en) 2023-07-06

Family

ID=86997253

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/143803 WO2023123397A1 (en) 2021-12-31 2021-12-31 Multi-cloud log processing method, storage method, and computer storage medium

Country Status (1)

Country Link
WO (1) WO2023123397A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103853743A (en) * 2012-11-29 2014-06-11 百度在线网络技术（北京）有限公司 Distributed system and log query method thereof
CN103929329A (en) * 2014-04-14 2014-07-16 百度在线网络技术（北京）有限公司 Log processing and configuration method, configuration server, service end device and system
CN111190875A (en) * 2019-12-27 2020-05-22 航天信息股份有限公司 Log aggregation method and device based on container platform
US20200293220A1 (en) * 2017-12-05 2020-09-17 Huawei Technologies Co., Ltd. Log processing method, and related device and system
CN113687974A (en) * 2021-10-22 2021-11-23 飞狐信息技术（天津）有限公司 Client log processing method and device and computer equipment

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103853743A (en) * 2012-11-29 2014-06-11 百度在线网络技术（北京）有限公司 Distributed system and log query method thereof
CN103929329A (en) * 2014-04-14 2014-07-16 百度在线网络技术（北京）有限公司 Log processing and configuration method, configuration server, service end device and system
US20200293220A1 (en) * 2017-12-05 2020-09-17 Huawei Technologies Co., Ltd. Log processing method, and related device and system
CN111190875A (en) * 2019-12-27 2020-05-22 航天信息股份有限公司 Log aggregation method and device based on container platform
CN113687974A (en) * 2021-10-22 2021-11-23 飞狐信息技术（天津）有限公司 Client log processing method and device and computer equipment

Similar Documents

Publication Publication Date Title
CN110321387B (en) Data synchronization method, equipment and terminal equipment
US10187461B2 (en) Configuring a system to collect and aggregate datasets
US9817859B2 (en) Collecting and aggregating log data with fault tolerance
US9817867B2 (en) Dynamically processing an event using an extensible data model
US9082127B2 (en) Collecting and aggregating datasets for analysis
US7685270B1 (en) Method and apparatus for measuring latency in web services
US9697104B2 (en) End-to end tracing and logging
CN104036025A (en) Distribution-base mass log collection system
WO2008064593A1 (en) A log analyzing method and system based on distributed compute network
US11030384B2 (en) Identification of sequential browsing operations
US11570078B2 (en) Collecting route-based traffic metrics in a service-oriented system
CN108737549A (en) A kind of log analysis method and device of big data quantity
CN110109660A (en) A kind of monitoring short message touching reaches the system and method for effect
US10397343B1 (en) Distributed trace storage in a service-oriented system
CN112948492A (en) Data processing system, method and device, electronic equipment and storage medium
CN104899134A (en) Automatic domain name registration server testing system and method
US10812346B1 (en) Application discovery and dependency mapping
CN109902070B (en) WiFi log data-oriented analysis storage search method
WO2023123397A1 (en) Multi-cloud log processing method, storage method, and computer storage medium
CN117076250A (en) Data processing method and device
CN117271584A (en) Data processing method and device, computer readable storage medium and electronic equipment
CN108959041B (en) Method for transmitting information, server and computer readable storage medium
CN111698109A (en) Method and device for monitoring log
CN115604343A (en) Data transmission method, system, electronic equipment and storage medium
CN114443602A (en) Multi-cloud log processing method, storage method and computer storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21969744

Country of ref document: EP

Kind code of ref document: A1