CN112988569A

CN112988569A - Method and system for viewing micro-service request response based on nginx

Info

Publication number: CN112988569A
Application number: CN202110164811.5A
Authority: CN
Inventors: 邢红瑞
Original assignee: Changyang Tech Beijing Co ltd
Current assignee: Changyang Tech Beijing Co ltd
Priority date: 2021-02-05
Filing date: 2021-02-05
Publication date: 2021-06-18

Abstract

The invention provides a micro-service request response checking method and system based on nginx, which comprises the steps of customizing the format of a nginx log by using the log _ format of the nginx, embedding the lua into the nginx by using a lua-nginx-module, and compiling and acquiring the configuration of response data returned by a back-end server in the protocol of the lua; the nginx receives the request data and the response data, outputs the request data and the response data by using an access _ log of the nginx to generate a nginx log, and outputs the nginx log to a back-end server; writing configuration supporting syslog in nginx. conf of nginx, and outputting the nginx log to a syslog server; the nginx log in the syslog server is stored into an elastic search, where it is viewed. The method does not influence the normal operation of the on-line microservice application, is convenient for data query, has strong applicability, is highly decoupled from the application at the rear end, and is easy for later maintenance and expansion. The method can help developers to quickly analyze and diagnose the performance bottleneck under the distributed application architecture, and improve the development and diagnosis efficiency under the micro-service era.

Description

Method and system for viewing micro-service request response based on nginx

Technical Field

The invention relates to the technical field of industrial control, in particular to a mini-service request response checking method and system based on nginx.

Background

With the continuous development of science and technology, B/S architecture has replaced C/S architecture to become the mainstream mode of software architecture, management systems and the like in various industries are developed by using B/S-based software architecture mode, and are gradually changed from early single-application architecture to micro-service architecture, API gateway is an architecture mode which is created along with micro-service, and is used for solving the problem that micro-service is dispersed and a unified gateway does not have to perform inlet traffic management.

The existing micro-service data viewing mode has the following defects:

1. http input and output modules are required to be applied to the microserver, requests and returned JSON data are printed, and running of a program is seriously influenced;

2. if tcpdump capture analysis is used on the API gateway machine, the workload to troubleshoot and locate a particular URL request is enormous because of the large amount of data requested.

Disclosure of Invention

The invention provides a mini-service request response checking method and system based on nginx, which aim to overcome the defects of the prior art.

In one aspect, the invention provides a mini-service request response viewing method based on nginx, which comprises the following steps:

s1: utilizing log _ format of nginx to self-define the format of a nginx log, utilizing lua-nginx-module to embed the lua into the nginx, and compiling and acquiring the configuration of response data returned by a back-end server in the coroutine of the lua;

s2: the nginx receives the request data and the response data, outputs the request data and the response data by using an access _ log of the nginx to generate a nginx log, and outputs the nginx log to a back-end server;

s3: writing configuration supporting syslog in nginx. conf of nginx, and outputting the nginx log to a syslog server;

s4: the nginx log in the syslog server is stored into an elastic search, where it is viewed.

According to the method, the nginx is used as the API gateway to check the output of the micro service data, the contents in the response data returned by the back-end server are extracted by combining the coroutine of the lua, the request data and the corresponding response data are written into the nginx log, and finally the nginx log is output and stored by sequentially utilizing syslog and elastic search. The method does not influence the normal operation of the on-line microservice application, is convenient for data query, has strong applicability, is highly decoupled from the application at the rear end, and is easy for later maintenance and expansion. The method can help developers to quickly analyze and diagnose the performance bottleneck under the distributed application architecture, and improve the development and diagnosis efficiency under the micro-service era.

In a specific embodiment, the customizing of the format of the nginx log by using the log _ format of the nginx specifically includes the following steps:

and configuring and acquiring the content of a post request body in the request data, the request for accessing the ip address of the client and the full path of http by using log _ format, and writing into the nginx log after configuring and converting an escape json field into a json format.

In a specific embodiment, the writing of the configuration for obtaining the response data returned by the back-end server in the protocol of the lua specifically includes:

defining a variable resp _ body used for acquiring a post request body in the response data in the location of nginx;

circularly acquiring an ngx.ctx.buffered variable of nginx in a body _ filter _ by _ lua running stage in the coroutine of the lua, accumulating the ngx.ctx.buffered variable, and assigning the accumulated data to the resp _ body variable;

and configuring a log file for viewing the response data in the location.

Because Nginx does not have a built-in httpbody variable for acquiring the post request, the method obtains the variable by using the body processing stage of the lua-Nginx-module in the coroutine process of the lua.

In a specific embodiment, the writing of the configuration supporting syslog in nginx. conf of nginx in S3 specifically includes:

adding a configuration for receiving the nginx log by a syslog server in nginx.conf;

conf adds the address of the syslog server that receives the nginx log.

The data are collected to a centralized log server, namely a syslog server, so that a large amount of user request data are prevented from being stored on an API gateway to occupy a hard disk of the gateway, the requested data are conveniently managed in a unified mode, and the workload and the management difficulty of operation and maintenance are reduced.

In a specific embodiment, said checking said nginx log in an ElasticSearch specifically includes:

looking up the nginx log through an index at the beginning of the nginx-access-log, thereby browsing the request data and the response data;

and searching the nginx log by configuring the kibana so as to browse the request data and the response data.

According to a second aspect of the present invention, a computer-readable storage medium is proposed, on which a computer program is stored, which computer program, when being executed by a computer processor, carries out the above-mentioned method.

According to a third aspect of the present invention, a mini-service request response viewing system based on nginx is provided, the system comprising:

nginx log format configuration module: configuring a format for customizing a nginx log by using log _ format of the nginx, embedding the lua into the nginx by using a lua-nginx-module, and compiling and acquiring configuration of response data returned by a back-end server in the coroutine of the lua;

the nginx log generation module: the nginx log generation method comprises the steps that the nginx is configured to receive request data and response data, the request data and the response data are output by using an access _ log of the nginx to generate a nginx log, and the nginx log is output to a back-end server;

nginx log output module: the configuration is used for writing configuration supporting syslog in nginx. conf of the nginx, and then outputting the nginx log to a syslog server;

nginx log storage module: configured to store the nginx log in the syslog server into an ElasticSearch, and view the nginx log in the ElasticSearch.

The method includes the steps that a format of a log _ format custom nginx log of the nginx is utilized, the lua is embedded into the nginx through the lua-nginx-module, and configuration of response data returned by a back-end server is compiled and obtained in the protocol of the lua; the nginx receives the request data and the response data, outputs the request data and the response data by using an access _ log of the nginx to generate a nginx log, and outputs the nginx log to a back-end server; writing configuration supporting syslog in nginx. conf of nginx, and outputting the nginx log to a syslog server; the nginx log in the syslog server is stored into an elastic search, where it is viewed. The method does not influence the normal operation of the on-line microservice application, is convenient for data query, has strong applicability, is highly decoupled from the application at the rear end, and is easy for later maintenance and expansion. The method can help developers to quickly analyze and diagnose the performance bottleneck under the distributed application architecture, and improve the development and diagnosis efficiency under the micro-service era.

Drawings

The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and together with the description serve to explain the principles of the invention. Other embodiments and many of the intended advantages of embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;

FIG. 2 is a flow chart of a nginx-based micro-service request response viewing method according to an embodiment of the present invention;

FIG. 3 is a block diagram of a nginx-based microservice request response viewing system in accordance with one embodiment of the present invention;

FIG. 4 is a block diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Fig. 1 shows an exemplary system architecture 100 to which a tiny service request response viewing method based on nginx according to an embodiment of the present application may be applied.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various applications, such as a data processing application, a data visualization application, a web browser application, etc., may be installed on the

terminal devices

101, 102, 103.

The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices including, but not limited to, smart phones, tablet computers, laptop portable computers, desktop computers, and the like. When the

terminal apparatuses

101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.

The server 105 may be a server that provides various services, such as a background information processing server that provides support for requested data presented on the

terminal devices

101, 102, 103. The background information processing server may process the acquired request data and generate a processing result (e.g., nginx log).

It should be noted that the method provided in the embodiment of the present application may be executed by the server 105, or may be executed by the

terminal devices

101, 102, and 103, and the corresponding apparatus is generally disposed in the server 105, or may be disposed in the

terminal devices

101, 102, and 103.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

According to an embodiment of the invention, a tiny service request response viewing method based on nginx is provided, and fig. 2 shows a flow chart of the tiny service request response viewing method based on nginx according to the embodiment of the invention. As shown in fig. 2, the method comprises the steps of:

s201: and customizing the format of the nginx log by using the log _ format of the nginx, embedding the lua into the nginx by using the lua-nginx-module, and compiling and acquiring the configuration of response data returned by the back-end server in the coroutine of the lua.

In a specific embodiment, the log _ format custom output format is used as follows:

log_format api escape＝json'$remote_addr"$request"request_body:"$request_body"resp_body:"$resp_body"'；

wherein, escape is converted into json format without the need of escape variable content; $ request _ body is the content of the request body; $ resp _ body is a variable of the post request body that we define by themselves; $ remote _ addr is the ip address of the accessing client; and $ request is a request for the full path for http.

and configuring a log file for viewing the response data in the location.

In a specific embodiment, since Nginx does not have an http body variable built in to get post request, this variable may be obtained at the body processing stage of the lua-Nginx-module. Therefore, for response data returned by the back-end server, the ngx.ctx.buffered variable of the nginx is needed to be used for obtaining, the nginx is added first to configure the lua _ need _ request _ body on, and the following specific configurations can be used:

1) define a variable, set $ resp _ body "" in the location of nginx;

2) and circularly acquiring ngx.ctx.buffered in the body _ filter _ by _ lua, wherein the specific codes are as follows:

local resp_body＝string.sub(ngx.arg[1],1,1000)

ngx.ctx.buffered＝(ngx.ctx.buffered or"")..resp_body

if ngx.arg[2]then

ngx.var.resp_body＝ngx.ctx.buffered

end

the accumulated data is placed in the previously defined variable resp _ body;

3) configuring access _ log/var/nignx/log/api.log api in location;

log files can be used to view the detailed information of each http request when access is requested.

S202: and the nginx receives the request data and the response data, outputs the request data and the response data by using an access _ log of the nginx to generate the nginx log, and outputs the nginx log to a back-end server.

S203: conif of nginx writes configuration supporting syslog, and outputs the nginx log to a syslog server.

In a specific embodiment, the writing of the configuration supporting syslog in nginx.conf of nginx in S203 specifically includes:

conf adds the address of the syslog server that receives the nginx log.

In a specific embodiment, the following syslog configuration may be used:

access_log syslog:server＝127.0.0.1:3303 api；

wherein, the syslog represents that the setting log is received by the syslog service; server represents the address of the syslog server receiving the syslog log, default to udp protocol, port 514, and in this embodiment we use port 3303.

S204: the nginx log in the syslog server is stored into an elastic search, where it is viewed.

In a specific embodiment, step S204 uses logstack to directly output the nginx log sent by the nginx to the ElasticSearch, and the specific method is as follows:

logstash monitors 3303 port to receive data by using input plug-in, and directly outputs the data to ElasticSearch by using output;

the implementation code is as follows:

with the above arrangement, the request and response data can be viewed by directly viewing the index at the beginning of the nginx-access-log in the elastic search, or by configuring the kibana.

Fig. 3 shows a frame diagram of a nginx-based microservice request response viewing system according to an embodiment of the invention. The system comprises a nginx log format configuration module 301, a nginx log generation module 302, a nginx log output module 303 and a nginx log storage module 304.

In a specific embodiment, the nginx log format configuration module 301 is configured to customize the format of the nginx log by using the log _ format of the nginx, embed the lua in the nginx by using the lua-nginx-module, and write and acquire a configuration of response data returned by the backend server in the coroutine of the lua;

the nginx log generation module 302 is configured to receive request data and response data by the nginx, output the request data and the response data by using an access _ log of the nginx to generate a nginx log, and output the nginx log to a back-end server;

the nginx log output module 303 is configured to write a configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to a syslog server;

the nginx log storage module 304 is configured to store the nginx log in the syslog server into an ElasticSearch where the nginx log is viewed.

The system uses nginx as an API gateway to check micro-service data output, simultaneously extracts content in response data returned by a back-end server by combining the coroutine of lua, simultaneously writes request data and corresponding response data into a nginx log, and finally sequentially outputs and stores the nginx log by using syslog and ElasticSearch. The method does not influence the normal operation of the on-line microservice application, is convenient for data query, has strong applicability, is highly decoupled from the application at the rear end, and is easy for later maintenance and expansion. The method can help developers to quickly analyze and diagnose the performance bottleneck under the distributed application architecture, and improve the development and diagnosis efficiency under the micro-service era.

Referring now to FIG. 4, shown is a block diagram of a computer system 400 suitable for use in implementing the electronic device of an embodiment of the present application. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 4, the computer system 400 includes a Central Processing Unit (CPU)401 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the system 400 are also stored. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 704.

The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a display such as a Liquid Crystal Display (LCD) and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. A driver 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 410 as necessary, so that a computer program read out therefrom is mounted into the storage section 408 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 409, and/or installed from the removable medium 411. The computer program performs the above-described functions defined in the method of the present application when executed by a Central Processing Unit (CPU) 401. It should be noted that the computer readable storage medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present application may be implemented by software or hardware. The units described may also be provided in a processor, and the names of the units do not in some cases constitute a limitation of the unit itself.

Embodiments of the present invention also relate to a computer-readable storage medium having stored thereon a computer program which, when executed by a computer processor, implements the method above. The computer program comprises program code for performing the method illustrated in the flow chart. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable medium or any combination of the two.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A mini-service request response viewing method based on nginx is characterized by comprising the following steps:

2. The method according to claim 1, wherein the customizing of the format of the nginx log by using log _ format of the nginx comprises the following specific steps:

3. The method according to claim 1, wherein the writing of the configuration for obtaining the response data returned by the back-end server in the protocol of the lua comprises the specific steps of:

and configuring a log file for viewing the response data in the location.

4. The method according to claim 1, wherein the writing of the configuration supporting syslog in nginx. conf of nginx in S3 specifically comprises:

conf adds the address of the syslog server that receives the nginx log.

5. The method of claim 1, wherein said viewing said nginx log in an ElasticSearch specifically comprises:

6. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a computer processor, carries out the method of any one of claims 1 to 5.

7. A nginx-based microservice request response viewing system, comprising: