CN113206797A - Flow control method and device, electronic equipment and storage medium - Google Patents
Flow control method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113206797A CN113206797A CN202110494968.4A CN202110494968A CN113206797A CN 113206797 A CN113206797 A CN 113206797A CN 202110494968 A CN202110494968 A CN 202110494968A CN 113206797 A CN113206797 A CN 113206797A
- Authority
- CN
- China
- Prior art keywords
- configuration parameters
- flow control
- log data
- request
- index value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000004458 analytical method Methods 0.000 claims abstract description 17
- 230000004044 response Effects 0.000 claims description 38
- 230000002159 abnormal effect Effects 0.000 claims description 16
- 230000000903 blocking effect Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 238000005096 rolling process Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/11—Identifying congestion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/20—Traffic policing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a flow control method, a flow control device, electronic equipment and a storage medium, wherein the method comprises the following steps: analyzing the acquired log information, and sending the log data obtained by analysis to kafka message middleware; performing index statistics on log data by using the log data in the flink consumption kafka message middleware according to the configuration parameters of the flink; and if the counted index value is larger than the corresponding preset index threshold value in the configuration parameters, intercepting the request corresponding to the counted index value. The method can timely detect the state of the server and intercept the request of the index value exceeding the preset index threshold, and the method can process according to the counted index value, find the source of the fault, and achieve rational data.
Description
Technical Field
The present application relates to the field of flow control technologies, and in particular, to a flow control method and apparatus, an electronic device, and a storage medium.
Background
Due to the fact that the external access amount is too large, great pressure is caused on the server, and even downtime occurs. Many current control methods for flow are passive, for example, when the memory or CPU of a server is detected to be fast-used, an alarm is generated, and related personnel deal with the alarm in an emergency. Usually, a simple and rough mode is adopted, some IP with high access quantity is directly forbidden, the fault tracing difficulty is high, and some more detailed index data and a more flexible, personalized and intelligent processing mode cannot be provided.
Disclosure of Invention
The application aims to provide a flow control method, a flow control device, electronic equipment and a storage medium, which can timely detect the state of a server and intercept a request of which an index value exceeds a preset index threshold value. The specific scheme is as follows:
in a first aspect, the present application discloses a flow control method, including:
analyzing the acquired log information, and sending the log data obtained by analysis to kafka message middleware;
consuming the log data in the kafka message middleware by using a flink, and carrying out index statistics on the log data according to the configuration parameters of the flink;
and if the counted index value is larger than the corresponding preset index threshold value in the configuration parameters, intercepting the request corresponding to the counted index value.
Optionally, before consuming the log data in the kafka message middleware by using a flink, the method further includes:
and creating the configuration parameters of the flink according to the service demand information, and sending the configuration parameters to the kafka message middleware.
Optionally, performing index statistics on the log data according to the configuration parameters of the flink, including:
dividing a time window according to a time window parameter in the configuration parameters of the flink;
and calculating the index value of each IP address requesting to access the same URL in the time window.
Optionally, if the counted index value is greater than the preset index threshold corresponding to the configuration parameter, intercepting the request corresponding to the counted index value, including:
and if the counted index value of the access target URL is larger than the preset index threshold value corresponding to the target URL in the configuration parameters, forbidding the IP address request accessing the target URL to access the target URL within the target blocking time length according to the target blocking time length corresponding to the target URL in the configuration parameters.
Optionally, if the counted index value is greater than the preset index threshold corresponding to the configuration parameter, intercepting the request corresponding to the counted index value, including:
and if the counted index value meets at least one of the condition that the request quantity in the rolling time window is greater than the maximum request quantity, the abnormal state code ratio is greater than a preset abnormal state code ratio threshold value and the response timeout time ratio is greater than a preset response timeout time ratio threshold value, intercepting the request corresponding to the counted index value.
Optionally, sending the log data obtained by analyzing the log information to the kafka message middleware, where the sending includes:
analyzing the log information according to the format of the log information, and taking the IP address, the request URL, the request time, the request response time and the response state code obtained by analysis as the log data;
sending the log data to a subject of the kafka message middleware.
In a second aspect, the present application discloses a flow control device comprising:
the analysis module is used for analyzing the acquired log information and sending the log data obtained through analysis to the kafka message middleware;
the statistical module is used for consuming the log data in the kafka message middleware by using a flink and carrying out index statistics on the log data according to the configuration parameters of the flink;
and the intercepting module is used for intercepting the request corresponding to the index value obtained by statistics if the index value obtained by statistics is larger than the corresponding preset index threshold value in the configuration parameters.
Optionally, the method further includes:
and the sending module is used for creating the configuration parameters of the flink according to the service demand information and sending the configuration parameters to the kafka message middleware.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
and a processor for implementing the steps of the flow control method when executing the computer program.
In a fourth aspect, the present application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the flow control method as described above.
The application provides a flow control method, which comprises the following steps: analyzing the acquired log information, and sending the log data obtained by analysis to kafka message middleware; consuming the log data in the kafka message middleware by using a flink, and carrying out index statistics on the log data according to the configuration parameters of the flink; and if the counted index value is larger than the corresponding preset index threshold value in the configuration parameters, intercepting the request corresponding to the counted index value.
Therefore, the log data are subjected to index statistics by using the flink real-time stream frame, the flink has the characteristics of high throughput and low time delay, a large amount of log data can be processed in real time, and when the index value is greater than the corresponding preset index threshold value through statistics calculation, the request corresponding to the index value is intercepted. This application utilizes flink real-time flow frame to carry out the index statistics to log data promptly, the real-time is high, the state that detects the server that can be timely, and intercept the request that the index value exceeds preset index threshold value, and this application can handle according to the index value of statistics, can find the trouble and trace to the source, accomplish rational basis promptly, only detect when server memory or CPU use light soon among the correlation technique, lead to the server to shut down, and simple rough storm directly forbids the IP that the access capacity is high, can't carry out the defect of trouble tracing to the source, this application can in time respond to the abnormal handling of server, and can realize the trouble and trace to the source, the basis carries out the fault handling, it is more intelligent. The application also provides a flow control device, an electronic device and a computer readable storage medium, which have the beneficial effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a flow control method according to an embodiment of the present application;
FIG. 2 is a system block diagram of a flow control system provided in an embodiment of the present application;
fig. 3 is a schematic structural diagram of a flow control device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Many of the current control methods for flow are passive. For example, only when the memory or the CPU of the server is detected to be running out, an alarm is generated, and the relevant personnel deal with the alarm in an emergency. The usual approach is to disable some high access IPs directly. The failure tracing difficulty is high, and some more detailed index data and more flexible, personalized and intelligent disposal modes can not be provided. Based on the foregoing technical problem, this embodiment provides a flow control method, which can respond to exception handling of a server in time, and can implement fault tracing, and based on fault handling, the flow control method is more intelligent, specifically refer to fig. 1, where fig. 1 is a flowchart of the flow control method provided in this embodiment of the present application, and specifically includes:
and S101, analyzing the acquired log information, and sending the log data obtained through analysis to the kafka message middleware.
It is understood that when the server provides an external web service, a request of the user is recorded in a log, and log information is generated. The present embodiment does not limit the specific content of the log information, and may include information such as an IP address, a request URL, a request time, a request response time, and a response status code. In this embodiment, the obtained log information is analyzed, and the analyzed log data is sent to kafka message middleware, where kafka is an open source stream processing platform, is a high-throughput distributed publish-subscribe message system, and is mainly used for processing stream data, and different data can be put into different topics (Topic). The present embodiment does not limit the specific content of the log data obtained by the analysis, and may include information such as an IP address, a request URL, request time, request response time, and a response status code. It is understood that the content stored in the log information contains log data, and there is a difference between the stored formats of the log information and the log data.
The embodiment does not limit the specific content of the log data, and can obtain the log data according to the actual business logic requirement. In a specific embodiment, sending log data obtained by parsing log information to the kafka message middleware may include:
analyzing the log information according to the format of the log information, and taking the IP address, the request URL, the request time, the request response time and the response state code obtained by analysis as log data;
sending the log data to the subject of the kafka message middleware.
The present embodiment does not limit the specific format of the log information, and it can be understood that there may be differences in the storage formats of the log information collected in different web services. The embodiment parses the log information according to the specific format of the log information, obtains fields of an IP address, a request URL, a request time, a request response time and a response status code, and sends the fields as log data to the subject of the kafka message middleware.
S102, utilizing the log data in the flash consumption kafka message middleware, and carrying out index statistics on the log data according to the configuration parameters of the flash.
The Flink is a real-time streaming framework and has the characteristics of simultaneously supporting high throughput, low delay and high performance. In this embodiment, the flink consumes the log data in the kafka message middleware, and performs index statistics on the log data according to the configuration parameters of the flink. The present embodiment does not limit the specific index for performing statistics, and may be statistics of the request amount in the time window, statistics of the fraction of the abnormal state code, statistics of the fraction of the response timeout time, or simultaneous statistics, and may be set according to specific service requirements.
The present embodiment does not limit the specific location where the configuration parameter of the flink exists, and may be stored in the kafka message middleware, or may be stored in another location of the server. In a specific embodiment, in order to improve the efficiency and reliability of the index statistics, before consuming log data in the kafka message middleware by using flink, the method may further include:
and creating a configuration parameter of the flink according to the service demand information, and sending the configuration parameter to the kafka message middleware.
The embodiment creates the configuration parameters of the flink according to the service requirement information and stores the configuration parameters in the kafka message middleware. The embodiment does not limit the specific content of the service requirement information, and can be set according to the actual situation. The present embodiment also does not limit the specific configuration parameter types of the flink created, e.g., the URL to be monitored may include the time window size, the URL to be monitored, the maximum request amount max _ qps within the time window (for example, 10 tens of thousands of requests in a minute, which may cause pressure on the server), the exception status code status (200 status codes for successful requests, and exception status codes for other status codes), the exception status code ratio status _ ratio (number of returned exception status codes/total number of accesses, and if the ratio of exception status codes exceeds a set threshold, the pressure on the server may be increased), the response timeout duration (setting a time threshold beyond which a timeout may affect the efficiency of the server), the response timeout duration _ ratio (number of response times/total number of accesses), the blocking time duration (the length of blocking the IP request), and so on. The configuration parameters are stored in the kafka message middleware, log data and the configuration parameters are consumed from the kafka message middleware by using a flink real-time streaming framework, and compared with the configuration parameters obtained from a server, the configuration parameters are more efficient and more reliable.
The embodiment does not limit the specific process of performing index statistics on the log data according to the configuration parameters of the flink. In a specific embodiment, performing index statistics on log data according to the configuration parameters of the flink may include:
dividing a time window according to a time window parameter in the flink configuration parameters;
and calculating the index value of each IP address requesting to access the same URL in the time window.
The specific size of the time window parameter is not limited in this embodiment, and may be 1 minute or 30 minutes, and may be set according to actual requirements. The embodiment divides the time window according to the time window parameters, and calculates the index value of each IP address requesting to access the same URL in the time window. The embodiment does not limit the specific URL requested, and the user can access the URL according to actual needs. Meanwhile, the embodiment does not limit the specific object of the calculated index value, and may count the maximum request amount of each IP address request access URL, the duty ratio of the abnormal state code, the response time timeout duty ratio, and other index data within the time window. In this embodiment, the number of calculated index values is not limited, and only one of the indexes may be counted, or several of the indexes may be counted at the same time, and may be set according to specific service requirements. For example, a scroll time window (or a time window) is set to 1 minute, and the index values of the maximum request amount max _ qps, the duty ratio of the abnormal state code, and the response time timeout duty ratio for each URL requested by the IP address are counted in one minute.
S103, if the counted index value is larger than the corresponding preset index threshold value in the configuration parameters, intercepting a request corresponding to the counted index value.
The embodiment does not limit a specific process of intercepting the request corresponding to the counted index value when the counted index value is greater than the corresponding preset index threshold value. In a specific embodiment, if the counted index value is greater than the preset index threshold corresponding to the configuration parameter, intercepting a request corresponding to the counted index value may include:
and if the counted index value of the access target URL is larger than the preset index threshold value corresponding to the target URL in the configuration parameters, forbidding the IP address of the access target URL to access the target URL within the target blocking time length according to the target blocking time length corresponding to the target URL in the configuration parameters.
In this embodiment, the specific website of the target URL is not limited, and any website may be selected from the URLs to be monitored as the target URL. The present embodiment also does not limit the specific size of each preset index threshold in the configuration parameters, for example, the maximum request amount max _ qps that can be set in a time window (e.g., 1 minute) is 10 ten thousand, the preset index threshold corresponding to the response timeout time is set to 50%, the preset index threshold corresponding to the response timeout time is set to 10s, and the like. It can be understood that the target blocking time length in this embodiment is a time length for prohibiting access, the specific size of the target blocking time length is not limited in this embodiment, and may be 10 minutes or 5 minutes, and different target blocking time lengths may be set for different index items.
The embodiment does not limit the specific satisfied condition that the counted index value is greater than the corresponding preset index threshold. In a specific embodiment, if the counted index value is greater than the preset index threshold corresponding to the configuration parameter, intercepting a request corresponding to the counted index value may include:
and if the counted index value meets at least one of the condition that the request quantity in the rolling time window is larger than the maximum request quantity, the abnormal state code ratio is larger than a preset abnormal state code ratio threshold value, and the response timeout time ratio is larger than a preset response timeout time ratio threshold value, intercepting the request corresponding to the counted index value.
It can be understood that, in the embodiment, the condition that the rolling time window is set according to the time window parameter and the interception is performed in this embodiment may be that when at least one of the request amount in the rolling time window is greater than the maximum request amount, the abnormal state code proportion is greater than the preset abnormal state code proportion threshold, and the response timeout proportion is greater than the preset response timeout proportion threshold, the interception is performed without simultaneously satisfying 2 or 3 of the conditions, so that the probability of server abnormality can be effectively reduced, and the reliability of stable operation of the server is improved.
Based on the technical scheme, the log data are subjected to index statistics by using the flink real-time streaming framework, the real-time performance is high, the state of the server can be timely detected, the request that the index value exceeds the preset index threshold value is intercepted, the fault tracing can be found according to the counted index value, and the defects that the server is down due to the fact that only the server memory or CPU fast utilization light is detected in the related technology, the IP with high access capacity is directly forbidden by a simple rough storm, and the fault tracing cannot be carried out are overcome.
Specific embodiments of a flow control system are provided below. Fig. 2 is a schematic diagram of a system framework of the flow control system provided in this embodiment, which sequentially shows from left to right that the request logs obtained by parsing, that is, log data and configuration parameters of the flink, are input into different topics (Topic) in the kafka message middleware, and the flink streaming processing framework consumes the corresponding topics from the kafka message middleware to obtain the configuration parameters and the request logs continuously generated from the source. In the flink streaming framework, the statistical indexes max _ qps, status _ ratio, duration _ ratio of IP address to request URL are calculated. And if the indexes exceed the corresponding preset index threshold values in the configuration parameters, intercepting the IP according to the set interception time length, namely the target interception time length, and forbidding to access the URL. The method comprises the following steps:
1. a client requesting a different web service from a server will generate different log information in the server. Collecting web logs of each place by using a log collection system flash, analyzing log information, and extracting required fields: IP address, request URL, request time, request response time, response status code, etc. And outputting the processed log data to the subject of the kafka message middleware.
2. Manually configuring the time window size, i.e., setting the time window parameter (e.g., 1 minute), the URL to be monitored (e.g., max. com), the maximum request amount within the time window max _ qps (e.g., 1 ten thousand), the exception status code status (e.g., status code 400 for successful request, exception if 500 is returned), the exception status code ratio status _ ratio (e.g., 75%), the response timeout duration (e.g., 10s), the response timeout duration ratio duration _ ratio (e.g., 50%), the barring duration, i.e., the length of the target barring time (e.g., 10 minutes), etc., and outputting the parameters to the subject in kafka.
3. And (3) using a flink streaming processing framework, consuming log data and configuration parameters from the kafka message middleware, and dividing the time window according to the set time window parameters. Within each time window, the statistical indices max _ qps, status _ ratio, duration _ ratio for each IP address request URL are calculated.
4. And if the calculated index in the time window exceeds a preset index threshold value in the configuration parameters, performing interception operation, and prohibiting the IP address from accessing the URL within a set time (for example, 10 minutes) which is the target interception time length.
Based on the above technical solution, this embodiment is based on the powerful computing power of the flink stream processing framework, the computation of various indexes is very fast (computation is completed within a few seconds in a time window of one minute), and requests of tens of thousands of IPs are monitored simultaneously, so that real-time and efficient response processing of abnormal problems is realized. And different URLs and corresponding processing modes can be configured according to different services, so that personalized operation is realized. Meanwhile, other services are not affected, and a rough mode of cutting and intercepting is avoided. And interception can be performed according to calculation indexes such as abnormal state codes and response overtime, so that rational and more intelligent effects are achieved.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a flow control device according to an embodiment of the present disclosure, which includes:
in some specific embodiments, the method specifically includes:
the analysis module 301 is configured to analyze the obtained log information, and send the log data obtained through analysis to the kafka message middleware;
the statistical module 302 is configured to perform index statistics on log data according to a configuration parameter of the flink by using the log data in the flink consumption kafka message middleware;
the intercepting module 303 is configured to intercept a request corresponding to the counted index value if the counted index value is greater than a preset index threshold corresponding to the configuration parameter.
In some specific embodiments, the method further comprises:
and the sending module is used for creating the configuration parameters of the flink according to the service demand information and sending the configuration parameters to the kafka message middleware.
In some specific embodiments, the statistics module 302 includes:
the dividing unit is used for dividing a time window according to a time window parameter in the flink configuration parameters;
and the calculation unit is used for calculating the index value of each IP address requesting to access the same URL in the time window.
In some specific embodiments, the intercepting module 303 includes:
and the forbidding unit is used for forbidding the IP address request of the access target URL to access the target URL within the target blocking time length according to the target blocking time length corresponding to the target URL in the configuration parameters if the counted index value of the access target URL is greater than the preset index threshold corresponding to the target URL in the configuration parameters.
In some specific embodiments, the intercepting module 303 includes:
and the intercepting unit is used for intercepting the request corresponding to the index value obtained by statistics if the index value obtained by statistics meets at least one of the condition that the request amount in the rolling time window is greater than the maximum request amount, the abnormal state code ratio is greater than a preset abnormal state code ratio threshold value and the response timeout time ratio is greater than a preset response timeout time ratio threshold value.
In some specific embodiments, the parsing module 301 includes:
the analysis unit is used for analyzing the log information according to the format of the log information and taking the IP address, the request URL, the request time, the request response time and the response state code obtained by analysis as log data;
and the sending unit is used for sending the log data to the theme of the kafka message middleware.
Since the embodiment of the flow control device portion corresponds to the embodiment of the flow control method portion, please refer to the description of the embodiment of the flow control method portion, and details thereof are not repeated here.
In the following, an electronic device provided by an embodiment of the present application is introduced, and the electronic device described below and the flow control method described above may be referred to correspondingly.
The application also discloses an electronic device, including:
a memory for storing a computer program;
and a processor for implementing the steps of the flow control method when executing the computer program.
Since the embodiment of the electronic device portion corresponds to the embodiment of the flow control method portion, please refer to the description of the embodiment of the flow control method portion for the embodiment of the electronic device portion, which is not repeated here.
In the following, a computer-readable storage medium provided by an embodiment of the present application is described, and the computer-readable storage medium described below and the flow control method described above may be referred to correspondingly.
The application also discloses a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the flow control method as described above.
Since the embodiment of the computer-readable storage medium portion corresponds to the embodiment of the flow control method portion, please refer to the description of the embodiment of the flow control method portion for the embodiment of the computer-readable storage medium portion, which is not repeated here.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The foregoing details a flow control method, an apparatus, an electronic device, and a computer-readable storage medium provided in the present application. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method of flow control, comprising:
analyzing the acquired log information, and sending the log data obtained by analysis to kafka message middleware;
consuming the log data in the kafka message middleware by using a flink, and carrying out index statistics on the log data according to the configuration parameters of the flink;
and if the counted index value is larger than the corresponding preset index threshold value in the configuration parameters, intercepting the request corresponding to the counted index value.
2. The flow control method according to claim 1, before consuming the log data in the kafka message middleware with a flink, further comprising:
and creating the configuration parameters of the flink according to the service demand information, and sending the configuration parameters to the kafka message middleware.
3. The flow control method according to claim 1, wherein performing index statistics on the log data according to the configuration parameters of the flink comprises:
dividing a time window according to a time window parameter in the configuration parameters of the flink;
and calculating the index value of each IP address requesting to access the same URL in the time window.
4. The flow control method according to claim 1, wherein intercepting the request corresponding to the statistically obtained index value if the statistically obtained index value is greater than a preset index threshold corresponding to the configuration parameter comprises:
and if the counted index value of the access target URL is larger than the preset index threshold value corresponding to the target URL in the configuration parameters, forbidding the IP address request accessing the target URL to access the target URL within the target blocking time length according to the target blocking time length corresponding to the target URL in the configuration parameters.
5. The flow control method according to claim 1, wherein intercepting the request corresponding to the statistically obtained index value if the statistically obtained index value is greater than a preset index threshold corresponding to the configuration parameter comprises:
and if the counted index value meets at least one of the condition that the request quantity in the rolling time window is greater than the maximum request quantity, the abnormal state code ratio is greater than a preset abnormal state code ratio threshold value and the response timeout time ratio is greater than a preset response timeout time ratio threshold value, intercepting the request corresponding to the counted index value.
6. The flow control method according to claim 1, wherein sending log data obtained by parsing the log information to kafka message middleware comprises:
analyzing the log information according to the format of the log information, and taking the IP address, the request URL, the request time, the request response time and the response state code obtained by analysis as the log data;
sending the log data to a subject of the kafka message middleware.
7. A flow control device, comprising:
the analysis module is used for analyzing the acquired log information and sending the log data obtained through analysis to the kafka message middleware;
the statistical module is used for consuming the log data in the kafka message middleware by using a flink and carrying out index statistics on the log data according to the configuration parameters of the flink;
and the intercepting module is used for intercepting the request corresponding to the index value obtained by statistics if the index value obtained by statistics is larger than the corresponding preset index threshold value in the configuration parameters.
8. The flow control device of claim 7, further comprising:
and the sending module is used for creating the configuration parameters of the flink according to the service demand information and sending the configuration parameters to the kafka message middleware.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the flow control method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the flow control method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110494968.4A CN113206797A (en) | 2021-05-07 | 2021-05-07 | Flow control method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110494968.4A CN113206797A (en) | 2021-05-07 | 2021-05-07 | Flow control method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113206797A true CN113206797A (en) | 2021-08-03 |
Family
ID=77030204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110494968.4A Pending CN113206797A (en) | 2021-05-07 | 2021-05-07 | Flow control method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113206797A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113660264A (en) * | 2021-08-16 | 2021-11-16 | 杭州安恒信息技术股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113905091A (en) * | 2021-09-15 | 2022-01-07 | 盐城金堤科技有限公司 | Method and device for processing access request |
| CN115396379A (en) * | 2022-08-24 | 2022-11-25 | 北京沃东天骏信息技术有限公司 | Flow control method, device, equipment and medium for service server |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9330051B1 (en) * | 2007-11-27 | 2016-05-03 | Sprint Communications Company L.P. | Collection of web server performance metrics to a centralized database for reporting and analysis |
| CN111163097A (en) * | 2019-12-31 | 2020-05-15 | 新浪网技术(中国)有限公司 | Web application firewall implementation system and method |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN112162903A (en) * | 2020-09-24 | 2021-01-01 | 常州微亿智造科技有限公司 | Method and system for monitoring state of service system based on Flink |
| CN112527601A (en) * | 2020-12-17 | 2021-03-19 | 航天信息股份有限公司 | Monitoring early warning method and device |
| CN112749056A (en) * | 2020-12-30 | 2021-05-04 | 广州品唯软件有限公司 | Application service index monitoring method and device, computer equipment and storage medium |
-
2021
- 2021-05-07 CN CN202110494968.4A patent/CN113206797A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9330051B1 (en) * | 2007-11-27 | 2016-05-03 | Sprint Communications Company L.P. | Collection of web server performance metrics to a centralized database for reporting and analysis |
| CN111163097A (en) * | 2019-12-31 | 2020-05-15 | 新浪网技术(中国)有限公司 | Web application firewall implementation system and method |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN112162903A (en) * | 2020-09-24 | 2021-01-01 | 常州微亿智造科技有限公司 | Method and system for monitoring state of service system based on Flink |
| CN112527601A (en) * | 2020-12-17 | 2021-03-19 | 航天信息股份有限公司 | Monitoring early warning method and device |
| CN112749056A (en) * | 2020-12-30 | 2021-05-04 | 广州品唯软件有限公司 | Application service index monitoring method and device, computer equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113660264A (en) * | 2021-08-16 | 2021-11-16 | 杭州安恒信息技术股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113660264B (en) * | 2021-08-16 | 2023-04-18 | 杭州安恒信息技术股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113905091A (en) * | 2021-09-15 | 2022-01-07 | 盐城金堤科技有限公司 | Method and device for processing access request |
| CN113905091B (en) * | 2021-09-15 | 2023-09-01 | 盐城天眼察微科技有限公司 | Method and device for processing access request |
| CN115396379A (en) * | 2022-08-24 | 2022-11-25 | 北京沃东天骏信息技术有限公司 | Flow control method, device, equipment and medium for service server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107992398B (en) | Monitoring method and monitoring system of service system | |
| CN113206797A (en) | Flow control method and device, electronic equipment and storage medium | |
| CN107370806B (en) | HTTP status code monitoring method, device, storage medium and electronic equipment | |
| CN102769549B (en) | The method and apparatus of network security monitoring | |
| US11093349B2 (en) | System and method for reactive log spooling | |
| CN111176879A (en) | Fault repairing method and device for equipment | |
| CN108900374B (en) | Data processing method and device applied to DPI equipment | |
| CN107204875B (en) | Data reporting link monitoring method and device, electronic equipment and storage medium | |
| CN111866016A (en) | Log analysis method and system | |
| CN112350854B (en) | Flow fault positioning method, device, equipment and storage medium | |
| CN110955586A (en) | System fault prediction method, device and equipment based on log | |
| CN112732405B (en) | JVM thread monitoring method and device and electronic equipment | |
| CN110929896A (en) | Security analysis method and device for system equipment | |
| CN114039900A (en) | Efficient network data packet protocol analysis method and system | |
| CN111324809A (en) | Hotspot information monitoring method, device and system | |
| CN113778810A (en) | Log collection method, device and system | |
| CN113760634A (en) | Data processing method and device | |
| CN113992378B (en) | Security monitoring method and device, electronic equipment and storage medium | |
| CN114238069A (en) | Web application firewall testing method and device, electronic equipment, medium and product | |
| CN114124555A (en) | Message playback method and device, electronic equipment and computer readable medium | |
| CN113254313A (en) | Monitoring index abnormality detection method and device, electronic equipment and storage medium | |
| CN115914064A (en) | Network system service performance evaluation method, device, computing equipment and storage medium | |
| CN113347045A (en) | Alarm message processing method and device | |
| CN111111211A (en) | Method, device, system, equipment and storage medium for reporting game data | |
| CN113920698B (en) | Early warning method, device, equipment and medium for interface abnormal call |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210803 |