CN113127319A - Information monitoring method, related device and computer storage medium - Google Patents
Information monitoring method, related device and computer storage medium Download PDFInfo
- Publication number
- CN113127319A CN113127319A CN202110368742.XA CN202110368742A CN113127319A CN 113127319 A CN113127319 A CN 113127319A CN 202110368742 A CN202110368742 A CN 202110368742A CN 113127319 A CN113127319 A CN 113127319A
- Authority
- CN
- China
- Prior art keywords
- user
- log data
- users
- log
- indexes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000012544 monitoring process Methods 0.000 title claims abstract description 48
- 230000002159 abnormal effect Effects 0.000 claims abstract description 69
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 23
- 230000006399 behavior Effects 0.000 claims description 47
- 230000003068 static effect Effects 0.000 claims description 42
- 238000005070 sampling Methods 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 7
- 238000012795 verification Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 5
- 238000013500 data storage Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012806 monitoring device Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses an information monitoring method, a related device and a computer storage medium, wherein the method comprises the following steps: acquiring log data of a plurality of users in a user node; determining log indexes of a plurality of users based on log data of the users and a preset index algorithm; determining at least one target user and abnormal indexes of at least one target user which do not meet an index threshold value based on log indexes of a plurality of users; and generating early warning information based on the abnormal indexes of the at least one target user. Therefore, the method and the device for processing the log data can directly obtain the log data of each user at the user node, determine the log indexes according to the log data, and further determine the abnormal log indexes and the target users corresponding to the abnormal log indexes in a mode of comparing the log indexes with the preset index threshold value, so that the early warning information is generated quickly, the problem of real-time monitoring at the user side is solved, and the user experience is effectively improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to an information monitoring method, a related apparatus, and a computer medium.
Background
With the development of network technology, users can learn online through programs of a webpage end or a mobile end without going out. However, in the present phase, once a page or a program is abnormal in the process of using the program by a user, the user needs to report the page or the program by himself, and the system cannot monitor the possible existence or occurrence of the program on the user side in real time.
Disclosure of Invention
The embodiment of the application provides an information monitoring method, a related device and a computer storage medium, which aim to solve the problem of real-time monitoring at a user side.
In a first aspect, an embodiment of the present application provides an information monitoring method, where the method includes:
acquiring log data of a plurality of users in user nodes;
determining log indexes of the users based on log data of the users and a preset index algorithm;
determining at least one target user and an abnormal index of the at least one target user which do not meet an index threshold value based on the log indexes of the plurality of users;
and generating early warning information based on the abnormal indexes of the at least one target user.
In a second aspect, an embodiment of the present application provides an information monitoring apparatus, including:
the acquisition module is used for acquiring log data of a plurality of users in the user nodes;
the first determination module is used for determining log indexes of the users based on log data of the users and a preset index algorithm;
a second determination module for determining at least one target user not meeting a metric threshold and an anomaly metric of the at least one target user based on log metrics of the plurality of users;
and the generating module is used for generating early warning information based on the abnormal indexes of the at least one target user.
In a third aspect, embodiments of the present application provide a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the above-mentioned method steps.
In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory;
wherein the memory stores a computer program adapted to be loaded by the processor and to perform the above-mentioned method steps.
The beneficial effects brought by the technical scheme provided by some embodiments of the application at least comprise:
in the embodiment of the application, log data of a plurality of users in the user node can be acquired; determining log indexes of a plurality of users based on log data of the users and a preset index algorithm; determining at least one target user and abnormal indexes of at least one target user which do not meet an index threshold value based on log indexes of a plurality of users; and generating early warning information based on the abnormal indexes of the at least one target user. Therefore, according to the embodiment of the application, the abnormal log indexes and the target users corresponding to the abnormal log indexes can be determined in a mode of directly obtaining the log data of each user at the user node, determining the log indexes according to the log data and further comparing the log indexes with the preset index threshold value, and therefore the early warning information is generated quickly, the problem of real-time monitoring at the user side is solved, and the user experience is effectively improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is an application scenario diagram of an information monitoring method according to an embodiment of the present application;
fig. 2 is a system architecture diagram of an information monitoring method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of an information monitoring method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another information monitoring method according to an embodiment of the present application;
fig. 5 is an application scenario diagram of another information monitoring method provided in the embodiment of the present application;
fig. 6 is a schematic structural diagram of an information monitoring apparatus according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.
In the description of the present application, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art. Further, in the description of the present application, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
Fig. 1 exemplarily shows an application scenario diagram of an information monitoring method provided in an embodiment of the present application. The information monitoring method provided by the embodiment of the application can monitor various functions of the product by acquiring behavior data generated by each user terminal in the user side. Wherein:
the products referred to in the embodiments of the present application may be applied to include, but are not limited to: browser page, WeChat applet page H5, mobile terminal program APP, personal computer terminal program pcapp. When a user triggers a certain function of a product in the user side, the behavior data of the user can be generated. For example, when a user inputs a website of a login page of a product in a browser, a user side may request interactive data from a link layer and a back-end server to render the login page, and if the user side receives or does not receive data returned by the link layer in a delayed manner, problems such as jamming, white screen, breakdown and the like may occur in a current page, so that the user cannot enter a normal login page, and corresponding behavior data of the user is also abnormal.
Possibly, the embodiment of the application may acquire the behavior data of the user in the following manner: behavior data of a user in an SDK (Software Development Kit) at a web page end can be obtained by intercepting a user request in a manner of adding JavaScript codes in a Header (Header) of a web page. The behavior data of the user in the mobile terminal SDK can be obtained by a mode of intercepting the user request by providing a calling method of a java or c language class library. The behavior data of the user in the personal computer side SDK can be obtained by a mode of intercepting the user request by providing a calling method of a class library of an Objective-C language.
The data types of the behavior data in the web page side SDK may include, but are not limited to: page loading, page exception, resource loading, interface request and service log. The data types of the behavior data in the mobile terminal SDK may include, but are not limited to: network monitoring, static resource loading, performance monitoring, service monitoring and service exception. The data types of behavior data in the personal computer side SDK may include, but are not limited to: starting page loading, finishing page loading, making page loading error, detecting network PING (Packet Internet Groper), detecting network CURL (Customized Uniform Resource Locator), and logging service.
Fig. 2 is a system architecture diagram of an information monitoring method applied to an embodiment of the present application. The information monitoring method provided by the embodiment of the application can be applied to a server. Specifically, the server may be connected to the terminal through a network. The network is used to provide a communication link between the terminal and the server. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. Terminals include, but are not limited to: wearable devices, monitoring devices, handheld devices, personal computers, tablet computers, in-vehicle devices, smart phones, computing devices or other processing devices connected to a wireless modem, and the like. The terminal devices in different networks may be called different names, for example: a monitoring device, a user equipment, an access terminal, a subscriber unit, a subscriber station, a mobile station, a remote terminal, a mobile device, a user terminal, a wireless communication device, a user agent or user equipment, a cellular telephone, a cordless telephone, a Personal Digital Assistant (PDA), a terminal device in a 5th generation mobile network or a future evolution network, etc. The terminal system is an operating system that can run on the terminal, is a program for managing and controlling terminal hardware and terminal applications, and is an indispensable system application of the terminal. The system comprises but is not limited to Android system, IOS system, Windows Phone (WP) system, Ubuntu mobile version operating system and the like.
It should be understood that the number of terminals, networks, and servers in fig. 2 is merely illustrative. There may be any number of terminals, networks and servers, as desired for the reality. For example, the server may be a server cluster composed of a plurality of servers. The user can use the server to interact with the terminal through the network so as to obtain the optimized version and the like.
Next, an application scenario diagram of the information monitoring method described in fig. 1 and a system architecture diagram described in fig. 2 are combined to describe the information monitoring method provided in the embodiment of the present application.
In one embodiment, as shown in fig. 3, a flow diagram of an information monitoring method is provided. As shown in fig. 3, the information monitoring method may include the steps of:
s301, log data of a plurality of users in the user node are obtained.
The user nodes can comprise domestic user nodes and overseas user nodes, wherein the domestic user nodes are used for representing user behavior data of various domestic areas acquired through a domestic server, and the overseas user nodes are used for representing user behavior data of various overseas areas acquired through an overseas server. The log data of the user is used for representing the behavior data obtained through processing.
Since hundreds of millions of data are transmitted back to domestic data centers through transoceanic links every day, if the data are transmitted in full, the problems of large network traffic cost, data loss in the transmission process and the like are faced. In order to solve the problem, the embodiment of the application can be respectively provided with a domestic user node and a overseas user node, wherein the domestic user node is responsible for domestic data collection and storage, and the overseas user node is responsible for overseas data collection and storage.
Possibly, the embodiment of the application can sample the log data of each user in the overseas user nodes based on a hierarchical sampling algorithm to obtain the log data of at least one user in the overseas user nodes. The log data of at least one user is the log data of a part of users in the log data of all users in the overseas user node. In addition, the overseas data center can be further inquired when the detailed log data of the overseas user is needed, so that the efficiency and the cost of the embodiment of the application are greatly optimized.
Possibly, in the embodiment of the application, the first message middleware is used for receiving the behavior data of each user in the domestic user node, and the second message middleware is used for receiving the behavior data of each user in the overseas user node. Further, after the behavior data of each user in the domestic user node and the behavior data of each user in the overseas user node are respectively logically processed, log data of each user in the domestic user node and log data of each user in the overseas user node are obtained. Wherein the logical processing may include any one or more of: field resolution, idempotent verification, user address resolution, user identification mapping and application information association.
The field analysis is used for analyzing the behavior data of the user reported by the user side into a field of the server, the idempotent verification is used for detecting whether the reported behavior data are repeated, and if adjacent data in the behavior data are repeated, filtering processing is needed. The user address resolution is used for determining user address identification according to the behavior data, and the user identification mapping is used for tracking the identification of the user in each level to determine the position of the existing problem of the behavior data of the user. The application information association is used to determine which specific services in the product the user has accessed.
S302, determining log indexes of a plurality of users based on log data of the users and a preset index algorithm.
The preset index algorithm may include any one or more of the following items: an interface request success rate index algorithm, a page performance index algorithm, a region monitoring index algorithm, a ring ratio index algorithm and a same ratio index algorithm.
It can be understood that the interface request success rate indicator algorithm is used to calculate the success rate of the user side requesting the backend interface from the server. The page performance index algorithm is used for quantifying the page performance according to a plurality of data such as page loading time, page stuck time, resource loading, visual stability and the like. The area monitoring index algorithm is used for calculating a certain index corresponding to the behavior data of all the users in each area, for example, the success rate of all the users in the area a accessing the operator network. The ring ratio index algorithm is used for comparing log data generated by triggering a certain function of a product by a user in two continuous statistical periods. The geometric index algorithm is used for comparing log data generated by triggering a certain function of a product by a user in the nth month of the year and the nth month of the last year.
Possibly, according to the embodiment of the application, corresponding log indexes can be generated directly according to log data such as page rendering time, resource loading time and the like. For example, rendering time index 2S for page a, resource loading time 1S for page a.
S303, determining at least one target user and abnormal indexes of the at least one target user which do not meet the index threshold value based on the log indexes of the plurality of users.
The index threshold is used for representing a preset log index in a normal range.
For example, the page performance index 80 generated when the domestic user a opens the page 1, and the ring ratio index 1.1 when the domestic user a opens the "digital library" to read the book 1; a page performance index 85 generated when the domestic user B opens the page 1, and a ring ratio index 0.9 when the domestic user B opens the digital library to read the book 1; the page performance index 90 of the domestic user C, and the ring ratio index 0.95 when the domestic user C opens the 'digital library' of the product 1 to read the book 1; the page performance index 70 generated when the overseas user D opens the page 1, the ring ratio index 1.2 when the overseas user D opens the "digital library" to read the book 1, the page performance index threshold value is 75-100, and the ring ratio index threshold value when the overseas user D opens the "digital library" to read the book 1 is 0.8-1.2, it can be determined that the target user that does not satisfy the page performance index threshold value is the overseas user D.
S304, generating early warning information based on the abnormal indexes of at least one target user.
The early warning information is used to represent prompt information for prompting that the log index of the user end is abnormal, for example, in the case that the page performance index of the target user D is abnormal, the generated early warning information may include: the method comprises the following steps of page performance index abnormal time, identification information of a target user D, a region where the target user is located, abnormal page identification information, abnormal field position information, abnormal page correlation information and the like.
In a specific example, the behavior data of a large number of domestic users can be acquired through kafka middleware of a domestic user node arranged in an access layer, and the behavior data of a large number of overseas users can be acquired through kafka middleware arranged in an overseas user node. Further, the logic layer may perform logic processing such as field analysis on the behavior data of the domestic user by using an ETL (Extract-Transform-Load, data warehouse technology) to obtain log data of the domestic user, and perform logic processing such as field analysis after sampling the behavior data of the overseas user to obtain log data of the overseas user. Further, the log processing is performed on the acquired domestic user log data and the extracted overseas user log data to obtain various log indexes (such as a ring ratio index of page loading time). And determining a target user with abnormal log data in domestic users and overseas users according to various log indexes and corresponding preset index thresholds, and generating and displaying corresponding early warning information according to the abnormal indexes of the target user so as to inform technicians to process the abnormal indexes of the target user.
In the embodiment of the application, log data of a plurality of users in the user node can be acquired; determining log indexes of a plurality of users based on log data of the users and a preset index algorithm; determining at least one target user and abnormal indexes of at least one target user which do not meet an index threshold value based on log indexes of a plurality of users; and generating early warning information based on the abnormal indexes of the at least one target user. Therefore, according to the embodiment of the application, the abnormal log indexes and the target users corresponding to the abnormal log indexes can be determined in a mode of directly obtaining the log data of each user at the user node, determining the log indexes according to the log data and further comparing the log indexes with the preset index threshold value, and therefore the early warning information is generated quickly, the problem of real-time monitoring at the user side is solved, and the user experience is effectively improved.
In some embodiments, fig. 4 illustrates a flowchart of an information monitoring method provided in an embodiment of the present application. As shown in fig. 4, the information monitoring method may include at least the following steps:
s401, obtaining static log data and/or dynamic log data of a plurality of users in user nodes.
The log data in the embodiment of the present application may include: static log data and/or dynamic log data. The static log data is used for representing log data generated by resources which need to be stored in advance when a user requests a webpage end or an application program, such as picture resources, Powerpoint resources, text resources and audio resources. The dynamic log data is used to represent log data generated by a user requesting a web page or an Interface in an Application program, such as an API (Application Programming Interface) request.
S402, determining the static indexes and/or the dynamic indexes of the multiple users based on the static log data and/or the dynamic log data of the multiple users and a preset static index algorithm and/or a preset dynamic index algorithm.
The preset static index algorithm may include a page performance index algorithm, and the page performance index may be calculated as: the comprehensive indexes of the picture index, the Powerpoint index, the character index and the vocal index. The dynamic index algorithm may include an interface index algorithm. For example, when the user 1 requests a plurality of API interfaces in the target page, the feedback time of the data when each API interface is requested may be determined first, and then the interface index of the target page and the average feedback time of the data of the plurality of API interfaces may be further calculated.
S403, determining static abnormal indexes and/or dynamic abnormal indexes of at least one target user and at least one target user which do not meet the static index threshold value and/or the dynamic index threshold value based on the static indexes and/or the dynamic indexes of the plurality of users.
The static index threshold is used for representing a numerical range of a preset static index under a normal condition. The dynamic index threshold is used for representing a numerical range of a preset dynamic index under a normal condition.
For example, the page performance index 80 generated when the domestic user a opens the page 1, and the interface index generated when the domestic user a clicks the book 1 in the "digital library" for 2 seconds; page performance index 85 generated when the domestic user B opens the page 1, and interface index 2.5 seconds generated when the domestic user B clicks the book 1 in the digital library; the page performance index 90 of the domestic user C and the interface index generated when the domestic user C clicks the book 1 in the digital library are 5 seconds; the page performance index 70 generated when the overseas user D opens the page 1, the interface index generated when the overseas user D clicks the book 1 in the "digital library" for 7 seconds, the page performance index threshold value is 75-100, and the interface index threshold value of the book 1 in the "digital library" is 0.1-3 seconds, so that it can be determined that the target user who does not satisfy the page performance index threshold value is the overseas user D, and the target users who do not satisfy the interface index threshold value are the domestic user C and the overseas user D.
S404, generating early warning information based on the static abnormal index and/or the dynamic abnormal index of at least one target user.
The early warning information is used for representing prompt information for prompting the static abnormal index and/or the dynamic abnormal index of the user side.
Continuing with the above example, in the case that the page performance index of the overseas user D is abnormal, the generated early warning information may include: the page performance index abnormal time, the identification information of the overseas user D, the specific position of the overseas user D, the identification information of the abnormal page, the position information of the abnormal field, the correlation information of the abnormal page and the like. Under the condition that the interface indexes of the domestic user C and the overseas user D are abnormal, the generated early warning information can comprise: the abnormal time of the interface index of the domestic user C, the identification information of the domestic user C, the specific position of the domestic user C, the abnormal interface identification corresponding to the domestic user C, the abnormal time of the interface index of the overseas user D, the identification information of the overseas user D, the specific position of the overseas user D and the abnormal interface identification corresponding to the domestic user D.
In some embodiments, the embodiment of the present application may further determine whether a failure service exists in log data of each user based on a preset buried point service and log data of a plurality of users; and determining the user with the failed service as a target user. The service embedding point is from the perspective of the service requirement of the product, for example, the product needs to count exposure and click of a certain page, and the click rate of a recommendation item needed by an algorithm person.
Specifically, the embedded point is an important way for collecting user behavior data, and is mainly used for collecting and recording interaction data between a user operation behavior at a specific position and a server, and the basic principle is to deploy collected SDK codes at terminals such as App/H5/PC. For example, when the user's behavior satisfies a certain condition, such as entering a certain page, clicking a certain button, etc., recording and storage are automatically triggered, and then the data is collected and transmitted to the terminal provider, or the requested data in the process of using the service by the user is collected through the back end.
In some embodiments, at least one piece of key index data of each user can be acquired based on log indexes of a plurality of users and a preset extraction rule; and generating and displaying an index large disk corresponding to at least one plurality of key index data based on at least one key index data of each user.
In some embodiments, the embodiment of the present application may further store log data of each user in the domestic user node in the first storage; and storing the log data of each user in the overseas user node into a second memory.
Referring to fig. 5, in a specific example, after analyzing DNS (Domain Name System) of products in mobile terminals or personal computers of users in domestic user nodes and westernish (overseas) user nodes, corresponding user behavior data is generated in SDK according to user operation behaviors. Further, through a Content Delivery Network (CDN) for video and audio on demand, live streaming media, total-station acceleration and secure acceleration in a link layer, an SLB (Server Load Balancing) for realizing Load Balancing among a plurality of servers, an Nginx for preventing a malicious attack on an intranet Server by an extranet, caching to reduce the pressure of the Server and access security control, and distributing a user request to a plurality of servers, behavior data of a user is respectively input to a Kafka middleware in the access layer and a Kafka middleware in meixi to transmit all behavior data of the user in a web page end and an application program end to a logic layer ETL in real time for field analysis, power equality verification, IP library processing (user address analysis), traceld mapping (user identification mapping), application information association and other logic processing to obtain behavior data of the domestic user and log data of the meixi user, and the log data are respectively stored in a domestic cloud storage Network The memory and the American cloud are stored, and then the log data of partial American users can be extracted to a log storage platform SLS of a domestic server by utilizing a hierarchical sampling algorithm. Furthermore, index extraction, log clustering and geometric comparison processing operations can be performed on a large number of domestic users and partial Meixi users by using Flink flow calculation in a log processing layer to obtain various index data of the users, technicians can observe various index data of each user through a Huatuo console in a display layer, comprehensive index data of all users in a certain area can also be observed through an index large disk, abnormal index data can be early warned through a service monitoring system, and when the system sends out early warning information, technicians can search for the early warning information of a target user in a monitoring terminal to quickly solve the problems existing at a user terminal.
Fig. 6 is a schematic structural diagram of an information monitoring apparatus according to an exemplary embodiment of the present application. The information monitoring apparatus may be disposed in a device such as a server, and execute the information monitoring method according to any of the embodiments described above. As shown in fig. 6, the information monitoring apparatus may include:
an obtaining module 61, configured to obtain log data of multiple users in a user node;
a first determining module 62, configured to determine log indexes of the multiple users based on log data of the multiple users and a preset index algorithm;
a second determining module 63, configured to determine, based on the log indicators of the plurality of users, at least one target user who does not meet an indicator threshold and an anomaly indicator of the at least one target user;
a generating module 64, configured to generate early warning information based on the abnormality indicator of the at least one target user.
In the embodiment of the application, log data of a plurality of users in the user node can be acquired; determining log indexes of a plurality of users based on log data of the users and a preset index algorithm; determining at least one target user and abnormal indexes of at least one target user which do not meet an index threshold value based on log indexes of a plurality of users; and generating early warning information based on the abnormal indexes of the at least one target user. Therefore, the method and the device for processing the log data can directly obtain the log data of each user at the user node, determine the log indexes according to the log data, and further determine the abnormal log indexes and the target users corresponding to the abnormal log indexes in a mode of comparing the log indexes with the preset index threshold value, so that the early warning information is generated quickly, the problem of real-time monitoring at the user side is solved, and the user experience is effectively improved.
In some embodiments, the user nodes include domestic user nodes and overseas user nodes;
the obtaining module 61 includes:
the receiving unit is used for receiving the behavior data of each user in the domestic user node by using a first message middleware and receiving the behavior data of each user in the overseas user node by using a second message middleware;
an obtaining unit, configured to obtain log data of each user in the domestic user node and log data of each user in the overseas user node after performing logic processing on behavior data of each user in the domestic user node and behavior data of each user in the overseas user node, respectively;
wherein the logical processing includes any one or more of: field resolution, idempotent verification, user address resolution, user identification mapping and application information association.
In some embodiments, after the obtaining unit, the method further includes:
the sampling unit is used for sampling the log data of each user in the overseas user nodes based on a hierarchical sampling algorithm to obtain the log data of at least one user in the overseas user nodes;
the log data of at least one user is the log data of a part of users in the log data of all users in the overseas user node.
In some embodiments, the log data comprises: static log data and/or dynamic log data;
the obtaining module 61 is specifically configured to: acquiring static log data and/or dynamic log data of a plurality of users in the user node;
the first determining module 62 is specifically configured to: determining static indexes and/or dynamic indexes of the multiple users based on the static log data and/or the dynamic log data of the multiple users and a preset static index algorithm and/or a preset dynamic index algorithm;
the second determining module 63 is specifically configured to: determining at least one target user which does not meet a static index threshold value and/or a dynamic index threshold value and static abnormal indexes and/or dynamic abnormal indexes of the at least one target user based on the static indexes and/or the dynamic indexes of the plurality of users;
the generating module 64 is specifically configured to: and generating early warning information based on the static abnormal index and/or the dynamic abnormal index of the at least one target user.
In some embodiments, before the generating module 64, the method further includes:
a third determining module, configured to determine whether a failure service exists in log data of each user based on a preset buried point service and the log data of the multiple users;
and the fourth determining module is used for determining the user with the failed service as the target user.
In some embodiments, the apparatus further comprises:
a plurality of key index data obtaining modules 61, configured to obtain a plurality of key index data based on the log indexes of the users and a preset extraction rule;
and the display module is used for generating and displaying the index large disks corresponding to the plurality of key index data.
In some embodiments, before the obtaining module 61, the apparatus further includes:
the storage module is used for storing the log data of each user in the domestic user node into a first storage; and storing the log data of each user in the overseas user node into a second memory.
It should be noted that, when the information monitoring apparatus provided in the foregoing embodiment executes the information monitoring method, only the division of the functional modules is illustrated, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the information monitoring apparatus and the information monitoring method provided by the above embodiments belong to the same concept, and details of implementation processes thereof are referred to as method embodiments, which are not described herein again.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 7, the electronic device 70 may include: at least one processor 71, at least one network interface 74, a user interface 73, a memory 75, at least one communication bus 72.
Wherein a communication bus 72 is used to enable the connection communication between these components.
The user interface 73 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 73 may also include a standard wired interface and a wireless interface.
The network interface 74 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.
The Memory 75 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 75 includes a non-transitory computer-readable medium. The memory 75 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 75 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like; the storage data area may store data and the like referred to in the above respective method embodiments. The memory 75 may alternatively be at least one memory device located remotely from the processor 71. As shown in fig. 7, the memory 75, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and an information monitoring application program.
In the electronic device 70 shown in fig. 7, the user interface 73 is mainly used as an interface for providing input for a user, and acquiring data input by the user; and processor 71 may be configured to invoke the information monitoring application stored in memory 75 and perform the following operations in particular:
acquiring log data of a plurality of users in user nodes;
determining log indexes of the users based on log data of the users and a preset index algorithm;
determining at least one target user and an abnormal index of the at least one target user which do not meet an index threshold value based on the log indexes of the plurality of users;
and generating early warning information based on the abnormal indexes of the at least one target user.
In some embodiments, the user nodes include domestic user nodes and overseas user nodes;
when executing acquiring log data of a plurality of users in a user node, the processor 71 specifically executes:
receiving behavior data of each user in the domestic user node by using a first message middleware, and receiving behavior data of each user in the overseas user node by using a second message middleware;
after the behavior data of each user in the domestic user node and the behavior data of each user in the overseas user node are respectively subjected to logic processing, log data of each user in the domestic user node and log data of each user in the overseas user node are obtained;
wherein the logical processing includes any one or more of: field resolution, idempotent verification, user address resolution, user identification mapping and application information association.
In some embodiments, the processor 71 further performs, after obtaining the log data of each user in the domestic user node and the log data of each user in the overseas user node:
sampling the log data of each user in the overseas user nodes based on a hierarchical sampling algorithm to obtain the log data of at least one user in the overseas user nodes;
the log data of at least one user is the log data of a part of users in the log data of all users in the overseas user node.
In some embodiments, the log data comprises: static log data and/or dynamic log data;
when the processor 71 executes the acquiring of the log data of the plurality of users in the user node, specifically: acquiring static log data and/or dynamic log data of a plurality of users in the user node;
when the processor 71 executes the log data based on the multiple users and the preset index algorithm to determine the log indexes of the multiple users, the following specific steps are executed: determining static indexes and/or dynamic indexes of the multiple users based on the static log data and/or the dynamic log data of the multiple users and a preset static index algorithm and/or a preset dynamic index algorithm;
when the processor 71 determines at least one target user not meeting the metric threshold and the abnormal metric of the at least one target user based on the log metrics of the plurality of users, specifically: determining at least one target user which does not meet a static index threshold value and/or a dynamic index threshold value and static abnormal indexes and/or dynamic abnormal indexes of the at least one target user based on the static indexes and/or the dynamic indexes of the plurality of users;
when executing the generation of the warning information based on the abnormal indicator of the at least one target user, the processor 71 specifically executes: and generating early warning information based on the static abnormal index and/or the dynamic abnormal index of the at least one target user.
In some embodiments, the processor 71 further performs, before performing the generating of the warning information based on the abnormality indicator of the at least one target user:
determining whether a failure service exists in the log data of each user based on a preset buried point service and the log data of the plurality of users;
and determining the user with the failed service as a target user.
In some embodiments, the processor 71 further performs:
acquiring a plurality of key index data based on the log indexes of the users and a preset extraction rule;
and generating and displaying the index large disks corresponding to the plurality of key index data.
In some embodiments, the processor 71 further performs, before performing the acquiring log data of a plurality of users in the user node:
storing log data of each user in the domestic user node into a first memory; and storing the log data of each user in the overseas user node into a second memory.
Embodiments of the present application also provide a computer-readable storage medium, which stores instructions that, when executed on a computer or a processor, cause the computer or the processor to perform one or more of the steps in the embodiments shown in fig. 3 to 4. The respective constituent modules of the information monitoring apparatus may be stored in the computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)), or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., Digital Versatile Disk (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. And the aforementioned storage medium includes: various media capable of storing program codes, such as a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk. The technical features in the present examples and embodiments may be arbitrarily combined without conflict.
The above-described embodiments are merely preferred embodiments of the present application, and are not intended to limit the scope of the present application, and various modifications and improvements made to the technical solutions of the present application by those skilled in the art without departing from the design spirit of the present application should fall within the protection scope defined by the claims of the present application.
Claims (10)
1. An information monitoring method, characterized in that the method comprises:
acquiring log data of a plurality of users in user nodes;
determining log indexes of the users based on log data of the users and a preset index algorithm;
determining at least one target user and an abnormal index of the at least one target user which do not meet an index threshold value based on the log indexes of the plurality of users;
and generating early warning information based on the abnormal indexes of the at least one target user.
2. The method of claim 1, wherein the user nodes comprise domestic user nodes and overseas user nodes;
the acquiring log data of a plurality of users in the user node includes:
receiving behavior data of each user in the domestic user node by using a first message middleware, and receiving behavior data of each user in the overseas user node by using a second message middleware;
after the behavior data of each user in the domestic user node and the behavior data of each user in the overseas user node are respectively subjected to logic processing, log data of each user in the domestic user node and log data of each user in the overseas user node are obtained;
wherein the logical processing includes any one or more of: field resolution, idempotent verification, user address resolution, user identification mapping and application information association.
3. The method of claim 2, wherein after obtaining the log data of each user in the domestic user node and the log data of each user in the overseas user node, further comprising:
sampling the log data of each user in the overseas user nodes based on a hierarchical sampling algorithm to obtain the log data of at least one user in the overseas user nodes;
the log data of at least one user is the log data of a part of users in the log data of all users in the overseas user node.
4. The method of claim 2, wherein the log data comprises: static log data and/or dynamic log data;
the acquiring log data of a plurality of users in the user node comprises: acquiring static log data and/or dynamic log data of a plurality of users in the user node;
the determining the log indexes of the plurality of users based on the log data of the plurality of users and a preset index algorithm comprises: determining static indexes and/or dynamic indexes of the multiple users based on the static log data and/or the dynamic log data of the multiple users and a preset static index algorithm and/or a preset dynamic index algorithm;
the determining, based on the log metrics of the plurality of users, at least one target user that does not meet a metric threshold and an anomaly metric of the at least one target user comprises: determining at least one target user which does not meet a static index threshold value and/or a dynamic index threshold value and static abnormal indexes and/or dynamic abnormal indexes of the at least one target user based on the static indexes and/or the dynamic indexes of the plurality of users;
the generating of the early warning information based on the abnormal index of the at least one target user comprises: and generating early warning information based on the static abnormal index and/or the dynamic abnormal index of the at least one target user.
5. The method of any one of claims 1-4, wherein prior to generating early warning information based on the anomaly indicators of the at least one target user, further comprising:
determining whether a failure service exists in the log data of each user based on a preset buried point service and the log data of the plurality of users;
and determining the user with the failed service as a target user.
6. The method of claim 1, wherein the method further comprises:
acquiring a plurality of key index data based on the log indexes of the users and a preset extraction rule;
and generating and displaying the index large disks corresponding to the plurality of key index data.
7. The method of claim 2, wherein prior to obtaining log data for a plurality of users in a user node, the method further comprises:
storing log data of each user in the domestic user node into a first memory; and storing the log data of each user in the overseas user node into a second memory.
8. An information monitoring apparatus, the apparatus comprising:
the acquisition module is used for acquiring log data of a plurality of users in the user nodes;
the first determination module is used for determining log indexes of the users based on log data of the users and a preset index algorithm;
a second determination module for determining at least one target user not meeting a metric threshold and an anomaly metric of the at least one target user based on log metrics of the plurality of users;
and the generating module is used for generating early warning information based on the abnormal indexes of the at least one target user.
9. A computer storage medium, characterized in that it stores a plurality of instructions adapted to be loaded by a processor and to perform the method steps according to any of claims 1-7.
10. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110368742.XA CN113127319A (en) | 2021-04-06 | 2021-04-06 | Information monitoring method, related device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110368742.XA CN113127319A (en) | 2021-04-06 | 2021-04-06 | Information monitoring method, related device and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127319A true CN113127319A (en) | 2021-07-16 |
Family
ID=76775190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110368742.XA Pending CN113127319A (en) | 2021-04-06 | 2021-04-06 | Information monitoring method, related device and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127319A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014153723A (en) * | 2013-02-04 | 2014-08-25 | Nippon Telegr & Teleph Corp <Ntt> | Log origination abnormality detection device and method |
| CN108491310A (en) * | 2018-03-26 | 2018-09-04 | 北京九章云极科技有限公司 | A kind of daily record monitoring method and system |
| CN110083575A (en) * | 2019-04-11 | 2019-08-02 | ***通信集团内蒙古有限公司 | Fulfilling monitoring method, device, equipment and computer readable storage medium |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN111756579A (en) * | 2020-06-24 | 2020-10-09 | 北京百度网讯科技有限公司 | Abnormity early warning method, device, equipment and storage medium |
| CN112054915A (en) * | 2019-06-06 | 2020-12-08 | 阿里巴巴集团控股有限公司 | Processing method, device and system for client abnormity early warning and computing equipment |
| CN112491602A (en) * | 2020-11-17 | 2021-03-12 | 中国平安财产保险股份有限公司 | Behavior data monitoring method and device, computer equipment and medium |
-
2021
- 2021-04-06 CN CN202110368742.XA patent/CN113127319A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2014153723A (en) * | 2013-02-04 | 2014-08-25 | Nippon Telegr & Teleph Corp <Ntt> | Log origination abnormality detection device and method |
| CN108491310A (en) * | 2018-03-26 | 2018-09-04 | 北京九章云极科技有限公司 | A kind of daily record monitoring method and system |
| CN110083575A (en) * | 2019-04-11 | 2019-08-02 | ***通信集团内蒙古有限公司 | Fulfilling monitoring method, device, equipment and computer readable storage medium |
| CN112054915A (en) * | 2019-06-06 | 2020-12-08 | 阿里巴巴集团控股有限公司 | Processing method, device and system for client abnormity early warning and computing equipment |
| CN110442503A (en) * | 2019-07-29 | 2019-11-12 | 深圳数位传媒科技有限公司 | A kind of alarm method and device using log index |
| CN111756579A (en) * | 2020-06-24 | 2020-10-09 | 北京百度网讯科技有限公司 | Abnormity early warning method, device, equipment and storage medium |
| CN112491602A (en) * | 2020-11-17 | 2021-03-12 | 中国平安财产保险股份有限公司 | Behavior data monitoring method and device, computer equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220229525A1 (en) | Dynamic user interface customization | |
| CN106503134B (en) | Browser jumps to the method for data synchronization and device of application program | |
| KR102504075B1 (en) | Matching and attributes of user device events | |
| CN109618176B (en) | Processing method, equipment and storage medium for live broadcast service | |
| CN110798380A (en) | Data acquisition method, system, device, test equipment and test server | |
| CN112506755B (en) | Log acquisition method, device, computer equipment and storage medium | |
| CN112953791B (en) | Network detection method and device, electronic equipment and computer readable storage medium | |
| US20180285242A1 (en) | Automated system for fixing and debugging software deployed to customers | |
| CN109828920A (en) | A kind of log analysis method, device and computer readable storage medium | |
| US10140377B2 (en) | Data processing, data collection | |
| CN111611140A (en) | Reporting verification method and device of buried point data, electronic equipment and storage medium | |
| US20150271026A1 (en) | End user performance analysis | |
| CN107766224B (en) | Test method and test device | |
| CN115705190A (en) | Method and device for determining dependence degree | |
| CN108062401A (en) | Using recommendation method, apparatus and storage medium | |
| JP6199844B2 (en) | Suspicious part estimation device and suspected part estimation method | |
| CN108345508A (en) | Interface calls test method and device | |
| CN113127319A (en) | Information monitoring method, related device and computer storage medium | |
| CN109698774A (en) | Method and device for monitoring device working condition | |
| CN112135199B (en) | Video playing method based on multiple types of video sources and related equipment | |
| CN112559278B (en) | Method and device for acquiring operation data | |
| KR102027759B1 (en) | Network-related new device registration method and apparatus | |
| CN111600769A (en) | Site detection method and device and storage medium | |
| CN111767447A (en) | Method and device for determining user traffic path | |
| CN109756393A (en) | Information processing method, system, medium and calculating equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |