CN117692665A - Live broadcast monitoring processing method, system, equipment and medium - Google Patents
Live broadcast monitoring processing method, system, equipment and medium Download PDFInfo
- Publication number
- CN117692665A CN117692665A CN202311689592.8A CN202311689592A CN117692665A CN 117692665 A CN117692665 A CN 117692665A CN 202311689592 A CN202311689592 A CN 202311689592A CN 117692665 A CN117692665 A CN 117692665A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- monitoring
- language model
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 185
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 153
- 238000012549 training Methods 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 52
- 238000001514 detection method Methods 0.000 claims abstract description 30
- 230000008569 process Effects 0.000 claims abstract description 18
- 238000004458 analytical method Methods 0.000 claims description 29
- 238000003860 storage Methods 0.000 claims description 21
- 238000004220 aggregation Methods 0.000 claims description 19
- 230000002776 aggregation Effects 0.000 claims description 19
- 238000013480 data collection Methods 0.000 claims description 16
- 230000005856 abnormality Effects 0.000 claims description 10
- 230000008447 perception Effects 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 7
- 230000008439 repair process Effects 0.000 claims description 7
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 abstract description 13
- 238000012423 maintenance Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 9
- 230000006855 networking Effects 0.000 description 8
- 238000013473 artificial intelligence Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000003058 natural language processing Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 101001072091 Homo sapiens ProSAAS Proteins 0.000 description 1
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 101150096185 PAAS gene Proteins 0.000 description 1
- 102100036366 ProSAAS Human genes 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000009411 base construction Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000004080 punching Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/232—Content retrieval operation locally within server, e.g. reading video streams from disk arrays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2404—Monitoring of server processing errors or hardware failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2407—Monitoring of transmitted content, e.g. distribution time, number of downloads
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a live broadcast monitoring processing method, a system, equipment and a medium, wherein the method comprises the steps of collecting and converging data of a live broadcast whole process to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data; performing model training processing by taking the monitoring source data as a training data set to obtain a large language model; performing anomaly detection processing on the monitoring source data through a large language model to obtain a fault result; and combining the large language model with an internal search engine to search and analyze the fault result in real time to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution. According to the embodiment of the invention, the large voice model and the internal search engine are applied to the real-time monitoring system, so that the data are rapidly processed and inquired, the abnormal data can be timely found and processed, and the method and the device can be widely applied to the technical field of operation and maintenance monitoring.
Description
Technical Field
The invention relates to the technical field of operation and maintenance monitoring, in particular to a live broadcast monitoring processing method, a live broadcast monitoring system, live broadcast monitoring equipment and a live broadcast monitoring medium.
Background
With the continuous development of live broadcast services of the video network, the complexity and the scale of the system are continuously increased, and the abnormal problem processing flow of the system becomes a main factor for restricting the stable operation of the system. Various abnormal conditions, such as server faults, user experience problems, abnormal platform operation and the like, may occur in the live broadcast process, and negative effects are brought to user experience and service quality. In view of the foregoing, there is a need for solving the technical problems in the related art.
Disclosure of Invention
In view of this, the embodiments of the present invention provide a live broadcast monitoring processing method, system, device, and medium, so as to improve the detection and processing efficiency of abnormal situations.
In one aspect, the present invention provides a live broadcast monitoring processing method, where the method includes:
data collection and aggregation processing are carried out on the live broadcast whole flow to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data;
performing model training processing by taking the monitoring source data as a training data set to obtain a large language model;
performing anomaly detection processing on the monitoring source data through the large language model to obtain a fault result;
and combining the large language model with an internal search engine, and carrying out real-time search and analysis processing on the fault result to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
Optionally, the method further comprises:
recording and archiving the alarm processing scheme, and establishing a history database and a knowledge sharing library;
updating the training data set according to the historical database and the knowledge sharing base;
and inputting the updated training data set into the large language model for iterative updating processing.
Optionally, the collecting and converging the data of the live broadcast whole process to obtain monitoring source data includes:
data collection processing is carried out on the server node to obtain service monitoring data;
data acquisition processing is carried out on the client nodes to obtain real-time perception data;
synchronizing the service monitoring data and the real-time perception data to a sink node to obtain monitoring data;
and acquiring historical data, and carrying out aggregation processing on the historical data and the monitoring data to obtain monitoring source data.
Optionally, the model training processing is performed by using the monitoring source data as a training data set to obtain a large language model, which includes:
performing data cleaning processing on the monitoring data in the monitoring source data, and increasing the weight of the monitoring data to obtain a training data set;
and carrying out parameter adjustment processing on the pre-training model according to the training data set to obtain a large language model.
Optionally, the performing anomaly detection processing on the monitoring source data through the large language model to obtain a fault result includes:
acquiring real-time monitoring data from the monitoring source data;
converting the monitoring data to obtain natural language text description;
and analyzing and processing the working state of the natural language text description through the large language model, and carrying out fault judgment by combining scenes to obtain a fault result.
Optionally, the combining the large language model with an internal search engine performs real-time search and analysis processing on the fault result to obtain an alarm processing scheme, including:
trend query processing is carried out on the fault result through the internal search engine, so that alarm trend information is obtained;
and carrying out root cause analysis and suggestion processing on the fault result through the large language model to obtain a solution.
On the other hand, the embodiment of the invention also provides a live broadcast monitoring processing system, which comprises:
the data acquisition, transmission and storage module is used for carrying out data collection and aggregation processing on the live broadcast whole process to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data;
the large model training module is used for carrying out model training processing by taking the monitoring source data as a training data set to obtain a large language model;
the monitoring abnormality detection module is used for carrying out abnormality detection processing on the monitoring source data through the large language model to obtain a fault result;
and the alarm analysis and repair module is used for combining the large language model with an internal search engine, searching and analyzing and processing the fault result in real time to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
Optionally, the system further comprises:
the exception handling recording module is used for recording and archiving the alarm handling scheme and establishing a history database and a knowledge sharing base; the history database and the knowledge sharing base are used for carrying out iterative updating processing on the large language model.
Optionally, the data acquisition, transmission and storage module is configured to perform data collection and aggregation processing on the live broadcast whole procedure to obtain monitoring source data, and includes:
data collection processing is carried out on the server node to obtain service monitoring data;
data acquisition processing is carried out on the client nodes to obtain real-time perception data;
synchronizing the service monitoring data and the real-time perception data to a sink node to obtain monitoring data;
and acquiring historical data, and carrying out aggregation processing on the historical data and the monitoring data to obtain monitoring source data.
Optionally, the large model training module is configured to perform model training processing with the monitoring source data as a training data set to obtain a large language model, and includes:
performing data cleaning processing on the monitoring data in the monitoring source data, and increasing the weight of the monitoring data to obtain a training data set;
and carrying out parameter adjustment processing on the pre-training model according to the training data set to obtain a large language model.
Optionally, the monitoring anomaly detection module is configured to perform anomaly detection processing on the monitoring source data through the large language model to obtain a fault result, and includes:
acquiring real-time monitoring data from the monitoring source data;
converting the monitoring data to obtain natural language text description;
and analyzing and processing the working state of the natural language text description through the large language model, and carrying out fault judgment by combining scenes to obtain a fault result.
Optionally, the alarm analysis and repair module is configured to combine the large language model with an internal search engine, perform real-time search and analysis processing on the fault result, and obtain an alarm processing scheme, where the alarm processing scheme includes:
trend query processing is carried out on the fault result through the internal search engine, so that alarm trend information is obtained;
and carrying out root cause analysis and suggestion processing on the fault result through the large language model to obtain a solution.
On the other hand, the embodiment of the invention also discloses electronic equipment, which comprises a processor and a memory;
the memory is used for storing programs;
the processor executes the program to implement the method as described above.
In another aspect, embodiments of the present invention also disclose a computer readable storage medium storing a program for execution by a processor to implement a method as described above.
In another aspect, embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the foregoing method.
Compared with the prior art, the technical scheme provided by the invention has the following technical effects:
the invention provides a live broadcast monitoring processing method, which comprises the following steps: data collection and aggregation processing are carried out on the live broadcast whole flow to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data; performing model training processing by taking the monitoring source data as a training data set to obtain a large language model; performing anomaly detection processing on the monitoring source data through a large language model to obtain a fault result; and combining the large language model with an internal search engine to search and analyze the fault result in real time to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
The live broadcast monitoring processing method provided by the invention is based on the combination of the large language model and the internal search engine to perform anomaly monitoring processing on the live broadcast whole flow, can rapidly find anomalies and accurately locate problems by utilizing the artificial intelligence and large data technology to monitor the system operation condition in real time, and obtains the historical trend of data through the internal search engine. Meanwhile, by combining the advantages of a large language model, the current warning trend, related solutions and repairing methods can be obtained rapidly, and automatic processing and repairing of system anomalies are realized. The invention can greatly improve the accuracy and efficiency of the system monitoring exception handling, reduce the system maintenance cost, and improve the performance and availability of the video networking end-to-end live broadcast service, thereby ensuring the stable operation and high availability of the video networking.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a live broadcast monitoring processing method provided in an embodiment of the present invention;
FIG. 2 is a flow chart of one implementation of step S101 in FIG. 1;
fig. 3 is a schematic node diagram of end-to-end live broadcast according to an embodiment of the present invention;
FIG. 4 is a flow chart of one implementation of step S102 in FIG. 1;
FIG. 5 is a flow chart of one implementation of step S103 in FIG. 1;
FIG. 6 is a flow chart of one implementation of step S104 in FIG. 1;
fig. 7 is a schematic structural diagram of a live broadcast monitoring processing system according to an embodiment of the present invention;
fig. 8 is a process flow diagram of a live broadcast monitoring processing system according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Fig. 10 is a schematic structural diagram of a storage medium according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
First, several nouns involved in the present invention are parsed:
artificial intelligence (Artificial Intelligence, AI) is a technical and scientific field that mimics human intelligence. It involves many different sub-fields including machine learning, natural language processing, computer vision, and expert systems. The goal of artificial intelligence is to enable computers to make intelligent decisions, learn, and solve problems like humans. Its technical basis includes using large amounts of data for learning and model training, automatic reasoning and decision making, pattern recognition and semantic understanding, etc. Through these techniques, artificial intelligence may accomplish a variety of tasks including image recognition, speech recognition, natural language processing, intelligent recommendation, and the like.
Training Models (TM) refer to Training a machine learning Model using existing data sets to enable learning features and rules from the data. The process of training the model generally includes inputting data into the model, calculating an output of the model, then comparing with the actual labels, and adjusting weights and parameters of the model according to the comparison result so as to gradually optimize the model. In general, a large amount of marked data is needed to train a model, and multiple times of training is performed in an iterative mode, so that the accuracy and generalization capability of the model are improved.
A large language model (Large Language Model, LLM), also known as a large language model or large model, is an artificial intelligence model intended to understand and generate human language. They train on a large amount of text data and can perform a wide range of tasks including text summarization, translation, emotion analysis, and so forth. Large language models are characterized by a large scale, containing billions of parameters, which help them learn complex patterns in language data. These models are typically based on deep learning architectures, such as translators, which help them to achieve impressive performance on various Natural Language Processing (NLP) tasks.
The video networking live broadcast service is live broadcast service based on the video networking, wherein the video networking is a video comprehensive service network, and service capabilities such as multi-brand video terminal access, video scheduling, cloud storage and AI application are realized by integrating the capabilities such as cloud network fusion resources and AI.
In the related art, various abnormal conditions may occur in the live broadcast process, such as server failure, user experience problems, abnormal platform operation and the like, which bring negative effects to user experience and service quality.
In view of this, the embodiment of the present invention provides a processing method for live broadcast monitoring, which may be applied to a terminal, a server, software running in a terminal or a server, and the like. The terminal may be, but is not limited to, a tablet computer, a notebook computer, a desktop computer, etc. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms.
Referring to fig. 1, an embodiment of the present invention provides a live broadcast monitoring processing method, where the method includes:
s101, carrying out data collection and aggregation processing on the live broadcast whole process to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data;
s102, taking the monitoring source data as a training data set to perform model training processing to obtain a large language model;
s103, performing anomaly detection processing on the monitoring source data through the large language model to obtain a fault result;
s104, combining the large language model with an internal search engine, and carrying out real-time search and analysis processing on the fault result to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
In the embodiment of the invention, the monitoring data and the historical data are collected by collecting the data of the live broadcast whole process, and the monitoring source data are obtained by gathering the monitoring data and the historical data. And then the monitoring source data is used as a training data set to be input into the model for training, so that a large language model is obtained. And then, performing anomaly detection processing on the real-time monitoring data in the monitoring source data through the large language model to obtain a fault result. And finally, combining the large language model with an internal search engine, and carrying out search analysis on the fault result to obtain an alarm processing scheme. According to the embodiment of the invention, the system operation condition is monitored in real time by utilizing artificial intelligence and big data technology, the problem is rapidly found out and accurately positioned, and the historical trend of the data is obtained through an internal search engine. The embodiment of the invention can greatly improve the accuracy and efficiency of the system monitoring exception handling, reduce the system maintenance cost, and improve the performance and availability of the video networking end-to-end live broadcast service, thereby ensuring the stable operation and high availability of the video networking.
It should be noted that, in each specific embodiment of the present invention, when related processing is required to be performed according to data related to the identity or characteristics of the target object, such as information of the target object, behavior data of the target object, history data of the target object, and position information of the target object, permission or consent of the target object is obtained first, and the collection, use, processing, etc. of the data complies with related laws and regulations and standards. In addition, when the embodiment of the invention needs to acquire the sensitive information of the target object, the independent permission or independent consent of the target object is acquired through a popup window or a jump to a confirmation page or the like, and after the independent permission or independent consent of the target object is explicitly acquired, the necessary target object related data for enabling the embodiment of the invention to normally operate is acquired.
Referring to fig. 1, further as an alternative embodiment, the method further includes:
s105, recording and archiving the alarm processing scheme, and establishing a history database and a knowledge sharing base;
s106, updating the training data set according to the historical database and the knowledge sharing base;
s107, inputting the updated training data set into the large language model for iterative updating processing.
In the embodiment of the invention, after detecting an abnormality at a certain moment or a certain time, the generated alarm processing scheme is recorded and archived, so that a history database and a knowledge sharing library are established and obtained, wherein the history database comprises history alarm information, and the knowledge sharing library comprises history solutions. The historical database and the knowledge sharing library are used for carrying out iterative updating processing on the large language model, after the historical database and the knowledge sharing library for exception processing are established, the training data set can be updated, and the updated training data set is input into the large language model for iterative updating processing. According to the embodiment of the invention, the historical database and the knowledge sharing base are established, the historical alarm information can be acquired through the historical database to train the large language model, and the previous solutions for the same kind or the same kind of fault result can be acquired through the knowledge sharing base, so that a reference basis can be provided for subsequent abnormal processing, and the processing efficiency of live broadcast monitoring is further improved.
Referring to fig. 2, in a further optional embodiment, in the step S101, the performing data collection and aggregation processing on the live broadcast whole procedure to obtain monitoring source data includes:
s201, data collection processing is carried out on the server node to obtain service monitoring data;
s202, data acquisition processing is carried out on the client nodes to obtain real-time perception data;
s203, synchronizing the service monitoring data and the real-time perception data to an aggregation node to obtain monitoring data;
s204, acquiring historical data, and carrying out aggregation processing on the historical data and the monitoring data to obtain monitoring source data.
In the embodiment of the invention, referring to fig. 3, the node distribution of the end-to-end live broadcast service based on the video networking includes a service end node, a client node and an aggregation node, where the service end node is deployed at the cloud, and may include a server, a network device, a (platform as a service) PAAS component, a (software as a service) SAAS service, and the like. The server node is used for monitoring and collecting data such as servers, network equipment, components, services, applications and the like in the server. The client node is deployed at a terminal, which may be a tablet computer, a notebook computer, a desktop computer, or the like. The client node is used for acquiring real-time sensing data from the client side buried point, and then synchronizing the data acquired from the server side and the client side to the sink node of the monitoring system to acquire the monitoring data. And collecting historical data, and converging the historical data with the monitoring data to obtain monitoring source data. Wherein the historical data includes historical alert and fault information. According to the embodiment of the invention, the monitoring source data is obtained by carrying out data cleaning on the operation log, the historical work order, the knowledge base construction and the fault report and converging the data into the source data of the live broadcast monitoring anomaly detection discovery and processing flow. According to the embodiment of the invention, the monitoring data and the historical data of the server side and the client side are subjected to aggregation processing, so that a data basis can be provided for subsequent abnormal monitoring detection.
Referring to fig. 4, in a further optional embodiment, in step S102, the performing model training processing on the monitoring source data as a training data set to obtain a large language model includes:
s401, performing data cleaning processing on the monitoring data in the monitoring source data, and increasing the weight of the monitoring data to obtain a training data set;
s402, carrying out parameter adjustment processing on the pre-training model according to the training data set to obtain a large language model.
In the embodiment of the invention, the monitoring data of the sink node in the monitoring system for training is cleaned, and the weight of the monitoring data of the fault time period is increased, so that the large language model is more concerned with the analysis and the processing of the monitoring data in the data training process. In the embodiment of the invention, a pre-trained GPT 2-xlage model is adopted as a large language model for data processing, wherein the GPT 2-xlage model is a Chinese text processing large language model and is obtained by multi-mode pre-training of a multi-modal pre-training framework tencentPretrain. The embodiment of the invention trains the model by taking monitoring source data as a training data set, wherein the monitoring source data comprises data such as monitoring data, historical alarm information, alarm processing flow, a knowledge base, alarm dispatch information, fault report and the like, and the trimmed large language model is obtained by verifying the large language model by adjusting the learning rate, the layer number, the number of hidden units, the attention head number and the batch processing size and then issuing the data to the production environment.
Referring to fig. 5, in a further optional embodiment, in step S103, the performing, by using the large language model, anomaly detection processing on the monitoring source data to obtain a fault result includes:
s501, acquiring real-time monitoring data from the monitoring source data;
s502, converting the monitoring data to obtain natural language text description;
s503, analyzing and processing the working state of the natural language text description through the large language model, and carrying out fault judgment by combining a scene to obtain a fault result.
In the embodiment of the invention, real-time monitoring data is obtained from monitoring source data, and the monitoring data comprises information such as real-time stream drawing number, stream drawing success number, forwarding stream drawing information, hole punching success rate, CPU and load information, memory service condition and the like. According to the embodiment of the invention, the monitoring data is subjected to text conversion processing to obtain natural language text description, and the analysis of the current monitoring data is performed through a large language model to obtain the root cause, influence, suggested processing steps and the like of the fault. The large language model can analyze whether the equipment and the service in the current state work normally or not, judge whether a fault exists currently or not according to the current scene, obtain a fault result, process the next step if the fault exists, otherwise, add abnormal-free data into the training data set to train the large language model. According to the embodiment of the invention, the large language model is applied to the field of monitoring and alarming, and the monitoring data is trained and learned by using the large language model trained by the neural network based on deep learning, so that the characteristics and modes of the data are extracted, and the abnormal data is detected. The embodiment of the invention can more accurately identify the abnormal data, reduce the situations of false alarm and missing report and improve the accuracy of abnormal detection.
Referring to fig. 6, in a further optional embodiment, in step S104, the combining the large language model with an internal search engine performs real-time search and analysis processing on the fault result to obtain an alarm processing scheme, where the method includes:
s601, carrying out trend query processing on the fault result through the internal search engine to obtain alarm trend information;
s602, performing root cause analysis and suggestion processing on the fault result through the large language model to obtain a solution.
In the embodiment of the invention, the internal search engine is combined with the large language model, and the internal search engine is used for inquiring real-time data and historical trends, such as server memory alarms, so as to inquire the memory utilization rate trend and draw a chart display. And processing data addresses with similar faults of the internal knowledge base according to the analysis conclusion of the large language model. The embodiment of the invention has the capability of actively inquiring real-time and historical monitoring data, can synchronously push the alarm trend information to form a visual chart when the alarm is generated, and calls a large language model to generate an alarm processing method and basis, thereby providing a solution for operation and maintenance personnel to process the alarm. According to the embodiment of the invention, by combining the large model with the internal search engine and utilizing the efficient real-time query capability and index structure of the search engine, the real-time monitoring data and the historical trend can be actively queried for graph display auxiliary analysis, so that the reliability of alarm analysis can be improved.
On the other hand, referring to fig. 7, the embodiment of the present invention further provides a live broadcast monitoring processing system, where the system includes:
the data acquisition, transmission and storage module 701 is configured to perform data collection and aggregation processing on the live broadcast whole process to obtain monitoring source data, where the monitoring source data includes monitoring data and historical data;
the big model training module 702 is configured to perform model training processing by using the monitoring source data as a training data set, so as to obtain a big language model;
the monitoring anomaly detection module 703 is configured to perform anomaly detection processing on the monitoring source data through the large language model, so as to obtain a fault result;
and the alarm analysis and repair module 704 is configured to combine the large language model with an internal search engine, and perform real-time search and analysis processing on the fault result to obtain an alarm processing scheme, where the alarm processing scheme includes alarm trend information and a solution.
Referring to fig. 7, further as a preferred embodiment, the system further comprises:
the exception handling record module 705 is configured to record and archive the alarm handling scheme, and establish a history database and a knowledge sharing base; the history database and the knowledge sharing base are used for carrying out iterative updating processing on the large language model.
Further as an optional implementation manner, the data acquisition, transmission and storage module is configured to perform data collection and aggregation processing on a live broadcast whole procedure to obtain monitoring source data, and includes:
data collection processing is carried out on the server node to obtain service monitoring data;
data acquisition processing is carried out on the client nodes to obtain real-time perception data;
synchronizing the service monitoring data and the real-time perception data to a sink node to obtain monitoring data;
and acquiring historical data, and carrying out aggregation processing on the historical data and the monitoring data to obtain monitoring source data.
Further as an optional implementation manner, the large model training module is configured to perform model training processing with the monitoring source data as a training data set to obtain a large language model, and includes:
performing data cleaning processing on the monitoring data in the monitoring source data, and increasing the weight of the monitoring data to obtain a training data set;
and carrying out parameter adjustment processing on the pre-training model according to the training data set to obtain a large language model.
Further as an optional implementation manner, the monitoring anomaly detection module is configured to perform anomaly detection processing on the monitoring source data through the large language model to obtain a fault result, and includes:
acquiring real-time monitoring data from the monitoring source data;
converting the monitoring data to obtain natural language text description;
and analyzing and processing the working state of the natural language text description through the large language model, and carrying out fault judgment by combining scenes to obtain a fault result.
Further as an optional implementation manner, the alarm analysis and repair module is configured to combine the large language model with an internal search engine, perform real-time search and analysis processing on the fault result, and obtain an alarm processing scheme, where the alarm processing scheme includes:
trend query processing is carried out on the fault result through the internal search engine, so that alarm trend information is obtained;
and carrying out root cause analysis and suggestion processing on the fault result through the large language model to obtain a solution.
It can be understood that the content in the above method embodiment is applicable to the system embodiment, and the functions specifically implemented by the system embodiment are the same as those of the above method embodiment, and the achieved beneficial effects are the same as those of the above method embodiment.
Referring to fig. 8, the process of the present invention specifically includes: the monitoring source data is acquired and converged through the monitoring data acquisition, transmission and storage module, the monitoring source data is input into the large model training model to train the large language model, the trained large language model is added into the monitoring anomaly detection module, and the monitoring source data is input into the monitoring anomaly detection module through the monitoring data acquisition, transmission and storage module to carry out anomaly detection. Judging whether an abnormality exists at present through a monitoring abnormality detection module, and if the abnormality exists, adding data into a data set for updating a large language model; if the abnormality exists, an alarm analysis and repair module is called. The alarm analysis and repair module combines the large language model with an internal search engine to perform alarm analysis on the abnormality to obtain an alarm processing scheme. Recording the alarm processing scheme of the detection processing through an exception processing recording module, adding the recorded data into a training set, and iteratively updating the large language model. The embodiment of the invention applies the large model and the internal search engine to the real-time monitoring system, and rapidly processes and inquires the data, thereby realizing the timely discovery and processing of abnormal data. The embodiment of the invention can realize the intellectualization of anomaly detection by combining the large voice model with the internal search engine, automatically find the anomaly mode and trend, reduce manual intervention and improve the detection accuracy and efficiency. The embodiment of the invention can rapidly process a large amount of monitoring data, improve the monitoring efficiency, reduce the resource waste, can fully utilize the parallel computing capacity of the large model and the internal search engine, can also process large-scale monitoring data, and is suitable for large-scale systems and complex network environments, including live broadcast and other business scenes.
Referring to fig. 9, an embodiment of the present invention further provides an electronic device, including a processor 901 and a memory 902; the memory 902 is used for storing a program; the processor 901 executes the program to implement the method as described previously.
Referring to fig. 10, an embodiment of the present invention further provides a computer-readable storage medium 1001, the storage medium 1001 storing a program 1002, the program 1002 being executed by a processor to implement a method as described above.
Embodiments of the present invention also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the method shown in fig. 1.
In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.
Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the described functions and/or features may be integrated in a single physical device and/or software module or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments described above, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are included in the scope of the present invention as defined in the appended claims.
Claims (10)
1. A live broadcast monitoring processing method, the method comprising:
data collection and aggregation processing are carried out on the live broadcast whole flow to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data;
performing model training processing by taking the monitoring source data as a training data set to obtain a large language model;
performing anomaly detection processing on the monitoring source data through the large language model to obtain a fault result;
and combining the large language model with an internal search engine, and carrying out real-time search and analysis processing on the fault result to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
2. The method according to claim 1, wherein the method further comprises:
recording and archiving the alarm processing scheme, and establishing a history database and a knowledge sharing library;
updating the training data set according to the historical database and the knowledge sharing base;
and inputting the updated training data set into the large language model for iterative updating processing.
3. The method of claim 1, wherein the performing data collection and aggregation on the live broadcast full-flow to obtain the monitoring source data includes:
data collection processing is carried out on the server node to obtain service monitoring data;
data acquisition processing is carried out on the client nodes to obtain real-time perception data;
synchronizing the service monitoring data and the real-time perception data to a sink node to obtain monitoring data;
and acquiring historical data, and carrying out aggregation processing on the historical data and the monitoring data to obtain monitoring source data.
4. The method of claim 1, wherein performing model training processing on the monitoring source data as a training data set to obtain a large language model, comprises:
performing data cleaning processing on the monitoring data in the monitoring source data, and increasing the weight of the monitoring data to obtain a training data set;
and carrying out parameter adjustment processing on the pre-training model according to the training data set to obtain a large language model.
5. The method according to claim 1, wherein the performing anomaly detection processing on the monitoring source data through the large language model to obtain a fault result includes:
acquiring real-time monitoring data from the monitoring source data;
converting the monitoring data to obtain natural language text description;
and analyzing and processing the working state of the natural language text description through the large language model, and carrying out fault judgment by combining scenes to obtain a fault result.
6. The method of claim 1, wherein the combining the large language model with an internal search engine performs real-time search and analysis processing on the fault result to obtain an alarm processing scheme, comprising:
trend query processing is carried out on the fault result through the internal search engine, so that alarm trend information is obtained;
and carrying out root cause analysis and suggestion processing on the fault result through the large language model to obtain a solution.
7. A live monitoring processing system, the system comprising:
the data acquisition, transmission and storage module is used for carrying out data collection and aggregation processing on the live broadcast whole process to obtain monitoring source data, wherein the monitoring source data comprises monitoring data and historical data;
the large model training module is used for carrying out model training processing by taking the monitoring source data as a training data set to obtain a large language model;
the monitoring abnormality detection module is used for carrying out abnormality detection processing on the monitoring source data through the large language model to obtain a fault result;
and the alarm analysis and repair module is used for combining the large language model with an internal search engine, searching and analyzing and processing the fault result in real time to obtain an alarm processing scheme, wherein the alarm processing scheme comprises alarm trend information and a solution.
8. The system of claim 7, wherein the system further comprises:
the exception handling recording module is used for recording and archiving the alarm handling scheme and establishing a history database and a knowledge sharing base; the history database and the knowledge sharing base are used for carrying out iterative updating processing on the large language model.
9. An electronic device comprising a memory and a processor;
the memory is used for storing programs;
the processor executing the program implements the method of any one of claims 1 to 6.
10. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311689592.8A CN117692665A (en) | 2023-12-08 | 2023-12-08 | Live broadcast monitoring processing method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311689592.8A CN117692665A (en) | 2023-12-08 | 2023-12-08 | Live broadcast monitoring processing method, system, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117692665A true CN117692665A (en) | 2024-03-12 |
Family
ID=90136521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311689592.8A Pending CN117692665A (en) | 2023-12-08 | 2023-12-08 | Live broadcast monitoring processing method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117692665A (en) |
-
2023
- 2023-12-08 CN CN202311689592.8A patent/CN117692665A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110717665B (en) | System and method for fault identification and trend analysis based on scheduling control system | |
| US9652318B2 (en) | System and method for automatically managing fault events of data center | |
| US9053436B2 (en) | Methods and system for providing simultaneous multi-task ensemble learning | |
| KR20210019564A (en) | Operation maintenance system and method | |
| CN111524118B (en) | Running state detection method and device of transformer, computer equipment and storage medium | |
| CN111064620A (en) | Power grid multimedia conference room equipment maintenance method and system based on operation and maintenance knowledge base | |
| US11620341B1 (en) | Industrial virtual assistant platform with robotic process automation for knowledge and insights management | |
| CN116225760A (en) | Real-time root cause analysis method based on operation and maintenance knowledge graph | |
| Cerquitelli et al. | Enhancing manufacturing intelligence through an unsupervised data-driven methodology for cyclic industrial processes | |
| Alem et al. | A hybrid intrusion detection system in industry 4.0 based on ISA95 standard | |
| US20220318681A1 (en) | System and method for scalable, interactive, collaborative topic identification and tracking | |
| CN117692665A (en) | Live broadcast monitoring processing method, system, equipment and medium | |
| Long et al. | 6G comprehensive intelligence: network operations and optimization based on Large Language Models | |
| US10762115B2 (en) | Automated response system using smart data | |
| CN113128837A (en) | Big data analysis system of rail transit power supply system | |
| Karimaa | Efficient video surveillance: performance evaluation in distributed video surveillance systems | |
| CN111538887B (en) | Big data graph and text recognition system and method based on artificial intelligence | |
| EP4149075B1 (en) | Automatic suppression of non-actionable alarms with machine learning | |
| CN114401398B (en) | Intelligent video operation and maintenance management system | |
| Li | [Retracted] The Application of AI‐Based Technology in Computer Network Operation and Maintenance | |
| Caravela et al. | A closed-loop automatic data-mining approach for preventive network monitoring | |
| Wang et al. | Ensembled Multi-classification Generative Adversarial Network for Condition Monitoring in Streaming Data with Emerging New Classes | |
| CN117997725A (en) | Multi-mode network intelligent supervision method based on knowledge graph | |
| Mithu | Incorporating a Knowledge Base into Machine Learning Techniques for Forensic Analysis of Control Systems | |
| CN117834202A (en) | SDWAN abnormal flow identification method based on XGBoost model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240318 Address after: Unit 1, Building 1, China Telecom Zhejiang Innovation Park, No. 8 Xiqin Street, Wuchang Street, Yuhang District, Hangzhou City, Zhejiang Province, 311100 Applicant after: Tianyi Shilian Technology Co.,Ltd. Country or region after: China Address before: 200000 room 1423, No. 1256 and 1258, Wanrong Road, Jing'an District, Shanghai Applicant before: Tianyi Digital Life Technology Co.,Ltd. Country or region before: China |
|
| TA01 | Transfer of patent application right |