CN114244691A - Video service fault positioning method and device and electronic equipment - Google Patents
Video service fault positioning method and device and electronic equipment Download PDFInfo
- Publication number
- CN114244691A CN114244691A CN202010939982.6A CN202010939982A CN114244691A CN 114244691 A CN114244691 A CN 114244691A CN 202010939982 A CN202010939982 A CN 202010939982A CN 114244691 A CN114244691 A CN 114244691A
- Authority
- CN
- China
- Prior art keywords
- data
- network
- index
- index parameter
- fault
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 230000008447 perception Effects 0.000 claims abstract description 45
- 238000011156 evaluation Methods 0.000 claims abstract description 8
- 238000012549 training Methods 0.000 claims description 43
- 230000008569 process Effects 0.000 claims description 26
- 230000005540 biological transmission Effects 0.000 claims description 18
- 238000007781 pre-processing Methods 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 2
- 230000004807 localization Effects 0.000 claims description 2
- 238000004891 communication Methods 0.000 abstract description 2
- 239000000523 sample Substances 0.000 description 43
- 238000012545 processing Methods 0.000 description 19
- 238000007726 management method Methods 0.000 description 15
- 230000002159 abnormal effect Effects 0.000 description 14
- 238000013480 data collection Methods 0.000 description 14
- 238000003066 decision tree Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000004140 cleaning Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 5
- 238000005520 cutting process Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
Abstract
The invention discloses a method and a device for positioning a video service fault and electronic equipment, and belongs to the technical field of communication. The specific implementation scheme comprises the following steps: acquiring data of a first index parameter and data of a second index parameter of a video service in a multi-domain network; analyzing to obtain user experience quality information of the video service based on a pre-trained service quality perception model and data of the first index parameter; evaluating the service quality of the video service according to the user experience quality information of the video service; under the condition that the service quality of the video service obtained through evaluation does not meet the preset requirement, fault location is carried out based on a pre-trained fault location model set and data of a second index parameter; the set of fault location models includes a plurality of fault location models, each corresponding to a network domain. Therefore, when the method is oriented to video services, the end-to-end fault location oriented to the global network can be automatically realized, and the location efficiency is improved.
Description
Technical Field
The invention belongs to the technical field of communication, and particularly relates to a method and a device for positioning a video service fault and electronic equipment.
Background
At present, in the video playing period, abnormal conditions such as long initial delay, frequent times of pause, screen blacking and the like often occur due to network faults. For the abnormal situation, the existing fault location method is that the maintenance personnel detects and inspects the fault points possibly existing in the network one by one through the probe. Therefore, the existing fault location method is low in efficiency.
Disclosure of Invention
The embodiment of the invention aims to provide a method, a device and electronic equipment for positioning a video service fault, so as to solve the problem that the existing fault positioning method is low in efficiency when the video service is oriented.
In order to solve the technical problem, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a method for locating a fault of a video service, where the method includes:
acquiring data of a first index parameter and data of a second index parameter of a video service in a multi-domain network;
analyzing to obtain user experience quality information of the video service based on a pre-trained service quality perception model and the data of the first index parameter; the service quality perception model is used for representing the incidence relation between the data of the first index parameter and the user experience quality information;
evaluating the service quality of the video service according to the user experience quality information of the video service;
under the condition that the service quality of the video service does not meet the preset requirement, fault location is carried out based on a pre-trained fault location model set and the data of the second index parameter;
wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain.
Optionally, the fault locating based on the pre-trained fault locating model set and the data of the second index parameter includes:
and respectively inputting the index parameter data of each network domain in the data of the second index parameters into the fault positioning model corresponding to each network domain to carry out fault reasoning.
Optionally, the first index parameter includes: index parameters in a wireless network, index parameters in a core network and index parameters in a bearer network.
Optionally, the second index parameter includes: index parameters in a wireless network, index parameters in a transmission network, index parameters in a core network, index parameters in a bearer network, and index parameters at a user data center.
Optionally, before the data of the first index parameter and the data of the second index parameter of the video service in the multi-domain network are obtained, the method includes:
for each network domain, respectively executing the following processes to obtain the fault location model set:
obtaining a first sample dataset of the network domain; wherein the first sample dataset comprises metric parameter data for the network domain;
and performing model training by using the first sample data set to obtain a fault positioning model corresponding to the network domain.
Optionally, before the data of the first index parameter and the data of the second index parameter of the video service in the multi-domain network are obtained, the method includes:
acquiring a second sample data set; the second sample data set comprises index parameter data of the video service in the multi-domain network and user experience quality information;
preprocessing the second sample data set to obtain a target data set;
and performing model training by using the target data set to obtain the service quality perception model.
Optionally, the preprocessing the second sample data set to obtain a target data set includes:
calculating a maximum information coefficient between the data of each type of index parameter in the index parameter data and the associated user experience quality information by using a maximum information coefficient method;
selecting the target data set from the second sample data set according to the maximum information coefficient obtained by calculation; wherein the target data set includes data of a first index parameter in the index parameter data and user experience quality information associated with the data of the first index parameter, and a maximum information coefficient between the data of the first index parameter and the user experience quality information is greater than a preset threshold.
Optionally, after performing fault location, the method further includes:
and sending the obtained fault information to the network domain with the fault.
In a second aspect, an embodiment of the present invention provides a fault location apparatus for video services, including:
the first acquisition module is used for acquiring data of a first index parameter and data of a second index parameter of the video service in the multi-domain network;
the analysis module is used for analyzing and obtaining user experience quality information of the video service based on a pre-trained service quality perception model and the data of the first index parameter; the service quality perception model is used for representing the incidence relation between the data of the first index parameter and the user experience quality information;
an evaluation module for evaluating the quality of the video service according to the user experience quality information of the video service
The fault positioning module is used for positioning faults based on a pre-trained fault positioning model set and the data of the second index parameter under the condition that the service quality of the video service does not meet the preset requirement;
wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain.
Optionally, the fault location module is specifically configured to:
and respectively inputting the index parameter data of each network domain in the data of the second index parameters into the fault positioning model corresponding to each network domain to carry out fault reasoning.
Optionally, the first index parameter includes: index parameters in a wireless network, index parameters in a core network and index parameters in a bearer network.
Optionally, the second index parameter includes: index parameters in a wireless network, index parameters in a transmission network, index parameters in a core network, index parameters in a bearer network, and index parameters at a user data center.
Optionally, the quality sensing apparatus further includes:
an execution module, configured to perform the following processes for each network domain, respectively, to obtain the fault location model set:
obtaining a first sample dataset of the network domain; wherein the first sample dataset comprises metric parameter data for the network domain;
and performing model training by using the first sample data set to obtain a fault positioning model corresponding to the network domain.
Optionally, the quality sensing apparatus further includes:
the second acquisition module is used for acquiring a second sample data set; the second sample data set comprises index parameter data of the video service in the multi-domain network and user experience quality information;
the preprocessing module is used for preprocessing the second sample data set to obtain a target data set;
and the training module is used for carrying out model training by utilizing the target data set to obtain the service quality perception model.
Optionally, the preprocessing module includes:
the calculating unit is used for calculating the maximum information coefficient between the data of each type of index parameter in the index parameter data and the associated user experience quality information by using a maximum information coefficient method;
a selecting unit, configured to select the target data set from the second sample data set according to the calculated maximum information coefficient; wherein the target data set includes data of a first index parameter in the index parameter data and user experience quality information associated with the data of the first index parameter, and a maximum information coefficient between the data of the first index parameter and the user experience quality information is greater than a preset threshold.
Optionally, the quality sensing apparatus further includes:
and the sending module is used for sending the obtained fault information to the network domain with the fault.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and when executed by the processor, the program or instructions implement the steps of the method according to the first aspect.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium on which a program or instructions are stored, which when executed by a processor implement the steps of the method according to the first aspect.
In the embodiment of the invention, data of a first index parameter and data of a second index parameter of a video service in a multi-domain network can be acquired, user experience quality information of the video service is obtained through analysis based on a pre-trained service quality perception model and the data of the first index parameter, the service quality of the video service is evaluated according to the user experience quality information of the video service, and fault location is carried out based on a pre-trained fault location model set and the data of the second index parameter under the condition that the service quality of the video service obtained through evaluation does not meet preset requirements. Therefore, when the video service is oriented, the end-to-end fault location oriented to the global network can be automatically realized, the location efficiency is improved, the timely perception of the network quality oriented to the user experience can be realized through the service quality perception model trained in advance, the abnormal condition can be accurately and comprehensively perceived in advance before a large number of complaints are made by the user, the fault location can be timely performed before the complaints are made in large numbers, and the user experience is improved.
Drawings
Fig. 1 is a flowchart of a method for locating a fault in a video service according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a business process in an embodiment of the invention;
fig. 3 is a schematic structural diagram of an end-to-end quality sensing and fault locating system for video services in an embodiment of the present invention;
FIG. 4 is a flow chart of a training process of a quality of service awareness model and a network fault location model in an embodiment of the invention;
FIG. 5 is a flow diagram of a quality of service awareness and network fault location inference process in an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a fault location device of a video service according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms first, second and the like in the description and in the claims of the present invention are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the invention may be practiced other than those illustrated or described herein, and that the objects identified as "first," "second," etc. are generally a class of objects and do not limit the number of objects, e.g., a first object may be one or more. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The method for locating a fault of a video service provided by the embodiment of the present invention is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.
Referring to fig. 1, fig. 1 is a flowchart of a method for locating a fault of a video service according to an embodiment of the present invention, where the method is applied to an electronic device, and as shown in fig. 1, the method includes the following steps:
step 101: and acquiring data of a first index parameter and data of a second index parameter of the video service in the multi-domain network.
In this embodiment, the first indicator parameter may include an indicator parameter in a wireless network, an indicator parameter in a core network, an indicator parameter in a bearer network, and the like. The data of the first index parameter is, for example, Deep Packet Inspection (DPI) data in a corresponding network.
Optionally, for the first indicator parameter, the indicator parameter in the wireless network includes, but is not limited to, an air interface delay, a signal strength, an uplink-downlink Round Trip Time (RTT), and the like; the index parameters of the core network include, but are not limited to, Transmission Control Protocol (TCP) link establishment duration, packet retransmission number, packet random number, uplink and downlink RTT delay, and the like; the index parameters of the bearer network include, but are not limited to, a TCP link establishment success rate, a bearer network bidirectional delay, a bearer network packet loss rate, and the like.
In this embodiment, the second index parameter may include an index parameter in a wireless network, an index parameter in a transmission network, an index parameter in a core network, an index parameter in a bearer network, and an index parameter at a user data center. The data of the second index parameter is, for example, network management data in the corresponding network.
Optionally, for the second indicator parameter, the indicator parameter in the wireless network includes but is not limited to: physical Resource Block (PRB) utilization, Radio Resource Control (RRC) user quantity, weak coverage indicators such as Reference Signal Received Power (RSRP) and the like, Interference indicators such as Signal to Interference plus Noise Ratio (SINR) and the like, handover indicators and the like; the index parameters in the transmission network include but are not limited to the indexes of bandwidth utilization rate, CPU utilization rate, time delay, packet loss rate and the like; the index parameters in the core network include but are not limited to the indexes of interface bandwidth utilization rate, CPU utilization rate, routing quality and the like; index parameters in the bearer network include, but are not limited to, indexes such as interface bandwidth utilization, CPU utilization, disk Input/output (I/O) capability, and routing performance; index parameters at the user data center include, but are not limited to, indexes such as server performance (e.g., CPU capability, I/O bus capability, scheduling capability, etc.), server bandwidth, Content Delivery Network (CDN) hit rate, Domain Name System (DNS) node load, etc.
Step 102: and analyzing to obtain the user experience quality information of the video service based on a pre-trained service quality perception model and the data of the first index parameter.
In this embodiment, the service quality awareness model is used to represent an association relationship between data of the first index parameter and the user experience quality information.
In one embodiment, the user quality of experience information may be selected as QoE data at the terminal side. The QoE data is, for example, Video quality Mean Opinion Score (vMOS).
In another embodiment, the user quality of experience information may be selected as a user quality of experience level. For example, the user experience quality level can be divided into five levels, i.e., level 1, level 2, level 3, level 4 and level 5.
Step 103: and evaluating the service quality of the video service according to the user experience quality information of the video service.
Optionally, in order to evaluate the service quality of the video service, a corresponding relationship between the user experience quality information and the service quality may be pre-established, so that after the user experience quality information is obtained through analysis, the corresponding service quality is evaluated. For example, the user experience quality level is divided into five levels, corresponding to good, better, good, bad and bad quality of service, respectively.
Step 104: and under the condition that the service quality of the video service obtained by evaluation does not meet the preset requirement, carrying out fault positioning based on a pre-trained fault positioning model set and data of the second index parameter.
Wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain. For example, the wireless network, the transmission network, the core network, the bearer network, and the user data center may respectively correspond to one fault location model, and all fault location models constitute a fault location model set.
Optionally, the step 104 may include: and respectively inputting the index parameter data of each network domain in the data of the second index parameters into a fault positioning model corresponding to each network domain to carry out fault reasoning. Therefore, end-to-end real-time accurate fault location facing to the global network can be realized.
In the embodiment of the invention, data of a first index parameter and data of a second index parameter of a video service in a multi-domain network can be acquired, user experience quality information of the video service is obtained through analysis based on a pre-trained service quality perception model and the data of the first index parameter, the service quality of the video service is evaluated according to the user experience quality information of the video service, and fault location is carried out based on a pre-trained fault location model set and the data of the second index parameter under the condition that the service quality of the video service obtained through evaluation does not meet preset requirements. Therefore, when the video service is oriented, the end-to-end fault location oriented to the global network can be automatically realized, the location efficiency is improved, the timely perception of the network quality oriented to the user experience can be realized through the service quality perception model trained in advance, the abnormal condition can be accurately and comprehensively perceived in advance before a large number of complaints are made by the user, the fault location can be timely performed before the complaints are made in large numbers, and the user experience is improved.
In the embodiment of the present invention, the service quality perception model may be selected from, but not limited to, a deep learning model, a neural network model, a decision tree model, and the like. The service quality perception model can be obtained by utilizing an XGboost algorithm to train the model. The XGboost algorithm is an integrated algorithm and is used for
To carry out the presentation of the contents,
representing the final classifier model, fkRepresenting decision trees, each decision tree being based on an input sample Xi＝{x1，x2，…xmAnd (6) generating a prediction result, and finally determining the final classification category through voting. For example, the result is classified into five classes of 1,2,3,4, and 5 according to the vMOS value, which respectively represent that the service quality is good, bad, and bad.
Optionally, before the step 101, a service quality perception model may be obtained through pre-training. The training process of the service quality perception model can comprise the following steps:
acquiring a second sample data set; the second sample data set comprises index parameter data of the video service in a multi-domain network and user experience quality information related to the index parameter data; the index parameter data is DPI data in a wireless network, a core network and a bearer network;
preprocessing the second sample data set to obtain a target data set;
and performing model training by using the target data set to obtain the service quality perception model.
Further, the preprocessing process for the second sample data set may include: firstly, data in a second sample data set are cleaned, de-duplicated, normalized and the like, and then data association is carried out on index parameter data and user experience quality information through quintuple, starting time, ending time and the like to obtain a target data set. The five-tuple is, for example, a set of a source IP address, a source port, a destination IP address, a destination port, and a transport layer protocol.
In addition, in order to improve the speed of model training, when the second sample data set is preprocessed, index parameter data with high relevance to the user experience quality information can be selected from the second sample data set, and model training can be performed based on the selected training data set. For example, when selecting the index parameter data, the index parameter data may be selected by using a Maximum Information Coefficient (MIC) obtained by the maximum information coefficient method.
Optionally, the preprocessing the sample data set may include: firstly, a maximum information coefficient MIC between data of each type of index parameters in the index parameter data and associated user experience quality information is calculated by using a maximum information coefficient method, and then a target data set is selected from a second sample data set according to the calculated maximum information coefficient. Wherein the target data set includes data of a first index parameter in the index parameter data and user experience quality information associated with the data of the first index parameter, and a maximum information coefficient between the data of the first index parameter and the user experience quality information is greater than a preset threshold. The preset threshold may be preset based on actual demand. Therefore, index parameter data with high relevance and user experience quality information can be selected by using a maximum information coefficient method, and the model training speed is increased on the premise of ensuring the model precision.
For example, assuming that a target data set is selected from a sample data set D by using a maximum information coefficient method, the following formulas (1) and (2) may be adopted to sequentially calculate MIC between data X of each index parameter of the data set D and tag data, i.e., user experience quality information Y, and if MIC [ X, Y ] is greater than a certain threshold (which may be set according to a scene requirement), the corresponding X and Y are selected until the cycle is completed, and the selected data constitutes a new data set for model training.
In the above formulas (1) and (2), p (X) represents the probability distribution of X; p (Y) represents the probability distribution of Y; p (X, Y) represents the joint probability distribution of X, Y; min (| X |, | Y |) represents a value that is smaller in the number of grids divided in the X direction and the number of grids divided in the Y direction.
Optionally, for the training of the service quality perception model, the selected new feature data with a large correlation coefficient may be used as input data, the user experience quality information such as the vMOS may be used as output data, and the XGBoost algorithm is used for model training. The specific process can comprise the following steps:
s1, given dataset D { (X)i,yi)}(|D|＝n,Xi∈Rm,yiE.g. R), the number of samples of the data set is n, the characteristic data volume is m, and the integrated decision tree is initialized according to the data set:
wherein w is the output leaf vector and T is the number of leaf nodes.
S2, determining a loss function, training the algorithm in an additive mode, and expressing an objective function in the training of the t-th round as follows:
wherein the content of the first and second substances,
for the regularization term, γ and λ are constants, and l () is the cross entropy loss function. Defining a node split loss reduction function as:
wherein, giIs the first order gradient of the loss function, hiIs the second order gradient of the loss function. I is the sample instance set, G, H is the sum of the first order gradient and the second order gradient corresponding to instance I, GL、HL、GR、HRRespectively corresponding left node instance ILAnd right node instance IRFirst order gradient and second order gradient sum.
And S3, performing split search on the decision tree nodes to generate an integrated decision tree. The specific steps for splitting the decision tree nodes are as follows: first, a first-order gradient G ═ Sigma is initialized according to a current node instance set Ii∈IgiSecond order gradient H ═ Sigmai∈IhiLeft split node first order gradient GL0, right split node first order gradient GRLeft split node second order gradient H ═ GL0, right split node second order gradient HRH; sample-by-sample, feature-by-feature update GL、HL、GR、HRUpdate compliance GL←GL+gi,HL←HL+hj，GR←G-GL,HR←H-HLAnd finding the maximum value of the reduction of the splitting loss as LsplitAnd splitting the nodes according to the splitting loss reduction value maximum node splitting condition. Repeating the process untilUntil the entire decision tree is constructed.
And S4, generating a prediction result for the generated decision tree by using the test sample, wherein the corresponding category of the leaf node with the highest probability is the final classification result. The trained XGboost model is a service quality perception model, and abnormal user experience can be accurately inferred through DPI data of a wireless network, a core network and a bearer network.
In an embodiment of the present invention, the training process of the fault location model set may include:
for each network domain, respectively executing the following processes to obtain the fault location model set:
obtaining a first sample dataset of the network domain; wherein the first sample dataset comprises metric parameter data for the network domain;
and performing model training by using the first sample data set to obtain a fault positioning model corresponding to the network domain.
That is, for each network domain such as a wireless network, a transmission network, a core network, a bearer network, or a user data center, a corresponding fault location model may be trained respectively. And integrating the trained fault positioning models to obtain a fault positioning model set. For example, when training the fault location model, model training can be performed through unsupervised learning such as the isolated forest iForest algorithm.
For example, taking a fault location model corresponding to a wireless network as an example, taking a sample data set on the wireless network side as Z, performing model training by using an isolated forest iForest algorithm, where the iForest is composed of t isolated trees (itrees), and each iTree is a binary Tree structure. The training process for the fault localization model for the wireless network may include:
a) randomly selecting psi sample points from a sample data set Z as subsamples, and putting the subsamples into a root node of a tree;
b) randomly appointing a dimension, and randomly generating a cutting point p in the current node data, wherein the cutting point is generated between the maximum value and the minimum value of the appointed dimension in the current node data;
c) generating a hyperplane by using the cutting point, dividing the data space of the current node into 2 subspaces, placing data with the specified dimension smaller than p on the left side of the current node, and placing data with the dimension larger than or equal to p on the right side of the current node;
d) recursion b) and c) in the child node, new child nodes are continuously constructed until only one data in the child node (cutting can no longer be continued) or the child node has reached a defined height. After t iTrees are obtained, the training of the iForest model is finished.
After that, the trained fault location model may be deployed to the cloud server for subsequent fault reasoning. The reasoning process is as follows: firstly, inputting data x in a relevant time period with abnormal user experience into a fault location model, enabling the fault location model to traverse each iTree, then calculating the data x finally falling on the layer number of each iTree (x is at the height of the tree), and finally calculating the height average value of the data x at each iTree. And if the height average value is lower than a preset threshold value, determining that the vehicle is in fault, otherwise, determining that no fault exists. The preset threshold may be preset based on actual demand.
Optionally, after the fault location is performed, the obtained fault information may be sent to the network domain in which the fault occurs, so as to send an alarm for the fault network, thereby implementing automatic operation and maintenance.
The present application is described below with reference to specific embodiments and the accompanying drawings.
As shown in fig. 2, the service processing procedure according to the embodiment of the present application may include:
step 21: and collecting data. The data collected in this step includes three types: 1) DPI data of a wireless network, a core network and a bearer network; 2) network management data of a wireless network, a transmission network, a core network, a bearer network and a user data center; 3) QoE data of the video terminal.
Step 22: preprocessing the data collected in step 21, such as cleaning, duplicate removal, normalization, and the like, and then performing data association on the wireless network, the core network, the bearer network DPI data, and the video QoE data through quintuple, a start-end timestamp, and the like.
Step 23: and judging whether the trained service quality perception model and the trained network fault positioning model are deployed or not. If the deployment is carried out, model reasoning can be directly carried out, and if the model initialization training is not carried out, the model training is started.
Step 24: and training a service quality perception model and a network fault positioning model, and deploying the service quality perception model and the network fault positioning model to a cloud server. For the training process of the service quality perception model and the network fault location model, reference may be made to the above contents, which are not described herein again.
Step 25: and monitoring the service by using the trained service quality perception module.
Step 26: and judging whether the user experience abnormity exists or not. If the user's abnormal experience exists, step 27 is executed, otherwise step 29 is executed.
Step 27: and carrying out fault location by utilizing the network fault location model set.
Step 28: and sending an alarm notice aiming at the fault network.
Step 29: ending the relevant process or starting the next monitoring period.
Referring to fig. 3, an embodiment of the present invention further provides an end-to-end quality awareness and fault location system for video services, which mainly includes a network physical facility layer, a data collection processing module deployed in a cloud server, a model training module deployed in the cloud server, and a model inference module deployed in the cloud server. The data collection and processing module comprises a data collection submodule and a data processing submodule, wherein the data collection submodule can collect data from a physical layer such as a wireless network, a transmission network, a core network, a carrying network and a user data center network, and the collected data can be preprocessed by the data processing submodule. The model training module comprises a service quality perception model and a network fault positioning model set, wherein the service quality perception model is obtained by training a supervised learning model based on a video vMOS label and service associated data, and the network fault positioning model set is obtained by performing unsupervised learning respectively based on data of each domain. The model reasoning module comprises a service quality perception module, a network fault positioning module and an alarm notification module, wherein the service quality perception module is used for identifying abnormal users, the network fault positioning module is used for positioning faults, and the alarm notification module is used for alarming and notifying. It should be noted that the data collection processing module and the model inference module may be disposed on one device, or disposed on different physical devices.
With reference to fig. 3 and fig. 4, the training process of the quality of service awareness model and the network fault location model in this embodiment may include:
s1: the data collection submodule in the data collection processing module collects data, and comprises: s1.1, collecting a vMOS value collected by a video terminal as a label; s1.2, collecting DPI data of the wireless network; s1.3, collecting DPI data of a core network; s1.3, collecting DPI data of the bearing network;
s2: the data processing submodule performs processing such as cleaning, deduplication and normalization on the data collected in the step S1, and then associates the data with key fields such as a quintuple and a start/end timestamp.
S3: the model training module performs supervised learning model training by using the associated data of each domain, namely, the associated data is used as model input data, the vMOS value is used as a label, and the supervised learning model is used for training to obtain a service quality perception model so as to identify abnormal users with poor experience.
S4: the data collection submodule in the data collection processing module collects data, and comprises: s4.1, collecting network management data of the wireless network; s4.2, collecting network management data of the transmission network; s4.3, collecting network management data of the core network; s4.4, collecting network management data of the bearer network; and S4.5, collecting network management data of the user data center.
S5: the data processing submodule performs processing such as cleaning, deduplication and normalization on the data collected in S4.
S6: the model training module utilizes the processed data of each domain to respectively perform model training by using an unsupervised learning method to obtain an end-to-end fault positioning model set consisting of five fault positioning models of a wireless network, a transmission network, a core network, a bearing network and a user data center.
As shown in fig. 3 and fig. 5, the quality of service sensing and network fault location inference process in this embodiment may include:
s1: the data collection submodule in the data collection processing module collects data, and comprises: s1.1, collecting a vMOS value collected by a video terminal as a label; s1.2, collecting DPI data of the wireless network; s1.3, collecting DPI data of a core network; s1.3, collecting DPI data of the bearing network;
s2: the data processing submodule performs processing such as cleaning, deduplication and normalization on the data collected in the step S1, and then associates the data with key fields such as a quintuple and a start/end timestamp.
S3: and inputting the associated data into a service quality perception module to perform abnormal user experience identification, namely inputting the associated data into the service quality perception module to perform reasoning.
S4: if the service quality sensing module judges that the user experience is abnormal, the service quality sensing module sends a service quality alarm to the network fault positioning model;
s5: the network fault positioning module sends a fault positioning related data request to the data collection module;
s6: the data collection submodule in the data collection processing module collects data, and comprises: s6.1, collecting network management data of the wireless network; s6.2, collecting network management data of the transmission network; s6.3, collecting network management data of the core network; s6.4, collecting network management data of the bearer network; s6.5, collecting network management data of the user data center.
S7: the data processing submodule performs processing such as cleaning, deduplication and normalization on the data collected in S7.
S8: inputting the processed data into each fault location model in the corresponding network fault location model set respectively for fault identification, and outputting the result to an alarm notification module;
S9.1-S9.5: the alarm notification sub-module passes the fault-related information to the network domain with the fault, such as a wireless network, a transmission network, a core network, a bearer network and a user data center.
In conclusion, the scheme can directly associate the multi-domain network indexes with the QoE of the user, build the model and construct the service quality perception model, thereby realizing the timely perception of the network quality facing the user experience. Furthermore, according to the abnormal perception of the network quality, the data acquisition and aggregation of the network management of the multi-domain network are integrally and uniformly analyzed, a fault intelligent positioning model set based on unsupervised learning is defined, and the end-to-end real-time accurate fault positioning facing to the global network is automatically realized.
The service quality perception model in the scheme can assist an operator to evaluate the QoE of the video user through network side data, perceive the network service quality provided for the user, find user abnormity in time so as to prevent a large amount of complaints of the user, extract relevant abnormal information such as time and position of the abnormal user and assist further fault location.
The network fault positioning model set in the scheme can automatically and accurately position the faults of the whole network domain in real time and send an alarm notice, so that the manual operation and maintenance cost and the time cost are greatly saved.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a video service fault location apparatus according to an embodiment of the present invention, where the apparatus is applied to an electronic device, and as shown in fig. 6, the video service fault location apparatus 60 may include:
the first obtaining module 61 is configured to obtain data of a first index parameter and data of a second index parameter of a video service in a multi-domain network;
an analysis module 62, configured to obtain user experience quality information of the video service through analysis based on a pre-trained service quality perception model and the data of the first index parameter; the service quality perception model is used for representing the incidence relation between the data of the first index parameter and the user experience quality information;
an evaluation module 63, configured to evaluate the quality of service of the video service according to the user experience quality information of the video service
The fault positioning module 64 is configured to perform fault positioning based on a pre-trained fault positioning model set and the data of the second index parameter when the service quality of the video service obtained through evaluation does not meet a preset requirement;
wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain.
Optionally, the fault location device 60 further includes:
optionally, the fault location module 64 is specifically configured to:
and respectively inputting the index parameter data of each network domain in the data of the second index parameters into the fault positioning model corresponding to each network domain to carry out fault reasoning.
Optionally, the first index parameter includes: index parameters in a wireless network, index parameters in a core network and index parameters in a bearer network.
Optionally, the second index parameter includes: index parameters in a wireless network, index parameters in a transmission network, index parameters in a core network, index parameters in a bearer network, and index parameters at a user data center.
Optionally, the quality sensing apparatus 60 further includes:
an execution module, configured to perform the following processes for each network domain, respectively, to obtain the fault location model set:
obtaining a first sample dataset of the network domain; wherein the first sample dataset comprises metric parameter data for the network domain;
and performing model training by using the first sample data set to obtain a fault positioning model corresponding to the network domain.
Optionally, the quality sensing apparatus 60 further includes:
the second acquisition module is used for acquiring a second sample data set; the second sample data set comprises index parameter data of the video service in the multi-domain network and user experience quality information;
the preprocessing module is used for preprocessing the second sample data set to obtain a target data set;
and the training module is used for carrying out model training by utilizing the target data set to obtain the service quality perception model.
Optionally, the preprocessing module includes:
the calculating unit is used for calculating the maximum information coefficient between the data of each type of index parameter in the index parameter data and the associated user experience quality information by using a maximum information coefficient method;
a selecting unit, configured to select the target data set from the second sample data set according to the calculated maximum information coefficient; wherein the target data set includes data of a first index parameter in the index parameter data and user experience quality information associated with the data of the first index parameter, and a maximum information coefficient between the data of the first index parameter and the user experience quality information is greater than a preset threshold.
Optionally, the quality sensing apparatus 60 further includes:
and the sending module is used for sending the obtained fault information to the network domain with the fault.
It can be understood that the fault location device 60 of the video service according to the embodiment of the present invention can implement the processes of the method embodiment shown in fig. 1, and can achieve the same technical effect, and is not described herein again to avoid repetition.
In addition, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the computer program, when executed by the processor, can implement each process of the method embodiment shown in fig. 1 and achieve the same technical effect, and is not described herein again to avoid repetition.
Referring to fig. 7, an electronic device 70 according to an embodiment of the present invention includes a bus 71, a transceiver 72, an antenna 73, a bus interface 74, a processor 75, and a memory 76.
In the embodiment of the present invention, the electronic device 70 further includes: a computer program stored on the memory 76 and executable on the processor 75. It is understood that the computer program can implement the processes of the embodiment of the method shown in fig. 1 when executed by the processor 75, and achieve the same technical effects, and therefore, the detailed description is omitted here to avoid repetition.
In fig. 7, a bus architecture (represented by bus 71), bus 71 may include any number of interconnected buses and bridges, bus 71 linking together various circuits including one or more processors, represented by processor 75, and memory, represented by memory 76. The bus 71 may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface 74 provides an interface between the bus 71 and the transceiver 72. The transceiver 72 may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 75 is transmitted over a wireless medium via the antenna 73, and further, the antenna 73 receives the data and transmits the data to the processor 75.
The processor 75 is responsible for managing the bus 71 and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And the memory 76 may be used to store data used by the processor 75 in performing operations.
Alternatively, the processor 75 may be a CPU, ASIC, FPGA or CPLD.
The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, can implement each process of the method embodiment shown in fig. 1 and achieve the same technical effect, and is not described herein again to avoid repetition.
Computer-readable media, which include both non-transitory and non-transitory, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention or the portions contributing to the prior art may be essentially embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a service classification device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (14)
1. A method for locating a fault of a video service is characterized by comprising the following steps:
acquiring data of a first index parameter and data of a second index parameter of a video service in a multi-domain network;
analyzing to obtain user experience quality information of the video service based on a pre-trained service quality perception model and the data of the first index parameter; the service quality perception model is used for representing the incidence relation between the data of the first index parameter and the user experience quality information;
evaluating the service quality of the video service according to the user experience quality information of the video service;
under the condition that the service quality of the video service does not meet the preset requirement, fault location is carried out based on a pre-trained fault location model set and the data of the second index parameter;
wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain.
2. The method of claim 1, wherein the fault locating based on the pre-trained fault location model set and the data of the second index parameter comprises:
and respectively inputting the index parameter data of each network domain in the data of the second index parameters into the fault positioning model corresponding to each network domain to carry out fault reasoning.
3. The method of claim 1, wherein the first metric parameter comprises: index parameters in a wireless network, index parameters in a core network and index parameters in a bearer network.
4. The method of claim 1, wherein the second indexing parameter comprises: index parameters in a wireless network, index parameters in a transmission network, index parameters in a core network, index parameters in a bearer network, and index parameters at a user data center.
5. The method according to claim 1, wherein the obtaining of the data of the first index parameter and the data of the second index parameter of the video service in the multi-domain network is preceded by:
for each network domain, respectively executing the following processes to obtain the fault location model set:
obtaining a first sample dataset of the network domain; wherein the first sample dataset comprises metric parameter data for the network domain;
and performing model training by using the first sample data set to obtain a fault positioning model corresponding to the network domain.
6. The method according to claim 1, wherein the obtaining of the data of the first index parameter and the data of the second index parameter of the video service in the multi-domain network is preceded by:
acquiring a second sample data set; the second sample data set comprises index parameter data of the video service in the multi-domain network and user experience quality information;
preprocessing the second sample data set to obtain a target data set;
and performing model training by using the target data set to obtain the service quality perception model.
7. The method of claim 6, wherein said pre-processing said second set of sample data to obtain a target data set comprises:
calculating a maximum information coefficient between the data of each type of index parameter in the index parameter data and the associated user experience quality information by using a maximum information coefficient method;
selecting the target data set from the second sample data set according to the maximum information coefficient obtained by calculation; wherein the target data set includes data of a first index parameter in the index parameter data and user experience quality information associated with the data of the first index parameter, and a maximum information coefficient between the data of the first index parameter and the user experience quality information is greater than a preset threshold.
8. The method of claim 1, wherein after fault locating, the method further comprises:
and sending the obtained fault information to the network domain with the fault.
9. A video service fault location apparatus, comprising:
the first acquisition module is used for acquiring data of a first index parameter and data of a second index parameter of the video service in the multi-domain network;
the analysis module is used for analyzing and obtaining user experience quality information of the video service based on a pre-trained service quality perception model and the data of the first index parameter; the service quality perception model is used for representing the incidence relation between the data of the first index parameter and the user experience quality information;
an evaluation module for evaluating the quality of the video service according to the user experience quality information of the video service
The fault positioning module is used for positioning faults based on a pre-trained fault positioning model set and the data of the second index parameter under the condition that the service quality of the video service does not meet the preset requirement;
wherein the set of fault location models includes a plurality of fault location models, each corresponding to a network domain.
10. The apparatus of claim 9,
the fault location module is specifically configured to: and under the condition that the service quality of the video service does not meet the preset requirement, respectively inputting the index parameter data of each network domain in the data of the second index parameter into a fault positioning model corresponding to each network domain for fault reasoning.
11. The apparatus of claim 9, wherein the first metric parameter comprises: index parameters in a wireless network, index parameters in a core network and index parameters in a bearer network.
12. The apparatus of claim 9, wherein the second indexing parameter comprises: index parameters in a wireless network, index parameters in a transmission network, index parameters in a core network, index parameters in a bearer network, and index parameters at a user data center.
13. An electronic device comprising a processor, a memory and a program or instructions stored on the memory and executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method of fault location of a video service according to any one of claims 1 to 8.
14. A computer-readable storage medium, characterized in that a program or instructions are stored on the computer-readable storage medium, which program or instructions, when executed by a processor, implement the steps of the method for fault localization of video services according to any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010939982.6A CN114244691A (en) | 2020-09-09 | 2020-09-09 | Video service fault positioning method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010939982.6A CN114244691A (en) | 2020-09-09 | 2020-09-09 | Video service fault positioning method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114244691A true CN114244691A (en) | 2022-03-25 |
Family
ID=80742577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010939982.6A Pending CN114244691A (en) | 2020-09-09 | 2020-09-09 | Video service fault positioning method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114244691A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115499302A (en) * | 2022-08-17 | 2022-12-20 | 中国电信股份有限公司 | Monitoring method and device of business system, readable storage medium and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577631A (en) * | 2008-05-07 | 2009-11-11 | 华为技术有限公司 | Method, system and network device for evaluating experience quality of user |
| CN102143389A (en) * | 2011-04-22 | 2011-08-03 | 赛特斯网络科技(南京)有限责任公司 | Quality assurance (QA) system and method of internet protocol television (IPTV) services |
| CN103856789A (en) * | 2014-03-13 | 2014-06-11 | 赛特斯信息科技股份有限公司 | System and method for achieving OTT service quality guarantee based on user behavior analysis |
| CN106022583A (en) * | 2016-05-12 | 2016-10-12 | 中国电力科学研究院 | Electric power communication service risk calculation method and system based on fuzzy decision tree |
| CN111163361A (en) * | 2018-11-07 | 2020-05-15 | 中国电信股份有限公司 | Service fault positioning method, device and system, network equipment and storage medium |
-
2020
- 2020-09-09 CN CN202010939982.6A patent/CN114244691A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577631A (en) * | 2008-05-07 | 2009-11-11 | 华为技术有限公司 | Method, system and network device for evaluating experience quality of user |
| CN102143389A (en) * | 2011-04-22 | 2011-08-03 | 赛特斯网络科技(南京)有限责任公司 | Quality assurance (QA) system and method of internet protocol television (IPTV) services |
| CN103856789A (en) * | 2014-03-13 | 2014-06-11 | 赛特斯信息科技股份有限公司 | System and method for achieving OTT service quality guarantee based on user behavior analysis |
| CN106022583A (en) * | 2016-05-12 | 2016-10-12 | 中国电力科学研究院 | Electric power communication service risk calculation method and system based on fuzzy decision tree |
| CN111163361A (en) * | 2018-11-07 | 2020-05-15 | 中国电信股份有限公司 | Service fault positioning method, device and system, network equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115499302A (en) * | 2022-08-17 | 2022-12-20 | 中国电信股份有限公司 | Monitoring method and device of business system, readable storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3968243A1 (en) | Method and apparatus for realizing model training, and computer storage medium | |
| CN111325463A (en) | Data quality detection method, device, equipment and computer readable storage medium | |
| CN110474786B (en) | Method and device for analyzing VoLTE network fault reason based on random forest | |
| CN108230016B (en) | Agricultural product market price transmission analysis method and analysis device | |
| CN108989075A (en) | A kind of network failure locating method and system | |
| CN107809766B (en) | Method and device for generating machine learning sample for network optimization | |
| CN113489619B (en) | Network topology inference method and device based on time series analysis | |
| US11197175B2 (en) | Forcasting time series data | |
| CN111901134B (en) | Method and device for predicting network quality based on recurrent neural network model (RNN) | |
| CN115622867A (en) | Industrial control system safety event early warning classification method and system | |
| CN114244691A (en) | Video service fault positioning method and device and electronic equipment | |
| US20230188440A1 (en) | Automatic classification of correlated anomalies from a network through interpretable clustering | |
| CN114827951A (en) | Vehicle network quality analysis method and system based on vehicle terminal and storage medium | |
| CN114490303B (en) | Fault root cause determination method and device and cloud equipment | |
| CN109831335A (en) | A kind of data monitoring method, monitor terminal, storage medium and data monitoring system | |
| CN113411236B (en) | Quality difference router detection method, quality difference router detection device, quality difference router detection equipment and storage medium | |
| CN116264575A (en) | Edge node scheduling method, device, computing equipment and storage medium | |
| CN117376084A (en) | Fault detection method, electronic equipment and medium thereof | |
| CN110544182B (en) | Power distribution communication network fusion control method and system based on machine learning technology | |
| CN114241401A (en) | Abnormality determination method, apparatus, device, medium, and product | |
| CN112734699A (en) | Article state warning method and device, storage medium and electronic device | |
| Wrench et al. | A rule induction approach to forecasting critical alarms in a telecommunication network | |
| Rozaki | Design and implementation for automated network troubleshooting using data mining | |
| CN114286196B (en) | Video quality difference cause analysis method and device based on wireless network index | |
| CN115731487A (en) | Data processing method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |