EP1689121B1 - Methods and apparatus for identifying chronic performance problems on data networks - Google Patents
Methods and apparatus for identifying chronic performance problems on data networks Download PDFInfo
- Publication number
- EP1689121B1 EP1689121B1 EP06002307A EP06002307A EP1689121B1 EP 1689121 B1 EP1689121 B1 EP 1689121B1 EP 06002307 A EP06002307 A EP 06002307A EP 06002307 A EP06002307 A EP 06002307A EP 1689121 B1 EP1689121 B1 EP 1689121B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- performance
- baseline
- operating
- level
- mean
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000001684 chronic effect Effects 0.000 title claims abstract description 83
- 238000000034 method Methods 0.000 title claims description 22
- 238000005259 measurement Methods 0.000 claims abstract description 81
- 238000000692 Student's t-test Methods 0.000 claims abstract description 31
- 238000012353 t test Methods 0.000 claims abstract description 31
- 238000012806 monitoring device Methods 0.000 claims description 39
- 238000012360 testing method Methods 0.000 claims description 39
- 230000004044 response Effects 0.000 claims description 7
- 238000012384 transportation and delivery Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 22
- 239000000523 sample Substances 0.000 description 39
- 230000005540 biological transmission Effects 0.000 description 17
- 238000004891 communication Methods 0.000 description 16
- 230000001052 transient effect Effects 0.000 description 13
- 238000004458 analytical method Methods 0.000 description 12
- 238000007726 management method Methods 0.000 description 10
- 238000001514 detection method Methods 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 9
- 230000008859 change Effects 0.000 description 7
- 230000002085 persistent effect Effects 0.000 description 6
- 230000015556 catabolic process Effects 0.000 description 5
- 238000006731 degradation reaction Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 4
- 238000013480 data collection Methods 0.000 description 4
- 230000001934 delay Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000007865 diluting Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000528 statistical test Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/022—Capturing of monitoring data by sampling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
- H04L43/0864—Round trip delays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
- H04L43/087—Jitter
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0888—Throughput
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Definitions
- the present invention relates to methods and apparatus for identifying chronic performance problems on data networks.
- Packetized data networks are in widespread use transporting mission critical data throughout the world.
- a typical data transmission system includes a plurality of customer (user) sites and a data packet switching network, which resides between the sites to facilitate communication among the sites via paths through the network.
- Packetized data networks typically format data into packets for transmission from one site to another.
- the data is partitioned into separate packets at a transmission site, wherein the packets usually include headers containing information relating to packet data and routing.
- the packets are transmitted to a destination site in accordance with any of several conventional data transmission protocols known in the art (e.g., Asynchronous Transfer Mode (ATM), Frame Relay, High Level Data Link Control (HDLC), X.25, IP, Ethernet, etc.), by which the transmitted data is restored from the packets received at the destination site.
- ATM Asynchronous Transfer Mode
- HDLC High Level Data Link Control
- X.25 IP, Ethernet, etc.
- the performance of these networks can be effectively measured using performance metrics such as packet latency, jitter (delay variation), and throughput. Persistent chronic network performance problems result in significant degradation in the performance of the network. Chronic latency, jitter, and throughput problems dramatically impact the transfer of important information and consequently impact business productivity and effectiveness.
- Chronic excessive latency is a significant increase in the time required for data to traverse the network and is one of the major underlying causes of user dissatisfaction with network performance and service levels. Durations of persistent chronic excessive latency may occur for hours or days. Users typically experience chronic excessive latency as substantial increases in application response time and/or failed transactions.
- Chronic excessive jitter may render voice or video streaming unintelligible. Excessive jitter will cause an unacceptable number of packets to be excluded from a reconstructed real-time output signal resulting in perceptible distortions in an audio or video output signal. Users typically experience chronic excessive jitter as a substantial disruption in their ability to understand the streaming media (e.g. voice or video) that they are receiving.
- streaming media e.g. voice or video
- Chronic excessive throughput problems may prevent critical backup or disaster recovery functionality. Users typically experience chronic excessive throughput problems as a substantial increase in the time required to access a remote resource.
- Network management tools allow the detection of excessive latency, excessive jitter, decreased throughput, and the like.
- Network management tools also provide for the generation of alerts informing network operators of certain problems. These techniques range in sophistication from simple ping echo tests to network monitoring devices performing accurate and precise testing and reporting, as described, for example, in U.S. Patent Nos. 5,521,907 , 6,058,102 , and 6,147,998 .
- EP-A- 1 206 085 With respect to the prior art attention is drawn to EP-A- 1 206 085 from which a method and apparatus is known for automated generation of a service level agreement delay value and monitoring of a network.
- Service level agreement delay values are generated by calculating a path delay and standard deviation for a path through the network.
- a confidence interval is determined for the path delay using the standard deviation.
- the service level agreement delay value is generated from the path delay and the confidence interval.
- the network is subsequently monitored in order to predict and track violations of the service level agreement delay value. The subsequent monitoring of the network is performed for both trunks and paths within the network.
- the present invention relates to a method of identifying a chronic performance problem on a data network as defined in claim 1. Further, the present invention relates to a system for identifying chronic performance problems on a data network as defined in claim 9.
- a system for identifying chronic performance problems on data networks includes remote network monitoring devices that provide measurements of performance metrics such as latency, jitter, and throughput, and a storage and processing system, such as a backend system, for collecting and storing data, monitoring and analyzing network performance, and notifying network operators when chronic performance problems are detected.
- the system provides accurate and reliable measurement of network latency, jitter and throughput; management of user performance requirements with configurable sensitivity; a context-sensitive data repository which enables the assessment and evaluation of network performance with respect to circuit, time, bandwidth and performance requirements; and predictions of chronic network performance problems by using statistical and heuristic analysis techniques.
- Chronic performance problems may be recognized via the following methodology. Network behavior is analyzed during an extended period where performance is considered normal or acceptable. This analysis is used to develop a baseline incorporating the average (mean) performance and the variability of the performance during normal operations. Criteria are then established for the network performance levels leading the user to perceive service levels degradations. Criteria are also established for the required confidence levels of chronic network performance problem predictions. These confidence levels are used to control the likelihood that detected problems will be deemed transient and not require specific corrective action. These criteria and confidence levels are established heuristically and can be dynamically adjusted.
- Continuous monitoring of ongoing network performance is then performed, comparing current performance with the baseline behavior, using the network performance criteria to filter out and identify network problems which are judged to be service-impacting and chronic.
- an alert and diagnosis are provided.
- Chronic excessive performance problems may be identified (and distinguished from transient problems) using the statistical "t-test" to estimate the likelihood that a detected degradation in network latency is persistent.
- a t-test assesses whether the means of two groups are statistically different from each other with a certain level of confidence by comparing the means of samples from the two groups, taking into consideration the variance of the samples.
- the t-test is used to compare the mean of one or more performance metrics under baseline conditions to the mean of the performance metric(s) over a recent period of operation and identifies the performance problem as chronic if the t-test suggests the mean performance is statistically different from the baseline.
- such a test may fail to detect a chronic performance problem or, conversely, falsely detect a chronic problem when one does not in fact exist.
- the selected confidence levels which translate to a significant or "alpha" level within the t-test, control the likelihood of a false detection of a chronic problem.
- FIG. 1 A system for monitoring performance for data communication networks is illustrated in Fig. 1 .
- an exemplary data transmission system 10 includes two sites (A and B) and a packet switching network 12 to facilitate communications between the sites.
- Site A is connected to the network by communication lines 20 and 22, which are accessible to a first monitoring device A
- site B is connected to the network by communication lines 24 and 26, which are accessible to a second monitoring device B.
- data network switching network
- network,: etc. refer to networks that carry virtually any kind of information and are not limited to any particular type of hardware, technology, application, or data (audio, video, digital data, telephony, etc.).
- the data transmission system 10 can include conventional communications line types, for example, T3, OC-3, North American T1 (1.544 Mbits/second), CCITT (variable rate), 56K or 64K North American Digital Dataphone Service (DDS), Ethernet, and a variety of data communications connections, such as V.35, RS-449, EIA 530, X.21 and RS-232.
- Sites A and B are each capable of transmitting and receiving data packets in various protocols utilized by the communication lines, such as Asynchronous Transfer Mode (ATM), Frame Relay, High Level Data Link Control (HDLC) and X.25, IP, Ethernet, etc.
- Each line 20, 22, 24, 26 represents a respective transmission direction as indicated by the arrows.
- the arrows on communication lines 20 and 22 represent transmissions from site A to the network and from the network to site A, respectively
- the arrows on communication lines 24 and 26 represent transmissions from site B to the network and from the network to site B, respectively.
- site A and site B utilize switching network 12 to communicate with each other, wherein each site is connected to switching network 12 that provides paths between the sites.
- switching network 12 For illustrative purposes, only two sites (A and B) are shown in Fig. 1 . However, it will be understood that the data communication system can include numerous sites, wherein each site is generally connected to multiple other sites over corresponding transmission circuits within the switching network.
- monitoring devices may be positioned at various points throughout the system, including endpoints associated with particular sites (as shown in Fig. 1 ) or at intermediate points in the switching network.
- the term "packet” (e.g., as used in "packetized switching network”) does not imply any particular transmission protocol and can refer to units or segments of data in a system using, for example, any one or combination of the above-listed data transmission protocols (or other protocols).
- packet is often associated with only certain data transmission protocols, to avoid any suggestion that the system of the present invention is limited to any particular data transmission protocols, the term “protocol data unit” (PDU) will be used herein to refer generically to the unit of data being transported by the communication network, including any discrete packaging of information.
- a PDU can be carried on a frame in the frame relay protocol, a related set of cells in the ATM protocol, a packet in an IP protocol, etc.
- monitoring devices A and B are respectively disposed between switching network 12 and sites A and B.
- Monitoring devices A and B can be located at sites A and B, at any point between switching network 12 and sites A and B, or at points within the switching network itself.
- the placement of the monitoring devices depends at least in part on the portion of the system or network over which a network service provider or other party wishes to monitor performance of data traffic flow.
- the service provider will want any performance commitments to be limited to equipment or portions of the network over which it has control.
- the service provider does not want to be responsible for performance problems or degradation caused by end-site equipment or portions of the network not owed or managed by the service provider.
- monitoring devices may be placed at virtually any point in the network or any point within an enterprise LAN.
- the monitoring devices can be any type of network monitoring device or probe and can comprise standalone hardware/software devices or software and/or hardware added to network equipment such as PCs, routers, CSU/DSUs (channel service unit/data service unit), FRADS, voice switches, phones, etc.
- Software embedded in the monitoring devices can collect network performance data for detailed analysis and report generation relating to any of a variety of performance metrics.
- a monitoring device can be a CSU/DSU that operates both as standard CSU/DSU and as managed devices capable of monitoring and inserting network management traffic; an inline device residing between a DSU and router, which monitors network traffic and inserts network management traffic; or a passive monitoring device that monitors network traffic only.
- the monitoring devices can be "active" monitoring devices capable of inserting PDUs into the data traffic (e.g., test PDUs) or "passive" monitoring devices that observe PDU data traffic without generating or inserting any test PDUs into the data traffic for the purpose of measuring performance metrics.
- passive monitoring devices can forward measurement data to each other via an out of band channel or through a link other than the monitored network channel, thereby permitting passive monitoring devices to directly coordinate testing periods.
- the monitoring devices can collect measurement data relating to any of a variety of performance metrics associated with operation of the network.
- performance metrics such as latency, jitter, and throughput are described.
- the invention is not limited to the measurement or analysis of any particular performance metric or any particular combination of metrics, and the principles of the invention can be applied to virtually any type of measurements associated with performance or operation of a network.
- latency relates to the amount of time required for signals or PDUs to traverse the network. Latency can be evaluated in terms of one-way delays between two points, or round-trip delays relating to the time required for an outbound signal to traverse the network from a first point to a second point plus the time required for a reply signal to traverse the network from the second point back to the first point. Such measurements can be made in any of a variety of ways, including active and passive measurement schemes, such as those described in U.S. Patent Nos. 5,521,907 and 6,058,102 .
- Jitter relates to the variation in delay among a set of PDUs.
- the quality of real-time data transmissions depends on the network's ability to deliver data with minimal variation in 5 the packet delay.
- PDUs of voice or video data are transmitted, a sequence of PDUs is sent to the network with fairly consistent time differences between successive PDUs, resulting in a relatively steady stream of PDUs.
- This stream of PDUs must essentially be reconstructed at the destination to accurately reproduce the audio or video signal. Due to conditions on the network, PDUs may experience different delays before arriving at a destination or may be dropped altogether and not reach the destination. PDUs arriving at the destination are buffered to compensate for some degree of delay variation.
- the output signal must be generated from the data in the PDUs within a reasonable period of time to avoid perceptible delays in the output audio or video signal. Consequently, PDUs not received within a predetermined period of time are considered to be dropped, and the output signal is reconstructed without such PDUs to keep voice calls static free and video running smoothly. Excessive jitter will cause an unacceptable number of PDUs to be excluded from the reconstructed real-time output signal resulting in perceptible distortions in the audio or video output signal. Any of a variety of techniques can be used to measure jitter.
- Throughput relates to the amount of data traffic received during a period of time, yielding a rate of data traffic or PDUs.
- the measured data rate may relate to a particular circuit, channel, site, application, user(s), or device, for example.
- the actual throughput rate can be compared to a committed information rate to determine how the level of usage relates to levels agreed upon by the service provider and customer (e.g., in a service level agreement).
- Centralized backup schemes involving transmission of volumes of data to backup sites (e.g., for daily, overnight system backups) are sensitive to loss of throughput, since reduced throughput levels prevent backups from being completed in a timely manner. Techniques such as those described in U.S. Patent No. 5,867,483 can be used to measure throughput.
- Related throughput metrics such as data delivery ratio can also be evaluated, as described for example in U.S. Patent No. 6,058,102 .
- the network monitoring system further includes a backend processing system 30 which receives measurement data either directly or indirectly from the monitoring devices, and collects and stores measurement data and processes the data to produce the various displays and reports required to monitor performance of the network and its components.
- the architecture depicted in Fig. 1 is a conceptual diagram illustrating major functional units and does not necessarily illustrate physical relationships or specific physical devices within the backend processing system or between the backend processing system and the monitoring devices.
- the configuration and components of the backend processing system can take many forms and are described herein only in general terms for context. Those skilled in the art will appreciate that the techniques described herein for identifying chronic performance problems can be implemented within any network monitoring system that collects data on performance metrics, regardless of the particular architecture of the system.
- backend processing system 30 includes a data storage capability represented by storage module 32 for storing measurement data as well as information generated by processing measurement data, such as aggregated report data, analysis results, and historical information.
- Processing system 30 further includes a management and processing capability represented in Fig. 1 by processor module 34, which performs various processing tasks, such as performing operations on raw measurement data to produce reports and performing analysis operations, such as the chronic performance analysis described herein.
- the system further includes a display, interface, and report capability represented by display/report module 36, which displays performance information in a tabular or graphical manner via an interactive graphical user interface, for example, and preferably includes the capability to generate various performance reports.
- the backend system may receive measurement data directly from the monitoring devices or may receive measurement data indirectly (e.g., the monitoring devices may supply measurement data to a storage device at the local site, which is subsequently supplied to the backend system.
- the links shown in Fig. 1 between the monitoring devices and the backend processing system are merely indicative of the fact that measurements from the monitoring devices are ultimately supplied in some manner to a backend processing system.
- the backend processing system may be located at a single site or may have components distributed throughout the network at multiple locations.
- storage module 32 may constitute storage capabilities at a number of local sites as well as a storage capability at one or more backend processing sites.
- various backend processing tasks nominally represented by processor 34 in Fig.
- the display/reporting capability may allow access to performance information via interfaces at a number of sites or via a web-based interface accessible to authorized customers or service provider personnel.
- the processing required to identify chronic performance problems according to the invention can be carried out in a backend processing system such that shown in Fig. 1 , although optionally at least some of the operations may be performed directly within the monitoring devices.
- the performance analysis scheme described herein can be incorporated into an auto-troubleshooting management system that enables system operators to identify and react more quickly to problems within a data network.
- Fig. 2 is a functional flow diagram of the operations performed to identify chronic performance problems on a data network in accordance with an exemplary embodiment of the invention.
- a first operation 200 baseline operating conditions are determined during normal operation, i.e., a period of time in which behavior of the monitored network or portion of the network is considered acceptable.
- a new network or portion of a network e.g., a new site, channel, circuit, link, etc.
- adjustments are required to achieve normal, acceptable operation.
- one or more monitoring devices periodically collect performance metric measurements on one or more performance metrics of interest.
- a monitoring device may make instantaneous measurements of PDU latency, throughput, and jitter at periodic increments of time (e.g., every second) over a measurement period (e.g., every fifteen minutes).
- the invention is not limited to any particular performance metrics, and other performance metrics could be evaluated, including but not limited to: data delivery ratio; and joint or composite performance metrics constructed from combinations of latency, jitter, and throughput, etc.
- a measurement value is computed for the performance metric(s) based on the instantaneous measurements (e.g., by computing the average value).
- each measurement period results in a single measurement value of the performance metric(s).
- one performance metric measurement is generated every fifteen minutes.
- the invention is not limited to measurement periods of any particular duration or any particular number of instantaneous measurements made during a measurement period.
- a measurement value could be taken and analysis performed every second, minute, hour, day, etc., or at irregular intervals (e.g., upon the occurrence of a particular event).
- a measurement value can be a "raw" measurement or determined from a set of measurements (e.g., by some computation or aggregation).
- Determining baseline conditions involves computing the mean and variance of a set of performance metric measurements.
- the sample size i.e., the number of measurements
- the baseline data collection period can be a week long which, in the case of a fifteen minute measurement period, would result in a sample size of 672 measurements.
- the invention is not limited to any particular baseline sample size or a baseline data collection period of any particular length.
- the mean, variance, and sample size of the baseline sample are used to perform statistical testing on samples taken during ongoing operation of the network, as described in greater detail below.
- baseline conditions could be derived from other sources. For example, if the expected behavior of performance metrics to be monitored can be accurately modeled (e.g., the mean and variance of the performance metric under acceptable operating conditions can be reliably predicted with a certain degree of accuracy), then expected baseline conditions could be substituted for actual, measured baseline conditions.
- thresholds to be used in testing of performance metrics to identify chronic problems are set.
- the first type of threshold is a service performance threshold that relates to the current value of the performance metric measurement relative to the baseline value or some absolute value.
- One of the criteria for identifying a chronic performance problem is that the current value of the performance metric measurement (or a mean of recent values) exceeds the service performance threshold (this is a necessary but not sufficient condition to declare a chronic problem).
- the service performance threshold can be selected in any number of ways.
- the threshold can be an absolute increase relative to the baseline mean of the performance metric. For example, for latency, the threshold could be set at a fixed value of 40 ms greater than the baseline mean latency.
- the service performance threshold could also be set as an absolute limit, irrespective of the baseline mean value of the performance metric. For example, the latency threshold could be set at a predetermined value of 100 ms.
- the service performance threshold could also be set as a relative increase with respect to the baseline mean value of the performance metric, for example, a 30% increase. Another option is to set a threshold that is a certain number of standard deviations away from a baseline operating mean value. Any one or combination of these criteria could be used to develop the service performance threshold.
- a combination of an absolute limit and a relative increase may produce a threshold better suited to real-world conditions.
- the baseline mean latency is 50 ms.
- the service performance threshold would be set at 65 ms.
- the absolute threshold should not be set below this level.
- the threshold could be set to a relative increase over the baseline mean latency or at an absolute value (e.g., 70 ms), which ever is greater.
- such thresholds may be context specific. For example, a 70 ms latency may be acceptable for local service, whereas a latency of 120 ms may be more suitable for national service.
- a confidence level threshold associated with statistical testing of the performance metric data is selected. As described in greater detail below, merely exceeding the service performance threshold is not sufficient to conclude that a chronic performance problem has developed. Essentially, the statistical testing attempts to identify a pattern of performance behavior that suggests the mean of performance metric measurements under ongoing operating conditions is significantly different from the baseline mean, thereby indicating that a chronic performance problem is developing.
- the confidence level threshold corresponds to a level of reliability of the statistical testing.
- a high confidence level threshold reduces the number of "false positives" from the statistical testing, such that there is high confidence that a chronic problem does indeed exist when the statistical testing indicates that the performance metric mean has changed from the baseline conditions.
- a high confidence level threshold increases the likelihood that a chronic problem may go undetected for a longer period of time. A lower confidence level threshold will result in more false alarms, but may detect a chronic problem more quickly.
- the confidence level threshold determines how sensitive the system is to indicating there is a chronic problem, and selection of an appropriate confidence level threshold may depend on several consideration. For example, the confidence level threshold should be set to reflect how sensitive the particular customer or service provider is to being disturbed by a false alarm or by the onset of a chronic performance problem. This, in turn, may relate to the type of operation being performed or data being transmitted, terms of a service level agreement between the customer and service provider, or the particular preferences of the customer. If the baseline conditions are more noisy (a relatively large baseline variance), a relatively high confidence level may be appropriate to avoid frequent false alarms.
- the confidence level threshold can be a configurable parameter that can be adjusted if necessary based on experience or past performance, such that the confidence level is heuristically determined (i.e., adjusted in response to past experience) either by human experience or by a software algorithm.
- the confidence level threshold can be initially set to a default sensitivity level and then subsequently adjusted by the customer or service provider via a graphical user interface. For example, a drop-down menu can be used to let a customer select one of several confidence levels (high, moderate, low). Another option is a slider bar, which allows the confidence level threshold to be set to a number of values on a sliding scale by moving an indicator on the slider.
- the selected confidence level threshold internally corresponds to an ⁇ value used in the statistical t-testing, as described in greater detail below.
- the network or portion of the network of interest is monitored by collecting measurements of one or more performance metrics (e.g., latency, throughput, jitter, etc.) from various monitoring devices, and these measurements are supplied to a processing system for analysis.
- performance metrics e.g., latency, throughput, jitter, etc.
- the mean and variance of recent measurements of the performance metric are periodically computed.
- the computation is repeated each time a new measurement is available so that testing for chronic performance problems can be performed with each new measurement, thereby enhancing the likelihood of early detection of a chronic problem.
- the set of measurements within a certain time window constitutes an operating sample (a sample of measurements taken during operation of the network) that is used to compute the mean and variance during network operation.
- the time window extends over a period of time that covers the set of measurements, terminating with the most recent measurement of the performance metric.
- the time window can be of a predetermined duration or can cover a predetermined number of measurements.
- the time window can be adjustable, variable, and/or user selectable.
- the number of measurements in the time window is the sample size n 2 of the operating sample. In general, as each new measurement becomes available, the time window slides forward and drops the oldest measurement from the operating sample used to compute the operating mean and variance, such that the operating sample slides forward in time.
- n 2 of the operating sample A number of factors should be considered in selecting the sample size n 2 of the operating sample. Increasing the sample size tends to produce a more stable result, making the analysis less prone to a false indication of a chronic performance problem. However, a larger sample size requires a longer time window, which may unduly increase the time required to detect a chronic performance problem, since a significant number of older measurements within the time window may reflect acceptable operation prior to the onset of a problem, thereby diluting the impact of more recent measurements that suggest a problem. Conversely, if the time window is too short (e.g., only a few measurements in the sample), there is an increased risk that a transient problem could cover a substantial portion of the window, thereby triggering false detections of chronic problems.
- setting the sample size is analogous to setting the gain of a filter, with a smaller sample size corresponding to a higher gain (which is highly responsive to recent input but more prone to "overreact” under certain conditions), and a larger sample size corresponding to a lower gain (which produces a smoother response but may not react quickly enough under certain conditions).
- a sample size of 96 measurements would correspond to a full day's worth of measurements.
- a time window of this duration may be too long to timely detect the onset of a chronic performance problem in some circumstances, and a shorter time window of a few hours may be more suitable in such circumstances (e.g., a window of 4 or 5 hours corresponds to 16 or 20 measurements). If performance metric measurements are available more frequently, somewhat larger sample sizes can be used without unduly delaying detection of chronic performance problems.
- the testing for detecting a chronic performance problem essentially involves two tests: a threshold test of the value of the performance metric; and a statistical test that indicates with some degree of confidence that the mean value of the performance metric has changed from the baseline value.
- a threshold test is conducted to determine whether the current value of the performance metric of interest exceeds the service performance threshold selected in the manner previously described. If the service performance threshold is not exceeded, performance is considered to be at an acceptable level (i.e., there is no performance problem detected, chronic or otherwise), and statistical testing is unnecessary to determine whether a performance problem is chronic. Accordingly, it is assumed that no chronic performance problem exists, and the testing ends until receipt of the next performance metric measurement.
- a t-test score is computed for the operating sample based on the baseline data and the measured performance data.
- the t-test essentially allows one to compare two sets of data (i.e., the baseline sample and the sample taken during ongoing network operation) and either "reject” or "fail to reject,” with a certain degree of confidence, the hypothesis that the current operating conditions remain the same as the baseline conditions.
- the t-test indicates whether the difference between the mean of the baseline sample and the mean of the operating sample is statistically significant to the point where it can be stated with a specified degree of confidence that the difference arose from an actual change in the performance metric being measured rather than by random chance.
- the baseline (first) random sample ( X 1 , X 2 ,..., X n1 ) of size n 1 whose sample mean is denote by X (given by equation 1) and whose variance is denoted by s 1 2 (given by equation 2) is compared to an operating (second) random sample ( Y 1 , Y 2, ..., Y n2 ) of size n 2 whose mean is denote by Y (given by equation 3) and whose variance is denoted by s 2 2 (given by equation 4).
- t X ⁇ - Y ⁇ S 1 2 n 1 + S 2 2 n 2
- the computation for the value t may vary somewhat based on statistical assumptions, and the invention is not limited to a particular computation for the value t .
- the value of t ⁇ is a function of the degrees of freedom, which is equal to n 1 + n 2 - 2, and of the confidence level threshold selected in the manner described above.
- the confidence level threshold corresponds to t-test level of confidence (1- ⁇ ). Essentially, the value of t ⁇ will result in a false alarm that there is a problem (1 - ⁇ )*100 percent of the time.
- the value of t ⁇ can be determined from the standard t distribution, based on the degrees of freedom and the value of ⁇ (e.g., such values are often provided in statistical tables).
- some of the assumptions associated with the t-test may not be met by the baseline and operating samples (e.g., the samples may not be normally distributed, the true variances may not be equal, the samples may not be random, the two samples may not be independent of each other, etc.). For this reason, the selected value of ⁇ (and hence t ⁇ ) may not yield the precise rate of false alarms ideally expected.
- the t-test still provides a good indication of whether a meaningful change in performance has occurred relative to the baseline.
- a heuristic approach can be used to obtain the desired rate of false alarms, wherein after empirically observing the rate of false alarms during actual operation using an ideal value for t ⁇ , the value of t ⁇ can be adjusted to a suitable value to obtain the desired false alarm rate.
- the analysis performed according to the present invention involves forecasting in the sense that detection of a chronic problem amounts to an assertion that a problem is not transient; the problem is going to persist in the future.
- historical data is used to forecast that a problem will be ongoing.
- the t-test is well-suited to providing such a forecast, since the t-test, in effect, yields a reliable, objective indication, with a controllable level of confidence, of whether observed changes in performance are likely the result of a change of conditions rather than the result of chance.
- the t-test gives a measure consistency or reliability. When the t-test value exceeds the significance threshold t ⁇ , it suggests that a performance metric value that is over a service performance threshold is likely to stay above the threshold.
- both the service performance threshold must be exceeded, indicating that some type of performance problem exist, and the t-test must indicate that the change in the mean value of the performance metric is significantly different from the baseline value. This combination of criteria filters out the circumstances that should not trigger detection of a chronic performance problem.
- the value of the performance metric may be above the service performance threshold, but only for a short time, meaning the problem could be sporadic or transient. This case will typically fail the confidence test provided by the t-test, effectively indicating that there is not sufficient confidence that the problem is going to persist. In another case, the t-test could theoretically indicate a significant statistical deviation from baseline; however, if the service performance threshold has not been exceeded by the current value of the performance metric, the performance is still considered acceptable, so there is no concern that the customer is experiencing unacceptable performance. A problem is reported only if the performance is poor with some persistence.
- Fig. 2 suggests that the t-test is performed only if the service performance threshold is exceeded.
- both the service performance threshold test and the t-test could be performed (performing of the t-test is not dependent on the outcome of the service performance threshold test), and the decision of whether a chronic performance problem exists is determined after the outcomes of the two tests are known (as opposed to the sequential logic shown in Fig. 2 ).
- the result of this approach is identical to the approach shown in Fig. 2 , but would generally require more processing.
- the sequence of the tests shown in Fig. 2 could be reversed, with the t-test being performed first and then the service performance threshold test being performed only if the t-test indicates a significant change of conditions.
- the mean and standard deviation of the operating sample are computed in operation 220.
- computation of the mean and standard deviation of the operating sample could be deferred until the occurrence of the current value of the performance metric exceeding the service performance threshold, since the mean and standard deviation are required only if the t-test is actually performed.
- Another option is to perform the service performance threshold testing by comparing the mean of the operating sample to the threshold rather than just the most recent (current) measurement of the performance metric (this would require the mean and standard deviation to be computed with each measurement, as shown in Fig. 2 ).
- using only the current value of the performance metric in the threshold test may have the advantage of permitting a more rapid detection of chronic problems, since the service performance threshold test fails every time a measurement exceeds the service performance threshold.
- the system can include a feature that allows historical data to be re-run through the chronic problem testing sequence using different thresholds to see if data that did not adequately trigger an alarm at a high threshold would have identified a chronic problem using a lower threshold (or, conversely, if a higher threshold would have avoided a false alarm). This feature assists in pro-actively adjusting thresholds.
- test results can be supplied to a management system for inclusion in graphical displays of network performance and for report generation.
- a management system can trigger an alarm or provide notice to an administrator of the condition.
- An alarm could be, for example, in the form of a visual indicator, an audible indicator or both.
- Fig. 3 illustrates a scenario in which a modest rise in latency on a network with stable performance and low variance results in a detection of a chronic performance problem.
- the higher latency measurements shown with cross-hatching will quickly result in the service performance threshold indicating a performance problem.
- the t-test will identify the latency performance problem as chronic and set an alarm.
- Fig. 3 by 6:00 am, corrective measures have been taken and the performance returns to normal. Subsequent slight increases in latency at 10:00 am and 2:00 are not identified as chronic problems.
- the invention has been described primarily in the context of "network performance," the techniques of the invention can be applied at various levels in the network architecture including specific applications and types or groups of applications. Thus, in this context, the invention is capable of monitoring "application performance" and can identify chronic performance problems with certain applications.
- One example of a suitable performance metric for applications would be TCP retries.
- the principles of the present invention may be applied not only to packetized communications networks (e.g. Frame Relay, SMDS, ATM, IP, etc.), but also to any communications network wherein performance metrics of transmitted data can be reliably measured.
- packetized communications networks e.g. Frame Relay, SMDS, ATM, IP, etc.
- performance metrics of transmitted data can be reliably measured.
- the principles of the present invention could be applied, for example, to measure performance metrics such as latency, throughput, or jitter in a non-packetized leased-line network.
- the term PDU encompasses virtually any identifiable portion of a data stream from which the same identifier can be generated at two points in a network.
- any data gathering devices capable of capturing and recording performance metrics can be used according to the principles of the present invention.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Debugging And Monitoring (AREA)
Abstract
Description
- The present invention relates to methods and apparatus for identifying chronic performance problems on data networks.
- Packetized data networks are in widespread use transporting mission critical data throughout the world. A typical data transmission system includes a plurality of customer (user) sites and a data packet switching network, which resides between the sites to facilitate communication among the sites via paths through the network.
- Packetized data networks typically format data into packets for transmission from one site to another. In particular, the data is partitioned into separate packets at a transmission site, wherein the packets usually include headers containing information relating to packet data and routing. The packets are transmitted to a destination site in accordance with any of several conventional data transmission protocols known in the art (e.g., Asynchronous Transfer Mode (ATM), Frame Relay, High Level Data Link Control (HDLC), X.25, IP, Ethernet, etc.), by which the transmitted data is restored from the packets received at the destination site.
- The performance of these networks can be effectively measured using performance metrics such as packet latency, jitter (delay variation), and throughput. Persistent chronic network performance problems result in significant degradation in the performance of the network. Chronic latency, jitter, and throughput problems dramatically impact the transfer of important information and consequently impact business productivity and effectiveness.
- Chronic excessive latency is a significant increase in the time required for data to traverse the network and is one of the major underlying causes of user dissatisfaction with network performance and service levels. Durations of persistent chronic excessive latency may occur for hours or days. Users typically experience chronic excessive latency as substantial increases in application response time and/or failed transactions.
- Chronic excessive jitter may render voice or video streaming unintelligible. Excessive jitter will cause an unacceptable number of packets to be excluded from a reconstructed real-time output signal resulting in perceptible distortions in an audio or video output signal. Users typically experience chronic excessive jitter as a substantial disruption in their ability to understand the streaming media (e.g. voice or video) that they are receiving.
- Chronic excessive throughput problems may prevent critical backup or disaster recovery functionality. Users typically experience chronic excessive throughput problems as a substantial increase in the time required to access a remote resource.
- Most packetized data networks exhibit predictable, stable behavior in terms of latency, jitter, and throughput. For example, most data packets traversing a domestic network may require 70 ms for a request and reply to traverse the network. This average latency may remain fairly stable over extended periods. Nevertheless, during these periods, networks can experience transient increases in traffic latency. For example, occasionally some data packets on the same network may require 200 ms or more to traverse the network. These transient increases in traffic latency are very different from chronic problems, because they typically do not affect perceived service levels. Transient occurrences of excessive latency, jitter, or throughput are a normal part of network operations and do not warrant dedicated corrective action.
- However, problems with chronic excessive performance problems and the consequential impact on business productivity and effectiveness do require explicit corrective action. This is particularly important, because the cost of resolving these problems can he a significant proportion of the support and management costs for network service providers.
- Currently, there is no reliable mechanism for effectively recognizing or predicting the onset of periods of chronic performance problems. Service providers cannot distinguish between transient and persistent problems. The requirement to avoid expensive and unnecessary responses to transient problems means that the required corrective action for persistent problems is delayed, resulting in reduced business productivity and effectiveness.
- Current network management tools allow the detection of excessive latency, excessive jitter, decreased throughput, and the like. Network management tools also provide for the generation of alerts informing network operators of certain problems. These techniques range in sophistication from simple ping echo tests to network monitoring devices performing accurate and precise testing and reporting, as described, for example, in
U.S. Patent Nos. 5,521,907 ,6,058,102 , and6,147,998 . - These patents describe how remote monitoring devices accurately and reliably measure performance metrics including point to point network latency, round-trip delay, throughput, and data delivery ratio. However, current network management tools cannot distinguish between chronic excessive problems and transient excessive problems. Nor can these tools predict the onset of chronic excessive problems. This deficiency degrades the ability of network managers to maintain high availability networks with minimal problems.
- From a service provider's standpoint, it would be desirable to detect the onset of a persistent problem that requires attention before the client complains about poor performance. Accordingly, there remains a need for the capability to identify chronic performance problems on data networks in an automated manner, while accurately distinguishing such chronic performance problems from transient problems.
- With respect to the prior art attention is drawn to
EP-A- 1 206 085 from which a method and apparatus is known for automated generation of a service level agreement delay value and monitoring of a network. Service level agreement delay values are generated by calculating a path delay and standard deviation for a path through the network. A confidence interval is determined for the path delay using the standard deviation. The service level agreement delay value is generated from the path delay and the confidence interval. The network is subsequently monitored in order to predict and track violations of the service level agreement delay value. The subsequent monitoring of the network is performed for both trunks and paths within the network. - The present invention relates to a method of identifying a chronic performance problem on a data network as defined in claim 1. Further, the present invention relates to a system for identifying chronic performance problems on a data network as defined in claim 9.
- Preferred embodiments of the invention are disclosed in the dependent claims.
- In accordance with the present invention, a system for identifying chronic performance problems on data networks includes remote network monitoring devices that provide measurements of performance metrics such as latency, jitter, and throughput, and a storage and processing system, such as a backend system, for collecting and storing data, monitoring and analyzing network performance, and notifying network operators when chronic performance problems are detected. The system provides accurate and reliable measurement of network latency, jitter and throughput; management of user performance requirements with configurable sensitivity; a context-sensitive data repository which enables the assessment and evaluation of network performance with respect to circuit, time, bandwidth and performance requirements; and predictions of chronic network performance problems by using statistical and heuristic analysis techniques.
- Chronic performance problems may be recognized via the following methodology. Network behavior is analyzed during an extended period where performance is considered normal or acceptable. This analysis is used to develop a baseline incorporating the average (mean) performance and the variability of the performance during normal operations. Criteria are then established for the network performance levels leading the user to perceive service levels degradations. Criteria are also established for the required confidence levels of chronic network performance problem predictions. These confidence levels are used to control the likelihood that detected problems will be deemed transient and not require specific corrective action. These criteria and confidence levels are established heuristically and can be dynamically adjusted.
- Continuous monitoring of ongoing network performance is then performed, comparing current performance with the baseline behavior, using the network performance criteria to filter out and identify network problems which are judged to be service-impacting and chronic. When a chronic network performance problem is identified, an alert and diagnosis are provided.
- Chronic excessive performance problems may be identified (and distinguished from transient problems) using the statistical "t-test" to estimate the likelihood that a detected degradation in network latency is persistent. Essentially, a t-test assesses whether the means of two groups are statistically different from each other with a certain level of confidence by comparing the means of samples from the two groups, taking into consideration the variance of the samples. In this case, when a transient reduction in performance occurs, the t-test is used to compare the mean of one or more performance metrics under baseline conditions to the mean of the performance metric(s) over a recent period of operation and identifies the performance problem as chronic if the t-test suggests the mean performance is statistically different from the baseline. At a given point in time, such a test may fail to detect a chronic performance problem or, conversely, falsely detect a chronic problem when one does not in fact exist. The selected confidence levels, which translate to a significant or "alpha" level within the t-test, control the likelihood of a false detection of a chronic problem.
- The above and still further objects, features and advantages of the present invention will become apparent upon consideration of the following definitions, descriptions and descriptive figures of specific embodiments thereof wherein like reference numerals in the various figures are utilized to designate like components. While these descriptions go into specific details of the invention, it should be understood that variations may and do exist and would be apparent to those skilled in the art based on the descriptions herein.
-
-
Fig. 1 is a functional block diagram of a data transmission system including monitoring devices located at different points in the system to measure data traffic performance metrics on a communication network, and a backend processing system. -
Fig. 2 is a functional flow chart illustrating operations performed to determine whether a chronic performance problem exists on a data network in accordance with an exemplary embodiment of the invention. -
Fig. 3 is a graph illustrating a scenario in which a modest rise in latency on a network with stable performance and a small latency variance results in a determination of chronic latency problem requiring attention. - The following detailed explanations of
Figs. 1-3 and of the preferred embodiments reveal the methods and apparatus of the present invention. A system for monitoring performance for data communication networks is illustrated inFig. 1 . Specifically, an exemplary data transmission system 10 includes two sites (A and B) and apacket switching network 12 to facilitate communications between the sites. Site A is connected to the network bycommunication lines communication lines - The data transmission system 10 can include conventional communications line types, for example, T3, OC-3, North American T1 (1.544 Mbits/second), CCITT (variable rate), 56K or 64K North American Digital Dataphone Service (DDS), Ethernet, and a variety of data communications connections, such as V.35, RS-449, EIA 530, X.21 and RS-232. Sites A and B are each capable of transmitting and receiving data packets in various protocols utilized by the communication lines, such as Asynchronous Transfer Mode (ATM), Frame Relay, High Level Data Link Control (HDLC) and X.25, IP, Ethernet, etc. Each
line communication lines communication lines - Generally, site A and site B utilize switching
network 12 to communicate with each other, wherein each site is connected to switchingnetwork 12 that provides paths between the sites. For illustrative purposes, only two sites (A and B) are shown inFig. 1 . However, it will be understood that the data communication system can include numerous sites, wherein each site is generally connected to multiple other sites over corresponding transmission circuits within the switching network. Likewise, monitoring devices may be positioned at various points throughout the system, including endpoints associated with particular sites (as shown inFig. 1 ) or at intermediate points in the switching network. - As used herein, the term "packet" (e.g., as used in "packetized switching network") does not imply any particular transmission protocol and can refer to units or segments of data in a system using, for example, any one or combination of the above-listed data transmission protocols (or other protocols). However, since the term "packet" is often associated with only certain data transmission protocols, to avoid any suggestion that the system of the present invention is limited to any particular data transmission protocols, the term "protocol data unit" (PDU) will be used herein to refer generically to the unit of data being transported by the communication network, including any discrete packaging of information. Thus, for example, a PDU can be carried on a frame in the frame relay protocol, a related set of cells in the ATM protocol, a packet in an IP protocol, etc.
- As shown in
Fig. 1 , monitoring devices A and B are respectively disposed between switchingnetwork 12 and sites A and B. Monitoring devices A and B can be located at sites A and B, at any point between switchingnetwork 12 and sites A and B, or at points within the switching network itself. The placement of the monitoring devices depends at least in part on the portion of the system or network over which a network service provider or other party wishes to monitor performance of data traffic flow. Typically, when service providers and customers enter into a service level agreement, the service provider will want any performance commitments to be limited to equipment or portions of the network over which it has control. The service provider does not want to be responsible for performance problems or degradation caused by end-site equipment or portions of the network not owed or managed by the service provider. On the other hand, a customer may desire to have monitoring devices relatively close to the actual destinations to assess overall end-to-end performance. Further, a customer or service provider may wish to have monitoring devices at the edges of the network and at intermediate points in the network to help pinpoint specific segments of the network or equipment causing a degradation in performance. More generally, monitoring devices can be placed at virtually any point in the network or any point within an enterprise LAN. - In general, the monitoring devices can be any type of network monitoring device or probe and can comprise standalone hardware/software devices or software and/or hardware added to network equipment such as PCs, routers, CSU/DSUs (channel service unit/data service unit), FRADS, voice switches, phones, etc. Software embedded in the monitoring devices can collect network performance data for detailed analysis and report generation relating to any of a variety of performance metrics. By way of non-limiting example, a monitoring device can be a CSU/DSU that operates both as standard CSU/DSU and as managed devices capable of monitoring and inserting network management traffic; an inline device residing between a DSU and router, which monitors network traffic and inserts network management traffic; or a passive monitoring device that monitors network traffic only. The monitoring devices can be "active" monitoring devices capable of inserting PDUs into the data traffic (e.g., test PDUs) or "passive" monitoring devices that observe PDU data traffic without generating or inserting any test PDUs into the data traffic for the purpose of measuring performance metrics. Optionally, passive monitoring devices can forward measurement data to each other via an out of band channel or through a link other than the monitored network channel, thereby permitting passive monitoring devices to directly coordinate testing periods.
- The monitoring devices can collect measurement data relating to any of a variety of performance metrics associated with operation of the network. In the examples provided herein, performance metrics such as latency, jitter, and throughput are described. However, it will be understood that the invention is not limited to the measurement or analysis of any particular performance metric or any particular combination of metrics, and the principles of the invention can be applied to virtually any type of measurements associated with performance or operation of a network.
- In general, latency relates to the amount of time required for signals or PDUs to traverse the network. Latency can be evaluated in terms of one-way delays between two points, or round-trip delays relating to the time required for an outbound signal to traverse the network from a first point to a second point plus the time required for a reply signal to traverse the network from the second point back to the first point. Such measurements can be made in any of a variety of ways, including active and passive measurement schemes, such as those described in
U.S. Patent Nos. 5,521,907 and6,058,102 . - Jitter relates to the variation in delay among a set of PDUs. The quality of real-time data transmissions depends on the network's ability to deliver data with minimal variation in 5 the packet delay. Typically, when PDUs of voice or video data are transmitted, a sequence of PDUs is sent to the network with fairly consistent time differences between successive PDUs, resulting in a relatively steady stream of PDUs. This stream of PDUs must essentially be reconstructed at the destination to accurately reproduce the audio or video signal. Due to conditions on the network, PDUs may experience different delays before arriving at a destination or may be dropped altogether and not reach the destination. PDUs arriving at the destination are buffered to compensate for some degree of delay variation. However, in real-time applications such as voice and video, the output signal must be generated from the data in the PDUs within a reasonable period of time to avoid perceptible delays in the output audio or video signal. Consequently, PDUs not received within a predetermined period of time are considered to be dropped, and the output signal is reconstructed without such PDUs to keep voice calls static free and video running smoothly. Excessive jitter will cause an unacceptable number of PDUs to be excluded from the reconstructed real-time output signal resulting in perceptible distortions in the audio or video output signal. Any of a variety of techniques can be used to measure jitter.
- Throughput relates to the amount of data traffic received during a period of time, yielding a rate of data traffic or PDUs. The measured data rate may relate to a particular circuit, channel, site, application, user(s), or device, for example. In the case of a shared resource, such as a virtual circuit or channel, the actual throughput rate can be compared to a committed information rate to determine how the level of usage relates to levels agreed upon by the service provider and customer (e.g., in a service level agreement). Centralized backup schemes involving transmission of volumes of data to backup sites (e.g., for daily, overnight system backups) are sensitive to loss of throughput, since reduced throughput levels prevent backups from being completed in a timely manner. Techniques such as those described in
U.S. Patent No. 5,867,483 can be used to measure throughput. Related throughput metrics, such as data delivery ratio can also be evaluated, as described for example inU.S. Patent No. 6,058,102 . - Referring again to
Fig. 1 , the network monitoring system further includes abackend processing system 30 which receives measurement data either directly or indirectly from the monitoring devices, and collects and stores measurement data and processes the data to produce the various displays and reports required to monitor performance of the network and its components. The architecture depicted inFig. 1 is a conceptual diagram illustrating major functional units and does not necessarily illustrate physical relationships or specific physical devices within the backend processing system or between the backend processing system and the monitoring devices. The configuration and components of the backend processing system can take many forms and are described herein only in general terms for context. Those skilled in the art will appreciate that the techniques described herein for identifying chronic performance problems can be implemented within any network monitoring system that collects data on performance metrics, regardless of the particular architecture of the system. - In general terms,
backend processing system 30 includes a data storage capability represented bystorage module 32 for storing measurement data as well as information generated by processing measurement data, such as aggregated report data, analysis results, and historical information.Processing system 30 further includes a management and processing capability represented inFig. 1 byprocessor module 34, which performs various processing tasks, such as performing operations on raw measurement data to produce reports and performing analysis operations, such as the chronic performance analysis described herein. The system further includes a display, interface, and report capability represented by display/report module 36, which displays performance information in a tabular or graphical manner via an interactive graphical user interface, for example, and preferably includes the capability to generate various performance reports. - The backend system may receive measurement data directly from the monitoring devices or may receive measurement data indirectly (e.g., the monitoring devices may supply measurement data to a storage device at the local site, which is subsequently supplied to the backend system. Thus, the links shown in
Fig. 1 between the monitoring devices and the backend processing system are merely indicative of the fact that measurements from the monitoring devices are ultimately supplied in some manner to a backend processing system. Further, the backend processing system may be located at a single site or may have components distributed throughout the network at multiple locations. For example,storage module 32 may constitute storage capabilities at a number of local sites as well as a storage capability at one or more backend processing sites. Likewise, various backend processing tasks, nominally represented byprocessor 34 inFig. 1 , may be performed by a number of different processors that carry out specific task and that may be distributed throughout the network. Similarly, the display/reporting capability may allow access to performance information via interfaces at a number of sites or via a web-based interface accessible to authorized customers or service provider personnel. - The processing required to identify chronic performance problems according to the invention can be carried out in a backend processing system such that shown in
Fig. 1 , although optionally at least some of the operations may be performed directly within the monitoring devices. The performance analysis scheme described herein can be incorporated into an auto-troubleshooting management system that enables system operators to identify and react more quickly to problems within a data network. -
Fig. 2 is a functional flow diagram of the operations performed to identify chronic performance problems on a data network in accordance with an exemplary embodiment of the invention. In afirst operation 200, baseline operating conditions are determined during normal operation, i.e., a period of time in which behavior of the monitored network or portion of the network is considered acceptable. Typically, when a new network or portion of a network (e.g., a new site, channel, circuit, link, etc.) is initially switched on, adjustments are required to achieve normal, acceptable operation. Once such conditions are established, and the variation in the performance metrics of interest is relatively small, the baseline conditions can be determined. - During the baseline data collection period, one or more monitoring devices periodically collect performance metric measurements on one or more performance metrics of interest. For example, a monitoring device may make instantaneous measurements of PDU latency, throughput, and jitter at periodic increments of time (e.g., every second) over a measurement period (e.g., every fifteen minutes). The invention is not limited to any particular performance metrics, and other performance metrics could be evaluated, including but not limited to: data delivery ratio; and joint or composite performance metrics constructed from combinations of latency, jitter, and throughput, etc.
- At the end of the measurement period, a measurement value is computed for the performance metric(s) based on the instantaneous measurements (e.g., by computing the average value). Thus, each measurement period results in a single measurement value of the performance metric(s). In the example of a fifteen minute measurement interval, one performance metric measurement is generated every fifteen minutes. The invention is not limited to measurement periods of any particular duration or any particular number of instantaneous measurements made during a measurement period. Thus, for example, a measurement value could be taken and analysis performed every second, minute, hour, day, etc., or at irregular intervals (e.g., upon the occurrence of a particular event). Likewise, a measurement value can be a "raw" measurement or determined from a set of measurements (e.g., by some computation or aggregation).
- Determining baseline conditions involves computing the mean and variance of a set of performance metric measurements. To develop a good, accurate baseline, the sample size (i.e., the number of measurements) used to compute the mean and variance is preferably a substantial number. That is, the longer the baseline data collection period, the greater the confidence that the baseline statistics are accurate. By way of a non-limiting example, the baseline data collection period can be a week long which, in the case of a fifteen minute measurement period, would result in a sample size of 672 measurements. The invention is not limited to any particular baseline sample size or a baseline data collection period of any particular length.
-
- The mean, variance, and sample size of the baseline sample are used to perform statistical testing on samples taken during ongoing operation of the network, as described in greater detail below.
- While it is preferable to develop baseline statistics from baseline conditions on the actual network or portion of the network that will later be monitored, as described above, optionally, baseline conditions could be derived from other sources. For example, if the expected behavior of performance metrics to be monitored can be accurately modeled (e.g., the mean and variance of the performance metric under acceptable operating conditions can be reliably predicted with a certain degree of accuracy), then expected baseline conditions could be substituted for actual, measured baseline conditions.
- Referring again to
Fig. 2 , inoperation 210, thresholds to be used in testing of performance metrics to identify chronic problems are set. Essentially, two types of thresholds are set. The first type of threshold is a service performance threshold that relates to the current value of the performance metric measurement relative to the baseline value or some absolute value. One of the criteria for identifying a chronic performance problem is that the current value of the performance metric measurement (or a mean of recent values) exceeds the service performance threshold (this is a necessary but not sufficient condition to declare a chronic problem). - The service performance threshold can be selected in any number of ways. The threshold can be an absolute increase relative to the baseline mean of the performance metric. For example, for latency, the threshold could be set at a fixed value of 40 ms greater than the baseline mean latency. The service performance threshold could also be set as an absolute limit, irrespective of the baseline mean value of the performance metric. For example, the latency threshold could be set at a predetermined value of 100 ms. The service performance threshold could also be set as a relative increase with respect to the baseline mean value of the performance metric, for example, a 30% increase. Another option is to set a threshold that is a certain number of standard deviations away from a baseline operating mean value. Any one or combination of these criteria could be used to develop the service performance threshold. For example, a combination of an absolute limit and a relative increase may produce a threshold better suited to real-world conditions. Consider an example where the baseline mean latency is 50 ms. Using only the criterion of a relative increase (e.g., 30%), the service performance threshold would be set at 65 ms. However, as a practical matter, it is also known that any customer line having a latency below 70 ms is considered to be operating acceptably; thus, the absolute threshold should not be set below this level. Using a combination of criteria, the threshold could be set to a relative increase over the baseline mean latency or at an absolute value (e.g., 70 ms), which ever is greater. Naturally, such thresholds may be context specific. For example, a 70 ms latency may be acceptable for local service, whereas a latency of 120 ms may be more suitable for national service.
- In addition to setting the service performance threshold, a confidence level threshold associated with statistical testing of the performance metric data is selected. As described in greater detail below, merely exceeding the service performance threshold is not sufficient to conclude that a chronic performance problem has developed. Essentially, the statistical testing attempts to identify a pattern of performance behavior that suggests the mean of performance metric measurements under ongoing operating conditions is significantly different from the baseline mean, thereby indicating that a chronic performance problem is developing. The confidence level threshold corresponds to a level of reliability of the statistical testing. A high confidence level threshold reduces the number of "false positives" from the statistical testing, such that there is high confidence that a chronic problem does indeed exist when the statistical testing indicates that the performance metric mean has changed from the baseline conditions. On the other hand, a high confidence level threshold increases the likelihood that a chronic problem may go undetected for a longer period of time. A lower confidence level threshold will result in more false alarms, but may detect a chronic problem more quickly.
- Thus, the confidence level threshold determines how sensitive the system is to indicating there is a chronic problem, and selection of an appropriate confidence level threshold may depend on several consideration. For example, the confidence level threshold should be set to reflect how sensitive the particular customer or service provider is to being disturbed by a false alarm or by the onset of a chronic performance problem. This, in turn, may relate to the type of operation being performed or data being transmitted, terms of a service level agreement between the customer and service provider, or the particular preferences of the customer. If the baseline conditions are more noisy (a relatively large baseline variance), a relatively high confidence level may be appropriate to avoid frequent false alarms.
- The confidence level threshold can be a configurable parameter that can be adjusted if necessary based on experience or past performance, such that the confidence level is heuristically determined (i.e., adjusted in response to past experience) either by human experience or by a software algorithm. Optionally, the confidence level threshold can be initially set to a default sensitivity level and then subsequently adjusted by the customer or service provider via a graphical user interface. For example, a drop-down menu can be used to let a customer select one of several confidence levels (high, moderate, low). Another option is a slider bar, which allows the confidence level threshold to be set to a number of values on a sliding scale by moving an indicator on the slider. The selected confidence level threshold internally corresponds to an α value used in the statistical t-testing, as described in greater detail below.
- Once a baseline has been established and thresholds set, the network or portion of the network of interest is monitored by collecting measurements of one or more performance metrics (e.g., latency, throughput, jitter, etc.) from various monitoring devices, and these measurements are supplied to a processing system for analysis. Referring again to
Fig. 2 , there is shown an exemplary embodiment of processing performed during ongoing operation of the network to analyze whether a chronic performance problem exists. Inoperation 220, for each performance metric of interest, the mean and variance of recent measurements of the performance metric are periodically computed. Preferably, the computation is repeated each time a new measurement is available so that testing for chronic performance problems can be performed with each new measurement, thereby enhancing the likelihood of early detection of a chronic problem. - Essentially, the set of measurements within a certain time window constitutes an operating sample (a sample of measurements taken during operation of the network) that is used to compute the mean and variance during network operation. The time window extends over a period of time that covers the set of measurements, terminating with the most recent measurement of the performance metric. The time window can be of a predetermined duration or can cover a predetermined number of measurements. Optionally, the time window can be adjustable, variable, and/or user selectable. The number of measurements in the time window is the sample size n2 of the operating sample. In general, as each new measurement becomes available, the time window slides forward and drops the oldest measurement from the operating sample used to compute the operating mean and variance, such that the operating sample slides forward in time.
-
- A number of factors should be considered in selecting the sample size n2 of the operating sample. Increasing the sample size tends to produce a more stable result, making the analysis less prone to a false indication of a chronic performance problem. However, a larger sample size requires a longer time window, which may unduly increase the time required to detect a chronic performance problem, since a significant number of older measurements within the time window may reflect acceptable operation prior to the onset of a problem, thereby diluting the impact of more recent measurements that suggest a problem. Conversely, if the time window is too short (e.g., only a few measurements in the sample), there is an increased risk that a transient problem could cover a substantial portion of the window, thereby triggering false detections of chronic problems. In this respect, setting the sample size is analogous to setting the gain of a filter, with a smaller sample size corresponding to a higher gain (which is highly responsive to recent input but more prone to "overreact" under certain conditions), and a larger sample size corresponding to a lower gain (which produces a smoother response but may not react quickly enough under certain conditions).
- Returning to the earlier example where a new measurement of the performance metric becomes available every fifteen minutes, a sample size of 96 measurements would correspond to a full day's worth of measurements. A time window of this duration may be too long to timely detect the onset of a chronic performance problem in some circumstances, and a shorter time window of a few hours may be more suitable in such circumstances (e.g., a window of 4 or 5 hours corresponds to 16 or 20 measurements). If performance metric measurements are available more frequently, somewhat larger sample sizes can be used without unduly delaying detection of chronic performance problems.
- Referring once again to
Fig. 2 , in the exemplary embodiment, the testing for detecting a chronic performance problem essentially involves two tests: a threshold test of the value of the performance metric; and a statistical test that indicates with some degree of confidence that the mean value of the performance metric has changed from the baseline value. Inoperation 230, a threshold test is conducted to determine whether the current value of the performance metric of interest exceeds the service performance threshold selected in the manner previously described. If the service performance threshold is not exceeded, performance is considered to be at an acceptable level (i.e., there is no performance problem detected, chronic or otherwise), and statistical testing is unnecessary to determine whether a performance problem is chronic. Accordingly, it is assumed that no chronic performance problem exists, and the testing ends until receipt of the next performance metric measurement. - If the current value of the performance metric of interest exceeds the service performance threshold, in
operation 240, a t-test score is computed for the operating sample based on the baseline data and the measured performance data. In the context of the invention, the t-test essentially allows one to compare two sets of data (i.e., the baseline sample and the sample taken during ongoing network operation) and either "reject" or "fail to reject," with a certain degree of confidence, the hypothesis that the current operating conditions remain the same as the baseline conditions. That is, the t-test indicates whether the difference between the mean of the baseline sample and the mean of the operating sample is statistically significant to the point where it can be stated with a specified degree of confidence that the difference arose from an actual change in the performance metric being measured rather than by random chance. More specifically, the baseline (first) random sample (X1 , X2 ,..., Xn1 ) of size n1 whose sample mean is denote byX (given by equation 1) and whose variance is denoted by s1 2 (given by equation 2) is compared to an operating (second) random sample (Y1 , Y2, ..., Yn2 ) of size n2 whose mean is denote byY (given by equation 3) and whose variance is denoted by s2 2 (given by equation 4). -
-
- The computation for the value t may vary somewhat based on statistical assumptions, and the invention is not limited to a particular computation for the value t.
- Essentially, what is being tested is that the actual, true mean µ 1 of the performance metric during the baseline conditions (which is unknown) is the same as the actual mean µ 2 of the performance metric during the period of operating in which the operating sample was taken (which is also unknown). The statistical hypothesis is that the means are the same. In
operation 250, if the computed value of t is less than a certain threshold tα, then the hypothesis is "not rejected," and it is assumed that the difference between the means of the baseline and operating samples is statistically insignificant and does not indicate a meaningful change in the performance metric from the baseline condition. Consequently, it is concluded that performance problem is not a chronic problem, and the testing ends until receipt of the next performance metric measurement. On the other hand, if the computed value of t exceeds the threshold tα, then the hypothesis is rejected, and it is assumed that the difference between the means of the baseline and operating samples is statistically significant and indicates a meaningful change in the performance metric from the baseline condition. Consequently, it is concluded that a chronic performance problem exists. - The value of tα is a function of the degrees of freedom, which is equal to n1 + n2 - 2, and of the confidence level threshold selected in the manner described above. The confidence level threshold corresponds to t-test level of confidence (1- α). Essentially, the value of tα will result in a false alarm that there is a problem (1 - α)*100 percent of the time.
- With an ideal or statistically rigorous t-test, the value of tα can be determined from the standard t distribution, based on the degrees of freedom and the value of α (e.g., such values are often provided in statistical tables). In practice, some of the assumptions associated with the t-test may not be met by the baseline and operating samples (e.g., the samples may not be normally distributed, the true variances may not be equal, the samples may not be random, the two samples may not be independent of each other, etc.). For this reason, the selected value of α (and hence tα) may not yield the precise rate of false alarms ideally expected. Nevertheless, even if these assumptions are not fully met, the t-test still provides a good indication of whether a meaningful change in performance has occurred relative to the baseline. A heuristic approach can be used to obtain the desired rate of false alarms, wherein after empirically observing the rate of false alarms during actual operation using an ideal value for tα, the value of tα can be adjusted to a suitable value to obtain the desired false alarm rate.
- The analysis performed according to the present invention involves forecasting in the sense that detection of a chronic problem amounts to an assertion that a problem is not transient; the problem is going to persist in the future. Thus, historical data is used to forecast that a problem will be ongoing. The t-test is well-suited to providing such a forecast, since the t-test, in effect, yields a reliable, objective indication, with a controllable level of confidence, of whether observed changes in performance are likely the result of a change of conditions rather than the result of chance. In effect, the t-test gives a measure consistency or reliability. When the t-test value exceeds the significance threshold tα, it suggests that a performance metric value that is over a service performance threshold is likely to stay above the threshold.
- As will be appreciated from the foregoing description and from
Fig. 2 , two things must occur for a chronic problem to be detected: both the service performance threshold must be exceeded, indicating that some type of performance problem exist, and the t-test must indicate that the change in the mean value of the performance metric is significantly different from the baseline value. This combination of criteria filters out the circumstances that should not trigger detection of a chronic performance problem. - In one case, the value of the performance metric may be above the service performance threshold, but only for a short time, meaning the problem could be sporadic or transient. This case will typically fail the confidence test provided by the t-test, effectively indicating that there is not sufficient confidence that the problem is going to persist. In another case, the t-test could theoretically indicate a significant statistical deviation from baseline; however, if the service performance threshold has not been exceeded by the current value of the performance metric, the performance is still considered acceptable, so there is no concern that the customer is experiencing unacceptable performance. A problem is reported only if the performance is poor with some persistence.
- The invention is not limited to the specific testing configuration shown in
Fig. 2 , and a number of variations are possible. For example,Fig. 2 suggests that the t-test is performed only if the service performance threshold is exceeded. Optionally, both the service performance threshold test and the t-test could be performed (performing of the t-test is not dependent on the outcome of the service performance threshold test), and the decision of whether a chronic performance problem exists is determined after the outcomes of the two tests are known (as opposed to the sequential logic shown inFig. 2 ). The result of this approach is identical to the approach shown inFig. 2 , but would generally require more processing. - According to another approach, the sequence of the tests shown in
Fig. 2 could be reversed, with the t-test being performed first and then the service performance threshold test being performed only if the t-test indicates a significant change of conditions. - In
Fig. 2 , the mean and standard deviation of the operating sample are computed inoperation 220. In the sequence of operations shown inFig. 2 , optionally, computation of the mean and standard deviation of the operating sample could be deferred until the occurrence of the current value of the performance metric exceeding the service performance threshold, since the mean and standard deviation are required only if the t-test is actually performed. - Another option is to perform the service performance threshold testing by comparing the mean of the operating sample to the threshold rather than just the most recent (current) measurement of the performance metric (this would require the mean and standard deviation to be computed with each measurement, as shown in
Fig. 2 ). However, using only the current value of the performance metric in the threshold test may have the advantage of permitting a more rapid detection of chronic problems, since the service performance threshold test fails every time a measurement exceeds the service performance threshold. - Optionally, the system can include a feature that allows historical data to be re-run through the chronic problem testing sequence using different thresholds to see if data that did not adequately trigger an alarm at a high threshold would have identified a chronic problem using a lower threshold (or, conversely, if a higher threshold would have avoided a false alarm). This feature assists in pro-actively adjusting thresholds.
- Once the testing for chronic excessive service problems is completed, the test results can be supplied to a management system for inclusion in graphical displays of network performance and for report generation. In the event that a chronic performance problem is identified, the system can trigger an alarm or provide notice to an administrator of the condition. An alarm could be, for example, in the form of a visual indicator, an audible indicator or both.
-
Fig. 3 illustrates a scenario in which a modest rise in latency on a network with stable performance and low variance results in a detection of a chronic performance problem. The higher latency measurements shown with cross-hatching, will quickly result in the service performance threshold indicating a performance problem. Assuming the baseline conditions exhibited the latency stability (low variance) exhibited by the measurements shown on the left side ofFig. 3 , after a relatively small number of higher-latency measurements, the t-test will identify the latency performance problem as chronic and set an alarm. InFig. 3 , by 6:00 am, corrective measures have been taken and the performance returns to normal. Subsequent slight increases in latency at 10:00 am and 2:00 are not identified as chronic problems. - While the invention has been described primarily in the context of "network performance," the techniques of the invention can be applied at various levels in the network architecture including specific applications and types or groups of applications. Thus, in this context, the invention is capable of monitoring "application performance" and can identify chronic performance problems with certain applications. One example of a suitable performance metric for applications would be TCP retries.
- It will be appreciated that the embodiments described above and illustrated in the drawings represent only a few of the many ways of utilizing the principles of the present invention to identify chronic performance problems in a communication network. The invention is not limited to any particular context and applies equally to all types of data and applications.
- The principles of the present invention may be applied not only to packetized communications networks (e.g. Frame Relay, SMDS, ATM, IP, etc.), but also to any communications network wherein performance metrics of transmitted data can be reliably measured. Thus the principles of the present invention could be applied, for example, to measure performance metrics such as latency, throughput, or jitter in a non-packetized leased-line network. In this respect, as used herein, the term PDU encompasses virtually any identifiable portion of a data stream from which the same identifier can be generated at two points in a network.
- Although the preferred embodiment discloses a particular functional representation of the monitoring devices, any data gathering devices capable of capturing and recording performance metrics can be used according to the principles of the present invention.
- Having described preferred embodiments of new and improved methods and apparatus for identifying chronic performance problems on data networks, it is believed that other modifications, variations and changes will be suggested to those skilled in the art in view of the teachings set forth herein. It is therefore to be understood that all such variations, modifications and changes are believed to fall within the scope of the present invention as defined by the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
Claims (19)
- A method of identifying a performance problem on a data network, comprising:establishing a baseline performance level of at least one performance metric;determining an operating performance level of the at least one performance metric during operation of the network; andidentifying a performance problem in response to the operating performance
level exceeding a performance threshold characterized by identifying the performance problem as being chronic by a determination that a difference between the baseline performance level and the operating performance level is statistically significant, wherein establishing a baseline performance level includes:collecting a baseline sample comprising a plurality of measurements of the at least one performance metric taken over a period of time; andcomputing a baseline mean and variance of the baseline sample, the baseline mean and variance being indicative of the baseline performance level,
wherein determining an operating performance level
includes:collecting an operating sample comprising a plurality of measurements of the at least one performance metric taken over a period of time; andcomputing an operating mean and variance of the operating sample, the operating mean and variance being indicative of the operating performance level,
and wherein the difference between the baseline performance level and the operating performance level is determined to be statistically significant by performing a t-test based on the baseline mean and variance and the operating mean and variance - The method of claim 1, wherein the determination that a difference between the baseline performance level and the operating performance level is statistically significant comprises statistical testing using a confidence level threshold.
- The method of claim 2, wherein the confidence level threshold corresponds to a level of reliability of the statistical testing.
- The method of any of claims 1 to 3, further comprising:selecting a confidence level threshold associated with performing the t-test.
- The method of any of claims 1 to 4, wherein the performance threshold is selected as a function of the baseline performance level.
- The method of any of claims 1 to 5, wherein the performance threshold is selected as a function of a predetermined performance level representing unacceptable performance.
- The method of any of claims 1 to 6, wherein the at least one performance metric includes at least one of: latency, jitter, throughput, data delivery ratio, and TCP retries.
- The method of any of claims 1 to 7, wherein the performance metric is associated with at least one of: a circuit, a channel, a link, a site, an application, a user, and a device.
- A system for identifying performance problems on a data network (12), comprising:means for measuring at least one performance metric at selected points in the network; andmeans for determining a baseline performance level of the at least one performance metric;means for determining an operating performance level of the at least one performance metric during operation of the network; andmeans for identifying a performance problem in response to the operating performance level exceeding a performance threshold characterized in that the means for identifying the performance problem identity the performance problem as being chronic by a determination that a difference between the baseline performance level and the operating performance level is statistically significant,
wherein establishing a baseline performance level includes:collecting a baseline sample comprising a plurality of measurements of the at least one performance metric taken over a period of time; andcomputing a baseline mean and variance of the baseline sample, the baseline mean and variance being indicative of the baseline performance level,wherein determining an operating performance level includes:collecting an operating sample comprising a plurality of measurements of the at least one performance metric taken over a period of time; andcomputing an operating mean and variance of the operating sample, the operating mean and variance being indicative of the operating performance level,and wherein the difference between the baseline performance level and the operating performance level is determined to be statistically significant by performing a t-test based on the baseline mean and variance and the operating mean and variance» - The system of claim 9, wherein the determination that a difference between the baseline performance level and the operating performance level is statistically significant comprises statistical testing using a confidence level threshold.
- The system of claim 10, wherein the confidence level threshold corresponds to a level of reliability of the statistical testing.
- The system of any of claims 9 to 11, wherein the means for measuring comprises
a plurality of monitoring devices
; and
wherein the means for determining a base line performance level, the means for determining an operating permance level and the means for identifying a performance problem are a processor (34) configured to receive performance metric measurement from the monitoring devices, wherein the processor determines a baseline performance level of the at least one performance metric, determines an operating performance level of the at least one performance metric during operation of the network, and identifies a chronic performance problem in response to the operating performance level exceeding a performance threshold and determining that a difference between the baseline performance level and the Operating performance level is statistically significant. - The system of claim 12, wherein the processor (34) receives from the plurality of monitoring devices the baseline sample comprising a plurality of measurements of the at least one performance metric taken over a period of time, and computes the baseline mean and variance of the baseline sample.
- The system of claim 12 or 13, wherein the processor (34) receives from the plurality of monitoring devices the operating sample comprising a plurality of measurements of the at least one performance metric taken over a period of time, and computes the operating mean and variance of the operating sample.
- The system of any of claims 12 to 14, wherein the processor (34) performs the t-test based on a selectable confidence level threshold.
- The system of any of claims 12 to 15, wherein the processor (34) selects the performance threshold as a function of the baseline performance level.
- The system of any of claims 12 to 16, wherein the processor (34) selects the performance threshold as a function of a predetermined performance level representing unacceptable performance.
- The system of any of claims 12 to 17, wherein the at least one performance metric includes at least one of: latency, jitter, throughput, data delivery ratio, and TCP retries.
- The system of any of claims 12 to 18, wherein the performance metric is associated with at least one of: a circuit, a channel, a link, a site, an application, a user, and a device.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US64958305P | 2005-02-04 | 2005-02-04 | |
US11/217,477 US7599308B2 (en) | 2005-02-04 | 2005-09-02 | Methods and apparatus for identifying chronic performance problems on data networks |
Publications (2)
Publication Number | Publication Date |
---|---|
EP1689121A1 EP1689121A1 (en) | 2006-08-09 |
EP1689121B1 true EP1689121B1 (en) | 2011-01-12 |
Family
ID=35911225
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP06002307A Active EP1689121B1 (en) | 2005-02-04 | 2006-02-03 | Methods and apparatus for identifying chronic performance problems on data networks |
Country Status (5)
Country | Link |
---|---|
US (1) | US7599308B2 (en) |
EP (1) | EP1689121B1 (en) |
AT (1) | ATE495604T1 (en) |
CA (1) | CA2535198C (en) |
DE (1) | DE602006019510D1 (en) |
Families Citing this family (107)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7580837B2 (en) | 2004-08-12 | 2009-08-25 | At&T Intellectual Property I, L.P. | System and method for targeted tuning module of a speech recognition system |
US7242751B2 (en) | 2004-12-06 | 2007-07-10 | Sbc Knowledge Ventures, L.P. | System and method for speech recognition-enabled automatic call routing |
US7751551B2 (en) | 2005-01-10 | 2010-07-06 | At&T Intellectual Property I, L.P. | System and method for speech-enabled call routing |
US8589508B2 (en) * | 2005-04-07 | 2013-11-19 | Opanga Networks, Inc. | System and method for flow control in an adaptive file delivery system |
US11258531B2 (en) | 2005-04-07 | 2022-02-22 | Opanga Networks, Inc. | System and method for peak flow detection in a communication network |
US8909807B2 (en) | 2005-04-07 | 2014-12-09 | Opanga Networks, Inc. | System and method for progressive download using surplus network capacity |
US7500010B2 (en) * | 2005-04-07 | 2009-03-03 | Jeffrey Paul Harrang | Adaptive file delivery system and method |
US9065595B2 (en) | 2005-04-07 | 2015-06-23 | Opanga Networks, Inc. | System and method for peak flow detection in a communication network |
US8719399B2 (en) * | 2005-04-07 | 2014-05-06 | Opanga Networks, Inc. | Adaptive file delivery with link profiling system and method |
US7657020B2 (en) | 2005-06-03 | 2010-02-02 | At&T Intellectual Property I, Lp | Call routing system and method of using the same |
US8175253B2 (en) * | 2005-07-07 | 2012-05-08 | At&T Intellectual Property I, L.P. | System and method for automated performance monitoring for a call servicing system |
US20140173102A1 (en) * | 2012-12-07 | 2014-06-19 | Cpacket Networks Inc. | Apparatus, System, and Method for Enhanced Reporting and Processing of Network Data |
US20070150949A1 (en) * | 2005-12-28 | 2007-06-28 | At&T Corp. | Anomaly detection methods for a computer network |
US7808916B1 (en) * | 2005-12-28 | 2010-10-05 | At&T Intellectual Property Ii, L.P. | Anomaly detection systems for a computer network |
JP4583312B2 (en) * | 2006-01-30 | 2010-11-17 | 富士通株式会社 | Communication status determination method, communication status determination system, and determination device |
US9110934B2 (en) * | 2006-06-02 | 2015-08-18 | International Business Machines Corporation | System and method for delivering an integrated server administration platform |
US7830814B1 (en) | 2006-06-02 | 2010-11-09 | Adobe Systems Incorporated | Providing information associated with network latency |
US20070282645A1 (en) * | 2006-06-05 | 2007-12-06 | Aaron Baeten Brown | Method and apparatus for quantifying complexity of information |
US8468042B2 (en) | 2006-06-05 | 2013-06-18 | International Business Machines Corporation | Method and apparatus for discovering and utilizing atomic services for service delivery |
US8289965B2 (en) | 2006-10-19 | 2012-10-16 | Embarq Holdings Company, Llc | System and method for establishing a communications session with an end-user based on the state of a network connection |
US7765294B2 (en) | 2006-06-30 | 2010-07-27 | Embarq Holdings Company, Llc | System and method for managing subscriber usage of a communications network |
US8488447B2 (en) | 2006-06-30 | 2013-07-16 | Centurylink Intellectual Property Llc | System and method for adjusting code speed in a transmission path during call set-up due to reduced transmission performance |
US9094257B2 (en) | 2006-06-30 | 2015-07-28 | Centurylink Intellectual Property Llc | System and method for selecting a content delivery network |
US8537695B2 (en) | 2006-08-22 | 2013-09-17 | Centurylink Intellectual Property Llc | System and method for establishing a call being received by a trunk on a packet network |
US8576722B2 (en) * | 2006-08-22 | 2013-11-05 | Centurylink Intellectual Property Llc | System and method for modifying connectivity fault management packets |
US8531954B2 (en) | 2006-08-22 | 2013-09-10 | Centurylink Intellectual Property Llc | System and method for handling reservation requests with a connection admission control engine |
US8144587B2 (en) | 2006-08-22 | 2012-03-27 | Embarq Holdings Company, Llc | System and method for load balancing network resources using a connection admission control engine |
US8274905B2 (en) | 2006-08-22 | 2012-09-25 | Embarq Holdings Company, Llc | System and method for displaying a graph representative of network performance over a time period |
US8064391B2 (en) | 2006-08-22 | 2011-11-22 | Embarq Holdings Company, Llc | System and method for monitoring and optimizing network performance to a wireless device |
US9479341B2 (en) | 2006-08-22 | 2016-10-25 | Centurylink Intellectual Property Llc | System and method for initiating diagnostics on a packet network node |
US8224255B2 (en) | 2006-08-22 | 2012-07-17 | Embarq Holdings Company, Llc | System and method for managing radio frequency windows |
US8130793B2 (en) | 2006-08-22 | 2012-03-06 | Embarq Holdings Company, Llc | System and method for enabling reciprocal billing for different types of communications over a packet network |
US8619600B2 (en) | 2006-08-22 | 2013-12-31 | Centurylink Intellectual Property Llc | System and method for establishing calls over a call path having best path metrics |
US9066348B2 (en) * | 2006-10-17 | 2015-06-23 | Intel Coporation | Device, system, and method for fast network entry for wireless devices in idle mode |
US20080112399A1 (en) * | 2006-11-13 | 2008-05-15 | British Telecommunications Public Limited Company | Telecommunications system |
EP2109940A4 (en) * | 2007-01-16 | 2013-10-09 | Opanga Networks Inc | Wireless data delivery management system and method |
US20080170508A1 (en) * | 2007-01-17 | 2008-07-17 | Abb Technology Ag | Channel integrity metric calculation |
US20080181102A1 (en) * | 2007-01-25 | 2008-07-31 | Verizon Services Organization Inc. | Network routing |
US20080188191A1 (en) * | 2007-02-06 | 2008-08-07 | British Telecommunications Public Limited Company | Network monitoring system |
EP1983477A1 (en) * | 2007-04-16 | 2008-10-22 | Hewlett-Packard Development Company, L.P. | Apparatus and method for processing management information |
US8204986B2 (en) * | 2007-07-27 | 2012-06-19 | Vmware, Inc. | Multi-hierarchy latency measurement in data centers |
US8326660B2 (en) * | 2008-01-07 | 2012-12-04 | International Business Machines Corporation | Automated derivation of response time service level objectives |
US8165862B2 (en) * | 2008-02-05 | 2012-04-24 | The Boeing Company | Methods and systems for predicting application performance |
US8385207B2 (en) * | 2008-05-27 | 2013-02-26 | International Business Machines Corporation | Method and apparatus for end-to-end network congestion management |
WO2010017205A2 (en) * | 2008-08-04 | 2010-02-11 | Jeffrey Harrang | Systems and methods for video bookmarking |
US8121816B2 (en) * | 2008-08-13 | 2012-02-21 | Verizon Patent And Licensing Inc. | Identifying causes of service level agreement performance violations |
WO2010033750A2 (en) * | 2008-09-18 | 2010-03-25 | Jeffrey Harrang | Systems and methods for automatic detection and coordinated delivery of burdensome media content |
CN102197624B (en) * | 2008-09-19 | 2016-10-12 | 交互数字专利控股公司 | Certification for secure wireless communication |
US9058259B2 (en) * | 2008-09-30 | 2015-06-16 | Vmware, Inc. | System and method for dynamic problem determination using aggregate anomaly analysis |
JP2012508424A (en) * | 2008-11-07 | 2012-04-05 | オパンガ ネットワークス インコーポレイテッド | Portable data storage device using host device to start data transfer |
US20100131385A1 (en) * | 2008-11-25 | 2010-05-27 | Opanga Networks, Llc | Systems and methods for distribution of digital media content utilizing viral marketing over social networks |
US8630193B2 (en) | 2009-06-11 | 2014-01-14 | Centurylink Intellectual Property Llc | System and method for indicating performance information to a user to enhance communications |
US20100153543A1 (en) * | 2008-12-17 | 2010-06-17 | Bo Lee | Method and System for Intelligent Management of Performance Measurements In Communication Networks |
US8205116B2 (en) * | 2009-02-18 | 2012-06-19 | At&T Intellectual Property I, L.P. | Common chronics resolution management |
US9210050B2 (en) * | 2009-07-09 | 2015-12-08 | Centurylink Intellectual Property Llc | System and method for a testing vector and associated performance map |
US8073946B1 (en) * | 2009-08-17 | 2011-12-06 | Google Inc. | Monitoring of metrics to identify abnormalities in a large scale distributed computing environment |
US8886790B2 (en) * | 2009-08-19 | 2014-11-11 | Opanga Networks, Inc. | Systems and methods for optimizing channel resources by coordinating data transfers based on data type and traffic |
EP2468067A4 (en) | 2009-08-19 | 2015-09-09 | Opanga Networks Inc | Optimizing media content delivery based on user equipment determined resource metrics |
US7978711B2 (en) * | 2009-08-20 | 2011-07-12 | Opanga Networks, Inc. | Systems and methods for broadcasting content using surplus network capacity |
US20130051244A1 (en) * | 2010-01-08 | 2013-02-28 | Nokia Siemens Networks Oy | Network optimisation |
US8495196B2 (en) | 2010-03-22 | 2013-07-23 | Opanga Networks, Inc. | Systems and methods for aligning media content delivery sessions with historical network usage |
US8774023B2 (en) * | 2010-09-22 | 2014-07-08 | At&T Intellectual Property I, Lp | Method and system for detecting changes in network performance |
US8705342B2 (en) * | 2010-10-15 | 2014-04-22 | Brookhaven Science Associates, Llc | Co-scheduling of network resource provisioning and host-to-host bandwidth reservation on high-performance network and storage systems |
US9374290B2 (en) * | 2010-12-13 | 2016-06-21 | Verizon Patent And Licensing Inc. | System and method for providing TCP performance testing |
JP5538652B2 (en) * | 2010-12-20 | 2014-07-02 | 三菱電機株式会社 | Network status monitoring method |
US8645530B2 (en) * | 2011-02-22 | 2014-02-04 | Kaseya International Limited | Method and apparatus of establishing computer network monitoring criteria |
US20120317266A1 (en) * | 2011-06-07 | 2012-12-13 | Research In Motion Limited | Application Ratings Based On Performance Metrics |
JP5782913B2 (en) * | 2011-08-18 | 2015-09-24 | 富士通株式会社 | Monitoring device, monitoring method, and monitoring program |
US9292408B2 (en) | 2011-09-21 | 2016-03-22 | Hewlett Packard Enterprise Development Lp | Automated detection of a system anomaly |
WO2013152787A1 (en) * | 2012-04-11 | 2013-10-17 | Nokia Siemens Networks Oy | Method and apparatus |
US9779260B1 (en) | 2012-06-11 | 2017-10-03 | Dell Software Inc. | Aggregation and classification of secure data |
US8832267B2 (en) * | 2012-08-07 | 2014-09-09 | Ca, Inc. | System and method for adaptive baseline calculation |
US8917596B2 (en) * | 2012-09-07 | 2014-12-23 | Accedian Networks Inc. | Automatic discovery and enforcement of service level agreement settings |
US8867545B2 (en) | 2012-09-25 | 2014-10-21 | Accedian Networks Inc. | Modified Ethernet preamble for inter line card communications in a modular communication chassis |
US9444674B2 (en) * | 2012-10-02 | 2016-09-13 | Microsoft Technology Licensing, Llc | Heuristic analysis of responses to user requests |
US10237144B2 (en) * | 2012-10-29 | 2019-03-19 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10952091B2 (en) | 2012-10-29 | 2021-03-16 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US9538409B2 (en) | 2012-10-29 | 2017-01-03 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10313905B2 (en) | 2012-10-29 | 2019-06-04 | T-Mobile Usa, Inc. | Contextual quality of user experience analysis using equipment dynamics |
US10412550B2 (en) | 2012-10-29 | 2019-09-10 | T-Mobile Usa, Inc. | Remote driving of mobile device diagnostic applications |
US9215181B2 (en) | 2012-11-06 | 2015-12-15 | Comcast Cable Communications, Llc | Systems and methods for managing a network |
US9280437B2 (en) * | 2012-11-20 | 2016-03-08 | Bank Of America Corporation | Dynamically scalable real-time system monitoring |
US10142236B2 (en) | 2013-03-14 | 2018-11-27 | Comcast Cable Communications, Llc | Systems and methods for managing a packet network |
US9535812B2 (en) * | 2013-06-28 | 2017-01-03 | Intel Corporation | Apparatus and method to track device usage |
GB2518151A (en) * | 2013-09-11 | 2015-03-18 | Ibm | Network anomaly detection |
US10122639B2 (en) * | 2013-10-30 | 2018-11-06 | Comcast Cable Communications, Llc | Systems and methods for managing a network |
GB201322571D0 (en) * | 2013-12-19 | 2014-02-05 | Bae Systems Plc | Network fault detection and location |
US20150281008A1 (en) * | 2014-03-25 | 2015-10-01 | Emulex Corporation | Automatic derivation of system performance metric thresholds |
US20150286978A1 (en) * | 2014-04-04 | 2015-10-08 | Stereologic Inc. | System and method for performance measurement and control |
US10326748B1 (en) | 2015-02-25 | 2019-06-18 | Quest Software Inc. | Systems and methods for event-based authentication |
US10417613B1 (en) | 2015-03-17 | 2019-09-17 | Quest Software Inc. | Systems and methods of patternizing logged user-initiated events for scheduling functions |
US9842220B1 (en) | 2015-04-10 | 2017-12-12 | Dell Software Inc. | Systems and methods of secure self-service access to content |
US10536352B1 (en) | 2015-08-05 | 2020-01-14 | Quest Software Inc. | Systems and methods for tuning cross-platform data collection |
US10181988B1 (en) | 2015-08-25 | 2019-01-15 | Control4 Corporation | Systems and methods for monitoring a network device |
US10218588B1 (en) | 2015-10-05 | 2019-02-26 | Quest Software Inc. | Systems and methods for multi-stream performance patternization and optimization of virtual meetings |
US10157358B1 (en) | 2015-10-05 | 2018-12-18 | Quest Software Inc. | Systems and methods for multi-stream performance patternization and interval-based prediction |
US10230604B2 (en) | 2016-02-29 | 2019-03-12 | Airmagnet, Inc. | System and method to monitor network delay |
US10142391B1 (en) * | 2016-03-25 | 2018-11-27 | Quest Software Inc. | Systems and methods of diagnosing down-layer performance problems via multi-stream performance patternization |
US10153980B2 (en) | 2016-07-19 | 2018-12-11 | Coco Communications Corp | Systems and methods for managing network congestion |
US10397065B2 (en) | 2016-12-16 | 2019-08-27 | General Electric Company | Systems and methods for characterization of transient network conditions in wireless local area networks |
US10439915B2 (en) * | 2017-04-14 | 2019-10-08 | Solarwinds Worldwide, Llc | Network status evaluation |
US10390238B1 (en) | 2018-10-30 | 2019-08-20 | Amdocs Development Limited | System, method, and computer program for quantifying real-time business and service impact of underperforming, overloaded, or failed cells and sectors, and for implementing remedial actions prioritization |
US20210392519A1 (en) * | 2018-10-31 | 2021-12-16 | Apple Inc. | Performance measurements related to pdu session and n4 session management |
US11277326B2 (en) * | 2020-03-31 | 2022-03-15 | Netflix, Inc. | Techniques for detecting changes to circuit delays in telecommunications networks |
GB202012610D0 (en) * | 2020-08-13 | 2020-09-30 | British Telecomm | Monitoring network connections |
CN113180697B (en) * | 2021-04-30 | 2022-11-01 | 深湾创新技术(深圳)有限公司 | Brain wave analysis method and terminal |
KR102676784B1 (en) * | 2021-11-15 | 2024-06-20 | 라인 야후 가부시키가이샤 | Method and system for evaluating service performance perceived by user |
Family Cites Families (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3813647A (en) | 1973-02-28 | 1974-05-28 | Northrop Corp | Apparatus and method for performing on line-monitoring and fault-isolation |
GB9019017D0 (en) | 1990-08-31 | 1990-10-17 | Hewlett Packard Co | Network fault analysis |
JPH06243064A (en) | 1993-02-12 | 1994-09-02 | Honda Motor Co Ltd | Fault detection system for computer network |
GB9303640D0 (en) | 1993-02-23 | 1993-04-07 | British Telecomm | Object-oriented correlation |
US5519830A (en) | 1993-06-10 | 1996-05-21 | Adc Telecommunications, Inc. | Point-to-multipoint performance monitoring and failure isolation system |
US5696764A (en) * | 1993-07-21 | 1997-12-09 | Fujitsu Limited | ATM exchange for monitoring congestion and allocating and transmitting bandwidth-guaranteed and non-bandwidth-guaranteed connection calls |
WO1996020548A1 (en) | 1994-12-23 | 1996-07-04 | British Telecommunications Public Limited Company | Fault monitoring |
US5521907A (en) | 1995-04-25 | 1996-05-28 | Visual Networks, Inc. | Method and apparatus for non-intrusive measurement of round trip delay in communications networks |
US5923849A (en) | 1996-05-07 | 1999-07-13 | International Network Services | Method of auditing communication traffic |
US5796633A (en) | 1996-07-12 | 1998-08-18 | Electronic Data Systems Corporation | Method and system for performance monitoring in computer networks |
US5790780A (en) | 1996-07-16 | 1998-08-04 | Electronic Data Systems Corporation | Analysis of failures in a computing environment |
US5937165A (en) | 1996-09-10 | 1999-08-10 | Ganymede Software, Inc | Systems, methods and computer program products for applications traffic based communications network performance testing |
US5867483A (en) | 1996-11-12 | 1999-02-02 | Visual Networks, Inc. | Method and apparatus for measurement of peak throughput in packetized data networks |
US6147998A (en) | 1997-08-26 | 2000-11-14 | Visual Networks Technologies, Inc. | Method and apparatus for performing in-service quality of service testing |
IL121898A0 (en) | 1997-10-07 | 1998-03-10 | Cidon Israel | A method and apparatus for active testing and fault allocation of communication networks |
US6058102A (en) | 1997-11-07 | 2000-05-02 | Visual Networks Technologies, Inc. | Method and apparatus for performing service level analysis of communications network performance metrics |
US6836800B1 (en) * | 1998-09-30 | 2004-12-28 | Netscout Systems, Inc. | Managing computer resources |
US6253339B1 (en) | 1998-10-28 | 2001-06-26 | Telefonaktiebolaget Lm Ericsson (Publ) | Alarm correlation in a large communications network |
US6397359B1 (en) | 1999-01-19 | 2002-05-28 | Netiq Corporation | Methods, systems and computer program products for scheduled network performance testing |
US6678250B1 (en) | 1999-02-19 | 2004-01-13 | 3Com Corporation | Method and system for monitoring and management of the performance of real-time networks |
US6711137B1 (en) | 1999-03-12 | 2004-03-23 | International Business Machines Corporation | System and method for analyzing and tuning a communications network |
GB2350983B (en) | 1999-06-10 | 2001-10-03 | 3Com Corp | Network problem indication |
US6424927B1 (en) | 1999-06-18 | 2002-07-23 | Sri International | Computer-based real-time transient pulse monitoring system and method |
US6564174B1 (en) | 1999-09-29 | 2003-05-13 | Bmc Software, Inc. | Enterprise management system and method which indicates chaotic behavior in system resource usage for more accurate modeling and prediction |
US6598183B1 (en) | 2000-01-04 | 2003-07-22 | Cisco Systems, Inc. | Software tool for automated diagnosis and resolution of problems of voice, data and VoIP communications networks |
US6763380B1 (en) | 2000-01-07 | 2004-07-13 | Netiq Corporation | Methods, systems and computer program products for tracking network device performance |
US7237138B2 (en) | 2000-05-05 | 2007-06-26 | Computer Associates Think, Inc. | Systems and methods for diagnosing faults in computer networks |
US20020129157A1 (en) | 2000-11-08 | 2002-09-12 | Shlomo Varsano | Method and apparatus for automated service level agreements |
US6738933B2 (en) | 2001-05-09 | 2004-05-18 | Mercury Interactive Corporation | Root cause analysis of server system performance degradations |
US20020174217A1 (en) | 2001-05-18 | 2002-11-21 | Gateway, Inc. | System and method for predicting network performance |
US6754854B2 (en) | 2001-06-04 | 2004-06-22 | Motorola, Inc. | System and method for event monitoring and error detection |
US6643613B2 (en) * | 2001-07-03 | 2003-11-04 | Altaworks Corporation | System and method for monitoring performance metrics |
US20030079011A1 (en) | 2001-10-23 | 2003-04-24 | Schilling Cynthia K. | System and method for displaying network status in a network topology |
US20030084146A1 (en) | 2001-10-25 | 2003-05-01 | Schilling Cynthia K. | System and method for displaying network status in a network topology |
US7133368B2 (en) | 2002-02-01 | 2006-11-07 | Microsoft Corporation | Peer-to-peer method of quality of service (QoS) probing and analysis and infrastructure employing same |
US7496655B2 (en) | 2002-05-01 | 2009-02-24 | Satyam Computer Services Limited Of Mayfair Centre | System and method for static and dynamic load analyses of communication network |
US6639538B1 (en) | 2002-05-14 | 2003-10-28 | Sri International | Real-time transient pulse monitoring system and method |
US7274670B2 (en) | 2002-09-27 | 2007-09-25 | Netiq Corporation | Methods, systems and computer program products for assessing network quality |
US7281041B2 (en) | 2002-10-31 | 2007-10-09 | Hewlett-Packard Development Company, L.P. | Method and apparatus for providing a baselining and auto-thresholding framework |
US7031878B2 (en) | 2002-12-16 | 2006-04-18 | General Electric Company | Systems and methods for diagnosing the cause of trend shifts in performance data |
US20060077902A1 (en) | 2004-10-08 | 2006-04-13 | Kannan Naresh K | Methods and apparatus for non-intrusive measurement of delay variation of data traffic on communication networks |
-
2005
- 2005-09-02 US US11/217,477 patent/US7599308B2/en active Active
-
2006
- 2006-02-03 DE DE602006019510T patent/DE602006019510D1/en active Active
- 2006-02-03 CA CA002535198A patent/CA2535198C/en not_active Expired - Fee Related
- 2006-02-03 AT AT06002307T patent/ATE495604T1/en not_active IP Right Cessation
- 2006-02-03 EP EP06002307A patent/EP1689121B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
US7599308B2 (en) | 2009-10-06 |
DE602006019510D1 (en) | 2011-02-24 |
CA2535198A1 (en) | 2006-08-04 |
ATE495604T1 (en) | 2011-01-15 |
CA2535198C (en) | 2009-12-29 |
EP1689121A1 (en) | 2006-08-09 |
US20060176824A1 (en) | 2006-08-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1689121B1 (en) | Methods and apparatus for identifying chronic performance problems on data networks | |
US6108800A (en) | Method and apparatus for analyzing the performance of an information system | |
EP3379419B1 (en) | Situation analysis | |
US7969893B2 (en) | List-based alerting in traffic monitoring | |
EP1422871B1 (en) | Network monitoring system responsive to changes in packet arrival variance and mean | |
EP1646183B1 (en) | Method and apparatus for non-intrusive measurement of delay variation of data traffic on communication networks | |
US8929382B2 (en) | Method, device, and computer program product for detecting and encoding states for accurate measurement | |
US7843815B2 (en) | Estimation of time-varying latency based on network trace information | |
EP1367771B1 (en) | Passive network monitoring system | |
US7313141B2 (en) | Packet sequence number network monitoring system | |
US8244862B2 (en) | Monitoring network performance to identify sources of network performance degradation | |
US7318178B2 (en) | Method and system for reducing false alarms in network fault management systems | |
EP1206085A2 (en) | Method and apparatus for automated service level agreements | |
US20150009840A1 (en) | Packet time stamp processing methods, systems, and apparatus | |
US9009305B1 (en) | Network host inference system | |
US20090262657A1 (en) | Data transfer path evaluation using filtering and change detection | |
US20080259806A1 (en) | Signature Matching Methods and Apparatus for Performing Network Diagnostics | |
Ciavattone et al. | Standardized active measurements on a tier 1 IP backbone | |
US7903657B2 (en) | Method for classifying applications and detecting network abnormality by statistical information of packets and apparatus therefor | |
US10447561B2 (en) | BFD method and apparatus | |
KR20080055744A (en) | A telecommuncations-based link monitoring system | |
US6704289B1 (en) | Method for monitoring service availability and maintaining customer bandwidth in a connectionless (IP) data network | |
US20060072479A1 (en) | Method and system for managing a network slowdown | |
US9306822B2 (en) | Method and system for silent trunk failure detection | |
Tlaiss et al. | Troubleshooting enhancement with automated slow-start detection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA HR MK YU |
|
17P | Request for examination filed |
Effective date: 20070129 |
|
17Q | First examination report despatched |
Effective date: 20070301 |
|
AKX | Designation fees paid |
Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REF | Corresponds to: |
Ref document number: 602006019510 Country of ref document: DE Date of ref document: 20110224 Kind code of ref document: P |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602006019510 Country of ref document: DE Effective date: 20110224 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: VDEP Effective date: 20110112 |
|
LTIE | Lt: invalidation of european patent or patent extension |
Effective date: 20110112 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110512 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110512 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110413 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110423 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110412 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110228 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 |
|
26N | No opposition filed |
Effective date: 20111013 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20111125 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110203 Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110314 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602006019510 Country of ref document: DE Effective date: 20111013 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20110203 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20110112 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602006019510 Country of ref document: DE Representative=s name: VERSCHT, THOMAS K., DIPL.-PHYS.(UNIV.), DE Ref country code: DE Ref legal event code: R081 Ref document number: 602006019510 Country of ref document: DE Owner name: FLUKE ELECTRONICS CORPORATION (N.D.GES.DES STA, US Free format text: FORMER OWNER: VISUAL NETWORKS OPERATIONS, INC., ROCKVILLE, MD., US Ref country code: DE Ref legal event code: R081 Ref document number: 602006019510 Country of ref document: DE Owner name: AIRMAGNET, INC. (N. D. GES. D. STAATES CALIF.), US Free format text: FORMER OWNER: VISUAL NETWORKS OPERATIONS, INC., ROCKVILLE, MD., US |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602006019510 Country of ref document: DE Representative=s name: VERSCHT, THOMAS K., DIPL.-PHYS.(UNIV.), DE Ref country code: DE Ref legal event code: R081 Ref document number: 602006019510 Country of ref document: DE Owner name: FLUKE ELECTRONICS CORPORATION (N.D.GES.DES STA, US Free format text: FORMER OWNER: VISUAL NETWORKS, INC. (N.D.GES.DES STAATES DELAWARE), ROCKVILLE, MD, US Ref country code: DE Ref legal event code: R081 Ref document number: 602006019510 Country of ref document: DE Owner name: AIRMAGNET, INC. (N. D. GES. D. STAATES CALIF.), US Free format text: FORMER OWNER: VISUAL NETWORKS, INC. (N.D.GES.DES STAATES DELAWARE), ROCKVILLE, MD, US |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20151008 AND 20151014 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20160519 AND 20160525 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602006019510 Country of ref document: DE Representative=s name: VERSCHT, THOMAS K., DIPL.-PHYS.(UNIV.), DE Ref country code: DE Ref legal event code: R081 Ref document number: 602006019510 Country of ref document: DE Owner name: AIRMAGNET, INC. (N. D. GES. D. STAATES CALIF.), US Free format text: FORMER OWNER: FLUKE ELECTRONICS CORPORATION (N.D.GES.DES STAATES DELAWARE), EVERETT, WASH., US |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602006019510 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: H04L0012240000 Ipc: H04L0041000000 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20231024 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240228 Year of fee payment: 19 Ref country code: GB Payment date: 20240227 Year of fee payment: 19 |