GB2596131A

GB2596131A - Estimating network performance requirements

Info

Publication number: GB2596131A
Application number: GB2009356.3A
Authority: GB
Inventors: Robert Blake Andrew; Robert John Shannon Michael
Original assignee: Spatialbuzz Ltd
Current assignee: Spatialbuzz Ltd
Priority date: 2020-06-18
Filing date: 2020-06-18
Publication date: 2021-12-22
Also published as: GB202009356D0

Abstract

A network performance requirement for a software application running on a user equipment (UE) connected to a network is estimated based on indications of user-experience and measurements of a network performance parameter. For each of a plurality of UEs connected to the network, an indication of a respective user’s experience of the software application during a usage period in which the software application was executed on the respective UE, and a measurement of a network performance parameter measured by the respective UE during the usage period are received, 405. A value of the network performance parameter required to achieve a predetermined level of user-experience for the application is determined based on the received indications of user experience and measurements of the network performance parameter, 406. The performance parameter may be one of: received signal strength (RSSI), signal to noise and distortion, Eb/N0, bit error rate, data rate, throughput or latency. The advantage of the invention is that measurement of performance data by the UEs improves the accuracy of data that is associated with a particular application. The invention may be applied to either wireless networks such as 4G, 5G, PMR etc or fixed line networks such as DSL.

Description

ESTIMATING NETWORK PERFORMANCE REQUIREMENTS

BACKGROUND

Software applications, or apps, arc widely used on user equipments (UEs), such as mobile communications devices, to perform a huge range of tasks, from purely providing entertainment through to furnishing useful data or a service. Each different application can place differing demands upon a mobile network to which the device is connected. For example a texting service places a relatively low demand on a network since it typically involves a small amount of data and the delivery of that data is not especially time-critical. By contrast, watching a movie, using for example a video streaming app, can place a much greater demand on a network, since the amount of data required to service the app is large and its transmission should be timely. Likewise, mobile gaming applications can require the transmission of large amounts of data and place particularly stringent requirements upon the latency of the network (even more so than with streaming video apps, which can take advantage of an amount of buffering to deal with the variability of data rates available when moving around a cell or from cell to cell).

App providers typically set minimum requirements for a network in order to guarantee' a certain minimum service quality to their users. For example, they may set a minimum sustainable data-rate and a maximum allowable latency.

Conventionally, a network operator may use these set minimum requirements when designing the network, to ensure that the network has sufficient capability to meet the demands placed on it by apps. However, these set minimum requirements may not provide an accurate picture of network requirements, potentially leading to networks being under-or over-resourced.

SUMMARY

According to a first aspect of the invention, there is provided a method of estimating a network performance requirement for a software application running on a user equipment, UE, connected to a utility supply network. The method comprises: receiving, for each of a plurality of UEs connected to the network, an indication of a respective user's experience of the software application during a usage period in which the software application was executed on the respective UE, and a measurement of a network performance parameter measured by the respective UE during the usage period; and determining, based on the received indications of user experience and measurements of the network performance parameter, a value of the network performance parameter required to achieve a predetermined level of user-experience for the application.

Such a method can provide an indication of the actual requirements for a mobile network, based upon a measurcablc network parameter, for a proportion of the users of the network to experience a certain minimum service quality for a given application. This is turn can enable the network to be planned or updated such that the network performs sufficiently well that a desired proportion of users of a given application on the network will be satisfied with the performance and usability of that app. The network operator can also learn about how certain network resources should be controlled or allocated in order to satisfy the needs of the network users. Thus the network operator does not have to rely solely on potentially inaccurate minimum requirements set by app designers when planning and operating the network.

The derivation of network performance parameter measurements purely from user devices can have the advantage of a lower volume of data needing to be processed than if this were to be gleaned from the network. Likewise the data can easily and accurately be associated with the app which was running; this can be difficult to do with any accuracy based purely upon network-derived data.

A further benefit of the technique is that it can take account of (or identify) evolving user perceptions. The invention described herein can spot trends as they are happening and allow the network operator to either feed-back to the app developer, improve their network, set user expectations more realistically (e.g. through feedback via the operator's app, the developer' s app, social media, etc.) or possibly throttle use of the app or differentially ration service to their higher-paying customers.

The predetermined level of user-experience for the application may be that required to ensure a certain minimum service quality to the users. The service quality may be regarded as a subjective measure of the performance of the application as perceived by the users while they are using the application. The indications of user-experience may be any measure enabling the operator to judge the user's perception of the performance of the app.

In some embodiments, the step of determining the value of the network performance parameter required to achieve the predetermined level of user-experience for the application may comprise: calculating, based on the received indications of user experience and measurements of the net work performance parameter, a threshold value of the network performance parameter for which the predetermined level of user-experience is indicated by a predetermined proportion of users.

The predetermined proportion may be, for example, the number of received indications for the application meeting or exceeding the predetermined level of user-experience as a proportion (e.g. percentage) of the total number of received indications for that application. The number of received indications may take into account whether one or more of the indications are received from the same or separate users, such that, for instance, the proportion can be the proportion of the different users for which user-experience indications and network performance measurements are received. This can provide an approximation of the relevant proportion of the total 'population' of users of the app on the network.

The threshold value of the network performance parameter may be a minimum value (for instance, where a greater value of the network parameter gives an improved performance of the app, such as data rate), or a maximum value (for instance, where a smaller value of the network parameter gives an improved performance of the app, such as latency).

In some embodiments, the method may further comprise receiving, for each of the plurality of UE's, an indication of the identity of the application, and/or application-type, that was executed on the respective UE during the respective usage period.

With this arrangement, the received indications and measurements can subsequently be easily associated with the application to which they relate.

A software application, which may hereon be referred to as an application or app, is a computer program or software application designed to run, at least partially, on a UE, which can be a mobile device such as a phone, tablet, or watch. Apps may be pre-installed on a UE, or otherwise downloaded from application distribution platforms operated by the owner of the mobile operating system.

An indication of the identity of the application can be any data identifying the specific application, or can be an indication of the type of application. The type of application can include a group of applications that are linked by a similar function/intended use and/or require a similar threshold of a given network performance parameter in order to function satisfactorily. Examples of application types can include voice applications, data-only applications, text/messaging applications, video-streaming applications, gaming applications, etc. In some embodiments, determining a network requirement for an app may comprise determining a network requirement for an app type. The network requirement for the app type may be taken to be the network requirement for the app.

In some embodiments, the identity of the application, or application-type, that was executed on the respective UE during the usage period may be determined by the respective UE.

For example, the app which was running can he identified from the mobile operating system of the respective UE. This can provide an efficient and reliable means of associating the software application with the measurements and user-experience indications.

In some embodiments, the received indications of user experience and identity of the application and/or application type, and measurements of the network performance parameter may be used to maintain a database of network performance measurements and corresponding user experience for one or more software applications and/or software application types. The required value of the network performance parameter for the application may be calculated using the data in the database for the application or corresponding application type.

Using the received measurement reports and user-experience indications to maintain a database can enable a larger number of measurements and user-experience indications to be built up over a period of time. Calculating the required value of the network performance parameter using the data in the database can then make use of a larger data-set, providing more accurate and reliable results. In addition, a database built up over a period of time enables the analysis of historic data to reveal time-evolving trends.

Maintaining the database may include, for example, populating the database with measurements and user-experience indications, or with data derived from the measurements and user-experience indications. Maintaining the database may also include updating existing database entries based on newly received measurements and user-experience indications (or data derived therefrom). It may also include the removal of old or no longer relevant data, such as data associated with an app or app-type which is no longer popular or no longer exists.

In some embodiments, receiving the indications of user experience and measurements of network performance parameters for the plurality of UEs comprises retrieving the indications of user experience and/or measurements of network performance parameters from a database.

In some embodiments, data may be removed from the database after a predetermined amount of time. This enables the data in the database to be kept up-to-date and relevant, to ensure that the analysis derived from the data is reflective of the current situation.

In some embodiments, the method may further comprise using the determined value of the network performance parameter to maintain a database of application requirements. In particulm embodiments, maintaining the database of application requirements may comprise: comparing a value of the network performance parameter stored in the application requirements database with the determined value of the network performance parameter; and replacing the stored value with the determined value if the two values differ by more than a predetermined amount.

In other words, a test can be performed to check whether or not the determined value of the network parameter, based on the received indications and measurements, agrees with (for instance, lies within a predetermined error-margin of) the stored value. The stored value can be, at least initially, the requirement for that parameter as published or notified by the app's developer. With this arrangement, processing and power consumption is reduced by only updating the application requirements database if a new value is sufficiently different from that currently stored.

In some embodiments, the method may further comprise: reporting, to a developer of the application, the determined value of the network performance parameter required to achieve the predetermined level of user-experience for the application.

With this arrangement, the developer is automatically made aware of any improved network requirement values, so that they can update their recorded or published network requirements for their app.

In some embodiments, the network performance parameter may be at least one of: received signal strength (RSSI); signal to noise and distortion (SINAD); energy per bit to noise power spectral density ratio (Eb/N0); bit-en-or rate; data-rate; data throughput; or latency.

In some embodiments, each indication of user experience may be based on feedback from the respective user about that user's satisfaction with the application during the respective usage period. The feedback may be any feedback enabling the operator to judge user-satisfaction, such as a user's answers to survey questions and/or user-selectable grading values, Or the user undertaking a net work status check using, for example, an app (different from the app to which the user-experience indications relate) provided by the network operator which incorporates such a facility. This can provide an improved insight into what value of a given network performance parameter actually provides a certain performance as experienced by the application user.

In some embodiments, the method may further comprise: receiving, for each of the plurality of UEs connected to the network, a measurement of a second network performance parameter measured by the respective UE during the usage period; and determining, based on the received indications of user experience and the measurements of the second network performance parameter, a value of the second network performance parameter required to achieve a predetermined level of user-experience for the application.

The second network performance parameter may be a different network performance parameter compared to the previously discussed (first) network performance parameter. For instance, it may be a different one of: received signal strength, RSS1; signal to noise and distortion, SINAD; energy per hit to noise power spectral density ratio. Eb/NO: bit-error rate; data-rate; data throughput; or latency.

Any of the methods described above in relation to the (first) network performance parameter 25 can also be performed for the second network performance parameter.

In some embodiments, the utility supply network may be a communications network.

In some embodiments, the method may further comprise controlling or adapting the utility supply network based on the determined network performance parameter. This may include, for example, building new infrastructure or changing how the network operates or limiting, rationing or otherwise controlling access to one or more apps.

According to a second aspect of the invention, there is provided a computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of any embodiment of the first aspect.

According to a third aspect of the invention, there is provided a network monitoring system for monitoring performance of a utility supply network, wherein the system is configured to receive, from each of a plurality of UEs connected to the network: an indication of a respective user's experience of the software application during a usage period in which a software application was executed on the respective UE; and a measurement of a network performance parameter measured by the respective UE during the usage period. The system comprises at least one processor and a memory storing instructions which, when executed by the at least one processor, cause the processor to perform the method of any embodiment of the first aspect.

BRIEF DESCRIPTION OF FIGURES

By way of example only, certain embodiments of the invention shall now be described by reference to the accompanying drawings, in which: figure 1 illustrates a communications network; figure 2 illustrates a UE; figure 3 illustrates a network monitoring system; figure 4 illustrates a method of estimating a network performance requirement for a software application running on a user equipment; and figure 5 illustrates a method of updating a database of network requirements.

DETAILED DESCRIPTION

As discussed above, a network operator may typically use minimum network requirements set by app providers when designing a network, in order to set the minimum requirements to ensure that the network has sufficient capability to meet the demands placed on it by apps. However, these set minimum requirements based on information from the app providers may not provide an accurate picture of network requirements, potentially leading to networks being under-or over-resourced.

Not all apps place requirements on all parameters, for example many will not be especially concerned with latency, however most will have some mandatory requirements in order to meet their mandated, published or just desired service levels to mobile users of their app.

In the absence of other information, a network operator is likely to take account of these requirements when planning or upgrading the network. For example, the network operator may assume a certain number and distribution of users for each of the top 10 apps and design to meet the published minima from the app providers, for this number and distribution of users in a cell.

Likewise, the network operator may use the published app requirements as metrics in assessing whether the network users are having a good experience on the network and report quality of service data accordingly.

However, a problem with this approach is that it takes no account of what actual users think about the experience they are having with particular apps. Using the minimum requirements set by the app producer can lead to either a significant over-design in the network or a large number of disgruntled and unhappy users, whose apps are not performing as they would like or expect. An app producer may therefore stipulate performance requirements which guarantee that an app operates faultlessly, thereby guaranteeing that the users will be happy with the app and continue to use it. As such, networks can often be over-designed to ensure that the app producer's performance requirements are met.

Referring firstly to Figure 1, a 40 cellular network is shown generally at 100. The network 100 comprises a number of cell-sites, or base-stations, 20-25 that are interconnected by a backhaul infrastructure 30. A number of user equipments (UEs) 10-15 (handsets, for example) are operating in the network 100 and are each connected to one or more of the cell-sites 20-25.

The network 100 also comprises a monitoring station 40 connected to the cell-sites 20-25 via the backhaul 30. In practice, the network 100 comprises many other elements besides those shown, though description of these elements is not necessary to understand the invention as embodied by the devices and processes that are described by reference to the drawings. Additionally. while Figure 1 and the remaining discussion of the invention is in the context of a 4G cellular network, the skilled person will appreciate that it could be any communications network or other utility supply network in which a number of users rely upon a network 'nodes' in order to receive, amalgamate, route or otherwise process and forward, communications traffic or other service flows.

Figure 2 illustrates a user equipment (UE) 10 that is a mobile subscriber device such as a smartphone, for example-that is operating in the 40 cellular telecommunications network 100. The UE 10 could be any of the UEs 10-15 shown in Figure 1. The UE 10 comprises a processor 112, a memory 114, a UPS receiver 116, a 40 transceiver 118, a WiFi0 transceiver 120 and three antennas 122-126. In practice, of course, the UE comprises many other elements besides, though description of these elements is not necessary to understand the invention as embodied by the devices and processes that arc described by reference to the drawings.

The processor 112 has overall control of the UE 10 and executes instructions, with the aid of the storage provided by memory 114 to which the processor 112 is operatively coupled, in order to perform such tasks as may be required of the UE 10 by its user and by the operator of the network 100. The processor 112 is operatively coupled to the UPS receiver 116, the 40 transceiver 118, and the WiFi0 transceiver 120 in order to control and operate those subsystems. In order to perform their respective functions, the UPS receiver 116, the 40 transceiver 118, and the WiFi0 transceiver 120 are operatively coupled to antennas 122-126 respectively.

The UPS receiver 116 receives signals from overhead UPS satellites (not shown), mathematically estimates the location of the UE 10 from those signals, and reports the estimated location to the processor 112.

The 40 transceiver 118 both transmits radio communications to, and receives radio communications from, the 40 cellular communications network 100. The signals transmitted by the 40 transceiver 118 may contain data generated by the user of the UE 10 and also signalling data-control information, measurement information and the like that is utilised by the 40 network 100 for the purpose of enabling the network 100 to at least adequately conduct radio communications with its users' UEs. Similarly, the signals received by the 40 transceiver 118 may contain data intended for user consumption voice-call data or video data, for example as well as signalling data for enabling the UE 10 to interact efficiently with the network 100.

The WiFi0 transceiver 120 both transmits radio communications to, and receives radio communications from, WiFi@ access points (not shown) in the vicinity of the UE 10. The signals transmitted by the WiFi@ transceiver 120 may contain data generated by the user of the UE 100 and also signalling data that enables or facilitates communication between the WiFi0 transceiver 120 and one or more WiFi@ access points. Similarly, the signals received by the WiFi@ transceiver 120 may contain data intended for user consumption voice-call data or video data, for example-as well as signalling data that enables or facilitates communication between the WiFi@ transceiver 120 and one or more WiFi@ access points.

The memory 114 contains inter alia programme code for execution by the processor 112 that constitutes an app 128. As mentioned above, the app 128 is a computer program or software application designed to run on a UE. Apps may be pre-installed on a UE, or otherwise downloaded from application distribution platforms operated by the owner of the mobile operating system. Only one app 128 is shown for clarity of illustration, but it will be appreciated that UE 10 may have multiple apps.

As discussed above, different apps can place different requirements on the network and it is an object of the invention to measure the values of one or more network performance parameters that deliver a particular level of performance for a given app for a typical user.

With this in mind, the UE 10 is configured to perform measurements on the 4G network 100 whilst a particular app, such as app 128, is being run on the UE 10. Such measurements may, for example, be of the following types: * The time it takes for round-trip communication from the LIE 10 to a predetermined interne resource via the 40 network 100 a so-called latency measurement.

* The rate at which user-type data (often called payload data) can be transmitted to the 40 network 100 from the UE 10.

* The rate at which user-type data can be received from the 40 network 100 by the UE 10.

* The throughput of user payload data (i.e. the data the user, or the app which the user is using, sends via the network) which the network can support at a given time, in a given location.

* The bit-error rate in user-type data transmitted to the UE 10 from the 40 network 100.

* RSSI of signals received from a 40 transmitter located at a cell site with which the TIE 1() is in communication.

* The ratio of the total signal power level received from a 40 transmitter to unwanted signal power -a so-called signal-to-noise and distortion ratio (SINAD); * The energy per bit to noise power spectral density ratio (Eb/No) of the signals received from a 40 transmitter.

Such measurements may be generally referred to as network performance measurements, ith each item in the list above representing a different network performance parameter.

The network performance measurements may be measurements that are ordinarily performed by the UE 10 during the course of its normal operation, or may alternatively be measurements performed (on request, for example) for the benefit of the operator of the 40 network 100 (i.e. specifically for the purpose of the methods described below). In either case, the results of the measurements may be stored by the operating system of the UE in the UE's memory. The UE may also be configured to record the identities of the apps (or app types) being executed on the UE. As will be understood, a Oven measurement can be associated with the execution of a particular app if the time that the measurement was performed falls within a period of time that the particular app was being run -referred to as a "usage period".

It may also be useful for the network operator to know die location in its network where a measurement was made, so that the operator can understand how different areas outs network are performing, and diagnose and remedy any underperformance. Therefore, optionally, the UE 10 may be configured estimate its location at the time the measurements were performed by, for example, using the UPS transceiver 116.

The UE 10 communicates the results of such network performance measurements to the operator of the 40 network 100. The network measurements can be transmitted over the 40 network 100 via the 40 transceiver 118, or over a Wi-Fi network via the Wi-Fi transceiver 120. Each measurement may be transmitted along with an indication of the app, or app-type, to which it relates (i.e. that was running at the time the measurement was performed), and/or an indication of the location where the measurement was made.

Figure 3 illustrates an example of the monitoring station 40. The monitoring station 40 comptises a processor 41 coupled to a memory 42, an input 43 representing a downstream connection between the backhaul infrastructure 30 and the monitoring station 40, and an output 44 representing an upstream connection from the monitoring station 40 to the backhaul infrastructure 30. In practice, the input and output may both be provided by a single connection, which connects the monitoring station 40 to the backhaul infrastructure 30. The monitoring station 40 also comprises many other elements besides those shown, though description of these elements is not necessary to understand the invention.

The monitoring station 40 is configured to receive, via the input 43, the network measurements sent from the UEs 10-15. As discussed above, each network measurement comprises the results of a measurement of a network performance parameter (such as one of those listed above) performed during the usage period of a given application on a respective UE. As discussed above, each network measurement can be accompanied by an indication of the identity of the app, or app type, that was running on the respective UE at the time the measurement was performed. The monitoring station 40 may also receive information identifying the location of a respective UE at the time the measurement was performed.

A network measurement can be sent from the UE 10 in response to a request sent to that UE from the monitoring station 40. A measurement can also (or alternatively) be sent from a UE automatically. In the case that network measurements are sent automatically, the UE can be configured to perform and transmit network measurements periodically according to a predetermined time interval. Multiple sets of measurements for different usage periods, different apps, or measurements of multiple network performance parameters, from a UE may be sent in a 'batch'. This can reduce power consumption by limiting the number of individual communications between the UEs and the monitoring station.

The monitoring station 40 is also configured to receive, via the input 43, user-experience indications. Each user-experience indication comprises an indication of a respective user's experience of the software application during the usage period of a given app on the respective UE. The indications of user-experience may be any measure enabling the operator to judge the user's perception of the performance of the app. For example, the indications may be based on (or derived from) a respective user's feedback, such as via a survey, about their experience of using the app. The survey may include questions and user-selectable grading values, allowing the operator to judge satisfaction. Alternatively, or additionally, user-experience indications may be, or may be based on 'network status checks' performed by respective users. The undertaking of a status check by a user may be an indication that the user is dissatisfied with the service being experienced by the user at that time whilst using a given app. The status check may be associated with a particular location, such as a particular region of the communications network or geographical location. For example, the status check may be an indication of the user's dissatisfaction with the app at a particular location. Knowing the location information may for example allow the network operator to exclude other causes of user dissatisfaction, such as known network problems in that location. Additionally or alternatively, knowing the location information may allow the network operator to deduce the experience of other users. If other users in the same/nearby locations/network regions have not submitted a status check, but are also not using the app (e.g. fewer uses than expected for a given time/day), this may be taken as an indication of dissatisfaction with the performance of the app. The network status checks may be submitted by users in a number of ways, as will be appreciated by a person skilled in the art, such as via app provided by the network operator incorporating such a facility (that app being separate from the app to which the user-experience indications actually relate). The user-experience indications can be associated with one or more network measurements based on an association with a respective usage period of the app (for instance, the user's feedback being obtained immediately after the usage period for the app).

Each user-experience indication may be obtained by and received from a respective UE (i.e. transmitted over the 4G network 100 or over a Wi-Fi network in the same way as the network measurements), or may be obtained by and received from a different device.

The received information may be stored in an app user database. The database may group the entries based on the indicated identity of the app or app type. The database may record information enabling different UE to be identified. The database can be updated regularly to ensure that each database entry is up-to-date. This can help to take account of network replanning exercises (e.g. antenna re-pointing) or the addition of new cell-sites. For example, the monitoring station 40 can be configured to remove an entry from the database where that entry relates to a usage period that was more than a predetermined period of time ago.

Figure 4 shows a flowchart of a method 400 for estimating app network performance requirements. Method 400 may be performed for example by the monitoring station 40.

The method begins and moves on to step 405, in which network performance measurements of a first network performance parameter and associated user-experience indications for a particular application are received. The measurements and user-experience indications may be received as discussed above in relation to Figure 3, or may be retrieved from a database (such as the app user database discussed above). The first network performance parameter may be chosen from a list of network parameters, such as the list presented above. The choice of this first network parameter may be based upon the above order, a different order, or at. random.

The method moves on to step 410 in which it is determined, based on the received indications of user experience and measurements of the network performance parameter, a value of the network performance parameter required to achieve a predetermined level of user-experience for the application. The determined value may he the minimum (or maximum, depending upon the network performance parameter) value of the network parameter at which at least a predetermined proportion of users were satisfied, after using the app. The level of satisfaction and the method of determining that satisfaction may vary depending on the network operator, but it is based at least in part on the received indications of user experience. As an example, the app user database may be queried to find the minimum (or maximum, depending upon the network performance parameter) stored value of the network performance parameter at which >Z% of users were satisfied with the particular app (as judged based on the user-experience indications). Z may be 90% or 95% or any other value determined by the network operator.

The determined value of the network performance parameter required to achieve the predetermined level of user-experience for the application can be said to represent the determined minimum requirement of that performance parameter for the app. As mentioned above, "minimum network requirement" does not necessarily imply a minimum value, but is simply a threshold value of the network performance parameter, which may be an upper threshold or a lower threshold, dependent on the particular parameter.

The determined minimum network requirement may then be stored in an app requirements database 420. This process is described further below in relation to Figure 5. Alternatively, or 30 additionally, the determined minimum network requirement may be reported to the developer of the app.

The method may then return to step 405 and repeat the process for a second network performance parameter. It will be appreciated that the method can be repeated as many times as necessary, until, for example, all the network performance parameters relevant to the app (for measurements and indications that are available) have been assessed. The method may include a test undertaken to determine if all relevant parameters have been checked by the method. If so, then the method ends at step 410; if not then the method returns to step 405 in which the next parameter is chosen, as discussed above. This process may then also be repeated for a different app.

Figure 5 illustrates the flow of a method 500 for updating the app user database 420. The method 500 may be performed as part of the method illustrated in Figure 4, or may be performed separately.

Once the minimum requirement for the network parameter has been determined, the method moves on to perform a test to check whether or not the determined minimum requirement, based upon user-experience indications, agrees with (e.g. lies within P% of) the minimum requirement for that parameter stored in the app requirements database 420. The app requirements database 420 is a database of minimum requirements for specific apps initially populated with requirements data for each relevant parameter as provided by the relevant app's developer, or with nominal/default values, determined by the network operator, where no such requirements are provided by the app developer. The minimum requirement stored in the app requirements database at the time the method 500 is performed may be the minimum acceptable level for that parameter as published or notified by the app's developer, a value determined by the network operator (e.g. the default/nominal value discussed above), or may be a minimum requirement that was determined in a previous instance of the method 500.

At step 505, the stored minimum requirement for the parameter is retrieved from the app requirements database, and at step 510, the retrieved stored value is compared to the determined value. If the determined minimum requirement for the network parameter is sufficiently different from (e.g. does not lie within a predetermined range/percent of the stored minimum requirement for that parameter) then the minimum requirement for that parameter is updated in app requirements database 420. In other words, the app's minimum requirements will be updated whether the new value is appreciably less than, or appreciably greater than, the existing (e.g. published) value. In this way, information is captured regarding whether the app developer is being overly optimistic or overly pessimistic regarding the network requirements for the app.

In the event that the parameter lies within P% of the existing (e.g. published) value, then that value may be left unchanged. P may, for example be 10% or any other appropriate value.

Although in the above examples the methods are said to be performed by the network monitoring station 40, it will be appreciated that some or all of the steps can be performed by the UE, or an entirely different device. Further, although illustrated as a separate component of the communications network, it is to be appreciated that in some embodiments the monitoring station 40 may be incorporated into other components of the network, such as a network controller. In general, the methods described above may be performed by any computing system associated with the communications network.

Application to Other Utility Supply Networks It is possible to apply the invention described above to diagnose faults in all kinds of communications networks, including 20, 30, 40, 50, PMR/SMR, Wi-Fi, etc. Equally, it is possible to apply the invention to a fixed-line data network, such as a broadband' internet network (e.g. using DSL, fibre optics or similar).

It is to be appreciated that any of the methods described above may be stored as instructions within a computer program, and/or on a computer readable medium, such as within a memory of a computing device such as a network monitoring system.

Although the invention has been described above with reference to one or more preferred embodiments, it will be appreciated that various changes or modifications may be made without departing from the scope of the invention as defined in the appended claims.

Claims

CLAIMS1. A method of estimating a network performance requirement for a software application running on a user equipment, UE, connected to a utility supply network, the method 5 comprising: receiving, for each of a plurality of UEs connected to the network: an indication of a respective user's experience of the software application during a usage period in which the software application was executed on the respective UE; and a measurement of a network performance parameter measured by the respective UE during the usage period; and deteimining, based on the received indications of user experience and measurements of the network performance parameter, a value of the network performance parameter required to achieve a predetermined level of user-experience for the application.
2. The method of claim 1, wherein the step of determining the value of the network performance parameter required to achieve the predetermined level of user-experience for the application comprises: calculating, based on the received indications of user experience and measurements of the network performance parameter, a threshold value of the network performance parameter for which the predetermined level of user-experience is indicated by a predetermined proportion of users.
3. The method of claim 1 or claim 2, further comprising: receiving, for each of the plurality of UE' s, an indication of the identity of the application, and/or application-type, that was executed on the respective UE during the respective usage period.
4. The method of claim 3, wherein the identity of the application, or application-type, that was executed on the respective UE during the usage period was determined by the respective 30 UE.
5. The method of claims 3 or 4, wherein: the received indications of user experience and identity of the application and/or application type, and measurements of the network performance parameter are used to maintain a database of network performance measurements and corresponding user experience for one or more software applications and/or software application types; and the required value of the network performance parameter for the application is calculated using the data in the database for the application or corresponding application type.
6. The method of claim 5, wherein data is removed from the database after a predetermined amount of time.
7. The method of any preceding claim, further comprising: using the determined value of the network performance parameter to maintain a database of application requirements.
8. The method of claim 7, wherein maintaining the database of application requirements 15 comprises: comparing a value of the network performance parameter stored in the application requirements database with the determined value of the network performance parameter; and replacing the stored value with the determined value if the two values differ by more than a predetermined amount.
9. The method of any preceding claim, further comprising: reporting, to a developer of the application, the deteimined value of the network performance parameter required to achieve the predetermined level of user-experience for the application.
10. The method of any preceding claim, wherein the network performance parameter is one of: received signal strength,RSSI; signal to noise and distortion, SINAD; energy per bit to noise power spectral density ratio, Eb/No; bit-error rate; data-rate; data throughput; or latency.
11. The method of any preceding claim, wherein each indication of user experience is based on feedback from the respective user about that user's satisfaction with the application during the respective usage period.
12. The method of any preceding claim, further comprising: receiving, for each of the plurality of UEs connected to the network: a measurement of a second network performance parameter measured by the respective UE during the usage period; and determining, based on the received indications of user experience and the measurements of the second network performance parameter, a value of the second network performance parameter required to achieve a predetermined level of user-experience for the 15 application.
13. The method of any preceding claim, wherein the utility supply network is a communications network.
14. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of claims 1-13.
15. A network monitoring system for monitoring performance of a utility supply network, wherein the system is configured to receive, from each of a plurality of UEs connected to the network: an indication of a respective user's experience of the software application during a usage period in which a software application was executed on the respective UE; and a measurement of a network performance parameter measured by the respective UE during the usage period, wherein the system comprises: at least one processor, and a memory storing instructions which, when executed by the at least one processor, cause the processor to perform the method of any of claims 1-13.