WO2005083571A1

WO2005083571A1 - A method of controlling the operation of a computing system arranged to interact with other entities

Info

Publication number: WO2005083571A1
Application number: PCT/AU2005/000288
Authority: WO
Inventors: David Mark Eagle; Mark Warren Davies; Christopher Franklin Johnson; Andrew James Wisener
Original assignee: Izuki Corporation
Priority date: 2004-03-01
Filing date: 2005-03-01
Publication date: 2005-09-09

Abstract

A method of controlling the operation of a computing system arranged to interact with other entities is disclosed, the method including the steps of monitoring the interactions of the computing system and determining whether they meet pre-determined conditions and saving data relating to at least one recent interaction of the computing system if at least one of the pre-determined conditions are not met.

Description

A METHOD OF CONTROLLING THE OPERATION OF A COMPUTING SYSTEM ARRANGED TO INTERACT WITH OTHER ENTITIES

Technical Field This invention relates to a method of controlling the operation of a computing system arranged to interact with other entities.

Background to the Invention Computing systems are used to interact with other entities, be they other computing systems or persons operating computing systems such as personal computers. Sometimes these interactions are part of a transaction. Transactions can occur in many forms including loan enquiries, grocery orders, credit card payments, share purchases, automated trading, fund transfers, flight bookings, and so forth. Growth in online transactions is exponential - Gartner and Forrester each forecasted over US$2 trillion in e-commerce in the US alone in 2004. These type of transactions are controlled by computing systems in the form of web-servers operating under the control of ecommerce software applications. Despite increasing sophistication in online services, applications continue to fail. In about the year 2003, research house IDC reported that 28% of attempted e- business transactions fail; and that 90% of e-businesses had application problems in the preceding 12 months that affected customer satisfaction. Many factors cause online application failures, including hardware failures, underestimation of loads, inadequate pre-production testing, programmer error, unexpected user behaviour, increasing technology complexity and inadequate change management practices. The combination of these factors means that online application failure is an enduring phenomenon of the modern economy. When online applications fail, transactions and revenue are lost, brand prestige is damaged, and internal IT and commercial management come under attack.

Summary of the Invention In a first aspect the present invention provides a method of controlling the operation of a computing system arranged to interact with other entities, the method including the steps of: monitoring the interactions of the computing system and determining whether they meet predetermined conditions; and saving data relating to at least one recent interaction of the computing system if at least one of the pre-determined conditions are not met. If an interaction does not meet pre-determined conditions then this indicates that there may be a problem with the interaction, and thus there may be a problem with the transaction that is being carried out. Data relating to the transaction, being data related to the at least one interaction, is saved. Corrective action may then be taken. The step of monitoring may further include the step of periodically making dummy interactions with the computing system, the dummy interactions mimicking those usually carried out between the computing system and another entity. This method of monitoring verifies that the computing system can successfully handle interactions that are similar to real interactions. Data arising from previous interactions of the computing system may be used as the basis for formulating the dummy interactions. The saved data may be used as the basis for formulating the dummy interactions. In this way, the system can verify whether a problem relating to a real transaction can be reproduced. A plurality of dummy interactions may be formulated from saved data relating to a single interaction, and in each of the formulated dummy interactions, at least one element of the data is varied. This allows for verification of the reproducibility of a problem over a range of similar interactions over a range of values.

This assists in identifying the root cause of a problem. The method may further include the step of taking corrective action to modify the operation of the computing system if the predetermined conditions are not met. This measure can avoid the current transaction failing and can prevent subsequent interactions from failing in the same way as the transaction involving the interaction that did not meet the pre-determined conditions. The method may further include the step of, if the pre-determined conditions are not met, repeatedly making that dummy interaction until the pre-determined conditions are met. This allows for identifying when a problem has been fixed. The step of monitoring may further include the step of monitoring the progress of real interactions made between the computing system and another entity. The method may further include the step of, if the pre-determined conditions are not met, formulating and making a dummy interaction based on the real interaction. The pre-determined conditions may relate to the content or timing of data sent or received by the computing system. The pre-determined conditions may include the presence within the data of a particular element of data that must fall within a pre-determined range of values. The particular element of data may be a particular value . The step of determining may further include the step of disregarding some of the data. The disregarded data may include date information, advertising information, or any other variable information that is not considered to be critical to the interaction being monitored. The computing system being controlled may be a web based ecommerce system. In a second aspect the present invention provides a system for controlling the operation of a computing system arranged to interact with other entities, the system including: monitoring means for monitoring at least some interactions of the computing system; determining means for determining whether the interactions meets predetermined conditions; and saving means for saving data relating to recent interactions of the computing system if at least one of the pre-determined conditions are not met. The monitoring means may further include interacting means for periodically making dummy interactions with the computing system, the dummy interactions mimicking those usually carried out between the computing system and another entity. The interacting means may use data arising from previous interactions of the computing system as the basis for formulating the dummy interactions. The interacting means uses the saved data as the basis for formulating the dummy interactions. The interacting means may be arranged to formulate a plurality of dummy interactions from saved data relating to a single interaction, and in each of the formulated dummy interactions, at least one element of the data is varied. The system may further include correcting means for taking corrective action to modify the operation of the computing system if the predetermined conditions are not met . The interacting means may be arranged to repeatedly make that dummy interaction until the pre-determined conditions are met. The monitoring means may be further arranged to monitor the progress of real interactions made between the computing system and another entity. The interaction means may be arranged to formulate and make a dummy interaction based on the real interaction. The pre-determined conditions may relate to the content or timing of data sent or received by the computing system. The pre-determined conditions may include the presence within the data of a particular element of data that must fall within a pre-determined range of values. The particular element of data may be a particular value . The determining means may further be arranged to disregard some of the data. The system may be for controlling the operation of a web based ecommerce system. In a third aspect the present invention provides a method of controlling the operation of a computing system arranged to interact with other entities, the method including the steps of: monitoring the interactions of the computing system and determining whether they meet pre- determined conditions; and if the pre-determined conditions are not met, taking corrective action to modify the operation of the computing system. In a fourth aspect the present invention provides a computer program arranged to instruct a computing system to conduct a method according to either the first or third aspects of the invention. In a fifth aspect the present invention provides a computer program arranged to instruct a computing system to operate as a system according to the second aspect of the invention. In a sixth aspect the present invention provides a computer readable medium including a computer program according to the fifth aspect of the invention.

Brief Description of the Drawings An embodiment of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 is a schematic view of an embodiment of a system for controlling the operation of a computing system according to the present invention.

Detailed Description of the Preferred Embodiment Referring to Figure 1 a system 10 is shown for monitoring the operation of a computing system in the form of e-commerce web-farm 13 that is arranged to interact with other entities in the form of web site users (not shown) who each interact with an e-commerce web-site hosted by web-farm 13 by way of a personal computer over an internet connection. System 10 is made up of two web servers 11, 12 each provided with connections to the internet. System 10 includes monitoring means and determining means in this example being in the form of Server 14 and Analyser 15 which both monitor interactions of computing system 13 and determine whether they meet pre-determined conditions. The system includes saving means in this example being in the form of Savers 16 which are arranged to save data relating to interactions of computing system 13 in Saver Data Store 26. Analyser 15 periodically analyses the data saved in Saver Data Store 26 and if at least one pre- determined condition is not met, it operates to take corrective action and/or to save the data. The Server Data Store 26 is periodically flushed of data that is no longer needed to be retained. Server 14 further includes interacting means embodied in software and hardware which periodically makes dummy interactions with web-farm 13. The dummy interactions mimic those usually carried out by the end users. Analyser 15 includes correcting means embodied in software and hardware which can take corrective action to modify the operation of web-farm 13 if at least one of the pre-determined conditions are not met by applying one or more remedial templates 17. The system 10 will hereinafter be referred to as an Application Safety Net. The aims of the Application Safety Net in this embodiment are:

(i) To detect production failures in http-based applications (ii) To instigate remedial steps to bypass causal problems until they are rectified (iii) To identify customers who fail to complete transactions (iv) To determine whether application failure (or other causes) resulted in the transaction failures (v) To capture customer data pertaining to uncompleted transactions and proactively contact these customers to salvage the transactions (vi) To proactively or reactively ensure that business objectives are maintained within the customer's environment

(vii) To protect and ensure ROI on business applications and company expenditure (viii) Assist in operational planning of Risk Management

The Application Safety Net includes components inside and outside a firewall 18 behind which web-farm 13 sits. The components are designed to run standalone or together depending on the requirements of the operator of web-farm 13.

Application Safety Net Framework The operational framework defines how the individual components of the Application Safety Net system combine to become a comprehensive application monitoring system, that is, both reactive and proactive, when failure (usability or technical) occurs 24 hours a day. There are seven components that combine to form the Application Safety Net Framework:

1. SCRIPTER 20

2. SERVER 14

3. SAVER 16 4. ANALYSER 15

5. REMEDIAL TEMPLATES 17

6. ALERT 22

7. REPORTER 24 1 ) SCRIPTER 20

Scripter 20 is a GUI tool. Scripter 20 records the navigation and actions performed on the site. Simple scripts can be recorded with no real understanding of how the application being monitored works.

As a user surfs through a site or application, scripter 20 records their actions and the source returned to the browser. This process generates the following:

• Comparison Templates which instruct server 14 how to determine if a response page is correct .

• Script Meta-Structure document, this document defines the steps involved in a script . The components of the Application Safety Net use this document to determine the flow of the script including which pages are at critical points and which pages should be compared using a comparison template. • Remedial Templates 17, this document defines the rules for what action to take when a problem is encountered.

• Test Data document, this contains sample test data to be used for the script Scripter 20 uses the Internet Explorer Object Model to request and stream http requests to and from the Scripter software. Session state is maintained, all header, get and post data is recorded. All scripted information, including the users instructions, is recorded in a number of documents.

Comparison Template The comparison template is a copy of the HTML page returned to the scripter tool with tags embedded into it which instruct server 14 how to determine if a response page meets predetermined conditions. For example, the following tags are supported: • Define areas on a page to ignore (Ad Banners, dates, session ids eg. <ASN ignoretags=12>)

• Define areas on a page that must change (todays date, today events eg. <ASN date=today fmt="dd-mmm-yyyy">)

• Define value ranges at a location (between 5% and 20%, >0 eg. <ASN fmt="0%" minvalue=5 maxvalue=20>)

• Describe layout and format of dynamic tables

Using these tags gives scripter 20 the power to be as definite or general about how to determine if a page response is correct. It also provides the ability to easily upgrade scripter 20 to incorporate new tests and functionality without compromising existing templates.

Script Meta-Data The script meta-data is an XML document, which maps out a script at both the transactional and page level .

At the page level it defines :

• The page URL • Description of step

• Where page sits in the frameset structure (for pages with frames)

• Response and alert instructions

• Maximum acceptable response times • Comparison Template details

• Comparison Instructions

• Details of page data fields The transactional level mapping is a way of defining the path through an application at the business transaction level. Scripter 20 identifies which steps of the script represent key steps in a business transaction. This includes critical points in the process that a customer must reach before they can be considered as completing a successful transaction. At the Transactional Level the Script meta-data defines: • Description of Transaction Step

• Maximum time to get to this step

• Response and alert instructions

• Privacy Instructions (which fields should be anonymous)

Test Data The test data XML document is used by server 14 to determine what data to apply to the script . In its ^■simplest form it contains the exact data that the user entered into all input fields, but it can be configured to select test data from a "database" of test values.

2 ) SERVER

Server 14 's main function is to detect problems in an application before a user does by repeatedly simulating an end user. Server 14 runs outside the firewall. It acts like an internet end user surfing the internet and confirms that the pages returned to the browser are the same as the pages recorded in scripter 20. The scripts run are those recorded in scripter 20, the rules used to determine if a page has been returned correctly are defined in the comparison templates generated by scripter 20. Server 14 is a non-intrusive component; it runs completely outside an organization's infrastructure. This method of monitoring has been found by the inventors to be a very successful method of finding and detecting application failure.

Server 14 uses the scripts generated by scripter 20. It uses the same architecture as scripter 20 to run scripts and uses the instructions embedded in the script meta-data file and comparison templates to confirm the results. The results are submitted to reporter 24.

Server 14 can be set up to always try the same test data or to select from a database of test values, or to randomly select values within a specified range.

Server 14 is a Web Service and uses this technology to communicate with the other components . Web Services use XML over the HTTP protocol to allow communication of distributed components over the internet. This allows the components to exchange data over a secure environment. For example, server 14 could pass analyser 15 a request informing it that a script has failed.

3) SAVER

Most http web and application servers do not log enough information to adequately track customer activity. The most commonly absent data includes html header, post data, cookies and session details. Saver 16 is a plug-in for a server to enhance its logging ability. Saver 16 can also be configured to perform actions as instructed by a remedial template 17. For example, it can be instructed to redirect to a different page for a given request, this could be a friendly "Site is down" splash page or a different URL.

Saver 16 is custom written for each type of application or web server it is required to run on, but the basic architecture does not change.

A desirable design requirement for these plug-ins is that they should have a minimal performance overhead and any failure in the plug-in should not cause server 14 to fail.

In this embodiment the plug-in is an IIS ISAPI filter 16. These are written to run on the Microsoft Internet Information Server. The filter obtains the list of requests to log from a file supplied by analyser 15. When a request is sent to the server that saver 16 is logging then the details are posted to a message queue. A service on the other end of the queue will pick up any messages and put them in the Saver Data Store 26. This asynchronous approach to logging ensures the system is robust and does not impact the web server. In a web farm a number web servers can all log to a single data storage area.

Two files are provided to saver 16 by analyser 15:

1. Transaction Metadata, an xml document detailing which requests to log and details on how session and form data is transmitted.

2. Remedial Templates 17, an xml document detailing any remedial action that is currently required to be taken on specific requests.

4) ANALYSER Analyser 15 's main function is to keep an eye on actual users currently using an application to check that they are able to use the application successfully. It does this by various means including: (a) comparing the requests the customer makes to the web server and confirming that the customer is reaching critical points in the application within a defined time limit. These critical points and time limits are mapped out during the scripting process. (b) Examining user data input for validity. (c) Examining the paths taken through the application for validity, (d) Examining the data input for values that may invoke other exception conditions in the application. "(e) Various other conditions.

Analyser 15 analyses the data logged by saver 16 and takes appropriate actions. This part of the system runs on a standalone server to minimize any impact on the installation.

This method of monitoring activity has been found by the inventors to be a very successful method of finding problems as customer encounter them. The transaction meta-data recorded in scripter 20 is used by analyser 15 to determine which pages it needs to analyse. Rules in the analyser 15 define how to determine if a problem has occurred and what action to take when a problem does occur.

Analyser 15 consists of 3 basic components

1. The Listener - A service continually running analyzing the data in the data store . Based on the rules established by the meta-data, 17 when the listener detects a problem or the cessation of a problem, it informs the Rules Based System (see below) .

2. Rules Based System - A program that determines which remedial templates to activate or deactivate based on conditions observed by the Listener.

3. Web Service Gateway - A component that allows communication with external components (Server) . The meta-data provided by scripter 20 gives analyser 15 detailed information on how customers should progress through an application and allows the Analyser to intelligently analyse live activity.

5) REMEDIAL TEMPLATES 17

Remedial templates 17 define actions to take when instructed by the Analyser 15. When a problem is encountered in an application either by server 14 or analyser 15, a range of remedial actions can be taken.

The simplest form of action would be to alert an administrator about the problem, this action is appropriate for problems that are not considered critical or for systems that require human intervention to confirm the exact nature of the problem before taking action.

More complex forms of action can include automatically diverting customers to an apology page, or even redirecting customers around a problem area.

Remedial templates 17 are xml documents containing detailed machine readable instructions on what actions to take when a system fails.

In one scenario, if the Analyser determines that an interaction between a user and the system does not satisfy the pre-determined conditions then it can re-submit the interactions making up the transaction up to the point just before the offending interaction. A customer service representative can be activated to make contact with the user whilst the transaction is still occurring, to guide the user through the rest of the transaction.

6) ALERT 22 '

The Alert 22 component performs the action of actually letting someone know when a problem is found. This can range from a simple email to an administrator through to detailed rules for sending various information to different stakeholders and also escalation procedures. The Alert 22 component informs people of problems using a number of mechanisms including:

• Email

• Fax

• SMS • Voice call

This component includes a comprehensive system notification and escalation system. 7) REPORTER 24

Reporter 24 is the central location for viewing and maintaining the following information:

• An overview and details of Scripts defined in the system

• Schedule details for Server Scripts

• Reports on Server Activity

• Reports on Analyser activity • Script Download. A function to download scripts to scripter 20 for upgrading.

Reporter 24 is an ASP.NET web site. It can be hosted on its own server. Like the other components its structure is based on the scripts generated by scripter 20.

Reporter 24 also uses Web Service technology to communicate with the other tools. Reporter 24 presents its information in real time, that is, as soon as one of the components completes its processing it sends its messages to reporter 24 ensuring the reporter 24 is displaying up to the minute information.

Because saver 16 is logging live activity, reporter 24 gives the administrator live information on how the site is being used and has been used from a business transaction viewpoint. This includes details on failed and incomplete transactions.

When both server 14 and Analyser are implemented, they are set up to work in tandem. Because both server 14 and Analyser are Web Services they can communicate and exchange information. This cross-communication includes:

Building Tests From Live Data During scripting, fields can be flagged as

"anonymous" . These fields are randomized, but all other fields can be passed to server 14 so it can construct its test data from real "live" data. Using this feature, tests gradually evolve to test the application the way actual users use it, not the way the person recording the initial script assumed it would be used. This method also allows a range of valid "paths" through the application to be compiled. The scripts produced can be subsequently analysed and any pages not seen as an impediment to the logical completion of the transaction can be tagged as "noise" pages to be ignored by the system. Further additional scripts can be derived by accumulating multiple distinct interactions by real users, and adding those paths not already mapped by a person using the scripter as a recorder.

This feature of enhancing the Server script library from live data is possible because scripter 20 can define complex rules to determine if the application is performing correctly.

The scripts used by the system can be updated from time to time to allow for developmental changes to the application being monitored.

Detecting Errors Before Customers When server 14 detects an error it can instruct analyser 15 of the problem and analyser 15 can analyse the logs to determine if other users are having a similar problem. Or analyser 15 can be configured to alert someone every time a real user encounters the same problem. Remedial Templates 17 can also be activated or updated either by an administrator or automatically.

What this effectively means is that the system can detect errors before any live customers do and take action to prevent real transactions failing until the application owners correct the problem.

Verifying Failures Externally Web applications are "stateless", ie. You cannot be certain whether the customer has encountered a genuine failure or has simply "walked away" from the application.

When analyser 15 detects a user who has failed to complete a path through an application it can instruct server 14 to take the same path through the application. It can also optionally instruct server 14 to use all or part of the actual data the customer entered. Although it is not possible to be 100% certain that the user has not walked away from the application, a much higher degree of certainty is provided.

Beyond the Web Server In many complex websites transactions can go offline for a period of time before returning to the website. An example of this might be a delivery system for an online ordering application. The customer may place an order and then get emailed once the stock has arrived so they can pick it up. Because of the high level of configurability of the Application Safety Net, business processes such as this can be monitored by both server 14 and the transaction analyser. This may require application specific customizations be done to some components. For example, server 14 can be instructed to sleep for a defined period and then check for an email, or log back on to the site to check progress.

The rules for matching an initial order with subsequent "checks for progress" by a customer are defined to the analyser 15 so it can confirm that customers are getting notified and logging on to confirm orders within a defined time limit .

Because server 14 and analyser 15 are both Web Services they can communicate with each other freely. Also, because they base their operation on the same scripts, they each know what the other component is interested in. The components for the Application Safety Net

Infrastructure are written using Microsoft technologies to run on Microsoft servers. This does not mean that the system only monitors Microsoft based http applications. Because most of the Application Safety Net framework runs completely independent of the client's infrastructure, the system will just as easily monitor any type of web application including UNIX based, Apache, Lotus Notes, Oracle and WebSphere . The saver 16 component to enhance the weblogs is specific to the webserver in use.

The Application Safety Net is designed to monitor http-based applications, this typically means internet based applications. However, there are a number of existing and emerging technologies that use the http protocol to transmit information. This includes Web Services. The B2B components of many complex systems can be monitored using the Application Safety Net system. Other types of applications that the Application Safety Net can be adapted to protect include SMTP email messaging systems, mobile phone applications, extranets, intranets and Virtual Private Network-based applications.

The Application Safety Net system can be installed as a complete system or a customer can select individual modules. For example, a customer may require only server 14 and not analyser 15, or only analyser 15 and not server 14.

The implementation of the Application Safety Net environment will vary depending on the requirements and infrastructure of the client. Here are the steps involved in a typical implementation:

• Identify Applications to Test

• Identify Alert Hierarchy

• Identify Reaction points • Determine which remedial actions to take based on identified exception conditions.

• Install Application Safety Net Scripter on PC and start recording Scripts, either manually or by analysing accumulated live transactions • Setup Server 11 and Reporter 24 outside firewall

• Setup Analyser 15

• Submit scripts to Server 11, Analyser 15 and Reporter 24 • Select and adapt relevant Remedial Templates

Case Studies To illustrate the operation of system 10 it will be convenient to discuss some illustrative case study problems. In these examples, web-farm 13 hosts a web-site dedicated to on-line selling of DVDs.

Problem 1 The payment gateway for the application is provided by an international bank, allowing online credit card payments. The gateway suddenly goes offline at 10:50am which means that customers can no longer make payment at the web-site.

Solution 1 Scripter 20 has been configured to perform a full test purchase on the site every 15 minutes and consequently it initiates an Alert to report a failed purchase attempt to support at 11:00am. The Support Team analyses the problem and determines that the gateway is down and that urgent remedial action is needed.

The Support Team turns on a Remedial Template 17 to perform 3 tasks:

1. it turns off the payment part of the site for all new users attempting to purchase DVDs;

2. it displays a payments gateway apology page for customers currently performing a purchase on the site and 3. it passes the purchase details to the call centre to be completed offline.

Analyser 15 compiles a report on all customers who have attempted to use the application since 10:45am (the last time a scheduled Script completed successfully) . It sends this report to the Customer Service Team, to examine each transaction and double-check whether each one completed satisfactorily.

Meanwhile scripter 20 keeps attempting the script until a purchase can successfully be completed. When the Script is running successfully, the Remedial Template 17 is disabled and business continues as normal.

Problem 2 Analyser 15 reports that less than 5% of customers from Europe who put DVDs in their cart are completing the transaction and going on to pay for the items. The average for all other regions is between 40% and 50%.

The Support Team views a Report on completed European transactions and finds the only completed transactions are for delivery to the UK. The Report shows that all other uses do not complete the process - the vast majority of these get to the delivery charges page and stop.

The Support Team instructs the Application Safety Net to send a simulated order for a customer in Belgium to scripter 20 to run as a test and report the results. Personal data from a real Belgian order is anonymised and sent to scripter 20 which runs the Script and reports an error at the delivery charges page and returns an error on the page "SQL Server Error: No Data found at line 131".

The problem is passed to IT Support to investigate. The Transaction Saver generates a report of European customers who got to that page in the last week but did not progress. This is sent to Customer Service Team to manage, in order to contact the intending customers and attempt to complete the transactions.

Problem 3 Scheduled delivery into inventory of stocks of a new release DVD has fallen through. Customer Support policy requires that all orders for DVDs that cannot be fulfilled must be blocked and an apology page displayed instead. This request can be sent through to IT Maintenance, but turn-around is typically 5 days.

Instead, a Remedial Template 17 can simply be set up to redirect all orders for the new release to an apology page. This redirection can be instigated automatically whenever the warehousing operation notifies an out-of- stock condition for a specified DVD. Scripter 20 can be set-up to test for stock status on every item, at periodic intervals. In this situation, the Remedial Templates 17 can pre-empt the user reaching an apology page, and instead propose alternates from the same artist, or the same genre, or alternately automatically post an offer of a discount if the customer is prepared to wait for back-orders to be shipped.

Problem 4 The application allows customers to view sample preview videos that are "streamed in" from some of the movie studios (ie served live from the movie studios) . These sometimes go offline. Scripter 20 is set to check the existence of the movies at periodic test intervals. If they are found to be offline it instructs the Remedial Template to display a "Movie Offline" page for that movie. This Remedial Template is turned on with no Support Team intervention required.

Problem 5 A new tax has been implemented for the Czech republic. The Transaction Analyser reports that customers are stalling at the point when the tax amount is displayed. A Remedial Template 17 is set up to notify the Call Centre whenever a customer reaches the tax page. The Call Centre staff then call the customer and assist them to complete the transaction.

Problem 6 The site's Support Team has forgotten to update the SSL key license, and secured pages are now reporting an invalid SSL key error. Scripter 20 picks up this anomaly automatically and reports it to the Support Team.

Problem 7 Analyser 15 reports that many customers are not making it from the shopping cart checkout page to the credit card information page, although they do make it to the final confirmation process to complete the process.

IT Support investigates and realizes that corporate customers can purchase without entering credit card info: they simply get sent an invoice at the end of the month. The original testers had overlooked this, but analyser 15 has discovered the new valid path through the system. IT Support instructs the Transaction Analyser to include this new path as a valid Script to both analyse, monitor and run periodic external tests .

Implementation Environments In the preferred embodiment, the computing system being controlled was a web-farm hosting an e-commerce website. The invention is not limited to this application and the controlled computing system could similarly include :

• A mobile device such as a laptop, PDA or mobile telephone

• An IVR (Interactive Voice Response) system

The entities interacting with the computer system in the above described preferred embodiment were people operating personal computers. Similarly, the other entities could be any of: • Another computing system running under control of a program, such as automated market trading applications

• A mobile device

• People using their voice and telephone to interact with an IVR

Where methods and apparatus of the present invention may be implemented by software applications, or partly implemented by software, then they may take the form of program code stored or available from computer readable media, such as CD-ROMS or any other machine readable media, the program code comprising instructions which, when loaded onto a machine such as a computer, the machine then becomes an apparatus for carrying out the invention. The computer readable media may include transmission media, such as cabling, fibre optics or any other form of transmission media. It will also be appreciated that, where methods and apparatus of the present invention are implemented by computing systems, or partly implemented by computing systems, then any appropriate computing system architecture may be utilised. This will include standalone computers, networked computers, and dedicated computing devices. Where the terms "computing system" and "computing device" are used, then these terms are intended to cover any appropriate arrangement of computer hardware for implementing the function described. Any reference to prior art contained herein is not to be taken as an admission that the information is common general knowledge, unless otherwise indicated. Finally, it is to be appreciated that various alterations or additions may be made to the parts previously described without departing from the spirit or ambit of the present invention.

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. A method of controlling the operation of a computing system arranged to interact with other entities, the method including the steps of : monitoring the interactions of the computing system and determining whether they meet pre-determined conditions; and saving data relating to at least one recent ^• interaction of the computing system if at least one of the pre-determined conditions are not met.

2. A method according to claim 1 wherein the step of monitoring further includes the step of periodically making dummy interactions with the computing system, ^• the dummy interactions mimicking those usually carried out between the computing system and another entity.

3. A method according to claim 2 wherein data arising from previous interactions of the computing system is used as the basis for formulating the dummy interactions .

4. A method according to either claim 2 or claim 3 wherein the saved data is used as the basis for formulating the dummy interactions.

5. A method according to claim 4 wherein a plurality of dummy interactions are formulated from saved data relating to a single interaction, and in each of the formulated dummy interactions, at least one element of the data is varied.

6. A method according to any one of claims 2 to 5 further including the step of taking corrective action to modify the operation of the computing system if the predetermined conditions are not met.

7. A method according to any one of claims 2 to 6 further including the step of, if the pre-determined conditions are not met, repeatedly making that dummy interaction until the pre-determined conditions are met .

8. A method according to any preceding claim wherein the step of monitoring further includes the step of monitoring the progress of real interactions made between the computing system and another entity.

9. A method according to claim 8 further including the step of, if the pre-determined conditions are not met, formulating and making a dummy interaction based on the real interaction.

10. A method according to any preceding claim wherein the pre-determined conditions relate to the content or timing of data sent or received by the computing system.

11. A method according to claim 10 wherein the predetermined conditions include the presence within the data of a particular element of data that must fall within a pre-determined range of values.

12. A method according to claim 11 wherein the particular element of data must be a particular value.

13. A method according to claim 10 wherein the step of determining further includes the step of disregarding some of the data.

14. A method according to any preceding claim wherein the computing system being controlled is a web based ecommerce system.

15. A system for controlling the operation of a computing system arranged to interact with other entities, the system including: monitoring means for monitoring at least some interactions of the computing system; determining means for determining whether the interactions meets pre-determined conditions; and saving means for saving data relating to recent interactions of the computing system if at least one of the pre-determined conditions are not met.

16. A system according to claim 15 wherein the monitoring means further includes : interacting means for periodically making dummy interactions with the computing system, the dummy interactions mimicking those usually carried out between the computing system and another entity.

17. A system according to claim 16 wherein the interacting means uses data arising from previous interactions of the computing system as the basis for formulating the dummy interactions.

18. A system according to either claim 16 or claim 17 wherein the interacting means uses the saved data as the basis for formulating the dummy interactions.

19. A system according to claim 18 wherein the interacting means is arranged to formulate a plurality of dummy interactions from saved data relating to a single interaction, and in each of the formulated dummy interactions, at least one element of the data is varied.

20. A system according to any one of claims 16 to 19 further including correcting means for taking corrective action to modify the operation of the computing system if the predetermined conditions are not met.

21. A system according to any one of claims 16 to 20 wherein the interacting means is arranged to repeatedly make that dummy interaction until the predetermined conditions are met.

22. A system according to any one of claims 15 to 21 wherein the monitoring means is further arranged to monitor the progress of real interactions made between the computing system and another entity.

23. A system according to claim 22 wherein the interaction means is arranged to formulate and make a dummy interaction based on the real interaction.

24. A system according to any one of claims 15 to 23 wherein the pre-determined conditions relate to the content or timing of data sent or received by the computing system.

25. A system according to claim 24 wherein the pre- determined conditions include the presence within the data of a particular element of data that must fall within a pre-determined range of values.

26. A system according to claim 25 wherein the particular element of data must be a particular value.

27. A system according to any one of claims 15 to 26 wherein the determining means is further arranged to disregard some of the data.

28. A system according to any one of claims 14 to 27 wherein the system is for controlling the operation of a web based ecommerce system.

29. A method of controlling the operation of a computing system arranged to interact with other entities, the method including the steps of : monitoring the interactions of the computing system and determining whether they meet pre-determined conditions; and if the pre-determined conditions are not met, taking corrective action to modify the operation of the computing system.

30. A method according to claim 29 wherein the computing system being controlled is a web-based ecommerce system.

31. A computer program arranged to instruct a computing system to conduct a method according to any one of claims 1 to 14 or claim 29.

32. A computer program arranged to instruct a computing system to operate as a system according to any one of claims 15 to 28.

33. A computer readable medium including a computer program according to either claim 31 or 32.