US20160255400A1 - Remote Display Performance Measurement Triggered by Application Display Upgrade - Google Patents
Remote Display Performance Measurement Triggered by Application Display Upgrade Download PDFInfo
- Publication number
- US20160255400A1 US20160255400A1 US15/150,119 US201615150119A US2016255400A1 US 20160255400 A1 US20160255400 A1 US 20160255400A1 US 201615150119 A US201615150119 A US 201615150119A US 2016255400 A1 US2016255400 A1 US 2016255400A1
- Authority
- US
- United States
- Prior art keywords
- display
- performance
- frame
- data
- remote
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005259 measurement Methods 0.000 title description 27
- 230000001960 triggered effect Effects 0.000 title 1
- 238000000034 method Methods 0.000 claims abstract description 78
- 238000009877 rendering Methods 0.000 claims abstract description 9
- 230000015654 memory Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 abstract description 14
- 239000003795 chemical substances by application Substances 0.000 description 20
- 238000012544 monitoring process Methods 0.000 description 15
- 239000000872 buffer Substances 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000011960 computer-aided design Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 238000013442 quality metrics Methods 0.000 description 7
- 238000003860 storage Methods 0.000 description 7
- 238000007906 compression Methods 0.000 description 6
- 230000006835 compression Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 239000003086 colorant Substances 0.000 description 5
- 238000013515 script Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000003780 insertion Methods 0.000 description 4
- 230000037431 insertion Effects 0.000 description 4
- 238000011056 performance test Methods 0.000 description 4
- 238000012384 transportation and delivery Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 108700026836 CAPABLE protocol Proteins 0.000 description 1
- 235000008733 Citrus aurantifolia Nutrition 0.000 description 1
- 235000011941 Tilia x europaea Nutrition 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005304 joining Methods 0.000 description 1
- 239000004571 lime Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000004513 sizing Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/4424—Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3495—Performance evaluation by tracing or monitoring for systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
- H04N21/4431—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB characterized by the use of Application Program Interface [API] libraries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/815—Virtual
Definitions
- VDI Virtual Desktop Infrastructure
- a user's desktop is typically hosted in a datacenter or cloud, and the user remotely interacts with her desktop via a variety of endpoint devices, including desktops, laptops, thin clients, smart phones, tablets, etc.
- endpoint devices including desktops, laptops, thin clients, smart phones, tablets, etc.
- Embodiments of the present invention provide methods, systems, and computer programs for measuring the quality of a remote display coupled to a remote desktop client. It should be appreciated that the present embodiments can be implemented in numerous ways, such as a process, an apparatus, a system, a device or a method on a computer readable medium. Several inventive embodiments are described below.
- a method in one embodiment, includes an operation for detecting calls from an application to an application programming interface (API), which is provided for rendering images on a display image. Each call causes an update of the display image. Further, the method includes an operation for embedding data for measuring performance in display frames of the display image based on the detecting. The embedding results in modified displayed frames with respective data for measuring performance. The modified displayed frames are transmitted to a remote client, which results in received modified display frames having respective received data for measuring the performance. In addition, the method includes an operation for calculating the remote display quality for the given application based on the received modified display frames and the respective received data for measuring performance.
- API application programming interface
- a system in another embodiment, includes a processor and a memory.
- the memory includes a display management program, a remote display program, and a performance measurement program.
- the display management program when executed by the processor, provides an application programming interface (API) provided for rendering images on a display image.
- API application programming interface
- the remote display program when executed by the processor, transmits the display image to a remote display.
- the program instructions cause the processor to: detect calls from an application to the API that causes an update of the display image; embed data for measuring performance in a plurality of display frames of the display image based on the detection; and calculate a remote display quality.
- the embedding results in a plurality of modified displayed frames with respective data for measuring performance.
- the calculation of the remote display quality for the application of the remote client is based on the received modified display frames with the respective received data for measuring performance.
- a computer program embedded in a non-transitory computer-readable storage medium when executed by one or more processors, for measuring performance of a computer system.
- the computer program includes program instructions for detecting calls from an application to an application programming interface (API) provided for rendering images on a display image, each call causing an update of the display image.
- the computer program includes program instructions for embedding data for measuring performance in a plurality of display frames of the display image based on the detecting, the embedding resulting in a plurality of modified displayed frames with respective data for measuring performance.
- the computer program further includes program instructions for transmitting the plurality of modified displayed frames to a remote client, the remote client receiving a plurality of received modified display frames with respective received data for measuring performance.
- the computer program further includes program instructions for calculating a remote display quality for the application at the remote client based on the received modified display frames and the respective received data for measuring performance.
- a method in one embodiment, includes an operation for embedding in display frames a respective frame identifier and a respective frame picture.
- the respective frame picture is placed in a respective region that depends of the value of the respective frame identifier.
- the method includes operations for sending the plurality of display frames to a remote client, and for extracting each received frame picture from the respective region of each received display frame at the remote client based on the received frame identifier.
- the received frame quality is calculated for each received display frame based on the comparison of the extracted received frame picture with the respective frame picture.
- the method includes an operation for calculating a remote display quality for the remote client based on the calculated received frame quality for the received display frames.
- the system includes a processor and a memory.
- the memory includes a plurality of known images, a remote display program, and a performance measurement program. Each known image is associated with a respective frame identifier, and the remote display program, when executed by the processor, receives a display image for a remote display.
- the program instructions from the performance measurement program when executed by the processor, cause the processor to: extract from a plurality of display frames a respective frame identifier and a respective frame picture, the respective frame picture being extracted from a respective region in each display frame that is based on the respective frame identifier; calculate a received frame quality for each received display frame based on a comparison of the extracted received frame picture with a respective known image that is associated with the frame identifier; and calculate a remote display quality for the remote display based on the calculated received frame quality for the received display frames.
- the computer program includes program instructions for embedding, in a plurality of display frames, a respective frame identifier and a respective frame picture. The respective frame picture is placed in a respective region that is based on the respective frame identifier.
- the computer program includes program instructions for sending, after the embedding, the plurality of display frames to a remote client, and program instructions for extracting each received frame picture from the respective region of each received display frame at the remote client based on the received frame identifier.
- the computer program includes program instructions for calculating a received frame quality for each received display frame based on a comparison of the extracted received frame picture with the respective frame picture that was embedded, and program instructions for calculating a frame rate quality based on the received frame identifiers.
- the computer program includes program instructions for calculating a remote display quality for the remote client based on the calculated received frame quality for the received display frames and the frame rate quality.
- FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment.
- FIG. 2 shows an architecture for measuring display performance using information embedded on the display pixels, in accordance with one embodiment.
- FIG. 3 illustrates the embedding of data for measuring performance in display frames, according to one embodiment.
- FIG. 4 illustrates how to embed performance data in display frames by trapping a call to an Application Programming Interface (API), according to one embodiment.
- API Application Programming Interface
- FIG. 5 is a flowchart for trapping a call to the API, according to one embodiment.
- FIG. 6A is a flowchart for embedding performance data at the server, according to one embodiment.
- FIG. 6B is a flowchart for extracting the performance data at the client, according to one embodiment.
- FIG. 7A is a flowchart for measuring performance of a remote display, according to one embodiment.
- FIG. 7B is a flowchart for calculating a remote display performance, in accordance with one embodiment.
- FIG. 8 is a simplified schematic diagram of a computer system for implementing embodiments presented herein.
- FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment.
- the environment depicted in FIG. 1 includes enterprise servers 102 that provide virtual desktop services to remote users 136 a - m .
- enterprise servers 102 that provide virtual desktop services to remote users 136 a - m .
- FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment.
- the environment depicted in FIG. 1 includes enterprise servers 102 that provide virtual desktop services to remote users 136 a - m .
- FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment.
- the environment depicted in FIG. 1 includes enterprise servers 102 that provide virtual desktop services to remote users 136 a - m .
- FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment.
- the environment depicted in FIG. 1 includes enterprise servers 102 that provide virtual desktop services to remote users 136 a - m .
- FIG. 1 depicts
- performance server 138 is also part of virtualization server 102 , and can be implemented as a virtual appliance.
- Performance server 138 collects performance data from servers and clients and analyzes the data collected for presentation to a user.
- a simplified architecture of virtualization server 102 is shown in FIG. 1 , with some components omitted for simplification purposes.
- a connection broker (not shown) manages the assignment of virtual machines (VM) 104 a - 104 n in enterprise server 102 to remote users.
- Remote users 136 a - 136 m are connected to computers 118 , 122 , 124 and 126 acting as clients in the virtual infrastructure.
- Computers 118 , 122 , 124 and 126 provide display presentation and input/output capabilities associated with virtual machines 104 a - 104 n .
- Clients include tablet 118 , PC 122 , laptop 124 , Personal Digital Assistance (PDA), mobile phone 126 , etc.
- Clients communicate with virtualization server 102 via network 120 .
- PDA Personal Digital Assistance
- Enterprise server 102 includes virtualization software 106 , which is also sometimes referred to as a “hypervisor,” and which manages physical resources of enterprise server 102 and emulates virtual hardware for each virtual machine 104 a - 104 n .
- Virtual machines 104 a - 104 n include a Guest Operating System (GOS) 136 , which may be a commodity operating system such as Microsoft Windows, which supports applications 108 that run on the guest Operating System (OS).
- a different view of virtual machine 104 n includes desktop view 110 , performance module 112 , and server agent 114 .
- Server Agent 114 is a process executing in the VM that processes action requests from local client 128 and provides action start and completion times.
- a local client 128 includes remote display 130 , and a remote desktop environment 132 that includes a remote agent 134 .
- Desktop view 110 may exist as a frame buffer (not separately shown) that corresponds to the graphical user interface display generated by the virtual machine.
- Display information for desktop view 110 which may vary according to the display protocol and include the entire contents of a frame buffer or updates (changes) to the frame buffer, or other display information such as drawing commands, primitives, and/or compressed video information such as macro block motion vectors (as known in the field of video compression) may be transmitted to remote display 130 at local client 128 .
- the display information is received by remote agent 134 and remote desktop 132 updates remote display 130 .
- Virtual Machine 104 n communicates with the associated local client over network 120 using a remote protocol, such as Remote Desktop Protocol (RDP) 116 .
- RDP Remote Desktop Protocol
- Remote Desktop Protocol 116 is a multichannel capable protocol that supports separate virtual channels for carrying presentation data, serial device communication, licensing information, highly encrypted data (keyboard, mouse activity), etc.
- the quality of the display images delivered to remote display 130 is measured. Additionally, the embodiments presented are independent of the communications protocols used to transfer display data, thus being able to reliably obtained performance measurements under different topologies and protocols and assess how different factors affect virtual desktop performance. Further still, the methods presented can scale to tens of thousands of clients and servers without unduly burdening the virtual infrastructure.
- FIG. 2 shows an architecture for measuring display performance using information embedded on the display pixels, in accordance with one embodiment.
- Embodiments presented herein may be used in a plurality of situations with a plurality of applications, as the embodiments presented are not dependent on the type of application being run on the virtual machine.
- rendering API calls made on the VDI desktop by an application or performance benchmark program are intercepted, a watermark is inserted to identify the frame being rendered, and a small reference image is added. On the client-side, the inserted watermark and reference image are extracted and compared against the original to measure the remote display quality.
- the inserted watermark is immune to compression and quantization errors to uniquely mark the identity of the frame being rendered.
- the identity and the position at which the selected reference image is inserted are indexed to the number of the frame being rendered by the application or benchmark.
- server 104 To measure performance of the remote display, server 104 notifies client 128 of particular events using a specific section of the screen.
- a specific area of the screen is designated to carry information in the form of encoded pixels that correspond to predefined messages.
- the area can be a pixel, a square, a rectangle, a line, a circle, etc.
- a call by application 108 to a graphics display API 204 is detected by the display performance application 112 .
- the display API call can be intercepted by display performance application 112 using known interception techniques, or display performance application 112 may subscribe to events raised by the operating system when the display API call is made.
- the API is provided by the operating system (OS) 136 , but the embodiments presented herein may be utilized with any API and any OS.
- the API call causes a change in the desktop view 110 for the virtual machine.
- the display performance application 112 after detecting the API call, inserts data for measuring performance in the desktop view 110 of the display frame, and then server agent 114 transmits the desktop view 110 to the remote client 128 .
- the remote desktop application 132 receives the display data and includes a remote agent 134 , which interacts with the server agent 114 to receive display buffer 202 updates.
- the remote desktop application 132 provides the content of the display buffer 202 to display 130 , which may be integrated with or connected to client 128 .
- remote agent 134 determines when data for measuring performance has been embedded in the received display frames to measure the performance of the remote display.
- the data for measuring performance refers to any data inserted in pixels of a display image, the data being used for measuring the quality of the remote display.
- the data for measuring performance includes a frame number encoded in pixels of the display frame, and a reference image also inserted in the display frame. It is noted that the embodiments described herein utilize a frame number encoded on the top part of the display image, and a reference image inserted in different parts of the display buffer.
- embodiments may utilize different ways of inserting data for the purpose of measuring the quality of the remote display, such as objects inserted within the display image (e.g., a circle, a square, a line, etc.), frame numbers inserted in different parts of the display image, frame numbers encoded with different colors or with different pixel areas, etc.
- objects inserted within the display image e.g., a circle, a square, a line, etc.
- frame numbers inserted in different parts of the display image e.g., a circle, a square, a line, etc.
- frame numbers encoded with different colors or with different pixel areas e.g., a circle, a square, a line, etc.
- the results of the remote display quality measurement may be used by the administrators to improve system resource allocation.
- VDI stacks have become more complex, and bottlenecks may occur in different places, such as in the network, the encoding algorithm, the Graphics Processor Unit (GPU), or somewhere else.
- the remote display quality may be acceptable for some functions, but not for other functions. For example, a clock on the desktop may be updating correctly, but a Computer Aided Design (CAD) application may not be performing well when manipulating 3D objects.
- CAD Computer Aided Design
- remote display quality measurements may be used by administrators to properly size their systems in order to support an adequate number of simultaneous users.
- the systems may be distributed over one or more servers, that provide virtual machines to a plurality of users that have a desktop, or some other computing device, to access a virtual machine instantiated on the one or more servers.
- a single server may be able to support a few users or many users (e.g., several to a hundred or so), depending on the resource availability (i.e., number and clock speed of CPU cores and GPUs, memory size, disk and networking throughput, etc.) of the server and the resource requirements of the users.
- the remote display quality is calculated at the client 128 based on the intercepted data for measuring performance.
- the remote agent 134 sends data to the display performance application 112 at the server 104 , and the display performance application 112 makes the necessary calculations for obtaining a value for the remote display quality.
- the remote agent 134 and the display performance application 112 cooperate to obtain the remote display quality, where some of the calculations are performed at the remote agent 132 and some calculations may be performed at the display performance application 112 .
- the remote display quality may be expressed in several ways.
- the remote display quality may include one or more of a received frame rate, a percentage of received frames, an image quality of the display, time required to receive remote display updates, etc., or any combination thereof.
- the process to enable measurement of the remote display quality may be started by an administrator, by request of a user, by request of a test script that is periodically executed, etc.
- the measurement initialization process includes setting up the mechanism for intercepting one or more calls that update the display.
- the initialization process also includes starting the remote agent in the client that will intercept the received display frames. Therefore, the actual measurement of the remote display performance will begin when the first call is made to a trapped function that updates the display.
- a single application is executed at the virtual machine, preferably in full-screen mode, and calls made by the application to update the display are trapped or intercepted to modify the display frames, before they are sent to the remote display.
- calls made by the application to update the display are trapped or intercepted to modify the display frames, before they are sent to the remote display.
- a performance measurement is started while more than one application is executing, and only one of the applications is being monitored, it can be the case that some updates to the display are not modified before being sent to the remote client because the updates are created by an application that is not being monitored. In this case, it is possible that some frames sent to the remote display have performance markers while other frames do not.
- the performance markers are inserted in the window of the application that is being monitored, but in other embodiments the performance markers may be embedded elsewhere in the screen (e.g., in a desktop area not covered by a window associated with an application).
- time periods allocated for measuring performance may vary. For example, a performance test may take one second, 10 seconds, 1 minute, 10 minutes, a day, etc. Further yet, the end of the performance test may be indicated by a request to end the performance test, such as when an administrator or a user selects an option to end the test. Additionally, the performance test may end after a predetermined amount of time, or after a predetermined number of display frames have been sent to the remote client, etc.
- an administrator may ask a user to press a key on the keyboard (e.g., F11) which causes the performance measurement to start. After a while, if the user presses the same key again (or another key) to stop the test, then the administrator may examine a performance log or some other saved data, to assess the quality of the remote display.
- a key on the keyboard e.g., F11
- FIG. 2 or otherwise described herein are exemplary. Other embodiments may utilize different modules or combine the functionality of several modules, etc. The embodiments illustrated in FIG. 2 should therefore not be interpreted to be exclusive or limiting, but rather exemplary or illustrative.
- FIG. 3 illustrates the embedding of data in display frames for measuring system performance, according to one embodiment.
- Display frame 302 shows a snapshot of a display frame that is to be transmitted to a remote display coupled to a remote client.
- display frame 302 includes a window 308 associated with an application executing on the virtual machine.
- the application may be a CAD Application, a word processing application, a spreadsheet, a game, a video rendering application, etc. that is currently being accessed by a user.
- the trigger to start measuring performance is a detected call to a graphics API, but other triggers arc also possible, such as a user input (e.g., via keyboard, mouse, voice, touch, gesture, etc.), an administrator input, a performance benchmark program instruction, etc.
- a performance frame identifier also referred to herein as a watermark or a frame counter, is inserted in the display frame. This insertion is performed on-the-fly in real time. In other words, the test may be performed during actual use of the system by a user without predetermined frame contents or user activity as would be the case for an automated test.
- a performance frame identifier also referred to herein as a watermark or a frame counter.
- the modified display frame 304 includes two types of embedded data for measuring performance: a frame identifier 310 and a frame picture 312 .
- the frame identifier or watermark 310 is inserted on the top left section of the display, but other embodiments may place the watermark in other areas of the display (e.g., top right corner, bottom left corner, etc.).
- the frame identifier is encoded as a set of tiles, each tile representing a bit of the frame identifier.
- each tile is 16 ⁇ 16 pixels in size, but other sizes are also possible.
- the tiles may have sizes in pixels of 4 ⁇ 4, 4 ⁇ 8, 8 ⁇ 8, 4 ⁇ 16, 8 ⁇ 16, 32 ⁇ 32, 64 ⁇ 64, etc.
- to encode a binary value each pixel in the tile is assigned the same color value, the pixels having a first value to encode a binary 0 and having a second value, different from the first value, to encode a binary 1.
- a binary 0 may be encoded as a black tile while a binary 1 may be encoded as a white tile, but other color values are also possible as long as they provide enough contrast between the two color values.
- the binary value should reliably survive even aggressive image compression and resulting image distortions caused by compression artifacts.
- the tiles are placed horizontally adjacent to each other.
- the tiles may be placed vertically, diagonally, or in any other disposition, as long as the client and the server use the same scheme for placing the tiles.
- not all the tiles are placed adjacent to each other (e.g., 4 tiles may be placed in each of the corners, etc.).
- each of the display frames to be transmitted to the remote client are encoded with the frame identifier, the frame identifier acting as a frame counter, where each frame has a value equal to the value of the previous frame plus 1.
- a frame picture 312 is inserted in the modified display frame 304 .
- the frame picture 312 changes on every frame.
- the location of the frame picture 312 also changes in every frame. The image inserted and the location of the frame picture 312 may be based on the frame number so that the agent at the client can determine where the frame picture is supposed to be situated after decoding the frame identifier. Also, the remote agent is able to determine which frame picture 312 was inserted based on the received frame identifier.
- the frame image and location may be specified in a table that associates a particular image and image location with the value of the last 8 bits of the frame number, the last two digits after converting to decimal notation, or some other value derived from the frame number, e.g., using a cryptographic hashing algorithm.
- the modified display frame 304 is transmitted to the remote client.
- the remote client receives the display frame, which is referred to herein as the received modified display frame 306 , because it corresponds to the original display frame after being modified and as received by the client.
- the received modified display frame 306 may be different from the modified display frame 304 due to alteration during transmission. For example, some of the pixels may have changed due to noise or a display frame may have been dropped by the network.
- the received modified display frame 306 includes a received frame identifier 314 and a receive frame picture 316 , as well as some of the other display elements, such as received application window 318 .
- the display quality is measured based on the received frame identifier 314 and the receive frame picture 316 .
- the information is transmitted to the server and the performance calculations are performed by the server.
- the received modified display frame may include noise artifacts resulting from image compression.
- the agent examines each of the tiles and assigns a binary value of zero or one based on the predominant pixel value within the tile. For example, a tile with 80% or 90% black pixels would be assumed to be a black tile, etc.
- the agent calculates the position of the expected received frame picture, which is based on the extracted frame counter. For example, the frame identifier with a binary value of 00110001 is associated with a location of ( 350 , 75 ) as measured in pixels with reference to the top left corner. This location identifies the top left corner of the inserted picture. In other embodiments, other types of coordinates and positions may be used, as long as both the client and the server use the same protocol to identify the location and identity of the frame pictures. Further, the size of the frame pictures may vary according to different embodiments. In some embodiments, the frame picture may have sizes of 10 ⁇ 10 pixels, 20 ⁇ 20 pixels, 32 ⁇ 32, 100 ⁇ 100 pixels, 100 ⁇ 50, etc.
- the identity of the received frame picture may be determined.
- the client and the agent share an image identity table with at least three columns: frame number, image identity, and image location. Once the frame number is identified, the image identity table is indexed by the frame number to determine the picture that was inserted at the server and the location where it was inserted.
- the received frame picture 316 is compared to the “golden” copy (e.g., the perfect copy that was inserted at the server), also referred to herein as the reference copy.
- the transmission protocols are not able to cache this image in order to avoid retransmitting the image. Under perfect conditions, the image must be sent with every frame update because the image is not in the previous display frame.
- PSNR Peak Signal-to-Noise Ratio
- MSE mean squared error
- PSNR is defined as:
- MAX I is the maximum possible pixel value of the image.
- PSNR the definition of PSNR is the same except the MSE is the sum over all squared value differences divided by image size and by three.
- the image quality for the display frames is measured while an application is executing, which allows the administrator to better understand the quality of image delivery for a given application.
- Other approaches to measure image quality use stock images that are displayed either before or after an application runs, but not while the application is running. Further, since the position is indexed by the value of the frame counter, it is possible to avoid being favored or penalized by compression schemes that may favor or penalize particular regions of the screen.
- the reference image is indexed utilizing the frame counter, which enables the measurement of image quality using different types of images (e.g., an image in black and white, an image with a limited number of colors, an image with a great variety of colors, and image where the colors change rapidly between adjacent pixels, an image where the colors change gradually between adjacent pixels, etc.).
- image quality e.g., an image in black and white, an image with a limited number of colors, an image with a great variety of colors, and image where the colors change rapidly between adjacent pixels, an image where the colors change gradually between adjacent pixels, etc.
- the quality metrics also includes a received frame rate which identifies the percentage of number of frames sent by the server that are received at the client.
- the same reference image is inserted in a plurality of consecutive display frames in the same location. By having the same picture in several frames, it is possible to determine how well the remote display protocol reacts to changes in the display. For example, it may take an average of three frames before the received image obtains a predetermined level of quality. If the number of frames is small, the user's perception will be that the display image updates quickly and with precision. This allows the network administrator to determine what an acceptable lag for updating the remote display is.
- the obtained performance measurements may be used to display an icon on the display (not shown) that informs the user of the level of quality of the remote display.
- the remote display icon may include 7 bars, and the number of displayed bars depends on the current remote display quality.
- the remote display quality is calculated periodically (e.g., every 30 seconds, although other values are also possible) and the icon is updated accordingly.
- FIG. 3 is exemplary. Other embodiments may utilize different types of data for measuring performance, different number of tiles, different locations for the encoded frame number and inserted image, etc.
- the embodiments illustrated in FIG. 3 should therefore not be interpreted to be exclusive or limiting, but rather exemplary or illustrative.
- FIG. 4 illustrates a mechanism for embedding performance data in display frames by trapping a call to an Application Programming Interface (API), according to one embodiment.
- the operating system provides one or more graphics APIs 204 for updating the display image.
- Each API provides one or more functions 406 that may be called by the applications to update the display image.
- the display performance application 112 traps one or more, or all, of the API calls that may cause an update to the display image, in order to embed data in the display frames to measure performance of the remote display quality.
- Embodiments perform real-time watermark insertion based on the APIs selected for measuring display quality.
- the watermarks and the reference images are inserted in real-time on the display frames, before the VDI encodes the data.
- the graphics API Before starting the performance measurement operation, the graphics API, or a subset of the calls within the graphics API, are trapped, which means that when a call is made to a trapped API call, control is transferred to the display performance application 112 .
- the display performance application 112 When control is transferred, the display performance application 112 performs the necessary operations to enable the system to measure the performance of the display on the remote client (e.g., inserting a frame identifier and a reference image into a frame buffer).
- application 108 makes a call 406 to the graphics API 204 .
- the API call causes an update of display image 402 , within a frame buffer that contains graphical display data.
- the trap created for the API call causes a transfer to the display performance application 112 , and the performance program 408 executing in therein, to perform the required insertion of data into the frame buffer for performance measurement.
- the display performance application 112 returns control to the graphics API to perform the requested display operation.
- the result is a modified display frame 306 , which is then transmitted to the client, as discussed previously with reference to FIG. 3 .
- trapping the API call includes setting a software “trampoline” 404 .
- trampoline 404 provides a software mechanism for inserting additional instructions into an existing instruction stream.
- the trampoline includes the replacement of an initial part of the program instructions associated with the API function 406 .
- a jump instruction is inserted to transfer control to the performance program 408 , also referred to as the API interceptor.
- the performance program 408 executes the required operations for setting up performance measurements, the performance program 408 executes the instructions that were removed from the API call, and then executes the jump back to the API code and continue the execution of the API call.
- the trampoline may be inserted by display performance application 112 or server agent 114 ( FIG. 1 ).
- Embodiments presented herein may utilize any type of API call interception or trap, such as utilizing a proxy-dB, code-overwriting or detouring, modifying the “import address table,” modifying an interrupt handler, etc.
- the only requirement is the performance program 408 is invoked when the API call is made.
- a notification is sent to the performance program 408 , while allowing the API call to be executed.
- the performance program may take appropriate action in order to execute the operations required to set up the performance monitoring operation. For example, the performance program 408 may “freeze” the program instructions of the API call until the performance program 408 finishes the set up for the monitoring operation.
- any method or device may be used as long as the performance program is able to set up the monitoring operation when the API call is made.
- OpenGL graphics API is used while running a version of the Windows operating system, and several calls are trapped, including “Present”, “SwapBuffers”, “glFlush”, “glutSwapBuffers”, and “glFinish.”
- the SwapBuffers function exchanges the front and back buffers if the current pixel format for the window referenced by the specified device context includes a back buffer.
- the syntax is:
- HDC specifies a device context.
- calls to the DirectX API are trapped. Any type of display update call may be trapped or intercepted by the display performance application 112 .
- the API interceptor keeps track of the frame number being rendered. The count of frame numbers begins with the first frame rendered when the API interception took place. It is noted that the process of intercepting the API call and inserting data for measuring performance is a low-overhead process, and does not unnecessarily perturb the system being monitored.
- performance data is inserted in response to a call by an application (e.g., a three-dimensional CAD application) it is possible to calculate an accurate performance benchmark for the given application.
- display updates may include other data being updated in the display, such as a desktop clock, an email notification message, a calendar event pop-up, etc.
- the performance measurements are performed on-the-fly, and the performance quality metric is obtained.
- the performance quality metric includes the received frame rate and the quality of the images received at the client.
- the user or the administrator are able to see how the system reacts to changes by the CAD application when the CAD application is interacting with the user.
- the CAD application is an example, and embodiments may be practiced with other applications, such as a word processor, a spreadsheet, a web browser, a video rendering application, etc.
- the performance metrics are also useful for sizing the resource requirements for a system. For example, an administrator may be rolling out a new CAD application to 50 users.
- the performance metrics allow the administrator to determine what are the resources (number of servers, processors, graphics processors, memory, network bandwidth, etc.) required to support all the users.
- FIG. 5 is a flowchart for trapping a call to the API, according to one embodiment.
- operation 502 an API call to be monitored is identified.
- the API-call trap is used to measure performance on a remote display attached to a remote client.
- operation 504 the API call identified in operation 502 is trapped.
- the code at the beginning of the API call is replaced with a jump instruction to a performance module program, but other embodiments may use other software traps.
- the method flows to operation 506 where a call is received from an application for the identified API call.
- This API call causes the jump instruction to be executed and the program instructions for the performance module to start being executed.
- the method flows to operation 508 where data for measuring performance is inserted in a display frame associated with the API call of operation 506 .
- this insertion includes embedding a frame identifier and a reference image in the display frame.
- the method flows to operation 510 where the code that was replaced in operation 504 is executed.
- the method flows to operation 512 where a jump back instruction is executed to transfer program control to the API call at the place that follows the code that was replaced in operation 504 .
- a jump back instruction is executed to transfer program control to the API call at the place that follows the code that was replaced in operation 504 .
- FIG. 5 is exemplary. Other methods may utilize other forms of trapping calls to an API call, as described above with reference to FIG. 4 .
- FIG. 6A is a flowchart for embedding performance data at the server, according to one embodiment.
- a trap is a set for trapping calls to an API call (or calls), where the API call causes an update of the display in a remote client.
- the method flows to operation 604 where the performance program monitor waits for a call from an application to the API call that was trapped in operation 602 .
- the performance monitor intercepts the API call made by the application.
- the method flows to operation 608 where performance monitoring is started if the performance monitoring is not already in process. For example, if the performance monitoring is beginning (i.e., the performance monitoring is starting with this intercepted call), the frame number is reset to an initial value (e.g., 0 or 1) in order to start counting the embedded frame numbers that will be transmitted to the remote display.
- the method flows to operation 610 where the performance monitor embeds performance data in the display frame that will be sent to the remote display. From operation 610 , the method flows to operation 612 where the frame counter is increased so when the next API called is trapped the next display frame sent will have a correct value. Of course, in another embodiment, the frame counter may be increased before sending the display frame instead. The frame counter is increased each frame. In one embodiment, the frame counter starts at one in the first frame and is increased by one in each successive frames. From operation 610 , the method flows back to operation 614 to continue trapping API calls and embedding performance data in successive display frames.
- the user determines the start of the performance monitoring operation, and in another embodiment an administrator, or a performance benchmark program, determines the start.
- scripts may be used to automate the performance monitoring process, where the scripts may be run periodically as determined by the configuration of the performance monitoring system.
- FIG. 6B is a flowchart for extracting the performance data at the client, according to one embodiment.
- a check is made to determine if a received display frame includes embedded data for measuring performance (e.g., a frame counter encoded in groups of pixels placed in a known location of the display frame).
- the method flows to operation 632 where a check is made to determine if performance data was found in the last received frame. If the performance data was not found, the method flows back to operation 630 , and if the performance data was found then the method flows to operation 634 .
- the performance measurement process is started at the client if there isn't an already-started performance monitoring process taking place.
- the performance data is extracted from the frame (e.g. frame identifier and reference image). From operation 636 , the method flows to operation 638 where one or more performance metrics are calculated based on the extracted performance data.
- the extracted performance data is sent to the server, and the server calculates the performance metrics (e.g., frame rate and image quality). The performance calculations may be done at any received frame, every certain number of frames, or periodically (e.g., every minute). For example, the image quality of the received inserted image may be calculated.
- the agent at the client keeps ongoing statistical values of the progression of the remote display quality. In addition, statistical values of the received frame rate may also be tracked.
- the performance monitoring ends when the administrator, or some other actor (e.g., a command from an automated script, a user), requests the termination of the performance measurement to the display performance application.
- the performance monitoring may also end when the administrator sends a command to the remote agent indicating the termination of the performance measurement.
- an automated script e.g., via View Planner tool
- a separate channel is used to notify the client that the operation has ended.
- the performance monitoring may also be terminated after a certain amount of time, manually by a command entered on the virtual machine, after a certain number of modified frames have been received, when the application executed in the virtual machine ends, when no modified display frames are received for a threshold period of Lime (i.e., timeout), or under some other condition established by the administrator.
- a threshold period of Lime i.e., timeout
- FIG. 7A is a flowchart for measuring performance of a remote display, according to one embodiment.
- calls, from an application to an API provided for rendering images on a display image are detected, where each call causes an update of the display image for the remote display.
- the method flows to operation 704 where data for measuring performance is embedded in a plurality of display frames of the display image based on the detecting.
- the embedding of the data for measuring performance results in a plurality of modified displayed frames (having respective data for measuring performance).
- the method flows to operation 706 where the plurality of modified displayed frames is transmitted to the remote client.
- the remote client receives the plurality of received modified display frames, each received modified display frame having respective received data for measuring performance.
- the method flows to operation 708 where the remote display quality for the application is calculated.
- the calculation is performed at the remote client based on the received modified display frames and the respective received data for measuring performance.
- the operations of the method are executed by a processor.
- the embodiments presented herein may be utilized by the administrator to assess the trade-offs between the received frame rate and the received quality of the display. For example, could the display quality improve if the frame rate was lowered? For example, transmitting 30 frames a second may result in better perceived quality than transmitting 60 frames a second, if fewer display frames get lost due to the lower bandwidth utilization.
- a self-learning program is used to dynamically adjust system resources based on the measured remote display performance. For example, the self-learning program may increase the network bandwidth when the remote display quality is below a predetermined threshold. The program then monitors the improvement in quality after the network bandwidth increase to identify the correlation between network bandwidth and remote display quality. Further, the monitoring program may adjust other system resources, such as number of virtual machines per server, number of processors, etc. In addition, the monitoring program may migrate a virtual machine from one server to another in order to avoid congestion.
- FIG. 7B is a flowchart for calculating a remote display performance, in accordance with one embodiment.
- a plurality of display frames are modified to embed a respective frame identifier and a respective frame picture in each of the display frames.
- the respective frame picture is placed in a respective region that is based on the respective frame identifier.
- the location of each frame picture varies from frame to frame, where frame pictures from consecutive frames to not have overlapping pixels on the display.
- the method flows to operation 754 where the plurality of display frames (after embedding the frame identifier and the frame pictures) are sent to a remote client. From operation 754 , the method flows to operation 756 where each received frame picture is extracted, at the remote client, from the respective region of each received display frame based on the received frame identifier.
- the method flows to operation 758 where performance quality metrics are calculated.
- the calculations include a received frame quality for each received display frame based on the comparison of the extracted received frame picture with the respective frame picture.
- the method flows to operation 760 where a second performance quality metric is calculated.
- the second performance quality metric identifies the remote display quality for the remote client, and is calculated based on the calculated received frame quality for the received display frames.
- the operations of the method are executed by a processor.
- FIG. 8 is a simplified schematic diagram of a computer system for implementing embodiments presented herein. It should be appreciated that the methods described herein may be performed with a digital processing system, such as a conventional, general-purpose computer system. Special purpose computers, which are designed or programmed to perform only one function may be used in the alternative.
- the computer system includes a central processing unit (CPU) 804 , which is coupled through bus 810 to random access memory (RAM) 806 , read-only memory (ROM) 812 , and mass storage device 814 .
- Program 808 resides in random access memory (RAM) 806 , but can also reside in mass storage 814 .
- Program 808 may include a performance manager program, such as the display performance application 112 of FIG. 2 .
- Mass storage device 814 represents a persistent data storage device such as a floppy disc drive or a fixed disc drive, which may be local or remote.
- Network interface 830 provides connections via network 832 , allowing communications with other devices.
- CPU 804 may be embodied in a general-purpose processor, a special purpose processor, or a specially programmed logic device.
- Input/Output (I/O) interface provides communication with different peripherals and is connected with CPU 804 , RAM 806 , ROM 812 , and mass storage device 814 , through bus 810 .
- Sample peripherals include display 818 , keyboard 822 , cursor control 824 , removable media device 834 , etc.
- Display 818 is configured to display the user interfaces described herein, such as remote desktop view 130 from FIG. 2 .
- Keyboard 822 , cursor control 824 , removable media device 834 , and other peripherals are coupled to I/O interface 820 in order to communicate information in command selections to CPU 804 .
- I/O interface 820 I/O interface 820 .
- data to and from external devices may be communicated through I/O interface 820 .
- Embodiments may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like.
- the embodiments can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.
- ESX software
- the example ESX(x) server is a hypervisor that partitions physical servers in multiple virtual machines. Each virtual machine may represent a complete system, with processors, memory, networking, storage and BIOS.
- VMware ESX enables multiple virtual machines to: (a) share physical resources, (b) run unmodified operating systems and applications, and (c) run the most resource-intensive applications side by side on the same server.
- Other servers made by other suppliers can have similar functionality or more or less functionality, depending on their target product. Accordingly, the systems, methods and computer readable media can be practiced using any product, software, or hardware configuration.
- the embodiments can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments are useful machine operations.
- the embodiments also relate to a device or an apparatus for performing these operations.
- the apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer.
- various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
- the embodiments can also be embodied as computer readable code on a computer readable medium.
- the computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices.
- the computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
- Data Mining & Analysis (AREA)
Abstract
Description
- This application is related to U.S. Pat. No. 7,831,661, issued Nov. 9, 2010, and entitled “MEASURING CLIENT INTERACTIVE PERFORMANCE USING A DISPLAY CHANNEL;” U.S. Pat. No. 8,347,344, filed on Dec. 18, 2008, and entitled “MEASURING REMOTE VIDEO PLAYBACK PERFORMANCE WITH EMBEDDED ENCODED PIXELS;” U.S. application Ser. No. 12/942,393, filed on Nov. 9, 2010, and entitled “MONITORING AUDIO FIDELITY AND AUDIO-VIDEO SYNCHRONIZATION;” and U.S. application Ser. No. 13/079,972, filed on Apr. 5, 2011, and entitled “QUALITY EVALUATION OF MULTIMEDIA DELIVERY IN CLOUD ENVIRONMENTS,” which are incorporated herein by reference for all purposes.
- Further, this application is related by subject matter to U.S. patent application Ser. No. ______ (Attorney Docket No. B119.02) filed on the same day as the instant application and entitled “MEASUREMENT OF REMOTE DISPLAY PERFORMANCE WITH IMAGE-EMBEDDED MARKERS,” which is incorporated herein by reference.
- In typical virtualized desktop infrastructure architectures, displays and input devices are local, and applications execute remotely in a server. Because applications execute remotely, a latency element is introduced due to the network travel time and the application response time. In Virtual Desktop Infrastructure (VDI) deployments, a user's desktop is typically hosted in a datacenter or cloud, and the user remotely interacts with her desktop via a variety of endpoint devices, including desktops, laptops, thin clients, smart phones, tablets, etc. There are many advantages to this approach, including cost savings, improved mobility, easier management and deployment of applications, etc. However, for VDI environments to become ubiquitous, the user should not be constrained in the type of application that can be successfully run. Accordingly, it is necessary to ensure that, when required, sufficient computational resources are made available in the data center, and, without comprising quality, there is sufficient network bandwidth to transmit the desired imagery and audio to the user's endpoint device. In order to ensure proper quality of delivery, it is necessary to automatically monitor video quality, audio quality, and the synchronization of audio and video.
- Embodiments of the present invention provide methods, systems, and computer programs for measuring the quality of a remote display coupled to a remote desktop client. It should be appreciated that the present embodiments can be implemented in numerous ways, such as a process, an apparatus, a system, a device or a method on a computer readable medium. Several inventive embodiments are described below.
- In one embodiment, a method includes an operation for detecting calls from an application to an application programming interface (API), which is provided for rendering images on a display image. Each call causes an update of the display image. Further, the method includes an operation for embedding data for measuring performance in display frames of the display image based on the detecting. The embedding results in modified displayed frames with respective data for measuring performance. The modified displayed frames are transmitted to a remote client, which results in received modified display frames having respective received data for measuring the performance. In addition, the method includes an operation for calculating the remote display quality for the given application based on the received modified display frames and the respective received data for measuring performance.
- In another embodiment, a system includes a processor and a memory. The memory includes a display management program, a remote display program, and a performance measurement program. The display management program, when executed by the processor, provides an application programming interface (API) provided for rendering images on a display image. Further, the remote display program, when executed by the processor, transmits the display image to a remote display. When the program instructions from the performance measurement program are executed by the processor, the program instructions cause the processor to: detect calls from an application to the API that causes an update of the display image; embed data for measuring performance in a plurality of display frames of the display image based on the detection; and calculate a remote display quality. The embedding results in a plurality of modified displayed frames with respective data for measuring performance. The calculation of the remote display quality for the application of the remote client is based on the received modified display frames with the respective received data for measuring performance.
- In yet another embodiment, a computer program embedded in a non-transitory computer-readable storage medium, when executed by one or more processors, for measuring performance of a computer system is provided. The computer program includes program instructions for detecting calls from an application to an application programming interface (API) provided for rendering images on a display image, each call causing an update of the display image. In addition, the computer program includes program instructions for embedding data for measuring performance in a plurality of display frames of the display image based on the detecting, the embedding resulting in a plurality of modified displayed frames with respective data for measuring performance. The computer program further includes program instructions for transmitting the plurality of modified displayed frames to a remote client, the remote client receiving a plurality of received modified display frames with respective received data for measuring performance. The computer program further includes program instructions for calculating a remote display quality for the application at the remote client based on the received modified display frames and the respective received data for measuring performance.
- In one embodiment, a method includes an operation for embedding in display frames a respective frame identifier and a respective frame picture. The respective frame picture is placed in a respective region that depends of the value of the respective frame identifier. Additionally, the method includes operations for sending the plurality of display frames to a remote client, and for extracting each received frame picture from the respective region of each received display frame at the remote client based on the received frame identifier. The received frame quality is calculated for each received display frame based on the comparison of the extracted received frame picture with the respective frame picture. Further, the method includes an operation for calculating a remote display quality for the remote client based on the calculated received frame quality for the received display frames.
- In another embodiment, the system includes a processor and a memory. The memory includes a plurality of known images, a remote display program, and a performance measurement program. Each known image is associated with a respective frame identifier, and the remote display program, when executed by the processor, receives a display image for a remote display. Further, the program instructions from the performance measurement program, when executed by the processor, cause the processor to: extract from a plurality of display frames a respective frame identifier and a respective frame picture, the respective frame picture being extracted from a respective region in each display frame that is based on the respective frame identifier; calculate a received frame quality for each received display frame based on a comparison of the extracted received frame picture with a respective known image that is associated with the frame identifier; and calculate a remote display quality for the remote display based on the calculated received frame quality for the received display frames.
- In yet another embodiment, the computer program includes program instructions for embedding, in a plurality of display frames, a respective frame identifier and a respective frame picture. The respective frame picture is placed in a respective region that is based on the respective frame identifier. In addition, the computer program includes program instructions for sending, after the embedding, the plurality of display frames to a remote client, and program instructions for extracting each received frame picture from the respective region of each received display frame at the remote client based on the received frame identifier. Further, the computer program includes program instructions for calculating a received frame quality for each received display frame based on a comparison of the extracted received frame picture with the respective frame picture that was embedded, and program instructions for calculating a frame rate quality based on the received frame identifiers. Further yet, the computer program includes program instructions for calculating a remote display quality for the remote client based on the calculated received frame quality for the received display frames and the frame rate quality.
- Other aspects of the embodiments will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the embodiments.
- The embodiments may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which:
-
FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment. -
FIG. 2 shows an architecture for measuring display performance using information embedded on the display pixels, in accordance with one embodiment. -
FIG. 3 illustrates the embedding of data for measuring performance in display frames, according to one embodiment. -
FIG. 4 illustrates how to embed performance data in display frames by trapping a call to an Application Programming Interface (API), according to one embodiment. -
FIG. 5 is a flowchart for trapping a call to the API, according to one embodiment. -
FIG. 6A is a flowchart for embedding performance data at the server, according to one embodiment. -
FIG. 6B is a flowchart for extracting the performance data at the client, according to one embodiment. -
FIG. 7A is a flowchart for measuring performance of a remote display, according to one embodiment. -
FIG. 7B is a flowchart for calculating a remote display performance, in accordance with one embodiment. -
FIG. 8 is a simplified schematic diagram of a computer system for implementing embodiments presented herein. - Methods, systems, and computer programs are presented for measuring quality of image delivery to a display connected to a remote desktop client. It should be noted, however, that the embodiments may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present embodiments.
- The problem of measuring the quality of an image delivered to a remote client has been addressed using a range of tools, and analysis techniques. However, many of these tools rely on measuring the quality of video delivered to a remote display by comparing the video as received at the client with a master copy of the video. However, these types of performance measurements do not allow for the measurement of quality as applications running on a computing device change the display output. In addition, some existing solutions do not allow for the possibility that a Virtual Desktop Infrastructure (VDI) protocol may compress different regions of the desktop screen differently, thereby, necessitating measuring image quality across different regions of the screen. Embodiments presented herein provide solutions for measuring performance on-the-fly on a display that is changing because of changes made by applications running in the computing device.
-
FIG. 1 depicts a remote desktop environment including virtual machine servers, according to one embodiment. The environment depicted inFIG. 1 includesenterprise servers 102 that provide virtual desktop services toremote users 136 a-m. Although embodiments are described within a virtual desktop system, the embodiments presented can be used in other environments with a remote display application transmitting display data to a remote client. - In one embodiment,
performance server 138 is also part ofvirtualization server 102, and can be implemented as a virtual appliance.Performance server 138 collects performance data from servers and clients and analyzes the data collected for presentation to a user. A simplified architecture ofvirtualization server 102 is shown inFIG. 1 , with some components omitted for simplification purposes. A connection broker (not shown) manages the assignment of virtual machines (VM) 104 a-104 n inenterprise server 102 to remote users.Remote users 136 a-136 m are connected tocomputers Computers virtual machines 104 a-104 n. Clients includetablet 118,PC 122,laptop 124, Personal Digital Assistance (PDA),mobile phone 126, etc. Clients communicate withvirtualization server 102 vianetwork 120. -
Enterprise server 102 includesvirtualization software 106, which is also sometimes referred to as a “hypervisor,” and which manages physical resources ofenterprise server 102 and emulates virtual hardware for eachvirtual machine 104 a-104 n.Virtual machines 104 a-104 n include a Guest Operating System (GOS) 136, which may be a commodity operating system such as Microsoft Windows, which supportsapplications 108 that run on the guest Operating System (OS). A different view ofvirtual machine 104 n includesdesktop view 110,performance module 112, andserver agent 114.Server Agent 114 is a process executing in the VM that processes action requests fromlocal client 128 and provides action start and completion times. On the other end, alocal client 128 includesremote display 130, and aremote desktop environment 132 that includes aremote agent 134.Desktop view 110 may exist as a frame buffer (not separately shown) that corresponds to the graphical user interface display generated by the virtual machine. Display information fordesktop view 110, which may vary according to the display protocol and include the entire contents of a frame buffer or updates (changes) to the frame buffer, or other display information such as drawing commands, primitives, and/or compressed video information such as macro block motion vectors (as known in the field of video compression) may be transmitted toremote display 130 atlocal client 128. The display information is received byremote agent 134 andremote desktop 132 updatesremote display 130. -
Virtual Machine 104 n communicates with the associated local client overnetwork 120 using a remote protocol, such as Remote Desktop Protocol (RDP) 116. In one embodiment,Remote Desktop Protocol 116 is a multichannel capable protocol that supports separate virtual channels for carrying presentation data, serial device communication, licensing information, highly encrypted data (keyboard, mouse activity), etc. - In some embodiments, the quality of the display images delivered to
remote display 130 is measured. Additionally, the embodiments presented are independent of the communications protocols used to transfer display data, thus being able to reliably obtained performance measurements under different topologies and protocols and assess how different factors affect virtual desktop performance. Further still, the methods presented can scale to tens of thousands of clients and servers without unduly burdening the virtual infrastructure. -
FIG. 2 shows an architecture for measuring display performance using information embedded on the display pixels, in accordance with one embodiment. Embodiments presented herein may be used in a plurality of situations with a plurality of applications, as the embodiments presented are not dependent on the type of application being run on the virtual machine. - In some embodiments, rendering API calls made on the VDI desktop by an application or performance benchmark program are intercepted, a watermark is inserted to identify the frame being rendered, and a small reference image is added. On the client-side, the inserted watermark and reference image are extracted and compared against the original to measure the remote display quality.
- The inserted watermark is immune to compression and quantization errors to uniquely mark the identity of the frame being rendered. In some embodiments, the identity and the position at which the selected reference image is inserted are indexed to the number of the frame being rendered by the application or benchmark.
- To measure performance of the remote display,
server 104 notifiesclient 128 of particular events using a specific section of the screen. A specific area of the screen is designated to carry information in the form of encoded pixels that correspond to predefined messages. For example, the area can be a pixel, a square, a rectangle, a line, a circle, etc. - Initially, a call by
application 108 to agraphics display API 204 is detected by thedisplay performance application 112. For example, the display API call can be intercepted bydisplay performance application 112 using known interception techniques, ordisplay performance application 112 may subscribe to events raised by the operating system when the display API call is made. In one embodiment, the API is provided by the operating system (OS) 136, but the embodiments presented herein may be utilized with any API and any OS. The API call causes a change in thedesktop view 110 for the virtual machine. Thedisplay performance application 112, after detecting the API call, inserts data for measuring performance in thedesktop view 110 of the display frame, and thenserver agent 114 transmits thedesktop view 110 to theremote client 128. - At the
client 128 side, theremote desktop application 132 receives the display data and includes aremote agent 134, which interacts with theserver agent 114 to receivedisplay buffer 202 updates. Theremote desktop application 132 provides the content of thedisplay buffer 202 to display 130, which may be integrated with or connected toclient 128. - In one embodiment,
remote agent 134 determines when data for measuring performance has been embedded in the received display frames to measure the performance of the remote display. The data for measuring performance refers to any data inserted in pixels of a display image, the data being used for measuring the quality of the remote display. In one embodiment, the data for measuring performance includes a frame number encoded in pixels of the display frame, and a reference image also inserted in the display frame. It is noted that the embodiments described herein utilize a frame number encoded on the top part of the display image, and a reference image inserted in different parts of the display buffer. However, other embodiments may utilize different ways of inserting data for the purpose of measuring the quality of the remote display, such as objects inserted within the display image (e.g., a circle, a square, a line, etc.), frame numbers inserted in different parts of the display image, frame numbers encoded with different colors or with different pixel areas, etc. The embodiments illustrated herein should therefore not be interpreted to be exclusive or limiting, but rather exemplary or illustrative. - The results of the remote display quality measurement may be used by the administrators to improve system resource allocation. VDI stacks have become more complex, and bottlenecks may occur in different places, such as in the network, the encoding algorithm, the Graphics Processor Unit (GPU), or somewhere else. Further, the remote display quality may be acceptable for some functions, but not for other functions. For example, a clock on the desktop may be updating correctly, but a Computer Aided Design (CAD) application may not be performing well when manipulating 3D objects.
- In addition, remote display quality measurements may be used by administrators to properly size their systems in order to support an adequate number of simultaneous users. The systems may be distributed over one or more servers, that provide virtual machines to a plurality of users that have a desktop, or some other computing device, to access a virtual machine instantiated on the one or more servers. A single server may be able to support a few users or many users (e.g., several to a hundred or so), depending on the resource availability (i.e., number and clock speed of CPU cores and GPUs, memory size, disk and networking throughput, etc.) of the server and the resource requirements of the users.
- It is noted that, in one embodiment, the remote display quality is calculated at the
client 128 based on the intercepted data for measuring performance. In another embodiment, theremote agent 134 sends data to thedisplay performance application 112 at theserver 104, and thedisplay performance application 112 makes the necessary calculations for obtaining a value for the remote display quality. In addition, it is also possible that theremote agent 134 and thedisplay performance application 112 cooperate to obtain the remote display quality, where some of the calculations are performed at theremote agent 132 and some calculations may be performed at thedisplay performance application 112. - The remote display quality may be expressed in several ways. For example, the remote display quality may include one or more of a received frame rate, a percentage of received frames, an image quality of the display, time required to receive remote display updates, etc., or any combination thereof.
- In one embodiment, the process to enable measurement of the remote display quality may be started by an administrator, by request of a user, by request of a test script that is periodically executed, etc. In one embodiment, the measurement initialization process includes setting up the mechanism for intercepting one or more calls that update the display. The initialization process also includes starting the remote agent in the client that will intercept the received display frames. Therefore, the actual measurement of the remote display performance will begin when the first call is made to a trapped function that updates the display.
- Typically, for benchmark performance measurements, a single application is executed at the virtual machine, preferably in full-screen mode, and calls made by the application to update the display are trapped or intercepted to modify the display frames, before they are sent to the remote display. If a performance measurement is started while more than one application is executing, and only one of the applications is being monitored, it can be the case that some updates to the display are not modified before being sent to the remote client because the updates are created by an application that is not being monitored. In this case, it is possible that some frames sent to the remote display have performance markers while other frames do not. In one embodiment, the performance markers are inserted in the window of the application that is being monitored, but in other embodiments the performance markers may be embedded elsewhere in the screen (e.g., in a desktop area not covered by a window associated with an application).
- In addition, the time periods allocated for measuring performance may vary. For example, a performance test may take one second, 10 seconds, 1 minute, 10 minutes, a day, etc. Further yet, the end of the performance test may be indicated by a request to end the performance test, such as when an administrator or a user selects an option to end the test. Additionally, the performance test may end after a predetermined amount of time, or after a predetermined number of display frames have been sent to the remote client, etc.
- For example, an administrator may ask a user to press a key on the keyboard (e.g., F11) which causes the performance measurement to start. After a while, if the user presses the same key again (or another key) to stop the test, then the administrator may examine a performance log or some other saved data, to assess the quality of the remote display.
- It is noted that the embodiments illustrated in
FIG. 2 or otherwise described herein are exemplary. Other embodiments may utilize different modules or combine the functionality of several modules, etc. The embodiments illustrated inFIG. 2 should therefore not be interpreted to be exclusive or limiting, but rather exemplary or illustrative. -
FIG. 3 illustrates the embedding of data in display frames for measuring system performance, according to one embodiment.Display frame 302 shows a snapshot of a display frame that is to be transmitted to a remote display coupled to a remote client. For purposes of illustration,display frame 302 includes awindow 308 associated with an application executing on the virtual machine. For example, the application may be a CAD Application, a word processing application, a spreadsheet, a game, a video rendering application, etc. that is currently being accessed by a user. - When a performance measurement trigger is activated, performance data is inserted in
display frame 302, resulting in modifieddisplay frame 304. In one embodiment, the trigger to start measuring performance is a detected call to a graphics API, but other triggers arc also possible, such as a user input (e.g., via keyboard, mouse, voice, touch, gesture, etc.), an administrator input, a performance benchmark program instruction, etc. - After detecting the call to the API, a performance frame identifier, also referred to herein as a watermark or a frame counter, is inserted in the display frame. This insertion is performed on-the-fly in real time. In other words, the test may be performed during actual use of the system by a user without predetermined frame contents or user activity as would be the case for an automated test. Thus, one implementation is to obtain the remote display quality or performance metric for the VDI client while an arbitrary application or benchmark is executing.
- In one embodiment, the modified
display frame 304 includes two types of embedded data for measuring performance: aframe identifier 310 and aframe picture 312. In one embodiment, the frame identifier orwatermark 310 is inserted on the top left section of the display, but other embodiments may place the watermark in other areas of the display (e.g., top right corner, bottom left corner, etc.). The frame identifier is encoded as a set of tiles, each tile representing a bit of the frame identifier. - In one embodiment, each tile is 16×16 pixels in size, but other sizes are also possible. For example, the tiles may have sizes in pixels of 4×4, 4×8, 8×8, 4×16, 8×16, 32×32, 64×64, etc. In one embodiment, to encode a binary value, each pixel in the tile is assigned the same color value, the pixels having a first value to encode a binary 0 and having a second value, different from the first value, to encode a binary 1. For example, a binary 0 may be encoded as a black tile while a binary 1 may be encoded as a white tile, but other color values are also possible as long as they provide enough contrast between the two color values. By using a plurality of pixels for each encoded bit, the binary value should reliably survive even aggressive image compression and resulting image distortions caused by compression artifacts.
- In one embodiment, the tiles are placed horizontally adjacent to each other. In other embodiments, the tiles may be placed vertically, diagonally, or in any other disposition, as long as the client and the server use the same scheme for placing the tiles. Further, in one embodiment, not all the tiles are placed adjacent to each other (e.g., 4 tiles may be placed in each of the corners, etc.).
- After starting the performance measurement, each of the display frames to be transmitted to the remote client are encoded with the frame identifier, the frame identifier acting as a frame counter, where each frame has a value equal to the value of the previous frame plus 1.
- In addition to the frame identifier, a
frame picture 312 is inserted in the modifieddisplay frame 304. In one embodiment, theframe picture 312 changes on every frame. In addition, the location of theframe picture 312 also changes in every frame. The image inserted and the location of theframe picture 312 may be based on the frame number so that the agent at the client can determine where the frame picture is supposed to be situated after decoding the frame identifier. Also, the remote agent is able to determine whichframe picture 312 was inserted based on the received frame identifier. For example, the frame image and location may be specified in a table that associates a particular image and image location with the value of the last 8 bits of the frame number, the last two digits after converting to decimal notation, or some other value derived from the frame number, e.g., using a cryptographic hashing algorithm. - After the display frame has been modified, the modified
display frame 304 is transmitted to the remote client. The remote client receives the display frame, which is referred to herein as the received modifieddisplay frame 306, because it corresponds to the original display frame after being modified and as received by the client. The received modifieddisplay frame 306 may be different from the modifieddisplay frame 304 due to alteration during transmission. For example, some of the pixels may have changed due to noise or a display frame may have been dropped by the network. - The received modified
display frame 306 includes a receivedframe identifier 314 and a receiveframe picture 316, as well as some of the other display elements, such as receivedapplication window 318. At the received client, the display quality is measured based on the receivedframe identifier 314 and the receiveframe picture 316. In some other embodiments, the information is transmitted to the server and the performance calculations are performed by the server. - As discussed above, the received modified display frame may include noise artifacts resulting from image compression. To extract the frame identifier, the agent examines each of the tiles and assigns a binary value of zero or one based on the predominant pixel value within the tile. For example, a tile with 80% or 90% black pixels would be assumed to be a black tile, etc.
- After extracting the value of the binary encoded frame counter by joining together all the binary values for each of the tiles, the agent calculates the position of the expected received frame picture, which is based on the extracted frame counter. For example, the frame identifier with a binary value of 00110001 is associated with a location of (350, 75) as measured in pixels with reference to the top left corner. This location identifies the top left corner of the inserted picture. In other embodiments, other types of coordinates and positions may be used, as long as both the client and the server use the same protocol to identify the location and identity of the frame pictures. Further, the size of the frame pictures may vary according to different embodiments. In some embodiments, the frame picture may have sizes of 10×10 pixels, 20×20 pixels, 32×32, 100×100 pixels, 100×50, etc.
- From the extracted frame identifier, the identity of the received frame picture may be determined. In one embodiment, the client and the agent share an image identity table with at least three columns: frame number, image identity, and image location. Once the frame number is identified, the image identity table is indexed by the frame number to determine the picture that was inserted at the server and the location where it was inserted.
- The received
frame picture 316 is compared to the “golden” copy (e.g., the perfect copy that was inserted at the server), also referred to herein as the reference copy. In one embodiment, by having the reference image location and/or identity change in every frame, the transmission protocols are not able to cache this image in order to avoid retransmitting the image. Under perfect conditions, the image must be sent with every frame update because the image is not in the previous display frame. - There are different methods for calculating the image quality based on the comparison between the received and the golden copy. In one embodiment, the Peak Signal-to-Noise Ratio (PSNR) is used. PSNR defines the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Because many signals have a very wide dynamic range, PSNR is usually expressed in terms of the logarithmic decibel scale. PSNR is defined via the mean squared error (MSE). Given a noise-free m×n monochrome image I and its noisy approximation K, MSE is defined as:
-
- PSNR is defined as:
-
- Where MAXI is the maximum possible pixel value of the image. For color images with three RUB values per pixel, the definition of PSNR is the same except the MSE is the sum over all squared value differences divided by image size and by three.
- In other embodiments, other methods may be utilized, such as Data Compression Ratio, the Perceptual Evaluation of Video Quality (PEVQ), Signal-to-Noise ratio (S/N), Structural Similarity (SSIM), etc. These and other image quality metrics are well known in the field, the use of any of which suitable ones is contemplated.
- The image quality for the display frames is measured while an application is executing, which allows the administrator to better understand the quality of image delivery for a given application. Other approaches to measure image quality use stock images that are displayed either before or after an application runs, but not while the application is running. Further, since the position is indexed by the value of the frame counter, it is possible to avoid being favored or penalized by compression schemes that may favor or penalize particular regions of the screen.
- In addition, the reference image is indexed utilizing the frame counter, which enables the measurement of image quality using different types of images (e.g., an image in black and white, an image with a limited number of colors, an image with a great variety of colors, and image where the colors change rapidly between adjacent pixels, an image where the colors change gradually between adjacent pixels, etc.). This flexibility allows obtaining remote display quality measurements that are not tilted towards one particular application type.
- The quality metrics also includes a received frame rate which identifies the percentage of number of frames sent by the server that are received at the client. In one embodiment, the same reference image is inserted in a plurality of consecutive display frames in the same location. By having the same picture in several frames, it is possible to determine how well the remote display protocol reacts to changes in the display. For example, it may take an average of three frames before the received image obtains a predetermined level of quality. If the number of frames is small, the user's perception will be that the display image updates quickly and with precision. This allows the network administrator to determine what an acceptable lag for updating the remote display is.
- In one embodiment, the obtained performance measurements may be used to display an icon on the display (not shown) that informs the user of the level of quality of the remote display. For example, the remote display icon may include 7 bars, and the number of displayed bars depends on the current remote display quality. In one embodiment, the remote display quality is calculated periodically (e.g., every 30 seconds, although other values are also possible) and the icon is updated accordingly.
- It is noted that the embodiments illustrated in or described with reference to
FIG. 3 are exemplary. Other embodiments may utilize different types of data for measuring performance, different number of tiles, different locations for the encoded frame number and inserted image, etc. The embodiments illustrated inFIG. 3 should therefore not be interpreted to be exclusive or limiting, but rather exemplary or illustrative. -
FIG. 4 illustrates a mechanism for embedding performance data in display frames by trapping a call to an Application Programming Interface (API), according to one embodiment. In one embodiment, the operating system provides one ormore graphics APIs 204 for updating the display image. Each API provides one ormore functions 406 that may be called by the applications to update the display image. Thedisplay performance application 112 traps one or more, or all, of the API calls that may cause an update to the display image, in order to embed data in the display frames to measure performance of the remote display quality. - Embodiments perform real-time watermark insertion based on the APIs selected for measuring display quality. The watermarks and the reference images are inserted in real-time on the display frames, before the VDI encodes the data.
- Before starting the performance measurement operation, the graphics API, or a subset of the calls within the graphics API, are trapped, which means that when a call is made to a trapped API call, control is transferred to the
display performance application 112. When control is transferred, thedisplay performance application 112 performs the necessary operations to enable the system to measure the performance of the display on the remote client (e.g., inserting a frame identifier and a reference image into a frame buffer). - In one embodiment,
application 108 makes acall 406 to thegraphics API 204. The API call causes an update ofdisplay image 402, within a frame buffer that contains graphical display data. The trap created for the API call causes a transfer to thedisplay performance application 112, and theperformance program 408 executing in therein, to perform the required insertion of data into the frame buffer for performance measurement. Afterwards, thedisplay performance application 112 returns control to the graphics API to perform the requested display operation. The result is a modifieddisplay frame 306, which is then transmitted to the client, as discussed previously with reference toFIG. 3 . - In one embodiment, trapping the API call includes setting a software “trampoline” 404. Broadly speaking,
trampoline 404 provides a software mechanism for inserting additional instructions into an existing instruction stream. The trampoline includes the replacement of an initial part of the program instructions associated with theAPI function 406. In its place, a jump instruction is inserted to transfer control to theperformance program 408, also referred to as the API interceptor. After theperformance program 408 executes the required operations for setting up performance measurements, theperformance program 408 executes the instructions that were removed from the API call, and then executes the jump back to the API code and continue the execution of the API call. The trampoline may be inserted bydisplay performance application 112 or server agent 114 (FIG. 1 ). - Embodiments presented herein may utilize any type of API call interception or trap, such as utilizing a proxy-dB, code-overwriting or detouring, modifying the “import address table,” modifying an interrupt handler, etc. The only requirement is the
performance program 408 is invoked when the API call is made. In another embodiment, a notification is sent to theperformance program 408, while allowing the API call to be executed. The performance program may take appropriate action in order to execute the operations required to set up the performance monitoring operation. For example, theperformance program 408 may “freeze” the program instructions of the API call until theperformance program 408 finishes the set up for the monitoring operation. In summary, any method or device may be used as long as the performance program is able to set up the monitoring operation when the API call is made. - In one embodiment, OpenGL graphics API is used while running a version of the Windows operating system, and several calls are trapped, including “Present”, “SwapBuffers”, “glFlush”, “glutSwapBuffers”, and “glFinish.” For example, the SwapBuffers function exchanges the front and back buffers if the current pixel format for the window referenced by the specified device context includes a back buffer. The syntax is:
-
- BOOL WINAPI SwapBuffers(HDC hdc);
- Where HDC specifies a device context. In another embodiment, calls to the DirectX API are trapped. Any type of display update call may be trapped or intercepted by the
display performance application 112. The API interceptor keeps track of the frame number being rendered. The count of frame numbers begins with the first frame rendered when the API interception took place. It is noted that the process of intercepting the API call and inserting data for measuring performance is a low-overhead process, and does not unnecessarily perturb the system being monitored. - Since the performance data is inserted in response to a call by an application (e.g., a three-dimensional CAD application) it is possible to calculate an accurate performance benchmark for the given application. For example, if the performance measurements are performed independently from application calls, display updates may include other data being updated in the display, such as a desktop clock, an email notification message, a calendar event pop-up, etc.
- The performance measurements are performed on-the-fly, and the performance quality metric is obtained. In one embodiment, the performance quality metric includes the received frame rate and the quality of the images received at the client. The user or the administrator are able to see how the system reacts to changes by the CAD application when the CAD application is interacting with the user. The CAD application is an example, and embodiments may be practiced with other applications, such as a word processor, a spreadsheet, a web browser, a video rendering application, etc.
- The performance metrics are also useful for sizing the resource requirements for a system. For example, an administrator may be rolling out a new CAD application to 50 users. The performance metrics allow the administrator to determine what are the resources (number of servers, processors, graphics processors, memory, network bandwidth, etc.) required to support all the users.
-
FIG. 5 is a flowchart for trapping a call to the API, according to one embodiment. Inoperation 502, an API call to be monitored is identified. The API-call trap is used to measure performance on a remote display attached to a remote client. Fromoperation 502, the method flows tooperation 504, where the API call identified inoperation 502 is trapped. In one embodiment, the code at the beginning of the API call is replaced with a jump instruction to a performance module program, but other embodiments may use other software traps. - From
operation 504, the method flows tooperation 506 where a call is received from an application for the identified API call. This API call causes the jump instruction to be executed and the program instructions for the performance module to start being executed. - From
operation 506, the method flows tooperation 508 where data for measuring performance is inserted in a display frame associated with the API call ofoperation 506. In one embodiment, this insertion includes embedding a frame identifier and a reference image in the display frame. Fromoperation 508, the method flows tooperation 510 where the code that was replaced inoperation 504 is executed. - From
operation 510, the method flows tooperation 512 where a jump back instruction is executed to transfer program control to the API call at the place that follows the code that was replaced inoperation 504. It is noted that the method described with reference toFIG. 5 is exemplary. Other methods may utilize other forms of trapping calls to an API call, as described above with reference toFIG. 4 . -
FIG. 6A is a flowchart for embedding performance data at the server, according to one embodiment. Inoperation 602, a trap is a set for trapping calls to an API call (or calls), where the API call causes an update of the display in a remote client. - From
operation 602, the method flows tooperation 604 where the performance program monitor waits for a call from an application to the API call that was trapped inoperation 602. Inoperation 606, the performance monitor intercepts the API call made by the application. Fromoperation 606, the method flows tooperation 608 where performance monitoring is started if the performance monitoring is not already in process. For example, if the performance monitoring is beginning (i.e., the performance monitoring is starting with this intercepted call), the frame number is reset to an initial value (e.g., 0 or 1) in order to start counting the embedded frame numbers that will be transmitted to the remote display. - From
operation 608, the method flows tooperation 610 where the performance monitor embeds performance data in the display frame that will be sent to the remote display. Fromoperation 610, the method flows tooperation 612 where the frame counter is increased so when the next API called is trapped the next display frame sent will have a correct value. Of course, in another embodiment, the frame counter may be increased before sending the display frame instead. The frame counter is increased each frame. In one embodiment, the frame counter starts at one in the first frame and is increased by one in each successive frames. Fromoperation 610, the method flows back to operation 614 to continue trapping API calls and embedding performance data in successive display frames. - In one embodiment, the user determines the start of the performance monitoring operation, and in another embodiment an administrator, or a performance benchmark program, determines the start. For example, scripts may be used to automate the performance monitoring process, where the scripts may be run periodically as determined by the configuration of the performance monitoring system.
-
FIG. 6B is a flowchart for extracting the performance data at the client, according to one embodiment. At the client, in operation 630 a check is made to determine if a received display frame includes embedded data for measuring performance (e.g., a frame counter encoded in groups of pixels placed in a known location of the display frame). - From
operation 630, the method flows tooperation 632 where a check is made to determine if performance data was found in the last received frame. If the performance data was not found, the method flows back tooperation 630, and if the performance data was found then the method flows tooperation 634. - In
operation 634, the performance measurement process is started at the client if there isn't an already-started performance monitoring process taking place. Inoperation 636, the performance data is extracted from the frame (e.g. frame identifier and reference image). Fromoperation 636, the method flows tooperation 638 where one or more performance metrics are calculated based on the extracted performance data. In another embodiment, the extracted performance data is sent to the server, and the server calculates the performance metrics (e.g., frame rate and image quality). The performance calculations may be done at any received frame, every certain number of frames, or periodically (e.g., every minute). For example, the image quality of the received inserted image may be calculated. In one embodiment, the agent at the client keeps ongoing statistical values of the progression of the remote display quality. In addition, statistical values of the received frame rate may also be tracked. - On the server side, the performance monitoring ends when the administrator, or some other actor (e.g., a command from an automated script, a user), requests the termination of the performance measurement to the display performance application. On the client side, the performance monitoring may also end when the administrator sends a command to the remote agent indicating the termination of the performance measurement. In one embodiment, if the performance measurement has been initiated by an automated script (e.g., via View Planner tool), a separate channel is used to notify the client that the operation has ended. Additionally, the performance monitoring may also be terminated after a certain amount of time, manually by a command entered on the virtual machine, after a certain number of modified frames have been received, when the application executed in the virtual machine ends, when no modified display frames are received for a threshold period of Lime (i.e., timeout), or under some other condition established by the administrator.
-
FIG. 7A is a flowchart for measuring performance of a remote display, according to one embodiment. Inoperation 702, calls, from an application to an API provided for rendering images on a display image, are detected, where each call causes an update of the display image for the remote display. - From
operation 702, the method flows tooperation 704 where data for measuring performance is embedded in a plurality of display frames of the display image based on the detecting. The embedding of the data for measuring performance results in a plurality of modified displayed frames (having respective data for measuring performance). - From
operation 704, the method flows tooperation 706 where the plurality of modified displayed frames is transmitted to the remote client. The remote client receives the plurality of received modified display frames, each received modified display frame having respective received data for measuring performance. - From
operation 706, the method flows tooperation 708 where the remote display quality for the application is calculated. The calculation is performed at the remote client based on the received modified display frames and the respective received data for measuring performance. In one embodiment, the operations of the method are executed by a processor. - The embodiments presented herein may be utilized by the administrator to assess the trade-offs between the received frame rate and the received quality of the display. For example, could the display quality improve if the frame rate was lowered? For example, transmitting 30 frames a second may result in better perceived quality than transmitting 60 frames a second, if fewer display frames get lost due to the lower bandwidth utilization.
- In one embodiment, a self-learning program is used to dynamically adjust system resources based on the measured remote display performance. For example, the self-learning program may increase the network bandwidth when the remote display quality is below a predetermined threshold. The program then monitors the improvement in quality after the network bandwidth increase to identify the correlation between network bandwidth and remote display quality. Further, the monitoring program may adjust other system resources, such as number of virtual machines per server, number of processors, etc. In addition, the monitoring program may migrate a virtual machine from one server to another in order to avoid congestion.
-
FIG. 7B is a flowchart for calculating a remote display performance, in accordance with one embodiment. Inoperation 752, a plurality of display frames are modified to embed a respective frame identifier and a respective frame picture in each of the display frames. The respective frame picture is placed in a respective region that is based on the respective frame identifier. In one embodiment, the location of each frame picture varies from frame to frame, where frame pictures from consecutive frames to not have overlapping pixels on the display. - From
operation 752, the method flows tooperation 754 where the plurality of display frames (after embedding the frame identifier and the frame pictures) are sent to a remote client. Fromoperation 754, the method flows tooperation 756 where each received frame picture is extracted, at the remote client, from the respective region of each received display frame based on the received frame identifier. - From
operation 756, the method flows tooperation 758 where performance quality metrics are calculated. In one embodiment, the calculations include a received frame quality for each received display frame based on the comparison of the extracted received frame picture with the respective frame picture. - From
operation 758, the method flows tooperation 760 where a second performance quality metric is calculated. The second performance quality metric identifies the remote display quality for the remote client, and is calculated based on the calculated received frame quality for the received display frames. In one embodiment, the operations of the method are executed by a processor. -
FIG. 8 is a simplified schematic diagram of a computer system for implementing embodiments presented herein. It should be appreciated that the methods described herein may be performed with a digital processing system, such as a conventional, general-purpose computer system. Special purpose computers, which are designed or programmed to perform only one function may be used in the alternative. The computer system includes a central processing unit (CPU) 804, which is coupled throughbus 810 to random access memory (RAM) 806, read-only memory (ROM) 812, andmass storage device 814.Program 808 resides in random access memory (RAM) 806, but can also reside inmass storage 814.Program 808 may include a performance manager program, such as thedisplay performance application 112 ofFIG. 2 . -
Mass storage device 814 represents a persistent data storage device such as a floppy disc drive or a fixed disc drive, which may be local or remote.Network interface 830 provides connections vianetwork 832, allowing communications with other devices. It should be appreciated thatCPU 804 may be embodied in a general-purpose processor, a special purpose processor, or a specially programmed logic device. Input/Output (I/O) interface provides communication with different peripherals and is connected withCPU 804,RAM 806,ROM 812, andmass storage device 814, throughbus 810. Sample peripherals includedisplay 818,keyboard 822,cursor control 824,removable media device 834, etc. -
Display 818 is configured to display the user interfaces described herein, such asremote desktop view 130 fromFIG. 2 .Keyboard 822,cursor control 824,removable media device 834, and other peripherals are coupled to I/O interface 820 in order to communicate information in command selections toCPU 804. IL should be appreciated that data to and from external devices may be communicated through I/O interface 820. - Embodiments may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The embodiments can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.
- Although virtualization operations can take on many forms, several concepts are important to ensure that physical constructs of physical machines are properly transferred to virtual machines. With this in mind, and by way of reference, more information on virtual system and methods can be found in U.S. patents owned by the assignee of the present application, VMware, Inc. Thus, U.S. Pat. Nos. 6,397,242, 6,496,847, 7,069,413, 7,356,679, and US Published Application 2008/0244028 A1, are each incorporated by reference for all purposes.
- For completeness, reference is made to an example server system, that is referred to by the trademark ESX™, which is provided by VMware, of Palo Alto, Calif. However, the methods, systems and embodiments are not limited to any one brand of server or virtualization servers. The example ESX(x) server is a hypervisor that partitions physical servers in multiple virtual machines. Each virtual machine may represent a complete system, with processors, memory, networking, storage and BIOS. VMware ESX enables multiple virtual machines to: (a) share physical resources, (b) run unmodified operating systems and applications, and (c) run the most resource-intensive applications side by side on the same server. Other servers made by other suppliers can have similar functionality or more or less functionality, depending on their target product. Accordingly, the systems, methods and computer readable media can be practiced using any product, software, or hardware configuration.
- With the above embodiments in mind, it should be understood that the embodiments can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments are useful machine operations. The embodiments also relate to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
- The embodiments can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
- Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times, or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in the desired way.
- Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/150,119 US9578373B2 (en) | 2010-11-09 | 2016-05-09 | Remote display performance measurement triggered by application display upgrade |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/942,393 US8788079B2 (en) | 2010-11-09 | 2010-11-09 | Monitoring audio fidelity and audio-video synchronization |
US13/079,972 US9674562B1 (en) | 2008-12-18 | 2011-04-05 | Quality evaluation of multimedia delivery in cloud environments |
US13/872,036 US8910228B2 (en) | 2010-11-09 | 2013-04-26 | Measurement of remote display performance with image-embedded markers |
US13/872,040 US9336117B2 (en) | 2010-11-09 | 2013-04-26 | Remote display performance measurement triggered by application display upgrade |
US15/150,119 US9578373B2 (en) | 2010-11-09 | 2016-05-09 | Remote display performance measurement triggered by application display upgrade |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/872,040 Continuation US9336117B2 (en) | 2010-11-09 | 2013-04-26 | Remote display performance measurement triggered by application display upgrade |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160255400A1 true US20160255400A1 (en) | 2016-09-01 |
US9578373B2 US9578373B2 (en) | 2017-02-21 |
Family
ID=51790265
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/872,040 Active 2034-06-18 US9336117B2 (en) | 2010-11-09 | 2013-04-26 | Remote display performance measurement triggered by application display upgrade |
US15/150,119 Active US9578373B2 (en) | 2010-11-09 | 2016-05-09 | Remote display performance measurement triggered by application display upgrade |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/872,040 Active 2034-06-18 US9336117B2 (en) | 2010-11-09 | 2013-04-26 | Remote display performance measurement triggered by application display upgrade |
Country Status (1)
Country | Link |
---|---|
US (2) | US9336117B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9674562B1 (en) | 2008-12-18 | 2017-06-06 | Vmware, Inc. | Quality evaluation of multimedia delivery in cloud environments |
US10453161B2 (en) | 2008-12-18 | 2019-10-22 | Vmware, Inc. | Watermarking and scalability techniques for a virtual desktop planning tool |
Families Citing this family (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8788079B2 (en) | 2010-11-09 | 2014-07-22 | Vmware, Inc. | Monitoring audio fidelity and audio-video synchronization |
US9336117B2 (en) | 2010-11-09 | 2016-05-10 | Vmware, Inc. | Remote display performance measurement triggered by application display upgrade |
US8863232B1 (en) | 2011-02-04 | 2014-10-14 | hopTo Inc. | System for and methods of controlling user access to applications and/or programs of a computer |
US9419848B1 (en) | 2012-05-25 | 2016-08-16 | hopTo Inc. | System for and method of providing a document sharing service in combination with remote access to document applications |
US8713658B1 (en) | 2012-05-25 | 2014-04-29 | Graphon Corporation | System for and method of providing single sign-on (SSO) capability in an application publishing environment |
US9792014B2 (en) | 2013-03-15 | 2017-10-17 | Microsoft Technology Licensing, Llc | In-place contextual menu for handling actions for a listing of items |
KR20150061470A (en) * | 2013-11-27 | 2015-06-04 | 한국전자통신연구원 | VDI service providing system and method |
US9600251B1 (en) * | 2014-10-13 | 2017-03-21 | Google Inc. | Enhancing API service schemes |
US10949075B2 (en) | 2014-11-06 | 2021-03-16 | Microsoft Technology Licensing, Llc | Application command control for small screen display |
US20160132301A1 (en) | 2014-11-06 | 2016-05-12 | Microsoft Technology Licensing, Llc | Programmatic user interface generation based on display size |
US10148584B2 (en) * | 2014-12-15 | 2018-12-04 | Ca, Inc. | Adaptive compression |
GB2527672B (en) * | 2015-05-22 | 2016-06-01 | Gamesys Ltd | Measuring device analysis |
US9866724B2 (en) | 2015-06-18 | 2018-01-09 | Vmware, Inc. | Pixel perturbation for transmission of meta-information |
US9842387B2 (en) * | 2015-06-18 | 2017-12-12 | Vmware, Inc. | Pixel perturbation for image quality measurement |
US9877064B2 (en) | 2015-07-24 | 2018-01-23 | GM Global Technology Operations LLC | Systems and methods for efficient event-based synchronization in media file transfer and real-time display rendering between a peripheral system and a host device |
US20170026684A1 (en) * | 2015-07-24 | 2017-01-26 | GM Global Technology Operations LLC | Communications between a peripheral system and a host device in efficient event-based synchronization of media transfer for real-time display rendering |
US11076143B1 (en) * | 2016-12-12 | 2021-07-27 | Facebook Technologies, Llc | In-band tear detection with compression |
WO2019135773A1 (en) * | 2018-01-08 | 2019-07-11 | Hewlett-Packard Development Company, L.P. | Brokering servers based on remote access performance |
US11500659B2 (en) * | 2018-12-10 | 2022-11-15 | Intuit, Inc. | System and method for virtual machine and virtual desktop accessibility |
US11930299B2 (en) | 2021-01-22 | 2024-03-12 | VMware LLC | Measuring audio and video latencies in virtual desktop environments |
US11816364B2 (en) * | 2022-01-13 | 2023-11-14 | Microsoft Technology Licensing, Llc | Performance evaluation of an application based on detecting degradation caused by other computing processes |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020138846A1 (en) * | 2001-01-11 | 2002-09-26 | Masami Mizutani | Communication system |
US20070271375A1 (en) * | 2004-09-27 | 2007-11-22 | Symphoniq Corporation | Method and apparatus for monitoring real users experience with a website capable of using service providers and network appliances |
Family Cites Families (74)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TW346391B (en) | 1996-09-20 | 1998-12-01 | Kuraray Co | Method of polymerizing photo-polymerizable composition for dental use and dental light-curing apparatus for use therewith |
US5933155A (en) * | 1996-11-06 | 1999-08-03 | Silicon Graphics, Inc. | System and method for buffering multiple frames while controlling latency |
US6029171A (en) | 1997-02-10 | 2000-02-22 | Actioneer, Inc. | Method and apparatus for group action processing between users of a collaboration system |
US7756892B2 (en) | 2000-05-02 | 2010-07-13 | Digimarc Corporation | Using embedded data with file sharing |
AR019458A1 (en) * | 1998-07-23 | 2002-02-20 | United Video Properties Inc | AN INTERACTIVE TELEVISION PROGRAMMING GUIDE PROVISION THAT SERVES AS AN ENTRY |
US6633685B1 (en) | 1998-08-05 | 2003-10-14 | Canon Kabushiki Kaisha | Method, apparatus, and storage media for image processing |
CZ2001721A3 (en) | 1998-08-27 | 2002-05-15 | International Business Machines Corporation | System and method for introducing additional information into video data |
US6357042B2 (en) | 1998-09-16 | 2002-03-12 | Anand Srinivasan | Method and apparatus for multiplexing separately-authored metadata for insertion into a video data stream |
US6618431B1 (en) | 1998-12-31 | 2003-09-09 | Texas Instruments Incorporated | Processor-based method for the acquisition and despreading of spread-spectrum/CDMA signals |
US6381362B1 (en) | 1999-04-08 | 2002-04-30 | Tata America International Corporation | Method and apparatus for including virtual ads in video presentations |
US6415438B1 (en) | 1999-10-05 | 2002-07-02 | Webtv Networks, Inc. | Trigger having a time attribute |
JP2001166209A (en) | 1999-12-07 | 2001-06-22 | Olympus Optical Co Ltd | Image optical system |
US6961765B2 (en) | 2000-04-06 | 2005-11-01 | Bbx Technologies, Inc. | System and method for real time monitoring and control of networked computers |
CN101820474B (en) | 2000-11-30 | 2013-11-06 | 因特拉松尼克斯有限公司 | Communication system |
US7774817B2 (en) | 2001-01-31 | 2010-08-10 | Microsoft Corporation | Meta data enhanced television programming |
US7155681B2 (en) | 2001-02-14 | 2006-12-26 | Sproqit Technologies, Inc. | Platform-independent distributed user interface server architecture |
US20020165757A1 (en) * | 2001-05-01 | 2002-11-07 | Lisser Charles Steven | Systems, methods and computer program products for comparing business performance |
US7287275B2 (en) | 2002-04-17 | 2007-10-23 | Moskowitz Scott A | Methods, systems and devices for packet watermarking and efficient provisioning of bandwidth |
US7706402B2 (en) | 2002-05-06 | 2010-04-27 | Ikanos Communications, Inc. | System and method for distributed processing of packet data containing audio information |
US20040086120A1 (en) | 2002-11-06 | 2004-05-06 | Akins Glendon L. | Selecting and downloading content to a portable player |
FI116016B (en) | 2002-12-20 | 2005-08-31 | Oplayo Oy | a buffering |
JPWO2004093448A1 (en) | 2003-04-17 | 2006-07-13 | 日本電信電話株式会社 | Digital watermarked video transmission system, digital watermarked video transmission method, information processing device, communication control device, digital watermarked video processing program, and storage medium storing digital watermarked video processing program |
US20040221315A1 (en) | 2003-05-01 | 2004-11-04 | Genesis Microchip Inc. | Video interface arranged to provide pixel data independent of a link character clock |
KR100499037B1 (en) | 2003-07-01 | 2005-07-01 | 엘지전자 주식회사 | Method and apparatus of dtv lip-sync test |
JP2005070738A (en) | 2003-08-04 | 2005-03-17 | Casio Comput Co Ltd | Imaging apparatus and method, and program |
US7395339B2 (en) | 2003-08-07 | 2008-07-01 | International Business Machines Corporation | Method and system for providing on-demand media streaming from a user's own library to a receiving device of the user |
WO2005054993A2 (en) | 2003-11-25 | 2005-06-16 | Computer Associates Think, Inc. | Web service performance index |
US7480698B2 (en) * | 2003-12-18 | 2009-01-20 | International Business Machines Corporation | Updating event data on a page using code understood natively by a browser |
US7532642B1 (en) | 2004-03-11 | 2009-05-12 | Sun Microsystems, Inc. | Methods and apparatus supporting adaptive bandwidth management |
JP4218573B2 (en) | 2004-04-12 | 2009-02-04 | ソニー株式会社 | Noise reduction method and apparatus |
KR20070085208A (en) | 2004-08-17 | 2007-08-27 | 캘리포니아 인스티튜트 오브 테크놀로지 | Method and apparatus for network congestion control using queue control and one-way delay measurements |
US7752325B1 (en) | 2004-10-26 | 2010-07-06 | Netapp, Inc. | Method and apparatus to efficiently transmit streaming media |
JP2006293284A (en) | 2005-03-14 | 2006-10-26 | Fujitsu Ltd | System for recovering mask, and recovery method |
US20070008108A1 (en) | 2005-07-07 | 2007-01-11 | Schurig Alma K | Unsynchronized beacon location system and method |
FR2889014A1 (en) | 2005-07-22 | 2007-01-26 | Nextamp Sa | METHOD AND DEVICE FOR TATOOTING HORODATES, METHOD AND DEVICE FOR DECODING HORODATES, APPLICATIONS AND CORRESPONDING COMPUTER PROGRAM PRODUCTS |
US7942340B2 (en) | 2005-11-24 | 2011-05-17 | Canon Kabushiki Kaisha | Two-dimensional code, and method and apparatus for detecting two-dimensional code |
US7593543B1 (en) | 2005-12-15 | 2009-09-22 | Nvidia Corporation | Apparatus, system, and method for tracing distribution of video content with video watermarks |
US7552467B2 (en) | 2006-04-24 | 2009-06-23 | Jeffrey Dean Lindsay | Security systems for protecting an asset |
US9432433B2 (en) | 2006-06-09 | 2016-08-30 | Qualcomm Incorporated | Enhanced block-request streaming system using signaling or block creation |
US7613949B1 (en) | 2006-06-30 | 2009-11-03 | Boone Lewis A | Fault isolation system and method |
EP1879347B1 (en) | 2006-07-14 | 2012-05-30 | Sony Europe Limited | System and method of audio/video streaming |
US8660152B2 (en) | 2006-09-25 | 2014-02-25 | Futurewei Technologies, Inc. | Multi-frame network clock synchronization |
US7693190B2 (en) | 2006-11-22 | 2010-04-06 | Cisco Technology, Inc. | Lip synchronization for audio/video transmissions over a network |
US8059207B2 (en) | 2007-02-23 | 2011-11-15 | Samsung Electronics Co., Ltd. | System and method for video noise reduction using an adaptive temporal method with motion detection and motion compensation |
US7969928B2 (en) | 2007-03-02 | 2011-06-28 | Motorola Solutions, Inc. | Method and apparatus for battery-aware dynamic bandwidth allocation for wireless sensor networks |
US9083943B2 (en) | 2007-06-04 | 2015-07-14 | Sri International | Method for generating test patterns for detecting and quantifying losses in video equipment |
US8230226B2 (en) | 2007-08-17 | 2012-07-24 | Intel Corporation | Advanced watermarking system and method |
KR101505519B1 (en) | 2007-10-16 | 2015-03-25 | 삼성전자 주식회사 | Apparatus and method for providing contents |
US8560634B2 (en) | 2007-10-17 | 2013-10-15 | Dispersive Networks, Inc. | Apparatus, systems and methods utilizing dispersive networking |
KR101178752B1 (en) | 2008-02-26 | 2012-09-10 | 브이엠웨어, 인코포레이티드 | Extending server-based desktop virtual machine architecture to client machines |
US8406254B2 (en) | 2008-04-11 | 2013-03-26 | Mobitv, Inc. | Network optimized distribution |
US8677399B2 (en) | 2008-04-15 | 2014-03-18 | Disney Enterprises, Inc. | Preprocessing video to insert visual elements and applications thereof |
US20090268709A1 (en) | 2008-04-23 | 2009-10-29 | Motorola, Inc. | Time and frequency correction for an access point in an ofdma communication system |
EP2261896B1 (en) | 2008-07-29 | 2017-12-06 | Yamaha Corporation | Performance-related information output device, system provided with performance-related information output device, and electronic musical instrument |
US8428301B2 (en) | 2008-08-22 | 2013-04-23 | Dolby Laboratories Licensing Corporation | Content identification and quality monitoring |
US8347344B2 (en) | 2008-12-18 | 2013-01-01 | Vmware, Inc. | Measuring remote video playback performance with embedded encoded pixels |
US7831661B2 (en) | 2008-12-18 | 2010-11-09 | Vmware, Inc. | Measuring client interactive performance using a display channel |
US8788079B2 (en) | 2010-11-09 | 2014-07-22 | Vmware, Inc. | Monitoring audio fidelity and audio-video synchronization |
US9214004B2 (en) | 2008-12-18 | 2015-12-15 | Vmware, Inc. | Watermarking and scalability techniques for a virtual desktop planning tool |
US8234236B2 (en) | 2009-06-01 | 2012-07-31 | International Business Machines Corporation | System and method for efficient allocation of resources in virtualized desktop environments |
US20120066711A1 (en) | 2009-08-24 | 2012-03-15 | Novara Technology, LLC | Virtualized home theater service |
US8644374B2 (en) | 2009-08-31 | 2014-02-04 | Cisco Technology, Inc. | Multiple description coding with spatial shifting |
US8780978B2 (en) | 2009-11-04 | 2014-07-15 | Qualcomm Incorporated | Controlling video encoding using audio information |
US8699351B2 (en) | 2009-12-04 | 2014-04-15 | At&T Intellectual Property I, L.P. | Method and system for detecting audio and video synchronization |
US8769428B2 (en) | 2009-12-09 | 2014-07-01 | Citrix Systems, Inc. | Methods and systems for generating a combined display of taskbar button group entries generated on a local machine and on a remote machine |
US20110225513A1 (en) | 2010-03-09 | 2011-09-15 | Avistar Communications Corporation | Scalable high-performance interactive real-time media architectures for virtual desktop environments |
US8918499B2 (en) | 2010-08-09 | 2014-12-23 | International Business Machines Corporation | Method and system for end-to-end quality of service in virtualized desktop systems |
US20120073344A1 (en) | 2010-09-28 | 2012-03-29 | Mario Fabris | Split cooler for a steel mill |
US8629913B2 (en) * | 2010-09-30 | 2014-01-14 | Apple Inc. | Overflow control techniques for image signal processing |
US8910228B2 (en) | 2010-11-09 | 2014-12-09 | Vmware, Inc. | Measurement of remote display performance with image-embedded markers |
US9336117B2 (en) | 2010-11-09 | 2016-05-10 | Vmware, Inc. | Remote display performance measurement triggered by application display upgrade |
KR101521478B1 (en) | 2010-12-07 | 2015-05-19 | 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 | Audio fingerprint differences for end-to-end quality of experience measurement |
US20120167145A1 (en) | 2010-12-28 | 2012-06-28 | White Square Media, LLC | Method and apparatus for providing or utilizing interactive video with tagged objects |
CA2734860A1 (en) * | 2011-03-21 | 2012-09-21 | Calgary Scientific Inc. | Method and system for providing a state model of an application program |
-
2013
- 2013-04-26 US US13/872,040 patent/US9336117B2/en active Active
-
2016
- 2016-05-09 US US15/150,119 patent/US9578373B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020138846A1 (en) * | 2001-01-11 | 2002-09-26 | Masami Mizutani | Communication system |
US20070271375A1 (en) * | 2004-09-27 | 2007-11-22 | Symphoniq Corporation | Method and apparatus for monitoring real users experience with a website capable of using service providers and network appliances |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9674562B1 (en) | 2008-12-18 | 2017-06-06 | Vmware, Inc. | Quality evaluation of multimedia delivery in cloud environments |
US10453161B2 (en) | 2008-12-18 | 2019-10-22 | Vmware, Inc. | Watermarking and scalability techniques for a virtual desktop planning tool |
US10623789B1 (en) | 2011-03-14 | 2020-04-14 | Vmware, Inc. | Quality evaluation of multimedia delivery in cloud environments |
Also Published As
Publication number | Publication date |
---|---|
US9578373B2 (en) | 2017-02-21 |
US9336117B2 (en) | 2016-05-10 |
US20140325054A1 (en) | 2014-10-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9578373B2 (en) | Remote display performance measurement triggered by application display upgrade | |
US8910228B2 (en) | Measurement of remote display performance with image-embedded markers | |
US7831661B2 (en) | Measuring client interactive performance using a display channel | |
US9674518B2 (en) | Measuring remote video display with embedded pixels | |
US9214004B2 (en) | Watermarking and scalability techniques for a virtual desktop planning tool | |
US9146884B2 (en) | Push pull adaptive capture | |
EP2663925B1 (en) | A method and mechanism for performing both server-side and client-side rendering of visual data | |
EP2248101B1 (en) | Graphics remoting architecture | |
US9674265B2 (en) | Filtering unnecessary display updates for a networked client | |
US8467631B2 (en) | Method and apparatus for identification of image uniqueness | |
US8600155B2 (en) | Classification and encoder selection based on content | |
US8924507B2 (en) | Cross-frame progressive spoiling support for reduced network bandwidth usage | |
US9201755B2 (en) | Real-time, interactive measurement techniques for desktop virtualization | |
US8872835B2 (en) | Prevention of DoS attack by a rogue graphics application | |
TWI495330B (en) | System and method for efficiently streaming digital video | |
US8055616B2 (en) | Application sharing smoothness | |
TW201426325A (en) | Method and system of displaying frame | |
Yadav et al. | Adaptive GPU resource scheduling on virtualized servers in cloud gaming | |
US9866724B2 (en) | Pixel perturbation for transmission of meta-information | |
US20150350295A1 (en) | System And Method For Loading Assets During Remote Execution | |
US9842387B2 (en) | Pixel perturbation for image quality measurement | |
CN115357340A (en) | Device and method for improving performance of virtual equipment | |
US8825739B2 (en) | Method and apparatus for controlling multiple systems in a low bandwidth environment | |
CN111506374A (en) | Cloud application UI interaction method and device based on different objects | |
WO2010150298A1 (en) | Image transmitting device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: VMWARE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:VMWARE, INC.;REEL/FRAME:067103/0030 Effective date: 20231121 |
|
AS | Assignment |
Owner name: UBS AG, STAMFORD BRANCH, CONNECTICUT Free format text: SECURITY INTEREST;ASSIGNOR:OMNISSA, LLC;REEL/FRAME:068118/0004 Effective date: 20240701 |