EP3292677B1 - Ultra-low latency remote application access - Google Patents
Ultra-low latency remote application access Download PDFInfo
- Publication number
- EP3292677B1 EP3292677B1 EP16790115.6A EP16790115A EP3292677B1 EP 3292677 B1 EP3292677 B1 EP 3292677B1 EP 16790115 A EP16790115 A EP 16790115A EP 3292677 B1 EP3292677 B1 EP 3292677B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- cursor
- ultra
- remote
- low latency
- browser
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 claims description 36
- 238000007906 compression Methods 0.000 claims description 26
- 230000006835 compression Effects 0.000 claims description 23
- 230000004044 response Effects 0.000 claims description 5
- 230000003139 buffering effect Effects 0.000 claims description 4
- 230000006854 communication Effects 0.000 description 21
- 238000004891 communication Methods 0.000 description 21
- 230000011664 signaling Effects 0.000 description 20
- 241000699666 Mus <mouse, genus> Species 0.000 description 19
- 238000009877 rendering Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 15
- 230000008859 change Effects 0.000 description 12
- 230000007246 mechanism Effects 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000004043 responsiveness Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000004091 panning Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 3
- 238000012163 sequencing technique Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000011960 computer-aided design Methods 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000010454 slate Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
- H04L65/613—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for the control of the source by the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/048—Indexing scheme relating to G06F3/048
- G06F2203/04806—Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
Definitions
- the classic access method is to have dedicated hardware, such as a remote terminal, or specialized software that would run on a remote device.
- the software for the remote device would communicate with a server and allow bidirectional communication.
- the remote device would have to download and install a native client application.
- the native client application would capture operating system-level input, such as mouse location and keyboard inputs, and transmit the information to the application server.
- the application server would treat the inputs from the client application as if the user were interacting with the application server directly.
- a remote application access system has a problem of latency, which may be perceived as responsiveness by the end user. In any remote access situation, there will be additional latencies from network connection issues.
- An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device.
- the remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience.
- the remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs.
- Executable code on the application server may encode screen images into video, which may be transmitted and rendered on the client device. In some cases, the rendered video may be larger than the displayable area of the client device, allowing for panning and zooming to be performed on the client device.
- An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device.
- the remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience.
- the remote access system may give a user the experience of operating a remote application with an experience similar to having the application operating natively on the client device.
- a powerful remote application server may be able to process and render complex applications that would not be possible or practical to implement on a client device.
- a remote application server may be executing a 3D Computer Aided Design (CAD) program, which may be accessed and controlled by a tablet computer.
- CAD Computer Aided Design
- the remote application server may have much more processing and rendering capability than the tablet computer, yet the user may interact with the 3D CAD program as if the program were running natively.
- the user experience simulates a native application. This is because the cursor movement comes from the operating system of the client device, where the responsiveness is the same as a native application.
- cursor location information may be captured on the client device, sent to the remote server, rendered on the remote server, then the rendering transferred to the client device.
- the responsiveness of the cursor may be severely limited by the round-trip travel time to the remote server and back.
- Rendering the cursor natively may eliminate the round-trip lag time or latency, which can give a user a much more comfortable experience, even if the remote application image updates still have the lag time of the round-trip.
- the remote access system may generate a video stream on the remote application server and transmit the video stream to a browser on the client device.
- the remote application server may render images that may be captured by a remote application service.
- the service may generate a video stream that may be sent to the client device.
- a video stream may be useful to send rendered images, as several video compression technologies can be used to compress and transmit the video stream.
- video streaming technologies are generally designed with inherent buffering technologies, any of which may add to latency. Some video compression technologies may be tuned to minimize buffering or other causes of latency.
- many video compression technologies may rely on differences between two or more images to eliminate the similar areas of the images.
- Such technologies inherently buffer multiple frames to achieve the compression goals, but in doing so introduce latency for each of the buffered frames.
- latency may be immaterial, but in a real-time or near-real time system where a user may be attempting to control a remote application through a graphical user interface, such latency may be detrimental.
- video streaming may be assisted by using hardware or software codecs on the remote application server and, in some cases, on the client device.
- codecs may operate without taking much, if any, processing cycles from either a central processing unit (CPU) or graphical processing unit (GPU).
- CPU central processing unit
- GPU graphical processing unit
- a hardware codec may not be available and a CPU or GPU may be employed for encoding or decoding a video stream.
- the remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs.
- the browser executable code may be JavaScript code that may be downloaded in a Hyper Text Markup Language (HTML) document.
- HTML Hyper Text Markup Language
- any browser can be used on any device to execute any application on a remote access server. This can be done without creating, supporting, and downloading native applications that would be specific to each hardware and software platform.
- the client-side browser-executable code may make deploying the remote access system much easier than when using a client-side native application.
- the rendered image on the remote application server may be sized to match the displayable area of a client's browser.
- the rendered image may be a pixel-to-pixel match to the client device, while in other cases, the rendered image may be larger or smaller than the displayable area of the client's browser.
- a pixel-to-pixel relationship between the rendered image on a remote application server and a displayable area of a client's browser may be useful in situations to maximize the resolution of the image.
- an application that may contain small text may remain readable and useful with a pixel-to-pixel relationship.
- a non-pixel-to-pixel relationship between the rendered image on a remote application server and the displayed image on a client device may be useful in several situations.
- a lower resolution rendered image may generate a video image that may be smaller than what may be rendered on a client device.
- Such systems may upscale the displayed video to match the viewable space on a client device.
- Such a configuration may be useful in low bandwidth situations as well as when the resolution of the image may not be of high importance.
- a real-time action game may be rendered in a lower resolution than may be displayed as a lower resolution of the rendered image may not take away from game play.
- a higher resolution rendered image may generate a video image that may be larger than what may be rendered on a client device. Such an example may be useful when the client device may be capable of panning the video while viewing, or when the client device may downscale the video image prior to viewing.
- the browser-based control of a remote computer may enable a multiuser system, where one user may operate a remote computer while other users passively view the video provided from the remote computer.
- two or more users may be able to control cursor movement and make cursor-related inputs to the remote computer.
- two or more users may collaborate by operating the same application on the remote device.
- Many such systems may have a shared audio channel, which may be delivered along with the video stream, or may be a separate channel. Some such systems may connect users through the telephone system for audio while the video and cursor controls may be provided through a digital data network connection.
- references to "a processor” include multiple processors. In some cases, a process that may be performed by “a processor” may be actually performed by multiple processors on the same device or on different devices. For the purposes of this specification and claims, any reference to “a processor” shall include multiple processors, which may be on the same device or different devices, unless expressly specified otherwise.
- the subject matter may be embodied as devices, systems, methods, and/or computer program products. Accordingly, some or all of the subject matter may be embodied in hardware and/or in software (including firmware, resident software, micro-code, state machines, gate arrays, etc.) Furthermore, the subject matter may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in connection with an instruction execution system.
- a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
- the computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.
- computer readable media may comprise computer storage media and communication media.
- Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
- Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by an instruction execution system.
- the computer-usable or computer-readable medium could be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, of otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
- the embodiment may comprise program modules, executed by one or more systems, computers, or other devices.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- functionality of the program modules may be combined or distributed as desired in various embodiments.
- FIG. 1 is a diagram illustration of an embodiment 100 showing an example system where a browser may be used to interactively control a remote computer.
- a server 102 may be controlled by a client 104, where an operating system desktop 108 and application window 110 may be rendered on the server, and the browser 112 may display at least a portion of the operating system desktop 108.
- the browser may capture and transmit keyboard, mouse, and other inputs to the server 102, which may send the inputs to the application on the server 102.
- the renderings from the display of the server 102 may be captured by an image capture component 114 and encoded with a video codec 116 to create a video stream.
- the video stream may be passed across a network 118 to the client 104, which may receive the video stream, decode the video stream with a video codec 120, and display the video stream using a video player 112 onto the browser window.
- Real time or “near-real time” control of an application on a server 102 by another device, in this case client 104 may introduce various complexities.
- the user may make an input, such as a mouse click, keyboard input, or other input, and the user would experience instantaneous response.
- the fact that the two devices communicate with each other introduces latencies, as well as the availability of limited or variable network connections.
- Video compression and transmission techniques are widely used, and any appropriate format, compression algorithms, codecs, and other protocols may be used. Video capture and compression mechanisms may be lossy, where some of the information may be lost during the compression routine, while other mechanisms may not be lossy.
- Video compression and transmission techniques may introduce latency into the system.
- Many video compression techniques may use data from one or more frames to compress information in later frames. Such techniques may inherently capture two, three, or more frames, perform compression using the frames, then transmit the video.
- One mechanism to reduce latencies may be to select a video compression and encoding technique that uses a minimum of frames during the compression process.
- Some video compression and encoding mechanisms may have configuration settings that may permit adjusting the encoding techniques.
- Video compression mechanisms that have been successfully implemented with such a system include H262, H263, H264, and others.
- a second mechanism by which user-perceived latency may be reduced may be to have cursor movement and rendering performed on the user's local device, rather than on a remote device. From the user's standpoint, the motion and feel of a cursor and its responsiveness to movement may give a perceived "real time" feel, even when the cursor inputs have a latency when they are transmitted and processed on a remote system.
- Local rendering of the cursor can cause the cursor to move with the same responsiveness within a browser window as with any other application on a client device.
- a responsiveness may give an acceptable experience, and in some cases, users may not realize that the operations performed within a browser window may actually be performed and rendered on a remote device.
- Embodiment 100 may illustrate a system where cursor information is not included in the video transmitted from the server 102, and where the cursor information may be rendered by the client 104.
- a cursor manager 126 may be capable of removing any rendered cursor from an image captured by the image capture component 114. Some such systems may capture a bitmap of the rendered screen, identify the cursor location, and change the bitmap to remove the cursor.
- the operating system of the server 102 may be configured so that the cursor may be very small or not rendered, which may obviate the step of removing a cursor from the rendered image.
- a cursor manager 126 may identify the specific cursor being used by the application or operating system of the server at a specific instance.
- the cursor manager 126 may transmit an image of the cursor in a bitmap or other form to the client 104.
- the client 104 may have a cursor manager 132 which may cause the cursor image to be rendered locally on the client 104.
- Some applications may have multiple cursors that may change based on the context of the cursor, a tool that may be selected, or some other feature of the application.
- an image editing system may have a cursor in the shape of an eyedropper for sampling colors, a bracket cursor used for editing text, a crosshair cursor for selecting a rectangular box, and so forth.
- the cursor manager 126 may identify the cursor and relay the cursor information over the server signaling channel 124 to the client 104.
- the cursor manager 126 may send an identifier for a specific cursor, and the client 104 may use one of several stored cursors 134 that corresponds to the identifier. Such systems may transmit an image of a cursor the first time that cursor may be encountered, then each time thereafter, may merely send the identifier instead of the image.
- the client 104 may have a cursor manager 132 that may cause the local cursor to change to a different cursor in response to a change in the cursor identifier.
- a controller 136 on the server 102 may use a server signaling channel 124 and a client signaling channel 144 to communicate with the client 104 during setup, tear down, and normal operations.
- the client 104 may have a server signaling channel 130 and a client signaling channel 140 connected to a client controller 138.
- the signaling channels may be illustrated here as separate communications channels.
- the channels may be separate sockets, ports, protocols, or be otherwise separate from the channel through which video may be transferred.
- some or all of the server signaling channel communications may be embedded or encoded into the video stream, thereby having a single port, channel, protocol, socket, or other channel from the server 102 to the client 104.
- the client 104 to server 102 communications may be a separate channel, port, protocol, socket, or other mechanism than from the server signaling channel.
- the client signaling channel may be a duplex version of the server signaling channel, where both directions of communication may use the same port, channel, protocol, socket, or other common mechanism.
- the client 104 may have a mechanism to capture mouse, keyboard, and other input 142. These inputs may be captured and transmitted through the client signaling channel 140 to the server 102.
- the server 102 may receive the data through the client signaling channel, and may provide the input to the operating system 146. These inputs may be passed to the application executing within the operating system and thereby the user of the browser 112 may operate an application on the server 102.
- the inputs collected on the client 104 may include any type of input, including keyboard, mouse, audio, as well as sensory input from gyroscopes, global positioning system (GPS) sensors, accelerometers, temperature sensors, pressure sensors, light sensors, and any other type of sensor that may be available on the client 104.
- the inputs may include camera input that may be collected from a camera mounted on the client 104 or connected to the client 104. The camera may be able to capture still images or video, and such inputs may be transmitted over the client signaling channel 140.
- video or audio inputs such inputs may be encoded using a codec and transmitted to the server 102 for decoding prior to use.
- the example of embodiment 100 may illustrate a system where a browser 112 may render the contents of an application window 110.
- This example is merely one use case, such as where the server 102 may be configured just to provide a specific application through the browser 112.
- Other systems may display multiple applications and even the entire operating system desktop 108 in the browser 112. Such applications may allow for the entire server 102 to be controlled by the client 104 through the browser 112.
- Figure 2 is a diagram of an embodiment 200 showing components that may be used for remote access of an application or system through a browser.
- the diagram of Figure 2 illustrates functional components of a system.
- the component may be a hardware component, a software component, or a combination of hardware and software.
- Some of the components may be application level software, while other components may be execution environment level components.
- the connection of one component to another may be a close connection where two or more components are operating on a single hardware platform. In other cases, the connections may be made over network connections spanning long distances.
- Each embodiment may use different hardware, software, and interconnection architectures to achieve the functions described.
- Embodiment 200 illustrates a device 202 that may have a hardware platform 204 and various software components.
- the device 202 as illustrated represents a conventional computing device, although other embodiments may have different configurations, architectures, or components.
- the device 202 may be a server computer. In some embodiments, the device 202 may still also be a desktop computer, laptop computer, netbook computer, tablet or slate computer, wireless handset, cellular telephone, game console or any other type of computing device. In some embodiments, the device 202 may be implemented on a cluster of computing devices, which may be a group of physical or virtual machines.
- the hardware platform 204 may include a processor 208, random access memory 210, and nonvolatile storage 212.
- the hardware platform 204 may also include a user interface 214 and network interface 216.
- the random access memory 210 may be storage that contains data objects and executable code that can be quickly accessed by the processors 208.
- the random access memory 210 may have a high-speed bus connecting the memory 210 to the processors 208.
- the nonvolatile storage 212 may be storage that persists after the device 202 is shut down.
- the nonvolatile storage 212 may be any type of storage device, including hard disk, solid state memory devices, magnetic tape, optical storage, or other type of storage.
- the nonvolatile storage 212 may be read only or read/write capable.
- the nonvolatile storage 212 may be cloud based, network storage, or other storage that may be accessed over a network connection.
- the user interface 214 may be any type of hardware capable of displaying output and receiving input from a user.
- the output display may be a graphical display monitor, although output devices may include lights and other visual output, audio output, kinetic actuator output, as well as other output devices.
- Conventional input devices may include keyboards and pointing devices such as a mouse, stylus, trackball, or other pointing device.
- Other input devices may include various sensors, including biometric input devices, audio and video input devices, and other sensors.
- the network interface 216 may be any type of connection to another computer.
- the network interface 216 may be a wired Ethernet connection.
- Other embodiments may include wired or wireless connections over various communication protocols.
- the software components 206 may include an operating system 218 on which various software components and services may operate.
- Various applications 220 may execute within the operating system environment. Some or all of the applications 220 may be made available for remote control through a client device 252 and a browser 258.
- the operating system 218 may have a rendering engine 222 which may render displayable images based on input from the operating system 218 and various applications 220.
- the rendering engine 222 may be an operating system function that may use a graphics processing unit 224, which may be a hardware or software accelerator that may assist in rendering graphics.
- the server 202 may have a server side executable 226, which may be a set of software components that may permit external control of the entire system or a subset of the system through a client device 252.
- the server side executable 226 is illustrated as a set of software components as a matter of convenience, but some or all of the components may be implemented in hardware.
- An image capture component 228 may capture the rendered images from the rendering engine 222.
- the image capture component 228 may be an operating system function in some systems or may be a separate component that may intercept rendered images that may be displayed on a monitor.
- a cursor remover component 230 may be used in systems that may render a cursor within the displayable image. Some systems may not have a cursor remover component 230 and may include the rendered cursor in the video stream. In such a case, the client device 252 may or may not have a separately rendered cursor.
- the video stream includes a rendered cursor and when the client device also has a separately rendered cursor, a user may experience two cursors on the screen.
- the cursor rendered on the server 202 and embedded in the video stream may not be as responsive as the cursor rendered by the client 104, however, in most cases, the client-rendered cursor may overlap or hide the server-rendered cursor.
- a user may experience that the server-rendered cursor may slightly lag behind the movement of the client-rendered cursor.
- a server 202 may render a cursor and the client device 252 may be configured not to render a cursor over the browser 258.
- the user interaction with the browser 258 may generate a location where a cursor may be at any given time, and the location may be transmitted to the server 202, then rendered in the video stream viewed on the client device 252.
- Such systems may have some latency or lag that may be experienced by the user of the client device 252. In cases where the network and video processing latencies are not exceptionally high, the user experience may still be acceptable.
- a server 202 may be configured to not render a cursor at all in the image generated by the rendering engine. In such cases, the rendered image may not have a cursor, so the only cursor a user of the client 252 may experience may be the cursor rendered by the client 252.
- An image sequence from the image capture component 228 may be sent to a video codec 236 and then communicated to the client device 252 by a communications manager 238.
- the communications manager 238 may handle a server signaling channel 240, which may send control information and other information to the client device 252.
- the server signaling channel 240 may be used to send cursor information to the client device 252.
- the cursor information may include the location of a cursor as well as the type of cursor being displayed.
- the cursor type may be an identifier for a specific cursor.
- the server signaling channel 240 may also be used to send a bitmap or other image of a cursor for rendering on the client device 252.
- the client device 252 may request a cursor image when a specified cursor may not be available on the client device 252, and the cursor manager 232 may be able to send the requested cursor from a group of stored cursors 234.
- a controller 242 may manage various administrative functions for the client/server interaction.
- the administrative functions may include setup and teardown of a session, monitoring and adjustment of communications settings, and other functions.
- the controller 242 of the server 202 may operate in conjunction with controller 272 of the client, and in some cases, either controller 242 or 272 may initiate or respond to one of the various administrative functions.
- controllers 242 or 272 provide authentication and configuration of a session.
- a client device 252 may request a connection from the server 202 or may be routed to a server 202 after making a request to a third device, such as a connection management system 280.
- the controller 242 may set up the applications 220 prior to establishing a video stream.
- a new instance of an application may be established and, when the instance becomes stable, a video stream may be created.
- a single server 202 may provide separate video connections to multiple devices, each device having a separate session with different instances of an application or with multiple devices sharing the same instance or session with a single instance of an application.
- a setup operation may involve starting a new instance of an application and configuring the application based on parameters or data that may be provided with a request for a new session. For example, an application may be configured to resume operations from a previously saved session, or may be started with a set of options requested by a user.
- Teardown of a session may involve terminating an application instance or reconfiguring the server 202 in some manner in preparation for a new session.
- the server 202 may maintain its configuration and various application instances in case the session termination was inadvertent.
- the server 202 may terminate one or more applications and free up resources for new sessions that may be requested at a later time.
- the controllers 242 and 272 may monitor and adjust communications settings due to network or other performance issues. For example, the controllers may identify a change in the network bandwidth, and may be able to adjust the video or other communications to take advantage of additional bandwidth or reduce transmissions to accommodate less bandwidth.
- the video communications may be adjusted by changing the video frame rate, density or compression of pixels within the images, or other parameter changes. The specific changes may depend on the specific video compression and encoding algorithm.
- the controllers 242 and 272 may be able to monitor and display connection quality and responsiveness to an end user, such as displaying a red/yellow/green light or bar graph for connection strength. When a connection problem may be recognized, the controllers may be able to present a dialog box, warning, notice, or other message on the user's screen on the browser 258.
- the client signaling channel 246 may handle inbound communications from the client device 202.
- One element of inbound communications may be keyboard/mouse input 278 as well as other input captured by the client device's browser 258. These inputs may be communicated from the communications manager 264 on the client device 252 to the communications manager 238 on the server 202 and the client signaling channel 246. These inputs may be transferred as inputs to the operating system 248.
- the keyboard/mouse input 278 may include any type of input captured by the client device 252. These inputs may include audio, video, Global Positioning System (GPS), accelerometer input, orientation sensors, temperature and pressure sensors, and any other type of input that may be captured by the client device 252. In some cases, the input may be data gathered by or presented by applications executing on the client device 252 or accessed through the network 250 to some other device. For the sake of convenience, the term keyboard/mouse input 278 includes all of the various inputs listed above.
- GPS Global Positioning System
- the client device 252 may have a hardware platform 254 on which an operating system 256 may operate.
- the hardware platform 254 may include any of the various options and characteristics described for the hardware platform 204.
- a browser 258 may execute on the operating system 256.
- the browser may have a display window 260, as well as an executable engine 262.
- the executable engine 262 may be a virtual machine, emulator, interpreter, compiler, or any other mechanism by which instructions may be executed.
- many browsers include a Javascript engine that may execute Javascript that may be contained in an HTML document that a browser 258 may render.
- Incoming video data may be decoded by a video codec 266 and shown on the browser's display window 260 by a video player 268.
- a video player 268 may include controls for starting, stopping, rewinding, fast forwarding, or other functions.
- the browser may be used to control a remote device, such as the server 202, such controls may be turned off or otherwise unavailable to the user.
- a client side executable 270 may contain code that may be executed on the browser's executable engine 262.
- the client side executable 270 may be downloaded as part of an HTML page, and in some instances, the components of the client side executable 270 may be embedded in Javascript or some other executable language.
- the client side executable 270 may include a controller 272, a communications manager 264, a cursor manager 274, and stored cursors 276.
- the client side executable 270 may include routines or other components that may capture keyboard/mouse inputs 278.
- the cursor manager 274 may determine which cursor is to be shown at a particular time.
- the cursor manager 274 may receive a bitmap or other image of a cursor and store the image in a set of stored cursors 276.
- the cursor manager 274 may receive an identifier which may identify the current cursor.
- the identifier may be any type of identifier, such as a set of numbers, letter, hashed values, or other identifier.
- the cursor manager 274 may look up the corresponding cursor image from the stored cursors 276, and may pass the cursor image to the browser 258 for rendering.
- a certain cursor may be applicable to a specific area of a rendered application, while a second cursor may be applicable to a different area of the rendered application.
- the cursor manager 274 may receive a map or boundary area identifying a specific cursor to use while other areas of a displayed video image may have a different cursor.
- Some applications may have several cursor images that may apply to different areas or zones of a rendered video image.
- the network 250 may be any type of network that may allow the server 202, client device 252, and other devices to communicate.
- a connection management system 280 may be a device that may help set up sessions between client devices and servers.
- the connection management system 280 may operate on a hardware platform 282 with an operating system 284.
- a connection manager 286 may receive an inbound request from a client device 252 for a session.
- the connection manager 286 may use an authentication service 288 to determine whether or not the requestor may have permission to establish a session.
- a load balancer 290 may select between multiple instances of a server 202 to direct the client to establish a session.
- connection manager 286 may setup and configure an application 220 on a server 202 specifically for an inbound request, then give the address of a server 202 for establishing a connection.
- Some servers 202 may be configured so that multiple client devices may connect to a single session. In some such cases, one or more client devices may be configured to provide input to an application executing on the server 202, while other client devices may be configured to merely monitor, watch, or view the display window without providing control to the window. In some cases, control of an application on a server 202 may be changed from one device to another.
- a monitoring client 292 may have a hardware platform 294 on which an operating system 296 may operate.
- a browser 298 may connect to the communications manager 238 and may render a display window 299, which may be the same image as rendered on the browser 258 of the client device 252.
- certain inputs may be received from the monitoring client 292, such as audio, keyboard, mouse, or other inputs, and may be transmitted to the server 202.
- Such inputs may be transmitted as inputs to the operation system 248, although in other instances, some of the inputs generated by monitoring client 292 may be transmitted to the client device 252 for rendering and display within the browser 258.
- FIG. 3 is a flowchart illustration of an embodiment 300 showing a method of setting up a communications session.
- Embodiment 300 is a simplified example of a sequence showing a load balancer 302 in the left hand column, a server 304 in the center column, and a client 306 in the right hand column.
- Embodiment 300 may illustrate one method by which a new session may be created.
- a load balancer 302 may provide initial setup functions, including authentication and setup, then a session may be created between the server 304 and client 308 where the client 308 may control the server 304 or an application running on the server 304.
- a client 306 may send a request for a web page in block 308 to the load balancer 302.
- the load balancer 302 may receive a request in block 310, then request authentication credentials in block 312.
- the authentication request may be received by the client 306 in block 314, and may send the credentials in block 316.
- the load balancer 302 may receive authentication credentials in block 318 and verify the credentials in block 320. In some cases, the credentials may be passed to a separate authentication service. After verifying the credentials in block 320, the load balancer 302 may send a web page with server information and authentication token in block 322 to the client 306.
- the load balancer 302 may also send set up or configuration instructions to the server 304, while in other cases, such configuration instructions may be included in the web page with server information.
- a web page is a shorthand notation for instructions for how to access the server 304 from the client 306.
- such instructions may be embodied in an HTML document, which may include executable scripts or other instructions.
- the web page may be received in block 324 by the client 306.
- the client 306 may sent a request with the authentication token in block 326, which may be received by the server 304 in block 328.
- the server 304 may verify the token in block 330, and may send client executable code in block 332 to the client 306.
- the client 306 may receive the client executable code in block 334 and may begin execution in block 336.
- the server 304 may send a video stream in block 340 that may represent the operating system desktop of the server 304, an application executing on the server 304, or some other "live" image of the server 304.
- the video stream may be received and rendered in block 338.
- the client 306 may generate and send client data in block 346, which may include keyboard, mouse, and other inputs, which may be received by the server 304 in block 348.
- the keyboard, mouse, and other inputs may be used to control the application or operating system running on the server 304, and thereby a user of the client 306 may operate the server 304 through a browser.
- Figure 4 is a flowchart illustration of an embodiment 400 showing a method of handling cursors in a browser-control system of a remote server.
- Embodiment 400 is a simplified example of a sequence showing a server 402 in the left hand column and a client 404 in the right hand column.
- Embodiment 400 may illustrate a process that may be performed between a server 402 and client 404 when the client 404 may render a cursor image locally, but may be providing input to and controlling the server 402 or at least an application on server 402.
- An outbound video stream may be created in block 406 by the server 402 and transmitted to the client 404.
- the client 404 may receive the video stream in block 408 and may render the video stream within a browser.
- the client 404 may gather and send a cursor location to the server 402, which may receive the cursor location in block 412.
- the cursor location may also include interactions or commands that may be executed with the cursor location. Such commands may be a left-click, right-click, center-click, scrolling, or other location-sensitive commands.
- the cursor location may be sent to the operating system in block 414, along with any commands or other input.
- the cursor image generated by the server 402 may be intercepted in block 416.
- the operating system of the server 402 or an application executing on the server 402 may have a signaling system to transmit a cursor image that may be displayed at the location selected by the user.
- Some applications may have cursor images that may reflect a function, command, setting, or other operational characteristic of the application. Some applications may change cursor images based on the location of the cursor within the application, and such image changes may signify different available operations to the user.
- a cursor image may have one or more transparent pixels or bits.
- the transparent pixels or bits may be partially transparent.
- the server 402 may generate a cursor identifier for the cursor image.
- the cursor identifier may be any shorthand notation or identifier for a particular cursor image.
- One mechanism for identifying a cursor image may be to perform a hash function on the image and generate an alpha-numeric or numeric designation.
- the hash function may be selected such that changes to the image may result in changes to the identifier, thereby signaling that the cursor has changed.
- the cursor identifier may be looked up in the cursor database in block 422. If the cursor image does not exist in the cursor database in block 424, the server 402 may send the cursor image and identifier in block 426 to the client 404.
- the client 404 may receive the cursor and identifier in block 428 and may store the cursor and identifier in a local cursor storage in block 430.
- the client may cause the cursor image to be used for the current location of the cursor in block 432 and the process may loop back to block 410.
- the cursor identifier may be transmitted in block 434.
- the client 404 may receive the cursor identifier and may look up the cursor identifier in the cursor database in block 436. If the cursor identifier and corresponding image exists locally in the cursor database in block 438, the client may use the cursor image in block 440.
- the client 404 may request a cursor image in block 442, which may be received by the server 402 in block 444.
- the server 402 may return to block 426 and may transmit the cursor image and identifier.
- a cursor image may be defined for a specific region within a rendered area of a video stream. For example, when a cursor hovers over a text object, the cursor may be changed to a text editing cursor, but such a cursor may be only valid for the region of a text box.
- a map, overlay, or other description of where certain cursors may be valid may be generated on the server 402 and transmitted to the client 404 in some embodiments.
- the server 402 may generate a map of the rendered area and may define which cursors may be valid at various locations within the map. Some such systems may have a default cursor that may be used wherever a special cursor may not be defined.
- Figure 5 is a diagram illustration of an example embodiment 500 showing an over-sized video transmission from a server 502 to a client 504.
- the over-sized video transmission may be a higher resolution or larger size than may be displayed in a browser on the client 504.
- Embodiment 500 may illustrate one use case of an over-sized video transmission.
- An over-sized video transmission may be any configuration where the server 502 may transmit a video stream that may be larger than the displayed area on the browser of the client 504.
- the over-sized video in this example may be useful for panning and zooming the video stream rendered on the client 504.
- any zooming or panning of the video stream may be performed on the client 504, which may be faster and more responsive than if the zoom or pan commands were transmitted to the server 502 and updates made to the rendered video stream.
- a zoom command may expand or contract the displayed area or viewport rendered in a browser on the client 504.
- a pan command may allow the client 504 to move the displayed viewport horizontally or vertically within the rendered area of the video.
- Zoom and pan commands may be issued on the client 504 by a user.
- the user may have a mouse navigation mechanism, such as a special mouse button and movement commands, or a mouse movement in conjunction with a keyboard command, such as a shift key, alt key, or control key.
- a zoom or pan command may be issued by moving the device, such as through accelerometer inputs when a user may move a device, such as a mobile device, through space.
- a server 502 may generate a rendered area 506, which may be captured and encoded into video using a video encoder 508.
- the video stream may be transmitted over a network 510 to a client 504.
- the client 504 may have a video decoder 512 and may generate a rendered area 514.
- the rendered area 514 may be the same size as the rendered area 506 on the server.
- the client 504 may display a subset of the rendered area 514 as a displayed viewport 516.
- the displayed viewport 516 may be moved around the rendered area 514 by zoom and pan commands issued by a user.
- the client 504 may capture keyboard, mouse, and other inputs 518, which may be transmitted over the network 510 to the server 502.
- the server 502 may use the keyboard, mouse, and other inputs 520 to input to the application, causing the rendered image of the application or operating system to be updated.
- the cursor locations transmitted back and forth between the client 504 and server 502 may be translated to compensate for the difference between the client's displayed viewport 516 and the rendered area 514.
- the server 502 may receive cursor locations based on an absolute horizontal and vertical location from one of the corners of the rendered area 506.
- the cursor of the client 504 may be limited to be within the displayed viewport 516.
- the client 504 may translate the cursor input captured on a browser to the rendered area dimensions in the keyboard, mouse, or other input 518 that may be transmitted to the server 502.
- Figure 6 is a flowchart illustration of an embodiment 600 showing a method of negotiating and changing the rendered area of a video stream that may be transmitted and displayed on a client.
- Embodiment 600 is a simplified example of a sequence showing a server 602 in the left hand column and a client 604 in the right hand column.
- Embodiment 600 may illustrate one method by which the rendered area of a video stream generated by a server 602 may be selected and changed.
- a client 604 may determine the displayable area on a browser in block 606.
- a zoom/pan allowance may be determined in block 608, which may determine how much extra size of the video stream may be allocated for locally performed zoom and pan commands.
- no zoom or pan allowance may be used. Such applications may typically take up the entire browser area and may not zoom or pan within the browser screen. For other applications, notably first person games, the zoom or pan actions may be a large part of the application's operation. Such applications may be allocated a large zoom or pan allowance.
- the full size of a rendered area may be determined in block 610 from the zoom or pan allowance and the displayable area of the browser.
- a requested rendered area may be larger or smaller than the viewable browser area.
- some systems may be configured to downsample at the browser, where a larger rendered area may be transmitted and the video may be shrunk to meet the viewable area on the browser. Such configurations may be useful when the downsampling may offer increased resolution, especially with lossy video compression.
- the rendered area may be two times the dimensions of the browser viewport.
- a browser may be configured to upsample a video stream, such that the displayed area on the browser may be larger than the video rendered area. In such a situation, the browser may expand the video to fit the browser screen.
- Such configurations may be useful for applications where less precise rendering may be acceptable or where the network bandwidth may be limiting.
- the network bandwidth may be analyzed to determine whether a video stream of a certain size may be possible. Any adjustments to the requested rendered area may be made in block 614.
- the client 604 may transmit a request in block 616, which may be received in block 618 by the server 602.
- the server 602 and client 604 may negotiate the rendered area size in blocks 620 and 622, respectively.
- the negotiation may involve identifying certain predetermined sizes that may be optimized on the server 602 or other considerations.
- the server 602 may begin transmitting a video stream in block 624, which may be received and rendered in block 626.
- the server 602 may detect that a change to the rendered area may be advisable in block 628, the server 602 may initiate a negotiation in block 620.
- the process may go back to block 606 to consider various factors that may be changed.
- the client 604 may determine that a change to the rendered area may be advised under several circumstances. For example, the client 604 may determine that a user may be exceeding the limits of the available room within the rendered area for zoom or pan commands. In such a case, the client 604 may request that the rendered area be expanded.
- a client 604 may detect that network bandwidth may be decreasing, causing the client 604 to request a smaller rendered area.
- the user may reposition the browser on the client 604, which may cause the browser's displayed viewport to change size or aspect ratio. Such a change may indicate that a rendered area change may be in order.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Description
- Remote access to computer systems has been around since the original mainframe computers. The classic access method is to have dedicated hardware, such as a remote terminal, or specialized software that would run on a remote device. The software for the remote device would communicate with a server and allow bidirectional communication.
- For a typical remote access system to work, the remote device would have to download and install a native client application. The native client application would capture operating system-level input, such as mouse location and keyboard inputs, and transmit the information to the application server. The application server would treat the inputs from the client application as if the user were interacting with the application server directly.
- In most use cases, a remote application access system has a problem of latency, which may be perceived as responsiveness by the end user. In any remote access situation, there will be additional latencies from network connection issues.
- An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device. The remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience. The remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs. Executable code on the application server may encode screen images into video, which may be transmitted and rendered on the client device. In some cases, the rendered video may be larger than the displayable area of the client device, allowing for panning and zooming to be performed on the client device.
- This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
- In the drawings,
-
FIGURE 1 is a diagram illustration of an embodiment showing browser control of a remote computer. -
FIGURE 2 is a diagram illustration of an embodiment showing a network environment with browser control of a remote computer. -
FIGURE 3 is a flowchart illustration of an embodiment showing a method for setting up a communications session. -
FIGURE 4 is a flowchart illustration of an embodiment showing a method for managing cursors. -
FIGURE 5 is a diagram illustration of an embodiment showing an oversized video transmission for zoom/pan support. -
FIGURE 6 is a flowchart illustration of an embodiment showing a method for selecting and adjusting a rendered area. - An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device. The remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience.
- The remote access system may give a user the experience of operating a remote application with an experience similar to having the application operating natively on the client device. By having a remote access system, a powerful remote application server may be able to process and render complex applications that would not be possible or practical to implement on a client device.
- For example, a remote application server may be executing a 3D Computer Aided Design (CAD) program, which may be accessed and controlled by a tablet computer. The remote application server may have much more processing and rendering capability than the tablet computer, yet the user may interact with the 3D CAD program as if the program were running natively.
- By rendering cursor movements on the client device, the user experience simulates a native application. This is because the cursor movement comes from the operating system of the client device, where the responsiveness is the same as a native application.
- Compare such a system to one where cursor location information may be captured on the client device, sent to the remote server, rendered on the remote server, then the rendering transferred to the client device. In such a system, the responsiveness of the cursor may be severely limited by the round-trip travel time to the remote server and back.
- Rendering the cursor natively may eliminate the round-trip lag time or latency, which can give a user a much more comfortable experience, even if the remote application image updates still have the lag time of the round-trip.
- The remote access system may generate a video stream on the remote application server and transmit the video stream to a browser on the client device. The remote application server may render images that may be captured by a remote application service. The service may generate a video stream that may be sent to the client device.
- A video stream may be useful to send rendered images, as several video compression technologies can be used to compress and transmit the video stream. However, video streaming technologies are generally designed with inherent buffering technologies, any of which may add to latency. Some video compression technologies may be tuned to minimize buffering or other causes of latency.
- For example, many video compression technologies may rely on differences between two or more images to eliminate the similar areas of the images. Such technologies inherently buffer multiple frames to achieve the compression goals, but in doing so introduce latency for each of the buffered frames. In a conventional video streaming situation, such latency may be immaterial, but in a real-time or near-real time system where a user may be attempting to control a remote application through a graphical user interface, such latency may be detrimental.
- In many cases, video streaming may be assisted by using hardware or software codecs on the remote application server and, in some cases, on the client device. Such codecs may operate without taking much, if any, processing cycles from either a central processing unit (CPU) or graphical processing unit (GPU). In some cases, a hardware codec may not be available and a CPU or GPU may be employed for encoding or decoding a video stream.
- The remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs. The browser executable code may be JavaScript code that may be downloaded in a Hyper Text Markup Language (HTML) document.
- By embedding the client-side executable code in an HTML document, any browser can be used on any device to execute any application on a remote access server. This can be done without creating, supporting, and downloading native applications that would be specific to each hardware and software platform. The client-side browser-executable code may make deploying the remote access system much easier than when using a client-side native application.
- The rendered image on the remote application server may be sized to match the displayable area of a client's browser. In some cases, the rendered image may be a pixel-to-pixel match to the client device, while in other cases, the rendered image may be larger or smaller than the displayable area of the client's browser.
- A pixel-to-pixel relationship between the rendered image on a remote application server and a displayable area of a client's browser may be useful in situations to maximize the resolution of the image. For example, an application that may contain small text may remain readable and useful with a pixel-to-pixel relationship.
- A non-pixel-to-pixel relationship between the rendered image on a remote application server and the displayed image on a client device may be useful in several situations.
- For example, a lower resolution rendered image may generate a video image that may be smaller than what may be rendered on a client device. Such systems may upscale the displayed video to match the viewable space on a client device. Such a configuration may be useful in low bandwidth situations as well as when the resolution of the image may not be of high importance. For example, a real-time action game may be rendered in a lower resolution than may be displayed as a lower resolution of the rendered image may not take away from game play.
- In another example, a higher resolution rendered image may generate a video image that may be larger than what may be rendered on a client device. Such an example may be useful when the client device may be capable of panning the video while viewing, or when the client device may downscale the video image prior to viewing.
- The browser-based control of a remote computer may enable a multiuser system, where one user may operate a remote computer while other users passively view the video provided from the remote computer. In some cases, two or more users may be able to control cursor movement and make cursor-related inputs to the remote computer. In some such cases, two or more users may collaborate by operating the same application on the remote device. Many such systems may have a shared audio channel, which may be delivered along with the video stream, or may be a separate channel. Some such systems may connect users through the telephone system for audio while the video and cursor controls may be provided through a digital data network connection.
- Throughout this specification, like reference numbers signify the same elements throughout the description of the figures.
- When elements are referred to as being "connected" or "coupled," the elements can be directly connected or coupled together or one or more intervening elements may also be present. In contrast, when elements are referred to as being "directly connected" or "directly coupled," there are no intervening elements present.
- In the specification and claims, references to "a processor" include multiple processors. In some cases, a process that may be performed by "a processor" may be actually performed by multiple processors on the same device or on different devices. For the purposes of this specification and claims, any reference to "a processor" shall include multiple processors, which may be on the same device or different devices, unless expressly specified otherwise.
- The subject matter may be embodied as devices, systems, methods, and/or computer program products. Accordingly, some or all of the subject matter may be embodied in hardware and/or in software (including firmware, resident software, micro-code, state machines, gate arrays, etc.) Furthermore, the subject matter may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in connection with an instruction execution system. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
- The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.
- Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by an instruction execution system. Note that the computer-usable or computer-readable medium could be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, of otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
- When the subject matter is embodied in the general context of computer-executable instructions, the embodiment may comprise program modules, executed by one or more systems, computers, or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.
-
Figure 1 is a diagram illustration of anembodiment 100 showing an example system where a browser may be used to interactively control a remote computer. In the example, aserver 102 may be controlled by aclient 104, where anoperating system desktop 108 andapplication window 110 may be rendered on the server, and thebrowser 112 may display at least a portion of theoperating system desktop 108. The browser may capture and transmit keyboard, mouse, and other inputs to theserver 102, which may send the inputs to the application on theserver 102. - The renderings from the display of the
server 102 may be captured by animage capture component 114 and encoded with avideo codec 116 to create a video stream. The video stream may be passed across anetwork 118 to theclient 104, which may receive the video stream, decode the video stream with avideo codec 120, and display the video stream using avideo player 112 onto the browser window. - "Real time" or "near-real time" control of an application on a
server 102 by another device, in thiscase client 104, may introduce various complexities. Ideally, the user may make an input, such as a mouse click, keyboard input, or other input, and the user would experience instantaneous response. The fact that the two devices communicate with each other introduces latencies, as well as the availability of limited or variable network connections. - Video compression and transmission techniques are widely used, and any appropriate format, compression algorithms, codecs, and other protocols may be used. Video capture and compression mechanisms may be lossy, where some of the information may be lost during the compression routine, while other mechanisms may not be lossy.
- Some of the video compression and transmission techniques may introduce latency into the system. Many video compression techniques may use data from one or more frames to compress information in later frames. Such techniques may inherently capture two, three, or more frames, perform compression using the frames, then transmit the video. One mechanism to reduce latencies may be to select a video compression and encoding technique that uses a minimum of frames during the compression process. Some video compression and encoding mechanisms may have configuration settings that may permit adjusting the encoding techniques.
- Video compression mechanisms that have been successfully implemented with such a system include H262, H263, H264, and others.
- A second mechanism by which user-perceived latency may be reduced may be to have cursor movement and rendering performed on the user's local device, rather than on a remote device. From the user's standpoint, the motion and feel of a cursor and its responsiveness to movement may give a perceived "real time" feel, even when the cursor inputs have a latency when they are transmitted and processed on a remote system.
- Local rendering of the cursor can cause the cursor to move with the same responsiveness within a browser window as with any other application on a client device. For cursors that may be controlled with mice or other pointing device, such a responsiveness may give an acceptable experience, and in some cases, users may not realize that the operations performed within a browser window may actually be performed and rendered on a remote device.
-
Embodiment 100 may illustrate a system where cursor information is not included in the video transmitted from theserver 102, and where the cursor information may be rendered by theclient 104. - In some systems, a
cursor manager 126 may be capable of removing any rendered cursor from an image captured by theimage capture component 114. Some such systems may capture a bitmap of the rendered screen, identify the cursor location, and change the bitmap to remove the cursor. - In other systems, the operating system of the
server 102 may be configured so that the cursor may be very small or not rendered, which may obviate the step of removing a cursor from the rendered image. - A
cursor manager 126 may identify the specific cursor being used by the application or operating system of the server at a specific instance. Thecursor manager 126 may transmit an image of the cursor in a bitmap or other form to theclient 104. Theclient 104 may have acursor manager 132 which may cause the cursor image to be rendered locally on theclient 104. - Some applications may have multiple cursors that may change based on the context of the cursor, a tool that may be selected, or some other feature of the application. For example, an image editing system may have a cursor in the shape of an eyedropper for sampling colors, a bracket cursor used for editing text, a crosshair cursor for selecting a rectangular box, and so forth. Each time the cursor may be changed, the
cursor manager 126 may identify the cursor and relay the cursor information over theserver signaling channel 124 to theclient 104. - In some cases, the
cursor manager 126 may send an identifier for a specific cursor, and theclient 104 may use one of several storedcursors 134 that corresponds to the identifier. Such systems may transmit an image of a cursor the first time that cursor may be encountered, then each time thereafter, may merely send the identifier instead of the image. Theclient 104 may have acursor manager 132 that may cause the local cursor to change to a different cursor in response to a change in the cursor identifier. - A
controller 136 on theserver 102 may use aserver signaling channel 124 and aclient signaling channel 144 to communicate with theclient 104 during setup, tear down, and normal operations. Theclient 104 may have aserver signaling channel 130 and aclient signaling channel 140 connected to aclient controller 138. - The signaling channels may be illustrated here as separate communications channels. In some cases, the channels may be separate sockets, ports, protocols, or be otherwise separate from the channel through which video may be transferred. In other cases, some or all of the server signaling channel communications may be embedded or encoded into the video stream, thereby having a single port, channel, protocol, socket, or other channel from the
server 102 to theclient 104. - The
client 104 toserver 102 communications may be a separate channel, port, protocol, socket, or other mechanism than from the server signaling channel. In some cases, the client signaling channel may be a duplex version of the server signaling channel, where both directions of communication may use the same port, channel, protocol, socket, or other common mechanism. - The
client 104 may have a mechanism to capture mouse, keyboard, andother input 142. These inputs may be captured and transmitted through theclient signaling channel 140 to theserver 102. Theserver 102 may receive the data through the client signaling channel, and may provide the input to theoperating system 146. These inputs may be passed to the application executing within the operating system and thereby the user of thebrowser 112 may operate an application on theserver 102. - The inputs collected on the
client 104 may include any type of input, including keyboard, mouse, audio, as well as sensory input from gyroscopes, global positioning system (GPS) sensors, accelerometers, temperature sensors, pressure sensors, light sensors, and any other type of sensor that may be available on theclient 104. In some cases, the inputs may include camera input that may be collected from a camera mounted on theclient 104 or connected to theclient 104. The camera may be able to capture still images or video, and such inputs may be transmitted over theclient signaling channel 140. In the case of video or audio inputs, such inputs may be encoded using a codec and transmitted to theserver 102 for decoding prior to use. - The example of
embodiment 100 may illustrate a system where abrowser 112 may render the contents of anapplication window 110. This example is merely one use case, such as where theserver 102 may be configured just to provide a specific application through thebrowser 112. Other systems may display multiple applications and even the entireoperating system desktop 108 in thebrowser 112. Such applications may allow for theentire server 102 to be controlled by theclient 104 through thebrowser 112. -
Figure 2 is a diagram of anembodiment 200 showing components that may be used for remote access of an application or system through a browser. - The diagram of
Figure 2 illustrates functional components of a system. In some cases, the component may be a hardware component, a software component, or a combination of hardware and software. Some of the components may be application level software, while other components may be execution environment level components. In some cases, the connection of one component to another may be a close connection where two or more components are operating on a single hardware platform. In other cases, the connections may be made over network connections spanning long distances. Each embodiment may use different hardware, software, and interconnection architectures to achieve the functions described. -
Embodiment 200 illustrates adevice 202 that may have ahardware platform 204 and various software components. Thedevice 202 as illustrated represents a conventional computing device, although other embodiments may have different configurations, architectures, or components. - In many embodiments, the
device 202 may be a server computer. In some embodiments, thedevice 202 may still also be a desktop computer, laptop computer, netbook computer, tablet or slate computer, wireless handset, cellular telephone, game console or any other type of computing device. In some embodiments, thedevice 202 may be implemented on a cluster of computing devices, which may be a group of physical or virtual machines. - The
hardware platform 204 may include aprocessor 208, random access memory 210, andnonvolatile storage 212. Thehardware platform 204 may also include auser interface 214 andnetwork interface 216. - The random access memory 210 may be storage that contains data objects and executable code that can be quickly accessed by the
processors 208. In many embodiments, the random access memory 210 may have a high-speed bus connecting the memory 210 to theprocessors 208. - The
nonvolatile storage 212 may be storage that persists after thedevice 202 is shut down. Thenonvolatile storage 212 may be any type of storage device, including hard disk, solid state memory devices, magnetic tape, optical storage, or other type of storage. Thenonvolatile storage 212 may be read only or read/write capable. In some embodiments, thenonvolatile storage 212 may be cloud based, network storage, or other storage that may be accessed over a network connection. - The
user interface 214 may be any type of hardware capable of displaying output and receiving input from a user. In many cases, the output display may be a graphical display monitor, although output devices may include lights and other visual output, audio output, kinetic actuator output, as well as other output devices. Conventional input devices may include keyboards and pointing devices such as a mouse, stylus, trackball, or other pointing device. Other input devices may include various sensors, including biometric input devices, audio and video input devices, and other sensors. - The
network interface 216 may be any type of connection to another computer. In many embodiments, thenetwork interface 216 may be a wired Ethernet connection. Other embodiments may include wired or wireless connections over various communication protocols. - The
software components 206 may include anoperating system 218 on which various software components and services may operate. -
Various applications 220 may execute within the operating system environment. Some or all of theapplications 220 may be made available for remote control through aclient device 252 and abrowser 258. - The
operating system 218 may have arendering engine 222 which may render displayable images based on input from theoperating system 218 andvarious applications 220. Therendering engine 222 may be an operating system function that may use agraphics processing unit 224, which may be a hardware or software accelerator that may assist in rendering graphics. - The
server 202 may have aserver side executable 226, which may be a set of software components that may permit external control of the entire system or a subset of the system through aclient device 252. Theserver side executable 226 is illustrated as a set of software components as a matter of convenience, but some or all of the components may be implemented in hardware. - An
image capture component 228 may capture the rendered images from therendering engine 222. Theimage capture component 228 may be an operating system function in some systems or may be a separate component that may intercept rendered images that may be displayed on a monitor. - A
cursor remover component 230 may be used in systems that may render a cursor within the displayable image. Some systems may not have acursor remover component 230 and may include the rendered cursor in the video stream. In such a case, theclient device 252 may or may not have a separately rendered cursor. - When the video stream includes a rendered cursor and when the client device also has a separately rendered cursor, a user may experience two cursors on the screen. The cursor rendered on the
server 202 and embedded in the video stream may not be as responsive as the cursor rendered by theclient 104, however, in most cases, the client-rendered cursor may overlap or hide the server-rendered cursor. A user may experience that the server-rendered cursor may slightly lag behind the movement of the client-rendered cursor. - A
server 202 may render a cursor and theclient device 252 may be configured not to render a cursor over thebrowser 258. In such a case, the user interaction with thebrowser 258 may generate a location where a cursor may be at any given time, and the location may be transmitted to theserver 202, then rendered in the video stream viewed on theclient device 252. Such systems may have some latency or lag that may be experienced by the user of theclient device 252. In cases where the network and video processing latencies are not exceptionally high, the user experience may still be acceptable. - A
server 202 may be configured to not render a cursor at all in the image generated by the rendering engine. In such cases, the rendered image may not have a cursor, so the only cursor a user of theclient 252 may experience may be the cursor rendered by theclient 252. - An image sequence from the
image capture component 228 may be sent to avideo codec 236 and then communicated to theclient device 252 by acommunications manager 238. - The
communications manager 238 may handle aserver signaling channel 240, which may send control information and other information to theclient device 252. - The
server signaling channel 240 may be used to send cursor information to theclient device 252. The cursor information may include the location of a cursor as well as the type of cursor being displayed. The cursor type may be an identifier for a specific cursor. Theserver signaling channel 240 may also be used to send a bitmap or other image of a cursor for rendering on theclient device 252. In some cases, theclient device 252 may request a cursor image when a specified cursor may not be available on theclient device 252, and thecursor manager 232 may be able to send the requested cursor from a group of storedcursors 234. - A
controller 242 may manage various administrative functions for the client/server interaction. The administrative functions may include setup and teardown of a session, monitoring and adjustment of communications settings, and other functions. Thecontroller 242 of theserver 202 may operate in conjunction withcontroller 272 of the client, and in some cases, eithercontroller - During setup and teardown of a communications sessions, some use cases may have
controllers client device 252 may request a connection from theserver 202 or may be routed to aserver 202 after making a request to a third device, such as aconnection management system 280. - When a connection request is received, the
controller 242 may set up theapplications 220 prior to establishing a video stream. In some cases, a new instance of an application may be established and, when the instance becomes stable, a video stream may be created. In some such cases, asingle server 202 may provide separate video connections to multiple devices, each device having a separate session with different instances of an application or with multiple devices sharing the same instance or session with a single instance of an application. - A setup operation may involve starting a new instance of an application and configuring the application based on parameters or data that may be provided with a request for a new session. For example, an application may be configured to resume operations from a previously saved session, or may be started with a set of options requested by a user.
- Teardown of a session may involve terminating an application instance or reconfiguring the
server 202 in some manner in preparation for a new session. In some cases, theserver 202 may maintain its configuration and various application instances in case the session termination was inadvertent. In other cases, theserver 202 may terminate one or more applications and free up resources for new sessions that may be requested at a later time. - The
controllers - The
controllers browser 258. - The
client signaling channel 246 may handle inbound communications from theclient device 202. One element of inbound communications may be keyboard/mouse input 278 as well as other input captured by the client device'sbrowser 258. These inputs may be communicated from thecommunications manager 264 on theclient device 252 to thecommunications manager 238 on theserver 202 and theclient signaling channel 246. These inputs may be transferred as inputs to theoperating system 248. - The keyboard/
mouse input 278 may include any type of input captured by theclient device 252. These inputs may include audio, video, Global Positioning System (GPS), accelerometer input, orientation sensors, temperature and pressure sensors, and any other type of input that may be captured by theclient device 252. In some cases, the input may be data gathered by or presented by applications executing on theclient device 252 or accessed through thenetwork 250 to some other device. For the sake of convenience, the term keyboard/mouse input 278 includes all of the various inputs listed above. - The
client device 252 may have ahardware platform 254 on which anoperating system 256 may operate. Thehardware platform 254 may include any of the various options and characteristics described for thehardware platform 204. Abrowser 258 may execute on theoperating system 256. - The browser may have a
display window 260, as well as anexecutable engine 262. Theexecutable engine 262 may be a virtual machine, emulator, interpreter, compiler, or any other mechanism by which instructions may be executed. In today's browser technologies, many browsers include a Javascript engine that may execute Javascript that may be contained in an HTML document that abrowser 258 may render. - Incoming video data may be decoded by a
video codec 266 and shown on the browser'sdisplay window 260 by avideo player 268. In many cases, avideo player 268 may include controls for starting, stopping, rewinding, fast forwarding, or other functions. In some cases when the browser may be used to control a remote device, such as theserver 202, such controls may be turned off or otherwise unavailable to the user. - A client side executable 270 may contain code that may be executed on the browser's
executable engine 262. The client side executable 270 may be downloaded as part of an HTML page, and in some instances, the components of the client side executable 270 may be embedded in Javascript or some other executable language. - The client side executable 270 may include a
controller 272, acommunications manager 264, acursor manager 274, and storedcursors 276. The client side executable 270 may include routines or other components that may capture keyboard/mouse inputs 278. - The
cursor manager 274 may determine which cursor is to be shown at a particular time. Thecursor manager 274 may receive a bitmap or other image of a cursor and store the image in a set of storedcursors 276. In many cases, thecursor manager 274 may receive an identifier which may identify the current cursor. The identifier may be any type of identifier, such as a set of numbers, letter, hashed values, or other identifier. Thecursor manager 274 may look up the corresponding cursor image from the storedcursors 276, and may pass the cursor image to thebrowser 258 for rendering. - In some cases, a certain cursor may be applicable to a specific area of a rendered application, while a second cursor may be applicable to a different area of the rendered application. In such cases, the
cursor manager 274 may receive a map or boundary area identifying a specific cursor to use while other areas of a displayed video image may have a different cursor. Some applications may have several cursor images that may apply to different areas or zones of a rendered video image. - The
network 250 may be any type of network that may allow theserver 202,client device 252, and other devices to communicate. - A
connection management system 280 may be a device that may help set up sessions between client devices and servers. Theconnection management system 280 may operate on a hardware platform 282 with anoperating system 284. Aconnection manager 286 may receive an inbound request from aclient device 252 for a session. Theconnection manager 286 may use anauthentication service 288 to determine whether or not the requestor may have permission to establish a session. Provided that the authentication is successful, aload balancer 290 may select between multiple instances of aserver 202 to direct the client to establish a session. - In some cases, the
connection manager 286 may setup and configure anapplication 220 on aserver 202 specifically for an inbound request, then give the address of aserver 202 for establishing a connection. - Some
servers 202 may be configured so that multiple client devices may connect to a single session. In some such cases, one or more client devices may be configured to provide input to an application executing on theserver 202, while other client devices may be configured to merely monitor, watch, or view the display window without providing control to the window. In some cases, control of an application on aserver 202 may be changed from one device to another. - A
monitoring client 292 may have ahardware platform 294 on which anoperating system 296 may operate. Abrowser 298 may connect to thecommunications manager 238 and may render adisplay window 299, which may be the same image as rendered on thebrowser 258 of theclient device 252. In some cases, certain inputs may be received from themonitoring client 292, such as audio, keyboard, mouse, or other inputs, and may be transmitted to theserver 202. Such inputs may be transmitted as inputs to theoperation system 248, although in other instances, some of the inputs generated by monitoringclient 292 may be transmitted to theclient device 252 for rendering and display within thebrowser 258. -
Figure 3 is a flowchart illustration of anembodiment 300 showing a method of setting up a communications session.Embodiment 300 is a simplified example of a sequence showing aload balancer 302 in the left hand column, aserver 304 in the center column, and a client 306 in the right hand column. - Other embodiments may use different sequencing, additional or fewer steps, and different nomenclature or terminology to accomplish similar functions. In some embodiments, various operations or set of operations may be performed in parallel with other operations, either in a synchronous or asynchronous manner. The steps selected here were chosen to illustrate some principals of operations in a simplified form.
-
Embodiment 300 may illustrate one method by which a new session may be created. Aload balancer 302 may provide initial setup functions, including authentication and setup, then a session may be created between theserver 304 andclient 308 where theclient 308 may control theserver 304 or an application running on theserver 304. - A client 306 may send a request for a web page in
block 308 to theload balancer 302. Theload balancer 302 may receive a request inblock 310, then request authentication credentials inblock 312. The authentication request may be received by the client 306 inblock 314, and may send the credentials inblock 316. - The
load balancer 302 may receive authentication credentials inblock 318 and verify the credentials inblock 320. In some cases, the credentials may be passed to a separate authentication service. After verifying the credentials inblock 320, theload balancer 302 may send a web page with server information and authentication token inblock 322 to the client 306. - In some cases, the
load balancer 302 may also send set up or configuration instructions to theserver 304, while in other cases, such configuration instructions may be included in the web page with server information. In this example, a web page is a shorthand notation for instructions for how to access theserver 304 from the client 306. In many cases, such instructions may be embodied in an HTML document, which may include executable scripts or other instructions. - The web page may be received in
block 324 by the client 306. The client 306 may sent a request with the authentication token inblock 326, which may be received by theserver 304 inblock 328. Theserver 304 may verify the token inblock 330, and may send client executable code inblock 332 to the client 306. The client 306 may receive the client executable code inblock 334 and may begin execution inblock 336. - The
server 304 may send a video stream inblock 340 that may represent the operating system desktop of theserver 304, an application executing on theserver 304, or some other "live" image of theserver 304. The video stream may be received and rendered inblock 338. The client 306 may generate and send client data inblock 346, which may include keyboard, mouse, and other inputs, which may be received by theserver 304 inblock 348. The keyboard, mouse, and other inputs may be used to control the application or operating system running on theserver 304, and thereby a user of the client 306 may operate theserver 304 through a browser. -
Figure 4 is a flowchart illustration of anembodiment 400 showing a method of handling cursors in a browser-control system of a remote server.Embodiment 400 is a simplified example of a sequence showing aserver 402 in the left hand column and aclient 404 in the right hand column. - Other embodiments may use different sequencing, additional or fewer steps, and different nomenclature or terminology to accomplish similar functions. In some embodiments, various operations or set of operations may be performed in parallel with other operations, either in a synchronous or asynchronous manner. The steps selected here were chosen to illustrate some principals of operations in a simplified form.
-
Embodiment 400 may illustrate a process that may be performed between aserver 402 andclient 404 when theclient 404 may render a cursor image locally, but may be providing input to and controlling theserver 402 or at least an application onserver 402. - An outbound video stream may be created in
block 406 by theserver 402 and transmitted to theclient 404. Theclient 404 may receive the video stream inblock 408 and may render the video stream within a browser. Inblock 410, theclient 404 may gather and send a cursor location to theserver 402, which may receive the cursor location inblock 412. - The cursor location may also include interactions or commands that may be executed with the cursor location. Such commands may be a left-click, right-click, center-click, scrolling, or other location-sensitive commands. The cursor location may be sent to the operating system in
block 414, along with any commands or other input. - The cursor image generated by the
server 402 may be intercepted inblock 416. In some cases, the operating system of theserver 402 or an application executing on theserver 402 may have a signaling system to transmit a cursor image that may be displayed at the location selected by the user. Some applications may have cursor images that may reflect a function, command, setting, or other operational characteristic of the application. Some applications may change cursor images based on the location of the cursor within the application, and such image changes may signify different available operations to the user. - In many cases, a cursor image may have one or more transparent pixels or bits. In some cases, the transparent pixels or bits may be partially transparent.
- The
server 402 may generate a cursor identifier for the cursor image. The cursor identifier may be any shorthand notation or identifier for a particular cursor image. One mechanism for identifying a cursor image may be to perform a hash function on the image and generate an alpha-numeric or numeric designation. The hash function may be selected such that changes to the image may result in changes to the identifier, thereby signaling that the cursor has changed. - The cursor identifier may be looked up in the cursor database in
block 422. If the cursor image does not exist in the cursor database inblock 424, theserver 402 may send the cursor image and identifier inblock 426 to theclient 404. - The
client 404 may receive the cursor and identifier inblock 428 and may store the cursor and identifier in a local cursor storage inblock 430. The client may cause the cursor image to be used for the current location of the cursor inblock 432 and the process may loop back to block 410. - If the cursor does exist in the server's cursor database in
block 424, the cursor identifier may be transmitted inblock 434. Theclient 404 may receive the cursor identifier and may look up the cursor identifier in the cursor database in block 436. If the cursor identifier and corresponding image exists locally in the cursor database inblock 438, the client may use the cursor image inblock 440. - If the lookup fails in
block 438 and the identified cursor is not available locally, theclient 404 may request a cursor image inblock 442, which may be received by theserver 402 inblock 444. Theserver 402 may return to block 426 and may transmit the cursor image and identifier. - In some cases, a cursor image may be defined for a specific region within a rendered area of a video stream. For example, when a cursor hovers over a text object, the cursor may be changed to a text editing cursor, but such a cursor may be only valid for the region of a text box. Such a map, overlay, or other description of where certain cursors may be valid may be generated on the
server 402 and transmitted to theclient 404 in some embodiments. In some cases, theserver 402 may generate a map of the rendered area and may define which cursors may be valid at various locations within the map. Some such systems may have a default cursor that may be used wherever a special cursor may not be defined. -
Figure 5 is a diagram illustration of anexample embodiment 500 showing an over-sized video transmission from aserver 502 to aclient 504. The over-sized video transmission may be a higher resolution or larger size than may be displayed in a browser on theclient 504. -
Embodiment 500 may illustrate one use case of an over-sized video transmission. An over-sized video transmission may be any configuration where theserver 502 may transmit a video stream that may be larger than the displayed area on the browser of theclient 504. The over-sized video in this example may be useful for panning and zooming the video stream rendered on theclient 504. - In such a configuration, any zooming or panning of the video stream may be performed on the
client 504, which may be faster and more responsive than if the zoom or pan commands were transmitted to theserver 502 and updates made to the rendered video stream. - A zoom command may expand or contract the displayed area or viewport rendered in a browser on the
client 504. A pan command may allow theclient 504 to move the displayed viewport horizontally or vertically within the rendered area of the video. - Zoom and pan commands may be issued on the
client 504 by a user. In some cases, the user may have a mouse navigation mechanism, such as a special mouse button and movement commands, or a mouse movement in conjunction with a keyboard command, such as a shift key, alt key, or control key. In other cases, a zoom or pan command may be issued by moving the device, such as through accelerometer inputs when a user may move a device, such as a mobile device, through space. - A
server 502 may generate a renderedarea 506, which may be captured and encoded into video using avideo encoder 508. The video stream may be transmitted over anetwork 510 to aclient 504. Theclient 504 may have avideo decoder 512 and may generate a renderedarea 514. The renderedarea 514 may be the same size as the renderedarea 506 on the server. - The
client 504 may display a subset of the renderedarea 514 as a displayedviewport 516. The displayedviewport 516 may be moved around the renderedarea 514 by zoom and pan commands issued by a user. - The
client 504 may capture keyboard, mouse, andother inputs 518, which may be transmitted over thenetwork 510 to theserver 502. Theserver 502 may use the keyboard, mouse, andother inputs 520 to input to the application, causing the rendered image of the application or operating system to be updated. - The cursor locations transmitted back and forth between the
client 504 andserver 502 may be translated to compensate for the difference between the client's displayedviewport 516 and the renderedarea 514. Typically, theserver 502 may receive cursor locations based on an absolute horizontal and vertical location from one of the corners of the renderedarea 506. However, the cursor of theclient 504 may be limited to be within the displayedviewport 516. Theclient 504 may translate the cursor input captured on a browser to the rendered area dimensions in the keyboard, mouse, orother input 518 that may be transmitted to theserver 502. -
Figure 6 is a flowchart illustration of anembodiment 600 showing a method of negotiating and changing the rendered area of a video stream that may be transmitted and displayed on a client.Embodiment 600 is a simplified example of a sequence showing aserver 602 in the left hand column and aclient 604 in the right hand column. - Other embodiments may use different sequencing, additional or fewer steps, and different nomenclature or terminology to accomplish similar functions. In some embodiments, various operations or set of operations may be performed in parallel with other operations, either in a synchronous or asynchronous manner. The steps selected here were chosen to illustrate some principals of operations in a simplified form.
-
Embodiment 600 may illustrate one method by which the rendered area of a video stream generated by aserver 602 may be selected and changed. - A
client 604 may determine the displayable area on a browser inblock 606. A zoom/pan allowance may be determined inblock 608, which may determine how much extra size of the video stream may be allocated for locally performed zoom and pan commands. - For some applications, no zoom or pan allowance may be used. Such applications may typically take up the entire browser area and may not zoom or pan within the browser screen. For other applications, notably first person games, the zoom or pan actions may be a large part of the application's operation. Such applications may be allocated a large zoom or pan allowance.
- The full size of a rendered area may be determined in
block 610 from the zoom or pan allowance and the displayable area of the browser. In some cases, even without a zoom or pan allowance, a requested rendered area may be larger or smaller than the viewable browser area. For example, some systems may be configured to downsample at the browser, where a larger rendered area may be transmitted and the video may be shrunk to meet the viewable area on the browser. Such configurations may be useful when the downsampling may offer increased resolution, especially with lossy video compression. In a typical downsampled configuration, the rendered area may be two times the dimensions of the browser viewport. - In another example, a browser may be configured to upsample a video stream, such that the displayed area on the browser may be larger than the video rendered area. In such a situation, the browser may expand the video to fit the browser screen. Such configurations may be useful for applications where less precise rendering may be acceptable or where the network bandwidth may be limiting.
- In
block 612, the network bandwidth may be analyzed to determine whether a video stream of a certain size may be possible. Any adjustments to the requested rendered area may be made inblock 614. - The
client 604 may transmit a request inblock 616, which may be received inblock 618 by theserver 602. Theserver 602 andclient 604 may negotiate the rendered area size inblocks server 602 or other considerations. After negotiation, theserver 602 may begin transmitting a video stream inblock 624, which may be received and rendered inblock 626. - When the
server 602 may detect that a change to the rendered area may be advisable inblock 628, theserver 602 may initiate a negotiation inblock 620. - Similarly, when the
client 604 may detect that a change to the rendered area may be advisable inblock 630, the process may go back to block 606 to consider various factors that may be changed. - The
client 604 may determine that a change to the rendered area may be advised under several circumstances. For example, theclient 604 may determine that a user may be exceeding the limits of the available room within the rendered area for zoom or pan commands. In such a case, theclient 604 may request that the rendered area be expanded. - In another example, a
client 604 may detect that network bandwidth may be decreasing, causing theclient 604 to request a smaller rendered area. In yet another example, the user may reposition the browser on theclient 604, which may cause the browser's displayed viewport to change size or aspect ratio. Such a change may indicate that a rendered area change may be in order.
Claims (15)
- A server system comprising:at least one processor;a first network connection; anda remote browser (112) a remote service capable of execution on said at least one processor, said remote service that:receives a rendered image of an application (110) executing on said at least one processor (108), the rendered image comprising a plurality of frames;generates a video stream (114) from said rendered image using video compression and encoding techniques (116) that use data from one or more frames to compress information in later frames, whereby latency is introduced into the system, the video compression and encoding techniques having configuration settings that permit adjusting the encoding techniques;transmits said video stream over said first network connection (118) to the remote browser (112); characterised in that the server system is an ultra low latency server system and the remote service is an ultra low latency remote service,said ultra low latency server system adapted to establish said first network connection by receiving a connection request from a client device (104) comprising said remote browser (112) having a JavaScript engine, and transmitting connection instructions to said remote browser (112), wherein said connection instructions comprise a hyper-text markup language, HTML, page and executable JavaScript,wherein embedding the executable JavaScript in the HTML page permits the remote browser on any client device to execute said application on said at least one processor without creating, supporting, and downloading native applications that would be specific to each hardware and software platform of any client device;and in that said video compression and encoding techniques are tuned to minimize buffering by using a minimum number of the frames during the compression process to reduce the latency in the system;said connection instructions further comprising executable code, said executable code that, when executed, captures at least a portion of a cursor information within said remote browser, receives cursor information (132) from said remote browser; and transmits said cursor information to said application (110);whereby, in use, a user of the remote browser experiences real-time or near real-time control of the application executing on the at least one processor of the ultra-low latency server system.
- The ultra-low latency server system of claim 1, said ultra-low latency remote service that further:
receives a connection request from said remote browser, said connection request comprising displayable screen dimensions on said remote browser, and optionally said displayable screen dimensions being smaller than dimensions of said rendered image. - The ultra-low latency server system of claim 2, said ultra-low latency remote service that further:
generates said video stream to match said displayable screen dimensions. - The ultra-low latency server system of claim 2 or 3, said ultra-low latency remote service that further:
causes said rendered image to match said displayable screen dimensions. - The ultra-low latency server system of claim 2, 3 or 4, said remote browser being configured to pan and/or zoom said video stream.
- The ultra-low latency server system of any preceding claim, said ultra-low latency remote service that further:
receives an updated displayable screen dimensions from said remote browser; and changes said video stream to match said updated displayable screen dimensions. - The ultra-low latency server system of any preceding claim, said ultra-low latency remote service further transmitting a cursor identifier, said cursor identifier identifying a displayed cursor by said application.
- The ultra-low latency server system of claim 7, said ultra-low latency remote service further transmitting a cursor boundary, said cursor boundary identifying a region within said video stream for which said cursor identifier is valid.
- The ultra-low latency server system of claim 8, said ultra -low latency remote service further receiving a cursor request and transmitting said current cursor image in response to said cursor request.
- A method performed on at least one processor of the ultra-low latency server system as claimed in any of claims 1 to 9, said method comprising:receiving a rendered image of an application, the rendered image comprising a plurality of frames;generating a video stream from said rendered image using video compression and encoding techniques that use data from one or more frames to compress information in later frames, whereby latency is introduced into the system, the video compression and encoding techniques having configuration settings that may permit adjusting the encoding techniques;transmitting said video stream over said first network connection to a remote browser;the ultra-low latency server system adapted to establish said first network connection by receiving a connection request from a client device comprising a remote browser having a javascript engine, and transmitting connection instructions to said remote browser, wherein said connection instructions comprise a hyper-text markup language, HTML,page and executable javascript, wherein embedding the executable javascript in the HTML page permits the remote browser on any client device to execute said application on said at least one processor without creating, supporting, and downloading native applications that would be specific to each hardware and software platform of any client device;and in that said video compression and encoding techniques are tuned to minimize buffering by using a minimum number of the frames during the compression process to reduce the latency in the system.said connection instructions further comprising executable code, said executable code that, when executed, captures at least a portion of said cursor information within said remote browser, receiving cursor information from said remote browser; and transmitting said cursor information to said application
- The method of claim 10, said application being executed on said at least one processor.
- The method of claim 10 or 11 further comprising:
receiving a connection request from said remote browser, said connection request comprising a displayable screen dimensions on said remote browser. - The method of claim 12 further comprising:
generating said video stream to match said displayable screen dimensions. - The method of claim 12 or 13 further comprising:
causing said rendered image to match said displayable screen dimensions. - The ultra-low latency server system of any of claims 2 to 5 further comprising:a connection management system executing on said at least one processor, said connection management system that:receives, and optionally authenticates, said connection request from said remote browser, said connection request being for remote access to an application server through said remote browser;said connection request comprising said displayable screen dimensions for said remote browser;determines a set of configuration settings for said application server;causes said application server to be configured with said set of configuration settings;determines a set of connection settings for said remote browser; and transmits said set of connection settings to said remote browser;the system optionally comprising a load balancer that selects said application server from a plurality of application servers.
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562157448P | 2015-05-05 | 2015-05-05 | |
US201562157451P | 2015-05-05 | 2015-05-05 | |
US201562218273P | 2015-09-14 | 2015-09-14 | |
PCT/US2016/031061 WO2016179436A1 (en) | 2015-05-05 | 2016-05-05 | Ultra-low latency remote application access |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3292677A1 EP3292677A1 (en) | 2018-03-14 |
EP3292677A4 EP3292677A4 (en) | 2019-01-02 |
EP3292677B1 true EP3292677B1 (en) | 2023-11-01 |
Family
ID=57218426
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP16790115.6A Active EP3292677B1 (en) | 2015-05-05 | 2016-05-05 | Ultra-low latency remote application access |
Country Status (3)
Country | Link |
---|---|
US (2) | US11750674B2 (en) |
EP (1) | EP3292677B1 (en) |
WO (1) | WO2016179436A1 (en) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2556940B (en) | 2016-11-28 | 2021-09-15 | Displaylink Uk Ltd | Displaying image data based on level of system performance |
US10868955B2 (en) | 2017-09-05 | 2020-12-15 | Facebook, Inc. | Modifying capture of video data by an image capture device based on video data previously captured by the image capture device |
EP3679722A1 (en) * | 2017-09-05 | 2020-07-15 | Facebook, Inc. | Modifying capture of video data by an image capture device based on video data previously captured by the image capture device |
US10866963B2 (en) * | 2017-12-28 | 2020-12-15 | Dropbox, Inc. | File system authentication |
US10552639B1 (en) | 2019-02-04 | 2020-02-04 | S2 Systems Corporation | Local isolator application with cohesive application-isolation interface |
US11880422B2 (en) | 2019-02-04 | 2024-01-23 | Cloudflare, Inc. | Theft prevention for sensitive information |
US10558824B1 (en) | 2019-02-04 | 2020-02-11 | S2 Systems Corporation | Application remoting using network vector rendering |
US10452868B1 (en) | 2019-02-04 | 2019-10-22 | S2 Systems Corporation | Web browser remoting using network vector rendering |
US11620797B2 (en) * | 2021-08-05 | 2023-04-04 | Bank Of America Corporation | Electronic user interface with augmented detail display for resource location |
KR102664915B1 (en) * | 2021-12-06 | 2024-05-17 | 엘지전자 주식회사 | Display device and operating method thereof |
US12039139B1 (en) * | 2022-06-24 | 2024-07-16 | Freedom Scientific, Inc. | Bifurcation of rendered and system pointing indicia to enable input via a viewport |
Family Cites Families (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6758755B2 (en) * | 1996-11-14 | 2004-07-06 | Arcade Planet, Inc. | Prize redemption system for games executed over a wide area network |
US6005550A (en) * | 1997-03-31 | 1999-12-21 | Compaq Computer Corporation | Automatic cursor parking to hide a mouse pointer |
US5995102A (en) * | 1997-06-25 | 1999-11-30 | Comet Systems, Inc. | Server system and method for modifying a cursor image |
US6538667B1 (en) * | 1999-07-23 | 2003-03-25 | Citrix Systems, Inc. | System and method for providing immediate visual response to user input at a client system connected to a computer system by a high-latency connection |
KR20010090291A (en) | 2000-03-24 | 2001-10-18 | 최영헌 | Advertising apparatus and method using pointer |
CA2442620A1 (en) * | 2001-03-30 | 2002-10-10 | David F. Becker | Remote collaboration technology design and methodology |
CN1213605C (en) * | 2002-08-07 | 2005-08-03 | 华为技术有限公司 | Method and device for transfering PC desk top in video communication terminal |
KR100611255B1 (en) * | 2004-10-15 | 2006-08-10 | 한국과학기술연구원 | Remote conference method of sharing work space |
US8319728B2 (en) * | 2005-10-17 | 2012-11-27 | Avocent Huntsville Corporation | Input-device movement to cursor movement correlator |
US10176272B2 (en) * | 2007-09-28 | 2019-01-08 | Excalibur Ip, Llc | System and method of automatically sizing and adapting a widget to available space |
US9740293B2 (en) | 2009-04-02 | 2017-08-22 | Oblong Industries, Inc. | Operating environment with gestural control and multiple client devices, displays, and users |
US8139099B2 (en) * | 2008-07-07 | 2012-03-20 | Seiko Epson Corporation | Generating representative still images from a video recording |
US9578113B2 (en) | 2009-04-15 | 2017-02-21 | Wyse Technology L.L.C. | Method and apparatus for transferring remote session data |
JP5378883B2 (en) * | 2009-05-28 | 2013-12-25 | 株式会社ソニー・コンピュータエンタテインメント | Image processing apparatus and image processing method |
US9294791B2 (en) | 2009-11-17 | 2016-03-22 | Broadcom Corporation | Method and system for utilizing switched digital video (SDV) for delivering dynamically encoded video content |
US9009219B2 (en) * | 2010-01-27 | 2015-04-14 | Vmware, Inc. | Native viewer use for service results from a remote desktop |
US9781477B2 (en) * | 2010-05-05 | 2017-10-03 | Cavium, Inc. | System and method for low-latency multimedia streaming |
US9684424B2 (en) * | 2010-07-08 | 2017-06-20 | Red Hat Israel, Ltd. | Transforming cursor graphics information |
US9798436B2 (en) * | 2010-07-08 | 2017-10-24 | Red Hat Israel, Ltd. | Remote computing with a low latency mouse mode |
US8949726B2 (en) * | 2010-12-10 | 2015-02-03 | Wyse Technology L.L.C. | Methods and systems for conducting a remote desktop session via HTML that supports a 2D canvas and dynamic drawing |
US9167020B2 (en) * | 2011-06-10 | 2015-10-20 | Microsoft Technology Licensing, Llc | Web-browser based desktop and application remoting solution |
US8903897B2 (en) | 2011-07-08 | 2014-12-02 | Google Inc. | System and method for providing interactive content to non-native application environments |
JP5737082B2 (en) * | 2011-08-31 | 2015-06-17 | 富士通株式会社 | Pointer position correction method, pointer position correction program, and server apparatus |
CN103959208A (en) * | 2011-10-28 | 2014-07-30 | 汤姆逊许可公司 | Method of internet browser-based remote user interface virtual mouse cursor positioning |
US9465572B2 (en) * | 2011-11-09 | 2016-10-11 | Microsoft Technology Licensing, Llc | Dynamic server-side image sizing for fidelity improvements |
US9009334B1 (en) * | 2011-12-09 | 2015-04-14 | Amazon Technologies, Inc. | Remote browsing session management |
US9503497B2 (en) * | 2011-12-10 | 2016-11-22 | LogMeln, Inc. | Optimizing transfer to a remote access client of a high definition (HD) host screen image |
US9392045B2 (en) * | 2012-01-31 | 2016-07-12 | Hewlett-Packard Development Company, L.P. | Remote graphics corresponding to region |
US8934887B2 (en) | 2012-05-31 | 2015-01-13 | Emblaze Ltd. | System and method for running mobile devices in the cloud |
US8880605B2 (en) * | 2012-06-27 | 2014-11-04 | RHUB Communications | Browser based web conferencing employing attendee side image scaling |
US9495065B2 (en) * | 2012-07-06 | 2016-11-15 | Navico Holding As | Cursor assist mode |
US9396002B2 (en) | 2013-01-18 | 2016-07-19 | Dell Products, Lp | Synchronizing a cursor from a managed system with a cursor from a remote system |
US20150365500A1 (en) * | 2013-01-31 | 2015-12-17 | Hewlett-Packard Development Company, L.P. | Remote client application |
US9563327B1 (en) * | 2013-03-14 | 2017-02-07 | Tapjoy, Inc. | Intelligent adjustment of graphical user interfaces |
WO2014143040A1 (en) | 2013-03-15 | 2014-09-18 | American Megatrends, Inc | System and method of web-based virtual media redirection |
EP2972936B1 (en) | 2013-03-15 | 2018-12-26 | American Megatrends, Inc. | System and method of web-based keyboard, video and mouse (kvm) redirection and application of the same |
WO2014155537A1 (en) | 2013-03-26 | 2014-10-02 | 株式会社東芝 | Display control device and display control method |
US10249018B2 (en) * | 2013-04-25 | 2019-04-02 | Nvidia Corporation | Graphics processor and method of scaling user interface elements for smaller displays |
US9716740B2 (en) * | 2013-06-14 | 2017-07-25 | Dell Products L.P. | Web-based transcoding to clients for client-server communication |
KR102107678B1 (en) * | 2013-07-03 | 2020-05-28 | 삼성전자주식회사 | Server for providing media information, apparatus, method and computer readable recording medium for searching media information related to media contents |
US9086837B1 (en) * | 2013-07-30 | 2015-07-21 | Microstrategy Incorporated | Collaboration sessions |
US10331621B1 (en) * | 2013-09-19 | 2019-06-25 | Trifacta Inc. | System and method for displaying a sample of uniform and outlier rows from a file |
CN103546821B (en) * | 2013-10-30 | 2017-01-11 | 乐视致新电子科技(天津)有限公司 | Method and device for regulating video playing interface |
US9537885B2 (en) * | 2013-12-02 | 2017-01-03 | At&T Intellectual Property I, L.P. | Secure browsing via a transparent network proxy |
US9674518B2 (en) * | 2013-12-20 | 2017-06-06 | Vmware, Inc. | Measuring remote video display with embedded pixels |
US10027764B2 (en) * | 2013-12-31 | 2018-07-17 | Google Llc | Associating network-hosted files with network-hosted applications |
US10404769B2 (en) * | 2013-12-31 | 2019-09-03 | Google Llc | Remote desktop video streaming alpha-channel |
US9380351B2 (en) * | 2014-01-17 | 2016-06-28 | Lg Display Co., Ltd. | Apparatus for transmitting encoded video stream and method for transmitting the same |
US20150212654A1 (en) * | 2014-01-28 | 2015-07-30 | Comikka, Inc. | Architecture for providing dynamically sized image sequences |
US9363305B2 (en) * | 2014-01-30 | 2016-06-07 | Vmware, Inc. | Remote contextual access to operating system desktops |
US11283866B2 (en) * | 2014-07-07 | 2022-03-22 | Citrix Systems, Inc. | Providing remote access to applications through interface hooks |
US9424627B2 (en) * | 2014-08-15 | 2016-08-23 | Bellevue Investments Gmbh & Co. Kgaa | System and method for high-performance client-side in-browser scaling of digital images |
-
2016
- 2016-05-05 US US15/147,808 patent/US11750674B2/en active Active
- 2016-05-05 WO PCT/US2016/031061 patent/WO2016179436A1/en active Application Filing
- 2016-05-05 EP EP16790115.6A patent/EP3292677B1/en active Active
-
2023
- 2023-07-24 US US18/357,466 patent/US20240007516A1/en active Pending
Non-Patent Citations (1)
Title |
---|
JANNICK BITSCH ET AL: "Ad-hoc symbiotic interactive displays through DLNA", PERVASIVE COMPUTING AND COMMUNICATIONS (PERCOM), 2012 IEEE INTERNATIONAL CONFERENCE ON, IEEE, 19 March 2012 (2012-03-19), pages 57 - 65, XP032180718, ISBN: 978-1-4673-0256-2, DOI: 10.1109/PERCOM.2012.6199849 * |
Also Published As
Publication number | Publication date |
---|---|
US20240007516A1 (en) | 2024-01-04 |
EP3292677A4 (en) | 2019-01-02 |
WO2016179436A1 (en) | 2016-11-10 |
US20160330260A1 (en) | 2016-11-10 |
EP3292677A1 (en) | 2018-03-14 |
US11750674B2 (en) | 2023-09-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20240007516A1 (en) | Ultra-low latency remote application access | |
US20230342500A1 (en) | Remoting application across a network using draw commands with an isolator application | |
EP3809706B1 (en) | Method and apparatus for transmitting scene image of virtual scene, computer device and computer readable storage medium | |
US20210089256A1 (en) | Method and system for browser-based screen sharing | |
JP6288594B2 (en) | Desktop cloud-based media control method and device | |
US10552639B1 (en) | Local isolator application with cohesive application-isolation interface | |
CN112463277A (en) | Computer system providing hierarchical display remoting with user and system prompt optimization and related methods | |
US11537777B2 (en) | Server for providing a graphical user interface to a client and a client | |
US20150039998A1 (en) | Screen sharing using scripting computer language code directly executable by web browser | |
JP2016511603A (en) | System and method for media streaming for multi-user control and shared display | |
CN114513520B (en) | Web three-dimensional visualization method based on synchronous rendering of client and server | |
US20180113663A1 (en) | Browser based display system for displaying image data | |
EP3090565B1 (en) | Methods, systems, and media for remote rendering of web content on a television device | |
AU2011354757B2 (en) | Three-dimensional earth-formulation visualization | |
US11900530B1 (en) | Multi-user data presentation in AR/VR | |
US11601490B2 (en) | Visualization and interaction of 3D models via remotely rendered video stream system and method | |
US20090274379A1 (en) | Graphical data processing | |
KR101480140B1 (en) | Multi injection server and method thereof | |
WO2019058111A1 (en) | Graphics streaming | |
Tamm et al. | Plugin free remote visualization in the browser | |
US20240194166A1 (en) | Plane-based screen capture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20171204 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20181203 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: H04L 29/06 20060101AFI20181127BHEP Ipc: H04L 12/12 20060101ALI20181127BHEP Ipc: H04L 29/08 20060101ALI20181127BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20200603 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: COLORADO CODE CRAFT, LTD. |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602016083860 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: H04L0029060000 Ipc: H04L0067025000 Ref country code: DE Ref legal event code: R079 Free format text: PREVIOUS MAIN CLASS: H04L0029060000 |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: PENGUIN COMPUTING, INC. |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/04812 20220101ALI20230420BHEP Ipc: H04L 65/70 20220101ALI20230420BHEP Ipc: H04L 65/613 20220101ALI20230420BHEP Ipc: H04L 9/40 20220101ALI20230420BHEP Ipc: G06F 3/04845 20220101ALI20230420BHEP Ipc: H04L 67/1004 20220101ALI20230420BHEP Ipc: H04L 67/025 20220101AFI20230420BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20230525 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230601 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602016083860 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240202 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240301 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1628533 Country of ref document: AT Kind code of ref document: T Effective date: 20231101 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240301 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240202 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240201 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240301 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240201 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20240526 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20240527 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240530 Year of fee payment: 9 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231101 |