WO2011022014A1

WO2011022014A1 - Configuration of additional display devices

Info

Publication number: WO2011022014A1
Application number: PCT/US2009/054480
Authority: WO
Inventors: Bernard J. Thompson; Andrew J. Fisher
Original assignee: Displaylink Corporation; Displaylink (Uk) Limited
Priority date: 2009-08-20
Filing date: 2009-08-20
Publication date: 2011-02-24

Abstract

A system comprises a processing device, a first display device, a second display device and a user interface device. A method of operating the system comprises initiating a position configuration process for the second display device, detecting a directional component of a user action in relation to the second display device, determining the position of the second display device with respect to the first display device from the directional component, and configuring a virtual desktop with the relative position of the first and second display devices. In one embodiment, the method further comprises displaying an icon on the second display device, and the detecting of a directional component of a user action in relation to the second display device comprises detecting the movement of a cursor to the icon.

Description

CONFIGURATION OF ADDITIONAL DISPLAY DEVICES

DESCRIPTION This invention relates to a display system, and to a method of operating the display system,

In desktop computing, it is now common to use more than one display device. Traditionally, a user would have a computer with a single display device attached, but now it is possible to have more than one display device attached to the computer, which increases the usable area for the worker. For example, International Patent Application Publication WO 2007/020408 discloses a display system which comprises a plurality of display devices, each displaying respectively an image, a data processing device connected to each display device and controlling the image displayed by each display device, and a user interface device connected to the data processing device. Connecting multiple display devices to a computer is a proven method for improving productivity.

When multiple display devices are attached to the same computer, the software controlling the display devices must know the physical placement of those display devices relative to each other. This allows the software to display correctly windows on multiple display devices and allows mouse movement between the display devices to seem natural. For example, consider a user who has a second display device to the left of their laptop and pushes the mouse cursor off the left of the laptop's display device. The user would expect the mouse cursor to appear on the second display device since it is physically to the left of the laptop. Similarly, if the user moved the cursor to the right side of the laptop's display device, they would expect the cursor to remain on the laptop display device. This natural behaviour is possible only when software has information about each display device's physical location.

Current configuration methods require the user to move virtual representations of their display devices around a virtual desktop in attempt to draw a map of the physical desktop. It is often difficult in these methods for the user to correlate the virtual representation of each display device with the physical display device it represents. Further, setting the location precisely, in the case that one display device is slightly higher than another for example, is often difficult. Finally, because display device positions are described by moving icons on the screen instead of tracking mouse movement, users must use trial and error to find a configuration that allows mouse movement to feel natural. That configuration can take a relatively long time.

It is therefore an object of the invention to improve upon the known art.

According to a first aspect of the present invention, there is provided a method of operating a display system, the system comprising a processing device, a first display device, a second display device and a user interface device, the method comprising the steps of initiating a position configuration process for the second display device, detecting a directional component of a user action in relation to the second display device, determining the position of the second display device with respect to the first display device from the directional component, and configuring a virtual desktop with the relative position of the first and second display devices.

According to a second aspect of the present invention, there is provided a display system comprising a processing device, a first display device, a second display device and a user interface device, the processing device arranged to initiate a position configuration process for the second display device, detect a directional component of a user action in relation to the second display device, determine the position of the second display device with respect to the first display device from the directional component, and configure a virtual desktop with the relative position of the first and second display devices.

According to a third aspect of the present invention, there is provided a computer program product on a computer readable medium for operating a display system, the system comprising a processing device, a first display device, a second display device and a user interface device, the product comprising instructions for initiating a position configuration process for the second display device, detecting a directional component of a user action in relation to the second display device, determining the position of the second display device with respect to the first display device from the directional component, and configuring a virtual desktop with the relative position of the first and second display devices.

Owing to the invention, it is possible to provide easy-to-use methods for users to describe to a display system the physical location of the display devices connected to a computer. Rather than requiring users to configure display device locations via an abstract user interface, the system uses behaviours to determine more naturally the location of a new display device. For example, in one embodiment, the system might draw a target on a newly added display device and instruct the user to click on the target. By noticing the direction of the mouse movement associated with an onscreen cursor, the system can determine where the display device has been physically placed. Another implementation asks the user to point the mouse cursor to the edge of an existing display device closest to the new display device. Both methods instruct the user to perform some natural task, and then infer the new display device's placement from the user's action.

The implementations described above use mouse movement to determine the cursor location, however the invention is not limited to mouse movements. For example, an implementation might make use of a web camera, prompting the user to look at or physically point their arm at a new display device. This implementation could use face tracking algorithms to determine the display device's location. The central idea of the system is to use some kind of natural gesture to infer the display device location of the newly added display device.

By using gestures, such as asking the user to click a target on a new display device, the system's software can quickly and accurately determine the location of a new display device with a single gesture. Configuration of the new display device takes only a few seconds and uses a method that feels very natural to users. One advantage of this method is using a person's natural behaviour to determine the location of a newly added display device. Further, in relation to implementations that use mouse movement to determine display device placement, the system's software will necessarily calibrate to allow smooth mouse movement between the display devices.

Currently, each operating system and graphic card vendor has their own methods for locating display devices. Al! use a method of drawing a virtual map of

5 display devices. The competitive advantage to this method is that users can configure new display devices in a matter of seconds with a simple, easy to understand method. Other methods take more time and are difficult to configure exactly right.

Though laptops can serve as examples of additional display devices, this io system is not limited to laptop computers. This system can be used on any device that has a pointing device such as a mouse or touch screen, and that includes a port, for connecting an external display device, using technology such as USB, VGA, DVI, or wifi. Thus, this system could be used on desktop computers, or even cell phones with touch screens. Further, the system could be used to indicate the location of

15 other computer peripherals whose location may be useful to the user interface, for example, web cameras, speakers, etc. In a system that uses one or more touch screens, the user could be asked to make one or more touching motions, rather than using the mouse cursor. Where at least one display device comprises a touch screen, the step of detecting a directional component of a user action in relation to

.0 the second display device comprises detecting a user touch of at least one of the touch screens.

Ideally, the methodology can be used when the second display device is a new display device that has been added to the display system. In this case, the step of initiating the position configuration process for the second display device is

>5 triggered by the detection of the connection of the second display device to the display system. As described above, the display system operates to detect a directional component of a user action in relation to the new display device, which is used to determine the position of the new display device with respect to the existing display device from the directional component, and accordingly configures a virtual

JO desktop with the relative position of the existing and new display devices. In a first embodiment, there is displayed an icon on the new display device, and the detecting of a directional component of a user action in relation to the new display device comprises detecting the movement of a cursor to the icon. In a second embodiment, the detecting of a directional component of a user action in relation to the new display device comprises detecting the movement of a cursor to an edge region of an existing display device. In a third embodiment, the detecting of a directional component of a user action in relation to the second display device comprises tracking one or more physiological features (such as eye line or face) of a user. In contrast to prior art systems, in which a user will move around on-screen virtual representations of the existing display device(s) and the new display device to form the virtual desktop, in the display system according to the invention, the user will perform a user action which contains a directional component in relation to the actual physical location of the new display device, for example by pointing their finger or moving the mouse cursor towards or onto that new display device.

Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which;

Figure 1 is a schematic diagram of a display system,

Figure 2 is a schematic diagram of a physical display devices and a virtual desktop,

Figures 3 to 8 are further schematic diagrams of the display system,

Figure 9 is a flowchart of a method of operating the display system, and Figure 10 is a schematic diagram of the display system using touch screens. A display system is shown in Figure 1. The system comprises a processing device 10, display devices 12 and user interface devices 14. The user interface devices are a keyboard 14a and a mouse 14b. The display system shown in Figure 1 is a standard desktop computer, with an additional display device 12b, composed of discrete components that are locally located but could equally be a device such as a laptop computer or suitably enabled handheld device such as a mobile phone or pda (personal digital assistant) all using an additional display 12b. Similarly, the display system may comprise part of a networked or mainframe computing system, in which case the processing device 10 may be located remotely from the user input devices 14 and the display devices 12 or indeed may have its function distributed amongst separate devices.

The display device 12a shows an image 16, and the display of that image 16 is controlled by the processing device 10. One or more applications are running on the processing device 10 and these are represented to the user by windows 18, with which the user can interact in a conventional manner. A cursor 20 is shown, and the user can control the movement of the cursor 20 about the image 16 shown on the display device 12 using the computer mouse 14b, again in a totally conventional manner. The user can perform actions with respect to the running application via the user interface device 14 and these actions result in corresponding changes in the image 16, displayed by the display device 12.

When the additional display device 12b, which might be an additional monitor 12b or the screen 12b of a laptop computer that has been connected to the processing device 10 by a USB or Ethernet connection, is first added to the display system, there exists the problem of the configuration of the system. As far as the processing device 10 is concerned, when the new display device 12b is added to the system a physical connection is made to that new display device 12b, but the processing device 10 has no information as to the actual location of the new display device 12b relative to the existing display device 12a. The processing device 10 does not know if the new display device 12b is located to the left or the right of the primary display 12a.

The operating system run by the processing device 10 uses virtual desktops to manage multiple display devices 12. Each physical display device 12 is represented by a frame buffer that contains everything currently shown on that display device 12. The operating system arranges these frame buffers into a single virtual desktop 22, as shown in Figure 2. When these frame buffers are arranged in the virtual desktop 22 in the same relative positions in which the physical display devices 12 are relatively placed, then the operating system can draw objects on all the display devices 12 in a natural way. The virtual desktop 22 is a combination of the respective images 16a and 16b being shown by the display devices 12.

If the user moves the mouse 14a such that the cursor 20 moves right off the edge of one display device 12a, then the cursor 20 appears on the left of the display device 12b to the right. Similarly a picture spread across several display devices 12 appears properly lined up between the display devices 12. The principle is similar to the windows of a house, where each window offers a view of the same room, though different parts of it. For this reason it is important for the processing device 10 to know the physical configuration of the display devices 12, in order that the virtual desktop 22 can be created in the correct manner from the respective display buffers for each display device 12.

The overall process of adding a new display device 12b to the display system is as follows. Firstly, the user attaches the new display device 12b. Software being run by the processing device 10 (for example within the operating system) detects the presence of the new display device 12b, the software instructs the user to indicate the new display device's position and the software determines the new display device's position and automatically configures that display device's location in the virtual desktop 22. Once the display system detects the connection of the second display device 12b to the system, the processing device 10 initiates the position configuration process for the second display device, and detects a directional component of a user action in relation to the second display device 12b.

The following describes various different implementations of the configuration method. The first embodiment is based around the user clicking a target, and is shown in Figure 3. This implementation determines a new display device's location by drawing a target icon 24 on the new display device 12b, then asking the user to click on the target 24. By noticing the direction of mouse movement, the software of the processing device 10 can determine where the new display device 12b has been physically placed. The overall process of adding a display device is as follows. The user attaches a new display device 12b. The software detects the new display device 12b, and configures the display device 12b enough to display an image on the new display device 12b. The software moves the cursor 20 to the centre of the primary display device 12a and draws a target 24 on the new display device 12b.

The software then instructs the user to move the cursor 10 to the new display device 12b. As the user does so, the software notes the path of the cursor 20 before the cursor 20 leaves the primary display device 12a, and uses this information to roughly position the new frame buffer on the virtual desktop 22. When the mouse cursor 20 leaves the primary display device 12a, the cursor 20 appears on the new display device 12b based on this placement. As the user continues to the centre of the target 24, the software continues to track the cursor movement to refine the position of display device 12b, since the user will naturally correct their movement for any inaccuracies. After the user clicks on the target 24, the software updates the positioning and allows the user to make further adjustments and confirm.

There are two general implementations of this method. A first assumes that new display devices 12b are added to the edge of the virtual desktop 22. In this case, the cursor 20 will travel through any other appropriate display devices 12 on its path to the new display device 12b. The advantage of this implementation is that if a user usually adds to existing configurations, setup is natural and smooth. The disadvantage of this technique comes when a user adds a new display device 12b between two existing display device 12a. In this case, detailed below, the cursor 20 will skip the new display device 12b and appear on another existing display device 12. The user will likely notice this and backtrack with the mouse 14b. The software will have to notice the user backtracking and assume that that new display device has been inserted.

The second implementation assumes that new display devices 12b are always adjacent to the primary display device 12. This assumption works well for small numbers of display devices 12. The disadvantage though, is that if the user adds a new display device 12b to the end of a chain of display devices 12, the mouse cursor 20 will jump past existing display devices 12 on its way to the target 24. The software will then have to employ a secondary location method or user interface to determine where the new display device 12b appears in the chain. Adding a second display device 12b to the display system is the simplest case as there are no other display devices 12 present to cause ambiguity problems. The software only needs to determine the angle to the new display device 12b relative to the existing display device 12a. The user adds the display device 12b and at the software's request, clicks on the target 24. The software determines the angle from the mouse movement and configures the display device.

For example, consider a user who adds a display device 12b to the right of their laptop 12a, as shown in Figure 3. The software paints the target 24 on the new display device 12b. The user moves the cursor 20 straight to the right. By the time the cursor 20 reaches the right edge of the primary display device 12a, the software has determined that the new display device 12b is on the right, and moves the cursor 20 to the centre of the left edge of the new display device 12b. To the user, this placement simply continues a single, unbroken movement. The user continues that movement to the target 24 and clicks. The software completes the configuration.

When adding further display devices 12 to a system that already has several display devices 12 proceeds just as in the simple case, except that the mouse cursor 20 may need to move through several display devices 12, on the way to the target 24. In this case, the software has more opportunities to refine the location of the new display device 12. For example, the previous user (of the Figure 3 configuration) adds a new (third) display device 12c to the right of the second display device 12b, as shown in Figure 4. The software paints a target icon 24 on the new display device 12c and centres the cursor 20 on the primary display device 12a. The user moves the cursor 20 straight to the right, as before, in order to move the cursor 20 towards the target 24. When the cursor 20 reaches the edge of the primary display device 12a, the processing device moves the cursor 20 to the next display device 12b. When the cursor 20 reaches the edge of that display device 12b, the cursor 20 appears on the left of the new display device 12c. The user clicks on the target and software completes the configuration of the virtual desktop 22, reflecting the relative position of the three display devices 12. In the above example of Figure 4, the user added the new display device 12c to the right of the two existing display device 12a and 12b. However it is possible that the user will create the special case of inserting the new display device 12c between the others. The preceding description uses the implementation that assumes that new display devices 12 are added to the edge of the virtual desktop 22. This assumption is not valid, however, when the user adds a display device 12c between two other display devices 12a and 12b. This case can be handled with a slight extension to the general method. The software will detect when the user radically changes direction, for example, going back the direction they came, to detect the case of an inserted display device 12c.

Consider if the previous user had added the new display device 12c between the laptop display 12a and the first additional display 12b. This is illustrated in Figure 5. The software paints the target 24 on the new display device 12c. The user moves their cursor 20 straight to the right. When the cursor leaves the laptop display device 12a, the cursor 20 appears on the left edge of the far display device 12b, which in this case is on the far side of the new display device 12c. The user will naturally steer the cursor 20 to the left in the other direction, in order to move it towards the target 24. The software will detect this radical change of direction and assume that the new display device 12c is between the two existing display devices 12a and 12b. When the cursor 20 reaches the left of the far right display device 12b, the cursor will appear on the right edge of the new display device 12c.

It may also be necessary to resolve ambiguities. If a large array of displays is being configured, the angle of mouse movements between displays that are far apart may not be distinct enough to accurately position the target display. This potential for mis-configuration can be determined by remembering the angle from the primary existing display device 12a to each additional display device 12b, and the path of which displays 12 were traversed to get to each target 12b. If the difference in angle between any two displays 12 on different paths falls below a certain threshold, then the user is asked to resolve any ambiguity by prompting the user to configure the ambiguous displays 12 again, this time from a different starting monitor 12. This additional set of paths and angles then allows the position of the new target monitor 12 to be more accurately triangulated.

A further embodiment of the method of configuring the new display device 12 is shown in Figures 6 and 7, which involves tapping adjacent display devices 12. In this implementation, the software run by the processing device 10 instructs the user to tap the mouse cursor 20 at the side of an existing display device 12, which is closest to the new display device 12c. The software configures the new display device 12c to be adjacent to that display device 12a. For example, the new display device 12c in Figure 6 is display device 12c. If the user taps the cursor 20 against the left hand side of display device 12a, this will set the position of the new display device 12c on the left hand side of display device 12a. If the new display device 12c was physically located above display device 12a, the cursor 20 could be tapped against the top side of display device 12a to position the new display device 12c accordingly. The virtual desktop 22 is updated according to the determined position. Consider a six display device arrangement with a new display device added in position 6, as shown in Figure 7. Either display device 5 or display device 3 could set the position of new display device 12d.

The overall process of adding the new display device 12 proceeds as follows: The user attaches a new display device 12. The software detects the new display device 12, and using existing display devices 12, prompts the user to tap the mouse cursor 20 against the side of the display device 12 closest to the new display device 12. The user does so, and the software configures the new display device 12 to be located in the virtual desktop 22 adjacent to the display device 12 on which the user has tapped, along the side of the display device 12 the user tapped.

Any method could be designed to indicate the display device 12. The above description uses tapping the side of the closest display device 12. Other possibilities include clicking on the mouse with the cursor icon 20 against the boundary of the adjoining display device 12, pointing the cursor 20 for a set period of time against the boundary of the adjoining display device 12 and using illuminating indication icons, for example arrows pointing to the edges of the screen, and performing any of the above.

Figure 8 shows a further embodiment of the system, in which the detecting of a directional component of a user action in relation to the second display device comprises tracking one or more physiological features of the user 26. A camera 28 is provided on the primary display 12a, which is capable of accurately capturing features of the user 26. The camera 28 is connected to the processing device 10 and the processing device 10 is able to process the image data received from the camera 28 to such an extent that it is possible to track one or more physiological features of the user 26, such the direction in which the user's eyes are facing. The eye line 30 of the user is shown in the Figure as looking at the new display device 12b, The software will prompt the user to look at the physical location of the new display device 12b and the eye line 30 will be determined by the processing device 10. From this information, the location of the new display device 12b will be determined, and the virtual desktop 22 will be configured accordingly.

The method of operating the display system, is summarised in Figure 9. The method comprises firstly, step S1 , which is the step of detecting the connection of a second display device 12 to the display system. This step is optional, as the process could be triggered manually by a user and not just by the detection of the connection of a new display device 12. The process continues with step S2, initiating a position configuration process for the second display device 12, step S3, detecting a directional component of a user action in relation to the second display device 12, step S4, determining the position of the second display device 12 with respect to the first display device 12, from the directional component, and finally, step S5, configuring the virtual desktop 22 with the relative position of the first and second display devices 12. In this way, the configuration process for the second display device 12 is completed from the directional user input, be that movement of the cursor 20 on one display device 12 or across multiple display devices 12, or the gathering of data from a user's directional action such as pointing or looking at the new display device 12. The directional user input is directed at the physical location of the second display device 12, not at any virtual representation of that display device 12.

The method can also be extended so that the processing device 10 stores the configuration that has been determined from the user input. Therefore if a given monitor configuration comes and goes, the processing device 10 will remember the last calibration, otherwise it could be annoying for a user if the configuration process kicked off every time they moved between office and home, for example. The configuration can also be stored with respect to a specific device which is identifiable, for example via a unique identifier that is passed during the connection of the new device. It is also possible that the configuration process can be repeated starting from the first pass calibration, in order to iterate to a more exact solution. It is possible that this iteration of the configuration is carried out as a continuous background process, for example by looking at the way a user compensates their mouse movements on each screen crossing. In this way, if the virtual desktop 22 is not properly aligned with the actual physical layout, this can be corrected over time, from the user's continued interactions.

As mentioned above, the methodology can also be used when one or both of the display devices 12 are touch screens. This allows an extension of the user input to gestures when touch screens are available. In this case touching two points to be joined would be a user-friendly option. This is shown in Figure 10. Two touching spots 32, for the user to touch, are illustrated on respective touch screens 12. The touch screens 12 are of course still display devices. The spots 32 do not need to be actually displayed on the devices 12, they are shown in the Figure to illustrate the concept of using touch to configure the position of the display device 12b relative to the display device 12a.

The user will be asked to touch the edge of the current display device 12a nearest the new display device 12b, and will use their finger 34 to touch the spot 32a. They will then be asked to touch the edge of the new display device 12b (which may display an icon for touching) and will touch the spot 32b. If only one of the devices 12 is a touch screen, such as the main display 12a, then the user action may comprise a stroke from the centre of the main display device 12a in the direction of the centre of the new display device 12b. This will be sufficient to configure the virtual desktop 22 will the physical location of the display devices 12 relative to each other.

Claims

1. A method of operating a display system, the system comprising a processing device, a first display device, a second display device and a user interface device, the method comprising the steps of:

o initiating a position configuration process for the second display device, o detecting a directional component of a user action in relation to the second display device,

o determining the position of the second display device with respect to the first display device from the directional component, and

o configuring a virtual desktop with the relative position of the first and second display devices.

2. A method according to claim 1 , and further comprising displaying an icon on the second display device, wherein the step of detecting a directional component of a user action in relation to the second display device comprises detecting the movement of a cursor to the icon.

3. A method according to claim 1 , wherein the step of detecting a directional component of a user action in relation to the second display device comprises detecting the movement of a cursor to an edge region of the first display device.

4. A method according to claim 1 , wherein the step of detecting a directional component of a user action in relation to the second display device comprises tracking one or more physiological features of a user.

5. A method according to claim 1 , wherein at least one display device comprises a touch screen, and the step of detecting a directional component of a user action in relation to the second display device comprises detecting a user touch of at least one touch screen.

6. A display system comprising a processing device, a first display device, a second display device and a user interface device, the processing device arranged to:

o initiate a position configuration process for the second display device, o detect a directional component of a user action in relation to the second display device,

o determine the position of the second display device with respect to the first display device from the directional component, and

o configure a virtual desktop with the relative position of the first and second display devices.

7. A system according to claim 6, wherein the processing device is further arranged to display an icon on the second display device, and is arranged, when detecting a directional component of a user action in relation to the second display device, to detect the movement of a cursor to the icon.

8. A system according to claim 6, wherein the processing device is arranged, when detecting a directional component of a user action in relation to the second display device, to detect the movement of a cursor to an edge region of the first display device.

9. A system according to claim 6, wherein the processing device is arranged, when detecting a directional component of a user action in relation to the second display device, to track one or more physiological features of a user.

10. A system according to claim 6, wherein at least one display device comprises a touch screen and the processing device is arranged, when detecting a directional component of a user action in relation to the second display device, to detect a user touch of at least one touch screen.

11. A computer program product on a computer readable medium for operating a display system, the system comprising a processing device, a first display device, a second display device and a user interface device, the product comprising instructions for:

12. A computer program product according to claim 11 , and further comprising instructions for displaying an icon on the second display device, wherein the instructions for detecting a directional component of a user action in relation to the second display device comprise instructions for detecting the movement of a cursor to the icon.

13. A computer program product according to claim 10, wherein the instructions for detecting a directional component of a user action in relation to the second display device comprise instructions for detecting the movement of a cursor to an edge region of the first display device.

14. A computer program product according to claim 10, wherein the instructions for detecting a directional component of a user action in relation to the second display device comprise instructions for tracking one or more physiological features of a user.

15. A computer program product according to claim 10, wherein at least one display device comprises a touch screen and the instructions for detecting a directional component of a user action in relation to the second display device comprise instructions for detecting a user touch of at least one touch screen.