GB2376397A

GB2376397A - Virtual or augmented reality

Info

Publication number: GB2376397A
Application number: GB0113559A
Authority: GB
Inventors: Stephen Philip Cheatle; Stephen Bernard Pollard; Maurizio Pilu
Original assignee: Hewlett Packard Co
Current assignee: HP Inc
Priority date: 2001-06-04
Filing date: 2001-06-04
Publication date: 2002-12-11
Also published as: GB0113559D0

Abstract

A portable head mounted display, HMD, (<B>10</B>) displays a virtual or augmented reality operating area e.g. for use in game playing. Two cameras (<B>12, 14</B>) placed on the HMD comprise a stereo vision depth sensing system to recognise and determine the distance to obstacles in the user's field of view. Real obstacles in the display may be 'cloaked' by forming virtual object around them. The user can therefore move around the physical environment, using only the displayed reality as a guide without fear of colliding onto real objects.

Description

VIRTUAL REALITY Field of the Invention This invention relates to virtual reality and, in particular, to creating virtual reality environments for use in virtual reality games and similar arrangements.

Background to the Invention It is, of course, well known to provide computer game apparatus comprising a twodimensional display screen on which is graphically defined game area which can be viewed by a player. The player can notionally move through the game area using controls such as those provided by a joystick, computer keyboard, cursor control device or the like. However, irrespective of how realistically the game area is portrayed on the screen, the fact that it is twodimensional effectively prevents the player from feeling as if they are actually within the game area itself and, in any event, the screen/control interface involves player movements which are very different to the movements which are being simulated in the game, i. e. , for example, the user presses a button to'jump'within the game.

In order to provide a more realistic game playing environment, virtual reality systems have been developed which generally comprise a head mounted display (HMD) through which a virtual world is three-dimensionally defined and can be viewed as such, giving the player a sensation of actually being present within the game area. The head mounted display typically has some form of motion sensor which causes the display to change as the player moves a part of their body, such as their head, thereby giving the user the sensation of looking around the virtual world which makes up the game area.

Some virtual reality systems permit the player to move around a physical area, such movement being translated into movement within the virtual game area. For example, US Patent No.

5,913, 727 describes an interactive contact and simulation game in which a player and a threedimensional computer-generated image interact in simulated physical contact. The apparatus includes control means for generating a simulated image of the player and displaying that image within the computer-generated game area, and a number of position sensing means and

impact generating means are secured to various parts of the player's body. The player moves around a predetermined physical space, and the position sensing means determine the player's position accordingly, in response to which determination, the computer-generated image is moved accordingly, to give the player the sensation of actually moving through the game area. When simulated contact between the player's image and virtual objects in the game area is determined by the control means, the impact generating means positioned at the apparent point of contact on the player's body is activated to apply pressure to the player at that point, thereby simulating contact between the player and the virtual game area.

One of the main disadvantages associated with such a system is that the player must be positioned in a relatively large, substantially empty physical space to prevent them from colliding with objects which they are moving around. In practice, this problem is generally overcome by providing some form of hand control which allows the user to simulate movement through the virtual reality game area by turning in the direction in which they wish to move and then using the hand control to give a'move forward'command.

The concept of"Z-keying"is known in the prior art as a technique for merging multiple graphics and video streams. The streams to be merged are assumed to come from coincident viewpoints. Each stream has a corresponding depth value. Pixel values for frames in the resulting merged stream are taken from the corresponding pixel location of the source stream with the lowest depth (z) value. As a result, virtual objects can be combined into real world scenes.

Augmented reality systems also exist whereby the head mounted display is effectively seethrough so that the user can see computer-generated graphical objects which are superimposed on physical objects present within the physical space in which the player is located, the computer-generated objects being substantially the same as the true physical objects in size, shape and nature. Such systems have the obvious advantage that the player can see the surrounding physical environment and is therefore free to move around without the fear of colliding with any objects which are present therein. However, one of the major disadvantages

is of course that the true physical environment (say a living room or the like) and the objects within it are not as stimulating and exciting as a typical graphical games environment.

We have now devised an arrangement which overcomes the problems outlined above.

Summary of the Invention Thus, in accordance with the present invention, there is provided Interactive image display apparatus, comprising portable display means for displaying a virtual scene, means for generating an at least partially virtual scene including one or more virtual objects, and means for rendering said virtual scene in said display means for viewing by a user, wherein the apparent distance of any point in the user's field of view within the virtual scene to the nearest virtual object in said virtual scene is no further than the actual distance in the same direction of view to the nearest physical object within the physical space surrounding the user.

In one embodiment of the present invention, the means for generating a virtual scene may comprise means for determining a depth map from the user's viewpoint of the physical space surrounding said user, and means for generating said virtual scene such that its virtual depth is no greater than the corresponding depth of said depth map of said physical space. The depth map preferably comprises a pixel image of the physical scene within the user's field of view, wherein each pixel represents the distance to the nearest physical object within said field of view.

In contrast to the concept of"Z-keying"the apparatus of an exemplary embodiment of the present invention operates in a different manner in that it uses the determined depth information relating the surrounding physical world to cover at least parts of that physical world completely. In one embodiment, the apparatus operates to cover up the real world completely by explicitly generating a graphical world which is always closer than the real world equivalent.

The apparatus beneficially comprises means for generating a new virtual scene in response to a change of the user's viewpoint, the new virtual scene preferably being displayed on display means as a smooth perturbation of the preceding rendered virtual scene from the preceding user viewpoint.

A preferred embodiment of the present invention comprises a motion sensor for providing a sensory input of the relative change of position of the user, the sensed change of position being translated into a corresponding change in the virtual position of the viewpoint of the virtual scene, the motion sensor preferably being located on a user's head, when in use.

In another embodiment of the invention, the means for generating a virtual scene comprises means for generating a (at least approximate) three-dimensional model of the physical space surrounding the user from the user's viewpoint, a three-dimensional virtual world being generated such that virtual objects appear to surround all of the physical objects within said physical space, and the virtual scene rendered in the display for view by the user provides a view of the said virtual world in accordance with the position of said user. The three dimensional model of the physical space may predetermined for a given physical space.

Alternatively, the apparatus may comprise means for determining depth information from a user's viewpoint at any given time, said depth information relating to the physical space surrounding said user, and sensor means carried or worn by the user for determining the physical location and/or orientation of said user, said depth information and said information relating to the physical location and/or orientation of the user being used to generate a threedimensional model of the physical space from the user's viewpoint.

In any event, the apparatus preferably comprises sensor means for determining the user's position and orientation relative to the surrounding physical space, and/or imaging means for determining the user's position relative to the surrounding physical space. Such imaging means may be arranged to identify one or more markers carried or worn by a user so that their position can be determined accordingly, or such imaging means may comprise image capturing means carried or worn by the user, and include means for recognising one or more elements or locations within an image captured thereby, means for determining its relative

location within a surrounding physical space and means for determining its relative location within a corresponding virtual scene by identifying the corresponding one or more elements or locations within said virtual scene. In any event, the imaging means may comprise stereo imaging means.

In one exemplary embodiment of the present invention, a three dimensional model of the physical space surrounding the user can be updated to take into account changes in the surrounding physical space since said three dimensional model was generated, and, optionally, the first three dimensional model can be stored and retrieved for use when required. In this case, the three dimensional virtual model corresponding to said first three dimensional model of the surrounding physical space can beneficially also be stored and retrieved when required.

In a first exemplary embodiment of the present invention, the virtual scene may comprise an entirely virtual environment which includes virtual obstacles or entities that appear to occupy at least the same space as the physical obstacles or entities within the surrounding physical space which a user would wish to avoid colliding with.

In another exemplary embodiment of the present invention, the virtual scene may comprise an augmented scene, with only moving entities (such as people) within the physical space being represented within the displayed scene as virtual entities. In this case, the display means may be substantially opaque, the scene rendered in said display means for view by the user comprising an image of the surrounding physical space captured by image capturing means carried or worn by the user. Alternatively, the display means may be substantially transparent with opaque pixels being rendered in the display means in the form of a partial virtual scene for masking one or more areas of the surrounding physical space viewed by the user through the display means.

In all cases, the display means preferably comprises a head mounted display (HMD).

If required, the virtual obstacles or entities may be, or appear to be, larger than the physical obstacles or entities they are intended to represent, and optionally, additional virtual obstacles or entities may be provided within the displayed scene which do not represent obstacles or entities within the surrounding physical space.

The apparatus preferably comprises means arranged to adapt said displayed scene, in real time, to the three-dimensional physical space in which a user is moving, and may include warning means, such as an audio or tactile signal, arranged to alert the user of any potential collision with a physical object or entity.

The apparatus may include means (particularly where the apparatus comprises apparatus for playing games) for providing virtual moving entities within said displayed scene, said entities being perceived to move through the virtual environment but being constrained by the same physical environment as is the user and, in the case where there are two or more users operating within the same virtual and physical environments (and each having their own display means), each user is preferably represented in the virtual scene by a different virtual character, the position of which within the virtual scene is determined by the user's position within the surrounding physical space. In this case, the apparatus is preferably arranged such that the or each user can select to be represented within the operating area by one of a plurality of virtual characters, such as monsters, robots, animals, soldiers, etc. , according to the nature of the game.

The apparatus beneficially comprises means for replacing one virtual scene with another virtual scene which is adapted to the same physical layout of the physical space in which the apparatus is being operated.

Further, the apparatus may comprise motion sensing means for sensing motion of a user within the surrounding physical space, means for magnifying said motion and means for translating such motion to motion within said virtual space such that motion of the user within said surrounding physical space appears as substantially faster motion within said virtual scene.

It will be apparent that, unlike the situation with totally augmented reality systems, registration between real physical objects or entities and the virtual reality cloaking objects or entities intended to mark their presence and position need not be exact, nor is the speed of rendering the virtual reality cloaking objects or entities quite as critical as it is with augmented reality alignment. It will be appreciated that it is sufficient to ensure that the virtual reality cloaking objects or entities appear to present virtual surfaces which effectively surround the physical objects or entities with a large enough safety margin to prevent collision. In practice, it is really only necessary to ensure that the clear space perceived by the user in the virtual scene is in fact clear within the physical environment in which they are moving. This would at least give the user an opportunity to safely move through the physical environment by ensuring that all physical obstacles or entities are marked as such in the displayed scene.

Of course, there is nothing to prevent the user from attempting to move through the virtual surface (thinking, perhaps, that it is simply a virtual object or entity which does not cloak a physical object or entity) and, therefore, colliding with a physical obstacle or entity in their path. For this reason, warning means, such as an audio or tactile signal, may be provided to alert the user of any potential collision with a physical object or entity.

Many different types of virtual environment are envisaged. In the case where the apparatus forms part of a games system, the virtual scene could comprise, for example, one or more caves, tunnels or rooms in a building. The apparatus provides a high degree of realism in this case, because the user can move freely within their physical environment (without fear of collision with any obstacles), and such movement is translated into simulated movement through the virtual environment which is displayed in three dimensions. Any games might include the provision of virtual characters, such as monsters, enemies, etc. which are also perceived to move through the virtual environment but which are preferably constrained by the same physical environment as is the user.

The present invention can, by its very nature, be used in a variety of different physical environments, including within a residential building, garden, garage, etc. In the case that the user only has use of a single confined space, such as a living room or bedroom, the apparatus

may comprise means for replacing one virtual scene with another virtual scene, as mentioned above, the second virtual scene being adapted to the same physical layout of the physical space in which the apparatus is being operated, i. e. both operating areas or virtual environments including virtual images which cloak the same physical environment layout.

Brief Description of the Drawings An embodiment of the present invention will now be described by way of example only and with reference to the accompanying drawing which is a perspective view of a head mounted display system for use in apparatus according to an exemplary embodiment of the present invention.

Detailed Description of the Invention In order to effectively'cloak'obstacles present in the physical space surrounding the apparatus of the present invention with virtual reality images appearing in appropriate positions within the virtual area as viewed by a user during use, there are three main issues to be considered.

Firstly, the three-dimensional free space around the user at any one time (in the direction in which they are nominally looking) needs to be mapped. Secondly, a three-dimensional virtual environment needs to be generated with its free space lying within the known free space of the surrounding physical space. Finally, any movement by the user within the physical environment needs to be monitored and fed back so that the virtual environment can be appropriately updated (while still conforming to the available free space within the surrounding physical environment).

Referring now to Figure 1 of the drawings, a head mounted display 10 for use in an exemplary embodiment of the present invention comprises a depth sensor which can determine the definite free space in the field of view of the player. Depth sensors for mapping the profile of a predetermined environment or area, and therefore determining the definite free space therein (using, for example, sonar, laser range-finding or stereo vision camera systems) are well known.

In the head mounted display of Figure 1, a stereo vision system is employed and, as such, two cameras 12,14 are mounted on opposing sides of the front panel 16 of the head mounted display 10. The cameras 12,14 capture images within their field of view, and a plurality of points on the images captured by both cameras 12,14 are searched until two points are found (one from each image captured by a respective camera) which correspond to the same point on a three dimensional object appearing in the images. The position coordinates of these corresponding points on the images and the relative positions of the two cameras 12,14 are used to provide data as to the specific point on the object in a three-dimensional space. By identifying a plurality of such corresponding points, the system can recognise the position, shape and size of a three dimensional object within its field of view. Using this method, all of the obstacles within the user's field of view can be recognised and the remaining free space can be determined. An operating area is generated having the same perceived free space as is available within the surrounding physical environment, any obstacles being marked or 'cloaked'with virtual images which are in keeping with the theme of the displayed scene.

A relative position sensor 18, comprising for example an accelerometer or gyroscope, is mounted generally centrally on the front panel 16 of the head mounted display 10. The relative position sensor measures relative motion of the user's head with respect to the evolving three-dimensional virtual environment. An (absolute) orientation sensor 20 is also mounted generally centrally on the front panel 16 of the head mounted display 10 to measure the orientation of the cameras 12,14 with respect to gravity.

The head mounted display 10 is retained on a user's head (so that the front panel 16 thereof covers the user's eyes) by means of a pair of side arms 22, similar to those used to retain conventional spectacles in position during use.

Consider a simple exemplary embodiment of the present invention comprising a game based on the exploration of a plurality of tunnels. Within each of the tunnels is a number of rocks, and aliens (or the like) appear from behind the rocks. The player is required to shoot the aliens as they appear.

The stereo vision camera system can be used to determine the largest area of free space available (in any one view) and the orientation of its ground plane within the physical space surrounding the apparatus, and the apparatus then creates virtually the largest tunnel corresponding thereto and populates it with rocks and aliens. As the wearer moves their head and body, the relative motion is measured and the position of the previously-constructed tunnel is updated and evolved to agree with the updated depth map created by the stereo vision camera system according to the updated head position. As the user'moves'through the tunnel, additional tunnel elements (such as branches, additional rocks, extensions, etc) can be added according to a simple, predefined tunnel modelling scheme and the evolving free space around the user. It should be noted that the evolution of tunnels need not be entirely geometrically consistent in order to give the impression of inhabiting a virtual world.

In the case where there are two or more players, each wearing their own head mounted display 10, each head mounted display 10 could be provided with an additional imaging system (similar to the stereo vision camera system described above and possibly sharing common elements therewith). Each player could be provided with an arrangement of markers (worn above the head, say) which are recognisable by the imaging system. Such an arrangement of markers would have a known geometry such that the imaging system could be used to determine the position and three-dimensional orientation of the markers (and therefore the associated player) with respect thereto (as is the case in automated motion capture systems).

Using this information, the additional players could be'cloaked'within the virtual environment by the generation of respective virtual images having the same approximate poses as the additional players. A similar marking and imaging system could be applied to cloak key pieces of equipment. For example, a stick used by a player could be portrayed in the virtual environment as a sword or similar weapon.

In an alternative embodiment of the invention, the game could be played within an augmented world, with only the players being cloaked. In other words, instead of generating a virtual environment, the players effectively see the real world through, for example, a head mounted video and display system, and within this environment, real players are replaced in real time by cloaked counterparts using substantially the same techniques as are described above.

In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be apparent to a person skilled in the art that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.

Claims

CLAIMS 1. Interactive image display apparatus, comprising portable display means for displaying a virtual scene, means for generating an at least partially virtual scene including one or more virtual objects, and means for rendering said virtual scene in said display means for viewing by a user, wherein the apparent distance of any point in the user's field of view within the virtual scene to the nearest virtual object in said virtual scene is no further than the actual distance in the same direction of view to the nearest physical object within the physical space surrounding the user.
2. Apparatus according to claim 1, wherein said means for generating a virtual scene comprises means for determining a depth map from the user's viewpoint of the physical space surrounding said user, and means for generating said virtual scene such that its virtual depth is no greater than the corresponding depth of said depth map of said physical space.
3. Apparatus according to claim 2, wherein said depth map comprises a pixel image of the physical scene within the user's field of view, wherein each pixel represents the distance to the nearest physical object within said field of view.
4. Apparatus according to any one of claims 1 to 3, comprising means for generating a new virtual scene in response to a change of the user's viewpoint.
5. Apparatus according to claim 4, wherein said new virtual scene is displayed on said display means as a smooth perturbation of the preceding rendered virtual scene from the preceding user viewpoint.

<Desc/Clms Page number 13>
6. Apparatus according to claim 4 or claim 5, comprising a motion sensor for providing a sensory input of the relative change of position of the user, the sensed change of position being translated into a corresponding change in the virtual position of the viewpoint of the virtual scene.
7. Apparatus according to claim 6, wherein said motion sensor is located on a user's head, when in use.
8. Apparatus according to claim 1, wherein said means for generating a virtual scene comprises means for generating a (at least approximate) three-dimensional model of the physical space surrounding the user from the user's viewpoint, a three- dimensional virtual world being generated such that virtual objects appear to surround all of the physical objects within said physical space, and the virtual scene rendered in the display for view by the user provides a view of the said virtual world in accordance with the position of said user.
9. Apparatus according to claim 8, wherein said three dimensional model of said physical space is predetermined for a given physical space.
10. Apparatus according to claim 8, comprising means for determining depth information from a user's viewpoint at any given time, said depth information relating to the physical space surrounding said user, and sensor means carried or worn by the user for determining the physical location and/or orientation of said user, said depth information and said information relating to the physical location and/or orientation of the user being used to generate a three-dimensional model of the physical space from the user's viewpoint.
11. Apparatus according to claim 8, comprising sensor means for determining the user's position and orientation relative to the surrounding physical space.

<Desc/Clms Page number 14>
12. Apparatus according to claim 8, comprising imaging means for determining the user's position relative to the surrounding physical space.
13. Apparatus according to claim 12, wherein said imaging means is arranged to identify one or more markers carried or worn by a user so that their position can be determined accordingly.
14. Apparatus according to claim 12, wherein said imaging means comprises image capturing means carried or worn by the user, and includes means for recognising one or more elements or locations within an image captured thereby, means for determining its relative location within a surrounding physical space and means for determining its relative location within a corresponding virtual scene by identifying the corresponding one or more elements or locations within said virtual scene.
15. Apparatus according to claim 12, wherein said imaging means comprises stereo imaging means.
16. Apparatus according to any one of claims 8 to 13, wherein a three dimensional model of the physical space surrounding the user can be updated to take into account changes in the surrounding physical space since said three dimensional model was generated.
17. Apparatus according to claim 14, wherein said first three dimensional model can be stored and retrieved for use when required.
18. Apparatus according to claim 15, wherein the three dimensional virtual model corresponding to said first three dimensional model ofthe surrounding physical space can also be stored and retrieved when required.

<Desc/Clms Page number 15>
19. Apparatus according to any one of the preceding claims, wherein said virtual scene comprises an entirely virtual environment which includes virtual obstacles or entities that appear to occupy at least the same space as the physical obstacles or entities within the surrounding physical space which a user would wish to avoid colliding with.
20. Apparatus according to claim 1, wherein said virtual scene comprises an augmented scene, with only moving entities (such as people) within the physical space being represented within the displayed scene as virtual entities
21. Apparatus according to claim 20, wherein said display means is substantially opaque, the scene rendered in said display means for view by the user comprising an image of the surrounding physical space captured by image capturing means carried or worn by the user, said apparatus including means for modifying said image after capture by replacing pixels in one or more areas thereof with their virtual image equivalents, prior to display.
22. Apparatus according to claim 20, wherein said display means is substantially transparent with opaque pixels being rendered in the display means in the form of a partial virtual scene for masking one or more areas of the surrounding physical space viewed by the user through the display means.
23. Apparatus according to any one of the preceding claims, wherein said display means comprises a head mounted display (HMD).
24. Apparatus according to any one of the preceding claims, wherein the virtual obstacles or entities are, or appear to be, larger than the physical obstacles or entities they are intended to represent.

<Desc/Clms Page number 16>
25. Apparatus according to any one of the preceding claims, wherein there are additional virtual obstacles or entities within said displayed operating area which do not represent obstacles or entities within the surrounding physical space.
26. Apparatus according to any one of the preceding claims, arranged to adapt said

displayed scene, in real time, to the three-dimensional physical space in which a user is moving.
27. Apparatus according to any one of the preceding claims, including warning means, such as an audio or tactile signal, arranged to alert the user of any potential collision with a physical object or entity.
28. Apparatus according to any one of the preceding claims, including means for providing virtual moving entities within said displayed scene, said entities being perceived to move through the virtual environment but being constrained by the same physical environment as is the user.
29. Apparatus according to any one of the preceding claims, wherein, in the case where there are two or more users operating within the same virtual and physical environments (and each having their own display means), each user is represented in the virtual scene by a different virtual character, the position of which within the virtual scene is determined by the user's position within the surrounding physical space.
30. Apparatus according to claim 29, arranged such that the or each user can select to be represented within the operating area by one of a plurality of virtual characters.
31. Apparatus according to any one of the preceding claims comprising means for replacing one virtual scene with another virtual scene which is adapted to the same physical layout of the physical space in which the apparatus is being operated.

<Desc/Clms Page number 17>
32. Apparatus according to any one of the preceding claims, comprising motion sensing means for sensing motion of a user within the surrounding physical space, means for magnifying said motion and means for translating such motion to motion within said virtual space such that motion of the user within said surrounding physical space appears as substantially faster motion within said virtual scene.
33. Interactive image display apparatus substantially as herein described with reference to the accompanying drawing.