GB2470597A

GB2470597A - Detecting movement or shape of a reflective element in an image

Info

Publication number: GB2470597A
Application number: GB0909227A
Authority: GB
Inventors: Jeffrey Raynor
Original assignee: STMicroelectronics Ltd Great Britain
Current assignee: STMicroelectronics Ltd Great Britain
Priority date: 2009-05-29
Filing date: 2009-05-29
Publication date: 2010-12-01
Also published as: GB0909227D0

Abstract

A method of tracking an object 20 comprises providing the object with a reflective or retro-reflective element 22, illuminating a space containing the object, imaging the space with a camera 18 having a pixel based image sensor 18b, and tracking movement and or shape of the radiation reflected from the reflective element. An example implementation is a gaming controller, with the illumination and tracking devices comprising an infra-red LED 14 and a digital image sensor provided at a display screen 10 on which a game to be controlled is displayed. The object may be a pointing device or a hand held gaming implement such as a gun that has a light source that is activated when a trigger is operated. The trigger may operate a mechanical shutter arranged to obscure part of the reflective member. The reflective member may have a particular shape so as to define indicia which is distinctive of one object among others. There may be two light sources 14 and two cameras 18.

Description

I

INPUT SYSTEM AND METHOD

This invention relates to a system and method for providing an input of information defining one or more of position, movement and identity to a computer system.

One field of application for the present invention is in computer gaming, where the user must be provided with an input device for interacting with the game. Currently most of these fall into three types: 1. Joystick or game controller, either a 4-axis joystick (digital or analog) or a controller dedicated to a particular console type. These offer a limited user experience. Also, they require a certain amount of circuitry in the controller and either a hard-wired or a wireless link to the computer.

2. Computer mouse. Although this is usually the preferred pointing device for PC use, it requires a flat, 2-D surface and is therefore not particularly suitable for games, especially console games which are played from a chair and without a suitable flat mousing surface available.

3. Gyro-controller, such as the Wil� controller. These devices use a MEMS accelerometer in the hand unit which measures force and can deduce the direction and velocity of motion, and also position. Therefore this type of input device is a closer mimic to the real world, which enhances the enjoyment of the game. This is a popular solution, but it has the disadvantage that the system intelligence and therefore cost is in the hand unit.

Many games involving weapons have a number of weapons available.

However, currently they are selected using a menu system and then the same controller is used to play the game.

One object of the present invention is to provide a user input system which addresses the above problems. Additionally the invention may be applied to non-gaming situations, such as the identification of goods or persons.

The invention is based on the concept of providing a device which interacts with a computer, for example as a pointing device, where the device itself requires no (or extremely limited) electronics, and no hard-wired or wireless data path between the device and the computer.

Accordingly, the invention provides a method of tracking and/or identifying an object, comprising: providing a reflecting area on the object, the reflecting area having a reflectance substantially higher than that of the object; illuminating a space containing the object; imaging said space on a pixel-based image sensor; and analysing the output of the image sensor to determine movement and/or shape of the reflecting area.

The method may be used in tracking the object, in which case the analysing step may comprise locating the position of the reflecting area within the image in successive frames.

Alternatively, the method may be used in identifying the object, in which case the analysing step may comprise identifying the shape or pattern of the reflecting area.

The method may also combine tracking and identifying.

From another aspect, the invention provides a system for tracking and/or identifying an object, comprising: a reflective member formed on or attachable to the object, the reflective member having a reflectance substantially higher than that of the object; a light source arranged to illuminate a space containing the object; a camera having a pixel-based image sensor; and image analysing means adapted to analyse the output of the image sensor to determine movement and/or shape of the reflecting area.

Preferably the reflective member is retroreflective.

In some embodiments of the invention, the reflective area of the reflective member is formed into a particular shape, or so as to define indicia, which is distinctive of one object among others.

In a particularly preferred application of the invention, the object is a hand-held gaming implement, and the camera is connected to a computer capable of running gaming software.

In this case the system typically includes a computer display screen, and the camera is preferably located adjacent the display screen and arranged to image a space occupied by a game user.

The camera preferably includes a lens having a field of view of 55°-80°.

Two cameras may be provided at spaced locations adjacent the display screen.

The object may be a pointing device provided with a user-operable trigger.

Activation of the trigger may be communicated to the camera by the trigger activating a second light source mounted on the pointing device, the second light source emitting light which is distinguishable from the first-mentioned light source. Alternatively, activation of the trigger may be communicated to the camera by the trigger operating a mechanical shutter arranged to obscure at least part of the reflective member.

Preferably, the image analysing means is operable to distinguish the reflective member from background by applying a single threshold level to the output of each pixel.

The light source preferably emits non-visible light, and may suitably be in the form of an infrared light-emitting diode.

Embodiments of the present invention will now be described, by way of example only, with reference to the drawings, in which: Fig. 1 is a schematic perspective view of one system embodying the invention; Fig. 2 shows examples of reflector patterns which may be used in embodiments of the invention; and Fig. 3 is a view similar to Fig. 1 of a modified embodiment.

Referring to Fig. 1, a display screen 10 of a computer is provided with at least one optical assembly 12 which comprises a light source 14 and a camera 18 formed by a lens 18a and a solid state image sensor 18b. The camera 18 is preferably mounted adjacent one corner of the screen 10 and approximately in the plane of the front face of the screen 10. The camera 18 points towards the area where a user would typically view the screen and play a game.

A suitable form of light source 14 is a light-emitting diode, most suitably an infrared LED to be invisible to the user. The camera 18 may be similar to those used in mobile telephones or as web cameras, and therefore small in size and cheap to manufacture. The lens 18a however is best modified from those used in such applications, as discussed below. Solid state image sensors, typically CMOS devices, sensitive to infrared are widely available at low cost.

In this embodiment the user device is an imitation gun 20, and it is desired to recognise the aiming axis of the gun 20 relative to the screen 10.

There are various well-known tracking algorithms which could in principle be used to track movement of the gun, for example face tracking systems used in video conferencing, or those used in optical mice. Such systems typically use non-cooperative targets, that is objects that do not have any added feature to enable or assist tracking, and they require large amounts of processing to determine the location of the desired object in the field of view. This processing comes at a cost -either in silicon (if the image processing is performed in a parallel manner) or in time (if the image processing is performed in a serial manner).

For an input system for a consumer gaming system, these trade-offs are undesirable as the input device must be both cheap and also responsive to the fact movements that are typical in computer game playing.

The present invention provides a cooperative system where the cooperating means on the handheld device (or equivalent) is very limited.

In Fig. 1, an area of retro-reflective material 22 is provided on the end of the gun 20. The retro-reflective material 22 may be glued, printed or embossed on the surface of the gun. Suitable retro-reflective materials are commonly available, for example Avery-Dennison� Ti 000 series, 3M Scotchlite� Engineer Grade 2200 and 3200, and others.

The retro-reflective material 22 is highly efficient at reflecting the infrared light from the source i4 and there is therefore a large contrast, typically iO:i or greater, between the retro-reflective material 22 and the rest of the scene. This large contrast greatly simplifies the task of the tracking system very simple methods to find and track the target; for example a simple threshold ing system can detect whether the pixel intensity at any point in the image sensor is greater than a defined threshold, if yes then the target is located.

During playing of the game, it is typical for the pointer (e.g. gun 20) to be moved rapidly and so it is necessary for the sensor system to be able to react quickly and to track objects that are moving quickly. One way to achieve this is to pulse the illumination source for a short duration as this will reduce the effect of motion blur. It is further advantageous to synchronize the operation of the sensor and the illumination source.

There are two ways of achieving this: One is by the use of a rolling blade shutter. Rolling blade shutters (named as the electronic version behaves in a similar manner to the mechanical version found on silver-halide film cameras) operate in a row-sequential manner. This is because rows share a common output conductor ("bitline") and so two rows cannot be read out at the same time they need to be sequential.

Hence the typical operation would be: For i = row 1 to NumberRows Reset Row (i) Integrate Row (i) Next i All the rows of the sensor are now in integrate mode Pulse IR LED illumination source For i = row ito NumberRows Readout Row (i) Reset Row (i) (Optional) Next i The other is by use of a global shutter. The problem with the rolling blade shutter approach is that the sensor is sensitive to light for a "long period" (at least the time to readout one frame). Hence the contrast ratio between the target and the rest of the scene is reduced (because the target is illuminated only for the few ps that the LED is on, but the scene is illuminated for the 1 Oms -lOOms of readout.

Hence, it is preferential to use a "global" shutter system -the well-known 4 transistors per pixel architecture is suitable (or even a modification which uses 5 transistors -the extra transistor providing an over-flow drain in case of high illumination levels).

The mode of operation would then be: Reset all pixels Integrate all pixels Flash LED Readout all pixels, i.e. transfer charge from photodiode to storage node Readout all the rows For highest spatial resolution of the pointer, it is preferable have a narrow field of view (FOV), i.e. observe only a small region of space. With such a system a movement (e.g. 1cm in object space) has a greater movement on the sensor (image space) when a narrow FOV is used. Unfortunately a narrow field of view implies that the camera needs to be accurately pointed at the player's weapon -something that is difficult or impractical to achieve.

For an easy-to-use consumer system, it is therefore advantageous to use an optical system (lens 1 8a) with a larger field of view (e.g. 550.800 diagonal FOV) than that used by webcams or mobile phone cameras (500 FOV).

A wider FOV makes it much easier for the user to install/set-up the system as there is greater tolerance in aiming the camera. This is at the expense of reduced spatial resolution/accuracy of tracking.

A further enhancement is to use a patterned retro-reflector, e.g. a cross, series of concentric circles to produce a target with clear and easily recognizable edges, such as shown in Fig. 2. This will help to restore some of the pointing accuracy I resolution lost when a larger FOV lens is used.

A pointing device alone (such as a drum stick), does not need a "trigger" or "mouse button", but a device such as a game-gun would need the functionality of a trigger. This could be implemented in many ways, such as an electrical switch for the trigger and a RF transmitter indicating that the trigger/switch button is depressed. This requires a RF receiver unit at the games console. An alternative method is to illuminate an LED on the gun when the trigger is pressed. This would be detected by the camera -removing the need for a separate RF receiver unit. The circuitry required on the gun is minimal, and a small battery would power this for a long time.

To distinguish between the light from the "trigger" and that from the pointing retro-reflector, it would be possible to pulse the light from the "trigger" at a rate lower (< 1/4 that) of the pointing illumination. Hence the "trigger" would appear to flash on and off and would be easy to differentiate the "trigger" light from the "pointing" light.

It would even be possible to produce a gun "trigger" which was completely (electrically) passive, by means of a mechanical shutter mechanism which obscures the tracking retroreflector when the trigger is depressed. The system would then "fire" the bullet when the retroreflector disappeared.

This type of toy would not require any electronics or power source.

Unfortunately, the "gun" would also "fire" if the tracker momentarily lost contact with the target (e.g. obscured by another player).

Optionally, the mechanical shutter could obscure part of the retroreflector target or even replace the retroreflector target with a different pattern to indicate that the gun has fired.

It is possible to include positional tracking information on the same substrate as the image sensor.[clarify?] The sensor then outputs only the position of the target. This has the benefit of having a low speed output (e.g. USB 1) and cheaper interconnections. Alternatively, it is possible to output the image (or cropped part of the image) and so a high speed interface is required (e.g. USB2, SM IA, COP, CSI2 or other streaming protocol standard).

Figure 1 shows the use of two cameras 18. This is the preferred system as there is less chance of a camera being obscured and also the possibility of increasing system accuracy by combining the positional data from both cameras.

Furthermore, if the cameras are capable of web-cam mode of operation (i.e. output an image instead of just positional information), then having two cameras allows for 3-D/stereoscopic image generation, such as 3D avatars etc. This is a key advantage of the optical pointing system (where the expense is in the fixed camera) over the MEMS accelerometer pointing system (where the expense is in the moving controller).

Figure 3 shows an improved system. Here there are two (or more) "different" weapons 20a and 20b. In each case the "weapon" is plastic in the shape of an (optionally) different gun design. There is no need for expensive tracking electronics in the gun. (There may be electronics associated with the trigger/fire button, but this is relatively cheap). The retroreflector is applied to the weapons such that the weapon type can be distinguished, in this example by means of the weapon 20a having a single retroreflective patch 22, and the weapon 22b having two retroreflective patches 22.

The image analysis in the camera 18 is arranged to detect the extra retroreflector 22 (optionally of a different design to the target retro-reflector) and then deduce that the user is holding and using weapon "B" instead of (or as well as!) "A". The software could then adapt (e.g. the velocity, trajectory, accuracy, momentum of the bullet) to the type of weapon the player is holding. To change weapons, the game player would have to physically carry the appropriate weapon; e.g. draw them from a holster and replace them when not required. This is closer to what is performed "in real life" and hence adds further realism to the game.

The marking differentiation may be as simple as the number of retro-reflectors on the object, as in Fig. 3, or it may be more complicated and include 1-D or 2-D barcodes of retro-reflecting material.

If there is enough spatial resolution, it may be possible to encode a relatively large number of bits in the image which would enable not only the production of a large range of "guns" (each type of gun having a common ID), but also the possibility of encoding a unique identification on each gun.

This ID could be used to identify the player and select the appropriate avatar in the game (e.g. the gaming system "knows" that weapon #98765 is owned by user #1 234 Mr. Joshua Rambow and so when it sees a valid weapon number, this is transmitted from the console to the server and the server instructs the console which avatar, gamer profile etc. to use or even to un-lock various aspects (levels, features etc.) of the game, thereby encouraging the sale of further weapons.

This need not be limited to weapons; it would be possible to have different markings on different sticks that were used as "virtual drums" or other "magic wands".

It may be that the markings on the items indicate their "hand" (i.e. 1 stripe for left, 2 for right) or the camera systems could deduce their positions from their relative locations in the image.

Although the invention has largely been described above with respect to computer games, the invention in its broader concept can be used in other applications.

It can, for example, be used as a non-gaming pointing device as a form of general computer input. One application would be for use by disabled persons to select characters or icons on a computer screen. It is known to use head-mounted pointing devices for this purpose, but the present invention would provide a simpler and lighter weight pointing device.

The invention can also be applied purely to identification of objects, rather than tracking movement.

It would be possible to mark items with retroreflecting barcodes which then would be read using a camera system similar to that described previously.

It would also be possible to use this technique to identify people. It is common for emergency services etc. to wear high-visibility jackets I vests etc. which usually incorporate a large retroreflector. It would be possible to use all or part of this large retroreflector as an ID area if there is structure (e.g. barcode, lettering) on the retroreflector, either by patterning the retro- reflector during manufacture of the garment, or even by applying a self-adhesive ("negative") label which obscured part of the retroreflector thereby producing a barcode.

Such an arrangement would have the advantage of a high level of contrast, allowing simple reading by pixel thresholding.

The invention thus provides an improved tracking and/or identification system in which a computer interacts with an input device, but where the input device is extremely simple and cheap to produce.

Claims

CLAIMS1. A method of tracking and/or identifying an object, comprising: providing a reflecting area on the object, the reflecting area having a reflectance substantially higher than that of the object; illuminating a space containing the object; imaging said space on a pixel-based image sensor; and analysing the output of the image sensor to determine movement and/or shape of the reflecting area.
2. A method according to claim 1, for use in tracking the object, in which the analysing step comprises locating the position of the reflecting area within the image in successive frames.
3. A method according to claim 1, for use in identifying the object, in which the analysing step comprises identifying the shape or pattern of the reflecting area.
4. A method which combines tracking in accordance with claim 2 and identifying in accordance with claim 3.
5. A system for tracking and/or identifying an object, comprising: a reflective member formed on or attachable to the object, the reflective member having a reflectance substantially higher than that of the object; a light source arranged to illuminate a space containing the object; a camera having a pixel-based image sensor; and image analysing means adapted to analyse the output of the image sensor to determine movement and/or shape of the reflecting area.
6. A system according to claim 5, in which the reflective member is retroreflective.
7. A system according to claim 5 or claim 6, in which the reflective area of the reflective member is formed into a particular shape, or so as to define indicia, which is distinctive of one object among others.
8. A system according to any of claims 5 to 7, in which the object is a hand-held gaming implement, and in which the camera is connected to a computer capable of running gaming software.
9. A system according to claim 8, including a computer display screen, and in which the camera is located adjacent the display screen and arranged to image a space occupied by a game user.
10. A system according to claim 9, in which the camera includes a lenshaving a field of view of 55°-80°.
11. A system according to claim 9 or claim 10, in which two cameras are provided at spaced locations adjacent the display screen.
12. A system according to any of claims 8 to 11, in which the object is a pointing device and is provided with a user-operable trigger.
13. A system according to claim 12, in which the trigger activates a second light source mounted on the pointing device, the second light source emitting light which is distinguishable from the first-mentioned light source.
14 A system according to claim 12, in which the trigger operates a mechanical shutter arranged to obscure at least part of the reflective member.
15. A system according to any of claims 5 to 14, in which the image analysing means is operable to distinguish the reflective member from background by applying a single threshold level to the output of each pixel.
16. A system according to any of claims 5 to 15, in which the light source emits non-visible light, preferably being in the form of an infrared light-emitting diode.