WO2010023348A1 - Interactive displays - Google Patents
Interactive displays Download PDFInfo
- Publication number
- WO2010023348A1 WO2010023348A1 PCT/FI2008/050475 FI2008050475W WO2010023348A1 WO 2010023348 A1 WO2010023348 A1 WO 2010023348A1 FI 2008050475 W FI2008050475 W FI 2008050475W WO 2010023348 A1 WO2010023348 A1 WO 2010023348A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- shadow
- image
- luminosity
- pointing object
- infrared
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
- G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
Definitions
- the present invention generally relates to interactive displays.
- Interactive displays which comprise a projector or a Liquid Crystal Display panel to form an image on a surface, a camera behind the surface and computer vision circuitry for detecting user input.
- US 2001/0012001 A1 discloses one such display system.
- This system comprises a semi-transparent screen and an infrared LED (light emitting diode) panel, a CCD (charge coupled device) camera and a projector provided on the back side of the semitransparent screen.
- the camera operates on infrared wavelengths and detects infrared light reflected from objects on the other side of the semitransparent screen, when the objects are relatively close to the screen or when they touch the screen.
- the projector projects visible image light onto the screen. Infrared component of the image projected by the projector may be filtered out. Thereby the projected image does not disturb the camera.
- infrared emitter/detector pairs placed behind backlight of an LCD (liquid crystal display) display are used to detect objects on or in the vicinity of the LCD display, the LCD display thus operating as an interactive display.
- the use of an LCD display may overcome many of the problems of the projector-based interactive screens.
- the disclosed technology however requires number of infrared emitter/detector pairs to be mounted on the system, if input is to be detected on a larger area, whereby production of such interactive displays is laborious and thus high costs may be involved.
- the display In the absence of ambient IR light, the display may illuminate objects on or near its surface by means of IR illumination, but it is very difficult to counter excessive ambient IR illumination.
- Detecting an object on the surface of the display with an IR camera is complicated by the presence of sunshine or other strongly IR- emitting illuminators such as halogen lamps. Namely, in this case the infrared- emission that the system produces may not be strong enough to cause near-by objects to appear as highlights.
- one known technology for detecting a finger touching the display surface is to detect when an image of the finger becomes crisp.
- a diffusive screen is attached onto the display surface so that a touching finger blocks IR light from being diffused by the diffusive screen under the finger and a relatively sharp shadow is formed conforming to the shape of the contact.
- any object casts a sharp shadow on the camera even from a distance. Hence, it may be impossible to detect a contact simply from the sharpness of the shadow.
- the camera signal may be flushed with measured light such that it is generally impossible or at least very difficult to distinguish any meaningful picture.
- US20050064936A1 describes a method in which a finger may be recognized from its shadow and distinguished from other shadows by using parameters which are characteristic to a finger, such as color, shade, reflectance, particular grey color change at the perimeter of the shadow caused by the skin of a finger, and by making use of the expected use based on the layout of usable radio buttons, see paragraphs 239 to 246.
- This publication lists some optical parameters without explaining how they should be used and moreover requires very accurate color cameras in order to detect fine effects such as the grey boundary through the skin at the edge of the finger.
- This publication deals with particular problems present in a car equipment, but does not address a situation in which there is a strong single backlight behind the finger.
- WO0184251 A2 presents a panel display with which movement of a finger is being detected by using histograms, see e.g. page 16 and Figs. 10a-11f.
- histograms is a known image processing method in which pixels are compared.
- the disclosure of this publication does not disclose whether it could be deduced whether a shadow is caused by a proximate or distant shadow.
- the publication neither discloses how to address problems caused by ambient light.
- the use of histograms of this publication loses the shape of the finger, whereas the shape would be needed for computing the width of a finger.
- the publication is focused on light areas rather than shadows.
- WO0184251 further requires computationally intensive histogram calculations and database comparisons so that its implementation requires relatively expensive and complex equipment.
- WO2004091956A2 discloses a vehicle display device panel. This publication discusses using of ambient light i.e. back illumination of a finger such that the finger is seen as a shadow, as described in the paragraph bridging pages 56 and 57. The size of a shadow caused by a finger may also be used for determining whether a particular pressure is applied against the screen, if a dedicated piezoelectric sensor is not used (page 51 to 52). Clearly, in order to identify the force of a finger tip pressing against the display, the finger should be pressed in a consistent angle against the display.
- the geometry between the driver's hand and a display mounted on the dashboard may determine that the angle of a finger is relatively consistent so that the width of the finger may indicate the force of the finger as the image caused by the finger then broadens.
- the inventors have realised that if the geometry is not that well defined, the finger may be placed in a widely varying angle in comparison to the display such that the shape and size of the shadow may vary in a very broad range.
- the disclosed detecting of the force of a finger against the screen by measuring the size of the shadow is not universally usable for detecting the point of touch by a finger.
- an interactive display device comprising: an image layer configured to provide a user with a perceivable visible image and configured to pass through infrared light; an image capture unit configured to capture an infrared image through the image layer; a recognizing unit configured to detect from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region; wherein the recognizing unit is configured to detect the contact areas under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.
- a method in an interactive display device comprising: providing a user with a perceivable visible image on an image layer configured to and configured to pass through infrared light; capturing an infrared image through the image layer; detecting from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region; wherein the detecting of the contact areas is performed under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.
- a computer program configured to cause when executed by a computer a method according to the first aspect of the invention.
- a computer readable memory medium embodied with a computer program which when executed by a computer causes a computer to perform a method according to the first aspect of the invention.
- Fig. 1 shows a block diagram of a system according to an embodiment of the invention
- Fig. 2 shows a simplified block diagram of the structure of a control unit shown in Fig. 1 according to an embodiment of the invention
- Fig. 3 shows a schematic flow chart according to an embodiment of the invention for illustrating a first phase in a process of determining a pointing object from an Infrared (IR) image captured by an IR image capture unit from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light;
- IR Infrared
- Fig. 4 illustrates according to an embodiment of the invention for illustrating a second phase in a process of determining a pointing object from an IR image captured by an IR image capture unit from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light;
- Fig. 5 shows one graph demonstrating a shadow gradient width test
- Fig. 6 shows another graph demonstrating a shadow gradient width test
- Fig. 7 shows an example of an infrared image of a pointing object when partly in contact with a display.
- an LCD screen is used as an example of an image layer.
- the LCD screen may comprise a film that forms a visible image and optionally also other elements such as background illumination, infrared (IR) illumination, incoming IR intensity detection across the LCD film, and/or one or more speakers for outputting sound. It is appreciated that the invention may likewise be applied with any other types of image layers as well, or screens for simpler expression.
- an image layer is any means for producing a visible image for a user, such as an electric display, a screen displaying a projected image or a substance carrying printed, dyed image, or woven image.
- Fig. 1 shows a block diagram of a system 100 according to an embodiment of the invention.
- Fig. 1 also shows a user 113 for facilitating the description of particular order of different elements.
- the system 100 is suited for use as an interactive user interface device e.g. as a built in dynamic menu in a restaurant, as a display screen at a ticket office, or generally wherever an interactive display and optical pointing recognition is desired.
- the system 100 comprises as an outmost element or as facing to the user 113 a touching layer 101 such as a toughened glass plate, then an LCD film 102 and a diffusion layer 103 behind the LCD film.
- a touching layer 101 such as a toughened glass plate
- the touching layer, the LCD film 102 and the diffusion layer 103 are all in a compact stack such that the distance from the touching layer to the diffusion layer is as low as possible for machine view performance reasons that will be described in more detail in this description.
- the purpose of the diffusion layer 103 is to spread the light coming from a background light source (described in the following), so that an image displayed on the LCD film appears even from many directions. This spreading can be achieved with a diffusion film or with a holographic rear-projection film.
- a background light source described in the following
- the diffusion layer 103 behind the LCD film 102 significant advantages may be achieved in comparison to placing the diffusion layer 103 in front of the LCD film 102 or to omitting the diffusion layer 103. Namely, the diffusion layer 103 reducing reflections from the typically glossy backside of the LCD film 102, which reflections may interfere with the recognizing of pointing objects. It is also advantageous to locate the diffusion layer 103 behind the LCD film 102 when seen by a user 113, because otherwise it provides an additional reflecting surface between the LCD film 102 and the viewer thus impairing the image quality or contrast especially.
- a casing 104 In order to support the aforementioned optical elements, there is provided a casing 104.
- the casing 104 comprises a back wall 105 attached to side walls 106.
- the side walls are attached from one end to the back wall 105 and from their opposite ends to the touching layer 101 , LCD film 102 and diffusion layer 103.
- a background light source 108 may be located in the casing 104 for background illumination of the LCD film 102.
- the background light source 108 may comprise, for example, one or more of: LED light, light conductor, fluorescent light, and luminescence light.
- the diffusion layer 103 is omitted. This may particularly be the case when an evenly illuminating background light is provided by a luminescence light that in itself provides an even field of illumination on the LCD film 102.
- the side walls 106 may be coated from the inner side with some reflective material in order to deliver maximum amount of light to the LCD film and finally to users of the system.
- the reflective side walls one may avoid or at least reduce shadows that may be caused to the background light in the IR image captured by a camera behind the LCD film 102.
- the reflective side walls may also help in delivering the background light to the LCD film in a way that the light can be spread to the users without allowing users to see details inside the system through the LCD film and at the same time improving the viewing angle of the system.
- the diffusion layer 103 may help to achieve this effect. More particularly, the side walls may be for example mirror walls, in which case a particularly good consistency of an image on the image layer may be achieved.
- an image capture unit 107 that is in this particular case an IR camera configured to see through the diffusion layer 103, LCD film 102 and the touching layer 101 any objects placed near or at the touching layer 101.
- Fig. 1 there is drawn as a dotted area an IR light reception space 107 for the IR camera.
- the camera is configured to detect signals that are outside the visible light wavelengths. There may be for example a filter in front of the lens of the camera providing this effect.
- the camera may be based on for example CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) technology.
- the image capture unit 107 may comprise more than one camera e.g.
- the distance between the background light source 108 and the diffusion layer 103 may generally depend on the space that the camera setup requires. The distance may be shortened for more compact sized system e.g. by moving the camera forward and/or turning the camera around (to point away from the screen) and using a mirror or prism in front of the camera to capture the activities on the screen.
- the image capture unit 107 may be formed of a planar sensing structure that has a number of sensors spread over a plane to recognize intensity of incoming light through each pixel or pixel block of the LCD film 102. Such a structure may also double as a visible light and/or IR illumination behind the LCD film 102 for the purpose of background illuminating the LCD screen and/or illuminating objects in front of the LCD screen under weak ambient IR illumination.
- a planar sensing structure operating as an image capture unit 107 there is no IR light reception space 107 as that drawn in Fig. 1 but instead there is merely a small parallel space between the LCD film and the image capture unit.
- the image capture unit may be integrated directly into the image plane, for example the LCD film.
- the camera 107 and the IR LEDs 110 may not be seen from the outside of the system if their shadow is negligent e.g. due to the highly diffused illumination coming from the background lights and as the reflective inner side of the system provides evens further illumination of the display surface. Further, the diffusion layer 103 may prevent the users from seeing any details from inside the system.
- a camera as the IR image capture unit typically provides a more economical solution than a plurality of emitter/detector pairs that are typically used in a planar illumination and sensing element.
- the area covered by one camera typically requires a relatively high number of emitter/detector pairs thus incurring a relatively high number of components and wires.
- production of the system may be simpler when a camera is used.
- the size of the system may be reduced and there is no need for accurately positioning and directing the camera in a desired angle with regard to the LCD film 102.
- the system 100 may further comprise an IR reflection inhibitor 109 configured to reduce reflection of IR light from the back wall and again from any elements in front of the camera 107.
- the IR reflection inhibitor 109 may comprise, for instance, a black plate such as a sooted aluminum plate placed around the background light source 108 or behind the background light source 108.
- IR reflection inhibitor 109 may comprise, for instance, a directionally refractive or reflective element such as one or more prisms configured to direct IR light to such a direction that the amount of IR light reflection to the camera 107 is significantly reduced.
- the IR reflection inhibitor 109 may comprise additionally or alternatively a selective screen between the background light source 108 and the IR light reception space 107.
- a screen that filters out IR light may be simply arranged e.g. in a plane configuration behind the camera parallel to the back wall 105 in sake of simple installation.
- the screen may be conically arranged partly on the front side of the camera 107 when seen from the direction of the user 113. While a planar configuration is economical and simple to arrange, conical or different curved arrangements may further enhance the operation of the IR reflection inhibitor 109, whether implemented using a filter or an absorbing surface. Namely, there is always an amount of reflection and by suitably shaping the IR reflection inhibitor 109 it may be possible further reduce reflections of IR light all the way from the outside of the case 104 through the LCD and back and forth inside the case 104 to the camera 104.
- the IR reflection inhibitor 109 represents an advantageous implementation that may be used in the present invention, but this document is more particularly focused on the way in which a pointing object 112 is determined from the IR image.
- a control unit 11 which is configured to control operation of the system and/or to detect the pointing object 112.
- the system 100 may comprise IR light sources 110 enabling input detection on IR wavelengths.
- the IR light sources may be for example IR LEDs placed outside the angle of view of the image capture unit.
- the IR light sources 110 may be located outside a cone formed by the view area of the camera.
- the LCD screen itself may be configured to provide the IR illumination across its view area or the IR illumination may be configured to be produced such that it illuminates objects at the LCD screen without first passing through the LCD screen.
- the system 100 may further comprise an audio speaker 114 for providing audible signals to the user 113.
- the system may be configured to e.g. provide a tapping sound to indicate determined tapping on the touching surface 101 for enhancing user experience of an operable system and to teach users that there is no need for applying substantial force against the touching surface when the recognition of the pointing object 112 is optically performed.
- Fig. 2 shows a simplified block diagram of the structure of the control unit 111.
- the control unit 111 may be based on, for example, a general purpose computer supplied with suitable software and / or on a particularly adapted computing device. While it is possible to implement the control unit 111 by purely hardware based a device, typically it is more economic and faster to produce by making use of software.
- control unit 111 is drawn to comprise a memory 201 that comprises a work memory 202, a non-volatile memory 203 that is configured to store software 204, presentation information 205 describing content to be presented by the system 100 and/or how pointing at different areas on the screen should be treated, and settings 206 needed e.g. for manual or automatic calibration of the system 100.
- the software 204 may comprise any one or more of the following items: operating system, device drivers, display presentation application, hypertext markup language parser, image processing software, and drivers for different external equipment that may be connected to the system such as printers, further displays, further interactive systems 100, audio systems, and external IR illumination equipment (not shown).
- the control unit 111 further comprises a processor 207 configured to control the operation of the control unit 111 according to the software 204 by executing computer executable program code contained by the software in the work memory 202. Alternatively, the control unit may be configured to execute the software in place in the non-volatile memory in which case the work memory may not be necessary.
- the control unit further comprises an input/output unit (I/O) 208 for exchanging signals with other elements of the system 100 and optionally also with external equipment.
- the I/O 208 may comprise e.g. any one or more of a universal serial bus port, a local area network port, an ISA bus, a PCI express port, an IR port, a Bluetooth element, and a parallel port.
- the system 100 may be provided with a transferable memory reception unit 209 such as a cd-rom or dvd- rom drive, memory card reader or memory stick reader which enables replacing part of the non-volatile memory e.g. for updating information to be displayed on the LCD screen 102.
- a transferable memory reception unit 209 such as a cd-rom or dvd- rom drive, memory card reader or memory stick reader which enables replacing part of the non-volatile memory e.g. for updating information to be displayed on the LCD screen 102.
- control unit In order to control the operation of various components of the system and to obtain the captured image, there are connections between the control unit or particularly its input/output unit 208 and other components of the system 100, while not shown in sake of clarity of the drawing.
- the control unit has generally the task of receiving a signal from the camera 107, detecting if and where the touching layer 101 is pointed at and typically also outputting the determination in a standard way e.g. emulating a computer drawing tablet, mouse or other known pointing device.
- control unit operation may comprise following acts: - controlling the LCD film to show desired images to the user 113 - controlling the IR lights 110 to produce IR light on demand for showing a pointing object 112 such as a user ' s 113 finger when brought close to the LCD film
- control unit may consist of one separate unit
- control unit 111 may alternatively be integrated with any other element or comprise two or more discreet elements each for one or more of the aforementioned acts.
- Fig. 3 shows a schematic flow chart according to an embodiment of the invention for illustrating a first phase in a process of determining a pointing object from an IR image captured by an IR image capture unit 107 from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light.
- a finger tip is sought as an example of a typical pointing object 112 at the touching layer 101 by the control unit 111.
- the system 100 operates by continually repeating the following process to determine a pointing object when brought at the touching layer 101.
- the flow chart presents the process for one single time of verifying if and where there are pointing objects at the touching layer 101.
- the process may alternatively be stopped once any one pointing object is found if so desired. It is an advantage of attempting to identify all pointing objects that if a user e.g. places many fingers on the touching layer 101 , many different contacts may be identified and all or none of respective areas may be detected as being pointed depending on the implementation.
- the process starts 301 when the camera operates and also typically the LCD film
- the 102 is presenting some still or motion picture to the user 113 and the control unit receives a digitized IR image from the camera or from an analogue-digital converter (possibly comprised by the control unit) in case that the image capture unit 107 has an analogue output.
- the digital IR image comprises a set of pixels typically such that the area corresponding to the touching layer 101 is divided into a matrix of x columns and y rows. In the matrix, each cell represents one pixel with a value that is typically of 8 bits depth i.e. has 256 different numerical values representative of the IR luminosity detected by the image capture unit for that cell.
- an initial pixel is selected.
- the initial cell may be chosen e.g. near an area corresponding to one corner of the touching layer such that the distance to nearest edges is about half the thickness of a thin finger tip.
- step 303 it is then checked whether the luminosity value of the selected cell is significantly below its neighboring pixels.
- This test may be carried out e.g. by dividing the luminosity value of the selected cell by the highest luminosity value in the cells at a given distance around the selected cell or by subtracting from the highest luminosity value of cells at a given distance around the selected cell and comparing the result to a predetermined threshold. Alternatively any other linear or non-linear comparison method may equally be applied.
- step 303 If the result the checking in step 303 is negative, then the selected cell is not likely one representing a finger tip and the process advances to step 304 to verify whether all feasible cells have been tested i.e. whether the testing should be finished. If yes, the process ends at step 305, alternatively the process advances to step 306 in which a next pixel is selected for testing.
- the next pixel may be selected by stepping onwards along a slice of the image by a distance of half the width of a narrow finger tip or less. Alternatively, coarser sampling may be employed to reduce required computation or denser sampling may be employed for better accuracy.
- step 303 for again checking whether a promising starting point would be selected for verifying whether it belongs to a shadow of a finger tip.
- a threshold luminosity value is defined between the background luminosity and the pixel luminosity.
- the value one may estimate the background luminosity with various methods. For example one can search the near-by pixels for the highest value, which would then be the luminosity value. Alternatively one can choose a subset of near-by pixels, for example by shooting short rays into different directions and collecting the brightest pixel along the rays. Further, one can take histogram of the near-by pixels and instead of taking the very brightest pixel, use a pixel value that is selected from the histogram at a given positioning.
- step 308 the control unit next shoots rays in different directions preferably with even angle offset e.g. into 8 different directions and checks the luminosity at each pixel along the rays, one by one until a pixel is found along the ray with a luminosity above a given threshold indicates that an edge region has been reached in the shadow of a suspected finger tip.
- step 309 it is tested whether a given number, here at least two, of the shadow lengths exceed or reach a given threshold. If yes, then the selected pixel is not likely to belong to a finger tip region and the process resumes to step 304. If no, the lengths of opposite rays are summed up to form the length of a shadow of given darkness in four different directions with 45 degree offset (in case of eight evenly distributed rays).
- the width of the finger tip is initially estimated 310 by choosing from the summed lengths of rays of opposite directions i.e. lengths along a direct line, by rejecting summed lengths which exceed a predetermined maximum length threshold and then taking the shortest of the remaining lengths.
- step 309 the order of the processing steps may be varied e.g. so that already in step 309 the summed lengths would be used. If all of the summed lengths exceed the predetermined maximum length threshold, the process may jump to step 304 in an embodiment in which such a finding is held as an indication that the selected pixel is not near an edge of a pointing object.
- step 309 when the estimated width of the finger tip (shadow) is calculated, the estimated width is stored and the selected pixel is marked as a potential finger tip pixel. The process then continues from step 401 in Fig. 4.
- Fig. 4 illustrates according to an embodiment of the invention for illustrating a second phase in a process of determining a pointing object from an IR image captured by an IR image capture unit 107 from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light.
- step 401 the process continues from step 401 at which potential finger tip pixels are merged into groups by using any commonly known flood fill technique, for example recursive fill, right-hand fill or scan-line fill.
- the flood fill produces a group of pixels representing an arbitrary shaped estimate area on the touching layer 101.
- width of the estimate area is calculated 402 in four adjacent directions with 45 degrees steps and an average of these four widths is calculated as a refined width estimate for the finger tip shadow. This calculation is done by averaging the length values of each of the pixels in the group.
- the merged area is tested 403 by comparing the number of pixels within the estimate area resulting from the flood fill to a given maximum area and if the estimate area is too small (or in one embodiment if the area is too large), the selected pixel is rejected from consideration as a likely pixel representing a finger tip, and the process resumes to step 304 of Fig. 3. Otherwise, the estimate area size testing is passed and the process may continue.
- the image can be presented in a number of ways. The methods may work on raw pixel data (as the algorithm in figure 3) or use for example low-pass (and/or high-pass) filtered versions of the image, or perform a global or local optimization or normalization on the image. Also any other commonly known image processing methods may be applied.
- a maximum allowable luminosity l max is calculated based on the refined width estimate.
- the maximum allowable luminosity may be derived for example by dividing the maximum luminosity value of the nearby pixels (that are clearly outside group) with some numeric constant, for example 0,3. Further l max may be limited so that it is never less than certain percentage of the maximum luminance value of camera image in the whole tracing area. In effect, this step dynamically adjusts the threshold for determining whether a shadow cast by the finger tip is dark enough. Naturally other strategies may be applied for this purpose.
- a histogram of the near-by pixel values may be calculated and the histogram may be used as an input to numeric analysis tools such as Bayesian estimator or artificial neural network that computes the value for l max . Also other linear- and non-linear methods may be used to calculate the value for l max .
- step 405 a central pixel or an average of a predetermined number of central pixels of the merged set of pixels is identified and a center luminosity value is derived from the central pixel or pixels.
- step 406 the center luminosity is compared to the maximum allowable luminosity and if the center luminosity does not exceed the maximum allowable luminosity l max , the process advances to step 407 wherein the merged group is accepted as a finger tip shadow and the process may in advance to step 304 to check whether the captured IR image is completely scanned or whether still further finger tips should be sought from the image. If, however, in step 406 the center luminosity is found to exceed the maximum allowable luminosity l max , the merged group of pixels is not considered as a finger tip shadow and the operation proceeds to step 304.
- the method takes implicitly into account the sharpness of the shadow.
- This sharpness information may be used in steps 308 and 309 in the following way: if the shadow is not sharp, then the lengths measured in 308 will be relatively long and the non-fingertip will be rejected in step 309, due to the long rays measured in step 308.
- a shadow gradient width test may be added.
- the shadow gradient width test may verify that the shadow is sharp enough so that it is not caused by far-away objects, under diffuse ambient illumination.
- One way to perform this test is to shoot rays into different directions from the estimated center of the fingertip as before the rays can be shot into eight directions. For each ray one then calculates two luminosity threshold values: start value Gb for the beginning of the gradient and end value Ge for the end of the gradient, based on the luminosity at the center of the finger and luminosity outside the center of the finger. Examples of how the luminosity may behave along the ray are presented in figures 5 and 6.
- shadow gradient width i.e. the distance between the start value Gb and the end value Ge crossing is short, so the shadow is sharp (thin gradient around the shadow center)
- distance between respective Gb and Ge crossings at respective distances dT and d2' is substantially longer, so the shadow gradient is weak along the ray.
- the gradient may be weak towards many of the rays, but strong towards other directions for example if the finger is not perpendicular to the screen.
- Such methods may be configured to present the darkness of the center of the potential fingertip and other relevant parameters such that calculations may identify which shadows are caused by objects in contact with the display, and which are caused by objects further away from the surface.
- the control unit may also be configured to automatically detect the edges of the LCD screens and to calibrate the parameters that govern the transformation from camera co-ordinates to screen-co-ordinates. Additionally the software may be configured to automatically eliminate the parts of the image that are not part of the active LCD film (for example LCD margins and any support structures) from the input processing.
- the control unit preferably provides identification of pointing object regions to an application that controls the content being showed on the LCD screen 102, or more generally to an application that is configured to make use of the pointing object location as a user input.
- Fig. 7 shows an example of an infrared image of a pointing object when partly in contact with a display.
- the pointing object 112 is a hand and five fingers are in contact with an external surface i.e. touching layer or image layer depending on implementation.
- the picture has been taken through a diffusive screen with near- infrared range light or radiation.
- words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.
- term light here is interchangeable with radiation. While infrared light has in occasions been used, this terming is merely for convenience of explanation the term light is not intended to imply suitability for perception by means of a human eye.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
- User Interface Of Digital Computer (AREA)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP08787748.6A EP2332027A4 (en) | 2008-08-26 | 2008-08-26 | INTERACTIVE DISPLAYS |
CN200880130919.5A CN102132239B (zh) | 2008-08-26 | 2008-08-26 | 交互式显示屏 |
PCT/FI2008/050475 WO2010023348A1 (en) | 2008-08-26 | 2008-08-26 | Interactive displays |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/FI2008/050475 WO2010023348A1 (en) | 2008-08-26 | 2008-08-26 | Interactive displays |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010023348A1 true WO2010023348A1 (en) | 2010-03-04 |
Family
ID=41720871
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/FI2008/050475 WO2010023348A1 (en) | 2008-08-26 | 2008-08-26 | Interactive displays |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP2332027A4 (zh) |
CN (1) | CN102132239B (zh) |
WO (1) | WO2010023348A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10589716B2 (en) | 2014-03-26 | 2020-03-17 | Magna Mirrors Of America, Inc. | Vehicle function control system using sensing and icon display module |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9198496B2 (en) * | 2012-07-02 | 2015-12-01 | Sony Corporation | Makeup TV |
JP6111706B2 (ja) * | 2013-02-01 | 2017-04-12 | セイコーエプソン株式会社 | 位置検出装置、調整方法、および調整プログラム |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010012001A1 (en) * | 1997-07-07 | 2001-08-09 | Junichi Rekimoto | Information input apparatus |
US6775014B2 (en) * | 2001-01-17 | 2004-08-10 | Fujixerox Co., Ltd. | System and method for determining the location of a target in a room or small area |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5483261A (en) * | 1992-02-14 | 1996-01-09 | Itu Research, Inc. | Graphical input controller and method with rear screen image detection |
GB9603330D0 (en) * | 1996-02-16 | 1996-04-17 | Thomson Training & Simulation | A method and system for determining the point of contact of an object with a screen |
EP2213501A3 (en) * | 2003-03-31 | 2012-05-09 | Timothy R. Pryor | Reconfigurable vehicle instrument panels |
NO323926B1 (no) * | 2004-11-12 | 2007-07-23 | New Index As | Visuelt system samt styringsobjekt og apparat til bruk i systemet. |
JP2008146203A (ja) * | 2006-12-07 | 2008-06-26 | Seiko Epson Corp | 電気光学装置、及び電子機器 |
CN100462900C (zh) * | 2007-05-24 | 2009-02-18 | 司秉玉 | 用于与显示***进行交互操作的方法及设备 |
-
2008
- 2008-08-26 EP EP08787748.6A patent/EP2332027A4/en not_active Withdrawn
- 2008-08-26 WO PCT/FI2008/050475 patent/WO2010023348A1/en active Application Filing
- 2008-08-26 CN CN200880130919.5A patent/CN102132239B/zh active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010012001A1 (en) * | 1997-07-07 | 2001-08-09 | Junichi Rekimoto | Information input apparatus |
US6775014B2 (en) * | 2001-01-17 | 2004-08-10 | Fujixerox Co., Ltd. | System and method for determining the location of a target in a room or small area |
Non-Patent Citations (2)
Title |
---|
See also references of EP2332027A4 * |
WILSON, A. D.: "PlayAnywhere: A Compact Interactive Tabletop Projection-Vision System", UIST '05: PROCEEDINGS OF THE 18TH ANNUAL ACM SYMPOSIUM ON USER INTERFACE SOFTWARE AND TECHNOLOGY. NEW YORK: ACM, - October 2005 (2005-10-01), NEW YORK: ACM, pages 83 - 92, XP002541983 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10589716B2 (en) | 2014-03-26 | 2020-03-17 | Magna Mirrors Of America, Inc. | Vehicle function control system using sensing and icon display module |
US10889265B2 (en) | 2014-03-26 | 2021-01-12 | Magna Mirrors Of America, Inc. | Vehicle function control system using sensing and icon display module |
US11427156B2 (en) | 2014-03-26 | 2022-08-30 | Magna Mirrors Of America, Inc. | Vehicular function control using sensing device with backlit icons |
Also Published As
Publication number | Publication date |
---|---|
CN102132239A (zh) | 2011-07-20 |
CN102132239B (zh) | 2015-11-25 |
EP2332027A4 (en) | 2016-08-03 |
EP2332027A1 (en) | 2011-06-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108700934B (zh) | 能够进行眼睛追踪的可穿戴设备 | |
US8847924B2 (en) | Reflecting light | |
EP2321697B1 (en) | Spatially adaptive photographic flash unit | |
EP2335141B1 (en) | Interactive display device with infrared capture unit | |
US9480397B2 (en) | Gaze tracking variations using visible lights or dots | |
US7593593B2 (en) | Method and system for reducing effects of undesired signals in an infrared imaging system | |
US10805543B2 (en) | Display method, system and computer-readable recording medium thereof | |
CN101836180B (zh) | 检测视觉***中的环境光级别 | |
TW201101140A (en) | Active display feedback in interactive input systems | |
EP2128693A1 (en) | Spatially Adaptive Photographic Flash Unit | |
US20120206416A1 (en) | Interactive Display | |
KR101507458B1 (ko) | 대화식 디스플레이 | |
EP2332027A1 (en) | Interactive displays | |
JP2006350452A (ja) | 撮影表示装置およびそのキャリブレーションデータ算出方法 | |
KR101385263B1 (ko) | 가상 키보드를 위한 시스템 및 방법 | |
US10928894B2 (en) | Eye tracking | |
JP2004326188A (ja) | 大画面タッチパネルシステムおよび検索・表示システム | |
KR20190027081A (ko) | 전자 장치, 그 제어 방법 및 컴퓨터 판독가능 기록 매체 | |
JP6197163B1 (ja) | 入出力システム、入力処理装置、入出力方法、及び、プログラム | |
JP2018113011A (ja) | 入出力システム、入力処理装置、入出力方法、及び、プログラム | |
JP2016158158A (ja) | 画像データ生成システム、プログラム及び画像データ生成方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 200880130919.5 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08787748 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REEP | Request for entry into the european phase |
Ref document number: 2008787748 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2008787748 Country of ref document: EP |