WO2011069157A2 - Methods and systems for position detection - Google Patents
Methods and systems for position detection Download PDFInfo
- Publication number
- WO2011069157A2 WO2011069157A2 PCT/US2010/059104 US2010059104W WO2011069157A2 WO 2011069157 A2 WO2011069157 A2 WO 2011069157A2 US 2010059104 W US2010059104 W US 2010059104W WO 2011069157 A2 WO2011069157 A2 WO 2011069157A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- set forth
- computing system
- space
- coordinate
- image
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
- G06F3/0428—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual
Definitions
- Touch-enabled computing devices have become increasingly popular. Such devices can use optical, resistive, and/or capacitive sensors to determine when a finger, stylus, or other object has approached or touched a touch surface, such as a display.
- a touch surface such as a display.
- the use of touch has allowed for a variety of interface options, such as so- called “gestures” based on tracking touches over time.
- Embodiments of the present subject matter include a computing device, such as a desktop, laptop, tablet computer, a mobile device, or a computing device integrated into another device (e.g., an entertainment device for gaming, a television, an appliance, kiosk, vehicle, tool, etc.).
- the computing device is configured to determine user input commands from the location and/or movement of one or more objects in a space.
- the object(s) can be imaged using one or more optical sensors and the resulting position data can be interpreted in any number of ways to determine a command.
- the commands include, but are not limited to, graphical user interface events within two-dimensional, three-dimensional, and other graphical user interfaces.
- an object such as a finger or stylus can be used to select on-screen items by touching a surface at a location mapped to the on-screen item or hovering over the surface near the location.
- the commands may relate to non-graphical events (e.g., changing speaker volume,
- Some embodiments may rely on other input in addition to the position data, such as a click of a physical button provided while a finger or object is at a given location.
- the same system may be able to interpret other input that does not feature a touch.
- the finger or stylus may be moved in a pattern that is then recognized as a particular input command, such as a gesture that is recognized based on or more heuristics that correlate the pattern of movement to particular commands.
- movement of the finger or stylus in free space may translate to movement in the graphical user interface. For instance, crossing a plane or reaching a specified area may be interpreted as a touch or selection action, even if nothing is physically touched.
- the object's location in space may influence how the object's position is interpreted as a command. For instance, a movement of an object within one part of the space may result in a different command than an identical movement of the object within another part of the space.
- a finger or stylus may be moved along one or two axes within the space (e.g., along a width and/or height of the space), with the movement in the one or two axes resulting in corresponding movement of the cursor in a graphical user interface.
- the same movement at different locations along a third axis (e.g., at a different depth) may result in different corresponding movement of the cursor.
- a left-to-right movement of a finger may result in faster movement of the cursor the farther the finger is from a screen of the device.
- This can be achieved in some embodiments by using a virtual volume (referred to as an "interactive volume" herein) defined by a mapping of space coordinates to screen/interface coordinates, with the mapping varying along the depth of the interactive volume.
- a first zone can be defined near a screen of the device and a second zone can be defined elsewhere.
- the second zone may lie between the screen and keys of a keyboard of a laptop computer, or may represent imageable space outside the first zone in the case of a tablet or mobile device.
- Input in the first zone may be interpreted as touch, hover, and other graphical user interface commands.
- Input in the second zone may be interpreted as gestures. For instance, a "flick" gesture may be provided in the second zone in order to move through a list of items, without need to select particular items/command buttons via the graphical user interface.
- aspects of various embodiments also include irradiation, detection, and device configurations that allow for image-based input to be provided in a responsive and accurate manner. For instance, detector
- fewer than all available pixels from the detector are sampled, such as by limiting the pixels to a projection of an interactive volume and/or determining an area of interest for detection by one detector of a feature detected by a second detector.
- FIGs. 1A-1D illustrate exemplary embodiments of a position detection system.
- Fig. 2 is a diagram showing division of an imaged space into a plurality of zones.
- FIG. 3 is a flowchart showing an example of handling input based on zone identification.
- Fig. 4 is a diagram showing an exemplary sensor configuration for providing zone-based detection capabilities.
- FIG. 5 is a cross-sectional view of an illustrative architecture for an optical unit.
- Fig. 6 is a diagram illustrating use of a CMOS-based sensing device in a position detection system.
- Fig. 7 is a circuit diagram illustrating one illustrative readout circuit for use in subtracting one image from another in hardware.
- FIGs. 8 and 9 are exemplary timing diagrams illustrating use of a sensor having hardware for subtracting a first and second image.
- Fig. 10 is a flowchart showing steps in an exemplary method for detecting one or more space coordinates.
- Fig. 11 is a diagram showing an illustrative hardware configuration and corresponding coordinate systems used in determining one or more space coordinates.
- Figs. 12 and 13 are diagrams showing use of a plurality of imaging devices to determine a space coordinate.
- Fig. 14 is a flowchart and accompanying diagram showing an illustrative method of identifying a feature in an image.
- Fig. 15A is a diagram of an illustrative system using an interactive volume.
- Figs. 15B-15E show examples of different cursor responses based on a variance in mapping along the depth of the interactive volume.
- Fig. 16 is a diagram showing an example of a user interface for configuring an interactive volume.
- Figs. 17A-17B illustrate techniques in limiting the pixels used in detection and/or image processing.
- Fig. 18 shows an example of determining a space coordinate using an image from a single camera.
- Fig. 1A is a view of an illustrative position detection system 100
- Fig. IB is a diagram showing an exemplary architecture for system 100.
- a position detection system can comprise one or more imaging devices and hardware logic that configures the position detection system access data from the at least one imaging device, the data comprising image data of an object in the space, access data defining an interactive volume within the space, determine a space coordinate associated with the object, and determine a command based on the space
- the position detection system is a computing system in which the hardware logic comprises a processor 102 interfaced to a memory 104 via bus 106.
- Program components 116 configure the processor to access data and determine the command.
- a software-based implementation is shown here, the position detection system could use other hardware (e.g., field
- FPGA programmable gate arrays
- PLA programmable logic arrays
- memory 104 can comprise RAM, ROM, or other memory accessible by processor 102 and/or another non-transitory computer- readable medium, such as a storage medium.
- System 100 in this example is interfaced via I/O components 107 to a display 108, a plurality of irradiation devices 110, and a plurality of imaging devices 112. Imaging devices 112 are configured to image a field of view including space 114.
- multiple irradiation and imaging devices are used, though it will be understood that a single imaging device could be used in some embodiments, and some embodiments could use a single irradiation device or could omit an irradiation device and rely on ambient light or other ambient energy.
- Memory 104 embodies one or more program components 116 that configure the computing system to access data from the imaging device (s) 112, the data comprising image data of one or more objects in the space, determine a space coordinate associated with the one or more objects, and determine a command based on the space coordinate.
- program components 116 that configure the computing system to access data from the imaging device (s) 112, the data comprising image data of one or more objects in the space, determine a space coordinate associated with the one or more objects, and determine a command based on the space coordinate.
- I/O interfaces 107 comprising a graphics interface (e.g., VGA, HDMI) can be used to connect display 108 (if used).
- I/O interfaces include universal serial bus (USB), IEEE 1394, and internal busses.
- networking components for communicating via wired or wireless communication can be used, and can include interfaces such as Ethernet, IEEE 802.11 (Wi-Fi), 802.16 (Wi-Max), Bluetooth, infrared, etc., CDMA, GSM, UMTS, or other cellular communication networks.
- Fig. 1A illustrates a laptop or netbook form factor.
- irradiation and imaging devices 110 and 112 are shown in body 101, which may also include the processor, memory, etc. However, any such components could be included in display 108.
- Fig. 1C shows another illustrative form factor of a position detection system 100'.
- a display device 108' has integrated irradiation devices 110 and imaging devices 112 in a raised area at the bottom of the screen. The area may be approximately 2mm in size.
- the imaging devices image a space 114' including the front area of display device 108'.
- Display device 108' can be interfaced to a computing system (not shown) including a processor, memory, etc.
- the processor and additional components could be included in the body of display 108'.
- the principles could be applied for other devices, such as tablet computers, mobile devices, and the like.
- Fig. ID shows another illustrative position detection system 100".
- imaging devices 112 can be positioned either side of an elongated irradiation device 110, which may comprise one or more light emitting diodes or other devices that emit light.
- space 114" includes a space above irradiation device 110 and between imaging devices 112.
- the image plane of each imaging device lies at an angle ⁇ between the bottom plane of space 114", and ⁇ can be equal or approximately equal to 45 degrees in some
- irradiation device 110 may not be centered on space 114".
- Irradiation device 110 and imaging devices 112 could be included in or mounted to a keyboard positioned in front of a separate screen as well.
- irradiation device 110 and imaging devices 112 could be included in or attached to a screen or tablet computer.
- irradiation device 110 and imaging devices 112 may be included in a separate body mounted to another device or used as a standalone peripheral with or without a screen.
- imaging devices 112 could be provided separately from irradiation device 110.
- imaging devices 112 could be positioned on either side of a keyboard, display screen, or simply on either side of an area in which spatial input is to be provided.
- Irradiation device(s) 110 could be positioned at any suitable location to provide irradiation as needed.
- imaging devices 112 can comprise area sensors that capture one or more frames depicting the field of view of the imaging devices.
- the images in the frames may comprise any representation that can be obtained using imaging units, and for example may depict a visual representation of the field of view, a representation of the intensity of light in the field of view, or another representation.
- the processor or other hardware logic of the position detection system can use the frame(s) to determine information about one or more objects in space 114, such as the location, orientation, direction of the object(s) and/or parts thereof. When an object is in the field of view, one or more features of the object can be identified and used to determine a coordinate within space 114 (i.e., a "space coordinate").
- the computing system can determine one or more commands based on the value of the space coordinate.
- the space coordinate is used in determining how to identify a particular command by using the space coordinate to determine a position, orientation, and/or movement of the object (or recognized feature of the object) over time.
- different ranges of space coordinates are treated differently in determining a command.
- the imaged space can be divided into a plurality of zones. This example shows an imaging device 112 and three zones, though more or fewer zones may be defined;
- the zones may vary along the length, width, and/or depth of the imaged space.
- An input command can be identified based on determining which one of a plurality of zones within the space contains the determined space coordinate. For example, if a coordinate lies in the zone ("Zone 1") proximate the display device 108, then the movement/position of the object associated with that coordinate can provide different input than if the coordinate were in Zones 2 or 3.
- the same imaging system can be used to determine a position component regardless of the zone in which the coordinate lies.
- multiple imaging systems are used to determine inputs.
- one or more imaging devices 112 further from the screen can be used to image zones 2 and/or 3.
- each imaging system passes a screen coordinate to a routine that determines a command in accordance with Fig. 3.
- one or more line or area sensors could be used to image the area at or around the screen, with a second system used for imaging one or both of zones 2 zone 3. If the second system images only one of zones 2 and 3, a third imaging system can image the other of zones 2 and 3.
- the imaging systems could each rely on one or more aspects described below to determine a space coordinate.
- multiple imaging systems could be used within one or more of the zones.
- zone 3 may be handled as a plurality of sub-zones, with each sub-zone imaged by a respective set of imaging devices. Zone coverage may overlap, as well.
- the imaging system for zone 1 could use triangulation principles to determine coordinates relative to the screen area, or each imaging system could use aspects of the position detection techniques noted herein. That same system could also determine distance from the screen. Additionally or alternatively, the systems could be used cooperatively. For example, the imaging system used to determine a coordinate in zone 1 could use triangulation for the screen coordinate and rely upon data from the imaging system used to image zone 3 in order to determine a distance from the screen.
- Fig. 3 is a flowchart showing an example of handling the input based on zone identification and can be carried out by program components 116 shown in Fig. 1 or by other hardware/software used to implement the position detection system.
- Block 302 represents determining one or more coordinates in the space. For example, as noted below a space coordinate associated with a feature of an object, such as a fingertip, point of a stylus, etc. can be identified by analyzing the location of the feature as depicted in images captured by different imaging devices 112 and the known geometry of the imaging devices.
- the routine can determine if the coordinate lies in zone 1 and, if so, use the coordinate in a determining touch input command as shown at 306.
- the touch input command may be identified using a routine that provides an input event such as a selection in a graphical user interface based on a mapping of space coordinates to screen coordinates.
- a click or other selection may be registered when the object touches or approaches a plane corresponding to the plane of the display. Additional examples of touch detection are discussed later below in conjunction with Fig. 18. Any of the examples discussed herein can respond to 2D touch inputs (e.g., identified by one or more contacts between an object and a surface of interest) as well as 3D coordinate inputs.
- Block 308 represents determining if the coordinate lies in Zone 2. If so, flow proceeds to block 310.
- Zone 2 lies proximate the keyboard/trackpad and therefore coordinates in zone 2 are used in determining touch pad commands.
- a set of 2-dimensional input gestures analogous to those associated with touch displays may be associated with the keyboard or trackpad. The gestures may be made during contact with the key(s) or trackpad or may occur near the keys or trackpad. Examples include, but are not limited to, finger waves, swipes, drags, and the like. Coordinate values can be tracked over time and one or more heuristics can be used to determine an intended gesture.
- the heuristics may identify one or more positions or points which, depending upon the gesture, may need to be identified in sequence. By matching patterns of movement and/or positions, the gesture can be identified. As another example, finger motion may be tracked and used to manipulate an on-screen cursor.
- Block 312 represents determining if the coordinate value lies in Zone 3. In this example, if the coordinate does not lie in any of the zones an error condition is defined, though a zone could be assigned by default in some embodiments or the coordinate could be ignored. However, if the coordinate does lay in Zone 3, then as shown at block 314 the coordinate is used to determine a three-dimensional gesture. Similarly to identifying two-dimensional gestures, three-dimensional gestures can be identified by tracking coordinate values over time and applying one or more heuristics in order to identify an intended input.
- Fig. 4 is a diagram showing an exemplary configuration for providing zone-based detection capabilities.
- an imaging device features an array 402 of pixels that includes portions corresponding to each zone of detection; three zones are shown here.
- Selection logic 404 can be used to sample pixel values and to provide the pixel values to an onboard controller 406 that formats/routes the data accordingly (e.g., via a USB interface in some embodiments).
- array 402 is steerable to adjust at least one of a field of view or a focus to include an identified one of the plurality of zones.
- the entire array or subsections thereof may be rotated and/or translated through use of suitable mechanical elements (e.g. micro electromechanical systems (MEMS) devices, etc.) in response to signals from selection logic 404.
- MEMS micro electromechanical systems
- the entire optical unit may be repositioned using a motor, hydraulic system, etc. rather than steering the sensor array or portions thereof.
- Fig. 5 is a cross-sectional view of an illustrative architecture for an optical unit 112 that can be used in a position detection system.
- the optical unit includes a housing 502 made of plastic or another suitable material and a cover 504.
- Cover 504 may comprise glass, plastic, or the like and includes at least a transparent portion over and/or in aperture 506.
- Light passes through aperture 506 to lens 508, which focuses light onto array 510, in this example through a filter 512.
- Array 510 and housing 502 are mounted to frame 514 in this example.
- frame 514 may comprise a printed circuit board in some embodiments.
- array 510 can comprise one or more arrays of pixels configured to provide image data. For example, if IR light is provided by an irradiation system, the array can capture an image by sensing IR light from the imaged space. As another example, ambient light or another wavelength range could be used.
- filter 512 is used to filter out one or more wavelength ranges of light to improve detection of other range (s) of light used in capturing images.
- filter 512 comprises a narrowband IR-pass filter to attenuate ambient light other than the intended wavelength(s) of IR before reaching array 510, which is configured to sense at least IR wavelengths.
- a suitable filter 512 can be configured to exclude ranges not of interest.
- Some embodiments utilize an irradiation system that uses one or more irradiation devices such as light emitting diodes (LEDs) to radiate energy (e.g., infrared (IR) 'light') over one or more specified wavelength ranges.
- LEDs light emitting diodes
- IR infrared
- SNR signal to noise ratio
- LEDs can be driven by a suitable signal to irradiate the space imaged by the imaging device (s) that capture one or more image frames used in position detection.
- the irradiation is modulated, such as by driving the irradiation devices at a known frequency.
- Image frames can be captured based on the timing of the modulation.
- Some embodiments use software filtering to eliminate background light by subtracting images, such as by capturing a first image when irradiation is provided and then capturing a second image without irradiation. The second image can be subtracted from the first and then the resulting "representative image" can be used for further processing.
- CMOS substrate 602 also includes signal conditioning logic (or a programmable CPU) 606 that can be used to facilitate detection by performing at least some image processing in hardware before the image is provided by the imaging device, such as by a hardware-implemented ambient subtraction, infinite impulse response (IIR) or finite impulse response (FIR) filtering, background- tracker-based touch detection, or the like.
- substrate 602 also includes logic to provide a USB output that is used to deliver the image to a computing device 610.
- a driver 612 embodied in memory of computing device 610 configures computing device 610 to process images to determine one or more commands based on the image data.
- components 604 and 606 may be physically separate, and 606 may be implemented in an FPGA, DSP, ASIC, or microprocessor.
- CMOS is discussed in this example, a sensing device could be implemented using any other suitable technology for constructing integrated circuits.
- Fig. 7 is a circuit diagram 700 illustrating one example of a readout circuit for use in subtracting one image from another in hardware. Such a circuit could be comprised in a position detection system.
- a pixel 702 can be sampled using on two different storage devices 704 and 706 (capacitors FD1 and FD2 in this example) by driving select transistors TXl and TX2, respectively.
- Buffer transistors 708 and 710 can then provide readout values when row select line 712 is driven, with the readout values provided to a differential amplifier 714.
- the output 716 of amplifier 714 represents the difference between the pixel as sampled when TXl is driven and the pixel as sampled when TX2 is driven.
- each pixel in a row of pixels could be configured with a corresponding readout circuit, with the pixels included in a row or area sensor.
- other suitable circuits could be configured whereby two (or more) pixel values can be retained using a suitable charge storage device or buffer arrangement for use in outputting a representative image or for applying another signal processing effect.
- Fig. 8 is a timing diagram 800 showing an example of sampling (by a position detection system) the pixels during a first and second time interval and taking a difference of the pixels to output a representative image.
- three successive frames (Frame n-1; Frame n; and Frame n+1) are sampled and output as representative images.
- Each row 1 through 480 is read over a time interval during which the irradiation is provided ("light on") (e.g., by driving TXl) and then read again not while light is not provided (“light off”) (e.g. by driving TX2). Then, a single output image can be provided.
- This method parallels software-based representative image sampling.
- Fig. 9 is a timing diagram 900 showing another sampling routine that can be used by a position detection system. This example features a higher modulation rate and rapid shuttering, with each row sampled during a given on-off cycle.
- the total exposure time for the frame can equal or approximately equal the number of rows multiplied by the time for a complete modulation cycle.
- Fig. 10 is a flowchart showing steps in an exemplary method 1000 for detecting one or more space coordinates.
- a position detection system such as one of the systems of Figs. 1A-1D may feature a plurality of imaging devices that are used to image a space and carry out a method in accordance with Fig. 10.
- Another example is shown at 1100 in Fig. 11.
- first and second imaging devices 112 are positioned proximate a display 108 and keyboard and are configured to image a space 114.
- space 114 corresponds to a rectangular space between display 108 and the keyboard.
- Fig. 11 also shows a coordinate system V (V x , V y , V z ) defined with respect to area 114, with the space coordinate(s) determined in terms of V.
- Each imaging device 112 also features its own coordinate system C defined relative to an origin of each respective camera (shown as 0 L and 0 R in Fig. 11), with 0 L defined as (-1, 0, 0) in coordinate system V and 0 R defined as (1, 0, 0) in coordinate system V.
- camera coordinates are specified in terms of (C L X , C L y , C L Z ) while right-side camera coordinates are specified in terms of (C R X , C R y , C R Z ).
- the x- and y- coordinate in each camera correspond to X and Y coordinates for each unit, while the z- coordinate (C L Z and C R Z ) is the normal or direction of the plane of the imaging unit in this example.
- acquiring the first and second image comprises acquiring a first difference image based on images from a first imaging device and acquiring a second difference image based on images from the second imaging device.
- Each difference image can be determined by subtracting a background image from a representative image.
- each of a first and a second imaging device can image the space while lit and while not lit.
- the first and second representative images can be determined by
- the imaging devices can be configured with hardware in accordance with Figs. 7-9 or in another suitable manner to provide a representative image based on modulation of the light source.
- the representative images can be used directly.
- the difference images can be obtained by subtracting a respective background image from each of the representative images so that the object whose feature(s) are to be identified (e.g., the finger, stylus, etc.) remains but background features are absent.
- Im t represents the output of the imaging device at imaging interval t.
- a series of representative images can be determined by alternatively capturing lit and unlit images to result in I I 2 , I 3 , I 4 , etc.
- the algorithm could be:
- the differential image can be obtained by:
- Block 1006 represents locating a feature in each of the first and second images.
- Any suitable technique can be used to identify the feature, including an exemplary method noted later below.
- Block 1008 represents determining camera coordinates for the feature and then converting the coordinates to virtual coordinates.
- Image pixel coordinates can be converted to camera coordinates C (in mm) using the following expression:
- ⁇ P x , P y ) is the principle center are the focal lengths of each camera from calibration.
- Coordinates from left imaging unit coordinates C L and right imaging unit coordinates C R can be converted to corresponding coordinates in coordinate system V according to the following expressions:
- V L M Left x C L
- V R M rightt x C R
- le ft and r i g ht are the transformation matrices from left and right camera coordinates to the virtual coordinates
- le ft and r i g ht can be calculated by the rotation matrix, R, and translation vector T from stereo camera calibration.
- a chessboard pattern can be imaged by both imaging device and used to calculate a homogenous transformation between cameras in order to derive a rotation matrix R and translation vector T.
- P R is a point in the right camera coordinate system and point P L is a point in the left camera coordinate system
- the origins of the cameras can be set along the x-axis of the virtual space, with the left camera origin at (-1, 0, 0) and the right camera origin at (0, 0, 1).
- the x-axis of the virtual coordinate, V x is defined along the origins of the cameras.
- the z-axis of the virtual coordinate, V z is defined as the cross product of the z-axes from the camera's local coordinates (i.e. by the cross product of C z z and Cf ).
- the y-axis of the virtual coordinate, V y is defined as the cross product of the x and z axes.
- V x R * [0,0,0] T + T
- V y V z x V x
- V ⁇ V x x V y
- V z is calculated twice in case C and Cf are not co-planar. Because the origin of the left camera is defined at [-1, 0, 0] T the homogenous transformation of points from the left camera coordinate to the virtual coordinate can be obtained using the following expression; similar computations can derive the homogonous transformation from the right camera coordinate to the virtual coordinate:
- M left [V x T V i-l,0,0,l] T ]
- M right [R x V x T R x V y T R x V z T [1,0,0,1]
- Block 1010 represents determining an intersection of a first line and a second line.
- the first line is projected from the first camera origin and through the virtual coordinates of the feature as detected at the first imaging device
- the second line is projected from the second camera origin and through the virtual coordinates of the feature as detected at the second imaging device.
- the feature as detected has a left-side coordinate P L in coordinate system V and a right-side coordinate P R in coordinate system V.
- a line can be projected from left-side origin 0 L through P L and from right-side origin 0 R through P R .
- the lines will intersect at or near a location corresponding to the feature as shown in Fig. 12.
- intersection point P is defined as the center of the smallest sphere to which both lines are tangential.
- the sphere n is tangential to the projected lines at points a and b and thus the center of sphere n is defined as the space coordinate.
- the center of the sphere can be calculated by:
- n is a unit vector from nodes b to a and is derived from the cross product of two rays (P L - 0 L )x(P R - 0 R ) .
- the three remaining unknowns, t L ,t R , and ⁇ can be derived from solving the following linear equation: (P L - 0 L
- Block 1012 represents an optional step of filtering the location P.
- the filter can be applied to eliminate vibration or minute movements in the position of P. This can minimize unintentional shake or movement of a pointer or the object being detected.
- Suitable filters include an infinite impulse response filter, a GHK filter, etc., or even a custom filter for use with the position detection system.
- a space coordinate P can be found based on identifying a feature as depicted in at least two images. Any suitable image processing technique can be used to identify the feature.
- An example of an image processing technique is shown in Fig. 14, which is a flowchart and accompanying diagram showing an illustrative method 1400 of identifying a fingertip in an image.
- Diagram 1401 depicts an example of a difference image under analysis according to method 1400.
- Block 1402 represents accessing the image data.
- the image may be retrieved directly from an imaging device or memory or may be subjected to background subtraction or other refinement to aid in the feature recognition process.
- Block 1404 represents summing the intensity of all pixels along each row and then maintaining a representation of the sum as a function of the row number.
- An example representation is shown as plot 1404A. Although shown here as a visual plot, an actual plot does not need to be provided in practice and the position detection system can instead rely on an array of values or another in-memory representation.
- the cameras are assumed to be oriented as shown in Fig. 11. Thus, the camera locations are fixed and a user of the system is presumed to enter space 114 using his or her hand (or another object) from the front side.
- this feature recognition method identifies an image coordinate [I x , I y ] as corresponding to the pointing fingertip when the coordinate lies at the bottom of the image.
- Block 1406 represents determining the bottom row of the largest segment of rows.
- the bottom row is shown at 1406 in the plot and only a single segment exists.
- the summed pixel intensities may be discontinuous due to variations in lighting, etc., and so multiple discontinuous segments could occur in plot 1404A; in such cases the bottommost segment is considered.
- the vertical coordinate I y can be approximated as the row at the bottommost segment.
- Block 1408 represents summing pixel intensity values starting from I y for columns of the image.
- a representation of the summed intensity values as a function of the column number is shown at 1408A, though as mentioned above in practice an actual plot need not be provided.
- the pixel intensity values are summed only for a maximum of h pixels from I y , with h equal to 10 pixels in one embodiment.
- Block 1410 represents approximating the horizontal coordinate I x of the fingertip can be approximated as the coordinate for the column having the largest value of the summed column intensities; this is shown at 1410A in the diagram.
- the approximated coordinates [I x , I y ] can be used to determine a space coordinate P according to the methods noted above (or any other suitable method). However, some embodiments proceed to block 1412, which represents one or more additional processing steps such as edge detection. For example, in one
- a Sobel edge detection is performed around [I x , I y ] (e.g., in a 40x40 pixel window) and a resulting edge image is stored in memory, with strength values for the edge image used across the entire image to determine edges of the hand.
- a location of the first fingertip can be defined as the pixel on the detected edge that is closest to the bottom edge of the image, and that location can be used in
- image coordinates of the remaining fingertips can be detected using suitable curvature algorithms, with corresponding space coordinates determined based on image coordinates of the remaining fingertips.
- the feature was recognized based on an assumption of a likely shape and orientation of the object in the imaged space. It will be understood that the technique can vary for different arrangements of detectors and other components of the position detection system. For instance, if the imaging devices are positioned differently, then the most likely location for the fingertip may be the topmost row or the leftmost column, etc.
- Fig. 15A illustrates use of an interactive volume in a position detection system.
- the processor(s) of a position detection system are configured to access data from the at least one imaging device, the data comprising image data of an object in the space, access data defining at least one interactive volume within the space, determine a space coordinate associated with the object, and determine a command based on the space coordinate and the interactive volume.
- the interactive volume is a three-dimensional geometrical object defined in the field of view of the imaging device(s) of the position detection system.
- FIG. 15A shows a position detection system 1500 featuring a display 108 and imaging devices 112.
- the space imaged by devices 112 features an interactive volume 1502, shown here as a trapezoidal prism.
- interactive volume 1502 defines a rear surface at or near the plane of display 108 and a front surface 1503 extending outward in the z+ direction. Corners of the rear surface of the interactive volume are mapped to corresponding corners of the display in this example, and a depth is defined between the rear and front surfaces.
- this mapping uses data regarding the orientation of the display—such information can be achieved in any suitable manner.
- an imaging device with a field of view of the display can be used to monitor the display surface and reflections thereon. Touch events can be identified based on inferring a touch surface from viewing an object and reflection of the object, with three touch events used to define the plane of the display. Of course, other techniques could be used to determine the location/orientation of the display.
- the computing device can determine a command by determining a value of an interface coordinate using a space coordinate and a mapping of coordinate values within the interactive volume to interface coordinates in order to determine at least first and second values for the interface coordinate.
- embodiments also include converting the position according to a more generalized approach.
- the generalized approach effectively allows for the conversion of space coordinates to interface coordinates to differ according to the value of the space coordinate, with the result that movement of an object over a distance within a first section of the interactive volume displaces a cursor by an amount less than (or more than) movement of the object over an identical distance within the second section.
- Figs. 15B-E illustrate one example of the resulting cursor displacement.
- Fig. 15B is a top view of the system shown in Fig. 15A showing the front and sides of interactive volume 1502 in cross-section.
- An object such as a finger or stylus is moved from point A to point B along distance 1, with the depth of both points A and B being near the front face 1503 of interactive volume 1502.
- Fig. 15C shows corresponding movement of a cursor from point a' to point b' over distance2.
- Fig. 15D again shows the cross sectional view, but although the object is moved from point C to point D along the same distancel along the x-axis, the movement occurs at a depth much closer to the rear face of interactive volume 1502.
- the resulting cursor movement is shown in Fig. 15E where the cursor moves distance3 from point c' to d'.
- mappings varied along the depth of the interactive volume but similar effects could be achieved in different directions through use of other mappings.
- a computing system can support a state in which the 3D coordinate detection system is used for 2D input.
- this is achieved by using an interactive volume with a short depth (e.g., 3cm) and a one-to- one mapping to screen coordinates.
- movement within the virtual volume can be used for 2D input, such as touch- and hover- based input commands.
- the click can be identified when the rear surface of the interactive volume is reached.
- cursor movement the effect can be used in any situation in which coordinates or other commands are determined based on movement of an object in the imaged space. For example, if three-dimensional gestures are identified, then the gestures may be at a higher spatial resolution at one part of the interactive volume as compared to another. As a specific example, if the interactive volume shown in Fig. 15A is used, a "flick" gesture may have higher magnitude at a location farther from the screen than if the same gesture were made closer to the screen.
- the interactive volume can be used in other ways.
- the rear surface of the interactive volume can be defined as the plane of the display or even outward from the plane of the display so that when the rear surface of the interactive volume is reached (or passed) a click or other selection command is provided at the corresponding interface coordinate. More generally, an encounter with any boundary of the interactive volume could be interpreted as a command.
- the interface coordinate is determined as a pointer position P according to the following trilinear interpolation:
- ⁇ ⁇ 0 .(1- ⁇ ⁇ ).(1- ⁇ ⁇ ).(1- ⁇ ⁇ ) + ⁇ ⁇ . ⁇ ⁇ .(1- ⁇ ⁇ ).(1- ⁇ ⁇ ) + ⁇ 2 .(1- ⁇ ⁇ ). ⁇ ⁇ .(1- ⁇ ⁇ ) + ⁇ 3 ⁇ ⁇ ⁇ , ⁇ (1- ⁇ ⁇ ) + ⁇ 4 ⁇ (1- ⁇ ⁇ ) ⁇ (1- ⁇ ,) ⁇ ⁇ + ⁇ 5 ⁇ ⁇ ⁇ (1- ⁇ ⁇ ) ⁇ ⁇
- mappings could be used to achieve the effects noted herein and the particular interpolation noted above is for purposes of example only. Still further, other types of mappings could be used. As an example, a plurality of rectangular sections of an imaged area can be defined along the depth of the imaged area. Each rectangular section can have a different x-y mapping of interface coordinates to space coordinates. [00101] Additionally, the interactive volume need not be a trapezoid— a rhombic prism could be used or an irregular shape could be provided. For example, an interactive volume could be defined so that x-y mapping varies according to depth (i.e. z-position) and/or x-z mapping varies according to height (i.e.
- y-position and/or y-z mapping varies according to width (i.e., x-position).
- width i.e., x-position.
- the shapes and behavior of the interactive volume here have been described with respect to a rectangular coordinate system but interactive volumes could be defined in terms of spherical or other coordinates, subject to the imaging capabilities and spatial arrangement of the position detection system.
- mapping of space coordinates to image coordinates can be calculated in real time by carrying out the corresponding calculations.
- an interactive volume can be implemented as a set of mapped coordinates calculated as a function of space coordinates, with the set stored in memory and then accessed during operation of the system once a space coordinate is determined.
- the size, shape, and/or position of the interactive volume can be adjusted by a user. This can allow the user to define multiple interactive volumes (e.g., for splitting the detectable space into sub-areas for multiple monitors) and to control how space coordinates are mapped to screen coordinate.
- Fig. 16 is an example of a graphical user interface 1600 that can be provided by a position detection system.
- interface 1600 provides a top view 1602 and a front view 1604 showing the relationship of the interactive volume to the imaging devices (represented as icons 1606) and the keyboard (represented as a graphic 1608).
- a side view could be provided as well.
- a user can adjust the size and position of the front and rear faces of the interactive volume. Additional embodiments may allow the user to define more complex interactive volumes, split the area into multiple interactive volumes, etc.
- This interface is provided for purposes of example only; in practice any suitable interface elements such as sliders, buttons, dialog boxes, etc. could be used to set parameters of the interactive volume. If the mapping calculations are carried out in real time or near real time, the adjustments in the interface can be used to make corresponding adjustments to the mapping parameters. If a predefined set is used, the interface can be used to select another pre-defined mapping and/or the set of coordinates can be calculated and stored in memory for use in converting space coordinates to interface coordinates.
- the interactive volume can also be used to enhance image processing and feature detection.
- Figs. 17A-B show use of one array of pixels 1702A from a first imaging device and a second array of pixels 1702B from a second imaging device.
- the processing device of the position detection system is configured to iteratively sample image data of the at least one imaging device and determine a space coordinate associated with an object in the space based on detecting an image of a feature of the object in the image data as noted above. Iteratively sampling the image data can comprise determining a range of pixels for use in sampling image data during the next iteration based on a pixel location of a feature during a current iteration.
- iteratively sampling can comprise using data regarding a pixel location of a feature as detected by one imaging device during one iteration to determine a range of pixels for use in locating the feature using another imaging device during that same iteration (or another iteration).
- a window 1700 of pixels is used, with the location of window 1700 updated based on the location of detected feature A. For example, during a first iteration (or series of iterations) feature A can be identified by sampling both arrays 1702A and 1702B, with feature A appearing in each; Fig. 17B shows feature A as it appears in array 1702B. However, once an initial location of feature A has been determined, window 1700 can be used to limit the area sampled in at least one of the arrays of pixels or, if the entire array is sampled, to limit the extent of the image searched during the next iteration.
- a fingertip or other feature For example, after a fingertip or other feature is identified, its image coordinates are kept in static memory so that detection in the next frame only passes a region of pixels (e.g., 40x40 pixels) around the stored coordinate for processing. Pixels outside the window may not be sampled at all or may be sampled at a lower resolution than the pixels inside the window. As another example, a particular row may be identified for use in searching for the feature.
- a region of pixels e.g. 40x40 pixels
- Pixels outside the window may not be sampled at all or may be sampled at a lower resolution than the pixels inside the window.
- a particular row may be identified for use in searching for the feature.
- the interactive volume is used in limiting the area searched or sampled.
- the interactive volume can be projected onto each camera's image plane as shown at 1704A and 1704B to define one or more regions within each array of pixels. Pixels outside the regions can be ignored during sampling and/or analysis to reduce the amount of data passing through the image processing steps or can be processed at a lower resolution than pixels inside the interactive volume.
- a relationship based on epipolar geometry for stereo vision can be used to limit the area searched or sampled.
- a detected fingertip in the first camera e.g., point A in array 1702A
- pixels in the second camera e.g., array 1702B
- This line will intersect with the interactive volume in a 3D line space.
- the 3D line space can be projected onto the image plane of the other camera (e.g., onto array 1702B) resulting in a 2D line segment (epipolar line) E that can be used in searching. For instance, pixels corresponding to the 2D line segment can be searched while the other pixels are ignored.
- a window along the epipolar line can be searched for the feature.
- the depiction of the epipolar line in this example is purely for purposes of illustration, in practice the direction and length of the line will vary according to the geometry of the system, location of the pointer, etc.
- the epipolar relationship is used to verify that the correct feature has been identified.
- the detected point in the first camera is validated if the detected point is found along the epipolar line in the second camera.
- some embodiments determine one or more space coordinates and use the space coordinate(s) in determining commands for a position detection system.
- the commands can include movement of a cursor position, hovers, clicks, and the like, the commands are not intended to be limited to only those cases. Rather, additional command types can be supported due to the ability to image objects, such as a user's hand, in space.
- multiple fingertips or even a hand model can be used to support 3D hand gestures.
- discriminative methods can be used to recover the hand gesture from a single frame through classification or regression techniques.
- generative methods can be used to fit a 3D hand model to the observed images. These techniques can be used in addition to or instead of the fingertip recognition technique noted above.
- fingertip recognition/cursor movement may be defined within a first observable zone while 3D and/or 2D hand gestures may be recognized for movement in one or more other observable zones.
- the position detection system uses a first set of pixels for use in sampling image data during a first state and a second set of pixels for use in sampling image data during a second state.
- the system can be configured to switch between the first and second states based on success or failure in detecting a feature in the image data. As an example, if a window, interactive volume, and/or epipolar geometry are used in defining a first set of pixels but the feature is not found in both images during an iteration, the system may switch to second state that uses all available pixels.
- states may be used to conserve energy and/or processing power. For example, in a "sleep" state one or more imaging devices are deactivated.
- One imaging device can be used to identify motion or other activity or another sensor can be used to toggle from the "sleep" state to another state.
- the position detection system may operate one or more imaging device using alternating rows or sets of rows during one state and switch to continuous rows in another state. This may provide enough detection capability to determine when the position detection system is to be used while conserving resources at other times.
- one state may use only a single row of pixels to identify movement and switch to another state in which all rows are used. Of course, when "all" rows are used one or more of the limiting techniques noted above could be applied.
- States may also be useful in conserving power by selectively disabling irradiation components.
- the default mode of operation is a low-power mode during which the position detection system is active but the irradiation components are deactivated.
- One or more imaging devices can act as proximity sensors using ambient light to determine whether to activate the IR irradiation system (or other irradiation used for position detection purposes). In other implementations, another type of proximity sensor could be used, of course.
- the irradiation system can be operated at full power until an event, such as lack of movement for a predetermined period of time.
- an area camera is used as a proximity sensor.
- anything entering one of the zones (zone 3, for example) detected with ambient light will cause the system to fully wake up.
- detection of objects entering the zone can be done at a much reduced frame rate, typically at lHz, to further save power.
- a computing device used with the position detection system may support a "sleep mode."
- sleep mode the irradiation system is inactive and only one row of pixels from one camera is examined. Movement can be found by measuring if any block of pixels significantly change in intensity at over a 1 or 2 second time interval or by more complex methods used to determine optical flow (e.g., phase correlation, differential methods such as Lucas-Kanade, Horn-Schunk, and/or discrete optimization methods). If motion is detected, then one or more other cameras of the position detection system can be activated to see if the object is actually in the interaction zone and not further out and, if an object is indeed in the interaction zone, the computing device can be woken from sleep mode.
- a position detection system can respond to 2D touch events.
- a 2D touch event can comprise one or more contacts between an object and a surface of interest.
- Fig. 18 shows an example 1800 of a computing system that provides for position detection in accordance with one or more of the examples above.
- the system includes a body 101, display 108, and at least one imaging device 112, though multiple imaging devices could be used.
- the imaged space includes a surface, which in this example corresponds to display 108 or a material atop the display.
- implementations may have another surface of interest (e.g., body 101, a peripheral device, or other input area) in view of imaging device(s) 112.
- determining a command comprises identifying whether a contact is made between the object and the surface. For example, a 3D space coordinate associated with a feature of object 1802 (in this example, a fingertip) can be determined using one or more imaging devices. If the space coordinate is at or near a surface of display 108, then a touch command may be inferred (either based on use of an interactive volume or some other technique).
- the surface is at least partially reflective and determining the space coordinate is based at least in part on image data
- object 1802 features a reflected image 1804.
- Object 1802 and reflected image 1804 can be imaged by imaging device 112.
- a space coordinate for the fingertip of object 1802 can be determined based on object 1802 and its reflection 1804, thereby allowing for use of a single camera to determine 3D coordinates.
- the position detection system searches for a feature (e.g., a fingertip) in one image and, if found, searches for a reflection of that feature.
- An image plane can be determined based on the image and its reflection.
- the position detection system may determine if a touch is in progress based on the proximity of the feature and its reflection— if the feature and its reflection coincide or are within a threshold distance of one another, this may be interpreted as a touch.
- a coordinate for point "A" between the fingertip and its reflection can determined based on the feature and its reflection.
- the location of the reflective surface is known from calibration (e.g., through three touches or any other suitable
- the position detection system can project a line 1806 from the camera origin, through the image plane coordinate corresponding to point "A" and determine where line 1806 intersects the plane of screen 108 to obtain 3D coordinates for point "A.”
- a line 1808 normal to screen 108 can be projected through A.
- a line 1810 can be projected from the camera origin through the fingertip as located in the image plane. The intersection of lines 1808 and 1810 represents the 3D coordinate of the fingertip (or the 3D coordinate of its reflection— the two can be distinguished based on their coordinate values to determine which one is in front of screen 108).
- a plurality of imaging devices are used, but a 3D coordinate for a feature (e.g., the fingertip of object 1802) is determined using each imaging device alone. Then, the images can be combined using stereo matching techniques and the system can attempt to match the fingertips from each image based on their respective epipolar lines and 3D coordinates. If the fingertips match, an actual 3D coordinate can be found using triangulation. If the fingertips do not match, then one view may be occluded, so the 3D coordinates from one camera can be used.
- a 3D coordinate for a feature e.g., the fingertip of object 1802
- the images can be combined using stereo matching techniques and the system can attempt to match the fingertips from each image based on their respective epipolar lines and 3D coordinates. If the fingertips match, an actual 3D coordinate can be found using triangulation. If the fingertips do not match, then one view may be occluded, so the 3D coordinates from one camera can be used.
- the fingertips as imaged using multiple imaging devices can be overlain (in memory) to determine finger coordinates. If one finger is occluded from being viewed by each imaging device, then a single-camera method can be used. The occluded finger and its reflection can be identified and then a line projected between the finger and its reflection— the center point of that line can be treated as the coordinate.
- a computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs.
- Suitable computing devices include multipurpose and specialized microprocessor-based computer systems accessing stored software, but also application-specific integrated circuits and other programmable logic, and combinations thereof. Any suitable programming, scripting, or other type of language or combinations of languages may be used to construct program components and code for implementing the teachings contained herein.
- Embodiments of the methods disclosed herein may be executed by one or more suitable computing devices.
- Such system(s) may comprise one or more computing devices adapted to perform one or more embodiments of the methods disclosed herein.
- such devices may access one or more computer- readable media that embody computer-readable instructions which, when executed by at least one computer, cause the at least one computer to implement one or more embodiments of the methods of the present subject matter.
- the software may comprise one or more components, processes, and/or applications.
- the computing device (s) may comprise circuitry that renders the device (s) operative to implement one or more of the methods of the present subject matter.
- Any suitable non-transitory computer-readable medium or media may be used to implement or practice the presently-disclosed subject matter, including, but not limited to, diskettes, drives, magnetic-based storage media, optical storage media, including disks (including CD-ROMS, DVD-ROMS, and variants thereof), flash, RAM, ROM, and other memory devices, and the like.
- IR infrared
- ambient light e.g., visible light
- IR light may be used in addition to or instead of IR light.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
- User Interface Of Digital Computer (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
Abstract
A computing device, such as a desktop, laptop, tablet computer, a mobile device, or a computing device integrated into another device (e.g., an entertainment device for gaming, a television, an appliance, kiosk, vehicle, tool, etc.) is configured to determine user input commands from the location and/or movement of one or more objects in a space. The object(s) can be imaged using one or more optical sensors and the resulting position data can be interpreted in any number of ways to determine a command, including 2-dimensional and 3-dimensional movmements with or without touch.
Description
Methods and Systems for Position Detection
Priority Claim
[0001] The present application claims priority to Australian Provisional
Application No. 2009905917, filed December 4, 2009 and entitled, "A Coordinate Input Device," which is incorporated by reference herein in its entirety; the present application also claims priority to Australian Provisional Application No.
2010900748, filed February 23, 2010 and entitled, "A Coordinate Input Device," which is incorporated by reference herein in its entirety; the present application also claims priority to Australian Provisional Application No. 2010902689, filed June 21, 2010 and entitled, "3D Computer Input System," which is incorporated by reference herein in its entirety.
Background
[0002] Touch-enabled computing devices have become increasingly popular. Such devices can use optical, resistive, and/or capacitive sensors to determine when a finger, stylus, or other object has approached or touched a touch surface, such as a display. The use of touch has allowed for a variety of interface options, such as so- called "gestures" based on tracking touches over time.
[0003] Despite the advantages of touch-enabled systems, drawbacks remain. Laptop and desktop computers benefit from touch-enabled screens, but the particular configuration or arrangement of the screen may require a user to reach or otherwise move in an uncomfortable manner. Additionally, some touch detection technologies remain expensive, particularly for larger screen areas.
Summary
[0004] Embodiments of the present subject matter include a computing device, such as a desktop, laptop, tablet computer, a mobile device, or a computing device integrated into another device (e.g., an entertainment device for gaming, a television, an appliance, kiosk, vehicle, tool, etc.). The computing device is configured to determine user input commands from the location and/or movement of one or more objects in a space. The object(s) can be imaged using one or more optical sensors and the resulting position data can be interpreted in any number of ways to determine a command.
[0005] The commands include, but are not limited to, graphical user interface events within two-dimensional, three-dimensional, and other graphical user interfaces. As an example, an object such as a finger or stylus can be used to select on-screen items by touching a surface at a location mapped to the on-screen item or hovering over the surface near the location. As a further example, the commands may relate to non-graphical events (e.g., changing speaker volume,
activating/deactivating a device or feature, etc.). Some embodiments may rely on other input in addition to the position data, such as a click of a physical button provided while a finger or object is at a given location.
[0006] However, the same system may be able to interpret other input that does not feature a touch. For instance, the finger or stylus may be moved in a pattern that is then recognized as a particular input command, such as a gesture that is recognized based on or more heuristics that correlate the pattern of movement to particular commands. As another example, movement of the finger or stylus in free
space may translate to movement in the graphical user interface. For instance, crossing a plane or reaching a specified area may be interpreted as a touch or selection action, even if nothing is physically touched.
[0007] The object's location in space may influence how the object's position is interpreted as a command. For instance, a movement of an object within one part of the space may result in a different command than an identical movement of the object within another part of the space.
[0008] As an example, a finger or stylus may be moved along one or two axes within the space (e.g., along a width and/or height of the space), with the movement in the one or two axes resulting in corresponding movement of the cursor in a graphical user interface. The same movement at different locations along a third axis (e.g., at a different depth) may result in different corresponding movement of the cursor. For instance, a left-to-right movement of a finger may result in faster movement of the cursor the farther the finger is from a screen of the device. This can be achieved in some embodiments by using a virtual volume (referred to as an "interactive volume" herein) defined by a mapping of space coordinates to screen/interface coordinates, with the mapping varying along the depth of the interactive volume.
[0009] As another example, different zones may be used for different types of input. In some embodiments, a first zone can be defined near a screen of the device and a second zone can be defined elsewhere. For instance, the second zone may lie between the screen and keys of a keyboard of a laptop computer, or may represent imageable space outside the first zone in the case of a tablet or mobile device. Input
in the first zone may be interpreted as touch, hover, and other graphical user interface commands. Input in the second zone may be interpreted as gestures. For instance, a "flick" gesture may be provided in the second zone in order to move through a list of items, without need to select particular items/command buttons via the graphical user interface.
[0010] As discussed below, aspects of various embodiments also include irradiation, detection, and device configurations that allow for image-based input to be provided in a responsive and accurate manner. For instance, detector
configuration and detector sampling can be used to provide higher image
processing throughput and more responsive detection. In some embodiments, fewer than all available pixels from the detector are sampled, such as by limiting the pixels to a projection of an interactive volume and/or determining an area of interest for detection by one detector of a feature detected by a second detector.
[0011] These illustrative embodiments are mentioned not to limit or define the limits of the present subject matter, but to provide examples to aid understanding thereof. Illustrative embodiments are discussed in the Detailed Description, and further description is provided there, including illustrative embodiments of systems, methods, and computer-readable media providing one or more aspects of the present subject matter. Advantages offered by various embodiments may be further understood by examining this specification and/or by practicing one or more embodiments of the claimed subject matter.
Brief Description of the Drawings
[0012] A full and enabling disclosure is set forth more particularly in the remainder of the specification. The specification makes reference to the following appended figures.
[0013] Figs. 1A-1D illustrate exemplary embodiments of a position detection system.
[0014] Fig. 2 is a diagram showing division of an imaged space into a plurality of zones.
[0015] Fig. 3 is a flowchart showing an example of handling input based on zone identification.
[0016] Fig. 4 is a diagram showing an exemplary sensor configuration for providing zone-based detection capabilities.
[0017] Fig. 5 is a cross-sectional view of an illustrative architecture for an optical unit.
[0018] Fig. 6 is a diagram illustrating use of a CMOS-based sensing device in a position detection system.
[0019] Fig. 7 is a circuit diagram illustrating one illustrative readout circuit for use in subtracting one image from another in hardware.
[0020] Figs. 8 and 9 are exemplary timing diagrams illustrating use of a sensor having hardware for subtracting a first and second image.
[0021] Fig. 10 is a flowchart showing steps in an exemplary method for detecting one or more space coordinates.
[0022] Fig. 11 is a diagram showing an illustrative hardware configuration and corresponding coordinate systems used in determining one or more space coordinates.
[0023] Figs. 12 and 13 are diagrams showing use of a plurality of imaging devices to determine a space coordinate.
[0024] Fig. 14 is a flowchart and accompanying diagram showing an illustrative method of identifying a feature in an image.
[0025] Fig. 15A is a diagram of an illustrative system using an interactive volume.
[0026] Figs. 15B-15E show examples of different cursor responses based on a variance in mapping along the depth of the interactive volume.
[0027] Fig. 16 is a diagram showing an example of a user interface for configuring an interactive volume.
[0028] Figs. 17A-17B illustrate techniques in limiting the pixels used in detection and/or image processing.
[0029] Fig. 18 shows an example of determining a space coordinate using an image from a single camera.
Detailed Description
[0030] Reference will now be made in detail to various and alternative exemplary embodiments and to the accompanying drawings. Each example is provided by way of explanation, and not as a limitation. It will be apparent to those skilled in the art that modifications and variations can be made. For instance, features illustrated or described as part of one embodiment may be used on another
embodiment to yield a still further embodiment. Thus, it is intended that this disclosure includes modifications and variations as come within the scope of the appended claims and their equivalents.
[0031] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the claimed subject matter. However, it will be understood by those skilled in the art that claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.
Illustrative System and Hardware Aspects of Position Detection Systems
[0032] Fig. 1A is a view of an illustrative position detection system 100, while Fig. IB is a diagram showing an exemplary architecture for system 100. Generally, a position detection system can comprise one or more imaging devices and hardware logic that configures the position detection system access data from the at least one imaging device, the data comprising image data of an object in the space, access data defining an interactive volume within the space, determine a space coordinate associated with the object, and determine a command based on the space
coordinate and the interactive volume.
[0033] In this example, the position detection system is a computing system in which the hardware logic comprises a processor 102 interfaced to a memory 104 via bus 106. Program components 116 configure the processor to access data and determine the command. Although a software-based implementation is shown here, the position detection system could use other hardware (e.g., field
programmable gate arrays (FPGA), programmable logic arrays (PLA), etc.).
[0034] Returning to Fig. 1, memory 104 can comprise RAM, ROM, or other memory accessible by processor 102 and/or another non-transitory computer- readable medium, such as a storage medium. System 100 in this example is interfaced via I/O components 107 to a display 108, a plurality of irradiation devices 110, and a plurality of imaging devices 112. Imaging devices 112 are configured to image a field of view including space 114.
[0035] In this example, multiple irradiation and imaging devices are used, though it will be understood that a single imaging device could be used in some embodiments, and some embodiments could use a single irradiation device or could omit an irradiation device and rely on ambient light or other ambient energy.
Additionally, although several examples herein use two imaging devices, a system could utilize more than two imaging devices in imaging an object and/or could use multiple different imaging systems for different purposes.
[0036] Memory 104 embodies one or more program components 116 that configure the computing system to access data from the imaging device (s) 112, the data comprising image data of one or more objects in the space, determine a space coordinate associated with the one or more objects, and determine a command based on the space coordinate. Exemplary configuration of the program
component(s) will be discussed in the examples below.
[0037] The architecture of system 100 shown in Fig. IB is not meant to be limiting. For example, one or more I/O interfaces 107 comprising a graphics interface (e.g., VGA, HDMI) can be used to connect display 108 (if used). Other examples of I/O interfaces include universal serial bus (USB), IEEE 1394, and internal busses. One or more networking components for communicating via wired or wireless communication can be used, and can include interfaces such as Ethernet, IEEE 802.11 (Wi-Fi), 802.16 (Wi-Max), Bluetooth, infrared, etc., CDMA, GSM, UMTS, or other cellular communication networks.
[0038] Fig. 1A illustrates a laptop or netbook form factor. In this example, irradiation and imaging devices 110 and 112 are shown in body 101, which may
also include the processor, memory, etc. However, any such components could be included in display 108.
[0039] For example, Fig. 1C shows another illustrative form factor of a position detection system 100'. In this example, a display device 108' has integrated irradiation devices 110 and imaging devices 112 in a raised area at the bottom of the screen. The area may be approximately 2mm in size. In this example, the imaging devices image a space 114' including the front area of display device 108'. Display device 108' can be interfaced to a computing system (not shown) including a processor, memory, etc. As another example, the processor and additional components could be included in the body of display 108'. Although shown as a display device (e.g., an LCD, plasma, OLED monitor, television, etc.), the principles could be applied for other devices, such as tablet computers, mobile devices, and the like.
[0040] Fig. ID shows another illustrative position detection system 100". In particular, imaging devices 112 can be positioned either side of an elongated irradiation device 110, which may comprise one or more light emitting diodes or other devices that emit light. In this example, space 114" includes a space above irradiation device 110 and between imaging devices 112. In this example the image plane of each imaging device lies at an angle Θ between the bottom plane of space 114", and Θ can be equal or approximately equal to 45 degrees in some
embodiments. Although shown here as a rectangular space, the actual size and extent of the space can depend upon the position, orientation, and capabilities of the imaging devices.
[0041] Additionally, depending upon the particular form factor, irradiation device 110 may not be centered on space 114". For example, if irradiation device 110 and imaging devices 112 are used with a laptop computer, they may be positioned approximately near the top or bottom of the keyboard, with space 114" corresponding to an area between the screen and keyboard. Irradiation device 110 and imaging devices 112 could be included in or mounted to a keyboard positioned in front of a separate screen as well. As a further example, irradiation device 110 and imaging devices 112 could be included in or attached to a screen or tablet computer. Still further, irradiation device 110 and imaging devices 112 may be included in a separate body mounted to another device or used as a standalone peripheral with or without a screen.
[0042] As yet another example, imaging devices 112 could be provided separately from irradiation device 110. For instance, imaging devices 112 could be positioned on either side of a keyboard, display screen, or simply on either side of an area in which spatial input is to be provided. Irradiation device(s) 110 could be positioned at any suitable location to provide irradiation as needed.
[0043] Generally speaking, imaging devices 112 can comprise area sensors that capture one or more frames depicting the field of view of the imaging devices. The images in the frames may comprise any representation that can be obtained using imaging units, and for example may depict a visual representation of the field of view, a representation of the intensity of light in the field of view, or another representation. The processor or other hardware logic of the position detection system can use the frame(s) to determine information about one or more objects in
space 114, such as the location, orientation, direction of the object(s) and/or parts thereof. When an object is in the field of view, one or more features of the object can be identified and used to determine a coordinate within space 114 (i.e., a "space coordinate"). The computing system can determine one or more commands based on the value of the space coordinate. In some embodiments, the space coordinate is used in determining how to identify a particular command by using the space coordinate to determine a position, orientation, and/or movement of the object (or recognized feature of the object) over time.
Illustrative Embodiments Featuring Multiple Detection Zones
[0044] In some embodiments, different ranges of space coordinates are treated differently in determining a command. For instance, as shown in Fig. 2 the imaged space can be divided into a plurality of zones. This example shows an imaging device 112 and three zones, though more or fewer zones may be defined;
additionally, the zones may vary along the length, width, and/or depth of the imaged space. An input command can be identified based on determining which one of a plurality of zones within the space contains the determined space coordinate. For example, if a coordinate lies in the zone ("Zone 1") proximate the display device 108, then the movement/position of the object associated with that coordinate can provide different input than if the coordinate were in Zones 2 or 3.
[0045] In some embodiments, the same imaging system can be used to determine a position component regardless of the zone in which the coordinate lies. However, in some embodiments multiple imaging systems are used to determine inputs. For example, one or more imaging devices 112 further from the screen can
be used to image zones 2 and/or 3. In one example, each imaging system passes a screen coordinate to a routine that determines a command in accordance with Fig. 3.
[0046] For example, for commands in zone 1, one or more line or area sensors could be used to image the area at or around the screen, with a second system used for imaging one or both of zones 2 zone 3. If the second system images only one of zones 2 and 3, a third imaging system can image the other of zones 2 and 3. The imaging systems could each rely on one or more aspects described below to determine a space coordinate. Of course, multiple imaging systems could be used within one or more of the zones. For example, zone 3 may be handled as a plurality of sub-zones, with each sub-zone imaged by a respective set of imaging devices. Zone coverage may overlap, as well.
[0047] The same or different position detection techniques could be used in conjunction with the various imaging systems. For example, the imaging system for zone 1 could use triangulation principles to determine coordinates relative to the screen area, or each imaging system could use aspects of the position detection techniques noted herein. That same system could also determine distance from the screen. Additionally or alternatively, the systems could be used cooperatively. For example, the imaging system used to determine a coordinate in zone 1 could use triangulation for the screen coordinate and rely upon data from the imaging system used to image zone 3 in order to determine a distance from the screen.
[0048] Fig. 3 is a flowchart showing an example of handling the input based on zone identification and can be carried out by program components 116 shown in
Fig. 1 or by other hardware/software used to implement the position detection system. Block 302 represents determining one or more coordinates in the space. For example, as noted below a space coordinate associated with a feature of an object, such as a fingertip, point of a stylus, etc. can be identified by analyzing the location of the feature as depicted in images captured by different imaging devices 112 and the known geometry of the imaging devices.
[0049] As shown at block 304, the routine can determine if the coordinate lies in zone 1 and, if so, use the coordinate in a determining touch input command as shown at 306. For example, the touch input command may be identified using a routine that provides an input event such as a selection in a graphical user interface based on a mapping of space coordinates to screen coordinates. As a particular example, a click or other selection may be registered when the object touches or approaches a plane corresponding to the plane of the display. Additional examples of touch detection are discussed later below in conjunction with Fig. 18. Any of the examples discussed herein can respond to 2D touch inputs (e.g., identified by one or more contacts between an object and a surface of interest) as well as 3D coordinate inputs.
[0050] Returning to Fig. 3, Block 308 represents determining if the coordinate lies in Zone 2. If so, flow proceeds to block 310. In this example, Zone 2 lies proximate the keyboard/trackpad and therefore coordinates in zone 2 are used in determining touch pad commands. For example, a set of 2-dimensional input gestures analogous to those associated with touch displays may be associated with the keyboard or trackpad. The gestures may be made during contact with the key(s)
or trackpad or may occur near the keys or trackpad. Examples include, but are not limited to, finger waves, swipes, drags, and the like. Coordinate values can be tracked over time and one or more heuristics can be used to determine an intended gesture. The heuristics may identify one or more positions or points which, depending upon the gesture, may need to be identified in sequence. By matching patterns of movement and/or positions, the gesture can be identified. As another example, finger motion may be tracked and used to manipulate an on-screen cursor.
[0051] Block 312 represents determining if the coordinate value lies in Zone 3. In this example, if the coordinate does not lie in any of the zones an error condition is defined, though a zone could be assigned by default in some embodiments or the coordinate could be ignored. However, if the coordinate does lay in Zone 3, then as shown at block 314 the coordinate is used to determine a three-dimensional gesture. Similarly to identifying two-dimensional gestures, three-dimensional gestures can be identified by tracking coordinate values over time and applying one or more heuristics in order to identify an intended input.
[0052] As another example, pattern recognition techniques could be applied to recognize gestures, even without relying directly on coordinates. For instance, the system could be configured to identify edges of a hand or other object in the area and perform edge analysis to determine a posture, orientation, and/or shape of a hand or other object. Suitable gesture recognition heuristics could be applied to recognize various input gestures based on changes in the recognized posture, orientation, and/or shape over time.
[0053] Fig. 4 is a diagram showing an exemplary configuration for providing zone-based detection capabilities. In this example, an imaging device features an array 402 of pixels that includes portions corresponding to each zone of detection; three zones are shown here. Selection logic 404 can be used to sample pixel values and to provide the pixel values to an onboard controller 406 that formats/routes the data accordingly (e.g., via a USB interface in some embodiments). In some embodiments, array 402 is steerable to adjust at least one of a field of view or a focus to include an identified one of the plurality of zones. For example, the entire array or subsections thereof may be rotated and/or translated through use of suitable mechanical elements (e.g. micro electromechanical systems (MEMS) devices, etc.) in response to signals from selection logic 404. As another example, the entire optical unit may be repositioned using a motor, hydraulic system, etc. rather than steering the sensor array or portions thereof.
Illustrative Embodiments of Imaging Devices
[0054] Fig. 5 is a cross-sectional view of an illustrative architecture for an optical unit 112 that can be used in a position detection system. In this example the optical unit includes a housing 502 made of plastic or another suitable material and a cover 504. Cover 504 may comprise glass, plastic, or the like and includes at least a transparent portion over and/or in aperture 506. Light passes through aperture 506 to lens 508, which focuses light onto array 510, in this example through a filter 512. Array 510 and housing 502 are mounted to frame 514 in this example. For instance, frame 514 may comprise a printed circuit board in some embodiments. In any event, array 510 can comprise one or more arrays of pixels configured to
provide image data. For example, if IR light is provided by an irradiation system, the array can capture an image by sensing IR light from the imaged space. As another example, ambient light or another wavelength range could be used.
[0055] In some embodiments, filter 512 is used to filter out one or more wavelength ranges of light to improve detection of other range (s) of light used in capturing images. For example, in one embodiment filter 512 comprises a narrowband IR-pass filter to attenuate ambient light other than the intended wavelength(s) of IR before reaching array 510, which is configured to sense at least IR wavelengths. As another example, if other wavelengths are of interest a suitable filter 512 can be configured to exclude ranges not of interest.
[0056] Some embodiments utilize an irradiation system that uses one or more irradiation devices such as light emitting diodes (LEDs) to radiate energy (e.g., infrared (IR) 'light') over one or more specified wavelength ranges. This can aid in increasing the signal to noise ratio (SNR), where the signal is the irradiated portion of the image and the noise is largely comprised of ambient light. For example, IR
LEDs can be driven by a suitable signal to irradiate the space imaged by the imaging device (s) that capture one or more image frames used in position detection. In some embodiments, the irradiation is modulated, such as by driving the irradiation devices at a known frequency. Image frames can be captured based on the timing of the modulation.
[0057] Some embodiments use software filtering to eliminate background light by subtracting images, such as by capturing a first image when irradiation is provided and then capturing a second image without irradiation. The second image
can be subtracted from the first and then the resulting "representative image" can be used for further processing. Mathematically, the operation can be expressed as Signal = (Signal+Noise) - Noise. Some embodiments improve SNR with high- intensity illuminating light such that any noise is swamped/dwarfed.
Mathematically, such situations can be described as Signal=Signal+Noise, where Signal > > Noise.
[0058] As shown in Fig. 6 some embodiments include hardware signal conditioning. Fig. 6 is a diagram 600 illustrating use of a CMOS-based sensing device 602 in a position detection system. In this example, sensor 604 comprises an array of pixels. CMOS substrate 602 also includes signal conditioning logic (or a programmable CPU) 606 that can be used to facilitate detection by performing at least some image processing in hardware before the image is provided by the imaging device, such as by a hardware-implemented ambient subtraction, infinite impulse response (IIR) or finite impulse response (FIR) filtering, background- tracker-based touch detection, or the like. In this example, substrate 602 also includes logic to provide a USB output that is used to deliver the image to a computing device 610. A driver 612 embodied in memory of computing device 610 configures computing device 610 to process images to determine one or more commands based on the image data. Although shown together in Fig. 6, components 604 and 606 may be physically separate, and 606 may be implemented in an FPGA, DSP, ASIC, or microprocessor. Although CMOS is discussed in this example, a sensing device could be implemented using any other suitable technology for constructing integrated circuits.
[0059] Fig. 7 is a circuit diagram 700 illustrating one example of a readout circuit for use in subtracting one image from another in hardware. Such a circuit could be comprised in a position detection system. In particular, a pixel 702 can be sampled using on two different storage devices 704 and 706 (capacitors FD1 and FD2 in this example) by driving select transistors TXl and TX2, respectively. Buffer transistors 708 and 710 can then provide readout values when row select line 712 is driven, with the readout values provided to a differential amplifier 714. The output 716 of amplifier 714 represents the difference between the pixel as sampled when TXl is driven and the pixel as sampled when TX2 is driven.
[0060] A single pixel is shown here, though it will be understood that each pixel in a row of pixels could be configured with a corresponding readout circuit, with the pixels included in a row or area sensor. Additionally, other suitable circuits could be configured whereby two (or more) pixel values can be retained using a suitable charge storage device or buffer arrangement for use in outputting a representative image or for applying another signal processing effect.
[0061] Fig. 8 is a timing diagram 800 showing an example of sampling (by a position detection system) the pixels during a first and second time interval and taking a difference of the pixels to output a representative image. As can be seen here, three successive frames (Frame n-1; Frame n; and Frame n+1) are sampled and output as representative images. Each row 1 through 480 is read over a time interval during which the irradiation is provided ("light on") (e.g., by driving TXl) and then read again not while light is not provided ("light off") (e.g. by driving TX2).
Then, a single output image can be provided. This method parallels software-based representative image sampling.
[0062] Fig. 9 is a timing diagram 900 showing another sampling routine that can be used by a position detection system. This example features a higher modulation rate and rapid shuttering, with each row sampled during a given on-off cycle. The total exposure time for the frame can equal or approximately equal the number of rows multiplied by the time for a complete modulation cycle.
Illustrative Embodiments of Coordinate Detection
[0063] Fig. 10 is a flowchart showing steps in an exemplary method 1000 for detecting one or more space coordinates. For example, a position detection system such as one of the systems of Figs. 1A-1D may feature a plurality of imaging devices that are used to image a space and carry out a method in accordance with Fig. 10. Another example is shown at 1100 in Fig. 11. In this example, first and second imaging devices 112 are positioned proximate a display 108 and keyboard and are configured to image a space 114. In this example, space 114 corresponds to a rectangular space between display 108 and the keyboard.
[0064] Fig. 11 also shows a coordinate system V (Vx, Vy, Vz) defined with respect to area 114, with the space coordinate(s) determined in terms of V. Each imaging device 112 also features its own coordinate system C defined relative to an origin of each respective camera (shown as 0L and 0R in Fig. 11), with 0L defined as (-1, 0, 0) in coordinate system V and 0R defined as (1, 0, 0) in coordinate system V. For the left-side camera, camera coordinates are specified in terms of (CL X, CL y, CL Z) while right-side camera coordinates are specified in terms of (CR X, CR y, CR Z). The x- and y-
coordinate in each camera correspond to X and Y coordinates for each unit, while the z- coordinate (CL Z and CR Z) is the normal or direction of the plane of the imaging unit in this example.
[0065] Back in Fig. 10, beginning at block 1002, the method moves to block 1004, which represents acquiring first and second images. In some embodiments, acquiring the first and second image comprises acquiring a first difference image based on images from a first imaging device and acquiring a second difference image based on images from the second imaging device.
[0066] Each difference image can be determined by subtracting a background image from a representative image. In particular, while a light source is modulated, each of a first and a second imaging device can image the space while lit and while not lit. The first and second representative images can be determined by
subtracting the unlit image from each device from the lit image from each device (or vice-versa, with the absolute value of the image taken). As another example, the imaging devices can be configured with hardware in accordance with Figs. 7-9 or in another suitable manner to provide a representative image based on modulation of the light source.
[0067] In some embodiments, the representative images can be used directly. However, in some embodiments the difference images can be obtained by subtracting a respective background image from each of the representative images so that the object whose feature(s) are to be identified (e.g., the finger, stylus, etc.) remains but background features are absent.
[0068] For example, in one embodiment a representative image is defined as
It = [Imf - Imf_1|
where Imt represents the output of the imaging device at imaging interval t.
[0069] A series of representative images can be determined by alternatively capturing lit and unlit images to result in I I2, I3, I4, etc. Background subtraction can be carried out by first initializing a background image B0 = Iv Then, the background image can be updated according to the following algorithm:
Then Bt[n] = Bt_l[n] + l;
Else Bt[n] = It[n]
[0070] As another example, the algorithm could be:
If It[n] >Bt_l[n],
Then Bt[n] = Bt_l[n] + l;
Else Bt [n] = Bt [n] - l
[0071] The differential image can be obtained by:
D, = It - B,
[0072] Of course, various embodiments can use any suitable technique to obtain suitable images. In any event, after the first and second images are acquired, the method moves to block 1006, which represents locating a feature in each of the first and second images. In practice, multiple different features could be identified, though embodiments can proceed starting from one common feature. Any suitable technique can be used to identify the feature, including an exemplary method noted later below.
[0073] Regardless of the technique used to identify the feature, the feature will be located in terms of two-dimensional image pixel coordinates I (IL X, IL y) and (IR X, IV) in each of the acquired images. Block 1008 represents determining camera coordinates for the feature and then converting the coordinates to virtual coordinates. Image pixel coordinates can be converted to camera coordinates C (in mm) using the following expression:
[0074] Coordinates from left imaging unit coordinates CL and right imaging unit coordinates CR can be converted to corresponding coordinates in coordinate system V according to the following expressions:
VL = MLeft x CL
VR = Mrightt x CR where left and right are the transformation matrices from left and right camera coordinates to the virtual coordinates; left and right can be calculated by the rotation matrix, R, and translation vector T from stereo camera calibration. A chessboard pattern can be imaged by both imaging device and used to calculate a homogenous transformation between cameras in order to derive a rotation matrix R and translation vector T. In particular, assuming PR is a point in the right camera
coordinate system and point PL is a point in the left camera coordinate system, the transformation from right to left can be defined as PL = R » PR + T. [0075] As before, the origins of the cameras can be set along the x-axis of the virtual space, with the left camera origin at (-1, 0, 0) and the right camera origin at (0, 0, 1). In this example, the x-axis of the virtual coordinate, Vx , is defined along the origins of the cameras. The z-axis of the virtual coordinate, Vz , is defined as the cross product of the z-axes from the camera's local coordinates (i.e. by the cross product of Cz z and Cf ). The y-axis of the virtual coordinate, Vy , is defined as the cross product of the x and z axes. [0076] With these definitions and the calibration data, each axis of the virtual coordinate system can be derived according to the following steps:
Vx = R * [0,0,0]T + T
Kz = ((R . [0,0,l]r = Γ) - Κχ) χ [0,0,1]Γ
Vy = Vz x Vx
V ∑ = V x x V y
Vz , is calculated twice in case C and Cf are not co-planar. Because the origin of the left camera is defined at [-1, 0, 0]T the homogenous transformation of points from the left camera coordinate to the virtual coordinate can be obtained using the following expression; similar computations can derive the homogonous transformation from the right camera coordinate to the virtual coordinate:
Mleft = [Vx TV i-l,0,0,l]T ] And
Mright = [R x Vx T R x Vy T R x Vz T [1,0,0,1]
[0077] Block 1010 represents determining an intersection of a first line and a second line. The first line is projected from the first camera origin and through the virtual coordinates of the feature as detected at the first imaging device, while the second line is projected from the second camera origin and through the virtual coordinates of the feature as detected at the second imaging device. [0078] As shown in Figs. 12-13, the feature as detected has a left-side coordinate PL in coordinate system V and a right-side coordinate PR in coordinate system V. A line can be projected from left-side origin 0L through PL and from right-side origin 0R through PR . Ideally, the lines will intersect at or near a location corresponding to the feature as shown in Fig. 12.
[0079] In practice, a perfect intersection may not be found— for example the projected lines may not be co-planar due to errors in calibration. Thus, in some embodiments the intersection point P is defined as the center of the smallest sphere to which both lines are tangential. As shown in Fig. 13, the sphere n is tangential to the projected lines at points a and b and thus the center of sphere n is defined as the space coordinate. The center of the sphere can be calculated by:
0L + {PL - 0L) » tL = Ρ+ λ· η
0R + (PR - 0R) » tR = Ρ - λ · η where n is a unit vector from nodes b to a and is derived from the cross product of two rays (PL - 0L)x(PR - 0R) . The three remaining unknowns, tL ,tR , and λ , can be derived from solving the following linear equation:
(PL - 0L
[tL tR λ] · -(PR -0R) = 0R -0L
-In
[0080] Block 1012 represents an optional step of filtering the location P. The filter can be applied to eliminate vibration or minute movements in the position of P. This can minimize unintentional shake or movement of a pointer or the object being detected. Suitable filters include an infinite impulse response filter, a GHK filter, etc., or even a custom filter for use with the position detection system.
[0081] As noted above, a space coordinate P can be found based on identifying a feature as depicted in at least two images. Any suitable image processing technique can be used to identify the feature. An example of an image processing technique is shown in Fig. 14, which is a flowchart and accompanying diagram showing an illustrative method 1400 of identifying a fingertip in an image. Diagram 1401 depicts an example of a difference image under analysis according to method 1400.
[0082] Block 1402 represents accessing the image data. For example, the image may be retrieved directly from an imaging device or memory or may be subjected to background subtraction or other refinement to aid in the feature recognition process. Block 1404 represents summing the intensity of all pixels along each row and then maintaining a representation of the sum as a function of the row number. An example representation is shown as plot 1404A. Although shown here as a visual plot, an actual plot does not need to be provided in practice and the position detection system can instead rely on an array of values or another in-memory representation.
[0083] In this example, the cameras are assumed to be oriented as shown in Fig. 11. Thus, the camera locations are fixed and a user of the system is presumed to enter space 114 using his or her hand (or another object) from the front side.
Therefore, the pixels at the pointing fingertip should be closer to the screen than any other pixels. Accordingly, this feature recognition method identifies an image coordinate [Ix , Iy] as corresponding to the pointing fingertip when the coordinate lies at the bottom of the image.
[0084] Block 1406 represents determining the bottom row of the largest segment of rows. In this example, the bottom row is shown at 1406 in the plot and only a single segment exists. In some situations, the summed pixel intensities may be discontinuous due to variations in lighting, etc., and so multiple discontinuous segments could occur in plot 1404A; in such cases the bottommost segment is considered. The vertical coordinate Iy can be approximated as the row at the bottommost segment.
[0085] Block 1408 represents summing pixel intensity values starting from Iy for columns of the image. A representation of the summed intensity values as a function of the column number is shown at 1408A, though as mentioned above in practice an actual plot need not be provided. In some embodiments, the pixel intensity values are summed only for a maximum of h pixels from Iy, with h equal to 10 pixels in one embodiment. Block 1410 represents approximating the horizontal coordinate Ix of the fingertip can be approximated as the coordinate for the column having the largest value of the summed column intensities; this is shown at 1410A in the diagram.
[0086] The approximated coordinates [Ix , Iy] can be used to determine a space coordinate P according to the methods noted above (or any other suitable method). However, some embodiments proceed to block 1412, which represents one or more additional processing steps such as edge detection. For example, in one
embodiment a Sobel edge detection is performed around [Ix , Iy] (e.g., in a 40x40 pixel window) and a resulting edge image is stored in memory, with strength values for the edge image used across the entire image to determine edges of the hand. A location of the first fingertip can be defined as the pixel on the detected edge that is closest to the bottom edge of the image, and that location can be used in
determining a space coordinate. Still further, image coordinates of the remaining fingertips can be detected using suitable curvature algorithms, with corresponding space coordinates determined based on image coordinates of the remaining fingertips.
[0087] In this example the feature was recognized based on an assumption of a likely shape and orientation of the object in the imaged space. It will be understood that the technique can vary for different arrangements of detectors and other components of the position detection system. For instance, if the imaging devices are positioned differently, then the most likely location for the fingertip may be the topmost row or the leftmost column, etc.
Illustrative Aspects of Position Detection Systems Utilizing Interactive volumes
[0088] Fig. 15A illustrates use of an interactive volume in a position detection system. In some embodiments, the processor(s) of a position detection system are
configured to access data from the at least one imaging device, the data comprising image data of an object in the space, access data defining at least one interactive volume within the space, determine a space coordinate associated with the object, and determine a command based on the space coordinate and the interactive volume. The interactive volume is a three-dimensional geometrical object defined in the field of view of the imaging device(s) of the position detection system.
[0089] Fig. 15A shows a position detection system 1500 featuring a display 108 and imaging devices 112. The space imaged by devices 112 features an interactive volume 1502, shown here as a trapezoidal prism. It will be understood that in various embodiments one or more interactive volumes can be used and the interactive volume(s) may be of any desired shape. In this example, interactive volume 1502 defines a rear surface at or near the plane of display 108 and a front surface 1503 extending outward in the z+ direction. Corners of the rear surface of the interactive volume are mapped to corresponding corners of the display in this example, and a depth is defined between the rear and front surfaces.
[0090] For best results, this mapping uses data regarding the orientation of the display— such information can be achieved in any suitable manner. As one example, an imaging device with a field of view of the display can be used to monitor the display surface and reflections thereon. Touch events can be identified based on inferring a touch surface from viewing an object and reflection of the object, with three touch events used to define the plane of the display. Of course, other techniques could be used to determine the location/orientation of the display.
[0091] In some embodiments, the computing device can determine a command by determining a value of an interface coordinate using a space coordinate and a mapping of coordinate values within the interactive volume to interface coordinates in order to determine at least first and second values for the interface coordinate.
[0092] Although a pointer could simply be mapped from a 3D coordinate to a 2D coordinate (or to a 2D coordinate plus a depth coordinate, in the case of a three- dimensional interface), embodiments also include converting the position according to a more generalized approach. In particular, the generalized approach effectively allows for the conversion of space coordinates to interface coordinates to differ according to the value of the space coordinate, with the result that movement of an object over a distance within a first section of the interactive volume displaces a cursor by an amount less than (or more than) movement of the object over an identical distance within the second section.
[0093] Figs. 15B-E illustrate one example of the resulting cursor displacement. Fig. 15B is a top view of the system shown in Fig. 15A showing the front and sides of interactive volume 1502 in cross-section. An object such as a finger or stylus is moved from point A to point B along distance 1, with the depth of both points A and B being near the front face 1503 of interactive volume 1502. Fig. 15C shows corresponding movement of a cursor from point a' to point b' over distance2.
[0094] Fig. 15D again shows the cross sectional view, but although the object is moved from point C to point D along the same distancel along the x-axis, the movement occurs at a depth much closer to the rear face of interactive volume
1502. The resulting cursor movement is shown in Fig. 15E where the cursor moves distance3 from point c' to d'.
[0095] In this example, because the front face of the interactive volume is smaller than the rear face of the interactive volume, a slower cursor movement results for a given movement in the imaged space as the movement occurs closer to the screen. A movement in a first cross-sectional plane of the interactive volume can result in a set of coordinate values that differ than the same movement if made in a second cross-sectional plane. In this example, the mappings varied along the depth of the interactive volume but similar effects could be achieved in different directions through use of other mappings.
[0096] For example, a computing system can support a state in which the 3D coordinate detection system is used for 2D input. In some implementations this is achieved by using an interactive volume with a short depth (e.g., 3cm) and a one-to- one mapping to screen coordinates. Thus, movement within the virtual volume can be used for 2D input, such as touch- and hover- based input commands. For instance, the click can be identified when the rear surface of the interactive volume is reached.
[0097] Although this example depicted cursor movement, the effect can be used in any situation in which coordinates or other commands are determined based on movement of an object in the imaged space. For example, if three-dimensional gestures are identified, then the gestures may be at a higher spatial resolution at one part of the interactive volume as compared to another. As a specific example, if the interactive volume shown in Fig. 15A is used, a "flick" gesture may have higher
magnitude at a location farther from the screen than if the same gesture were made closer to the screen.
[0098] In addition to varying mapping of coordinates along the depth (and/or another axis of the interactive volume), the interactive volume can be used in other ways. For example, the rear surface of the interactive volume can be defined as the plane of the display or even outward from the plane of the display so that when the rear surface of the interactive volume is reached (or passed) a click or other selection command is provided at the corresponding interface coordinate. More generally, an encounter with any boundary of the interactive volume could be interpreted as a command.
[0099] In one embodiment, the interface coordinate is determined as a pointer position P according to the following trilinear interpolation:
Ρ = Ρ0.(1-ξχ).(1-ξγ).(1-ξζ) + Ρι.ξχ.(1-ξγ).(1-ξζ) + Ρ2.(1-ξχ).ξγ.(1-ξζ) + Ρ3·ξχ·ξ,·(1-ξζ) + Ρ4·(1-ξχ)·(1-ξ,)·ξζ+Ρ5·ξχ·(1-ξζ)·ξζ
+ ρ6.(\-ξχ).ξγ.ξζ+ρΊ.ξχ.ξγ.ξζ where the vertices of the interactive volume are Ρ[0_Ί] and ξ = [ζχ,ζγ,ζζ] is the determined space coordinate in the range of [0, 1].
[00100] Of course, other mappings could be used to achieve the effects noted herein and the particular interpolation noted above is for purposes of example only. Still further, other types of mappings could be used. As an example, a plurality of rectangular sections of an imaged area can be defined along the depth of the imaged area. Each rectangular section can have a different x-y mapping of interface coordinates to space coordinates.
[00101] Additionally, the interactive volume need not be a trapezoid— a rhombic prism could be used or an irregular shape could be provided. For example, an interactive volume could be defined so that x-y mapping varies according to depth (i.e. z-position) and/or x-z mapping varies according to height (i.e. y-position) and/or y-z mapping varies according to width (i.e., x-position). The shapes and behavior of the interactive volume here have been described with respect to a rectangular coordinate system but interactive volumes could be defined in terms of spherical or other coordinates, subject to the imaging capabilities and spatial arrangement of the position detection system.
[00102] In practice, the mapping of space coordinates to image coordinates can be calculated in real time by carrying out the corresponding calculations. As another example, an interactive volume can be implemented as a set of mapped coordinates calculated as a function of space coordinates, with the set stored in memory and then accessed during operation of the system once a space coordinate is determined.
[00103] In some embodiments the size, shape, and/or position of the interactive volume can be adjusted by a user. This can allow the user to define multiple interactive volumes (e.g., for splitting the detectable space into sub-areas for multiple monitors) and to control how space coordinates are mapped to screen coordinate. Fig. 16 is an example of a graphical user interface 1600 that can be provided by a position detection system. In this example, interface 1600 provides a top view 1602 and a front view 1604 showing the relationship of the interactive
volume to the imaging devices (represented as icons 1606) and the keyboard (represented as a graphic 1608). A side view could be provided as well.
[00104] By dragging or otherwise manipulating elements 1620, 1622, 1624, and 1626, a user can adjust the size and position of the front and rear faces of the interactive volume. Additional embodiments may allow the user to define more complex interactive volumes, split the area into multiple interactive volumes, etc. This interface is provided for purposes of example only; in practice any suitable interface elements such as sliders, buttons, dialog boxes, etc. could be used to set parameters of the interactive volume. If the mapping calculations are carried out in real time or near real time, the adjustments in the interface can be used to make corresponding adjustments to the mapping parameters. If a predefined set is used, the interface can be used to select another pre-defined mapping and/or the set of coordinates can be calculated and stored in memory for use in converting space coordinates to interface coordinates.
[00105] The interactive volume can also be used to enhance image processing and feature detection. Figs. 17A-B show use of one array of pixels 1702A from a first imaging device and a second array of pixels 1702B from a second imaging device. In some embodiments, the processing device of the position detection system is configured to iteratively sample image data of the at least one imaging device and determine a space coordinate associated with an object in the space based on detecting an image of a feature of the object in the image data as noted above. Iteratively sampling the image data can comprise determining a range of pixels for use in sampling image data during the next iteration based on a pixel
location of a feature during a current iteration. Additionally or alternatively, iteratively sampling can comprise using data regarding a pixel location of a feature as detected by one imaging device during one iteration to determine a range of pixels for use in locating the feature using another imaging device during that same iteration (or another iteration).
[00106] As shown in Fig. 17A, a window 1700 of pixels is used, with the location of window 1700 updated based on the location of detected feature A. For example, during a first iteration (or series of iterations) feature A can be identified by sampling both arrays 1702A and 1702B, with feature A appearing in each; Fig. 17B shows feature A as it appears in array 1702B. However, once an initial location of feature A has been determined, window 1700 can be used to limit the area sampled in at least one of the arrays of pixels or, if the entire array is sampled, to limit the extent of the image searched during the next iteration.
[00107] For example, after a fingertip or other feature is identified, its image coordinates are kept in static memory so that detection in the next frame only passes a region of pixels (e.g., 40x40 pixels) around the stored coordinate for processing. Pixels outside the window may not be sampled at all or may be sampled at a lower resolution than the pixels inside the window. As another example, a particular row may be identified for use in searching for the feature.
[00108] Additionally or alternatively, in some embodiments the interactive volume is used in limiting the area searched or sampled. Specifically, the interactive volume can be projected onto each camera's image plane as shown at 1704A and 1704B to define one or more regions within each array of pixels. Pixels outside the
regions can be ignored during sampling and/or analysis to reduce the amount of data passing through the image processing steps or can be processed at a lower resolution than pixels inside the interactive volume.
[00109] As another example, a relationship based on epipolar geometry for stereo vision can be used to limit the area searched or sampled. A detected fingertip in the first camera, e.g., point A in array 1702A, has a geometrical relationship to pixels in the second camera (e.g., array 1702B) found by running a line from the origin of the first camera through the detected fingertip in 3-D space. This line will intersect with the interactive volume in a 3D line space. The 3D line space can be projected onto the image plane of the other camera (e.g., onto array 1702B) resulting in a 2D line segment (epipolar line) E that can be used in searching. For instance, pixels corresponding to the 2D line segment can be searched while the other pixels are ignored. As another example, a window along the epipolar line can be searched for the feature. The depiction of the epipolar line in this example is purely for purposes of illustration, in practice the direction and length of the line will vary according to the geometry of the system, location of the pointer, etc.
[00110] In some embodiments, the epipolar relationship is used to verify that the correct feature has been identified. In particular, the detected point in the first camera is validated if the detected point is found along the epipolar line in the second camera.
Embodiments with Enhanced Recognition Capability
[00111] As noted above some embodiments determine one or more space coordinates and use the space coordinate(s) in determining commands for a
position detection system. Although the commands can include movement of a cursor position, hovers, clicks, and the like, the commands are not intended to be limited to only those cases. Rather, additional command types can be supported due to the ability to image objects, such as a user's hand, in space.
[00112] For example, in one embodiment multiple fingertips or even a hand model can be used to support 3D hand gestures. For example, discriminative methods can be used to recover the hand gesture from a single frame through classification or regression techniques. Additionally or alternatively generative methods can be used to fit a 3D hand model to the observed images. These techniques can be used in addition to or instead of the fingertip recognition technique noted above. As another example, fingertip recognition/cursor movement may be defined within a first observable zone while 3D and/or 2D hand gestures may be recognized for movement in one or more other observable zones.
Use of Multiple States in Position Detection Systems
[00113] In some embodiments the position detection system uses a first set of pixels for use in sampling image data during a first state and a second set of pixels for use in sampling image data during a second state. The system can be configured to switch between the first and second states based on success or failure in detecting a feature in the image data. As an example, if a window, interactive volume, and/or epipolar geometry are used in defining a first set of pixels but the feature is not found in both images during an iteration, the system may switch to second state that uses all available pixels.
[00114] Additionally or alternatively, states may be used to conserve energy and/or processing power. For example, in a "sleep" state one or more imaging devices are deactivated. One imaging device can be used to identify motion or other activity or another sensor can be used to toggle from the "sleep" state to another state. As another example, the position detection system may operate one or more imaging device using alternating rows or sets of rows during one state and switch to continuous rows in another state. This may provide enough detection capability to determine when the position detection system is to be used while conserving resources at other times. As another example, one state may use only a single row of pixels to identify movement and switch to another state in which all rows are used. Of course, when "all" rows are used one or more of the limiting techniques noted above could be applied.
[00115] States may also be useful in conserving power by selectively disabling irradiation components. For example, when running on batteries in portable devices it is a disadvantage to provide IR light on a continuous basis. Therefore, in some implementations, the default mode of operation is a low-power mode during which the position detection system is active but the irradiation components are deactivated. One or more imaging devices can act as proximity sensors using ambient light to determine whether to activate the IR irradiation system (or other irradiation used for position detection purposes). In other implementations, another type of proximity sensor could be used, of course. The irradiation system can be operated at full power until an event, such as lack of movement for a predetermined period of time.
[00116] In one implementation, an area camera is used as a proximity sensor. Returning to the example of Fig. 2, during a low-power mode, anything entering one of the zones (zone 3, for example) detected with ambient light will cause the system to fully wake up. During the low-power mode, detection of objects entering the zone can be done at a much reduced frame rate, typically at lHz, to further save power.
[00117] Additional power reduction measures can be used as well. For example, a computing device used with the position detection system may support a "sleep mode." During sleep mode, the irradiation system is inactive and only one row of pixels from one camera is examined. Movement can be found by measuring if any block of pixels significantly change in intensity at over a 1 or 2 second time interval or by more complex methods used to determine optical flow (e.g., phase correlation, differential methods such as Lucas-Kanade, Horn-Schunk, and/or discrete optimization methods). If motion is detected, then one or more other cameras of the position detection system can be activated to see if the object is actually in the interaction zone and not further out and, if an object is indeed in the interaction zone, the computing device can be woken from sleep mode.
Touch Detection
[00118] As noted above, a position detection system can respond to 2D touch events. A 2D touch event can comprise one or more contacts between an object and a surface of interest. Fig. 18 shows an example 1800 of a computing system that provides for position detection in accordance with one or more of the examples above. Here, the system includes a body 101, display 108, and at least one imaging device 112, though multiple imaging devices could be used. The imaged space
includes a surface, which in this example corresponds to display 108 or a material atop the display. However, implementations may have another surface of interest (e.g., body 101, a peripheral device, or other input area) in view of imaging device(s) 112.
[00119] In some implementations, determining a command comprises identifying whether a contact is made between the object and the surface. For example, a 3D space coordinate associated with a feature of object 1802 (in this example, a fingertip) can be determined using one or more imaging devices. If the space coordinate is at or near a surface of display 108, then a touch command may be inferred (either based on use of an interactive volume or some other technique).
Single Camera Coordinate Determination
[00120] In some implementations, the surface is at least partially reflective and determining the space coordinate is based at least in part on image data
representing a reflection of the object. For example, as shown in Fig. 18, object 1802 features a reflected image 1804. Object 1802 and reflected image 1804 can be imaged by imaging device 112. A space coordinate for the fingertip of object 1802 can be determined based on object 1802 and its reflection 1804, thereby allowing for use of a single camera to determine 3D coordinates.
[00121] For example, in one implementation, the position detection system searches for a feature (e.g., a fingertip) in one image and, if found, searches for a reflection of that feature. An image plane can be determined based on the image and its reflection. The position detection system may determine if a touch is in progress based on the proximity of the feature and its reflection— if the feature and
its reflection coincide or are within a threshold distance of one another, this may be interpreted as a touch.
[00122] Regardless of whether a touch occurs, a coordinate for point "A" between the fingertip and its reflection can determined based on the feature and its reflection. The location of the reflective surface (screen 108 in this example) is known from calibration (e.g., through three touches or any other suitable
technique), and it is known that "A" must lie on the reflective surface.
[00123] The position detection system can project a line 1806 from the camera origin, through the image plane coordinate corresponding to point "A" and determine where line 1806 intersects the plane of screen 108 to obtain 3D coordinates for point "A." Once the 3D coordinate for "A" is known, a line 1808 normal to screen 108 can be projected through A. A line 1810 can be projected from the camera origin through the fingertip as located in the image plane. The intersection of lines 1808 and 1810 represents the 3D coordinate of the fingertip (or the 3D coordinate of its reflection— the two can be distinguished based on their coordinate values to determine which one is in front of screen 108).
[00124] Additional examples of using a single camera for 3D position detection can be found in U.S. Patent Application No. 12/704,949, filed February 12, 2010 naming Bo Li and John Newton as inventors, which is incorporated by reference herein in its entirety.
[00125] In some implementations, a plurality of imaging devices are used, but a 3D coordinate for a feature (e.g., the fingertip of object 1802) is determined using each imaging device alone. Then, the images can be combined using stereo
matching techniques and the system can attempt to match the fingertips from each image based on their respective epipolar lines and 3D coordinates. If the fingertips match, an actual 3D coordinate can be found using triangulation. If the fingertips do not match, then one view may be occluded, so the 3D coordinates from one camera can be used.
[00126] For example, when detecting multiple contacts (e.g., two fingertips spaced apart), the fingertips as imaged using multiple imaging devices can be overlain (in memory) to determine finger coordinates. If one finger is occluded from being viewed by each imaging device, then a single-camera method can be used. The occluded finger and its reflection can be identified and then a line projected between the finger and its reflection— the center point of that line can be treated as the coordinate.
General Considerations
[00127] Examples discussed herein are not meant to imply that the present subject matter is limited to any specific hardware architecture or configuration discussed herein. As was noted above, a computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multipurpose and specialized microprocessor-based computer systems accessing stored software, but also application-specific integrated circuits and other programmable logic, and combinations thereof. Any suitable programming, scripting, or other type of language or combinations of languages may be used to construct program components and code for implementing the teachings contained herein.
[00128] Embodiments of the methods disclosed herein may be executed by one or more suitable computing devices. Such system(s) may comprise one or more computing devices adapted to perform one or more embodiments of the methods disclosed herein. As noted above, such devices may access one or more computer- readable media that embody computer-readable instructions which, when executed by at least one computer, cause the at least one computer to implement one or more embodiments of the methods of the present subject matter. When software is utilized, the software may comprise one or more components, processes, and/or applications. Additionally or alternatively to software, the computing device (s) may comprise circuitry that renders the device (s) operative to implement one or more of the methods of the present subject matter.
[00129] Any suitable non-transitory computer-readable medium or media may be used to implement or practice the presently-disclosed subject matter, including, but not limited to, diskettes, drives, magnetic-based storage media, optical storage media, including disks (including CD-ROMS, DVD-ROMS, and variants thereof), flash, RAM, ROM, and other memory devices, and the like.
[00130] Examples of infrared (IR) irradiation were provided. It will be understood that any suitable wavelength range (s) of energy can be used for position detection, and the use of IR irradiation and detection is for purposes of example only. For example, ambient light (e.g., visible light) may be used in addition to or instead of IR light.
[00131] While the present subject matter has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in
the art, upon attaining an understanding of the foregoing may readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, it should be understood that the present disclosure has been presented for purposes of example rather than limitation, and does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.
Claims
1. A computing system, comprising:
a processor;
a memory; and
at least one imaging device configured to image a space,
wherein the memory comprises at least one program component that configures the processor to:
access data from the at least one imaging device, the data comprising image data of an object in the space,
identify at least one feature of the object imaged by the at least one imaging device,
determine a space coordinate associated with the feature based on an image coordinate of the feature, and
determine a command based on the space coordinate.
2. The computing system set forth in claim 1, wherein determining a command comprises determining an interface coordinate based on the space coordinate.
3. The computing system set forth in claim 2, wherein the interface coordinate is determined using an interactive volume.
4. The computing system set forth in claim 1, 2, or 3, wherein determining a command comprises identifying a gesture by tracking at least the space coordinate over time.
5. The computing system set forth in claim 4, wherein the gesture is a two- dimensional gesture.
6. The computing system set forth in claim 4, wherein the gesture is a three- dimensional gesture.
7. The computing system set forth in claim 1, 2, or 3, wherein determining a command comprises performing an edge analysis to determine at least one of a shape or posture of the object.
8. The computing system set forth in claim 7, wherein determining a command comprises tracking the shape or posture over time to identify a three- dimensional gesture.
9. The computing system set forth in claim 1, 2, or 3, wherein determining a command is based at least in part on determining which one of a plurality of zones within the space contains the determined space coordinate.
10. The computing system set forth in claim 9, wherein the at least one imaging device comprises:
a first imaging device configured to image at least a first zone but not a second zone, and
a second imaging device configured to image at least the second zone.
11. The computing system set forth in claim 9, wherein the plurality of zones include a first zone proximate a display device and a second zone,
wherein the input command is identified as a selection in a graphical user interface if the space coordinate is contained in the first zone, and wherein the input command is identified as a gesture if the space coordinate is contained in the second zone.
12. The computing system set forth in claim 9, wherein at least one imaging device is steerable to adjust at least one of a field of view or a focus to include an identified one of the plurality of zones.
13. The computing system set forth in claim 1, 2, or 3 wherein the imaged space comprises a surface and determining a command comprises identifying whether a contact is made between the object and the surface.
14. The computing system set forth in claim 13, wherein the surface corresponds to a display interfaced to the processor or a layer of material atop the display.
15. The computing system set forth in claim 13, wherein the surface is at least partially reflective and determining the space coordinate is based at least in part on image data representing a reflection of the object.
16. The computing system set forth in claim 1, 2, or 3, wherein the computing system comprises a tablet computer and the at least one imaging device is positioned at an edge of the tablet computer.
17. The computing system set forth in claim 1, 2, or 3, wherein the computing system comprises a laptop computer and at least one imaging device comprises a plurality of imaging devices positioned along a keyboard of the laptop computer.
18. The computing system set forth in claim 1, 2, or 3, wherein the at least one imaging device is included in at least one peripheral device interfaced to the processor.
19. The computing device set forth in claim 1, 2, or 3 wherein the memory comprises at least one program component that configures the processor to deactivate an irradiation system during a low-power mode and to exit the low- power mode in response to detecting motion in the space.
20. The computing device set forth in claim 19, wherein the at least one imaging device comprises a plurality of imaging devices and, during the low-power mode, one of the plurality of imaging devices is used.
21. The computing device set forth in claim 1, 2, or 3 wherein the memory comprises at least one program component that configures the processor to wake the computing device from a sleep mode in response to detecting motion in the space.
22. The computing device set forth in claim 21, wherein, during the sleep mode, the irradiation system is deactivated.
23. The computing device set forth in claim 21, wherein, during the sleep mode, a single row of pixels in one imaging device is used.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201080063123XA CN102741782A (en) | 2009-12-04 | 2010-12-06 | Methods and systems for position detection |
EP10798871A EP2507684A2 (en) | 2009-12-04 | 2010-12-06 | Methods and systems for position detection |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2009905917A AU2009905917A0 (en) | 2009-12-04 | A coordinate input device | |
AU2009905917 | 2009-12-04 | ||
AU2010900748 | 2010-02-23 | ||
AU2010900748A AU2010900748A0 (en) | 2010-02-23 | A coordinate input device | |
AU2010902689A AU2010902689A0 (en) | 2010-06-21 | 3D computer input system | |
AU2010902689 | 2010-06-21 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2011069157A2 true WO2011069157A2 (en) | 2011-06-09 |
WO2011069157A3 WO2011069157A3 (en) | 2011-07-28 |
Family
ID=43706427
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2010/059050 WO2011069148A1 (en) | 2009-12-04 | 2010-12-06 | Methods and systems for position detection using an interactive volume |
PCT/US2010/059104 WO2011069157A2 (en) | 2009-12-04 | 2010-12-06 | Methods and systems for position detection |
PCT/US2010/059082 WO2011069152A2 (en) | 2009-12-04 | 2010-12-06 | Imaging methods and systems for position detection |
PCT/US2010/059078 WO2011069151A2 (en) | 2009-12-04 | 2010-12-06 | Sensor methods and systems for position detection |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2010/059050 WO2011069148A1 (en) | 2009-12-04 | 2010-12-06 | Methods and systems for position detection using an interactive volume |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2010/059082 WO2011069152A2 (en) | 2009-12-04 | 2010-12-06 | Imaging methods and systems for position detection |
PCT/US2010/059078 WO2011069151A2 (en) | 2009-12-04 | 2010-12-06 | Sensor methods and systems for position detection |
Country Status (4)
Country | Link |
---|---|
US (4) | US20110205155A1 (en) |
EP (4) | EP2507683A1 (en) |
CN (4) | CN102754048A (en) |
WO (4) | WO2011069148A1 (en) |
Families Citing this family (182)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9471170B2 (en) | 2002-11-04 | 2016-10-18 | Neonode Inc. | Light-based touch screen with shift-aligned emitter and receiver lenses |
US8902196B2 (en) * | 2002-12-10 | 2014-12-02 | Neonode Inc. | Methods for determining a touch location on a touch screen |
US9195344B2 (en) * | 2002-12-10 | 2015-11-24 | Neonode Inc. | Optical surface using a reflected image for determining three-dimensional position information |
US7629967B2 (en) | 2003-02-14 | 2009-12-08 | Next Holdings Limited | Touch screen signal processing |
US8508508B2 (en) | 2003-02-14 | 2013-08-13 | Next Holdings Limited | Touch screen signal processing with single-point calibration |
US8456447B2 (en) | 2003-02-14 | 2013-06-04 | Next Holdings Limited | Touch screen signal processing |
US7538759B2 (en) | 2004-05-07 | 2009-05-26 | Next Holdings Limited | Touch panel display system with illumination and detection provided from a single edge |
KR20100075460A (en) | 2007-08-30 | 2010-07-02 | 넥스트 홀딩스 인코포레이티드 | Low profile touch panel systems |
US8432377B2 (en) | 2007-08-30 | 2013-04-30 | Next Holdings Limited | Optical touchscreen with improved illumination |
US8405636B2 (en) | 2008-01-07 | 2013-03-26 | Next Holdings Limited | Optical position sensing system and optical position sensor assembly |
US9035876B2 (en) | 2008-01-14 | 2015-05-19 | Apple Inc. | Three-dimensional user interface session control |
SE533704C2 (en) | 2008-12-05 | 2010-12-07 | Flatfrog Lab Ab | Touch sensitive apparatus and method for operating the same |
US9063614B2 (en) | 2009-02-15 | 2015-06-23 | Neonode Inc. | Optical touch screens |
WO2011047505A1 (en) * | 2009-10-20 | 2011-04-28 | Hewlett-Packard Development Company, L.P. | Data processing system and method |
WO2011085023A2 (en) | 2010-01-06 | 2011-07-14 | Celluon, Inc. | System and method for a virtual multi-touch mouse and stylus apparatus |
US8587422B2 (en) | 2010-03-31 | 2013-11-19 | Tk Holdings, Inc. | Occupant sensing system |
US9007190B2 (en) | 2010-03-31 | 2015-04-14 | Tk Holdings Inc. | Steering wheel sensors |
JP5759230B2 (en) | 2010-04-02 | 2015-08-05 | ティーケー ホールディングス,インコーポレーテッド | Steering wheel with hand sensor |
US20150153715A1 (en) * | 2010-09-29 | 2015-06-04 | Google Inc. | Rapidly programmable locations in space |
US9477302B2 (en) * | 2012-08-10 | 2016-10-25 | Google Inc. | System and method for programing devices within world space volumes |
US8730190B2 (en) * | 2011-01-13 | 2014-05-20 | Qualcomm Incorporated | Detect motion generated from gestures used to execute functionality associated with a computer system |
US9335793B2 (en) * | 2011-01-31 | 2016-05-10 | Apple Inc. | Cover attachment with flexible display |
US10025388B2 (en) * | 2011-02-10 | 2018-07-17 | Continental Automotive Systems, Inc. | Touchless human machine interface |
US8497838B2 (en) * | 2011-02-16 | 2013-07-30 | Microsoft Corporation | Push actuation of interface controls |
GB201103346D0 (en) | 2011-02-28 | 2011-04-13 | Dev Ltd | Improvements in or relating to optical navigation devices |
US8619049B2 (en) | 2011-05-17 | 2013-12-31 | Microsoft Corporation | Monitoring interactions between two or more objects within an environment |
GB2491870B (en) * | 2011-06-15 | 2013-11-27 | Renesas Mobile Corp | Method and apparatus for providing communication link monito ring |
GB201110159D0 (en) * | 2011-06-16 | 2011-07-27 | Light Blue Optics Ltd | Touch sensitive display devices |
US9377865B2 (en) | 2011-07-05 | 2016-06-28 | Apple Inc. | Zoom-based gesture user interface |
US9459758B2 (en) | 2011-07-05 | 2016-10-04 | Apple Inc. | Gesture-based interface with enhanced features |
US8881051B2 (en) | 2011-07-05 | 2014-11-04 | Primesense Ltd | Zoom-based gesture user interface |
WO2013021385A2 (en) | 2011-08-11 | 2013-02-14 | Eyesight Mobile Technologies Ltd. | Gesture based interface system and method |
US9030498B2 (en) | 2011-08-15 | 2015-05-12 | Apple Inc. | Combining explicit select gestures and timeclick in a non-tactile three dimensional user interface |
US9218063B2 (en) | 2011-08-24 | 2015-12-22 | Apple Inc. | Sessionless pointing user interface |
DE102011112618A1 (en) * | 2011-09-08 | 2013-03-14 | Eads Deutschland Gmbh | Interaction with a three-dimensional virtual scenario |
CN103019391A (en) * | 2011-09-22 | 2013-04-03 | 纬创资通股份有限公司 | Input device and method using captured keyboard image as instruction input foundation |
TW201316240A (en) * | 2011-10-06 | 2013-04-16 | Rich Ip Technology Inc | Touch processing method and system using graphic user interface image |
GB2509651B (en) * | 2011-10-11 | 2015-07-08 | Ibm | Object designation method, device and computer program |
GB2496378B (en) * | 2011-11-03 | 2016-12-21 | Ibm | Smart window creation in a graphical user interface |
US20130135188A1 (en) * | 2011-11-30 | 2013-05-30 | Qualcomm Mems Technologies, Inc. | Gesture-responsive user interface for an electronic device |
US9395820B2 (en) | 2011-12-02 | 2016-07-19 | Intel Corporation | Techniques for notebook hinge sensors |
US9679215B2 (en) | 2012-01-17 | 2017-06-13 | Leap Motion, Inc. | Systems and methods for machine control |
US9501152B2 (en) | 2013-01-15 | 2016-11-22 | Leap Motion, Inc. | Free-space user interface and control using virtual constructs |
US9070019B2 (en) | 2012-01-17 | 2015-06-30 | Leap Motion, Inc. | Systems and methods for capturing motion in three-dimensional space |
US8638989B2 (en) | 2012-01-17 | 2014-01-28 | Leap Motion, Inc. | Systems and methods for capturing motion in three-dimensional space |
US8693731B2 (en) | 2012-01-17 | 2014-04-08 | Leap Motion, Inc. | Enhanced contrast for object detection and characterization by optical imaging |
US11493998B2 (en) | 2012-01-17 | 2022-11-08 | Ultrahaptics IP Two Limited | Systems and methods for machine control |
US10691219B2 (en) | 2012-01-17 | 2020-06-23 | Ultrahaptics IP Two Limited | Systems and methods for machine control |
US20150253428A1 (en) | 2013-03-15 | 2015-09-10 | Leap Motion, Inc. | Determining positional information for an object in space |
KR20130085094A (en) * | 2012-01-19 | 2013-07-29 | 삼성전기주식회사 | User interface device and user interface providing thereof |
US20130207962A1 (en) * | 2012-02-10 | 2013-08-15 | Float Hybrid Entertainment Inc. | User interactive kiosk with three-dimensional display |
US9229534B2 (en) * | 2012-02-28 | 2016-01-05 | Apple Inc. | Asymmetric mapping for tactile and non-tactile user interfaces |
US8928590B1 (en) * | 2012-04-03 | 2015-01-06 | Edge 3 Technologies, Inc. | Gesture keyboard method and apparatus |
WO2013154720A1 (en) | 2012-04-13 | 2013-10-17 | Tk Holdings Inc. | Pressure sensor including a pressure sensitive material for use with control systems and methods of using the same |
TWI485577B (en) * | 2012-05-03 | 2015-05-21 | Compal Electronics Inc | Electronic apparatus and operating method thereof |
US9652043B2 (en) * | 2012-05-14 | 2017-05-16 | Hewlett-Packard Development Company, L.P. | Recognizing commands with a depth sensor |
US10168835B2 (en) | 2012-05-23 | 2019-01-01 | Flatfrog Laboratories Ab | Spatial resolution in touch displays |
WO2013183938A1 (en) * | 2012-06-08 | 2013-12-12 | 주식회사 케이엠티글로벌 | User interface method and apparatus based on spatial location recognition |
US20130335378A1 (en) * | 2012-06-18 | 2013-12-19 | Tzyy-Pyng Lin | Touch device |
KR101925412B1 (en) * | 2012-07-03 | 2018-12-05 | 삼성전자주식회사 | Method and apparatus for controlling sleep mode in portable terminal |
US8497841B1 (en) * | 2012-08-23 | 2013-07-30 | Celluon, Inc. | System and method for a virtual keyboard |
WO2014043664A1 (en) | 2012-09-17 | 2014-03-20 | Tk Holdings Inc. | Single layer force sensor |
CN103713735B (en) * | 2012-09-29 | 2018-03-16 | 华为技术有限公司 | A kind of method and apparatus that terminal device is controlled using non-contact gesture |
US9164625B2 (en) | 2012-10-14 | 2015-10-20 | Neonode Inc. | Proximity sensor for determining two-dimensional coordinates of a proximal object |
US10324565B2 (en) | 2013-05-30 | 2019-06-18 | Neonode Inc. | Optical proximity sensor |
US10282034B2 (en) | 2012-10-14 | 2019-05-07 | Neonode Inc. | Touch sensitive curved and flexible displays |
US9741184B2 (en) | 2012-10-14 | 2017-08-22 | Neonode Inc. | Door handle with optical proximity sensors |
US9207800B1 (en) | 2014-09-23 | 2015-12-08 | Neonode Inc. | Integrated light guide and touch screen frame and multi-touch determination method |
US10585530B2 (en) | 2014-09-23 | 2020-03-10 | Neonode Inc. | Optical proximity sensor |
US9921661B2 (en) | 2012-10-14 | 2018-03-20 | Neonode Inc. | Optical proximity sensor and associated user interface |
RU2012145783A (en) * | 2012-10-26 | 2014-05-10 | Дисплаир, Инк. | METHOD AND DEVICE FOR RIGID CONTROL FOR MULTIMEDIA DISPLAY |
FR2997771A1 (en) * | 2012-11-06 | 2014-05-09 | H2I Technologies | Method for non-contact detection of e.g. hand by infrared radiation, for operating human computer interface in car, involves determining position of target by triangulation, and using relative coordinates of reflection points and signal |
US9285893B2 (en) | 2012-11-08 | 2016-03-15 | Leap Motion, Inc. | Object detection and tracking with variable-field illumination devices |
KR102070778B1 (en) * | 2012-11-23 | 2020-03-02 | 엘지전자 주식회사 | Rgb-ir sensor with pixels array and apparatus and method for obtaining 3d image using the same |
TWI581127B (en) * | 2012-12-03 | 2017-05-01 | 廣達電腦股份有限公司 | Input device and electrical device |
US20140168372A1 (en) * | 2012-12-17 | 2014-06-19 | Eminent Electronic Technology Corp. Ltd. | Sensing apparatus and sensing method for generating three-dimensional image information |
FR3000243B1 (en) * | 2012-12-21 | 2015-02-06 | Dav | INTERFACE MODULE |
TWI533256B (en) * | 2013-01-07 | 2016-05-11 | 義明科技股份有限公司 | Gesture sensing device and method of sensing three-dimensional gestures |
US9667883B2 (en) | 2013-01-07 | 2017-05-30 | Eminent Electronic Technology Corp. Ltd. | Three-dimensional image sensing device and method of sensing three-dimensional images |
US10609285B2 (en) | 2013-01-07 | 2020-03-31 | Ultrahaptics IP Two Limited | Power consumption in motion-capture systems |
US9465461B2 (en) | 2013-01-08 | 2016-10-11 | Leap Motion, Inc. | Object detection and tracking with audio and optical signals |
US9141198B2 (en) * | 2013-01-08 | 2015-09-22 | Infineon Technologies Ag | Control of a control parameter by gesture recognition |
US9223442B2 (en) * | 2013-01-10 | 2015-12-29 | Samsung Display Co., Ltd. | Proximity and touch sensing surface for integration with a display |
US9459697B2 (en) | 2013-01-15 | 2016-10-04 | Leap Motion, Inc. | Dynamic, free-space user interactions for machine control |
US10241639B2 (en) | 2013-01-15 | 2019-03-26 | Leap Motion, Inc. | Dynamic user interactions for display control and manipulation of display objects |
CA2898452C (en) * | 2013-01-16 | 2021-02-09 | Blackberry Limited | Electronic device with touch-sensitive display and gesture-detection |
US9335922B2 (en) | 2013-01-16 | 2016-05-10 | Research In Motion Limited | Electronic device including three-dimensional gesture detecting display |
US9323380B2 (en) | 2013-01-16 | 2016-04-26 | Blackberry Limited | Electronic device with touch-sensitive display and three-dimensional gesture-detection |
US10152135B2 (en) * | 2013-03-15 | 2018-12-11 | Intel Corporation | User interface responsive to operator position and gestures |
US10620709B2 (en) | 2013-04-05 | 2020-04-14 | Ultrahaptics IP Two Limited | Customized gesture interpretation |
SE537579C2 (en) * | 2013-04-11 | 2015-06-30 | Crunchfish Ab | Portable device utilizes a passive sensor for initiating contactless gesture control |
WO2014168567A1 (en) | 2013-04-11 | 2014-10-16 | Flatfrog Laboratories Ab | Tomographic processing for touch detection |
US9916009B2 (en) | 2013-04-26 | 2018-03-13 | Leap Motion, Inc. | Non-tactile interface systems and methods |
US9747696B2 (en) | 2013-05-17 | 2017-08-29 | Leap Motion, Inc. | Systems and methods for providing normalized parameters of motions of objects in three-dimensional space |
US9423913B2 (en) | 2013-07-01 | 2016-08-23 | Blackberry Limited | Performance control of ambient light sensors |
US9398221B2 (en) | 2013-07-01 | 2016-07-19 | Blackberry Limited | Camera control using ambient light sensors |
US9342671B2 (en) | 2013-07-01 | 2016-05-17 | Blackberry Limited | Password by touch-less gesture |
US9367137B2 (en) | 2013-07-01 | 2016-06-14 | Blackberry Limited | Alarm operation by touch-less gesture |
US9489051B2 (en) | 2013-07-01 | 2016-11-08 | Blackberry Limited | Display navigation using touch-less gestures |
US9405461B2 (en) | 2013-07-09 | 2016-08-02 | Blackberry Limited | Operating a device using touchless and touchscreen gestures |
WO2015005919A1 (en) * | 2013-07-10 | 2015-01-15 | Hewlett-Packard Development Company, L.P. | Sensor and tag to determine a relative position |
US9874978B2 (en) | 2013-07-12 | 2018-01-23 | Flatfrog Laboratories Ab | Partial detect mode |
US9465448B2 (en) | 2013-07-24 | 2016-10-11 | Blackberry Limited | Backlight for touchless gesture detection |
US9280259B2 (en) | 2013-07-26 | 2016-03-08 | Blackberry Limited | System and method for manipulating an object in a three-dimensional desktop environment |
EP2829949B1 (en) * | 2013-07-26 | 2019-11-06 | BlackBerry Limited | System and method for manipulating an object in a three-dimensional desktop environment |
US10281987B1 (en) | 2013-08-09 | 2019-05-07 | Leap Motion, Inc. | Systems and methods of free-space gestural interaction |
US9721383B1 (en) | 2013-08-29 | 2017-08-01 | Leap Motion, Inc. | Predictive information for free space gesture control and communication |
CN104423564B (en) | 2013-09-11 | 2018-03-27 | 联想(北京)有限公司 | Input information identifying method, device and electronic equipment |
US9390598B2 (en) | 2013-09-11 | 2016-07-12 | Blackberry Limited | Three dimensional haptics hybrid modeling |
JP2015060296A (en) * | 2013-09-17 | 2015-03-30 | 船井電機株式会社 | Spatial coordinate specification device |
US9632572B2 (en) | 2013-10-03 | 2017-04-25 | Leap Motion, Inc. | Enhanced field of view to augment three-dimensional (3D) sensory space for free-space gesture interpretation |
EP2876526B1 (en) * | 2013-10-10 | 2019-01-16 | Elmos Semiconductor Aktiengesellschaft | Device for gesture recognition and method for recognition of gestures |
WO2015065341A1 (en) * | 2013-10-29 | 2015-05-07 | Intel Corporation | Gesture based human computer interaction |
US9996638B1 (en) | 2013-10-31 | 2018-06-12 | Leap Motion, Inc. | Predictive information for free space gesture control and communication |
CN103616955A (en) * | 2013-12-10 | 2014-03-05 | 步步高教育电子有限公司 | Calligraphy or gesture recognition method and device |
CN104714630B (en) * | 2013-12-12 | 2017-12-29 | 联想(北京)有限公司 | Gesture identification method, system and computer |
US9989942B2 (en) * | 2013-12-30 | 2018-06-05 | Qualcomm Incorporated | Preemptively triggering a device action in an Internet of Things (IoT) environment based on a motion-based prediction of a user initiating the device action |
US9613262B2 (en) | 2014-01-15 | 2017-04-04 | Leap Motion, Inc. | Object detection and tracking for providing a virtual device experience |
US10126882B2 (en) | 2014-01-16 | 2018-11-13 | Flatfrog Laboratories Ab | TIR-based optical touch systems of projection-type |
US10146376B2 (en) | 2014-01-16 | 2018-12-04 | Flatfrog Laboratories Ab | Light coupling in TIR-based optical touch systems |
JP6287382B2 (en) * | 2014-03-12 | 2018-03-07 | オムロン株式会社 | Gesture recognition device and method for controlling gesture recognition device |
US9563956B2 (en) * | 2014-03-26 | 2017-02-07 | Intel Corporation | Efficient free-space finger recognition |
WO2015147863A1 (en) * | 2014-03-28 | 2015-10-01 | Hewlett-Packard Development Company, L.P. | Computing device |
TWI509488B (en) * | 2014-04-30 | 2015-11-21 | Quanta Comp Inc | Optical touch system |
CN105266759A (en) * | 2014-05-26 | 2016-01-27 | 义明科技股份有限公司 | Physiological signal detection device |
US9864470B2 (en) * | 2014-05-30 | 2018-01-09 | Flatfrog Laboratories Ab | Enhanced interaction touch system |
WO2015199602A1 (en) | 2014-06-27 | 2015-12-30 | Flatfrog Laboratories Ab | Detection of surface contamination |
CN106462252B (en) * | 2014-06-30 | 2019-05-21 | 歌乐株式会社 | Touchless manipulation detection device |
US9866820B1 (en) * | 2014-07-01 | 2018-01-09 | Amazon Technologies, Inc. | Online calibration of cameras |
CN104375700A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Electronic device |
CN104375640A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Touch control device |
CN104375716A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Touch sensing system, control device and mobile device |
CN104375638A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Sensing equipment, mobile terminal and air sensing system |
CN104375717A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Portable device, touch control system and touch device |
CN104375698A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Touch control device |
CN104375718A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Aerial induction device, aerial induction system and electronic equipment |
CN104375639A (en) * | 2014-07-17 | 2015-02-25 | 深圳市钛客科技有限公司 | Aerial sensing device |
WO2016018416A1 (en) * | 2014-07-31 | 2016-02-04 | Hewlett-Packard Development Company, L.P. | Determining the location of a user input device |
WO2016018355A1 (en) * | 2014-07-31 | 2016-02-04 | Hewlett-Packard Development Company, L.P. | Virtual reality clamshell computing device |
CN204480228U (en) | 2014-08-08 | 2015-07-15 | 厉动公司 | motion sensing and imaging device |
CN104216560B (en) * | 2014-08-19 | 2018-01-16 | 深圳市魔眼科技有限公司 | Mobile device and realize the system of the aerial touch-control of mobile device, control device |
KR102263064B1 (en) * | 2014-08-25 | 2021-06-10 | 삼성전자주식회사 | Apparatus and method for recognizing movement of a subject |
JP6337715B2 (en) * | 2014-09-19 | 2018-06-06 | コニカミノルタ株式会社 | Image forming apparatus and program |
KR101601951B1 (en) * | 2014-09-29 | 2016-03-09 | 주식회사 토비스 | Curved Display for Performing Air Touch Input |
FR3028967B1 (en) * | 2014-11-21 | 2017-12-15 | Renault Sas | GRAPHICAL INTERFACE AND METHOD FOR MANAGING THE GRAPHICAL INTERFACE WHEN TACTILE SELECTING A DISPLAYED ELEMENT |
WO2016122385A1 (en) | 2015-01-28 | 2016-08-04 | Flatfrog Laboratories Ab | Dynamic touch quarantine frames |
US10318074B2 (en) | 2015-01-30 | 2019-06-11 | Flatfrog Laboratories Ab | Touch-sensing OLED display with tilted emitters |
EP3537269A1 (en) | 2015-02-09 | 2019-09-11 | FlatFrog Laboratories AB | Optical touch system |
US9696795B2 (en) | 2015-02-13 | 2017-07-04 | Leap Motion, Inc. | Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments |
US10401546B2 (en) | 2015-03-02 | 2019-09-03 | Flatfrog Laboratories Ab | Optical component for light coupling |
JP6485160B2 (en) * | 2015-03-27 | 2019-03-20 | セイコーエプソン株式会社 | Interactive projector and interactive projector control method |
US20160357260A1 (en) * | 2015-06-03 | 2016-12-08 | Stmicroelectronics (Research & Development) Limited | Distance independent gesture detection |
CN105072430B (en) * | 2015-08-19 | 2017-10-03 | 海信集团有限公司 | A kind of method and apparatus for adjusting projected image |
US10606468B2 (en) | 2015-11-20 | 2020-03-31 | International Business Machines Corporation | Dynamic image compensation for pre-touch localization on a reflective surface |
US9823782B2 (en) * | 2015-11-20 | 2017-11-21 | International Business Machines Corporation | Pre-touch localization on a reflective surface |
US20170153708A1 (en) * | 2015-11-29 | 2017-06-01 | Tusher Chakraborty | Secured and Noise-suppressed Multidirectional Gesture Recognition |
US10775937B2 (en) | 2015-12-09 | 2020-09-15 | Flatfrog Laboratories Ab | Stylus identification |
CN106060391B (en) | 2016-06-27 | 2020-02-21 | 联想(北京)有限公司 | Processing method and device for working mode of camera and electronic equipment |
CN106502570B (en) * | 2016-10-25 | 2020-07-31 | 科世达(上海)管理有限公司 | Gesture recognition method and device and vehicle-mounted system |
US10761657B2 (en) | 2016-11-24 | 2020-09-01 | Flatfrog Laboratories Ab | Automatic optimisation of touch signal |
EP3667475B1 (en) | 2016-12-07 | 2022-09-07 | FlatFrog Laboratories AB | A curved touch device |
CN110300950B (en) | 2017-02-06 | 2023-06-16 | 平蛙实验室股份公司 | Optical coupling in touch sensing systems |
WO2018174788A1 (en) | 2017-03-22 | 2018-09-27 | Flatfrog Laboratories | Object characterisation for touch displays |
EP4036697A1 (en) | 2017-03-28 | 2022-08-03 | FlatFrog Laboratories AB | Optical touch sensing apparatus |
WO2019045629A1 (en) | 2017-09-01 | 2019-03-07 | Flatfrog Laboratories Ab | Improved optical component |
WO2019172826A1 (en) | 2018-03-05 | 2019-09-12 | Flatfrog Laboratories Ab | Improved touch-sensing apparatus |
US11875012B2 (en) | 2018-05-25 | 2024-01-16 | Ultrahaptics IP Two Limited | Throwable interface for augmented reality and virtual reality environments |
WO2020028826A1 (en) * | 2018-08-02 | 2020-02-06 | Firefly Dimension, Inc. | System and method for human interaction with virtual objects |
US10921854B2 (en) * | 2018-09-06 | 2021-02-16 | Apple Inc. | Electronic device with sensing strip |
US10524461B1 (en) * | 2018-09-25 | 2020-01-07 | Jace W. Files | Pest detector to identify a type of pest using machine learning |
WO2020153890A1 (en) | 2019-01-25 | 2020-07-30 | Flatfrog Laboratories Ab | A videoconferencing terminal and method of operating the same |
US10977821B2 (en) * | 2019-06-12 | 2021-04-13 | The Boeing Company | High accuracy extrinsic calibration procedure for cameras and range sensors |
US11698457B2 (en) * | 2019-09-04 | 2023-07-11 | Pixart Imaging Inc. | Object detecting system and object detecting method |
US11281337B1 (en) * | 2019-09-24 | 2022-03-22 | Apple Inc. | Mirror accessory for camera based touch detection |
US10838049B1 (en) * | 2019-12-17 | 2020-11-17 | The Boeing Company | Calibration procedure for establishing an extrinsic relationship between lidar and camera sensors |
JP2023504590A (en) | 2019-12-31 | 2023-02-03 | ネオノード インコーポレイテッド | Contactless touch input system |
KR20210100850A (en) * | 2020-02-07 | 2021-08-18 | 삼성전자주식회사 | Electronic device and system for processing user input and method thereof |
WO2021162602A1 (en) | 2020-02-10 | 2021-08-19 | Flatfrog Laboratories Ab | Improved touch-sensing apparatus |
CN111880676B (en) * | 2020-06-22 | 2022-03-15 | 深圳市鸿合创新信息技术有限责任公司 | Partition touch control method and device, electronic equipment and storage medium |
JP2023544332A (en) | 2020-09-30 | 2023-10-23 | ネオノード インコーポレイテッド | optical touch sensor |
US11900631B2 (en) * | 2021-01-22 | 2024-02-13 | Lenovo (Singapore) Pte. Ltd. | Operating mode change by image compare |
US12026317B2 (en) | 2021-09-16 | 2024-07-02 | Apple Inc. | Electronic devices with air input sensors |
Family Cites Families (142)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US844152A (en) * | 1906-02-21 | 1907-02-12 | William Jay Little | Camera. |
US3025406A (en) * | 1959-02-05 | 1962-03-13 | Flightex Fabrics Inc | Light screen for ballistic uses |
US3563771A (en) * | 1968-02-28 | 1971-02-16 | Minnesota Mining & Mfg | Novel black glass bead products |
US3784813A (en) * | 1972-06-06 | 1974-01-08 | Gen Electric | Test apparatus for pneumatic brake system |
US3860754A (en) * | 1973-05-07 | 1975-01-14 | Univ Illinois | Light beam position encoder apparatus |
US4144449A (en) * | 1977-07-08 | 1979-03-13 | Sperry Rand Corporation | Position detection apparatus |
CA1109539A (en) * | 1978-04-05 | 1981-09-22 | Her Majesty The Queen, In Right Of Canada, As Represented By The Ministe R Of Communications | Touch sensitive computer input device |
US4243879A (en) * | 1978-04-24 | 1981-01-06 | Carroll Manufacturing Corporation | Touch panel with ambient light sampling |
US4243618A (en) * | 1978-10-23 | 1981-01-06 | Avery International Corporation | Method for forming retroreflective sheeting |
US4568912A (en) * | 1982-03-18 | 1986-02-04 | Victor Company Of Japan, Limited | Method and system for translating digital signal sampled at variable frequency |
US4507557A (en) * | 1983-04-01 | 1985-03-26 | Siemens Corporate Research & Support, Inc. | Non-contact X,Y digitizer using two dynamic ram imagers |
US4893120A (en) * | 1986-11-26 | 1990-01-09 | Digital Electronics Corporation | Touch panel using modulated light |
US4811004A (en) * | 1987-05-11 | 1989-03-07 | Dale Electronics, Inc. | Touch panel system and method for using same |
US4990901A (en) * | 1987-08-25 | 1991-02-05 | Technomarket, Inc. | Liquid crystal display touch screen having electronics on one side |
US5196835A (en) * | 1988-09-30 | 1993-03-23 | International Business Machines Corporation | Laser touch panel reflective surface aberration cancelling |
US5179369A (en) * | 1989-12-06 | 1993-01-12 | Dale Electronics, Inc. | Touch panel and method for controlling same |
JPH0458316A (en) * | 1990-06-28 | 1992-02-25 | Toshiba Corp | Information processor |
US5097516A (en) * | 1991-02-28 | 1992-03-17 | At&T Bell Laboratories | Technique for illuminating a surface with a gradient intensity line of light to achieve enhanced two-dimensional imaging |
US5196836A (en) * | 1991-06-28 | 1993-03-23 | International Business Machines Corporation | Touch panel display |
US6141000A (en) * | 1991-10-21 | 2000-10-31 | Smart Technologies Inc. | Projection display system with touch sensing on screen, computer assisted alignment correction and network conferencing |
US5483261A (en) * | 1992-02-14 | 1996-01-09 | Itu Research, Inc. | Graphical input controller and method with rear screen image detection |
DE69331433T2 (en) * | 1992-10-22 | 2002-10-02 | Advanced Interconnection Technology, Inc. | Device for the automatic optical inspection of printed circuit boards with wires laid therein |
US5751355A (en) * | 1993-01-20 | 1998-05-12 | Elmo Company Limited | Camera presentation supporting system |
US5502568A (en) * | 1993-03-23 | 1996-03-26 | Wacom Co., Ltd. | Optical position detecting unit, optical coordinate input unit and optical position detecting method employing a pattern having a sequence of 1's and 0's |
US5729704A (en) * | 1993-07-21 | 1998-03-17 | Xerox Corporation | User-directed method for operating on an object-based model data structure through a second contextual image |
US5490655A (en) * | 1993-09-16 | 1996-02-13 | Monger Mounts, Inc. | Video/data projector and monitor ceiling/wall mount |
US5729404A (en) * | 1993-09-30 | 1998-03-17 | Seagate Technology, Inc. | Disc drive spindle motor with rotor isolation and controlled resistance electrical pathway from disc to ground |
US7310072B2 (en) * | 1993-10-22 | 2007-12-18 | Kopin Corporation | Portable communication display device |
US5739850A (en) * | 1993-11-30 | 1998-04-14 | Canon Kabushiki Kaisha | Apparatus for improving the image and sound processing capabilities of a camera |
US5484966A (en) * | 1993-12-07 | 1996-01-16 | At&T Corp. | Sensing stylus position using single 1-D image sensor |
US5712658A (en) * | 1993-12-28 | 1998-01-27 | Hitachi, Ltd. | Information presentation apparatus and information display apparatus |
US6141485A (en) * | 1994-11-11 | 2000-10-31 | Mitsubishi Denki Kabushiki Kaisha | Digital signal recording apparatus which utilizes predetermined areas on a magnetic tape for multiple purposes |
EP0716391B1 (en) * | 1994-12-08 | 2001-09-26 | Hyundai Electronics America | Electrostatic pen apparatus and method |
US5594469A (en) * | 1995-02-21 | 1997-01-14 | Mitsubishi Electric Information Technology Center America Inc. | Hand gesture machine control system |
JP3098926B2 (en) * | 1995-03-17 | 2000-10-16 | 株式会社日立製作所 | Anti-reflective coating |
US5591945A (en) * | 1995-04-19 | 1997-01-07 | Elo Touchsystems, Inc. | Acoustic touch position sensor using higher order horizontally polarized shear wave propagation |
US6191773B1 (en) * | 1995-04-28 | 2001-02-20 | Matsushita Electric Industrial Co., Ltd. | Interface apparatus |
US5734375A (en) * | 1995-06-07 | 1998-03-31 | Compaq Computer Corporation | Keyboard-compatible optical determination of object's position |
US6031524A (en) * | 1995-06-07 | 2000-02-29 | Intermec Ip Corp. | Hand-held portable data terminal having removably interchangeable, washable, user-replaceable components with liquid-impervious seal |
US5709910A (en) * | 1995-11-06 | 1998-01-20 | Lockheed Idaho Technologies Company | Method and apparatus for the application of textile treatment compositions to textile materials |
US6015214A (en) * | 1996-05-30 | 2000-01-18 | Stimsonite Corporation | Retroreflective articles having microcubes, and tools and methods for forming microcubes |
US6208329B1 (en) * | 1996-08-13 | 2001-03-27 | Lsi Logic Corporation | Supplemental mouse button emulation system, method and apparatus for a coordinate based data input device |
JP3624070B2 (en) * | 1997-03-07 | 2005-02-23 | キヤノン株式会社 | Coordinate input device and control method thereof |
US5801919A (en) * | 1997-04-04 | 1998-09-01 | Gateway 2000, Inc. | Adjustably mounted camera assembly for portable computers |
US6346966B1 (en) * | 1997-07-07 | 2002-02-12 | Agilent Technologies, Inc. | Image acquisition system for machine vision applications |
US20020036617A1 (en) * | 1998-08-21 | 2002-03-28 | Timothy R. Pryor | Novel man machine interfaces and applications |
JP3794180B2 (en) * | 1997-11-11 | 2006-07-05 | セイコーエプソン株式会社 | Coordinate input system and coordinate input device |
US6031531A (en) * | 1998-04-06 | 2000-02-29 | International Business Machines Corporation | Method and system in a graphical user interface for facilitating cursor object movement for physically challenged computer users |
US6313853B1 (en) * | 1998-04-16 | 2001-11-06 | Nortel Networks Limited | Multi-service user interface |
US6020878A (en) * | 1998-06-01 | 2000-02-01 | Motorola, Inc. | Selective call radio with hinged touchpad |
JP4033582B2 (en) * | 1998-06-09 | 2008-01-16 | 株式会社リコー | Coordinate input / detection device and electronic blackboard system |
JP2000043484A (en) * | 1998-07-30 | 2000-02-15 | Ricoh Co Ltd | Electronic whiteboard system |
US7268774B2 (en) * | 1998-08-18 | 2007-09-11 | Candledragon, Inc. | Tracking motion of a writing instrument |
JP2000089913A (en) * | 1998-09-08 | 2000-03-31 | Gunze Ltd | Touch panel input coordinate converting device |
US6690357B1 (en) * | 1998-10-07 | 2004-02-10 | Intel Corporation | Input device using scanning sensors |
US6504634B1 (en) * | 1998-10-27 | 2003-01-07 | Air Fiber, Inc. | System and method for improved pointing accuracy |
DE19856007A1 (en) * | 1998-12-04 | 2000-06-21 | Bayer Ag | Display device with touch sensor |
US6147678A (en) * | 1998-12-09 | 2000-11-14 | Lucent Technologies Inc. | Video hand image-three-dimensional computer interface with multiple degrees of freedom |
US6335724B1 (en) * | 1999-01-29 | 2002-01-01 | Ricoh Company, Ltd. | Method and device for inputting coordinate-position and a display board system |
US6179426B1 (en) * | 1999-03-03 | 2001-01-30 | 3M Innovative Properties Company | Integrated front projection system |
JP2001014091A (en) * | 1999-06-30 | 2001-01-19 | Ricoh Co Ltd | Coordinate input device |
JP3986710B2 (en) * | 1999-07-15 | 2007-10-03 | 株式会社リコー | Coordinate detection device |
JP2001060145A (en) * | 1999-08-23 | 2001-03-06 | Ricoh Co Ltd | Coordinate input and detection system and alignment adjusting method therefor |
US6512838B1 (en) * | 1999-09-22 | 2003-01-28 | Canesta, Inc. | Methods for enhancing performance and data acquired from three-dimensional image systems |
JP4052498B2 (en) * | 1999-10-29 | 2008-02-27 | 株式会社リコー | Coordinate input apparatus and method |
US6690397B1 (en) * | 2000-06-05 | 2004-02-10 | Advanced Neuromodulation Systems, Inc. | System for regional data association and presentation and method for the same |
US6690363B2 (en) * | 2000-06-19 | 2004-02-10 | Next Holdings Limited | Touch panel display system |
US6803906B1 (en) * | 2000-07-05 | 2004-10-12 | Smart Technologies, Inc. | Passive touch system and method of detecting user input |
US7227526B2 (en) * | 2000-07-24 | 2007-06-05 | Gesturetek, Inc. | Video-based image control system |
JP3851763B2 (en) * | 2000-08-04 | 2006-11-29 | 株式会社シロク | Position detection device, position indicator, position detection method, and pen-down detection method |
JP2002073268A (en) * | 2000-09-04 | 2002-03-12 | Brother Ind Ltd | Coordinate reader |
US7058204B2 (en) * | 2000-10-03 | 2006-06-06 | Gesturetek, Inc. | Multiple camera control system |
US6897853B2 (en) * | 2000-11-10 | 2005-05-24 | Microsoft Corp. | Highlevel active pen matrix |
US6518600B1 (en) * | 2000-11-17 | 2003-02-11 | General Electric Company | Dual encapsulation for an LED |
JP4037128B2 (en) * | 2001-03-02 | 2008-01-23 | 株式会社リコー | Projection display apparatus and program |
JP4768143B2 (en) * | 2001-03-26 | 2011-09-07 | 株式会社リコー | Information input / output device, information input / output control method, and program |
US6517266B2 (en) * | 2001-05-15 | 2003-02-11 | Xerox Corporation | Systems and methods for hand-held printing on a surface or medium |
US7259747B2 (en) * | 2001-06-05 | 2007-08-21 | Reactrix Systems, Inc. | Interactive video display system |
US8035612B2 (en) * | 2002-05-28 | 2011-10-11 | Intellectual Ventures Holding 67 Llc | Self-contained interactive video display system |
GB2378073B (en) * | 2001-07-27 | 2005-08-31 | Hewlett Packard Co | Paper-to-computer interfaces |
US6927384B2 (en) * | 2001-08-13 | 2005-08-09 | Nokia Mobile Phones Ltd. | Method and device for detecting touch pad unit |
US7007236B2 (en) * | 2001-09-14 | 2006-02-28 | Accenture Global Services Gmbh | Lab window collaboration |
DE10163992A1 (en) * | 2001-12-24 | 2003-07-03 | Merck Patent Gmbh | 4-aryl-quinazolines |
US7821541B2 (en) * | 2002-04-05 | 2010-10-26 | Bruno Delean | Remote control apparatus using gesture recognition |
US7038659B2 (en) * | 2002-04-06 | 2006-05-02 | Janusz Wiktor Rajkowski | Symbol encoding apparatus and method |
US7015418B2 (en) * | 2002-05-17 | 2006-03-21 | Gsi Group Corporation | Method and system for calibrating a laser processing system and laser marking system utilizing same |
US7170492B2 (en) * | 2002-05-28 | 2007-01-30 | Reactrix Systems, Inc. | Interactive video display system |
US7348963B2 (en) * | 2002-05-28 | 2008-03-25 | Reactrix Systems, Inc. | Interactive video display system |
US7330184B2 (en) * | 2002-06-12 | 2008-02-12 | Smart Technologies Ulc | System and method for recognizing connector gestures |
US20040001144A1 (en) * | 2002-06-27 | 2004-01-01 | Mccharles Randy | Synchronization of camera images in camera-based touch system to enhance position determination of fast moving objects |
US20090143141A1 (en) * | 2002-08-06 | 2009-06-04 | Igt | Intelligent Multiplayer Gaming System With Multi-Touch Display |
JP2004078613A (en) * | 2002-08-19 | 2004-03-11 | Fujitsu Ltd | Touch panel system |
CA2502235A1 (en) * | 2002-10-10 | 2004-04-22 | Waawoo Technology Inc. | Pen-shaped optical mouse |
US6954197B2 (en) * | 2002-11-15 | 2005-10-11 | Smart Technologies Inc. | Size/scale and orientation determination of a pointer in a camera-based touch system |
US20040095311A1 (en) * | 2002-11-19 | 2004-05-20 | Motorola, Inc. | Body-centric virtual interactive apparatus and method |
US6995748B2 (en) * | 2003-01-07 | 2006-02-07 | Agilent Technologies, Inc. | Apparatus for controlling a screen pointer with a frame rate based on velocity |
TW594662B (en) * | 2003-06-03 | 2004-06-21 | Chunghwa Picture Tubes Ltd | Method for restraining noise when flat display turn on/off |
JP4125200B2 (en) * | 2003-08-04 | 2008-07-30 | キヤノン株式会社 | Coordinate input device |
CN1918532A (en) * | 2003-12-09 | 2007-02-21 | 雷阿卡特瑞克斯***公司 | Interactive video window display system |
US7492357B2 (en) * | 2004-05-05 | 2009-02-17 | Smart Technologies Ulc | Apparatus and method for detecting a pointer relative to a touch surface |
US7893920B2 (en) * | 2004-05-06 | 2011-02-22 | Alpine Electronics, Inc. | Operation input device and method of operation input |
JP4274997B2 (en) * | 2004-05-06 | 2009-06-10 | アルパイン株式会社 | Operation input device and operation input method |
US7644493B2 (en) * | 2004-07-02 | 2010-01-12 | Seagate Technology Llc | Adjustable head stack comb |
JP4442877B2 (en) * | 2004-07-14 | 2010-03-31 | キヤノン株式会社 | Coordinate input device and control method thereof |
WO2006015514A1 (en) * | 2004-08-12 | 2006-02-16 | Dong Li | An inductive keyboard of the portable terminal and the controlling method thereof |
EP1645944B1 (en) * | 2004-10-05 | 2012-08-15 | Sony France S.A. | A content-management interface |
US7616231B2 (en) * | 2005-01-06 | 2009-11-10 | Goodrich Corporation | CMOS active pixel sensor with improved dynamic range and method of operation for object motion detection |
EP1844388A2 (en) * | 2005-02-04 | 2007-10-17 | PolyVision Corporation | Apparatus and method for mounting interactive unit to flat panel display |
US7577925B2 (en) * | 2005-04-08 | 2009-08-18 | Microsoft Corporation | Processing for distinguishing pen gestures and dynamic self-calibration of pen-based computing systems |
US20070019103A1 (en) * | 2005-07-25 | 2007-01-25 | Vkb Inc. | Optical apparatus for virtual interface projection and sensing |
US9395905B2 (en) * | 2006-04-05 | 2016-07-19 | Synaptics Incorporated | Graphical scroll wheel |
US8587526B2 (en) * | 2006-04-12 | 2013-11-19 | N-Trig Ltd. | Gesture recognition feedback for a dual mode digitizer |
US20070257891A1 (en) * | 2006-05-03 | 2007-11-08 | Esenther Alan W | Method and system for emulating a mouse on a multi-touch sensitive surface |
US7333094B2 (en) * | 2006-07-12 | 2008-02-19 | Lumio Inc. | Optical touch screen |
US7333095B1 (en) * | 2006-07-12 | 2008-02-19 | Lumio Inc | Illumination for optical touch panel |
WO2008007372A2 (en) * | 2006-07-12 | 2008-01-17 | N-Trig Ltd. | Hover and touch detection for a digitizer |
US8441467B2 (en) * | 2006-08-03 | 2013-05-14 | Perceptive Pixel Inc. | Multi-touch sensing display through frustrated total internal reflection |
US8564544B2 (en) * | 2006-09-06 | 2013-10-22 | Apple Inc. | Touch screen device, method, and graphical user interface for customizing display of content category icons |
WO2008042310A2 (en) * | 2006-10-03 | 2008-04-10 | Dow Global Technologies Inc. | Improved atmospheric pressure plasma electrode |
KR100783552B1 (en) * | 2006-10-11 | 2007-12-07 | 삼성전자주식회사 | Input control method and device for mobile phone |
US20090030853A1 (en) * | 2007-03-30 | 2009-01-29 | De La Motte Alain L | System and a method of profiting or generating income from the built-in equity in real estate assets or any other form of illiquid asset |
WO2008137708A1 (en) * | 2007-05-04 | 2008-11-13 | Gesturetek, Inc. | Camera-based user input for compact devices |
JP5430572B2 (en) * | 2007-09-14 | 2014-03-05 | インテレクチュアル ベンチャーズ ホールディング 67 エルエルシー | Gesture-based user interaction processing |
WO2009045861A1 (en) * | 2007-10-05 | 2009-04-09 | Sensory, Incorporated | Systems and methods of performing speech recognition using gestures |
WO2009092599A1 (en) * | 2008-01-25 | 2009-07-30 | Sensitive Object | Touch-sensitive panel |
KR20100121512A (en) * | 2008-02-11 | 2010-11-17 | 넥스트 홀딩즈 리미티드 | Systems and methods for resolving multitouch scenarios for optical touchscreens |
US9772689B2 (en) * | 2008-03-04 | 2017-09-26 | Qualcomm Incorporated | Enhanced gesture-based image manipulation |
US8392847B2 (en) * | 2008-05-20 | 2013-03-05 | Hewlett-Packard Development Company, L.P. | System and method for providing content on an electronic device |
US20090327955A1 (en) * | 2008-06-28 | 2009-12-31 | Mouilleseaux Jean-Pierre M | Selecting Menu Items |
TW201009671A (en) * | 2008-08-21 | 2010-03-01 | Tpk Touch Solutions Inc | Optical semiconductor laser touch-control device |
JP2010050903A (en) * | 2008-08-25 | 2010-03-04 | Fujitsu Ltd | Transmission apparatus |
WO2010039663A2 (en) * | 2008-10-02 | 2010-04-08 | Next Holdings, Inc. | Stereo optical sensors for resolving multi-touch in a touch detection system |
US8339378B2 (en) * | 2008-11-05 | 2012-12-25 | Smart Technologies Ulc | Interactive input system with multi-angle reflector |
US8957865B2 (en) * | 2009-01-05 | 2015-02-17 | Apple Inc. | Device, method, and graphical user interface for manipulating a user interface object |
US9652030B2 (en) * | 2009-01-30 | 2017-05-16 | Microsoft Technology Licensing, Llc | Navigation of a virtual plane using a zone of restriction for canceling noise |
US20120044143A1 (en) * | 2009-03-25 | 2012-02-23 | John David Newton | Optical imaging secondary input means |
JP5256535B2 (en) * | 2009-07-13 | 2013-08-07 | ルネサスエレクトロニクス株式会社 | Phase-locked loop circuit |
US20110019204A1 (en) * | 2009-07-23 | 2011-01-27 | Next Holding Limited | Optical and Illumination Techniques for Position Sensing Systems |
US8438500B2 (en) * | 2009-09-25 | 2013-05-07 | Apple Inc. | Device, method, and graphical user interface for manipulation of user interface objects with activation regions |
US20110221666A1 (en) * | 2009-11-24 | 2011-09-15 | Not Yet Assigned | Methods and Apparatus For Gesture Recognition Mode Control |
US20110176082A1 (en) * | 2010-01-18 | 2011-07-21 | Matthew Allard | Mounting Members For Touch Sensitive Displays |
US20110234542A1 (en) * | 2010-03-26 | 2011-09-29 | Paul Marson | Methods and Systems Utilizing Multiple Wavelengths for Position Detection |
-
2010
- 2010-12-06 EP EP10798867A patent/EP2507683A1/en not_active Withdrawn
- 2010-12-06 EP EP10795126A patent/EP2507682A2/en not_active Withdrawn
- 2010-12-06 US US12/960,759 patent/US20110205155A1/en not_active Abandoned
- 2010-12-06 CN CN201080063109XA patent/CN102754048A/en active Pending
- 2010-12-06 EP EP10795511A patent/EP2507692A2/en not_active Withdrawn
- 2010-12-06 WO PCT/US2010/059050 patent/WO2011069148A1/en active Application Filing
- 2010-12-06 CN CN2010800631070A patent/CN102754047A/en active Pending
- 2010-12-06 WO PCT/US2010/059104 patent/WO2011069157A2/en active Application Filing
- 2010-12-06 US US12/960,900 patent/US20110205185A1/en not_active Abandoned
- 2010-12-06 US US12/961,199 patent/US20110205151A1/en not_active Abandoned
- 2010-12-06 CN CN201080063123XA patent/CN102741782A/en active Pending
- 2010-12-06 EP EP10798871A patent/EP2507684A2/en not_active Withdrawn
- 2010-12-06 WO PCT/US2010/059082 patent/WO2011069152A2/en active Application Filing
- 2010-12-06 WO PCT/US2010/059078 patent/WO2011069151A2/en active Application Filing
- 2010-12-06 US US12/961,175 patent/US20110205186A1/en not_active Abandoned
- 2010-12-06 CN CN2010800631117A patent/CN102741781A/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20110205155A1 (en) | 2011-08-25 |
US20110205151A1 (en) | 2011-08-25 |
CN102741782A (en) | 2012-10-17 |
EP2507684A2 (en) | 2012-10-10 |
CN102754048A (en) | 2012-10-24 |
US20110205186A1 (en) | 2011-08-25 |
WO2011069148A1 (en) | 2011-06-09 |
WO2011069151A3 (en) | 2011-09-22 |
EP2507692A2 (en) | 2012-10-10 |
WO2011069157A3 (en) | 2011-07-28 |
CN102754047A (en) | 2012-10-24 |
CN102741781A (en) | 2012-10-17 |
US20110205185A1 (en) | 2011-08-25 |
WO2011069152A3 (en) | 2012-03-22 |
EP2507682A2 (en) | 2012-10-10 |
WO2011069151A2 (en) | 2011-06-09 |
EP2507683A1 (en) | 2012-10-10 |
WO2011069152A2 (en) | 2011-06-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20110205151A1 (en) | Methods and Systems for Position Detection | |
US11720181B2 (en) | Cursor mode switching | |
US9619042B2 (en) | Systems and methods for remapping three-dimensional gestures onto a finite-size two-dimensional surface | |
US9454260B2 (en) | System and method for enabling multi-display input | |
US9747696B2 (en) | Systems and methods for providing normalized parameters of motions of objects in three-dimensional space | |
WO2011119154A1 (en) | Gesture mapping for display device | |
KR20140086805A (en) | Electronic apparatus, method for controlling the same and computer-readable recording medium | |
TWI444875B (en) | Multi-touch input apparatus and its interface method using data fusion of a single touch sensor pad and imaging sensor | |
TW201504925A (en) | Method for operating user interface and electronic device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201080063123.X Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10798871 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2010798871 Country of ref document: EP |