WO2017057107A1 - 入力装置、入力方法、及びプログラム - Google Patents
入力装置、入力方法、及びプログラム Download PDFInfo
- Publication number
- WO2017057107A1 WO2017057107A1 PCT/JP2016/077730 JP2016077730W WO2017057107A1 WO 2017057107 A1 WO2017057107 A1 WO 2017057107A1 JP 2016077730 W JP2016077730 W JP 2016077730W WO 2017057107 A1 WO2017057107 A1 WO 2017057107A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- input
- marker
- user
- captured image
- image
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 70
- 239000003550 marker Substances 0.000 claims abstract description 200
- 238000001514 detection method Methods 0.000 claims abstract description 75
- 230000033001 locomotion Effects 0.000 claims description 70
- 238000000605 extraction Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 abstract description 48
- 238000010586 diagram Methods 0.000 description 35
- 239000011521 glass Substances 0.000 description 29
- 230000008859 change Effects 0.000 description 18
- 238000003384 imaging method Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 11
- 238000007689 inspection Methods 0.000 description 10
- 230000007704 transition Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 239000000284 extract Substances 0.000 description 5
- 230000001133 acceleration Effects 0.000 description 4
- 230000010365 information processing Effects 0.000 description 4
- 238000010079 rubber tapping Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 210000000707 wrist Anatomy 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 125000002066 L-histidyl group Chemical group [H]N1C([H])=NC(C([H])([H])[C@](C(=O)[*])([H])N([H])[H])=C1[H] 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0172—Head mounted characterised by optical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
Definitions
- the present invention relates to an input device, an input method, and a program.
- Patent Document 1 discloses a technique for displaying a menu screen on a palm or switching the menu screen according to a gesture of a user's palm (for example, a gesture for opening a palm).
- Patent Document 2 discloses a technique for performing input using a virtual keyboard displayed on a head-mounted display. Patent Document 2 discloses a technique for fixing the virtual keyboard to an object in the real space so that the position of the virtual keyboard does not change even if the head mounted display wearer moves his / her head. ing.
- the present invention has been made in view of the above problems.
- the objective of this invention is providing the technique which improves the recognition precision at the time of recognizing a user's input operation from a captured image.
- the input device includes: 1) a position calculating unit that calculates a marker position at a timing based on a detection result of a sensor attached to a user's body using a captured image including the marker; and 2) the calculation.
- Input recognition means for recognizing an input designating a location on the captured image based on the position of the marker.
- the designated place is a place away from the marker.
- the marker is attached to the user's body or is a part of the user's body.
- the input method of the present invention is an input method executed by a computer.
- the input method includes: 1) a position calculating step of calculating a marker position at a timing based on a detection result of a sensor attached to the user's body using a captured image including the marker; and 2) the calculated An input recognition step for recognizing an input designating a place on the captured image away from the marker based on the position of the marker.
- the designated place is a place away from the marker.
- the marker is attached to the user's body or is a part of the user's body.
- a technique for improving the recognition accuracy when recognizing a user input operation from a captured image is provided.
- FIG. 1 is a block diagram illustrating an input device according to a first embodiment. It is a figure for demonstrating operation
- 3 is a flowchart illustrating a flow of processing executed by the input device according to the first embodiment. It is a figure which illustrates a mode that the marker image currently displayed on the display screen of a device is changed when the left arm is tapped by the user. It is a figure which illustrates an input position. It is a figure which illustrates the relationship between the input position on the plane of a captured image, and the position of a marker.
- FIG. 6 is a block diagram illustrating an input device according to a second embodiment.
- FIG. 10 is a diagram illustrating an operation of the input device according to the second embodiment.
- FIG. 6 is a block diagram illustrating an input device according to a third embodiment.
- FIG. 10 is a diagram illustrating a hardware configuration of a computer that implements an input device according to a third embodiment. It is a figure which illustrates a mode that the display position of a partial image does not change even if the position of a marker changes.
- FIG. 10 is a block diagram illustrating an input device according to a fourth embodiment. It is a figure which illustrates input operation by a user. It is a figure which illustrates the method of determining the start time of detection object time using a sensor timing. It is a figure which illustrates the picked-up image containing the marker of the blurred state. It is a figure which illustrates a mode that the shape defined by the motion of a marker is recognized as an input. It is a figure which illustrates gesture input. It is a figure which illustrates glass and a watch.
- FIG. 1 is a block diagram illustrating an input device 2000 according to the first embodiment.
- each block represents a functional unit configuration, not a hardware unit configuration.
- the input device 2000 includes a position calculation unit 2020 and an input recognition unit 2040.
- the position calculation unit 2020 calculates the position of the marker at the timing based on the detection result of the sensor attached to the user's body using the captured image including the marker.
- the marker is attached to the user's body or is a part of the user's body.
- the input recognition unit 2040 recognizes an input for designating a location on the captured image based on the calculated marker position. The place specified by this input is a place away from the marker.
- FIG. 2 is a diagram for conceptually explaining the operation of the input device 2000.
- the captured image 22 is a captured image generated by a camera that captures the periphery of the user.
- a marker is worn on the body of the user who uses the input device 2000.
- a watch-type device 10 is worn on the left arm 50 of the user.
- the device 10 displays the marker image 16 on the touch panel 14.
- This marker image 16 has a role of a marker.
- the user uses the input device 2000 to perform an input operation for designating a place such as a point or a range at a position away from the marker.
- a point on the captured image 22 that is away from the marker image 16 is designated based on the position of the marker image 16.
- an area on the captured image 22 that is away from the marker image 16 is specified based on the position of the marker image 16.
- the device that is operated by this input may be the input device 2000 or may be other than the input device 2000.
- the input device 2000 uses a sensor worn by the user to recognize the input operation by the user.
- the sensor 12 is built in the device 10.
- the sensor 12 is a vibration sensor, for example.
- the user taps the left arm 50 at the timing of performing an input operation.
- the position calculation unit 2020 calculates the position of the marker from the captured image 22 generated at the timing when the vibration is detected by the vibration sensor or at a timing close thereto. Then, the input device 2000 recognizes an input for designating a certain place on the captured image 22 away from the marker based on the calculated marker position.
- the input device 2000 can grasp that the input operation by the user has been performed by grasping that the tap or the like has been detected by the sensor. Therefore, according to the input device 2000, it is possible to prevent the input operation from being recognized even though the user has performed the input operation, or the input operation to be erroneously recognized even though the user has not performed the input operation. And the input operation intended by the user can be correctly recognized.
- the operation of the input device 2000 described with reference to FIG. 2 is an example for facilitating understanding of details of the input device 2000 described below, and the operation of the input device 2000 is limited to the above-described example. Do not mean. Details and variations of the operation of the input device 2000 will be described in the following description.
- Each functional component of the input device 2000 may be realized by hardware (eg, a hard-wired electronic circuit) that implements each functional component, or a combination of hardware and software (eg, electronic A combination of a circuit and a program for controlling the circuit may be realized.
- hardware eg, a hard-wired electronic circuit
- software eg, electronic A combination of a circuit and a program for controlling the circuit may be realized.
- a case where each functional component of the input device 2000 is realized by a combination of hardware and software will be further described.
- FIG. 3 is a diagram illustrating a configuration of a computer 1000 that implements the input device 2000.
- the computer 1000 is a variety of computers such as a head-mounted display, a tablet terminal, a smartphone, a personal computer (PC), or a server machine.
- the computer 1000 may be various computers designed to realize the input device 2000, or may be a general-purpose computer.
- the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage 1080, and an input / output interface 1100.
- the bus 1020 is a data transmission path for the processor 1040, the memory 1060, and the storage 1080 to transmit / receive data to / from each other.
- the method of connecting the processors 1040 and the like is not limited to bus connection.
- the processor 1040 is an arithmetic processing device such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).
- the memory 1060 is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory).
- the storage 1080 is a storage device such as a hard disk, an SSD (Solid State Drive), or a memory card. Further, the storage 1080 may be a memory such as a RAM or a “ROM”.
- the input / output interface 1100 is an interface for connecting the computer 1000 and an input / output device.
- the camera 20 is connected to the input / output interface 1100.
- the camera 20 is an arbitrary camera that can capture an image of the user's surroundings.
- the camera 20 repeatedly captures images and generates captured images 22 representing the respective imaging results.
- the camera 20 may be a 2D (two-dimensional) camera or a 3D (three-dimensional) camera.
- the position where the camera 20 is provided is arbitrary.
- the camera 20 is attached to an object worn by the user.
- the items worn by the user are, for example, a head-mounted display or an employee ID card provided from the neck.
- the camera 20 may be provided on a wall of a room where the user performs an input operation on the input device 2000. In the latter case, it is preferable that the shooting range (shooting direction, zoom ratio, etc.) of the camera 20 can be changed by remote control using a remote controller or the like.
- FIG. 4 is a diagram illustrating a state in which the camera 20 is provided on the head mounted display.
- the head mounted display 170 illustrated in FIG. 4 includes a display 172 that is a transmissive display. The user can see the image superimposed on the scenery on the real world by looking at the image displayed on the display 172 while viewing the scenery on the real world via the display 172.
- the camera 20 is provided near the display 172.
- the scene included in the captured image 22 generated by the camera 20 is changed to a scene close to the scene viewed by the user by the camera 20 capturing an image in the same direction as or near to the user's view direction. Can do.
- the imaging direction of the camera 20 is arbitrary, and is not limited to the user's visual field direction or a direction close thereto. Further, the position where the camera 20 is provided is not limited to the vicinity of the display 172.
- the head mounted display 170 provided with the camera 20 is not limited to a head mounted display having a transmissive display, and may be a head mounted display having a non-transmissive display. The relationship between the type of the head mounted display and the mounting method of the input device 2000 will be described later as necessary.
- the storage 1080 stores program modules for realizing the functions of the input device 2000.
- the processor 1040 implements each function corresponding to the program module by executing each program module.
- the processor 1040 may execute the modules after reading them onto the memory 1060 or without reading them onto the memory 1060.
- each program module may be stored in the memory 1060.
- the computer 1000 may not include the storage 1080.
- FIG. 5 is a flowchart illustrating the flow of processing executed by the input device 2000 according to the first embodiment.
- the position calculation unit 2020 grasps the detection result of the sensor (S102).
- the position calculation unit 2020 acquires the captured image 22 (S104).
- the position calculation unit 2020 calculates the position of the marker included in the captured image 22 (106).
- the input recognizing unit 2040 recognizes an input for designating a location on the captured image 22 away from the marker (S108).
- the sensor worn by the user is an arbitrary sensor that can be used to grasp the timing of the input operation by the user, and is not limited to the vibration sensor described above.
- the sensor may be an acceleration sensor, a vibration switch, a pressure sensor, a capacitance sensor, or the like. Similar to the vibration sensor, the acceleration sensor and the vibration switch are provided, for example, inside the device 10.
- the pressure sensor and the capacitance sensor are provided on the touch panel of the device 10, for example. Further, for example, the pressure sensor or the capacitance sensor may be provided on a sheet or the like that is attached to or wrapped around the user's arm or hand. Further, for example, the pressure sensor and the capacitance sensor may be provided on a user's clothes (such as a sleeve).
- the position calculation unit 2020 grasps the detection result of the sensor attached to the body of the user of the input device 2000 (S102). There are various methods for the position calculation unit 2020 to grasp the detection result of the sensor. The method will be exemplified below. Hereinafter, a case where the sensor is the sensor 12 (vibration sensor) described in FIG. 2 will be described as an example.
- the position calculation unit 2020 grasps the detection result of the sensor 12 by acquiring information indicating the detection result of the sensor 12 by performing wireless communication with the device 10 including the sensor 12. For example, the device 10 transmits a predetermined signal to the position calculation unit 2020 at a timing when the sensor 12 detects a vibration having a predetermined magnitude or more. In this case, the position calculation unit 2020 can acquire the detection result of the sensor 12 that “vibration is detected by the sensor 12” by receiving the predetermined signal.
- the device 10 may transmit information indicating the time point when the vibration is detected to the position calculation unit 2020.
- the device 10 may change the appearance of the device 10 in accordance with the vibration detected by the sensor 12.
- the position calculation unit 2020 grasps the detection result of the sensor 12 (vibration is detected by the sensor 12) by detecting this predetermined change using the image generated by the camera 20.
- the device 10 changes the display on the display screen when the sensor 12 detects a vibration of a predetermined magnitude or more. More specifically, the device 10 changes the image displayed on the display screen of the device 10 or nothing is displayed when vibration of a predetermined magnitude or more is detected by the sensor 12. A new image is displayed on the display screen.
- the position calculation unit 2020 detects that the display on the display screen of the device 10 has been changed by analyzing an image repeatedly generated by the camera 20. Thereby, the position calculation unit 2020 grasps that the vibration is detected by the sensor 12 built in the device 10.
- FIG. 6 is a diagram illustrating a state in which the marker image 16 displayed on the display screen (touch panel 14) of the device 10 is changed when the left arm 50 is tapped by the user.
- the device 10 may turn on or blink a light such as a backlight of the display screen of the device 10 or a light emitting diode (LED) light provided on the device 10. Good.
- the position calculation unit 2020 detects lighting and blinking of these lights by analyzing images repeatedly generated by the camera 20. Thereby, the position calculation unit 2020 grasps that the vibration is detected by the sensor 12.
- the position calculation unit 2020 grasps the detection of the vibration by changing the appearance of the device 10 according to the detection of the vibration by the sensor 12, wireless communication is performed between the device 10 and the input device 2000. There is no need. Therefore, the device 10 and the input device 2000 do not have to have a wireless communication function unless there is another necessity for wireless communication.
- the method by which the position calculation unit 2020 grasps the detection result by a sensor other than the vibration sensor is the same as the method described above for the vibration sensor.
- the sensor is an acceleration sensor or a vibration switch
- the device 10 or the position calculation unit 2020 detects that the vibration sensor has a predetermined magnitude or more when the sensor detects acceleration or vibration having a predetermined magnitude or more.
- the same processing as when detecting is performed.
- the sensor is a pressure sensor
- the device 10 and the position calculation unit 2020 are configured such that when the pressure sensor detects a pressure greater than a predetermined magnitude, the vibration sensor detects a vibration greater than a predetermined magnitude. Similar processing is performed.
- the device 10 or the position calculation unit 2020 determines that the vibration sensor has a predetermined size when the capacitance sensor detects a displacement of the capacitance greater than or equal to a predetermined size. The same processing as that when the above vibration is detected is performed.
- the position calculation unit 2020 acquires the captured image 22 (S104). There are various methods by which the position calculation unit 2020 acquires the captured image 22. For example, the position calculation unit 2020 acquires the captured image 22 from the camera 20. In this case, the input device 2000 and the camera 20 are communicably connected.
- the position calculation unit 2020 acquires the captured image 22 from the storage device.
- the position calculation unit 2020 is connected to be able to communicate with the storage device.
- the position calculation unit 2020 calculates the position of the marker included in the captured image 22 (S106). For the calculation of the marker position, the captured image 22 including the marker at the timing based on the detection result of the sensor is used.
- timing based on the detection result of the sensor is referred to as sensor timing.
- Sensor timing is timing when vibration is detected by the sensor or timing close thereto. For example, when a predetermined signal is transmitted from the device 10 to the position calculation unit 2020 when vibration or the like is detected by the sensor 12, the sensor timing is a point in time when the position calculation unit 2020 receives the predetermined signal. Further, for example, when information indicating a time point when vibration or the like is detected by the sensor 12 is transmitted from the device 10 to the position calculation unit 2020, the sensor timing is a time point indicated by this information. Further, for example, when a predetermined change is added to the appearance of the device 10 when vibration or the like is detected by the sensor 12, the sensor timing is a point in time when the predetermined change is detected by the position calculation unit 2020.
- the sensor timing may be a time point before or after a predetermined time from each time point described above (such as a time point when the above signal is received from the device 10).
- the predetermined time may be set in advance in the position calculation unit 2020 or may be stored in a storage device accessible from the position calculation unit 2020.
- the imaging generated by the camera 20 a predetermined time before the time when the above-described signal is received from the device 10.
- Image 22 is required. Therefore, the camera 20 starts imaging before vibration or the like is detected by the sensor 12. For example, the camera 20 continuously captures an image from when the user starts using the input device 2000 until the user ends.
- the captured image 22 generated by the camera 20 is stored in a storage device or the like continuously for a predetermined period.
- the camera 20 may start imaging after vibration is detected by the sensor 12. Good.
- the camera 20 receives a signal indicating that vibration has been detected by the sensor 12 from the device 10 or the input device 2000, and starts imaging at the timing when the signal is received.
- the position calculation unit 2020 calculates the position of the marker using the captured image 22 generated immediately before or after the sensor timing, for example.
- the marker is an arbitrary marker whose position can be specified at least in the captured image 22.
- the marker is a marker that can be used to determine a three-dimensional coordinate system.
- a marker that can be used for determining a three-dimensional coordinate system is, for example, an AR (Augmented Reality) marker.
- AR Augmented Reality
- any marker that can be used to determine the three-dimensional coordinate system is not limited to the reference direction, as long as it can obtain three directions perpendicular to each other from a certain reference point. Is not limited.
- the marker When the marker is attached to the user's body, the marker can be attached to any part of the user's body. For example, the marker is attached to the user's arm or hand.
- the marker is an image displayed on the display screen of the device attached to the user's arm.
- This device is an arbitrary electronic device having a function of displaying an image on a display screen.
- the device may be directly attached to the user's arm or may be attached to the user's arm from above the clothes.
- the marker is a marker image 16 displayed on the touch panel 14 of the watch-type device 10 attached to the wrist of the user.
- the marker image 16 may be an image stored in advance in the device 10 or an image stored in a storage device outside the device 10. In the latter case, the device 10 acquires the marker image 16 from the storage device and displays it.
- the marker is not limited to that displayed on the device as described above.
- the marker may be drawn directly on the user's arm or the like, or may be drawn on an arbitrary object on the user's arm or the like. In the latter case, for example, the marker is drawn on a ring attached to the user's finger, a wristband attached to the user's wrist, or a sleeve of clothes worn by the user.
- the marker may be drawn by hand or printed.
- the marker is a part of the user's body
- the marker can be any part of the user's body.
- the marker is a user's finger or wrist.
- the position calculation unit 2020 detects a marker from the acquired captured image 22 and calculates the position of the marker.
- the position calculation unit 2020 holds information about the shape, size, color, and the like relating to the marker in advance, and uses such known information to detect the marker from the acquired captured image 22 and to detect the position. Is calculated.
- a technique for detecting a predetermined object from the image and calculating the position of the object in the image is a known technique, details of this technique are omitted.
- the input recognizing unit 2040 Based on the calculated marker position, the input recognizing unit 2040 recognizes an input for designating a location on the captured image 22 away from the marker (S108). As described above, the place specified by the input may be one point on the captured image 22 or an area on the captured image 22.
- an input is recognized as a point or an area. For example, whether the input is recognized as one point or a region is set in the position calculation unit 2020 in advance. For example, information indicating whether an input is recognized as one point or an area is stored in advance in a storage device accessible from the input recognition unit 2040. Further, for example, before the input recognition unit 2040 recognizes the input, an input operation for selecting whether the input is recognized as one point or an area is received from the user. The method of accepting this input operation is arbitrary. For example, this input operation is performed using the touch panel 14 of the device 10.
- FIG. 7 is a diagram illustrating an input position.
- the input recognition unit 2040 recognizes the input position 60 as a place on the input captured image 22 (FIG. 7A).
- the input recognition unit 2040 recognizes the region 62 determined by the input position 60 as a place on the input captured image 22 (FIG. 7B).
- the input position 60 is relatively determined in advance in relation to the marker position on the captured image 22.
- the relationship between the input position and the marker position is determined by a two-dimensional positional relationship on the plane of the captured image 22.
- FIG. 8 is a diagram illustrating the relationship between the input position 60 and the marker position on the plane of the captured image 22.
- the relationship between the input position 60 and the marker position is defined by the following formula (1).
- (A x , a y ) are the coordinates of the input position 60 in the captured image 22.
- (M x, m y) are the coordinates of the center position of the marker in the captured image 22.
- (C x , c y ) is a relative position of the input position 60 with respect to the center position of the marker.
- the position of the input position 60 is determined in advance as a relative position in the three-dimensional space with respect to the position of the marker.
- This three-dimensional space is a three-dimensional space determined by a marker included in the captured image 22.
- FIG. 9 is a diagram illustrating the relationship between the region 62 and the position of the marker in the three-dimensional space.
- the input recognition unit 2040 calculates the coordinate axis 140 from the shape of the marker included in the captured image 22.
- the input recognizing unit 2040 uses the center of the marker as the origin, the plane determined by the marker as the XY plane, and the coordinate axis having the direction perpendicular to the plane as the Z direction as the coordinate axis 140.
- the relationship between the input position 60 and the marker position is determined by, for example, the following formula (2) on a three-dimensional space determined by the coordinate axis 140.
- (A x , a y , a z ) are the coordinates of the input position 60 on the three-dimensional space determined by the coordinate axis 140.
- (M x, m y, m y) are the coordinates of the center position of the marker on the 3-dimensional space defined by axes 140.
- (C x , c y , c z ) is a relative position of the input position 60 with respect to the center position of the marker in the three-dimensional space determined by the coordinate axis 140.
- the user when the relationship between the input position 60 and the marker position is determined by the relationship in the three-dimensional space, for example, the user performs an operation of changing the posture of the device 10 in FIG. 2 (an operation of twisting the wrist of the left hand) or the like. By doing so, the input position 60 can be changed without greatly moving the left arm 50.
- the position of the area 62 is determined by the input position 60.
- the input recognition unit 2040 sets a region where the position of the center or one vertex is determined by the input position 60 as the region 62.
- the shape of the region 62 is an arbitrary shape such as a rectangle or a circle.
- the size of the region 62 is also arbitrary.
- the shape and size of the area 62 are preset in the input recognition unit 2040.
- the shape and size of the area 62 are stored in advance in a storage device accessible from the input recognition unit 2040.
- the input device 2000 may accept an input designating the shape and size of the region 62 from the user. This input is performed using the touch panel 14 of the device 10, for example.
- FIG. 10 is a block diagram illustrating an input device 2000 according to the second embodiment.
- each block represents a functional unit configuration, not a hardware unit configuration.
- the input device 2000 of the second embodiment has the same functions as the input device 2000 of the first embodiment unless otherwise described.
- the input device 2000 according to the second embodiment recognizes an input specifying the area 62 on the captured image 22 by the method described in the first embodiment. Furthermore, the input device 2000 according to the second embodiment extracts a partial image in the region 62 from the captured image 22.
- FIG. 11 is a diagram illustrating the operation of the input device 2000 according to the second embodiment.
- the user inputs an area 62 using the marker image 16 displayed on the touch panel 14 of the device 10. Then, the input device 2000 extracts the partial image 24 in the region 62 from the captured image 22.
- the input device 2000 of the second embodiment further includes an extraction unit 2060.
- the extraction unit 2060 extracts a partial image included in the region 62 recognized by the input recognition unit 2040 from the captured image 22.
- a technique for extracting a portion included in a predetermined range from a certain image is a known technique, a detailed description of the technique is omitted.
- the captured image 22 used for recognizing the region 62 and the captured image 22 from which the partial image 24 is extracted may be the same or different. In the latter case, for example, the extraction unit 2060 extracts the partial image 24 from the latest captured image 22 when the input recognition unit 2040 finishes the process of recognizing the input operation.
- the input device 2000 of the second embodiment is realized by the computer 1000.
- the storage 1080 of the computer 1000 according to the second embodiment further includes a program module that implements each function of the input device 2000 described above.
- a region for extracting a partial image from a captured image can be designated by a user's intuitive method of using a marker attached to the user's body. Therefore, when it is desired to extract a part of the captured image, the user can easily and intuitively specify a part that the user wants to extract.
- extracting a partial image from a captured image can be performed by, for example, storing or displaying the portion of the captured image that is particularly focused on or performing image analysis on the portion.
- FIG. 12 is a block diagram illustrating an input device 2000 according to the third embodiment.
- each block represents a functional unit configuration, not a hardware unit configuration.
- the input device 2000 according to the third embodiment has the same functions as those of the input device 2000 according to the second embodiment, unless otherwise specified.
- the input device 2000 displays the partial image 24 on the display screen.
- the input device 2000 displays the partial image 24 so that the partial image 24 extracted by the extraction unit 2060 is superimposed on the scenery included in the captured image 22 or a scenery close thereto. Is displayed on the display screen.
- the input device 2000 controls the display of the partial image 24 so that the partial image 24 can be seen independently of the change in the scenery included in the captured image 22. .
- the input device 2000 of the third embodiment has a display control unit 2080.
- the display control unit 2080 displays the partial image 24 extracted by the extraction unit 2060 at a predetermined position on the display screen.
- the display control unit 2080 displays the partial image at the center of the display screen.
- the display control unit 2080 displays the partial image 24 so that the partial image 24 remains at the position when extracted by the extraction unit 2060.
- the display control unit 2080 continues to display the partial image 24 at the predetermined position on the display screen until a predetermined time elapses or a predetermined operation is performed by the user.
- the predetermined time may be preset in the display control unit 2080 or may be stored in a storage device accessible from the display control unit 2080. Further, the input device 2000 may accept an input designating the predetermined time from the user.
- the predetermined operation may be an arbitrary operation. For example, the predetermined operation is performed using the touch panel 14 of the device 10.
- the display control unit 2080 displays the partial image 24 on the display screen of the head mounted display 170 (display 172 in FIG. 4).
- the display control unit 2080 displays the partial image 24 on a projection surface such as a wall on which the captured image 22 is projected by the projector. In this case, the projection surface serves as a display screen.
- the display control unit 2080 displays the partial image 24 on an arbitrary display screen such as a liquid crystal display.
- the head-mounted display 170 includes a display 172 that is a non-transmissive display and a display that is a transmissive display.
- the display 172 is a non-transmissive display
- the head mounted display 170 superimposes the captured image 22 generated by the camera 20 and an image representing another object on the display 172.
- the display control unit 2080 of the present embodiment displays the captured image 22 on which the partial image 24 is superimposed on the display 172 so that the user can see the partial image 24 superimposed on the surrounding scenery.
- the display 172 is a transmissive display
- the user superimposes an image on the surrounding scenery by viewing both the real object (real world scenery) that can be seen at the front of the display 172 and the image displayed on the display 172. You can see what was done.
- the display control unit 2080 displays the partial image 24 on the display 172 to show the partial image 24 to the user.
- the real-world scene viewed by the user is a scene captured by the camera 20 or a scene close thereto. Therefore, the user can see the partial image 24 superimposed on the landscape included in the captured image 22 or a landscape close thereto.
- the display control unit 2080 includes a portion of the captured image 22 as in the case of the head mounted display 170 having a non-transmissive display.
- the image 24 is superimposed and displayed on the display screen.
- a method for the display control unit 2080 to determine the display position of the partial image 24 when the partial image 24 is displayed so as to remain at the position (input position 60) when the partial image 24 is extracted by the extraction unit 2060 is as follows. It depends on the type. When using a display screen on which the captured image 22 is displayed, such as a head-mounted display having a non-transmissive display, the display control unit 2080 stores the input position 60 when the partial image 24 is extracted, and then The partial image 24 is superimposed on the input position 60 on each captured image 22 displayed on the display screen.
- the display control unit 2080 is the input position 60 from which the partial image 24 is extracted, which is the position on the captured image 22. Is converted to a position on the display screen. Then, the display control unit 2080 displays the partial image 24 at the position on the display screen calculated by this conversion.
- the correspondence between the coordinates on the captured image 22 and the coordinates on the display screen is determined based on various parameters (view angle and focal length) related to the camera 20 and the positional relationship between the display screen and the camera 20. This correspondence relationship may be calculated by the display control unit 2080 using these parameters or the like, or may be predetermined as a set value.
- FIG. 13 is a diagram illustrating a hardware configuration of a computer 1000 that implements the input device 2000 according to the third embodiment.
- the configuration of the computer 1000 is the same as the configuration of the computer 1000 that implements the input device 2000 of the first embodiment.
- the storage 1080 included in the computer 1000 according to the third embodiment further includes a program module for realizing each function included in the input device 2000 according to the present embodiment.
- the input / output interface 1100 is connected to a display 172 of the head mounted display 170 on which the partial image 24 is displayed, a projector 160 that projects the partial image 24 onto the projection surface, and the like.
- FIG. 14 is a diagram illustrating a state in which the display position of the partial image 24 does not change even if the marker position changes.
- FIG. 14A shows a state before moving the marker
- FIG. 14B shows a state after moving the marker.
- the user can move the marker in a state where the partial image 24 is displayed at a fixed position on the display screen. Therefore, for example, the user can newly perform an input operation to be recognized by the input recognition unit 2040 in a state where the partial image 24 is displayed at a fixed position. Therefore, the degree of freedom of input operation by the user is improved.
- FIG. 15 is a diagram illustrating a state in which the display position of the partial image 24 does not change even when the imaging range of the camera 20 changes.
- FIG. 15A shows a captured image 22 generated by the camera 20 when the partial image 24 is extracted
- FIG. 15B shows a captured image 22 generated by the camera 20 whose imaging range has changed thereafter.
- the user can change the imaging range of the camera 20 in a state where the partial image 24 is displayed at a fixed position on the display screen.
- the user can change the viewing direction with the partial image 24 displayed at a fixed position. Therefore, the convenience of the input device 2000 by the user is improved.
- FIG. 16 is a block diagram illustrating an input device 2000 according to the fourth embodiment.
- each block represents a functional unit configuration, not a hardware unit configuration. Except when specifically described, the input device 2000 of the present embodiment has the same functions as the input device 2000 of the third embodiment.
- the input device 2000 recognizes an input operation by the user based on the movement of the marker included in the captured image 22.
- FIG. 17 is a diagram illustrating an input operation by the user.
- the user draws a line 70 on the partial image 24 by moving the marker.
- the line 70 is a line for emphasizing a particularly notable part of the state included in the partial image 24. Note that the input performed by the user is not limited to the input to the partial image 24.
- the input device 2000 further includes a motion detection unit 2100 and a second input recognition unit 2120.
- the motion detection unit 2100 detects the motion of the marker included in the captured image 22.
- the detected movement of the marker is the movement of the marker in time including the timing based on the detection result of the sensor attached to the body of the user of the input device 2000.
- the second input recognition unit 2120 recognizes an input based on the detected marker movement. This input may be for the input device 2000 or for other devices.
- the sensor may be a sensor used by the position calculation unit 2020 or a different sensor.
- the motion detection unit 2100 acquires the detection result of the sensor 12.
- the method for the movement detection unit 2100 to grasp the detection result of the sensor 12 is the same as the method for the position calculation unit 2020 to grasp the detection result of the sensor.
- the movement detection unit 2100 detects the movement of the marker in a period determined based on the detection result of the sensor 12.
- this “period determined based on the detection result of the sensor 12” is referred to as a detection target period.
- the motion detection unit 2100 determines the detection target period using one or two sensor timings described in the first embodiment. When the detection target period is determined using one sensor timing, the motion detection unit 2100 determines both the start time and the end time of the detection target period using one sensor timing.
- the motion detection unit 2100 determines the start point of the detection target period using the earlier sensor timing of the two sensor timings, and sets the later sensor timing. Used to determine the end time of the detection target period.
- FIG. 18 is a diagram illustrating a method for determining the start time of the detection target period using the sensor timing.
- the motion detection unit 2100 sets the sensor timing as the start time of the detection target period (FIG. 18A). Further, for example, the motion detection unit 2100 sets a time point a predetermined time before the sensor timing as the start time point of the detection target period (FIG. 18B). Further, for example, the motion detection unit 2100 sets a time point after a predetermined time from the sensor timing as the start time point of the detection target period (FIG. 18C).
- the method for determining the end point of the detection target period using the sensor timing is the same as the method for determining the start point of the detection target period using the sensor timing.
- the information representing each predetermined time described above may be set in advance in the motion detection unit 2100 or may be stored in a storage device accessible from the motion detection unit 2100. Further, the predetermined time used for determining the start time of the detection target period and the predetermined time used for the end time of the detection target period may be the same or different.
- the motion detection unit 2100 detects the movement of the marker using the captured image 22 generated in the detection target period.
- the motion detection unit 2100 performs marker recognition for each of the plurality of captured images 22 and calculates the position of the marker included in each captured image 22. Then, the motion detection unit 2100 uses the information representing the change in the marker position as information representing the marker motion.
- the information indicating the change in the marker position is, for example, information in which the marker positions are arranged in time series.
- the number of captured images 22 generated in the detection target period may be one.
- the motion detection unit 2100 calculates the motion of the marker from one captured image 22 that includes the blurred marker.
- FIG. 19 is a diagram illustrating a captured image 22 including a blurred marker.
- the marker image 16 is blurred so as to move from the marker image 16-A to the marker image 16-B.
- the motion detection unit 2100 detects a change in the position of a feature point (for example, the four corners or the center of the marker image 16) common to the marker image 16-A and the marker image 16-B as a marker motion.
- the second input recognition unit 2120 recognizes an input based on the detected marker movement.
- the second input recognition unit 2120 may recognize 1) a shape based on the detected marker movement as an input, or 2) recognize a predetermined input operation corresponding to the detected marker movement as an input.
- FIG. 20 is a diagram illustrating a state in which a shape determined by the marker movement 40 is recognized as an input.
- the second input recognition unit 2120 recognizes the shape 41 represented by the marker motion 40-A and the shape 42 represented by the motion 40-B as inputs. For example, this input is used when the user performs handwriting input.
- the second input recognition unit 2120 recognizes as an input a shape whose shape and size are determined by the movement of the marker, although the shape is different from the movement of the marker. Specifically, the second input recognizing unit 2120 recognizes the rectangle 44 having both ends of the motion 40-C as opposite ends of the diagonal line and the circle 46 having both ends of the motion 40-C as both ends of the diameter as inputs. For example, this input is used when the user performs an input (selection operation or the like) representing a certain range or draws a predetermined figure.
- Whether to use the method shown in FIG. 20 (a) or the method shown in FIG. 20 (b) may be fixed in advance or may be settable by the user.
- Information indicating which method is used may be set in advance in the second input recognition unit 2120 or may be stored in a storage device accessible from the second input recognition unit 2120.
- FIG. 21 is a diagram illustrating gesture input.
- FIG. 21A shows a flick operation
- FIG. 21B shows a rotation operation.
- Information associating the movement of the marker with a predetermined input operation corresponding to the movement may be preset in the second input recognition unit 2120, or stored in advance in an internal or external storage device of the input device 2000. It may be.
- the second input recognition unit 2120 may recognize only the movement of the marker as an input regardless of the position where the movement of the marker is performed, or 2) the movement of the marker and the movement of the marker. A combination with the position at which is performed may be recognized as an input. In the former case, the same marker movement represents the same input no matter where the movement of the marker is performed on the captured image 22. On the other hand, in the latter case, it is meaningful where the marker is moved on the captured image 22.
- the input operation illustrated in FIG. 17 is an operation of drawing a line 70 so as to surround a rust portion of a pipe included in the partial image 24. Therefore, the second input recognition unit 2120 needs to recognize not only the shape of the line 70 but also the place where the line 70 is drawn as an input.
- the second input recognition unit 2120 recognizes, as an input, the shape determined by the movement of the marker detected by the motion detection unit 2100 and the gesture determined by the movement of the marker.
- the second input recognition unit 2120 recognizes the input based on the transition of the coordinates of the marker on the captured image 22 detected by the motion detection unit 2100.
- the second input recognition unit 2120 recognizes the transition of the coordinate of the marker on the captured image 22 as an input. Therefore, the second input recognition unit 2120 recognizes not only the shape of the line 70 but also the position where the line 70 is drawn as an input.
- the second input recognition unit 2120 may recognize not the place where the marker operation is performed but a place away from the place as the place where the input is performed by the user. This is similar to the case where the input recognition unit 2040 recognizes a place away from the marker position as an input. In FIG. 17 described above, an input is performed for a position away from the marker by moving the marker.
- the second input recognition unit 2120 determines the transition of the marker coordinates on the captured image 22 detected by the motion detection unit 2100. It is converted into a transition of coordinates at a place away from the marker by a predetermined method.
- the second input recognition unit 2120 uses the relationship between the marker position and the input position 60 described using Expression (1). Specifically, the second input recognition unit 2120 converts the positions of the markers on the plurality of captured images 22 to the input positions 60 using Equation (1). And the 2nd input recognition part 2120 recognizes the information which arranged the calculated several input position 60 in time series as an input.
- the second input recognizing unit 2120 uses the relationship between the marker position and the input position 60 described using Expression (2), and sets the marker positions on the plurality of captured images 22 to the input position 60. May be converted to
- the second input recognition unit 2120 recognizes the input by 1) or 2) may be set in advance in the second input recognition unit 2120 or accessible from the second input recognition unit 2120 It may be stored in the storage device or may be selectable by the user.
- the input device 2000 recognizes user input by either the input recognition unit 2040 or the second input recognition unit 2120. There are various methods for determining whether the input device 2000 recognizes the user input by the input recognition unit 2040 or the second input recognition unit 2120.
- the input device 2000 receives an operation from the user to select which of the input recognition unit 2040 and the second input recognition unit 2120 should recognize the input.
- the input device 2000 may recognize the input by the second input recognition unit 2120 while the partial image 24 extracted by the extraction unit 2060 is displayed on the display screen by the display control unit 2080. . By doing so, the above-described line 70 can be drawn on the partial image 24 without the user performing an input method selection operation.
- the user in addition to an input operation (an input operation recognized by the input recognition unit 2040) that specifies a location away from the marker, the user performs an input operation (second input recognition unit 2120) using the movement of the marker.
- the input operation recognized by the user can be performed. Therefore, the variation of the input operation by the user increases, and the convenience of the input device 2000 is improved.
- FIG. 22 is a diagram illustrating the glass 200 and the watch 210.
- the glass 200 is a head mounted display provided with a camera 202.
- the input device 2000 is built in the glass 200.
- the user uses the glasses 200 by wearing them like glasses.
- the camera 202 is a video camera and corresponds to the camera 20 in each embodiment described above.
- the display 204 is a transmissive display in the shape of a lens.
- the glass 200 displays various information on the display 204. As a result, various information is superimposed on the real world scene for the user's eyes.
- the watch 210 has a built-in vibration sensor 211.
- the watch 210 has a touch panel 212.
- the watch 210, the vibration sensor 211, and the touch panel 212 correspond to the device 10, the sensor 12, and the touch panel 14 in each embodiment 1, respectively.
- the touch panel 212 displays a marker image 216.
- the glass 200 and the watch 210 constitute an input system that functions as an input interface for the information processing apparatus.
- the information processing apparatus that is the operation target of the input system may be the glass 200 or the watch 210, or may be a computer other than that.
- the information processing apparatus to be operated is a glass 200.
- FIG. 23 is a diagram illustrating an overview of the inspection work performed by the user.
- the user reads a meter indicating the state of each device according to the check list, or records a memo or takes a picture of a portion of the device that is anxious.
- the input operation for the glass 200 is roughly divided into three.
- the first input operation is an input for designating a place away from the marker image 216 included in the imaging range of the camera 202. This operation is the input described in the first to third embodiments.
- the state in which the glass 200 accepts this operation is called a first input mode.
- the second input operation is an input operation performed by moving a predetermined object within the shooting range of the camera 202.
- This operation is the input operation described in the fourth embodiment.
- the input device 2000 recognizes not only the marker image 216 but also movements of various objects such as a user's finger as an input.
- the state in which the glass 200 accepts this operation is referred to as a second input mode.
- the third input operation is an input operation performed by adding an operation to the operation image displayed on the display 204.
- the state in which the glass 200 accepts this operation is called a third input mode.
- the specific contents of the operation in the third input mode will be described later.
- FIG. 24 is a diagram illustrating an operation selection menu 300 included in the user's field of view.
- the operation selection menu 300 includes three operation images of an input button 302, a shooting button 304, and a recording button 306.
- the glass 200 displays this item so that the first item of the checklist can be seen in the vicinity of the left arm 50 to which the watch 210 is attached as seen from the user's eyes.
- the glass 200 displays an operation selection menu so that the operation selection menu can be seen on the left arm 50 to which the watch 210 is attached as viewed from the user's eyes.
- the user performs an operation of selecting an operation image. Specifically, the user performs an operation of tapping an operation image to be selected. Then, the watch 210 that has detected the vibration of the tap changes the marker image displayed on the touch panel 212. Then, the glass 200 detects that the image displayed on the touch panel 212 has changed, and performs a process of recognizing input accordingly.
- the state of the glass 200 when the operation selection menu 300 is displayed is the third input mode for accepting selection of an operation image. Therefore, the glass 200 determines which operation image has been selected by the user when detecting that the image displayed on the touch panel 212 has changed. Specifically, the glass 200 detects the position of the user's finger from the captured image 22 captured by the camera 202, and indicates which position of the operation image included in the operation selection menu 300 corresponds to the position. Find out.
- FIG. 25 is a diagram illustrating a guide 80 included in the user's field of view.
- the guide 80 represents the partial image 24 (the area 62 recognized by the input recognition unit 2040) extracted by the display control unit 2080 when the user inputs to the input device 2000.
- the user can visually confirm which part is extracted as the partial image 24 before extracting the partial image 24.
- the user moves the device 10 so that the guide 80 is displayed at a place where the user wants to save the image. Thereafter, the user performs input to the input device 2000 by tapping the left arm 50.
- the state of the input device 2000 transitions to a second mode in which the input recognition unit 2040 accepts an input operation. Therefore, when the user taps the left arm 50, the input recognition unit 2040 recognizes the area 62. Then, the display control unit 2080 extracts the partial image 24 using this area 62.
- the partial image 24 extracted by the region 62 is displayed at the center of the display 204 by the display control unit 2080.
- the pipe included in the partial image 24 is partially rusted, so the user draws 70 on the partial image 24 by moving the marker image 216. In this way, the user can save the partial image 24 while highlighting points of particular interest.
- FIG. 26 illustrates how the user draws the line 70.
- the input method selection menu 310 includes three operation images including a meter reading button 312, a numerical value input button 314, and a memo button 316.
- FIG. 27 is a diagram illustrating an input method selection menu 310 included in the user's field of view.
- the input method selection menu 310 includes three operation images including a meter reading button 312, a numerical value input button 314, and a memo button 316.
- the meter reading button 312 is an input operation for automatically reading the meter value from the captured image 22 captured by the camera 202.
- the glass 200 transitions to the first input mode.
- the automatic reading of the meter value is completed, the glass 200 transitions to the second input mode again.
- Numeral input button 314 is an input operation in which the user himself inputs the meter value.
- FIG. 28 is a diagram illustrating a state after the numerical value input button 314 is selected.
- the glass 200 displays the numeric keypad image 318 on the display 204 so that the numeric keypad image 318 is superimposed on the left arm 50.
- the user inputs the numerical value of the meter by tapping each key on the numeric keypad image 318.
- the memo button 316 is an input operation for inputting a memo.
- FIG. 29 is a diagram illustrating a state after the memo button 316 is selected.
- the glass 200 displays the keyboard image 320 on the display 204 so that the keyboard image 320 is superimposed on the left arm 50.
- the user inputs a memo by tapping each key of the keyboard image 320.
- the input operation for the current inspection item is completed, and the operation proceeds to the next inspection item.
- the user also performs the various input operations described above for the next inspection item. In this way, the user performs an input operation on each inspection item.
- the input device 2000 includes a second display control unit 2140 for displaying the guide 80.
- FIG. 30 is a block diagram illustrating an input device 2000 having the second display control unit 2140.
- the position calculation unit 2020 acquires the captured images 22 in time series regardless of whether the sensor detects vibration or the like, and is included in each captured image 22. Each marker position is calculated.
- the second display control unit 2140 sequentially displays the guides 80 corresponding to the positions of the markers arranged in time series.
- the display position of the guide 80 is a position corresponding to the input position 60 calculated from the marker position by the input recognition unit 2040.
- the second display control unit 2140 uses the same method as the input recognition unit 2040 to calculate the input position 60 from the marker position, and guides 80 from the positions of the markers calculated by the position calculation unit 2020. The display position of is calculated.
- the second display control unit 2140 displays the guide 80 at the display position on the captured image 22 so that the guide 80 is superimposed on the captured image 22.
- the second display control unit 2140 displays the display screen corresponding to the display position on the captured image 22. The upper position is calculated, and a guide is displayed at that position.
- the correspondence between the coordinates on the captured image 22 and the coordinates on the display screen includes various parameters relating to the camera 20 (view angle and focal length), the positional relationship between the display screen and the camera 20, and the like. Determined based on This correspondence relationship may be calculated by the second display control unit 2140 using these parameters or the like, or may be predetermined as a set value.
- the second display control unit 2140 displays a guide 80 representing the input position 60 on the display screen.
- the display representing the input position 60 is an arbitrary mark such as a point, a cross point, a cross mark, or a circle mark having the input position 60 as the center position.
- the second display control unit 2140 displays a guide 80 representing the region 62 corresponding to the input position 60 on the display screen.
- the guide display representing the region 62 is a frame line representing the region 62, for example.
- the frame line can be an arbitrary line such as a solid line, a dotted line, or a one-dot chain line.
- the guide display representing the area 62 may indicate only the apex of the area 62.
- the vertex can be represented by an arbitrary mark such as a point, a cross point, a cross mark, or a circle mark.
- the guide display representing the region 62 may be both a frame line and a mark representing the vertex.
- the color of the guide 80 can be any color.
- the inspection work is targeted, but the application range of the input device 2000 is not limited to the inspection work.
- the input device 2000 can be used for device assembly work, nursing work by a nurse, and other various work records.
- Position calculating means for calculating the position of the marker at the timing based on the detection result of the sensor attached to the user's body, using a captured image including the marker;
- Input recognition means for recognizing an input designating a location on the captured image based on the calculated marker position; The specified location is a location away from the marker;
- the input device wherein the marker is attached to the user's body or is a part of the user's body.
- the input recognition means recognizes an area at a position away from the calculated marker position as an input, 1. having extraction means for extracting a partial image included in the region from the captured image; The input device described in 1. 3.
- the display control unit displays the partial image extracted from the captured image on the display screen so as to be superimposed on another captured image.
- the display screen is a display screen of a head-mounted display or a projection surface of a projector. Or 4.
- the display control means displays a display representing the designated place on the display screen; To 5.
- Movement detection means for detecting movement of the marker at a time including a timing based on a detection result of a sensor attached to the body of the user;
- second input recognition means for recognizing the detected movement of the marker as an input operation.
- An input method performed by a computer A position calculating step of calculating the position of the marker in a period determined based on the detection result of the sensor attached to the user's body, using a captured image including the marker; An input recognition step for recognizing an input designating a location on the captured image based on the calculated marker position; The specified location is a location away from the marker; The input method, wherein the marker is attached to a user's body or is a part of the user's body. 9. In the input recognition step, an area located away from the calculated marker position is recognized as an input, 7. an extraction step of extracting a partial image included in the region from the captured image; The input method described in. 10. 8.
- the display control step the partial image extracted from the captured image is superimposed on another captured image and displayed on the display screen.
- the display screen is a display screen of a head mounted display or a projection surface of a projector. Or 11.
- the input method described in. 13. 9.
- a display representing the designated place is displayed on the display screen.
- the input method according to any one of the above.
- the input method according to any one of the above.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Optics & Photonics (AREA)
- Geometry (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
図1は、実施形態1に係る入力装置2000を例示するブロック図である。図1において、各ブロックは、ハードウエア単位の構成ではなく、機能単位の構成を表している。
入力装置2000の各機能構成部は、各機能構成部を実現するハードウエア(例:ハードワイヤードされた電子回路など)で実現されてもよいし、ハードウエアとソフトウエアとの組み合わせ(例:電子回路とそれを制御するプログラムの組み合わせなど)で実現されてもよい。以下、入力装置2000の各機能構成部がハードウエアとソフトウエアとの組み合わせで実現される場合について、さらに説明する。
図5は、実施形態1の入力装置2000によって実行される処理の流れを例示するフローチャートである。位置算出部2020はセンサの検出結果を把握する(S102)。位置算出部2020は撮像画像22を取得する(S104)。位置算出部2020は、撮像画像22に含まれるマーカの位置を算出する(106)。入力認識部2040は、算出されたマーカの位置に基づいて、そのマーカから離れた撮像画像22上の場所を指定する入力を認識する(S108)。
ユーザに身に付けられるセンサは、ユーザによる入力操作のタイミングを把握するために利用できる任意のセンサであり、前述した振動センサに限定されない。例えばセンサは、加速度センサ、振動スイッチ、圧力センサ、又は静電容量センサなどでもよい。加速度センサや振動スイッチは、振動センサと同様に、例えばデバイス10の内部に設けられる。圧力センサや静電容量センサは、例えばデバイス10のタッチパネルに設けられる。また例えば、圧力センサや静電容量センサは、ユーザの腕や手などに貼られたり巻き付けられたりするシートなどに設けられてもよい。また例えば、圧力センサや静電容量センサは、ユーザの衣服(袖など)に設けられてもよい。
<<センサの検出結果を把握する方法>>
位置算出部2020は、入力装置2000のユーザの身体に取り付けられたセンサの検出結果を把握する(S102)。位置算出部2020がセンサの検出結果を把握する方法は様々である。以下、その方法を例示する。なお以下では、センサが図2で説明したセンサ12(振動センサ)である場合を例として説明する。
例えば位置算出部2020は、センサ12を内蔵するデバイス10と無線通信を行うことにより、センサ12の検出結果を示す情報を取得することで、センサ12の検出結果を把握する。例えば、デバイス10は、センサ12が所定の大きさ以上の振動を検出したタイミングで、位置算出部2020に対して所定の信号を送信する。この場合、位置算出部2020は、この所定の信号を受信することにより、「センサ12によって振動が検出された」というセンサ12の検出結果を取得することができる。
デバイス10は、センサ12による振動の検出に応じてデバイス10の外観を変化させてもよい。この場合、位置算出部2020は、カメラ20によって生成された画像を用いてこの所定の変化を検出することで、センサ12の検出結果(センサ12によって振動が検出されたこと)を把握する。
位置算出部2020は撮像画像22を取得する(S104)。位置算出部2020が撮像画像22を取得する方法は様々である。例えば位置算出部2020は、カメラ20から撮像画像22を取得する。この場合、入力装置2000とカメラ20とは通信可能に接続されている。
位置算出部2020は、撮像画像22に含まれるマーカの位置を算出する(S106)。マーカの位置の算出には、センサの検出結果に基づくタイミングにおけるマーカが含まれる撮像画像22が利用される。以下、「センサの検出結果に基づくタイミング」をセンサタイミングと表記する。
マーカは、撮像画像22において少なくともその位置が特定できる任意のマーカである。例えばマーカは、3次元の座標系の決定に利用できるマーカである。3次元の座標系の決定に利用できるマーカは、例えば AR(Augmented Reality)マーカである。ただし、3次元の座標系の決定に利用できるマーカは、参照方向によらず、或る基準点からの相互に直交する3つの方向を一定に得ることができるものであればよく、AR マーカには限定されない。
位置算出部2020は、取得した撮像画像22からマーカを検出し、そのマーカの位置を算出する。例えば、位置算出部2020は、マーカに関する形状、サイズ、色などについての情報を予め保持しており、このような既知の情報を用いて、取得された撮像画像22からマーカを検出し、その位置を算出する。ここで、画像内から所定の物を検出し、さらにその物の画像内における位置を算出する技術は既知の技術であるため、この技術の詳細については省略する。
入力認識部2040は、算出されたマーカの位置に基づいて、そのマーカから離れた撮像画像22上の場所を指定する入力を認識する(S108)。前述したように、入力によって指定される場所は、撮像画像22上の一点であってもよいし、撮像画像22上の領域であってもよい。
領域62の位置は、入力位置60によって定まる。例えば入力認識部2040は、中心や1つの頂点の位置が入力位置60で定まる領域を領域62とする。
図10は、実施形態2に係る入力装置2000を例示するブロック図である。図10において、各ブロックは、ハードウエア単位の構成ではなく、機能単位の構成を表している。特に説明する場合を除き、実施形態2の入力装置2000は、実施形態1の入力装置2000と同様の機能を有する。
実施形態2の入力装置2000は、実施形態1の入力装置2000と同様に、計算機1000で実現される。実施形態2の計算機1000のストレージ1080は、上述した入力装置2000の各機能を実現するプログラムモジュールをさらに有する。
本実施形態の入力装置2000によれば、ユーザの身体に取り付けられたマーカを用いるという、ユーザにとって直感的な方法で、撮像画像から部分画像を抽出する領域を指定できる。よって、撮像画像からその一部を抽出したい場合に、ユーザが直感的に容易に抽出したい部分を指定できる。なお、撮像画像から部分画像を抽出することには、撮像画像の内の特に注目したい部分について、その部分の保存や表示を行ったり、その部分について画像解析を行ったりすることができるなど、様々な利点がある。
図12は、実施形態3に係る入力装置2000を例示するブロック図である。図12において、各ブロックは、ハードウエア単位の構成ではなく、機能単位の構成を表している。実施形態3の入力装置2000は、特に説明する点を除き、実施形態2の入力装置2000と同様の機能を有する。
図13は、実施形態3の入力装置2000を実現する計算機1000のハードウエア構成を例示する図である。この計算機1000の構成は、実施形態1の入力装置2000を実現する計算機1000の構成と同様である。ただし、実施形態3の計算機1000が有するストレージ1080は、本実施形態の入力装置2000が有する各機能を実現するためのプログラムモジュールをさらに有する。
本実施形態の入力装置2000によれば、撮像画像22から抽出された部分画像24が、表示画面上の所定位置に固定して表示される。そのため、ユーザがマーカを動かしても(撮像画像22に含まれるマーカの位置が変化しても)、部分画像24が表示される表示画面上の位置が変わらない。図14は、マーカの位置が変わっても部分画像24の表示位置が変わらない様子を例示する図である。図14(a)はマーカを動かす前の様子を表し、図14(b)はマーカを動かした後の様子を表す。
図16は、実施形態4に係る入力装置2000を例示するブロック図である。図16において、各ブロックは、ハードウエア単位の構成ではなく、機能単位の構成を表している。特に説明する場合を除き、本実施形態の入力装置2000は、実施形態3の入力装置2000と同様の機能を有する。
動き検出部2100は、センサ12の検出結果を取得する。動き検出部2100がセンサ12の検出結果を把握する方法は、位置算出部2020がセンサの検出結果を把握する方法と同様である。
第2入力認識部2120は、検出したマーカの動きに基づいて入力を認識する。第2入力認識部2120は、1)検出したマーカの動きに基づく形状を入力として認識としてもよいし、2)検出したマーカの動きに対応する所定の入力操作を入力として認識してもよい。
図20は、マーカの動き40によって定まる形状を入力として認識する様子を例示する図である。図20(a)では、第2入力認識部2120は、マーカの動き40-Aが表す形状41や、動き40-Bが表す形状42を入力として認識する。例えばこの入力は、ユーザが手書き入力を行う際に利用される。
検出したマーカの動きに対応する所定の入力操作は、いわゆるジェスチャ入力による入力操作である。図21は、ジェスチャ入力を例示する図である。図21(a)はフリック動作を表し、図21(b)は回転動作を表している。
第2入力認識部2120は、1)マーカを動かす動作が行われた位置に関係なく、そのマーカの動きのみを入力として認識してもよいし、2)マーカの動きと、そのマーカを動かす動作が行われた位置との組み合わせを入力として認識してもよい。前者の場合、マーカを動かす動作が撮像画像22上のどの場所で行われても、同じマーカの動きは同じ入力を表す。一方後者の場合、マーカを動かす動作が撮像画像22上のどの場所で行われたかが意味を持つこととなる。
実施形態4の入力装置2000は、入力認識部2040と第2入力認識部2120のいずれかによってユーザの入力を認識する。入力装置2000が、入力認識部2040と第2入力認識部2120のどちらによってユーザの入力を認識するかを決定する方法は様々である。
本実施形態によれば、ユーザは、マーカから離れた場所を指定する入力操作(入力認識部2040によって認識される入力操作)に加え、マーカの動きを用いた入力操作(第2入力認識部2120によって認識される入力操作)を行えるようになる。よって、ユーザによる入力操作のバリエーションが増え、入力装置2000の利便性が向上する。
以下、上述した入力装置2000を、より具体的な実施例に沿って説明する。なお、この実施例は入力装置2000の利用方法の一例であり、入力装置2000の利用方法を限定するものではない。
実施例の入力装置2000は、ガイド80を表示するために、第2表示制御部2140を有する。図30は、第2表示制御部2140を有する入力装置2000を例示するブロック図である。
1. ユーザの身体に取り付けられたセンサの検出結果に基づくタイミングにおけるマーカの位置を、前記マーカが含まれる撮像画像を用いて算出する位置算出手段と、
前記算出されたマーカの位置に基づいて、前記撮像画像上の場所を指定する入力を認識する入力認識手段と、を有し、
前記指定される場所は、前記マーカから離れた場所であり、
前記マーカは、ユーザの身体に取り付けられているか、又は前記ユーザの身体の一部である、入力装置。
2. 前記入力認識手段は、前記算出されたマーカの位置から離れた位置にある領域を入力として認識し、
前記撮像画像から、前記領域に含まれる部分画像を抽出する抽出手段を有する、1.に記載の入力装置。
3. 前記部分画像を表示画面の所定位置に表示する表示制御手段を有する、2.に記載の入力装置。
4. 前記表示制御手段は、前記撮像画像から抽出された前記部分画像を、他の撮像画像に重畳させて前記表示画面に表示する、3.に記載の入力装置。
5. 前記表示画面は、ヘッドマウントディスプレイの表示画面又はプロジェクタの投影面である、3.又は4.に記載の入力装置。
6. 前記表示制御手段は、前記指定される場所を表す表示を前記表示画面に表示する、3.乃至5.いずれか一つに記載の入力装置。
7. 前記ユーザの身体に取り付けられたセンサの検出結果に基づくタイミングを含む時間における前記マーカの動きを検出する動き検出手段と、
前記検出したマーカの動きを入力操作として認識する第2入力認識手段と、を有する1.乃至6.いずれか一つに記載の入力装置。
ユーザの身体に取り付けられたセンサの検出結果に基づいて定まる期間におけるマーカの位置を、前記マーカが含まれる撮像画像を用いて算出する位置算出ステップと、
前記算出されたマーカの位置に基づいて、前記撮像画像上の場所を指定する入力を認識する入力認識ステップと、を有し、
前記指定される場所は、前記マーカから離れた場所であり、
前記マーカは、ユーザの身体に取り付けられているか、又は前記ユーザの身体の一部である、入力方法。
9. 前記入力認識ステップにおいて、前記算出されたマーカの位置から離れた位置にある領域を入力として認識し、
前記撮像画像から、前記領域に含まれる部分画像を抽出する抽出ステップを有する、8.に記載の入力方法。
10. 前記部分画像を表示画面の所定位置に表示する表示制御ステップを有する、9.に記載の入力方法。
11. 前記表示制御ステップにおいて、前記撮像画像から抽出された前記部分画像を、他の撮像画像に重畳させて前記表示画面に表示する、10.に記載の入力方法。
12. 前記表示画面は、ヘッドマウントディスプレイの表示画面又はプロジェクタの投影面である、10.又は11.に記載の入力方法。
13. 前記表示制御ステップにおいて、前記指定される場所を表す表示を前記表示画面に表示する、10.乃至12.いずれか一つに記載の入力方法。
14. 前記ユーザの身体に取り付けられたセンサの検出結果に基づくタイミングを含む時間における前記マーカの動きを検出する動き検出ステップと、
前記検出したマーカの動きを入力操作として認識する第2入力認識ステップと、を有する8.乃至13.いずれか一つに記載の入力方法。
Claims (9)
- ユーザの身体に取り付けられたセンサの検出結果に基づくタイミングにおけるマーカの位置を、前記マーカが含まれる撮像画像を用いて算出する位置算出手段と、
前記算出されたマーカの位置に基づいて、前記撮像画像上の場所を指定する入力を認識する入力認識手段と、を有し、
前記指定される場所は、前記マーカから離れた場所であり、
前記マーカは、ユーザの身体に取り付けられているか、又は前記ユーザの身体の一部である、入力装置。 - 前記入力認識手段は、前記算出されたマーカの位置から離れた位置にある領域を入力として認識し、
前記撮像画像から、前記領域に含まれる部分画像を抽出する抽出手段を有する、請求項1に記載の入力装置。 - 前記部分画像を表示画面の所定位置に表示する表示制御手段を有する、請求項2に記載の入力装置。
- 前記表示制御手段は、前記撮像画像から抽出された前記部分画像を、他の撮像画像に重畳させて前記表示画面に表示する、請求項3に記載の入力装置。
- 前記表示画面は、ヘッドマウントディスプレイの表示画面又はプロジェクタの投影面である、請求項3又は4に記載の入力装置。
- 前記表示制御手段は、前記指定される場所を表す表示を前記表示画面に表示する、請求項3乃至5いずれか一項に記載の入力装置。
- 前記ユーザの身体に取り付けられたセンサの検出結果に基づくタイミングを含む時間における前記マーカの動きを検出する動き検出手段と、
前記検出したマーカの動きを入力操作として認識する第2入力認識手段と、を有する請求項1乃至6いずれか一項に記載の入力装置。 - コンピュータによって実行される入力方法であって、
ユーザの身体に取り付けられたセンサの検出結果に基づいて定まる期間におけるマーカの位置を、前記マーカが含まれる撮像画像を用いて算出する位置算出ステップと、
前記算出されたマーカの位置に基づいて、前記撮像画像上の場所を指定する入力を認識する入力認識ステップと、を有し、
前記指定される場所は、前記マーカから離れた場所であり、
前記マーカは、ユーザの身体に取り付けられているか、又は前記ユーザの身体の一部である、入力方法。 - 請求項8に記載の入力方法に係る各ステップをコンピュータに実行させるプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017543168A JP6390799B2 (ja) | 2015-09-28 | 2016-09-20 | 入力装置、入力方法、及びプログラム |
CN201680053037.8A CN108027654B (zh) | 2015-09-28 | 2016-09-20 | 输入设备、输入方法和程序 |
US15/759,323 US10234955B2 (en) | 2015-09-28 | 2016-09-20 | Input recognition apparatus, input recognition method using maker location, and non-transitory computer-readable storage program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2015-190239 | 2015-09-28 | ||
JP2015190239 | 2015-09-28 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017057107A1 true WO2017057107A1 (ja) | 2017-04-06 |
Family
ID=58427555
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2016/077730 WO2017057107A1 (ja) | 2015-09-28 | 2016-09-20 | 入力装置、入力方法、及びプログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US10234955B2 (ja) |
JP (1) | JP6390799B2 (ja) |
CN (1) | CN108027654B (ja) |
WO (1) | WO2017057107A1 (ja) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019046455A (ja) * | 2018-07-06 | 2019-03-22 | 株式会社コロプラ | 仮想体験を提供するためにコンピュータで実行される方法、プログラム及びコンピュータ |
WO2023067836A1 (ja) * | 2021-10-18 | 2023-04-27 | 東芝デジタルソリューションズ株式会社 | コンテンツ表示装置、コンテンツ表示プログラム、コンテンツ表示方法、及びコンテンツ表示システム |
JP2023060799A (ja) * | 2021-10-18 | 2023-04-28 | 東芝デジタルソリューションズ株式会社 | コンテンツ表示装置、コンテンツ表示プログラム、コンテンツ表示方法、及びコンテンツ表示システム |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017057107A1 (ja) * | 2015-09-28 | 2017-04-06 | 日本電気株式会社 | 入力装置、入力方法、及びプログラム |
CN108027655A (zh) * | 2016-02-25 | 2018-05-11 | 日本电气株式会社 | 信息处理***、信息处理设备、控制方法和程序 |
US11487353B2 (en) * | 2016-11-14 | 2022-11-01 | Logitech Europe S.A. | Systems and methods for configuring a hub-centric virtual/augmented reality environment |
US10545578B2 (en) * | 2017-12-22 | 2020-01-28 | International Business Machines Corporation | Recommending activity sensor usage by image processing |
US10546426B2 (en) * | 2018-01-05 | 2020-01-28 | Microsoft Technology Licensing, Llc | Real-world portals for virtual reality displays |
US10839603B2 (en) | 2018-04-30 | 2020-11-17 | Microsoft Technology Licensing, Llc | Creating interactive zones in virtual environments |
NL2022598B1 (en) * | 2019-02-19 | 2020-08-31 | Easee Health B V | A method of performing an eye examination test for examining eyes of a user as well as a corresponding computing device. |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000102036A (ja) * | 1998-09-22 | 2000-04-07 | Mr System Kenkyusho:Kk | 複合現実感提示システム、複合現実感提示方法、マン・マシーンインタフェース装置、およびマン・マシーンインタフェース方法 |
JP2011257943A (ja) * | 2010-06-08 | 2011-12-22 | Canon Inc | ジェスチャ操作入力装置 |
JP2013114467A (ja) * | 2011-11-29 | 2013-06-10 | Nikon Corp | 表示システム、表示方法およびプログラム |
JP2013190941A (ja) * | 2012-03-13 | 2013-09-26 | Nikon Corp | 情報入出力装置、及び頭部装着表示装置 |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5799099A (en) * | 1993-02-12 | 1998-08-25 | George S. Allen | Automatic technique for localizing externally attached fiducial markers in volume images of the head |
SE507680C2 (sv) * | 1997-01-13 | 1998-07-06 | Qualisys Ab | Metod och anordning för att bestämma positionen av ett objekt |
US9619143B2 (en) * | 2008-01-06 | 2017-04-11 | Apple Inc. | Device, method, and graphical user interface for viewing application launch icons |
US8269842B2 (en) * | 2008-06-11 | 2012-09-18 | Nokia Corporation | Camera gestures for user interface control |
CN102402680B (zh) * | 2010-09-13 | 2014-07-30 | 株式会社理光 | 人机交互***中手部、指示点定位方法和手势确定方法 |
EP2620849B1 (en) * | 2010-09-22 | 2019-08-28 | Shimane Prefectural Government | Operation input apparatus, operation input method, and program |
EP2761362A4 (en) | 2011-09-26 | 2014-08-06 | Microsoft Corp | VIDEO DISPLAY CHANGE BASED ON SENSOR INPUT FOR TRANSPARENT EYE DISPLAY |
CN103135755B (zh) * | 2011-12-02 | 2016-04-06 | 深圳泰山在线科技有限公司 | 交互***及方法 |
JP2013200793A (ja) * | 2012-03-26 | 2013-10-03 | Sony Corp | 情報処理装置、情報処理方法およびプログラム |
JP2014092940A (ja) * | 2012-11-02 | 2014-05-19 | Sony Corp | 画像表示装置及び画像表示方法、並びにコンピューター・プログラム |
JP5962547B2 (ja) * | 2013-03-08 | 2016-08-03 | ソニー株式会社 | 情報処理装置、情報処理方法およびプログラム |
US10025486B2 (en) * | 2013-03-15 | 2018-07-17 | Elwha Llc | Cross-reality select, drag, and drop for augmented reality systems |
ITTO20130541A1 (it) * | 2013-06-28 | 2014-12-29 | St Microelectronics Srl | Dispositivo a semiconduttore integrante un partitore resistivo e procedimento di fabbricazione di un dispositivo a semiconduttore |
US20150186708A1 (en) * | 2013-12-31 | 2015-07-02 | Sagi Katz | Biometric identification system |
CN104199602B (zh) * | 2014-08-26 | 2018-11-09 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
WO2017057107A1 (ja) * | 2015-09-28 | 2017-04-06 | 日本電気株式会社 | 入力装置、入力方法、及びプログラム |
US10168769B2 (en) * | 2015-09-28 | 2019-01-01 | Nec Corporation | Input apparatus, input method, and program |
US10296101B2 (en) * | 2016-02-08 | 2019-05-21 | Nec Corporation | Information processing system, information processing apparatus, control method, and program |
JP6256497B2 (ja) * | 2016-03-04 | 2018-01-10 | 日本電気株式会社 | 情報処理システム、情報処理装置、制御方法、及びプログラム |
-
2016
- 2016-09-20 WO PCT/JP2016/077730 patent/WO2017057107A1/ja active Application Filing
- 2016-09-20 JP JP2017543168A patent/JP6390799B2/ja active Active
- 2016-09-20 US US15/759,323 patent/US10234955B2/en active Active
- 2016-09-20 CN CN201680053037.8A patent/CN108027654B/zh active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000102036A (ja) * | 1998-09-22 | 2000-04-07 | Mr System Kenkyusho:Kk | 複合現実感提示システム、複合現実感提示方法、マン・マシーンインタフェース装置、およびマン・マシーンインタフェース方法 |
JP2011257943A (ja) * | 2010-06-08 | 2011-12-22 | Canon Inc | ジェスチャ操作入力装置 |
JP2013114467A (ja) * | 2011-11-29 | 2013-06-10 | Nikon Corp | 表示システム、表示方法およびプログラム |
JP2013190941A (ja) * | 2012-03-13 | 2013-09-26 | Nikon Corp | 情報入出力装置、及び頭部装着表示装置 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019046455A (ja) * | 2018-07-06 | 2019-03-22 | 株式会社コロプラ | 仮想体験を提供するためにコンピュータで実行される方法、プログラム及びコンピュータ |
JP7166813B2 (ja) | 2018-07-06 | 2022-11-08 | 株式会社コロプラ | 仮想体験を提供するためにコンピュータで実行される方法、プログラム及びコンピュータ |
WO2023067836A1 (ja) * | 2021-10-18 | 2023-04-27 | 東芝デジタルソリューションズ株式会社 | コンテンツ表示装置、コンテンツ表示プログラム、コンテンツ表示方法、及びコンテンツ表示システム |
JP2023060799A (ja) * | 2021-10-18 | 2023-04-28 | 東芝デジタルソリューションズ株式会社 | コンテンツ表示装置、コンテンツ表示プログラム、コンテンツ表示方法、及びコンテンツ表示システム |
JP7330323B2 (ja) | 2021-10-18 | 2023-08-21 | 東芝デジタルソリューションズ株式会社 | コンテンツ表示装置、コンテンツ表示プログラム、コンテンツ表示方法、及びコンテンツ表示システム |
Also Published As
Publication number | Publication date |
---|---|
JPWO2017057107A1 (ja) | 2018-06-14 |
US20180260032A1 (en) | 2018-09-13 |
CN108027654A (zh) | 2018-05-11 |
CN108027654B (zh) | 2021-01-12 |
JP6390799B2 (ja) | 2018-09-19 |
US10234955B2 (en) | 2019-03-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6390799B2 (ja) | 入力装置、入力方法、及びプログラム | |
JP6344530B2 (ja) | 入力装置、入力方法、及びプログラム | |
JP6288372B2 (ja) | インタフェース制御システム、インタフェース制御装置、インタフェース制御方法、及びプログラム | |
JP5412227B2 (ja) | 映像表示装置、および、その表示制御方法 | |
KR101844390B1 (ko) | 사용자 인터페이스 제어를 위한 시스템 및 기법 | |
JP5802667B2 (ja) | ジェスチャ入力装置およびジェスチャ入力方法 | |
JP5900393B2 (ja) | 情報処理装置、操作制御方法及びプログラム | |
JP6057396B2 (ja) | 3次元ユーザインタフェース装置及び3次元操作処理方法 | |
JP6350772B2 (ja) | 情報処理システム、情報処理装置、制御方法、及びプログラム | |
US20180315246A1 (en) | Information processing device, information processing method, and program | |
KR101343748B1 (ko) | 포인터를 표시하지 않는 투명 디스플레이 가상 터치 장치 | |
US10713488B2 (en) | Inspection spot output apparatus, control method, and storage medium | |
JP2012238293A (ja) | 入力装置 | |
JP6341343B2 (ja) | 情報処理システム、情報処理装置、制御方法、及びプログラム | |
JP2016126687A (ja) | ヘッドマウントディスプレイ、操作受付方法および操作受付プログラム | |
KR20160062906A (ko) | 고글형 디바이스를 위한 증강현실 키보드 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16851281 Country of ref document: EP Kind code of ref document: A1 |
|
DPE2 | Request for preliminary examination filed before expiration of 19th month from priority date (pct application filed from 20040101) | ||
WWE | Wipo information: entry into national phase |
Ref document number: 15759323 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2017543168 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16851281 Country of ref document: EP Kind code of ref document: A1 |