WO2014027666A1 - 入力装置、入力方法、及び記憶媒体 - Google Patents
入力装置、入力方法、及び記憶媒体 Download PDFInfo
- Publication number
- WO2014027666A1 WO2014027666A1 PCT/JP2013/071893 JP2013071893W WO2014027666A1 WO 2014027666 A1 WO2014027666 A1 WO 2014027666A1 JP 2013071893 W JP2013071893 W JP 2013071893W WO 2014027666 A1 WO2014027666 A1 WO 2014027666A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- hand
- movement
- shape
- user
- input
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04842—Selection of displayed objects or displayed text elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Definitions
- the present invention relates to an input device, an input method, and a storage medium.
- Patent Document 1 discloses an image recognition device that determines a user operation based on the movement of a user's hand operating a virtual operation surface formed by a three-dimensional display.
- Patent Document 2 discloses a display control apparatus that determines a method for arranging images displayed on a display based on a photographed user's gesture.
- Patent Document 3 discloses an information processing apparatus that determines whether a user's touch operation is a pressing operation (tap operation) or a gesture operation based on a movement amount of the touch operation of the touch panel.
- a captured image is associated with an operation screen in which a selectable region (for example, a button or icon display region) that is a target of a selection operation is defined, and the user in the captured image
- a selectable region for example, a button or icon display region
- An apparatus for recognizing an operation for selecting a selectable region by a user based on a hand movement (for example, stop of a hand for a certain period of time at a position corresponding to the selectable region). is there.
- selection operation for selecting a selectable region by a user based on a hand movement (for example, stop of a hand for a certain period of time at a position corresponding to the selectable region).
- the input device can easily determine whether the touch operation is a selection operation (tap operation or the like) or a gesture operation based only on the movement amount information of the user's hand captured by the touch panel. Can be distinguished.
- the movement of the user's hand moving to the selectable area is captured by the camera. That is, it is not possible to determine the hand movement and the gesture operation prior to the selection operation based on the presence or absence of detection. For this reason, the method disclosed in the cited document 3 cannot discriminate between the selection operation to the selectable region and the gesture operation, and there is a problem that the variation of the operation that can be discriminated by the input device cannot be increased.
- the present invention has been made in view of such a problem, and an object thereof is to provide an input device, an input method, and a storage medium that can discriminate a gesture operation in addition to a selection operation.
- An input device includes: An input device that discriminates a user's input based on a user's hand movement in a captured image associated with an operation screen in which a selectable area is defined, Input determination for determining whether a user input is a selection operation of the selectable region or a gesture operation different from the selection operation based on a change in the shape of the hand in the plurality of captured images caused by the movement of the hand Means are provided.
- the input method is: An input method for determining a user's input based on a user's hand movement in a captured image associated with an operation screen on which a selectable area to be selected is displayed, Determining whether a user input is a selection operation of the selectable area or a gesture operation different from the selection operation based on a change in the shape of the hand in the plurality of captured images caused by the movement of the hand.
- a storage medium is: A computer-readable non-transitory storage medium for storing a program, wherein the program is based on a user's hand movement in a captured image associated with an operation screen on which a selectable area is displayed.
- an input device an input method, and a storage medium that determine a gesture operation in addition to a selection operation.
- FIG. 4 is a diagram for explaining a gesture operation.
- FIG. 4 is a diagram for explaining a gesture operation.
- FIG. 4 is a diagram for explaining a selection operation. It is a block diagram which shows the internal structure of the input device shown in FIG. It is a functional block diagram for demonstrating the function of the control part shown in FIG.
- FIG. 1 It is a figure for demonstrating a hand area
- FIG. 1 It is a figure for demonstrating a hand area
- FIG. 4 is a diagram showing a change in the shape of a user's hand during a gesture operation.
- FIG. 6 is a diagram for explaining an effect of the input device according to the first embodiment.
- FIG. 5 is a diagram for explaining a FIFO queue prepared in the storage unit illustrated in FIG. 4.
- 10 is a flowchart of input processing according to the second embodiment. It is a figure which shows a mode that the change of a user's hand moving direction is specified in specifying the moving direction and shape change of a user's hand. It is a figure which shows a mode that the change of a user's hand shape is specified in specifying the moving direction and shape change of a user's hand.
- FIG. 10 is a diagram for explaining a modified example of the input device, and is a diagram for explaining a direction comparison map created based on the direction of the selectable region and the position of the user's hand at the start of movement.
- FIG. 4 is a diagram illustrating a state in which a direction difference degree is specified based on a direction comparison map. It is a figure which shows a mode that the movement locus
- the input device 100 is a selectable area in which an input to a user device (for example, an operation command, a command, an icon, etc. for the user device is displayed based on the movement of the user's hand imaged by the camera. Etc.) are provided.
- the input device 100 is a personal computer having an appearance as shown in FIG. First, an overview of processing performed by the input device 100 will be described with reference to FIGS. 2 and 3.
- the input device 100 displays a display unit 120 that displays an operation screen as shown in FIG. 2A in which images (icons and buttons) to be selected are displayed in the selectable area, and an image as shown in FIG. 2B, for example. And an imaging unit 130 for imaging.
- the captured image (hereinafter referred to as “captured image”) is associated with the operation screen as shown in FIG. 2C.
- the input device 100 associates each coordinate point on the captured image with a coordinate point on the operation screen by coordinate transformation such as affine transformation. And if a user's fingertip is reflected in a certain part on a picked-up image, it will accept as operation which points to the part corresponding on an operation picture.
- the selectable area is set on the operation screen according to the state of the application that is the target of the input operation.
- an image of a button that can be pressed, an image of an icon to be selected, or the like may be displayed at the same time.
- the input device 100 communicates with an application, for example, acquires the coordinates of the selectable area, and stores them in the RAM.
- the input device 100 determines whether the user input is a selection operation or a gesture operation different from the selection operation based on the movement of the user's hand in the captured image.
- the “gesture operation” means a predetermined operation such as gesture gestures that are predetermined for device operation, for example, a hand-circulating operation as shown in FIG. 3A, or a finger as shown in FIG. 3B, for example.
- the “selection operation” (also referred to as “pointing operation”) refers to an operation in which the user attempts to select a selectable area displayed on the screen.
- the “selection operation” includes the movement of the user's hand moving to the point area.
- the user's hand moves in a direction where there is a selectable area.
- the user since the user often moves the hand in parallel while looking at the operation screen, for example, as shown in FIG.
- the user in the case of a gesture operation, the user often moves his / her hand so as to wave his / her hand with the wrist as a fulcrum.
- the input device 100 determines whether the user input is a selection operation or a gesture operation based on a change in the shape of the user's hand caused by the movement.
- the input device 100 includes an operation unit 110, a display unit 120, an imaging unit 130, an image storage unit 140, a control unit 150, and a storage unit 160.
- the operation unit 110 includes an input device such as a keyboard.
- the operation unit 110 receives a process start command or the like from the user, and transmits it to the control unit 150 described later.
- the display unit 120 includes a display device such as a liquid crystal display.
- the display unit 120 displays various images according to the control of the control unit 150.
- the imaging unit 130 includes an imaging device such as a CCD (Charge Coupled Device Image Sensor) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera.
- the imaging unit 130 is installed in the upper center of the display unit 120, for example.
- the imaging unit 130 sequentially captures images (including video) of the user's hand, adds a shooting time to each frame, and sequentially transmits the frames to the image storage unit 140.
- the image storage unit 140 includes a video memory such as a RAM (Random Access Memory).
- the image storage unit 140 stores the image data transmitted from the imaging unit 130.
- the control unit 150 includes, for example, an arithmetic device such as a processor, a RAM that can be used as a work area by the arithmetic device, and a ROM (Read Only Memory) that stores data such as programs and initial values.
- the control unit 150 operates according to a program loaded from the ROM to the RAM, and executes various operations including “input processing” described later.
- the control unit 150 operates according to “input processing”, and as illustrated in FIG. 5, the image acquisition unit 151, the movement detection unit 152, the direction comparison unit 153, the shape comparison unit 154, the input determination unit 155, and the transmission unit 156. Function as. Note that these functions will be described later in the description of “input processing”.
- the storage unit 160 includes a storage device capable of reading and writing data such as a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), a flash memory, and a hard disk.
- the storage unit 160 stores various data such as a “gesture discrimination table” and a “command table”.
- the “gesture discrimination table” is information for discriminating which of the plurality of movement patterns stored in advance is the user's gesture, that is, the user's hand trajectory.
- “movement pattern information” is composed of information in which “identification information” and “feature data” are associated with each other.
- Identification information is a movement pattern of the user's hand, and is information (identifier) for uniquely identifying each of a plurality of movement patterns stored in advance.
- identifier information for uniquely identifying each of a plurality of movement patterns stored in advance.
- FIG. 6 as the movement of the user's hand, as shown in FIG. 7A, “round (clockwise)” with respect to the clockwise rotation as shown in FIG. 7A, and with respect to the counterclockwise rotation as shown in FIG. “Circulation (counterclockwise)”, “linear movement (left direction)” with respect to the flicking operation in the left direction as shown in FIG. 7C, and “linear movement (to the left direction)” with respect to the flicking operation in the right direction as shown in FIG. Right direction) ”, unique identification information is assigned to different hand movements.
- the format of the identification information is not limited to a specific format. For example, numerical values such as 1, 2, 3,..., And symbols such as A, B, C.
- Feature data is information for specifying the movement pattern of the user's hand.
- the feature data includes feature amount data obtained by extracting and storing features of a movement locus drawn by a user's hand in advance.
- the feature data is stored in advance in a storage device such as a ROM, and is used for determining a gesture operation by pattern recognition using a movement vector.
- the control unit 150 determines which movement pattern the user's hand movement locus corresponds to by comparing the user's hand locus with the feature data.
- the “command table” is information for determining which of a plurality of operation commands the user's gesture operation corresponds to. As shown in FIG. 6, the command table includes information in which “identification information” and “control command” are associated with each other.
- Control command is control information for controlling the device.
- the “control command” is control information such as playback, stop, fast forward, skip, decision, cancel, volume change, etc., which are input to application software such as music playback software, operating system, or the like.
- the control unit 150 commands the imaging unit 130 to start imaging.
- the imaging unit 130 sequentially transmits the captured images to the image storage unit 140.
- the control unit 150 executes application software and starts “input processing” for inputting various control information to the application software.
- the “input processing” will be described below with reference to the flowchart of FIG.
- the image acquisition unit 151 of the control unit 150 receives from the image storage unit 140 a captured image for the latest predetermined period in which a human hand is captured (for example, as shown in FIG. 9, the shooting time is 1:23:40). (Multiple frames of image data for 20 seconds from 50 to 1:24:00 to 50) are acquired (step S101). That is, the control unit 150 acquires a plurality of captured images with different shooting timings from the image acquisition unit 151.
- the movement detection unit 152 of the control unit 150 extracts a region (hereinafter, referred to as “hand region”) in which the hand of the person with the index finger raised is shown for each of the plurality of frames of captured images.
- the hand region to be extracted may be the shape of the hand itself (for example, a portion PA surrounded by a broken line in FIG. 10), or the smallest rectangular region that includes the hand (for example, surrounded by a broken line in FIG. 10).
- Portion PB portion of the hand itself (for example, a portion PA surrounded by a broken line in FIG. 10), or the smallest rectangular region that includes the hand (for example, surrounded by a broken line in FIG. 10).
- the movement detection unit 152 determines whether or not the movement start is detected based on the change in the position of the hand region in the captured image of each frame (step S103). As a specific example, the movement detection unit 152 compares the positions of the hand areas of the plurality of captured images acquired in step S101, and compares the position of the hand area (the amount of movement of the hand) and the difference in shooting timing (time difference) in each frame. ) To determine the hand movement speed. Then, based on the obtained speed, the user's hand moves from the stopped state (the moving speed of the hand is equal to or lower than the fixed speed) to the moving state (the moving speed of the hand is constant) within the predetermined period in which the image is acquired in step S101.
- step S103 It is determined whether or not there is a time point when the state has shifted to a state above the speed. If there is no transition time point, the movement detection unit 152 determines that the user's hand has not shifted to the movement state (step S103: No), returns the process to step S101, and detects the transition point. Steps S101 to S103 are repeated until. On the other hand, if the movement state has been entered (step S103: Yes), the shooting time (hereinafter referred to as the movement start time T1) at the time of the change to the movement state is stored in a RAM or the like, and the process proceeds to step S104.
- the shooting time T1 hereinafter referred to as the movement start time T1
- the movement detection unit 152 Based on the change in the position of the hand area of each frame after the start of movement, the movement detection unit 152 includes a time point at which the user's hand has shifted from the moving state to the stopped state in the captured image after the movement start time T1. (Step S104).
- the movement detecting unit 152 changes the moving state from the moving state to the stopped state, contrary to step S103, based on the moving speed of the hand determined based on the difference in the position of the hand region (the amount of movement of the hand) and the difference in the photographing timing (time difference). Determine if there is a transition point in time.
- step S104 when the user's hand has not shifted to the stop state (step S104: No), the process returns to step S101, and steps S101 to S104 are repeated.
- the shooting time at the time of shifting to the movement stop state (hereinafter referred to as movement stop time T2) is stored in the RAM or the like, and the process proceeds to step S105.
- the movement detection unit 152 acquires the captured image at the movement start time T1 and the captured image at the movement stop time T2 from the image storage unit 140, and stores them in the RAM (step S105). For example, if 1: 23: 45: 67 is detected as the movement start time T1 in step S103 and 1: 23: 46: 67 is detected as the movement stop time T2 in step S104, the two frames (1:23) are detected. The captured image of a frame of 45 minutes 67 minutes and a frame of 1:23 minutes 46 seconds 67) is stored in the RAM.
- the direction comparison unit 153 of the control unit 150 identifies the direction of the selectable region with respect to the position of the user's hand at the movement start time T1 based on the captured image at the movement start time T1 of the user's hand (step S106).
- This direction can be obtained based on, for example, the coordinates of the center of gravity of the selectable area recorded in the predetermined area of the RAM and the center of gravity of the hand area extracted in step S102.
- the direction is specified for each selectable area. For example, as shown in FIG.
- the direction comparison unit 153 connects the center-of-gravity position G1 of the hand area at the movement start time T1 and the centers (C1 to C4) of the selectable areas. (For example, a clockwise angle from a predetermined reference direction such as the upward direction in the figure, and the same applies to the respective drawings for explaining the angles on the operation screen hereinafter) as the direction of the selectable area. Good.
- the direction comparison unit 153 compares the direction of the selectable area acquired in step S106 with the moving direction of the user's hand. Then, the comparison result is acquired as a “direction difference degree” indicating the degree of direction difference. Note that the movement direction of the user's hand can be obtained based on the position of the hand region in the frame at the movement start time T1 and the movement stop time T2 stored in step S105, for example. When there are a plurality of selectable areas, the direction difference is obtained for each selectable area (step S107).
- the direction comparison unit 153 may determine the angle of a line connecting the gravity center position G1 of the user's hand at the movement start time T1 and the gravity center position G2 of the user's hand at the movement stop time T2 (for example, as illustrated in FIG. The angle when the middle-up direction is 0 ° is specified. Then, as shown in FIG. 12B, the difference between the specified angle and the direction angle of the selectable area is calculated, and the difference having the smallest value among the calculated differences (17 ° in the example of FIG. 12B) is set as the direction. Get as the degree of difference.
- the user moves to any of the selectable areas from the movement start point (in this example, movement start point T1) to the movement stop point (in this example, movement stop point T2). Move your hand toward.
- the movement direction is independent of the direction from the movement start point toward the selectable area. For this reason, the direction difference indicating whether or not the user's hand is moving toward one of the selectable areas set on the operation screen is whether or not the movement of the hand is part of the selection operation. This is a coefficient indicating the certainty.
- the degree of direction difference is smaller. That is, the smaller the direction difference, the higher the probability that the user's hand movement is part of the selection operation. Conversely, the greater the degree of direction difference, the higher the probability that the user's hand movement is for a gesture operation rather than a selection operation.
- the shape comparison unit 154 of the control unit 150 compares the shape of the user's hand when the movement is started with the shape of the user's hand when the movement is stopped. Then, the comparison result is acquired as “shape difference” indicating the degree of change in shape (step S108). For example, the shape comparison unit 154 compares the positional relationship between the gravity center position G1 of the hand region at the movement start time T1 and the fingertip F1 and the positional relationship between the gravity center position G2 of the hand region at the movement stop time T2 and the fingertip F2. To calculate the shape difference.
- the shape comparison unit 154 determines the angle of the line connecting the centroid position G1 of the hand region at the movement start time T1 and the fingertip F1, the centroid position G2 of the hand region at the movement stop time T2, and the fingertip F2.
- the difference between the angles of the lines connecting the lines is calculated as the shape difference degree.
- the shape difference calculated in this way is an indicator of the probability that the user's hand is moving as part of a predetermined gesture operation, unlike the movement to any of the selectable areas. This is because, in general, when a hand is moved as part of the selection operation, the user moves the hand without deforming it.
- the gesture operation as shown in FIG.
- the hand is deformed more than in the case of the selection operation. That is, the greater the degree of shape difference, the greater the probability that the user's hand is moving as part of the gesture operation.
- the input determination unit 155 of the control unit 150 adds the direction difference degree and the shape difference degree with a predetermined weight (step S109). For example, if the direction difference degree acquired in step S107 is D and the shape difference degree acquired in step S108 is S, the weighting coefficients of the direction difference degree D and the shape difference degree S are respectively set as w1 and w2, and the following ( The addition value ADD is obtained by equation (1). Note that the value of each weight is a value obtained in advance by experiments and stored in the ROM as a set value. Or it is good also as a user being able to set according to the habit of own operation. Here, the values of w1 and w2 are both positive.
- the input determining unit 155 determines whether the added value (for example, the added value ADD) is equal to or less than a predetermined threshold (step S110). If it is less than or equal to the predetermined threshold (step S110: Yes), it is determined that the user's hand movement is a selection operation, and the process proceeds to step S111. Otherwise (step S110: No), it is determined that the user's hand movement is a gesture operation, and the input determination unit 155 advances the process to step S112.
- This added value ADD is obtained based on a direction difference degree D indicating the degree to which the user's hand is directed to one of the selectable areas and a shape difference degree S indicating the magnitude of the change in the user's shape.
- a coefficient indicating whether the movement of the user's hand is for a gesture operation or part of a selection operation When w1 and w2 are positive values, the probability of being a gesture operation is higher as ADD is larger. Therefore, it is possible to determine input according to the user's intention by obtaining ADD and determining that it is a gesture operation when the size exceeds a threshold value.
- step S110: Yes When the selection operation is determined (step S110: Yes), the input determination unit 155, the direction of the selectable area acquired in step S106, the gravity center position G1 of the user's hand at the movement start time T1, and the movement stop time T2 It is determined which selectable area has been selected based on the center of gravity position G2 of the user's hand. Then, the transmission unit 156 notifies the application software of a command indicating that the selectable area has been selected together with the information determined by the input determination unit 155 (step S111).
- step S110 when it is determined that the operation is a gesture operation (step S110: No), the input determining unit 155 displays the movement trajectory drawn by the center of gravity of the hand from the start of movement to the stop of movement in the gesture determination table shown in FIG. The user's gesture is determined by collating with the feature data. Then, the transmission unit 156 acquires a control command assigned to the determined gesture (that is, identification information) from the command table, and transmits the acquired control command to the application software (step S112). That is, the transmission unit 156 transmits the determined input result to the application to be operated. If there is no corresponding feature data in the gesture determination table, the transmission unit 156 ends the step S112 as it is without inputting a control command.
- a control command assigned to the determined gesture that is, identification information
- control unit 150 returns to step S101 and repeats the processing from step S101 to step S111.
- the input device 100 determines whether the movement of the user's hand is a selection operation. It is possible to accurately determine whether it is a gesture operation. For example, as shown in FIGS. 13A and 13B, even if the user's hand moves in the direction of the selectable area in the same manner, by using the information on the change in the shape of the user's hand, for example, The case can be determined as a “selection operation”, and the case of FIG. 13B can be determined as a “gesture operation”. Therefore, the input device 100 can simultaneously recognize a gesture operation in addition to a selectable operation. As a result, the input device 100 can realize various valuation input operations, and the convenience of input is high for the user. .
- the input device 100 can further determine whether the movement of the user's hand is a selection operation or a gesture operation. Can be distinguished well. For example, as shown in FIG. 14, even if the shape of the hand does not change much due to the movement, the movement of the user's hand that has not moved toward the selectable area is changed to “gesture” instead of “selection operation”. “Operation”.
- the input device 100 determines whether the user's input is a selection operation or a gesture operation based on a change in hand shape between the movement start time and the movement stop time. Instead, it is also effective to discriminate the user input based on a stepwise or continuous change in the shape of the hand from the start of movement to the stop of movement.
- an input device is disclosed that discriminates a user's input based on a stepwise or continuous change in time series of hand shapes.
- the input device 100 includes an operation unit 110, a display unit 120, an imaging unit 130, an image storage unit 140, and a control unit 150, as illustrated in FIG. And a storage unit 160.
- a storage area (hereinafter referred to as FIFO (First In In First Out) format) (hereinafter referred to as a FIFO (First In In First Out) format) capable of storing image data for a plurality of frames (for example, three frames), as shown in FIG. , Referred to as “FIFO queue”).
- FIFO queue a storage area capable of storing image data for a plurality of frames (for example, three frames), as shown in FIG. , Referred to as “FIFO queue”).
- FIFO queue a storage area capable of storing image data for a plurality of frames (for example, three frames), as shown in FIG. , Referred to as “FIFO queue”).
- a photographed image obtained by photographing the current (latest) hand a photographed image n-2 before the time T, a photographed image n-3 before the time 2T, are stored.
- the control unit 150 acquires the latest one frame of image data from the image storage unit 140 at the time interval T, and enqueues the
- the captured images stored in the FIFO queue are referred to as image data A, image data B, and image data C in order from the newest.
- the time interval T at which the control unit 150 acquires images from the image storage unit 140 is not limited to a specific interval and can be set arbitrarily. For example, the interval may be 1 second or 1 frame based on the factory default setting or the setting operation by the user.
- the control unit 150 commands the imaging unit 130 to start imaging.
- the imaging unit 130 sequentially transmits the captured images to the image storage unit 140.
- the control unit 150 executes application software and starts “input processing” for inputting various control information to the application software.
- the “input processing” will be described below with reference to the flowchart of FIG.
- the movement detection unit 152 determines whether the user's hand has shifted from the stopped state to the moving state using, for example, the same method as in step S103 of the first embodiment (step S201). When not having shifted to the moving state (step S201: No), step S201 is repeated until the moving state is shifted. When it has shifted to the moving state (step S201: Yes), the process proceeds to step S202.
- the image acquisition unit 151 determines whether the contents of the FIFO queue have been updated by the control unit 150, that is, whether the time T has elapsed since the last FIFO queue update (step S202). When the FIFO queue has not been updated, or when step S202 is executed for the first time after the user's hand moves to the moving state (step S202: No), step S202 is repeated until the FIFO queue is updated. If the FIFO queue has been updated (step S202: Yes), the process proceeds to step S203.
- the direction comparison unit 153 identifies a change in the movement direction of the user's hand for each time interval T. More specifically, the direction comparison unit 153 refers to the movement direction of the user's hand from time 2T before time T specified by the images A and B (hereinafter referred to as “movement direction at the previous time point”). The movement direction of the user's hand from before the time T specified by the images B and C to the current time (hereinafter referred to as “current movement direction”) is compared. Then, the comparison result is acquired as a “direction difference degree” indicating the degree of difference in the moving direction between the previous time point and the current time point (step S203). For example, direction comparing unit 153, as shown in FIG.
- the angle (figure line connecting the center of gravity position G B of the user's hand at the time of the gravity center position G A and the image B of the user's hand at the time of image A In this example, 88 °) is specified as the moving direction at the previous time point. Furthermore, it identifies the angle of a line connecting the user's hand at the time of the gravity center position G B and the image C of a user's hand at the time of the image B and the center of gravity position G C (96 ° in the illustrated example) as the moving direction of the current To do. Then, the difference between the two specified angles is acquired as the direction difference degree D.
- the direction difference degree D is numbered in the order of acquisition D0, D1, D2,... After movement detection as shown in FIG.
- the probability that the movement of the hand is a part of the selection operation is higher as the degree of direction difference D in each period and the accumulated addition value are smaller.
- the larger the cumulative addition value the higher the probability that the movement of the hand is for a gesture operation different from the selection operation.
- the shape comparison unit 154 identifies a change in the shape of the user's hand for each fixed time interval T. More specifically, the shape comparison unit 154 includes the shape of the user's hand at the time of the image B (hereinafter referred to as “the shape at the previous time point”) and the shape of the user's hand at the time of the image C (hereinafter referred to as “the shape of the user”). Compare the current shape. Then, the comparison result is acquired as a “shape difference” indicating the degree of change in shape (step S204). For example, as shown in FIG.
- shape comparison unit 154 calculates the angle of the line connecting the center of gravity position G B and the fingertip F B of the hand region at the time of image B (351 ° in the illustrated example) (in the example of FIG. 53 °) angle of a line connecting the center of gravity position G C and the fingertip F C of the hand region at the time of the image C to calculate the difference between the shape dissimilarity S.
- shape difference degree S is numbered as S0, S1, S2,... In the order of acquisition after movement detection as shown in FIG.
- Si: i 0, 1, 2, 3,...
- the user when a hand is moved as part of the selection operation, the user does not change the shape of the hand from the movement start point (in this example, movement start point T1) to the movement stop point (in this example, movement stop point T2). Move your hand. Therefore, the probability that the movement of the hand is a part of the selection operation is higher as the shape difference degree S in each period and the accumulated addition value are smaller. On the other hand, the greater the cumulative addition value of the shape difference degree S, the higher the probability that the movement of the hand is for a gesture operation different from the selection operation.
- the input determination unit 155 cumulatively adds all the addition values ADD acquired after the movement start is detected (step S206). For example, the input determination unit 155 determines the accumulated value ACC by the following (Equation 3) using the most recently calculated addition value as ADDn.
- the input determining unit 155 determines whether the accumulated value ACC is equal to or greater than a predetermined threshold (step S207). When the accumulated value ACC is equal to or greater than the predetermined threshold (step S207: Yes), the input determination unit 155 determines that the user input is a gesture operation, and proceeds to step S208. Otherwise (step S207: No), the process proceeds to step S210.
- This accumulated value ACC is calculated based on the degree of change of the hand movement direction and the degree of change of the hand shape (angle) by the user from the movement start time T1 to the movement stop time T2. A coefficient indicating whether the hand movement is for a gesture operation or part of a selection action.
- step S207: Yes the movement detection unit 152 uses the same method as that in step S104 of the first embodiment, for example, to move the user's hand from the moving state to the stopped state. It is determined whether or not it is being performed (step S208). When not having shifted to the stop state (step S208: No), step S208 is repeated until the stop state is shifted. When it has shifted to the stop state (step S208: Yes), the process proceeds to step S209.
- the input determination unit 155 compares the movement locus drawn by the center of gravity of the hand from the start of movement to the stop of movement with the feature data of the gesture determination table shown in FIG. 6 to determine the user's gesture. Then, the control command assigned to the gesture determined by the transmission unit 156 is acquired from the command table and input to the application software (step S209).
- step S207 when the accumulated value ACC is less than the predetermined threshold (step S207: No), the movement detection unit 152 uses, for example, the same method as that in step S104 in the first embodiment, Is determined to have shifted from the moving state to the stopped state (step S210).
- step S210: No the control unit 150 returns to step S202 and repeats steps S202 to S210.
- step S210: Yes the process proceeds to step S211.
- the input determination unit 155 determines which selectable region has been selected using the same method as in step S111 of the first embodiment. Then, the transmission unit 156 notifies the application software of information (control signal) indicating that the selectable area has been selected (step S211).
- the shape of the hand at the start of movement or the stop of movement is used to determine the user's input. ing.
- the user moves the hand from the movement start time to the movement stop time without changing the movement direction so much. Also, move the hand without changing the shape of the hand.
- the gesture operation the movement direction and the shape of the hand are changed as shown in FIG. 3B (flick operation) and FIG. 19 (rolling operation).
- the shape of the hand at the movement start time T1 and the shape of the hand at the movement stop time T2 are the same shape, and the movement direction is the direction of the selectable region. In some cases.
- the input device 100 according to the above-described embodiment can determine whether the selection operation or the gesture operation is preferable even in such a case, and thus the determination accuracy is high.
- the input device 100 discriminates the user's input based on information on the movement direction of the hand and the change in the shape of the hand, but the change in shape without using the information on the movement direction.
- the user input may be determined based only on the information.
- the input determination unit 155 determines that the movement of the user's hand is a selection operation when the shape difference S acquired in step S108 is equal to or less than a predetermined threshold, and so Otherwise, it may be determined that the user's hand movement is a gesture operation.
- the input determining unit 155 determines the user input based on the accumulated values from the shape difference degrees S0 to Sn, not the accumulated added values from the added values ADD0 to ADDn. May be. Further, an average value of the shape difference degrees S0 to Sn may be used instead of the cumulative addition value.
- step S107 and step S203 a “direction difference” indicating the degree of difference in the moving direction is obtained.
- the direction comparison unit 153 indicates the degree of coincidence between the direction of the selectable area and the movement direction of the user's hand, or indicates the degree of coincidence between the current movement direction and the previous movement direction. May be acquired.
- the direction comparison unit 153 may calculate the direction coincidence by adding the movement distance to the movement direction of the user's hand.
- the direction comparison unit 153 creates a direction comparison map in which regions are set in stages based on the direction of the selectable region acquired in step S106 and the position of the user's hand at the start of movement (for example, As shown in FIG. 20A, rectangular areas A to E having diagonal lines on the line connecting the center of gravity of each selectable area with the center of gravity G1 of the user's hand at the movement start time T1 are set stepwise. Then, the direction comparison unit 153 assigns a value to each region such that the region farther from the center of gravity of the user's hand has a larger value.
- the direction comparison unit 153 acquires the value assigned to the region where the center of gravity G2 of the user's hand at the movement stop time T2 is located (region C in the example of FIG. 20B) as the degree of direction coincidence. That is, the weight value is set so that the closer the direction is to the selectable area, the easier it is to discriminate the selection operation. According to such a configuration, input discrimination with high accuracy is possible.
- a “shape difference” indicating the degree of change in the shape of the hand is acquired.
- the shape comparison unit 154 stops the movement and the shape of the hand at the movement start time T1.
- the “shape coincidence degree” indicating the degree of coincidence with the hand shape at the time point T2 or the degree of coincidence between the hand shape at the previous time point and the current hand shape may be acquired.
- the weight value is set so that the higher the shape matching degree, the easier it is to determine that the movement of the hand is part of the selection operation.
- step S109 and step S205 the direction difference degree (direction difference degree D or Di) and the shape difference degree (shape difference degree S or Si) are added with a predetermined weight to obtain an addition value.
- it is configured to determine whether the movement of the user's hand is a selection operation or a gesture operation.
- the input determination unit 155 may determine whether the user's hand movement is a selection operation or a gesture operation based on a value obtained by adding the direction matching degree and the shape matching degree with a predetermined weight.
- a configuration in which the input device 100 displays a guide indicating the ideal gesture operation hand activation on the operation screen is also effective.
- a belt-like semi-transparent guidance display for guiding the ideal hand movement to the user is output.
- the control unit 150 may include a feedback unit having a function of outputting a movement locus drawn by the user's hand so as to be visible along the guidance display. For example, when the user's hand moves along the band near the guidance display band, the feedback unit moves along the band in accordance with the position of the user's hand as shown in FIG. 21A. The light trace of is displayed.
- the feedback unit may change the display mode of the guide depending on the movement of the hand by a gesture operation.
- the transmittance of the guidance display is changed as needed based on the change in the cumulative value ACC calculated in step S206.
- the feedback unit lowers the transmittance of the guidance display as the cumulative value ACC increases, as indicated by RC in FIG. 21B. According to such a configuration, it is possible to make the user recognize that the gesture operation is determined.
- a change in the shape of the hand is obtained based on a change in the angle of the line connecting the center of gravity of the hand and the fingertip.
- the shape comparison unit 154 may capture a change in the shape of the hand based on the distance between the center of gravity position of the hand and the fingertip.
- the shape comparison unit 154 includes a distance L1 between the center of gravity position of the hand and the fingertip at the movement start time T1 or the previous time point, and a distance L2 between the center of gravity position of the hand and the fingertip at the movement stop time T2 or the current time. May be calculated as the degree of shape difference.
- information on the change in the angle of the line connecting the center of gravity of the hand and the fingertip may be used. According to such a configuration, a change in the shape of the hand can be accurately captured.
- the shape comparison unit 154 may correct the shape dissimilarity or the shape coincidence calculated based on the movement distance of the hand. For example, the shape comparison unit 154 calculates the distance from the center of gravity position of the hand at the previous time point to the current hand center of gravity position as the movement distance of the hand, Alternatively, the calculated shape dissimilarity or shape coincidence is corrected so that the shape coincidence increases as the movement distance increases.
- the shape difference degree may be divided by the movement distance to obtain a new shape difference degree.
- a new shape matching degree may be obtained by multiplying the shape matching degree by the movement distance. For example, as shown in FIG.
- an input discrimination system can be improved by a discrimination process based on a comprehensive evaluation value of a moving speed and a change in hand shape.
- the transmission unit 156 is configured to transmit the determined control command to the application software.
- software that is a target for transmitting a control command based on an input determination result is not limited to application software.
- the input determination unit 155 may input the determined control command to, for example, an operating system.
- the determined control command may be configured to be input to a device other than the input device 100 via a communication line or wireless.
- the input device may be different from the device for executing the operation and the operation system.
- an output device for transmitting the input result to another device that executes the application via a parallel port or a USB (Universal Serial Bus) terminal as a transmission unit is provided. Just do it.
- a personal computer is shown as an example of the input device 100.
- the input device 100 is not limited to a personal computer, and may be another type of electric device having a function of determining input from a user. May be.
- the input device 100 may be a television, a recorder, a game machine, a fixed phone, a mobile phone, a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), or the like.
- the input device 100 may be realized by a dedicated system or an ordinary computer system.
- the input device 100 may be configured by storing and distributing a program for performing the above-described operation in a computer-readable recording medium, installing the program in a computer, and executing the above-described processing. Good.
- it may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer, for example.
- the above-described functions may be realized by joint operation of the OS and application software. In this case, only the part other than the OS may be stored and distributed in a medium, or may be downloaded to a computer.
- Recording media for recording the above programs include USB memory, flexible disk, CD (Compact Disc), DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), MO (Magneto-Optical disk), SD memory card ( Computer-readable non-transitory recording media such as Secure Digital Memory card, Memory Stick (registered trademark), and other magnetic disks, optical disks, magneto-optical disks, semiconductor memories, magnetic tapes, and the like can be used. It is also possible to superimpose a computer program on a carrier wave and distribute the computer program via a communication network. For example, the computer program is posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program is distributed via the network. And you may comprise so that each function of the input device 1 may be implement
- BBS Bulletin Board System
- An input device that discriminates a user's input based on a user's hand movement in a captured image associated with an operation screen in which a selectable area is defined, Input determination for determining whether a user input is a selection operation of the selectable region or a gesture operation different from the selection operation based on a change in the shape of the hand in the plurality of captured images caused by the movement of the hand Comprising means, An input device characterized by that.
- the input determination means determines whether the user input is a selection operation of the selectable region or a gesture operation based on a direction in which the hand moves in the plurality of captured images and a change in the shape of the hand caused by the movement. To determine, The input device according to Supplementary Note 1, wherein:
- the input determining means includes a change in the shape of the hand specified by comparing the shape of the hand at the start of movement and the shape of the hand at the stop of movement, and the position of the hand at the start of movement. And the direction of movement of the hand specified based on the position of the hand at the time of the movement stop, and determining whether the user input is a selection operation or a gesture operation of the selectable region,
- the input device according to Supplementary Note 2, wherein
- a movement detecting means for detecting a movement start time and a movement stop time of the hand;
- the input determining means includes a change in the shape of the hand at predetermined time intervals from the movement start time to the movement stop time, and the hand at each predetermined time interval from the movement start time to the movement stop time. Determining whether the user input is a selection operation of the selectable area or a gesture operation based on a change in the movement direction;
- the input device according to Supplementary Note 2, wherein
- Shape comparison means for comparing the shape of the hand at the start of movement and the shape of the hand at the stop of movement to calculate a shape dissimilarity indicating the degree of shape difference or a shape coincidence indicating the degree of coincidence
- the movement direction of the hand is specified based on the position of the hand at the start of movement and the position of the hand at the stop of movement, and the selectable area based on the position of the hand at the start of movement
- a direction in which a direction is specified, and the direction in which the hand is moved is compared with the direction in which the selectable area is specified to calculate a direction difference indicating the degree of direction difference or a direction coincidence indicating the degree of match.
- Shape comparison means Direction comparison means for comparing the current movement direction and the previous movement direction at each predetermined time interval and calculating a direction difference indicating a degree of direction difference or a direction coincidence indicating a degree of match; Further comprising
- the input determining means includes Adding the shape dissimilarity or the shape coincidence and the direction dissimilarity or the direction coincidence with a predetermined weight; With the movement start time as the accumulation start time, the added value is cumulatively added at each predetermined time interval, When the cumulative added value exceeds a predetermined threshold value, the user's input is determined as a gesture operation, and the cumulative added value is set to the predetermined value before the movement is stopped or within a predetermined time after the cumulative start. When the threshold value is not exceeded, the user input is determined as the selection operation of the selectable area,
- the input device includes Adding the shape dissimilarity or the shape coincidence and the direction dissimilarity or the direction coincidence with a predetermined weight;
- the hand is a user's hand with a finger raised;
- the shape comparison means compares the current positional relationship between the center of gravity position of the hand and the fingertip and the positional relationship between the center of gravity position of the hand and the fingertip before the predetermined time, Calculating the shape matching degree,
- the input device according to Supplementary Note 6, wherein:
- To the computer that controls the input device to determine the input of Input determination for determining whether a user input is a selection operation of the selectable region or a gesture operation different from the selection operation based on a change in the shape of the hand in the plurality of captured images caused by the movement of the hand A storage medium characterized by realizing a function.
- the present invention can be used for an input device that determines user input based on an image of a user's hand.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- General Health & Medical Sciences (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
- Image Analysis (AREA)
Abstract
Description
選択可能領域が定義された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別手段
を備える、ことを特徴とする。
選択操作の対象となる選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力方法であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する
ことを特徴とする。
プログラムを記憶するコンピュータ読み取り可能な非一時的な記憶媒体であって、当該プログラムが、選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置を制御するコンピュータに、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別機能、を実現させる
ことを特徴とする。
本実施の形態の入力装置100は、カメラで撮像されたユーザの手の動きに基づいて、ユーザの機器に対する入力(例えば、ユーザの機器に対する操作指令、コマンド、アイコン等が表示された選択可能領域の選択等)を判別する機能を備える。入力装置100は、例えば図1に示すような外観を持つパーソナルコンピュータである。最初に、図2と図3を参照して、入力装置100が行う処理の概要を説明する。
実施の形態1の入力装置100は、移動開始時点と移動停止時点の手の形状の変化に基づいてユーザの入力が選択操作かジェスチャ操作か判別した。その代わりに、移動開始から移動停止までの手の形状の時系列にそった段階的な、あるいは連続的な変化に基づいてユーザの入力を判別する構成も効果的である。本実施形態では、手の形状の時系列にそった段階的なあるいは連続的な変化に基づいて、ユーザの入力を判別する入力装置が開示される。
上記の実施形態の一部または全部は、以下の付記のようにも記載されうるが、以下には限られない。
選択可能領域が定義された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別手段
を備える、
ことを特徴とする入力装置。
前記入力判別手段は、前記複数の撮像画像において前記手が移動した方向と該移動により生じる前記手の形状の変化とに基づいて、ユーザの入力が前記選択可能領域の選択動作かジェスチャ動作かを判別する、
ことを特徴とする付記1に記載の入力装置。
前記手の移動開始時点及び移動停止時点を検出する移動検出手段、を更に備え、
前記入力判別手段は、前記移動開始時点の前記手の形状と前記移動停止時点の前記手の形状とを比較して特定される前記手の形状の変化と、前記移動開始時点の前記手の位置と前記移動停止時点の前記手の位置とに基づいて特定される前記手の移動方向と、に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする付記2に記載の入力装置。
前記手の移動開始時点及び移動停止時点を検出する移動検出手段、を更に備え、
前記入力判別手段は、前記移動開始時点から前記移動停止時点までの所定時間間隔毎の前記手の形状の変化と、前記移動開始時点から前記移動停止時点までの前記所定時間間隔毎の前記手の移動方向の変化と、に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする付記2に記載の入力装置。
前記移動開始時点の前記手の形状と前記移動停止時点の前記手の形状とを比較して、形状の相違の度合いを示す形状相違度または一致の度合いを示す形状一致度を算出する形状比較手段と、
前記移動開始時点の前記手の位置と前記移動停止時点の前記手の位置とに基づいて前記手の移動方向を特定し、前記移動開始時点の前記手の位置を基準とした前記選択可能領域の方向を特定し、特定した前記手の移動方向と特定した前記選択可能領域の方向とを比較して、方向の相違の度合いを示す方向相違度または一致の度合いを示す方向一致度を算出する方向比較手段と、を更に備え、
前記入力判別手段は、前記形状相違度または前記形状一致度と、前記方向相違度または前記方向一致度と、を所定の重みで加算し、該加算値に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする付記3に記載の入力装置。
前記所定時間間隔毎に、現時点の前記手の形状と前記所定時間前の前記手の形状とを比較して、形状の相違の度合いを示す形状相違度または一致の度合いを示す形状一致度を算出する形状比較手段と、
前記所定時間間隔毎に、現時点の移動方向と前時点の移動方向とを比較して、方向の相違の度合いを示す方向相違度または一致の度合いを示す方向一致度を算出する方向比較手段と、を更に備え、
前記入力判別手段は、
前記形状相違度または前記形状一致度と、前記方向相違度または前記方向一致度と、を所定の重みで加算し、
前記移動開始時点を累積開始時点として、前記加算した値を前記所定時間間隔毎に累積加算し、
前記累積加算値が所定の閾値を超えた場合、ユーザの入力はジェスチャ操作と判別し、前記移動停止時点までにもしくは前記累積開始後予め定められた時間内に、前記累積加算値が前記所定の閾値を超えなかった場合、ユーザの入力は前記選択可能領域の選択操作と判別する、
ことを特徴とする付記4に記載の入力装置。
前記手は指を立てた状態のユーザの手であり、
前記形状比較手段は、前記現時点の前記手の重心位置と指先との位置関係と、前記所定時間前の前記手の重心位置と指先との位置関係と、を比較して、前記形状相違度または前記形状一致度を算出する、
ことを特徴とする付記6に記載の入力装置。
前記操作画面には、ユーザに理想的な前記手の移動軌跡を案内する帯状の半透明の案内表示が表示されており、
前記手が描いた移動軌跡を前記案内表示に沿って視認可能に出力するとともに、前記累積加算値に基づいて案内表示の透過率を随時変更するフィードバック手段、を更に備える、
ことを特徴とする付記6または7に記載の入力装置。
選択操作の対象となる選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力方法であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する、
ことを特徴とする入力方法。
プログラムを記憶するコンピュータ読み取り可能な非一時的な記憶媒体であって、当該プログラムが、選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置を制御するコンピュータに、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別機能、を実現させる
ことを特徴とする記憶媒体。
110 操作部
120 表示部
130 撮像部
140 画像記憶部
150 制御部
151 画像取得部
152 移動検出部
153 方向比較部
154 形状比較部
155 入力判別部
156 伝達部
160 記憶部
Claims (10)
- 選択可能領域が定義された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別手段
を備える、
ことを特徴とする入力装置。 - 前記入力判別手段は、前記複数の撮像画像において前記手が移動した方向と該移動により生じる前記手の形状の変化とに基づいて、ユーザの入力が前記選択可能領域の選択動作かジェスチャ動作かを判別する、
ことを特徴とする請求項1に記載の入力装置。 - 前記手の移動開始時点及び移動停止時点を検出する移動検出手段、を更に備え、
前記入力判別手段は、前記移動開始時点の前記手の形状と前記移動停止時点の前記手の形状とを比較して特定される前記手の形状の変化と、前記移動開始時点の前記手の位置と前記移動停止時点の前記手の位置とに基づいて特定される前記手の移動方向と、に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする請求項2に記載の入力装置。 - 前記手の移動開始時点及び移動停止時点を検出する移動検出手段、を更に備え、
前記入力判別手段は、前記移動開始時点から前記移動停止時点までの所定時間間隔毎の前記手の形状の変化と、前記移動開始時点から前記移動停止時点までの前記所定時間間隔毎の前記手の移動方向の変化と、に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする請求項2に記載の入力装置。 - 前記移動開始時点の前記手の形状と前記移動停止時点の前記手の形状とを比較して、形状の相違の度合いを示す形状相違度または一致の度合いを示す形状一致度を算出する形状比較手段と、
前記移動開始時点の前記手の位置と前記移動停止時点の前記手の位置とに基づいて前記手の移動方向を特定し、前記移動開始時点の前記手の位置を基準とした前記選択可能領域の方向を特定し、特定した前記手の移動方向と特定した前記選択可能領域の方向とを比較して、方向の相違の度合いを示す方向相違度または一致の度合いを示す方向一致度を算出する方向比較手段と、を更に備え、
前記入力判別手段は、前記形状相違度または前記形状一致度と、前記方向相違度または前記方向一致度と、を所定の重みで加算し、該加算値に基づいてユーザの入力が前記選択可能領域の選択操作かジェスチャ操作かを判別する、
ことを特徴とする請求項3に記載の入力装置。 - 前記所定時間間隔毎に、現時点の前記手の形状と前記所定時間前の前記手の形状とを比較して、形状の相違の度合いを示す形状相違度または一致の度合いを示す形状一致度を算出する形状比較手段と、
前記所定時間間隔毎に、現時点の移動方向と前時点の移動方向とを比較して、方向の相違の度合いを示す方向相違度または一致の度合いを示す方向一致度を算出する方向比較手段と、を更に備え、
前記入力判別手段は、
前記形状相違度または前記形状一致度と、前記方向相違度または前記方向一致度と、を所定の重みで加算し、
前記移動開始時点を累積開始時点として、前記加算した値を前記所定時間間隔毎に累積加算し、
前記累積加算値が所定の閾値を超えた場合、ユーザの入力はジェスチャ操作と判別し、前記移動停止時点までにもしくは前記累積開始後予め定められた時間内に、前記累積加算値が前記所定の閾値を超えなかった場合、ユーザの入力は前記選択可能領域の選択操作と判別する、
ことを特徴とする請求項4に記載の入力装置。 - 前記手は指を立てた状態のユーザの手であり、
前記形状比較手段は、前記現時点の前記手の重心位置と指先との位置関係と、前記所定時間前の前記手の重心位置と指先との位置関係と、を比較して、前記形状相違度または前記形状一致度を算出する、
ことを特徴とする請求項6に記載の入力装置。 - 前記操作画面には、ユーザに理想的な前記手の移動軌跡を案内する帯状の半透明の案内表示が表示されており、
前記手が描いた移動軌跡を前記案内表示に沿って視認可能に出力するとともに、前記累積加算値に基づいて案内表示の透過率を随時変更するフィードバック手段、を更に備える、
ことを特徴とする請求項6または7に記載の入力装置。 - 選択操作の対象となる選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力方法であって、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する
ことを特徴とする入力方法。 - プログラムを記憶するコンピュータ読み取り可能な非一時的な記憶媒体であって、当該プログラムが、選択可能領域が表示された操作画面に対応付けられる撮像画像の中のユーザの手の動きに基づいて、ユーザの入力を判別する入力装置を制御するコンピュータに、
前記手が移動したことにより生じる複数の前記撮像画像における前記手の形状の変化に基づいて、ユーザの入力が前記選択可能領域の選択操作か該選択操作とは異なるジェスチャ操作かを判別する入力判別機能、を実現させる
ことを特徴とする記憶媒体。
Priority Applications (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA2882004A CA2882004A1 (en) | 2012-08-17 | 2013-08-13 | Input device, input method, and storage medium |
CN201380043996.8A CN104584080B (zh) | 2012-08-17 | 2013-08-13 | 输入设备和输入方法 |
US14/422,124 US9753545B2 (en) | 2012-08-17 | 2013-08-13 | Input device, input method, and storage medium |
KR1020157003798A KR101624215B1 (ko) | 2012-08-17 | 2013-08-13 | 입력 장치, 입력 방법, 및 기억 매체 |
EP13879402.9A EP2887318A4 (en) | 2012-08-17 | 2013-08-13 | INPUT DEVICE, INPUT METHOD AND STORAGE MEDIUM |
IL237225A IL237225A0 (en) | 2012-08-17 | 2015-02-15 | Input device, input method and storage medium |
HK15107654.5A HK1207192A1 (en) | 2012-08-17 | 2015-08-08 | Input device, input method, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012-181223 | 2012-08-17 | ||
JP2012181223A JP5665140B2 (ja) | 2012-08-17 | 2012-08-17 | 入力装置、入力方法、及びプログラム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014027666A1 true WO2014027666A1 (ja) | 2014-02-20 |
Family
ID=50286613
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2013/071893 WO2014027666A1 (ja) | 2012-08-17 | 2013-08-13 | 入力装置、入力方法、及び記憶媒体 |
Country Status (10)
Country | Link |
---|---|
US (1) | US9753545B2 (ja) |
EP (1) | EP2887318A4 (ja) |
JP (1) | JP5665140B2 (ja) |
KR (1) | KR101624215B1 (ja) |
CN (1) | CN104584080B (ja) |
CA (1) | CA2882004A1 (ja) |
HK (1) | HK1207192A1 (ja) |
IL (1) | IL237225A0 (ja) |
TW (1) | TWI543018B (ja) |
WO (1) | WO2014027666A1 (ja) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019061334A (ja) * | 2017-09-25 | 2019-04-18 | Kddi株式会社 | 機器制御装置、機器制御方法及び機器制御システム |
WO2019168190A1 (ja) * | 2018-03-02 | 2019-09-06 | Necソリューションイノベータ株式会社 | 操作推定装置、操作推定方法、及びコンピュータ読み取り可能な記録媒体 |
CN111062312A (zh) * | 2019-12-13 | 2020-04-24 | RealMe重庆移动通信有限公司 | 手势识别方法、手势控制方法、装置、介质与终端设备 |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9070019B2 (en) | 2012-01-17 | 2015-06-30 | Leap Motion, Inc. | Systems and methods for capturing motion in three-dimensional space |
US10846942B1 (en) | 2013-08-29 | 2020-11-24 | Ultrahaptics IP Two Limited | Predictive information for free space gesture control and communication |
JP6178741B2 (ja) * | 2013-09-24 | 2017-08-09 | 京セラドキュメントソリューションズ株式会社 | 電子機器 |
US9996638B1 (en) * | 2013-10-31 | 2018-06-12 | Leap Motion, Inc. | Predictive information for free space gesture control and communication |
CN106030460B (zh) * | 2014-02-18 | 2017-12-19 | 三菱电机株式会社 | 移动体用手势引导装置、移动体用手势引导***及移动体用手势引导方法 |
US9785247B1 (en) | 2014-05-14 | 2017-10-10 | Leap Motion, Inc. | Systems and methods of tracking moving hands and recognizing gestural interactions |
JP6579866B2 (ja) * | 2015-08-31 | 2019-09-25 | キヤノン株式会社 | 情報処理装置とその制御方法、プログラム、記憶媒体 |
JP6650595B2 (ja) * | 2015-09-24 | 2020-02-19 | パナソニックIpマネジメント株式会社 | 機器制御装置、機器制御方法、機器制御プログラムおよび記録媒体 |
EP3182250B1 (en) * | 2015-12-18 | 2019-10-30 | Aptiv Technologies Limited | System and method for monitoring 3d space in front of an output unit for the control of the output unit |
US10025376B2 (en) | 2016-04-27 | 2018-07-17 | Rovi Guides, Inc. | Methods and systems for displaying additional content on a heads up display displaying a virtual reality environment |
AU2017255567A1 (en) * | 2016-04-27 | 2018-07-12 | Rovi Guides, Inc. | Methods and systems for displaying additional content on a heads up display displaying a virtual reality environment |
JP2018005815A (ja) * | 2016-07-08 | 2018-01-11 | 大日本印刷株式会社 | 入力システム及びその入力装置 |
US10162422B2 (en) * | 2016-10-10 | 2018-12-25 | Deere & Company | Control of machines through detection of gestures by optical and muscle sensors |
EP4160530A4 (en) * | 2020-06-01 | 2024-06-26 | National Institute Of Advanced Industrial Science and Technology | GESTURE RECOGNITION DEVICE, SYSTEM AND PROGRAM THEREFOR |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009151691A (ja) | 2007-12-21 | 2009-07-09 | Sony Corp | 情報処理装置及びタッチ動作認識方法 |
JP2011175623A (ja) | 2010-01-29 | 2011-09-08 | Shimane Prefecture | 画像認識装置および操作判定方法並びにプログラム |
JP2011204019A (ja) * | 2010-03-25 | 2011-10-13 | Sony Corp | ジェスチャ入力装置、ジェスチャ入力方法およびプログラム |
WO2011142317A1 (ja) * | 2010-05-11 | 2011-11-17 | 日本システムウエア株式会社 | ジェスチャー認識装置、方法、プログラム、および該プログラムを格納したコンピュータ可読媒体 |
JP2011248606A (ja) | 2010-05-26 | 2011-12-08 | Canon Inc | 表示制御装置および表示制御方法 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8555207B2 (en) * | 2008-02-27 | 2013-10-08 | Qualcomm Incorporated | Enhanced input using recognized gestures |
US9772689B2 (en) * | 2008-03-04 | 2017-09-26 | Qualcomm Incorporated | Enhanced gesture-based image manipulation |
US8428368B2 (en) * | 2009-07-31 | 2013-04-23 | Echostar Technologies L.L.C. | Systems and methods for hand gesture control of an electronic device |
IL204436A (en) * | 2010-03-11 | 2016-03-31 | Deutsche Telekom Ag | A system and method for remote control of online TV by waving hands |
US20110289455A1 (en) * | 2010-05-18 | 2011-11-24 | Microsoft Corporation | Gestures And Gesture Recognition For Manipulating A User-Interface |
US9262015B2 (en) * | 2010-06-28 | 2016-02-16 | Intel Corporation | System for portable tangible interaction |
CN102236414A (zh) * | 2011-05-24 | 2011-11-09 | 北京新岸线网络技术有限公司 | 三维显示空间中的图片操作方法和*** |
-
2012
- 2012-08-17 JP JP2012181223A patent/JP5665140B2/ja active Active
-
2013
- 2013-08-13 CA CA2882004A patent/CA2882004A1/en not_active Abandoned
- 2013-08-13 EP EP13879402.9A patent/EP2887318A4/en not_active Withdrawn
- 2013-08-13 WO PCT/JP2013/071893 patent/WO2014027666A1/ja active Application Filing
- 2013-08-13 KR KR1020157003798A patent/KR101624215B1/ko active IP Right Grant
- 2013-08-13 CN CN201380043996.8A patent/CN104584080B/zh active Active
- 2013-08-13 US US14/422,124 patent/US9753545B2/en not_active Expired - Fee Related
- 2013-08-16 TW TW102129418A patent/TWI543018B/zh not_active IP Right Cessation
-
2015
- 2015-02-15 IL IL237225A patent/IL237225A0/en unknown
- 2015-08-08 HK HK15107654.5A patent/HK1207192A1/xx not_active IP Right Cessation
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2009151691A (ja) | 2007-12-21 | 2009-07-09 | Sony Corp | 情報処理装置及びタッチ動作認識方法 |
JP2011175623A (ja) | 2010-01-29 | 2011-09-08 | Shimane Prefecture | 画像認識装置および操作判定方法並びにプログラム |
JP2011204019A (ja) * | 2010-03-25 | 2011-10-13 | Sony Corp | ジェスチャ入力装置、ジェスチャ入力方法およびプログラム |
WO2011142317A1 (ja) * | 2010-05-11 | 2011-11-17 | 日本システムウエア株式会社 | ジェスチャー認識装置、方法、プログラム、および該プログラムを格納したコンピュータ可読媒体 |
JP2011248606A (ja) | 2010-05-26 | 2011-12-08 | Canon Inc | 表示制御装置および表示制御方法 |
Non-Patent Citations (1)
Title |
---|
See also references of EP2887318A4 |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019061334A (ja) * | 2017-09-25 | 2019-04-18 | Kddi株式会社 | 機器制御装置、機器制御方法及び機器制御システム |
WO2019168190A1 (ja) * | 2018-03-02 | 2019-09-06 | Necソリューションイノベータ株式会社 | 操作推定装置、操作推定方法、及びコンピュータ読み取り可能な記録媒体 |
CN111062312A (zh) * | 2019-12-13 | 2020-04-24 | RealMe重庆移动通信有限公司 | 手势识别方法、手势控制方法、装置、介质与终端设备 |
CN111062312B (zh) * | 2019-12-13 | 2023-10-27 | RealMe重庆移动通信有限公司 | 手势识别方法、手势控制方法、装置、介质与终端设备 |
Also Published As
Publication number | Publication date |
---|---|
CA2882004A1 (en) | 2014-02-20 |
JP5665140B2 (ja) | 2015-02-04 |
IL237225A0 (en) | 2015-04-30 |
KR20150034258A (ko) | 2015-04-02 |
EP2887318A1 (en) | 2015-06-24 |
HK1207192A1 (en) | 2016-01-22 |
TW201423482A (zh) | 2014-06-16 |
JP2014038527A (ja) | 2014-02-27 |
TWI543018B (zh) | 2016-07-21 |
US9753545B2 (en) | 2017-09-05 |
CN104584080A (zh) | 2015-04-29 |
US20150234469A1 (en) | 2015-08-20 |
CN104584080B (zh) | 2017-03-01 |
EP2887318A4 (en) | 2016-04-27 |
KR101624215B1 (ko) | 2016-06-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2014027666A1 (ja) | 入力装置、入力方法、及び記憶媒体 | |
JP5802667B2 (ja) | ジェスチャ入力装置およびジェスチャ入力方法 | |
KR102194272B1 (ko) | 제스처들을 이용한 터치 입력들의 향상 | |
US10732742B2 (en) | Information processing program and method for causing a computer to transform a displayed object based on input area and force of a touch input | |
KR101302910B1 (ko) | 제스처 인식 장치, 제스처 인식 방법, 제어 프로그램을 기록한 컴퓨터로 판독 가능한 기록 매체 | |
US9547391B2 (en) | Method for processing input and electronic device thereof | |
EP2864932B1 (en) | Fingertip location for gesture input | |
KR102479052B1 (ko) | 복수의 컨트롤러를 이용한 전자 장치의 디스플레이 제어 방법 및 그 장치 | |
US8922351B2 (en) | Display apparatus, information processing system, recording medium and television receiver | |
US20220382377A1 (en) | Systems and methods for controlling virtual widgets in a gesture-controlled device | |
JP2024520943A (ja) | キー機能実行方法、キー機能実行システム、キー機能実行装置、電子機器、及びコンピュータプログラム | |
US11886643B2 (en) | Information processing apparatus and information processing method | |
EP2899623A2 (en) | Information processing apparatus, information processing method, and program | |
CN114360047A (zh) | 举手手势识别方法、装置、电子设备及存储介质 | |
CN104714736A (zh) | 一种退出全屏锁定状态的控制方法及终端 | |
CN104951211B (zh) | 一种信息处理方法和电子设备 | |
JP2014109888A (ja) | 入力装置およびプログラム | |
CN112051746B (zh) | 获取服务的方法以及装置 | |
KR102194778B1 (ko) | 공간상의 상호 작용을 이용한 단말의 제어 방법 및 그 단말 | |
JP5565886B2 (ja) | 入力装置、入力方法、及びプログラム | |
EP3568745B1 (en) | Hover interaction using orientation sensing | |
CN118368357A (zh) | 界面控制方法、装置、终端及存储介质 | |
CN116246335A (zh) | 追踪扩展现实的输入手势的方法和使用其的*** | |
CN105867806A (zh) | 一种输入方法及电子设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13879402 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20157003798 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2882004 Country of ref document: CA |
|
WWE | Wipo information: entry into national phase |
Ref document number: 237225 Country of ref document: IL |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 14422124 Country of ref document: US Ref document number: 2013879402 Country of ref document: EP |