CN107918507A - A kind of virtual touchpad method based on stereoscopic vision - Google Patents

A kind of virtual touchpad method based on stereoscopic vision Download PDF

Info

Publication number
CN107918507A
CN107918507A CN201610898268.0A CN201610898268A CN107918507A CN 107918507 A CN107918507 A CN 107918507A CN 201610898268 A CN201610898268 A CN 201610898268A CN 107918507 A CN107918507 A CN 107918507A
Authority
CN
China
Prior art keywords
finger tip
virtual touchpad
gesture
stereoscopic vision
method based
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610898268.0A
Other languages
Chinese (zh)
Inventor
郝刚
梁鹏
郑振兴
吴玉婷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Polytechnic Normal University
Original Assignee
Guangdong Polytechnic Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Polytechnic Normal University filed Critical Guangdong Polytechnic Normal University
Priority to CN201610898268.0A priority Critical patent/CN107918507A/en
Publication of CN107918507A publication Critical patent/CN107918507A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present invention proposes a kind of virtual touchpad method based on stereoscopic vision, the described method comprises the following steps：S1, merges the tracking that a plurality of visual cues realize instruction gesture；S2, the three-dimensional position of finger tip is calculated with reference to depth information and camera imaging model；S3, provides the definition of virtual touchpad, and combines the three-dimensional position of finger tip and the depth information of virtual touchpad, realizes man-machine interactive operation.

Description

A kind of virtual touchpad method based on stereoscopic vision
Technical field
The invention belongs to computer vision interaction, is related to a kind of virtual touchpad method based on stereoscopic vision.
Background technology
In recent years, although portable keyboard and mouse can complete most interactive task, need to fully take up use The both hands and notice at family, there are significant limitation in terms of interactive naturality and flexibility.And view-based access control model is man-machine Interaction obtains the video image for including people in interaction scenarios by imaging device, is anticipated using behavior of the vision algorithm to people and interaction Figure is identified, understands and describes, there is provided a kind of natural man-machine interaction mode focusing on people.
The mankind are most accustomed to the hands carrying out most of interactive operation with the external world, therefore the gesture interaction of view-based access control model is also subject to Extensive concern and research.By years of researches, the human-computer interaction technology of view-based access control model obtains many progress, but still deposits In many difficult and challenge.
Gesture interaction system based on wearable vision allows user's free movement anywhere, can also be with computer point The visual experience of wearer is enjoyed, but dresses the factor such as naturality, diversity and hommization of man-machine interaction mode under environment and all can Important influence is produced to interactive experience.
Patent " a kind of gesture tracking method based in mixed and disorderly background, application number CN201510401525.0 ", using Histogram divion is carried out to each two field picture in LUV color spaces, then the different zones of segmentation are big according to the possibility of the colour of skin Small sequence, by calculate the different zones of present frame with previous frame gesture area apart from size and corresponding each region In movement pixel number, determine the gesture area of present frame, and record gesture information.
" the finger tip localization method of instruction gesture, application number CN200910198196.9 ", are according to instruction gesture to patent The hand-characteristic of instruction gesture in behavior, automatically determines the finger tip position of instruction gesture.Using background subtraction, extraction refers to Show gesture object, with skin color segmentation method, the hand region of extraction instruction gesture, according to the finger tip of instruction gesture positioned at finger Show the hand edge profile of gesture and apart from the finger areas center of gravity farthest of instruction gesture, automatically determine finger cusp.
" gesture operation method, utilize gesture positioning screen light calibration method and gesture system, application number to patent CN201510731772.7 ", first obtains at least one characteristic portion of user, using the characteristic portion position as reference, The position of generation and the characteristic portion has the pseudo operation space of fixed relative position relation, and it is empty to detect the pseudo operation Between in gesture, and perform corresponding operation.
Above-mentioned the first two invention mainly using the advanced pedestrian's hand segmentation of features of skin colors and then is identified and tracks, although base It is easy to use in the hand tracking algorithm of color, but the change to illumination condition is sensitive, it is difficult to adapt to changeable during wearing calculates Environment.
Although above-mentioned 3rd is invented the three-dimensional information that can obtain gesture, at least one object of reference need to be selected in advance, and The behavior that cannot be distinguished by wearer is intended to.
The content of the invention
Present invention aims at using virtual touchpad provide space constraint, by it is a kind of naturally in a manner of distinguish wearer Interbehavior and voluntary behavior, solve the problems, such as to cannot be distinguished by wearer's behavior intention in visual interactive.In addition, pin To illumination condition under wearing environment and the uncertainty of background change, it is proposed that based on ICONDENSATION frame fusion profiles The tracking of instruction gesture is realized with the CL-PGT algorithms of LBP features.
In order to solve the above technical problems, the present invention adopts the following technical scheme that：It is a kind of virtual tactile based on stereoscopic vision Template method, comprises the following steps：
S1, merges the tracking that a plurality of visual cues realize instruction gesture；
S2, the three-dimensional position of finger tip is calculated with reference to depth information and camera imaging model；
S3, provides the definition of virtual touchpad, and combines the three-dimensional position of finger tip and the depth information of virtual touchpad, real Existing man-machine interactive operation.
Further, it is by special based on ICONDENSATION frame fusions profile and LBP in step S1 of the invention CL-PGT (i.e. Contour and Lbp based Pointing Gesture Tracking) algorithm of sign realizes instruction gesture Tracking.
ICONDENSATION frames are (to come from paper to CONDENSATION algorithms by introducing important sampling mechanism CONDENSATION-Conditional density propagation for visual tracking, author M.Isard, A.Blake, International Journal of Computer Vision (IJCV), 1998,29 (1)：5-28.) carry out Extension, and CONDENSATION algorithms are actually particle filter (comes from paper Novel approach to non- Linear and non-gaussion Bayesian state estimation, author Gordon N, Salmond D, ProcofInstitute Electric Engineering, 1993,140 (2)：107-113.) a kind of deformation.
This track algorithm is at the same time predicted the state of particle using motion model and important sampling two ways, i.e., comprehensive Close and consider space-time restriction and the accuracy and efficiency of particle sampler is improved based on the heuristic information of Depth cue.
The strategy carries out the prediction of particle state using two kinds of clues, only two kinds of clues at the same time prediction of failure when can just lead Cause tracking mistake.In addition, important sampling technology can be used for dbjective state reinitializing, for track algorithm provide one kind from The mechanism recovered automatically in mistake.
Further, in step S2 of the invention, using the tracking result for indicating gesture, the two-dimensional position of finger tip is obtained, Then human hand plane and the intersection point of finger tip straight line of image formation are calculated.
Wherein, human hand plane is fitted the obtained 3 d-dem point set of Hand Gesture Segmentation by least square method Arrive, and finger tip point is in the coordinate of imaging plane, by indicating that gesture tracking algorithm is calculated.
Further.In the step S3 of the present invention, virtual touchpad is not actual hardware, is defined in stereoscopic vision Certain distance and perpendicular to the plane of camera optical axis immediately ahead of machine, can not the person of being worn see, but can person's of being worn " touching " Arrive.
Further, in step S3 of the invention, the decision procedure that " touching " with virtual touchpad operates is to judge hand Whether the depth value of gesture finger tip is greater than or equal to virtual touchpad to the distance of stereoscopic vision machine.
Wherein, man-machine interactive operation includes：Finger tip clicks on the click event of touch pad analog mouse, and finger tip is on a touchpad Draw stroke and realize that handwriting input and finger tip draw the circle extract operation that closed curve realizes region on a touchpad.
Example is taken as with aiming circle, the interaction based on virtual touchpad is summarised as following steps：(1) will instruction gesture contact void Intend touch pad, (2) mobile instruction gesture draws closed curve circle and takes target area, and (3) will indicate that gesture is moved away from virtual touchpad End figure takes.
The present invention has following beneficial effect compared with prior art：
(1) present invention by it is a kind of naturally in a manner of distinguish the interbehavior and voluntary behavior of wearer, efficiently solve and regard Feel and cannot be distinguished by the problem of wearer's behavior is intended in interaction；
(2) the CL-PGT algorithms based on ICONDENSATION frame fusions profile and LBP features realize instruction gesture with Track, solves illumination condition and the uncertainty of background change under wearing environment.
Brief description of the drawings
Fig. 1 is the group method flow chart of the present invention.
Fig. 2 is the method flow diagram for the track algorithm CL-PGT for indicating gesture.
Fig. 3 is to calculate finger tip three-dimensional position schematic diagram.
Fig. 4 is virtual touchpad schematic diagram.
Embodiment
Below in conjunction with the accompanying drawings and specific embodiment the present invention is carried out in further detail with complete explanation.It is appreciated that It is that specific embodiment described herein is only used for explaining the present invention, rather than limitation of the invention.
As shown in Figure 1, the virtual touchpad method based on stereoscopic vision of the present invention, comprises the following steps：
S1：It is as shown in Fig. 2, real by the CL-PGT algorithms based on ICONDENSATION frame fusions profile and LBP features The track algorithm of existing instruction gesture, specific implementation step are as follows：
1st, initialize
The SVM classifier SVM of LBP features is obtained using the positive and negative sample training manually markedlbp, initialize particle assembly State.
2nd, particle filter tracking
1) new particle assembly is produced using factored sampling and important sampling：To each particle n=1：N
A. produce equally distributed random number α ∈ [0,1)；
B. if meeting α≤γ, using factored sampling, that is, benchmark particle is passed throughNew particle is produced with motion modelAnd makeWherein, γ is threshold value, and interval is (0,1), important to control Sampling and the ratio of factored sampling；T is represented sometime；N represents particle numbering；Benchmark particle is represented, by t-1 N-th of particle at momentWith probabilitySelect generation；Represent the zero-mean gaussian white noise independently of dbjective state Sound；
C. if meeting α ＞ γ, using important sampling, i.e., according to important function gt(Xt) produce new particleAnd makeWherein, the function prototype of important function is g (p)=N (p, σ), and N () is represented Gaussian Profile, σ are corresponding covariances；Status switch Xt={ x1... xt}；
2) the new weight for producing particle is calculated using observational equation：To each particle
A. the LBP features of target area are extracted, and utilize following formula
Calculate the weight p of particlelbp, wherein, d represents to calculate this feature point to separation using the SVM classifier that training obtains The distance of hyperplane；
B. profile clue and following formula are utilized
Calculate the weight p of particlecon, wherein, vmBe on gesture curve between control point m and nearest edge feature away from From θm∈ [0, pi/2] is the angle of the matched gradient direction of normal direction of control point m, and μ, σ are analogous to Gaussian Profile Control parameter, M be on gesture curve control count out；
C. the granular Weights Computing of two kinds of features of fusion is
Wherein, αlbpAnd αconFor the weight of different clues, can rule of thumb set or by setting verification collection total Obtain；Represented and posteriority is general for ensureing that as N → ∞ important sampling function does not interfere with the uniformity of particle assembly The correct estimation of rate.
3) end-state of target t moment can be estimated to obtain by particle assembly
3rd, repeat step 2 continues to track, or algorithm terminates.
S2：As shown in figure 3, according to the tracking result of instruction gesture, the two-dimensional position of finger tip can be obtained, and then can pass through Depth map directly obtains the depth information of finger tip.But interaction scenarios and illumination condition are in dynamic change under wearing environment, very The segmental defect of depth image is easy to cause, so as to cause the result of calculation of mistake.
To realize the calculating of finger tip three-dimensional position, problem is converted into and calculates human hand plane 1 and the friendship of finger tip straight line of image formation 2 Point.
Human hand plane 1 can be fitted to obtain by least square method to the obtained 3 d-dem point set of Hand Gesture Segmentation, It is marked as H：a1x+b1y+c1z+d1=0.
Finger tip straight line of image formation 2 is to represent the process that light reaches imaging point 4 from finger tip point through camera optical center 3, is marked It is denoted as L：A [R, t] (x, y, z, 1)T=λ (u, v, 1)T, wherein A is video camera internal reference matrix, and R is spin matrix, and t is translation square Battle array, λ is constant.Above-mentioned parameter can be determined by the strong calibration process of video camera.(u, v, l) represents finger tip point in imaging plane Coordinate, can be by indicating that gesture tracking algorithm is calculated.
S3：As shown in figure 4, virtual touchpad 1 is defined within 2 front certain distance of stereoscopic vision machine and perpendicular to taking the photograph The plane of camera optical axis.Assuming that D represents that virtual touchpad 1 arrives the distance of stereoscopic vision machine 2, d represents the depth of gesture finger tip 3 Value, touch event are converted into whether Rule of judgment d >=D sets up.Therefore, by setting suitable distance threshold D to can be achieved to touch Touch the reliable detection of event.
In addition, to provide intuitively visual feedback to wearer, it is automatic when system detectio is to finger tip contacts virtual touchpad In one green cross of finger tip image position display.
The foregoing is merely the preferred embodiment of the present invention, is not intended to limit the invention, for those skilled in the art For, the present invention can have various modifications and changes.All any modifications made within spirit and principles of the present invention, be equal Replace, improve etc., it should all be included in the protection scope of the present invention.

Claims (8)

  1. A kind of 1. virtual touchpad method based on stereoscopic vision, it is characterised in that comprise the following steps：
    S1, merges the tracking that a plurality of visual cues realize instruction gesture；
    S2, the three-dimensional position of finger tip is calculated with reference to depth information and camera imaging model；
    S3, provides the definition of virtual touchpad, and combines the three-dimensional position of finger tip and the depth information of virtual touchpad, realizes people Machine interactive operation.
  2. A kind of 2. virtual touchpad method based on stereoscopic vision according to claim 1, it is characterised in that the step S1 passes through CL-PGT (the i.e. Contour and Lbp based based on ICONDENSATION frame fusions profile Yu LBP features Pointing Gesture Tracking) algorithm realize instruction gesture tracking.
  3. A kind of 3. virtual touchpad method based on stereoscopic vision according to claim 1, it is characterised in that the step S2 is obtained the two-dimensional position of finger tip, is then calculated human hand plane and finger tip straight line of image formation using the tracking result for indicating gesture Intersection point.
  4. A kind of 4. virtual touchpad method based on stereoscopic vision according to claim 3, it is characterised in that the step Human hand plane can be fitted to obtain by least square method to the obtained 3 d-dem point set of Hand Gesture Segmentation in S2.
  5. A kind of 5. virtual touchpad method based on stereoscopic vision according to claim 3, it is characterised in that the step Finger tip point, can be by indicating that gesture tracking algorithm is calculated in the coordinate of imaging plane in S2.
  6. A kind of 6. virtual touchpad method based on stereoscopic vision according to claim 1, it is characterised in that the step The virtual touchpad of S3 is not actual hardware, is defined in stereoscopic vision machine front certain distance and perpendicular to video camera The plane of optical axis, can not the person of being worn see, but can person's of being worn " touching " arrive.
  7. A kind of 7. virtual touchpad method based on stereoscopic vision according to claim 6, it is characterised in that the step The decision procedure that " touching " operates in S3 is whether the depth value for judging gesture finger tip is greater than or equal to virtual touchpad to solid The distance of vision machine.
  8. A kind of 8. virtual touchpad method based on stereoscopic vision according to claim 1, it is characterised in that the step Man-machine interactive operation includes in S3：Finger tip clicks on the click event of touch pad analog mouse, and finger tip draws stroke on a touchpad Realize that handwriting input and finger tip draw the circle extract operation that closed curve realizes region on a touchpad.
CN201610898268.0A 2016-10-10 2016-10-10 A kind of virtual touchpad method based on stereoscopic vision Pending CN107918507A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610898268.0A CN107918507A (en) 2016-10-10 2016-10-10 A kind of virtual touchpad method based on stereoscopic vision

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610898268.0A CN107918507A (en) 2016-10-10 2016-10-10 A kind of virtual touchpad method based on stereoscopic vision

Publications (1)

Publication Number Publication Date
CN107918507A true CN107918507A (en) 2018-04-17

Family

ID=61892864

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610898268.0A Pending CN107918507A (en) 2016-10-10 2016-10-10 A kind of virtual touchpad method based on stereoscopic vision

Country Status (1)

Country Link
CN (1) CN107918507A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110471577A (en) * 2018-08-17 2019-11-19 中山叶浪智能科技有限责任公司 A kind of virtual touch control method of 360 degrees omnidirection, system, platform and storage medium
CN110471576A (en) * 2018-08-16 2019-11-19 中山叶浪智能科技有限责任公司 A kind of nearly screen touch method of single camera, system, platform and storage medium
CN111443831A (en) * 2020-03-30 2020-07-24 北京嘉楠捷思信息技术有限公司 Gesture recognition method and device
WO2021227628A1 (en) * 2020-05-14 2021-11-18 华为技术有限公司 Electronic device and interaction method therefor

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110471576A (en) * 2018-08-16 2019-11-19 中山叶浪智能科技有限责任公司 A kind of nearly screen touch method of single camera, system, platform and storage medium
CN110471576B (en) * 2018-08-16 2023-11-17 中山叶浪智能科技有限责任公司 Single-camera near-screen touch method, system, platform and storage medium
CN110471577A (en) * 2018-08-17 2019-11-19 中山叶浪智能科技有限责任公司 A kind of virtual touch control method of 360 degrees omnidirection, system, platform and storage medium
CN110471577B (en) * 2018-08-17 2023-08-22 中山叶浪智能科技有限责任公司 360-degree omnibearing virtual touch control method, system, platform and storage medium
CN111443831A (en) * 2020-03-30 2020-07-24 北京嘉楠捷思信息技术有限公司 Gesture recognition method and device
WO2021227628A1 (en) * 2020-05-14 2021-11-18 华为技术有限公司 Electronic device and interaction method therefor

Similar Documents

Publication Publication Date Title
WO2021129064A1 (en) Posture acquisition method and device, and key point coordinate positioning model training method and device
Kumar et al. A multimodal framework for sensor based sign language recognition
Sharp et al. Accurate, robust, and flexible real-time hand tracking
Wang et al. Real-time hand-tracking with a color glove
Pan et al. A real-time multi-cue hand tracking algorithm based on computer vision
Prisacariu et al. 3D hand tracking for human computer interaction
JP5117452B2 (en) Marker recognition method using dynamic threshold and learning system based on augmented reality using the same
CN110688965B (en) IPT simulation training gesture recognition method based on binocular vision
CN104992171A (en) Method and system for gesture recognition and man-machine interaction based on 2D video sequence
CN105739702A (en) Multi-posture fingertip tracking method for natural man-machine interaction
Wu et al. Robust fingertip detection in a complex environment
CN107918507A (en) A kind of virtual touchpad method based on stereoscopic vision
CN113220114A (en) Embedded non-contact elevator key interaction method integrating face recognition
CN104714650B (en) A kind of data inputting method and device
Yong et al. Emotion recognition in gamers wearing head-mounted display
Chen et al. Depth-based hand gesture recognition using hand movements and defects
Hammer et al. Robust hand tracking in realtime using a single head-mounted rgb camera
Brancati et al. Robust fingertip detection in egocentric vision under varying illumination conditions
Schlattmann et al. Markerless 4 gestures 6 DOF real‐time visual tracking of the human hand with automatic initialization
Shaker et al. Real-time finger tracking for interaction
Ghosh et al. Real-time 3d markerless multiple hand detection and tracking for human computer interaction applications
Yeh et al. Vision-based virtual control mechanism via hand gesture recognition
Shin et al. Welfare interface implementation using multiple facial features tracking for the disabled people
CN103995586B (en) Non- wearing based on virtual touch screen refers to gesture man-machine interaction method
Bellarbi et al. Hand gesture recognition using contour based method for tabletop surfaces

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20180417