WO2020009715A3 - Tracking user movements to control a skeleton model in a computer system - Google Patents

Tracking user movements to control a skeleton model in a computer system Download PDF

Info

Publication number
WO2020009715A3
WO2020009715A3 PCT/US2018/052092 US2018052092W WO2020009715A3 WO 2020009715 A3 WO2020009715 A3 WO 2020009715A3 US 2018052092 W US2018052092 W US 2018052092W WO 2020009715 A3 WO2020009715 A3 WO 2020009715A3
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
artificial neural
sensor module
computer system
control
Prior art date
Application number
PCT/US2018/052092
Other languages
French (fr)
Other versions
WO2020009715A2 (en
Inventor
Viktor Vladimirovich ERIVANTCEV
Alexander Sergeevich LOBANOV
Alexey Ivanovich Kartashov
Daniil Olegovich Goncharov
Original Assignee
Finch Technologies Ltd.
Ward, John P.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/973,137 external-priority patent/US11474593B2/en
Priority claimed from US15/996,389 external-priority patent/US10416755B1/en
Application filed by Finch Technologies Ltd., Ward, John P. filed Critical Finch Technologies Ltd.
Publication of WO2020009715A2 publication Critical patent/WO2020009715A2/en
Publication of WO2020009715A3 publication Critical patent/WO2020009715A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/005—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 with correlation of navigation data from several sources, e.g. map or contour matching
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/04—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by terrestrial means
    • G01C21/08—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by terrestrial means involving use of the magnetic field of the earth
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
    • G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
    • G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
    • G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
    • A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
    • A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Automation & Control Theory (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Environmental & Geological Engineering (AREA)
  • General Life Sciences & Earth Sciences (AREA)
  • Geology (AREA)
  • Computer Hardware Design (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Analysis (AREA)

Abstract

A system having sensor modules and a computing device. Each sensor module has an inertial measurement unit attached to a portion of a user. An artificial neural network is used to make predictions of orientations of parts of the user based on the measurements for the inertial-based modules. For example, the artificial neural network can be trained to predict orientations measured using an optical tracking system based on orientations measured using inertial measurement units and/or to prediction orientation measurements of some rigid parts in a kinematic chain based on orientation measurements of other rigid parts in the kinematic chain. For example, the sensor modules can include different subsets that share a common sensor module; and the artificial neural network can be used to combine different predictions, made separately from the different subsets, for the common sensor module.
PCT/US2018/052092 2018-05-07 2018-09-20 Tracking user movements to control a skeleton model in a computer system WO2020009715A2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US15/973,137 2018-05-07
US15/973,137 US11474593B2 (en) 2018-05-07 2018-05-07 Tracking user movements to control a skeleton model in a computer system
US15/996,389 2018-06-01
US15/996,389 US10416755B1 (en) 2018-06-01 2018-06-01 Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system

Publications (2)

Publication Number Publication Date
WO2020009715A2 WO2020009715A2 (en) 2020-01-09
WO2020009715A3 true WO2020009715A3 (en) 2020-02-20

Family

ID=68480436

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/052092 WO2020009715A2 (en) 2018-05-07 2018-09-20 Tracking user movements to control a skeleton model in a computer system

Country Status (2)

Country Link
CN (1) CN110456902A (en)
WO (1) WO2020009715A2 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111340211B (en) * 2020-02-19 2020-11-24 腾讯科技（深圳）有限公司 Training method of action control model, related device and storage medium
US11237632B2 (en) * 2020-03-03 2022-02-01 Finch Technologies Ltd. Ring device having an antenna, a touch pad, and/or a charging pad to control a computing device based on user motions
WO2021211002A1 (en) * 2020-04-13 2021-10-21 Общество С Ограниченной Ответственностью "Психо Технолоджи" Method for optical-inertial tracking of a user
JP7435357B2 (en) * 2020-08-18 2024-02-21 トヨタ自動車株式会社 Operating state monitoring system, training support system, operating state monitoring system control method, and control program

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5515477A (en) * 1991-04-22 1996-05-07 Sutherland; John Neural networks
US20120214594A1 (en) * 2011-02-18 2012-08-23 Microsoft Corporation Motion recognition
US20180020978A1 (en) * 2016-07-25 2018-01-25 Patrick Kaifosh System and method for measuring the movements of articulated rigid bodies

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100194879A1 (en) * 2007-07-10 2010-08-05 Koninklijke Philips Electronics N.V. Object motion capturing system and method
GB0720412D0 (en) * 2007-10-18 2007-11-28 Melexis Nv Combined mems accelerometer and gyroscope
KR101483713B1 (en) * 2008-06-30 2015-01-16 삼성전자 주식회사 Apparatus and Method for capturing a motion of human
US8279418B2 (en) * 2010-03-17 2012-10-02 Microsoft Corporation Raster scanning for depth detection
US9599634B2 (en) * 2012-12-03 2017-03-21 Vibrado Technologies, Inc. System and method for calibrating inertial measurement units
US9676098B2 (en) * 2015-07-31 2017-06-13 Heinz Hemken Data collection from living subjects and controlling an autonomous robot using the data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5515477A (en) * 1991-04-22 1996-05-07 Sutherland; John Neural networks
US20120214594A1 (en) * 2011-02-18 2012-08-23 Microsoft Corporation Motion recognition
US20180020978A1 (en) * 2016-07-25 2018-01-25 Patrick Kaifosh System and method for measuring the movements of articulated rigid bodies

Also Published As

Publication number Publication date
CN110456902A (en) 2019-11-15
WO2020009715A2 (en) 2020-01-09

Similar Documents

Publication Publication Date Title
WO2020009715A3 (en) Tracking user movements to control a skeleton model in a computer system
Fang et al. 3D human gesture capturing and recognition by the IMMU-based data glove
Harms et al. Ethos: Miniature orientation sensor for wearable human motion analysis
EP4327776A3 (en) A toothbrush system
Berret et al. The inactivation principle: mathematical solutions minimizing the absolute work and biological implications for the planning of arm movements
Wu et al. Learning to combine high variability with high precision: Lack of transfer to a different task
Abraham et al. Real time conversion of sign language to speech and prediction of gestures using Artificial Neural Network
DE502006009447D1 (en) ARTIFICIAL BATTERY MODULE SYSTEM
Khademi et al. Convolutional neural networks for environmentally aware locomotion mode recognition of lower-limb amputees
CN109549635B (en) Dynamic online measuring method for human body temperature and wearable equipment
CN110974242B (en) Gait abnormal degree evaluation method for wearable device and wearable device
Tomlinson et al. Dynamic dominance persists during unsupported reaching
Rezaei et al. Unobtrusive human fall detection system using mmwave radar and data driven methods
Choi et al. Development of a low-cost wearable sensing glove with multiple inertial sensors and a light and fast orientation estimation algorithm
AU2021332455A8 (en) Systems and methods for measuring, learning, and using emergent properties of complex adaptive systems
Qi et al. Lower extremity joint angle tracking with wireless ultrasonic sensors during a squat exercise
Hernandez et al. Force feasible set prediction with artificial neural network and musculoskeletal model
Cerfoglio et al. Machine learning-based estimation of ground reaction forces and knee joint kinetics from inertial sensors while performing a vertical drop jump
Scibilia et al. Human control model estimation in physical human–machine interaction: A survey
Matos-Carvalho et al. Sensitivity analysis of LSTM networks for fall detection wearable sensors
Pan et al. A sensor glove for the interaction with a nursing-care assistive robot
Jiang et al. A postural control model incorporating multisensory inputs for maintaining a musculoskeletal model in a stance posture
Obukhov et al. The model of the automatic control system for a treadmill based on neural networks
CN111079927B (en) Patella pain detection system based on extreme learning machine
Hu et al. mmPose-FK: A Forward Kinematics Approach to Dynamic Skeletal Pose Estimation Using mmWave Radars

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18925072

Country of ref document: EP

Kind code of ref document: A2

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18925072

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 18925072

Country of ref document: EP

Kind code of ref document: A2

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 16/07/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18925072

Country of ref document: EP

Kind code of ref document: A2