CN117503120B - Human body posture estimation method and system - Google Patents
Human body posture estimation method and system Download PDFInfo
- Publication number
- CN117503120B CN117503120B CN202311736609.0A CN202311736609A CN117503120B CN 117503120 B CN117503120 B CN 117503120B CN 202311736609 A CN202311736609 A CN 202311736609A CN 117503120 B CN117503120 B CN 117503120B
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- virtual
- matrix
- pose
- measurement unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 239000011159 matrix material Substances 0.000 claims abstract description 144
- 238000005259 measurement Methods 0.000 claims abstract description 83
- 230000009466 transformation Effects 0.000 claims abstract description 82
- 210000004197 pelvis Anatomy 0.000 claims abstract description 5
- 238000013507 mapping Methods 0.000 claims description 15
- 238000013519 translation Methods 0.000 claims description 15
- 210000001624 hip Anatomy 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 210000000689 upper leg Anatomy 0.000 claims description 4
- 210000000245 forearm Anatomy 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 2
- 230000009471 action Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000003814 drug Substances 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 208000027418 Wounds and injury Diseases 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000000386 athletic effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 244000309466 calf Species 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 238000007654 immersion Methods 0.000 description 2
- 208000014674 injury Diseases 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 1
- 230000005358 geomagnetic field Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1116—Determining posture transitions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1118—Determining activity level
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6813—Specially adapted to be attached to a specific body part
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Public Health (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Physiology (AREA)
- Dentistry (AREA)
- Geometry (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention provides a human body posture estimation method and a system, which relate to the technical field of space data processing and comprise the steps of arranging an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating a posture of the human body after the calibration of a heading direction (Z axis), determining an initial Z axis direction of the inertial sensor, and initializing the inertial sensor; under the standing posture, the rotation matrix is used as a virtual transformation matrix during the standing posture calibration, the posture is calibrated based on a Z axis, and the virtual transformation matrix is combined to complete the virtual human posture calibration; selecting virtual joints, respectively establishing a coordinate system, setting pelvis as a base coordinate system, constructing a resolving serial branch by using other joint coordinate systems with the base coordinate system as a starting point, respectively calculating pose matrixes of the joints through homogeneous coordinate transformation to obtain a space pose settlement result of the joints of the virtual person, and corresponding the real human body space pose and the virtual human space pose to obtain a human body pose result.
Description
Technical Field
The present invention relates to spatial data processing technology, and in particular, to a human body posture estimation method and system.
Background
In order to monitor the motion data of each joint of a human body, a motion gesture simulation algorithm is researched, a motion gesture mapping model relation between real human body gesture information (namely sensor information) and a virtual human body is required to be established, and the real human joint gestures are in one-to-one correspondence with the joint gestures of the virtual human body model, so that real-time joint gesture, speed and force information is provided for a subsequent balance rehabilitation training platform control system;
CN201911159199.1, discloses a human body posture estimating method, comprising: acquiring a single image from an original graph by adopting a human body detection network, wherein the original graph comprises at least one single image; performing key point detection on each single image by adopting a human body posture estimation network to obtain a distribution heat map of the positions of each key point; acquiring the overlapping degree of the frame of each single image and the frames of other single images in the original graph, and judging whether the single images have shielding problems or not; if the single image has no shielding problem, obtaining the final coordinates of key points in the single image by using the distribution heat map; if the single image has the shielding problem, forming all the distribution heat maps in the single image into a map form, optimizing each distribution heat map through a map neural network, and obtaining the final coordinates of key points in the single image from the optimized distribution heat maps. The overall structure information of the graph is fully extracted by using the graph neural network, so that the correction effect on the positions of the key points is realized, and the detection accuracy is improved;
in summary, in the prior art, the algorithm for estimating the human body posture based on the graph structure model is based on the common optical image, is interfered by the ambient light and the color, and is blocked by a plurality of human bodies in the image, so that the posture estimation accuracy is insufficient, which is inferior to the accuracy of the data of the sensor, in addition, the existing posture sensor, the heading direction, namely the Z-axis direction, is mostly determined by means of the geomagnetic field, but is easily interfered by the electromagnetic interference of the electronic equipment when in use, so that the sensor Z-axis angle is inaccurate to measure and has huge drift. The application of the present invention solves at least some of the problems of the prior art.
Disclosure of Invention
The embodiment of the invention provides a human body posture estimation method and system, which at least can solve part of problems in the prior art.
In a first aspect of an embodiment of the present invention,
provided is a human body posture estimation method, including:
setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the calibrated gesture of the inertial measurement unit in the heading direction (Z axis) under a world coordinate system, determining the initial Z axis direction of the inertial sensor, and initializing the inertial sensor;
under standing posture, the standing posture calibration time rotation matrix of the inertial measurement unit is used as a virtual transformation matrix, and based on the Z-axis calibration post-posture of the inertial measurement unit, the virtual transformation matrix is combined to realize mapping from a real space to a virtual space, so that virtual human posture calibration is completed;
selecting 15 joints of a virtual human head, a chest, a waist, a left large arm, a right large arm, a small arm, a hand, a thigh, a shank and a foot, respectively establishing a coordinate system, setting a pelvis as a basic coordinate system, taking the basic coordinate system as a starting point for other 14 coordinate systems, constructing a resolving serial branch, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose settlement result of the virtual human joint, and corresponding a real human space pose and the virtual human space pose to obtain a human body pose result.
In an alternative embodiment of the present invention,
setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor, wherein the method comprises the following steps of:
setting an inertial sensor, wherein the inertial sensor comprises a gyroscope and an accelerometer, and combining the gyroscope and the accelerometer into an inertial measurement unit;
under a world coordinate system, determining a Z-axis calibration transformation matrix according to the Z-axis calibration time gesture matrix of the inertial measurement unit and the unified gesture matrix after Z-axis calibration;
and according to the original rotation matrix of the inertial measurement unit, the Z-axis calibration transformation matrix is combined, the Z-axis calibration post-rotation matrix of the inertial measurement unit is calculated, the initial Z-axis direction of the inertial sensor is determined, and the inertial sensor is initialized.
In an alternative embodiment of the present invention,
further comprises:
according to the Z-axis calibration time attitude matrix and the Z-axis calibration post unified attitude matrix of the inertial measurement unit, the Z-axis calibration transformation matrix is calculated by multiplying the unified attitude matrix by the inverse of the Z-axis calibration time attitude matrix, and the formula is as follows:
wherein ZR i The Z-axis calibration transformation matrix of the ith joint coordinate system is represented, w represents the coordinate system of the world as a reference, ZA represents the Z-axis calibration,indicating a unified posture matrix after Z-axis calibration, wherein Zp indicates +.>Representing the inverse of the gesture matrix during Z-axis calibration of the ith joint coordinate system;
the original rotation matrix of the inertial measurement unit is multiplied by the Z-axis calibration transformation matrix, and the Z-axis calibrated rotation matrix of the inertial measurement unit is calculated, wherein the formula is as follows:
wherein,the Z-axis post-calibration rotation matrix representing the ith joint coordinate system, w R i representing the original rotation matrix of the ith joint coordinate system.
In an alternative embodiment of the present invention,
under standing posture, the standing posture calibration time rotation matrix of the inertial measurement unit is used as a virtual transformation matrix, the posture after Z-axis calibration based on the inertial measurement unit is combined with the virtual transformation matrix, the mapping from a real space to a virtual space is realized, and the completion of the virtual person posture calibration comprises:
under the standing posture, the rotation matrix is calibrated by the standing posture and is used as a virtual transformation matrix, and the formula is as follows:
wherein,virtual transformation matrix representing the ith joint coordinate system relative to the world coordinate system, < ->A rotation matrix when standing posture is marked is represented;
and multiplying the gesture right after the Z-axis calibration by the inverse of the virtual transformation matrix, and calculating to obtain a virtual rotation matrix, wherein the formula is as follows:
wherein, V R i representing the virtual rotation matrix of the object,representing the inverse of the virtual transformation matrix.
In an alternative embodiment of the present invention,
selecting virtual human joints, respectively establishing a coordinate system, setting a base coordinate system, constructing a resolving serial branch by using the base coordinate system as a starting point by other coordinate systems, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose resolving result of the virtual human joints, and corresponding a real human space pose and the virtual human space pose to obtain a human body pose result, wherein the obtaining the human body pose result comprises the following steps:
selecting 15 joints of a virtual person, establishing 15 coordinate systems corresponding to the 15 joints, setting a hip coordinate system as a base coordinate system, constructing 5 resolving serial branches by taking the base coordinate system as a starting point in other coordinate systems, and respectively calculating the resolving serial branches;
the homogeneous coordinate transformation is provided with two forms which are respectively expressed as a pure translation type and a pure rotation type, and based on the homogeneous coordinate transformation, a coordinate system is wound and moved to construct the pose matrix representation of each joint;
and (5) corresponding the real human body space pose and the virtual human body space pose to obtain a human body pose result.
In an alternative embodiment of the present invention,
further comprises:
the homogeneous coordinate transformation is respectively expressed as a pure translational type and a pure rotational type, wherein the pure translational type is as follows:
wherein,representing a translation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing translation of the origin of the j-coordinate system relative to the origin of the I-coordinate system, I 3×3 Representing a 3 x 3 identity matrix;
the pure rotation is as follows:
wherein,representing a rotation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing a rotation of the j-coordinate system relative to the i-coordinate system,/-coordinate system>Representing a 3 x 3 rotation matrix;
solving the pose matrix of each joint, wherein the formula is as follows:
wherein, V T i representing a pose matrix from the virtual coordinate system V to the ith joint coordinate system, V P 1 representing a translation vector from the virtual coordinate system V to the base coordinate system, V R n-1 representing the rotation from the virtual coordinate system V to the n-1 th joint coordinate system, V P n represents translation vector from virtual coordinate system V to nth joint coordinate system, rot # V R n-1 ) ' represents the inverse of the rotation matrix from the virtual coordinate system V to the n-1 th joint coordinate system, V R i representing rotation from the virtual coordinate system V to the ith joint coordinate system.
In a second aspect of an embodiment of the present invention,
there is provided a human body posture estimation system including:
the first unit is used for setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor;
the second unit is used for rotating the matrix in the standing posture calibration of the inertial measurement unit to serve as a virtual transformation matrix, and based on the Z-axis calibrated posture of the inertial measurement unit, mapping from a real space to a virtual space is realized by combining the virtual transformation matrix, so that the virtual human posture calibration is completed;
and the third unit is used for selecting 15 joints of the virtual head, chest, waist, left and right large arms, small arms, hands, thighs, calves and feet, respectively establishing a coordinate system, setting pelvis as a base coordinate system, taking the base coordinate system as a starting point in other 14 coordinate systems, constructing a resolving serial branch, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation, obtaining a space pose settlement result of the virtual joints, and corresponding the real human space pose and the virtual human space pose to obtain a human body pose result.
In a third aspect of an embodiment of the present invention,
there is provided an electronic device including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
By combining gyroscope and accelerometer data, high-precision gesture measurement and tracking can be realized, and the method is very useful for applications such as virtual reality, augmented reality, motion analysis and medical fields. The mounting of the inertial measurement unit at the joints of the human body can help to monitor and analyze the movements of the joints, which has an important role in rehabilitation medicine, athletic training and biomedical research. By means of the Z-axis calibration and the initial Z-axis direction determination, accurate attitude calibration of the sensor can be achieved, which is very useful for improving measurement accuracy when the sensor is used in different environments.
By mapping the motion of the real space to the virtual space, the user can interact more naturally in the virtual environment, and the immersion of the virtual reality experience is improved; virtual person pose calibration this technique allows the pose of a virtual person or object to be calibrated so that it mimics the actions of a user in the real world. By mapping actual movements to virtual space, movement analysis can be performed for physical training, rehabilitation medicine, and biomedical research, which helps monitor athlete's skills and movement processes to improve performance and reduce risk of injury.
By using multiple joints and coordinate systems, and homogeneous coordinate transformation of pure translational and pure rotational, more realistic imitation of the pose of a virtual person can be achieved, and the motion and pose of the virtual person in the virtual environment will be more realistic and fluent. The accuracy of pose matching can be improved by using multiple joints and coordinate systems, as well as different types of coordinate transformations. By matching the gestures of the virtual person and the real person, more natural action interaction can be realized.
The existing 6-axis inertial sensor has the problems of unfixed gesture of the starting course direction (Z axis) and the problems of complicated operation and low calibration efficiency of a single sensor calibration method, while the existing 9-axis inertial sensor has the problems of large electromagnetic interference, easy drift and difficult detection and correction of the course direction, and the problem is solved by the centralized batch Z axis calibration algorithm under comprehensive consideration; the existing virtual person gesture calibration is complex in action (11 preset actions are required to be completed), and the problem of low matching quality is solved through the standing gesture.
Drawings
FIG. 1 is a schematic flow chart of a human body posture estimation method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a coordinate system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a human body posture estimation system according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 1 is a schematic flow chart of a human body posture estimation method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
s101, setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor;
in an alternative embodiment of the present invention,
setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor, wherein the method comprises the following steps of:
setting an inertial sensor, wherein the inertial sensor comprises a gyroscope and an accelerometer, and combining the gyroscope and the accelerometer into an inertial measurement unit;
under a world coordinate system, determining a Z-axis calibration transformation matrix according to the Z-axis calibration time gesture matrix of the inertial measurement unit and the unified gesture matrix after Z-axis calibration;
and according to the original rotation matrix of the inertial measurement unit, the Z-axis calibration transformation matrix is combined, the Z-axis calibration post-rotation matrix of the inertial measurement unit is calculated, the initial Z-axis direction of the inertial sensor is determined, and the inertial sensor is initialized.
For example, the data of gyroscopes, accelerometers may be calibrated to eliminate sensor errors. The calibrated sensor data are combined into a comprehensive inertial measurement unit. Typically, this involves using a pose estimation algorithm (e.g., a Kalman filter or quaternion filter) to fuse the gyroscope, accelerometer data to obtain more accurate pose information.
The inertial measurement unit is mounted at a human joint in order to measure the movement and posture of the joint.
In the absence of human motion, the initial pose of the inertial measurement unit is recorded (typically by reading the output of the sensor). In the case of human body movement, sensor data is continuously acquired and a posture estimation algorithm is used to calculate a posture after Z-axis calibration. And comparing the initial gesture determined under the condition of no human body movement with the gesture determined under the condition of human body movement, and calculating a Z-axis calibration transformation matrix.
And multiplying the initial attitude rotation matrix by the Z-axis calibration transformation matrix to determine the initial Z-axis direction of the inertial sensor. The initial Z-axis direction of the inertial sensor is used for subsequent motion and attitude measurements. This direction can be used to identify and track the movement of the joint.
In an alternative embodiment of the present invention,
further comprises:
according to the Z-axis calibration time attitude matrix and the Z-axis calibration post unified attitude matrix of the inertial measurement unit, the Z-axis calibration transformation matrix is calculated by multiplying the unified attitude matrix by the inverse of the Z-axis calibration time attitude matrix, and the formula is as follows:
wherein ZR i The Z-axis calibration transformation matrix of the ith joint coordinate system is represented, w represents the coordinate system of the world as a reference, ZA represents the Z-axis calibration,indicating a unified posture matrix after Z-axis calibration, wherein Zp indicates +.>Representing the inverse of the gesture matrix during Z-axis calibration of the ith joint coordinate system;
the original rotation matrix of the inertial measurement unit is multiplied by the Z-axis calibration transformation matrix, and the Z-axis calibrated rotation matrix of the inertial measurement unit is calculated, wherein the formula is as follows:
wherein,the Z-axis post-calibration rotation matrix representing the ith joint coordinate system, w R i representing the original rotation matrix of the ith joint coordinate system.
It will be appreciated that by combining gyroscope, accelerometer data, high accuracy attitude measurement and tracking can be achieved, which is very useful for applications in virtual reality, augmented reality, motion analysis, and medical fields, among others. The mounting of the inertial measurement unit at the joints of the human body can help to monitor and analyze the movements of the joints, which has an important role in rehabilitation medicine, athletic training and biomedical research. By means of the Z-axis calibration and the initial Z-axis direction determination, accurate attitude calibration of the sensor can be achieved, which is very useful for improving measurement accuracy when the sensor is used in different environments.
S102, under a standing posture, using a standing posture calibration time rotation matrix of the inertial measurement unit as a virtual transformation matrix, and combining the virtual transformation matrix based on the Z-axis calibrated posture of the inertial measurement unit to realize mapping from a real space to a virtual space so as to complete the posture calibration of a virtual person;
in an alternative embodiment of the present invention,
under standing posture, the standing posture calibration time rotation matrix of the inertial measurement unit is used as a virtual transformation matrix, the posture after Z-axis calibration based on the inertial measurement unit is combined with the virtual transformation matrix, the mapping from a real space to a virtual space is realized, and the completion of the virtual person posture calibration comprises:
under the standing posture, the rotation matrix is calibrated by the standing posture and is used as a virtual transformation matrix, and the formula is as follows:
wherein,virtual transformation matrix representing the ith joint coordinate system relative to the world coordinate system, < ->A rotation matrix when standing posture is marked is represented;
and multiplying the gesture right after the Z-axis calibration by the inverse of the virtual transformation matrix, and calculating to obtain a virtual rotation matrix, wherein the formula is as follows:
wherein, V R i representing the virtual rotation matrix of the object,representing the inverse of the virtual transformation matrix.
Illustratively, a person to be tested can stand at the position where the inertial measurement unit is installed, so that the standing posture is ensured to be stable; recording sensor data in a standing posture by using an inertial measurement unit, wherein the sensor data comprises posture information and a rotation matrix; this virtual transformation matrix will be used to map the motion of the real space to the virtual space, based on the rotation matrix at stance.
Sensor data of the inertial measurement unit is continuously acquired, including attitude information and a rotation matrix. And mapping the gesture information and the rotation matrix acquired in real time into a virtual space according to the generated virtual transformation matrix. This will translate the motion in real space into motion of a virtual person or object.
A virtual person or object is displayed in the virtual space, and its initial pose is determined by the virtual transformation matrix and the pose information acquired in real time. The tested person simulates the actions of the virtual person or the object and records the gesture information of the virtual person or the object in real time. And adjusting the gesture of the virtual person or the object by using the recorded gesture information so as to realize the gesture calibration of the virtual person. The accuracy of the calibration can be assessed by comparing the pose of the virtual person or object and the person under test, and the necessary calibration is performed.
It can be appreciated that by mapping the motion of the real space to the virtual space, the user can interact more naturally in the virtual environment, improving the immersion of the virtual reality experience; virtual person pose calibration this technique allows the pose of a virtual person or object to be calibrated so that it mimics the actions of a user in the real world. By mapping actual movements to virtual space, movement analysis can be performed for physical training, rehabilitation medicine, and biomedical research, which helps monitor athlete's skills and movement processes to improve performance and reduce risk of injury.
S103, selecting virtual human joints, respectively establishing a coordinate system, setting a base coordinate system, constructing a resolving serial branch by using the base coordinate system as a starting point by other coordinate systems, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose settlement result of the virtual human joints, and corresponding the real human space pose and the virtual human space pose to obtain a human body pose result.
Illustratively, a local coordinate system is established for each joint of the virtual person. These coordinate systems are typically associated with the skeletal structure of the virtual human model, one for each joint. A basic coordinate system, typically the world coordinate system, is selected as the reference coordinate system for the entire system. By correlating the skeleton structure of the virtual human model, a solution series branch is constructed, which is a tree structure, each joint is connected to a parent node, and there may be a plurality of child nodes representing the relationship between the joints. For each joint, its parent node, and the relative motion relationship with the parent node, is determined, typically using a rotation matrix or quaternion.
Fig. 2 is a schematic diagram of a coordinate system according to an embodiment of the present invention, as shown in fig. 2, the coordinate system according to an embodiment of the present invention may include any one of an IMU (inertial measurement unit ) coordinate system, northeast day coordinate system, virtual space coordinate system, and joint coordinate system.
And (3) starting from the base coordinate system, calculating the pose matrix of each joint step by step according to the structure of the calculated serial branches. This may be accomplished by a homogeneous coordinate transformation, multiplying the pose matrix of each joint with the pose matrix of the parent node. Repeating the steps until the pose matrixes of all joints are calculated.
For each virtual human joint, the final pose matrix represents its spatial pose in the virtual human model, i.e. its pose relative to the coordinate system of the virtual human model.
The pose data of a real human body is obtained, typically by a sensor (e.g., inertial measurement unit) or other pose capturing device. The pose between the virtual person and the real human body is matched by using the spatial pose of the virtual person joint and the pose data of the real human body, which can be realized by comparing the pose difference between the virtual person joint and the real human body joint. According to the matching result, the gesture synchronization of the virtual person and the real human body can be realized, so that the action and the gesture of the real human body can be simulated in the virtual environment.
In an alternative embodiment of the present invention,
selecting virtual human joints, respectively establishing a coordinate system, setting a base coordinate system, constructing a resolving serial branch by using the base coordinate system as a starting point by other coordinate systems, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose resolving result of the virtual human joints, and corresponding a real human space pose and the virtual human space pose to obtain a human body pose result, wherein the obtaining the human body pose result comprises the following steps:
selecting 15 joints of a virtual person, establishing 15 coordinate systems corresponding to the 15 joints, setting a hip coordinate system as a base coordinate system, constructing 5 resolving serial branches by taking the base coordinate system as a starting point in other coordinate systems, and respectively calculating the resolving serial branches;
the homogeneous coordinate transformation is provided with two forms which are respectively expressed as a pure translation type and a pure rotation type, and based on the homogeneous coordinate transformation, a coordinate system is wound and moved to construct the pose matrix representation of each joint;
and (5) corresponding the real human body space pose and the virtual human body space pose to obtain a human body pose result.
By using multiple joints and coordinate systems, and homogeneous coordinate transformation of pure translational and pure rotational, more realistic imitation of the pose of a virtual person can be achieved, and the motion and pose of the virtual person in the virtual environment will be more realistic and fluent. The accuracy of pose matching can be improved by using multiple joints and coordinate systems, as well as different types of coordinate transformations. By matching the gestures of the virtual person and the real person, more natural action interaction can be realized.
In an alternative embodiment of the present invention,
further comprises:
the homogeneous coordinate transformation is respectively expressed as a pure translational type and a pure rotational type, wherein the pure translational type is as follows:
wherein,representing a translation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing translation of the origin of the j-coordinate system relative to the origin of the I-coordinate system, I 3×3 Representing a 3 x 3 identity matrix;
the pure rotation is as follows:
wherein,representing a rotation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing j sittingRotation of the frame relative to the i frame, +.>Representing a 3 x 3 rotation matrix;
solving the pose matrix of each joint, wherein the formula is as follows:
wherein, V T i representing a pose matrix from the virtual coordinate system V to the ith joint coordinate system, V P 1 representing a translation vector from the virtual coordinate system V to the base coordinate system, V R n-1 representing the rotation from the virtual coordinate system V to the n-1 th joint coordinate system, V P n represents translation vector from virtual coordinate system V to nth joint coordinate system, rot # V R n-1 ) ' represents the inverse of the rotation matrix from the virtual coordinate system V to the n-1 th joint coordinate system, V R i representing rotation from the virtual coordinate system V to the ith joint coordinate system.
Fig. 3 is a schematic structural diagram of a human body posture estimation system according to an embodiment of the present invention, as shown in fig. 3, the system includes:
the first unit is used for setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor;
the second unit is used for rotating the matrix in the standing posture calibration of the inertial measurement unit to serve as a virtual transformation matrix, and based on the Z-axis calibrated posture of the inertial measurement unit, mapping from a real space to a virtual space is realized by combining the virtual transformation matrix, so that the virtual human posture calibration is completed;
and the third unit is used for selecting 15 joints of the virtual head, chest, waist, left and right large arms, small arms, hands, thighs, calves and feet, respectively establishing a coordinate system, setting pelvis as a base coordinate system, taking the base coordinate system as a starting point in other 14 coordinate systems, constructing a resolving serial branch, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation, obtaining a space pose settlement result of the virtual joints, and corresponding the real human space pose and the virtual human space pose to obtain a human body pose result.
In a third aspect of an embodiment of the present invention,
there is provided an electronic device including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
The present invention may be a method, apparatus, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for performing various aspects of the present invention.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (6)
1. A human body posture estimation method, characterized by comprising:
an inertial sensor is arranged, the inertial sensor is combined into an inertial measurement unit, the inertial measurement unit is installed at a joint of a human body, the Z-axis calibrated posture of the inertial measurement unit is calculated under a world coordinate system, the initial Z-axis direction of the inertial sensor is determined, and the inertial sensor is initialized;
under standing posture, the standing posture calibration time rotation matrix of the inertial measurement unit is used as a virtual transformation matrix, and based on the Z-axis calibration post-posture of the inertial measurement unit, the virtual transformation matrix is combined to realize mapping from a real space to a virtual space, so that virtual human posture calibration is completed;
selecting 15 joints of a virtual human head, a chest, a waist, a left large arm, a right large arm, a forearm, a hand, a thigh, a shank and a foot, respectively establishing a coordinate system, setting a pelvis as a basic coordinate system, taking the basic coordinate system as a starting point of the other 14 joint coordinate systems, constructing a resolving serial branch, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose settlement result of the virtual human joint, and corresponding a real human space pose and the virtual human space pose to obtain a human body pose result;
setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor, wherein the method comprises the following steps of:
setting an inertial sensor, wherein the inertial sensor comprises a gyroscope and an accelerometer, and combining the gyroscope and the accelerometer into an inertial measurement unit;
under a world coordinate system, determining a Z-axis calibration transformation matrix according to the Z-axis calibration time gesture matrix of the inertial measurement unit and the unified gesture matrix after Z-axis calibration;
according to the original rotation matrix of the inertial measurement unit and the Z-axis calibration transformation matrix, calculating a Z-axis calibration post-rotation matrix of the inertial measurement unit, determining an initial Z-axis direction of the inertial sensor, and initializing the inertial sensor;
selecting virtual human joints, respectively establishing a coordinate system, setting a base coordinate system, constructing a resolving serial branch by using the base coordinate system as a starting point by other coordinate systems, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose resolving result of the virtual human joints, and corresponding a real human space pose and the virtual human space pose to obtain a human body pose result, wherein the obtaining the human body pose result comprises the following steps:
selecting 15 joints of a virtual person, establishing 15 coordinate systems corresponding to the 15 joints, setting a hip coordinate system as a base coordinate system, constructing 5 resolving serial branches by taking the base coordinate system as a starting point in other coordinate systems, and respectively calculating the resolving serial branches;
the homogeneous coordinate transformation is provided with two forms which are respectively expressed as a pure translation type and a pure rotation type, and based on the homogeneous coordinate transformation, a coordinate system is wound and moved to construct the pose matrix representation of each joint;
corresponding the real human body space pose and the virtual human space pose to obtain a human body pose result;
further comprises:
the homogeneous coordinate transformation is respectively expressed as a pure translational type and a pure rotational type, wherein the pure translational type is as follows:
wherein,representing a translation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing translation of the origin of the j-coordinate system relative to the origin of the I-coordinate system, I 3×3 Representing a 3 x 3 identity matrix;
the pure rotation is as follows:
wherein,representing a rotation transformation from the j-coordinate system to the i-coordinate system,/-coordinate system>Representing a rotation of the j-coordinate system relative to the i-coordinate system,/-coordinate system>Representing a 3 x 3 rotation matrix;
solving the pose matrix of each joint, wherein the formula is as follows:
wherein, V T i representing a pose matrix from the virtual coordinate system V to the ith joint coordinate system, V P 1 representing a translation vector from the virtual coordinate system V to the base coordinate system, V R n-1 representing the rotation from the virtual coordinate system V to the n-1 th joint coordinate system, V P n represents translation vector from virtual coordinate system V to nth joint coordinate system, rot # V R n-1 ) ' represents the inverse of the rotation matrix from the virtual coordinate system V to the n-1 th joint coordinate system, V R i representing rotation from the virtual coordinate system V to the ith joint coordinate system.
2. The method as recited in claim 1, further comprising:
according to the Z-axis calibration time attitude matrix and the Z-axis calibration post unified attitude matrix of the inertial measurement unit, the Z-axis calibration transformation matrix is calculated by multiplying the unified attitude matrix by the inverse of the Z-axis calibration time attitude matrix, and the formula is as follows:
wherein ZR i The Z-axis calibration transformation matrix of the ith joint coordinate system is represented, w represents the coordinate system of the world as a reference, ZA represents the Z-axis calibration,indicating a unified posture matrix after Z-axis calibration, wherein Zp indicates +.>Representing the inverse of the gesture matrix during Z-axis calibration of the ith joint coordinate system;
the original rotation matrix of the inertial measurement unit is multiplied by the Z-axis calibration transformation matrix, and the Z-axis calibrated rotation matrix of the inertial measurement unit is calculated, wherein the formula is as follows:
wherein,the Z-axis post-calibration rotation matrix representing the ith joint coordinate system, w R i representing the original rotation matrix of the ith joint coordinate system.
3. The method of claim 1, wherein in the standing posture, using the standing posture calibration time rotation matrix of the inertial measurement unit as a virtual transformation matrix, based on the Z-axis calibrated posture of the inertial measurement unit, and combining the virtual transformation matrix to realize mapping from a real space to a virtual space, and completing the virtual person posture calibration comprises:
under the standing posture, the rotation matrix is calibrated by the standing posture and is used as a virtual transformation matrix, and the formula is as follows:
wherein,virtual transformation matrix representing the ith joint coordinate system relative to the world coordinate system, < ->A rotation matrix when standing posture is marked is represented;
and multiplying the gesture right after the Z-axis calibration by the inverse of the virtual transformation matrix, and calculating to obtain a virtual rotation matrix, wherein the formula is as follows:
wherein, V R i representing the virtual rotation matrix of the object,representing the inverse of the virtual transformation matrix.
4. A human body posture estimation system, characterized by being adapted to implement the human body posture estimation method of any of the preceding claims 1-3, comprising:
the first unit is used for setting an inertial sensor, combining the inertial sensor into an inertial measurement unit, installing the inertial measurement unit at a joint of a human body, calculating the Z-axis calibrated posture of the inertial measurement unit under a world coordinate system, determining the initial Z-axis direction of the inertial sensor, and initializing the inertial sensor;
the second unit is used for rotating the matrix in the standing posture calibration of the inertial measurement unit to serve as a virtual transformation matrix, and based on the Z-axis calibrated posture of the inertial measurement unit, mapping from a real space to a virtual space is realized by combining the virtual transformation matrix, so that the virtual human posture calibration is completed;
and the third unit is used for selecting the virtual human joints, respectively establishing a coordinate system, setting a base coordinate system, constructing a resolving serial branch by using the base coordinate system as a starting point by other coordinate systems, respectively calculating pose matrixes of all joints in the resolving serial branch through homogeneous coordinate transformation to obtain a space pose settlement result of the virtual human joints, and corresponding the real human space pose and the virtual human space pose to obtain a human body pose result.
5. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any of claims 1 to 3.
6. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311736609.0A CN117503120B (en) | 2023-12-18 | 2023-12-18 | Human body posture estimation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311736609.0A CN117503120B (en) | 2023-12-18 | 2023-12-18 | Human body posture estimation method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117503120A CN117503120A (en) | 2024-02-06 |
| CN117503120B true CN117503120B (en) | 2024-04-16 |
Family
ID=89751414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311736609.0A Active CN117503120B (en) | 2023-12-18 | 2023-12-18 | Human body posture estimation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117503120B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104117206A (en) * | 2014-08-01 | 2014-10-29 | 天津恒威先创科技发展有限公司 | Method for realizing virtual reality all-directional action based on action capturing system |
| CN107898466A (en) * | 2017-10-17 | 2018-04-13 | 深圳大学 | A kind of limb motion based on inertial sensor catches system and method |
| CN109949228A (en) * | 2019-03-14 | 2019-06-28 | 北京航空航天大学 | A kind of on-line proving device and method of the AR glasses of optical perspective |
| CN110097639A (en) * | 2019-03-18 | 2019-08-06 | 北京工业大学 | A kind of 3 D human body Attitude estimation method |
| CN113253748A (en) * | 2021-06-08 | 2021-08-13 | 深圳市优必选科技股份有限公司 | State estimation method, device, equipment and medium for parallel humanoid robot |
| CN113793360A (en) * | 2021-08-31 | 2021-12-14 | 大连理工大学 | Three-dimensional human body reconstruction method based on inertial sensing technology |
| CN114974506A (en) * | 2022-05-17 | 2022-08-30 | 重庆大学 | Human body posture data processing method and system |
| CN115373511A (en) * | 2022-07-12 | 2022-11-22 | 浙江大学 | Human motion digital twinning construction method based on inertial motion capture technology |
| CN116563497A (en) * | 2023-02-10 | 2023-08-08 | 上海科技大学 | Virtual person driving method, device, equipment and readable storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101214227B1 (en) * | 2011-05-25 | 2012-12-20 | 한국과학기술연구원 | method of motion tracking. |
| US11954801B2 (en) * | 2022-04-11 | 2024-04-09 | Microsoft Technology Licensing, Llc | Concurrent human pose estimates for virtual representation |
-
2023
- 2023-12-18 CN CN202311736609.0A patent/CN117503120B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104117206A (en) * | 2014-08-01 | 2014-10-29 | 天津恒威先创科技发展有限公司 | Method for realizing virtual reality all-directional action based on action capturing system |
| CN107898466A (en) * | 2017-10-17 | 2018-04-13 | 深圳大学 | A kind of limb motion based on inertial sensor catches system and method |
| CN109949228A (en) * | 2019-03-14 | 2019-06-28 | 北京航空航天大学 | A kind of on-line proving device and method of the AR glasses of optical perspective |
| CN110097639A (en) * | 2019-03-18 | 2019-08-06 | 北京工业大学 | A kind of 3 D human body Attitude estimation method |
| CN113253748A (en) * | 2021-06-08 | 2021-08-13 | 深圳市优必选科技股份有限公司 | State estimation method, device, equipment and medium for parallel humanoid robot |
| CN113793360A (en) * | 2021-08-31 | 2021-12-14 | 大连理工大学 | Three-dimensional human body reconstruction method based on inertial sensing technology |
| CN114974506A (en) * | 2022-05-17 | 2022-08-30 | 重庆大学 | Human body posture data processing method and system |
| CN115373511A (en) * | 2022-07-12 | 2022-11-22 | 浙江大学 | Human motion digital twinning construction method based on inertial motion capture technology |
| CN116563497A (en) * | 2023-02-10 | 2023-08-08 | 上海科技大学 | Virtual person driving method, device, equipment and readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 基于惯性传感器的人体动作捕捉***;吴天强等;自动化与信息工程;20230628;第44卷(第3期);第34-39页 * |
| 基于惯性测量***的人体运动捕捉技术;纪俐;姬晓飞;;计算机工程与设计;20160416(第04期);第1098-1101页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117503120A (en) | 2024-02-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Robertson et al. | Research methods in biomechanics | |
| Borst et al. | Realistic virtual grasping | |
| CN101579238B (en) | Human motion capture three dimensional playback system and method thereof | |
| Zhang et al. | A novel hierarchical information fusion method for three-dimensional upper limb motion estimation | |
| Piao et al. | Real-time visual–inertial SLAM based on adaptive keyframe selection for mobile AR applications | |
| Chen et al. | Real‐time human motion capture driven by a wireless sensor network | |
| JP2001504605A (en) | Method for tracking and displaying a user's location and orientation in space, method for presenting a virtual environment to a user, and systems for implementing these methods | |
| RU2107328C1 (en) | Method for tracing and displaying of position and orientation of user in three-dimensional space and device which implements said method | |
| CN109781104B (en) | Motion attitude determination and positioning method and device, computer equipment and medium | |
| Miller | A computer simulation model of the airborne phase of diving | |
| CN114417738B (en) | Sparse IMU real-time human body motion capture and joint stress prediction method and system | |
| CN114663463A (en) | Method, system, device, electronic device and storage medium for measuring joint mobility | |
| KR102150172B1 (en) | Relative movement based motion recognition method and apparatus | |
| Scherfgen et al. | Estimating the pose of a medical manikin for haptic augmentation of a virtual patient in mixed reality training | |
| WO2022221249A1 (en) | Video-based hand and ground reaction force determination | |
| Callejas-Cuervo et al. | Capture and analysis of biomechanical signals with inertial and magnetic sensors as support in physical rehabilitation processes | |
| CN115919250A (en) | Human dynamic joint angle measuring system | |
| Chakravarthi et al. | Real-time human motion tracking and reconstruction using IMU sensors | |
| CN117503120B (en) | Human body posture estimation method and system | |
| CN116740618A (en) | Motion video action evaluation method, system, computer equipment and medium | |
| Farsoni et al. | A low-cost high-fidelity ultrasound simulator with the inertial tracking of the probe pose | |
| US10549426B2 (en) | Method for estimating movement of a poly-articulated mass object | |
| JP2024508782A (en) | Methods to improve markerless motion analysis | |
| O-larnnithipong et al. | Real-time implementation of orientation correction algorithm for 3D hand motion tracking interface | |
| JP2014117409A (en) | Method and apparatus for measuring body joint position |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |