CN115331153B - Posture monitoring method for assisting vestibule rehabilitation training - Google Patents
Posture monitoring method for assisting vestibule rehabilitation training Download PDFInfo
- Publication number
- CN115331153B CN115331153B CN202211243707.6A CN202211243707A CN115331153B CN 115331153 B CN115331153 B CN 115331153B CN 202211243707 A CN202211243707 A CN 202211243707A CN 115331153 B CN115331153 B CN 115331153B
- Authority
- CN
- China
- Prior art keywords
- initial key
- key point
- key points
- standard
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000012544 monitoring process Methods 0.000 title claims abstract description 23
- 238000012549 training Methods 0.000 title claims abstract description 23
- 230000033001 locomotion Effects 0.000 claims abstract description 77
- 230000008569 process Effects 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000009471 action Effects 0.000 claims abstract description 18
- 230000001720 vestibular Effects 0.000 claims abstract description 17
- 238000010606 normalization Methods 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 7
- 238000009432 framing Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 3
- 230000036544 posture Effects 0.000 description 63
- 210000003739 neck Anatomy 0.000 description 25
- 210000001508 eye Anatomy 0.000 description 10
- 238000009826 distribution Methods 0.000 description 9
- 230000000737 periodic effect Effects 0.000 description 6
- 239000013598 vector Substances 0.000 description 5
- 238000001914 filtration Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 210000003128 head Anatomy 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000002146 bilateral effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 210000003414 extremity Anatomy 0.000 description 2
- 230000006698 induction Effects 0.000 description 2
- 230000001939 inductive effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 210000001513 elbow Anatomy 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 210000003811 finger Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Rehabilitation Tools (AREA)
Abstract
The invention relates to the technical field of data processing, in particular to a posture monitoring method for assisting vestibular rehabilitation training. Firstly, acquiring a posture image and an initial key point in the image in the motion process of a patient; forming an initial key point group by using any two initial key points, and clustering the initial key points of the attitude images of a plurality of cycles based on the spatial distance from the initial key point group to the standard key point group to obtain a plurality of clusters; carrying out binarization processing on the initial key points in a plurality of periods to update the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points after binarization processing; calculating a scaling factor based on the spatial distance group and the imbalance coefficient; and based on the zoom factor, acquiring a final key point in the attitude image by using a SimDR algorithm, and connecting the final key point to obtain the motion attitude. According to the invention, the corresponding scaling factor is calculated according to the different key points corresponding to each action, so that the accuracy rate of monitoring and positioning the key points is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a posture monitoring method for assisting vestibular rehabilitation training.
Background
In the vestibular rehabilitation training process, the effectiveness of the rehabilitation training of the patient is generally evaluated by judging whether the posture of the patient in the reciprocating motion process is qualified or not and whether the position of a key point is abnormal or not. However, this evaluation method needs the medical specialist to observe the whole process of the patient's rehabilitation exercise on site, which not only takes a long time but also depends on professional medical rehabilitation knowledge and resources.
Through the rapid development of a computer vision technology and a deep learning technology, limb key points can be accurately identified by monitoring the limb key points in the process of human motion and training by utilizing the learning capacity of a neural network at the present stage, and the posture of a human body is evaluated through the key points. Aiming at the problem that the size of a Gaussian heat map generated by Heatmap-based through 2D Gaussian distribution is often smaller than that of an original image, quantization errors are easily generated in the coordinate restoration process, and the positioning of key points is influenced, the SimDR introduces a scaling factor to accurately enhance the positioning to a level smaller than that of a single pixel. However, the fixed-size scaling factor determines that the number of each pixel block divided into a plurality of bins is not changed, which means that in the subsequent positioning after coordinate classification, the prediction probability is the same in magnitude regardless of whether the prediction position is correct, and the scaling factor cannot be adjusted according to the characteristics of the key points in the image.
Disclosure of Invention
In order to solve the technical problems, the invention aims to provide a posture monitoring method for assisting vestibular rehabilitation training, which adopts the following technical scheme:
acquiring a motion video of a patient in a motion process, and preprocessing the motion video to obtain a posture image;
acquiring a plurality of initial key points in the attitude image, determining neck key points from the initial key points, establishing a coordinate system by taking the neck key points as a coordinate origin, and updating the coordinates of each initial key point; acquiring a plurality of periodic attitude images, and normalizing the coordinates of initial key points in the plurality of periodic attitude images; for any attitude image, forming an initial key point group by using any two initial key points, and clustering the initial key points of the attitude images of a plurality of cycles based on a spatial distance group formed by the spatial distances from the initial key point group to a standard key point group formed by any other two standard key points in the standard image to obtain a plurality of clusters; carrying out binarization processing on the initial key points in a plurality of periods, and updating the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points in the same cluster after binarization processing; calculating a scaling factor based on the set of spatial distances and the imbalance coefficient;
and based on the scaling factor, acquiring a final key point in the attitude image and a final key point coordinate corresponding to the final key point by using a SimDR algorithm, and connecting the final key point to obtain the motion attitude.
Preferably, the preprocessing the motion video to obtain a pose image includes:
and performing framing processing on the motion video to obtain a plurality of single-frame motion images, and denoising the single-frame motion images to obtain a posture image.
Preferably, the acquiring a plurality of initial key points in the pose image includes:
and acquiring a plurality of initial key points in the attitude image by using a SimDR algorithm.
Preferably, the method for obtaining the spatial distance group formed by the spatial distances from the initial keypoint group to the standard keypoint group formed by any other two standard keypoints in the standard image comprises:
mapping the initial key points and the standard key points in the standard image to the same image, taking any initial key point in the initial key point group as a first initial key point, and calculating the distance between the first initial key point and any standard key point in the standard image as a first spatial distance; taking another initial key point in the initial key point group as a second initial key point, and calculating the distance between the second initial key point and another standard key point in the standard image as a second spatial distance; the first spatial distance and the second spatial distance form a spatial distance group.
Preferably, the binarizing processing the initial key points in the multiple cycles to update the pixel values of the initial key points includes:
when the initial key points in the clusters corresponding to the initial key points of two initial key points in the initial key point group are uniformly distributed, updating the pixel values of the pixel points corresponding to all the initial key points in the initial key point group to be 1;
and when the initial key points in the cluster corresponding to at least one of the two initial key points in the initial key point group are non-uniformly distributed, updating the pixel values of the pixel points corresponding to all the initial key points in the initial key point group to be 0.
Preferably, the constructing an imbalance coefficient according to the pixel values of the initial key points in the same cluster after the binarization processing includes:
the calculation formula of the imbalance coefficient is as follows:
wherein,
is the imbalance coefficient;
is a selection function;
the number of initial key points which are 0 in the pixel values after binarization and correspond to the pixel points of the standard key points;
and the number of initial key points which are 1 in the pixel values after binarization of the pixel points corresponding to the standard key points.
Preferably, the calculating the scaling factor based on the spatial distance group and the imbalance coefficient includes:
the calculation formula of the scaling factor is as follows:
wherein,
is an initial key point
A corresponding scaling factor;
rounding the calculation result in brackets;
is the imbalance coefficient;
normalizing the position coordinates of the standard key points except the neck key point in any standard key point group;
removing neck keypoints and standard keypoints from standard keypoint group
The position coordinate of another standard key point after normalization;
is the number of cycles;
the included angle formed by any two standard key points in all periods is equal to the number of the included angles formed by any two initial key points;
the abscissa after normalization for the jth initial key point in the pth period;
the vertical coordinate after normalization for the jth initial key point in the pth period;
the abscissa after normalization for the j +1 st initial key point in the p period;
the normalized ordinate of the j +1 st initial key point in the p period;
is the number of initial keypoints.
Preferably, after obtaining the motion gesture, the method further includes:
and comparing the motion posture with a standard posture formed by connecting standard key points in the standard image, and correcting the action of the patient when the motion posture is not completely matched with the standard posture.
The embodiment of the invention at least has the following beneficial effects:
according to the method, an initial key point group is formed according to initial key points in a posture image corresponding to a motion video in the motion process of a patient, a spatial distance group of the initial key point group is obtained according to the spatial distance between the initial key point group and a standard key point group corresponding to the standard image, and the spatial distance group represents the positioning accuracy of the initial key points relative to the standard key points. The method comprises the steps of constructing an unbalance coefficient according to the difference of the space distances of a standard key point group and an initial key point group, calculating scaling factors according to the space distances and the unbalance coefficient, obtaining final key points in the attitude image and final key point coordinates corresponding to the final key points by using a SimDR algorithm according to different scaling factors corresponding to the key points, corresponding the key points with different accuracies to different scaling factors, obtaining the final key point coordinates according to different scaling factors, and improving the accuracy rate of key point monitoring and positioning.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a method for assisting a posture monitoring method of vestibular rehabilitation training according to an embodiment of the present invention.
Detailed Description
In order to further illustrate the technical means and effects of the present invention for achieving the predetermined objects, the following detailed description of the posture monitoring method for assisting vestibular rehabilitation training according to the present invention with reference to the accompanying drawings and the preferred embodiments thereof will be provided with specific embodiments, structures, features and effects thereof. In the following description, the different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The embodiment of the invention provides a specific implementation method of a posture monitoring method for assisting vestibular rehabilitation training, which is suitable for a posture monitoring scene of vestibular rehabilitation training. In the scene, a motion video of the patient during rehabilitation motion is acquired through the video recorder. The method aims to solve the problem that the scaling factor with a fixed size determines that the number of each pixel block divided into a plurality of bins is not changed, and the scaling factor cannot be adjusted according to the characteristics of key points in an image. According to the invention, the pixels with different prediction probabilities are corresponding to different scaling factors according to different key points corresponding to each action, so that the accuracy rate of monitoring and positioning the key points is improved.
The following specifically describes a specific scheme of a posture monitoring method for assisting vestibular rehabilitation training, which is provided by the invention, with reference to the accompanying drawings.
Referring to fig. 1, a flowchart of a method for assisting posture monitoring method of vestibular rehabilitation training according to an embodiment of the present invention is shown, where the method includes the following steps:
step S100, collecting a motion video of a patient in a motion process, and preprocessing the motion video to obtain a posture image.
And respectively acquiring various posture data of the patient in the rehabilitation exercise process and the standard exercise process by utilizing recording equipment, and manufacturing corresponding labels of the posture data. Common vestibule rehabilitation actions comprise head-to-hand immobilization, hand-to-head immobilization, straight-to-go up-and-down viewing, straight-to-left-and-right viewing and the like, the motion process and the posture of a human body are different, and the positions of key points corresponding to the motion process and the posture of the human body are also continuously changed. These rehabilitation movements include multiple key points corresponding to different body parts such as eyes, wrists, fingers, necks, elbows, and the like.
In the motion process, the key points of the human body are in a state of being possibly changed at any time, and the instability of the motion duration and the instability of the motion range are considered, and the recording equipment is utilized to respectively acquire the whole motion process of standard actions and the motion video in the motion process of the vestibular rehabilitation patient during training. The camera of the recording device needs to cover a certain spatial range, secondly the higher the camera resolution the better. The method comprises the following steps of obtaining a motion video of a patient in a motion process, preprocessing the motion video to obtain a posture image, specifically: the motion video is subjected to framing processing to obtain a plurality of single-frame motion images, and the single-frame motion images are subjected to denoising to obtain attitude images.
The method comprises the steps of performing framing processing on each motion video collected by a recording device, wherein each frame corresponds to a picture of a single action, and the purpose of framing is to convert video data into image data so as to facilitate the positioning of subsequent key points. The same framing processing is performed on each acquired motion video of the patient, and it should be noted that the quantity of image data acquired after the motion video of the same motion is framed needs to be consistent.
In order to eliminate the influence of noise on image quality in the recording process, the acquired image data is denoised, common denoising techniques include median filtering denoising, bilateral filtering denoising, mean filtering denoising and the like, and the noise is eliminated by using bilateral filtering denoising in the invention, which is a known technique and is not described in detail herein.
Step S200, acquiring a plurality of initial key points in the attitude image, determining neck key points from the initial key points, establishing a coordinate system by taking the neck key points as a coordinate origin, and updating the coordinates of each initial key point; acquiring a plurality of periodic attitude images, and normalizing the coordinates of initial key points in the plurality of periodic attitude images; for any attitude image, forming an initial key point group by using any two initial key points, and clustering the initial key points of the attitude images of a plurality of cycles based on a spatial distance group formed by the spatial distances from the initial key point group to a standard key point group formed by any other two standard key points in the standard image to obtain a plurality of clusters; carrying out binarization processing on the initial key points in a plurality of periods, and updating the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points after binarization processing in the same cluster; a scaling factor is calculated based on the set of spatial distances and the imbalance coefficient.
After the attitude image is obtained, firstly, a plurality of initial key points in the attitude image are obtained by using the CNN-based in the SimDR algorithm. Determining neck key points from the initial key points, establishing a coordinate system by taking the neck key points as the origin of coordinates, and updating the coordinates of each initial key point. In the invention, the initial key point coordinate parameters in the posture image corresponding to the patient and the initial key point coordinate parameters in the standard image in the standard motion process are subjected to data processing, and the relative position between the initial key points is calculated by utilizing the coordinate information of the initial key points and the inverse trigonometric function.
Because the body parameters of different people are different, for example, different heights of different patients lead to different positions of key points of the neck, that is, in a static state, the positions of the initial key points between the patient and the rehabilitation teacher may be different, the heights and shoulder widths of the two people are different, so that when the same action is performed, the positions of the key points of the body are also different, but the included angles between the relative initial key points are the same, and subsequent posture assessment is performed through the relative positions between the initial key points.
Since the standard posture corresponding to each action is fixed even though there is a difference in the physical parameters of the two persons, assuming that the action of looking straight up and down is the standard posture, assuming that 45 ° of head elevation and 30 ° of overlooking are specified, the movement angle between the key point corresponding to the neck and the key point corresponding to the eyes is fixed, the key point positions corresponding to the eyes of two persons with different heights are certainly different, but the angles of neck elevation and overlooking are consistent when each person reaches the standard posture. The standard key point 1 is the key point position corresponding to the eyes when the head is raised by 45 degrees, the standard key point 2 is the key point position corresponding to the eyes when the head is overlooked by 30 degrees,
is the angle of variation between the criteria keypoints 1, 2; the initial key point 1 of the patient is the key point position corresponding to the eyes of the patient when the patient raises 45 degrees, the initial key point 2 of the patient is the key point position corresponding to the eyes of the patient when the patient overlooks 30 degrees,
is the angle of change between the patient's key points 1, 2. If the patient achieves the rehabilitation standard during the movement, the corresponding key points should be satisfied
。
Therefore, the normalization processing is required to be performed on the initial key point data in the acquired pose image, because the feature values of the key points used for training the neural network should only be related to the behavior and the action of the current moving object, and are not related to the proportional size of the human body in the picture, the normalization processing is performed on the initial key point data:
wherein (x, y) are the coordinates of the initial keypoint, ((x, y))
) Coordinates of the neck key points, w and h are the length and width of the pose image, respectively: (
) Coordinates of the initial keypoints after normalization for the keypoints.
The SimDR first extracts the keypoint representation using a CNN-based network. Given the obtained keypoint representation, the vertical and horizontal coordinates are then separately coordinate-sorted to produce the final prediction. In this process, simDR solves the problem of severe quantization error generated back to the original image by the coordinate method obtained by argmax in the conventional Heatmap-based method by setting a scaling factor k.
In the SimDR, the horizontal and vertical coordinates of key points are represented by two independent one-dimensional vectors, further, the positioning accuracy of the key points is enhanced to a sub-pixel level through a scaling factor k, but the size of the scaling factor k is fixed in the process, the scaling factor k has the effect of enhancing the positioning accuracy to a level smaller than that of a single pixel, the quantization error of the SimDR is smaller when the scaling factor k is larger, but the training difficulty of a model is increased when the value of the scaling factor k is too large, and therefore the scaling factor k can be regulated and changed.
In the invention, the closer the data information after the initial key point normalization in the patient movement process is to the data information after the initial key point normalization in the standard posture, the more obvious the characteristics of the pixel points corresponding to the patient key points are, and the weaker the characteristics of the pixel points far away from the data information after the initial key point normalization in the standard posture are. Because the closer to the initial keypoint data information at the standard pose, the more standard the motion made during the patient's motion is. The data of the key points comprise coordinate information after the key points are normalized and included angles between the key points.
In the image by the encodern key points correspond to n embedding, that is, the encoder structure outputs n one-dimensional vectors, and further, the n one-dimensional vectors are converted into n SimDR representations through linear projection, and further, for a given ith key point representation (for a given ith key point representation: (b) (c))
) As input to the coordinate classifier, the horizontal coordinate classifier generates the abscissa of the ith keypoint
The vertical coordinate classifier generates the ordinate of the ith key point
The classification result is that the key point is positioned in the original image by dividing the position of the maximum point on the one-dimensional vector by the scaling factor.
Wherein,
is the abscissa of the predicted point;
is the ordinate of the predicted point; k is a scaling factor;
and (5) representing the SimDR corresponding to the original key point. It should be noted that, based on the scaling factor, locating the keypoints by using SimDR is a well-known technique for those skilled in the art, and is not described herein in detail.
The degree to which each pixel point should be scaled is measured by considering the similarity between the initial keypoint normalized image data distribution of the patient and the standard keypoint normalized image data distribution of the standard pose, as how many bins each pixel should be divided into.
Selecting a scaling factor, specifically:
acquiring attitude images of multiple periods, and normalizing the coordinates of initial key points in the attitude images of the multiple periods; for any attitude image, any two initial key points form an initial key point group, and the initial key points of the attitude images in a plurality of cycles are clustered based on a spatial distance group formed by the spatial distances from the initial key point group to a standard key point group formed by any two other standard key points in the standard image to obtain a plurality of clusters.
The method comprises the steps of carrying out periodic induction on a framed posture image according to a motion period of a standard image, wherein the periodic induction refers to inducing image data belonging to the same period into the same period according to acquisition time, recording the frame number (n) of each period in the standard image, inducing the posture image with the same frame number (n) into one period for framed patient image data, namely corresponding posture images, and analyzing the standard image data of 1 period and the posture image data of a plurality of periods in each action. In the embodiment of the invention, 10 periods of attitude image data are selected for analysis.
Because each attitude image comprises more than one key point, the motion trend and the corresponding attitude of each key point are different, the key points at different positions can be regarded as independent pixel points. And for any attitude image, forming an initial key point group by using any two initial key points, and forming a spatial distance group based on the spatial distances from the initial key point group to a standard key point group formed by any other two standard key points in the standard image. The method for acquiring the space distance group comprises the following steps: mapping the initial key points and the standard key points in the standard image to the same image, taking any initial key point in the initial key point group as a first initial key point, and calculating the distance between the first initial key point and any standard key point in the standard image as a first spatial distance; taking another initial key point in the initial key point group as a second initial key point, and calculating the distance between the second initial key point and another standard key point in the standard image as a second spatial distance; the first spatial distance and the second spatial distance form a spatial distance group.
Because whether an action is standard or not can not be accurately determined by matching or not of a single initial key point, the posture of the patient is considered to reach the standard posture only when the position coordinates of all the normalized key points can be matched one by one. A spatial distance is constructed, namely the distance from each initial key point in the attitude image data to a standard key point in the standard image, and the scaling degree of each pixel point is evaluated subsequently through the spatial distance.
And clustering the initial key points of the attitude images of the multiple cycles based on the space distance group to obtain multiple clusters. Because the clustering is a comparison of the spatial distance groups from the initial key point groups to the standard key point groups, each initial key point corresponds to one spatial distance from the standard key point, that is, each initial key point in each initial key point group has a respective spatial distance from each initial key point to the corresponding standard key point, so each cluster corresponds to two standard key points, and each standard key point has a respective initial key point group.
For the clustering result, the number of the initial key points distributed around the standard key points reflects the degree of the initial key points close to the standard key points in each period, if the standard posture is reached, the normalized spatial distance of each group of the initial key points is equal to the spatial distance between the standard key points and the neck key points, and the angle is fixed. Suppose that the eye key points of the same motion posture in different periods are respectively
Then the key point of the neck and the key point of the standard
Spatial distribution and neck and eye key points
The spatial distribution is uniformMeanwhile, the key point of the neck and the key point of the standard
Spatial distribution of (A) and neck and eye key points
Is consistent, which means that in three-dimensional space, pixel points are
The surrounding clustering results are evenly distributed, and the pixel points are
The surrounding clustering results also satisfy uniform distribution, and the two distributions are independent from each other as a whole. If the motion posture of the patient does not reach the standard, the initial key points of the corresponding posture cannot reach the correct positions, and the spatial distribution between the key points of the neck and the key points of the neck is different from the spatial distribution between the key points of the neck and the standard key points.
Performing binarization processing on the initial key points in a plurality of periods, and updating the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points in the same cluster after binarization processing; a scaling factor is calculated based on the set of spatial distances and the imbalance coefficient.
When the initial key points in the clusters corresponding to the initial key points of two initial key points in the initial key point group are uniformly distributed, updating the pixel values of the pixel points corresponding to all the initial key points in the initial key point group to be 1;
and when the initial key points in the cluster corresponding to at least one of the two initial key points in the initial key point group are non-uniformly distributed, updating the pixel values of the pixel points corresponding to all the initial key points in the initial key point group to be 0.
That is, if two initial keypoints in one initial keypoint group respectively satisfy the initial keypoints
The initial key points in the corresponding clusters are uniformly distributed around the initial key points
Uniformly distributing around the initial key points in the corresponding clusters, and recording pixel points corresponding to all the initial key points in the initial key point group as 1; if only one or zero pixel points in an initial key point group are satisfied at the initial key point
Uniformly distributed or initial keypoints around the initial keypoints within the corresponding cluster
And uniformly distributing the initial key points in the corresponding clusters around, recording pixel points corresponding to all the initial key points in the initial key point group as 0, traversing 10 periods of the initial key points, and obtaining a binarization processing result.
Further, an imbalance coefficient is constructed according to pixel values corresponding to initial key points around the standard key points of the standard image after binarization processing in the same cluster. It can also be said that the imbalance coefficient reflects whether the initial keypoints are uniformly distributed around the standard keypoints, and its role is to give a smaller scaling factor to the uniformly distributed initial keypoints, that is, the initial keypoints with the binarization result of 1, so that a more accurate result can be obtained in SimDR positioning.
The calculation formula of the imbalance coefficient is as follows:
wherein,
is the imbalance coefficient;
is a selection function;
the number of initial key points which are 0 in the pixel values after binarization and correspond to the pixel points of the standard key points;
and the number of initial key points which are 1 in the pixel values after binarization of the pixel points corresponding to the standard key points.
The imbalance coefficient A reflects whether the initial key points of the posture of the patient can be uniformly and independently distributed around the standard key points, and the larger the imbalance coefficient A is, the less the initial key points of the posture of the patient at the current moment can reach the standard posture.
A scaling factor is calculated based on the set of spatial distances and the imbalance coefficient. The purpose of the scaling factor is to calculate the scaling factor because the scaling factor is located at the denominator in the SimDR positioning calculation, so that the smaller the scaling factor, the greater the degree of scaling that will be achieved.
The calculation formula of the scaling factor is as follows:
wherein,
is an initial key point
A corresponding scaling factor;
rounding the calculation result in brackets;
is said toEqualizing the coefficients;
normalizing the position coordinates of the standard key points except the neck key point in any standard key point group;
removing neck keypoints and standard keypoints from standard keypoint group
The position coordinate of another standard key point after normalization;
is the number of cycles;
the included angle formed by any two standard key points in all periods is equal to the number of the included angles formed by any two initial key points;
the abscissa after normalization for the jth initial key point in the pth period;
the normalized ordinate of the jth initial key point in the pth period;
the abscissa after normalization is carried out on the j +1 st initial key point in the p period;
and (4) normalizing the vertical coordinate of the j +1 st initial key point in the p period.
Wherein if the initial key point
Binarization methodAnd the latter is 1, then the corresponding imbalance coefficient A takes the second parameter, otherwise, the first parameter is taken.
Is an initial key point
Key point combinations to standard poses
The spatial distance of (2) reflects the distance from the initial key point group to the clustering center in the re-clustering process, i.e. the distance from the initial key point group to the standard key point, and the smaller the spatial distance is, the more likely the initial key point is clustered to the cluster where the standard key point is located.
And
is calculated from the difference between the abscissa and ordinate of the standard keypoint and the abscissa and ordinate of the initial keypoint, and is therefore considered to be
And
the likelihood of whether the initial keypoint location is correct can be approximately characterized,
and
the smaller the probability that the initial keypoint is located closer to the standard keypoint is. Since there is not only one keypoint in each action, the initial set of keypoints characterizing this action needs to be jointly computed, and the criterion distance reflects the distance between the criterion keypoint and the initial keypoint under the same actionThe larger the standard distance is, the less the initial key point of the posture of the patient at the current moment reaches the standard posture, and similarly, the larger the unbalance coefficient A is, the more the key point of the posture of the patient at the current moment does not reach the standard posture, the smaller the scaling coefficient A is, the more the key point of the posture of the patient at the current moment does not reach the standard posture, and thus the SimDR can lock the position of the key point more accurately in the subsequent coordinate classification process.
And S300, based on the scaling factor, acquiring a final key point in the attitude image and a final key point coordinate corresponding to the final key point by using a SimDR algorithm, and connecting the final key point coordinate to obtain a motion attitude.
And calculating the spatial distance of each initial key point input into the SimDR depth model according to the steps, further obtaining two one-dimensional vectors output by the model by using the spatial distance to obtain a corresponding scaling factor k, and calculating the coordinates of the predicted point according to the coordinate classifier, namely obtaining the final key point and the corresponding final key point coordinates thereof.
And monitoring the motion postures of the patients by combining the coordinate positions of the final key points according to the obtained final key point coordinates of the motion postures of the patients under various motions. Because each action cannot be completely represented by the position of only one key point, the gesture monitoring purpose is achieved. After the final key points corresponding to each frame of posture image are obtained, the final key points in each image are connected to obtain the motion posture of the current moment, further, the motion posture of the patient in the motion process is compared with the standard posture formed by connecting the standard key points in the standard image, and when the motion posture is completely matched with the standard posture, the motion posture of the current patient is considered to be standard, so that the function of assisting vestibular rehabilitation training can be exerted; when the motion posture and the standard posture are not completely matched, the current motion posture is regarded as nonstandard, and action correction is carried out on the patient.
In summary, the present invention relates to the field of data processing technology. The method comprises the steps of collecting a motion video of a patient in a motion process, and preprocessing the motion video to obtain a posture image; acquiring a plurality of initial key points in the attitude image, determining neck key points from the initial key points, establishing a coordinate system by taking the neck key points as a coordinate origin, and updating the coordinates of each initial key point; acquiring attitude images of a plurality of periods, and normalizing the coordinates of initial key points in the attitude images of the plurality of periods; for any attitude image, forming an initial key point group by using any two initial key points, and clustering the initial key points of the attitude images in a plurality of cycles based on a spatial distance group formed by the spatial distances from the initial key point group to a standard key point group formed by any two other standard key points in the standard image to obtain a plurality of clusters; carrying out binarization processing on the initial key points in a plurality of periods, and updating the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points after binarization processing in the same cluster; calculating a scaling factor based on the set of spatial distances and the imbalance coefficient; and based on the scaling factor, acquiring a final key point in the attitude image and a final key point coordinate corresponding to the final key point by using a SimDR algorithm, and connecting the final key point to obtain the motion attitude. According to the method, the pixels with different prediction probabilities are corresponding to different scaling factors according to different key points corresponding to each action, so that the accuracy of key point monitoring and positioning is improved.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. The processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.
Claims (6)
1. A posture monitoring method for assisting vestibular rehabilitation training is characterized by comprising the following steps:
acquiring a motion video of a patient in a motion process, and preprocessing the motion video to obtain a posture image;
acquiring a plurality of initial key points in the attitude image, determining neck key points from the initial key points, establishing a coordinate system by taking the neck key points as an origin of coordinates, and updating the coordinates of each initial key point; acquiring attitude images of a plurality of periods, and normalizing the coordinates of initial key points in the attitude images of the plurality of periods; for any attitude image, forming an initial key point group by using any two initial key points, and clustering the initial key points of the attitude images of a plurality of cycles based on a spatial distance group formed by the spatial distances from the initial key point group to a standard key point group formed by any other two standard key points in the standard image to obtain a plurality of clusters; carrying out binarization processing on the initial key points in a plurality of periods, and updating the pixel values of the initial key points; constructing an imbalance coefficient according to the pixel values of the initial key points after binarization processing in the same cluster; calculating a scaling factor based on the set of spatial distances and the imbalance coefficient;
based on the scaling factor, acquiring a final key point in the attitude image and a final key point coordinate corresponding to the final key point by using a SimDR algorithm, and connecting the final key point to obtain a motion attitude;
wherein, the calculation formula of the imbalance coefficient is as follows:
wherein,
is the imbalance coefficient;
is a selection function;
the number of initial key points which are 0 in the pixel values after binarization of the pixel points corresponding to the standard key points;
the number of initial key points which are 1 in the pixel values after binarization of the pixel points corresponding to the standard key points;
wherein, the calculation formula of the scaling factor is as follows:
wherein,
is an initial key point
A corresponding scaling factor;
rounding the calculation result in brackets;
is the imbalance coefficient;
normalizing the position coordinates of the standard key points except the neck key point in any standard key point group;
is a key of the standardRemoving neck key points and standard key points in point group
The position coordinate of another standard key point after normalization;
is the number of cycles;
the included angle formed by any two standard key points in all periods is equal to the number of the included angles formed by any two initial key points;
the abscissa after normalization for the jth initial key point in the pth period;
the vertical coordinate after normalization for the jth initial key point in the pth period;
the abscissa after normalization is carried out on the j +1 st initial key point in the p period;
the vertical coordinate after normalization for the j +1 st initial key point in the p period;
is the number of initial keypoints.
2. The method for monitoring the posture for assisting vestibular rehabilitation training according to claim 1, wherein the preprocessing the motion video to obtain the posture image comprises:
and performing framing processing on the motion video to obtain a plurality of single-frame motion images, and denoising the single-frame motion images to obtain a posture image.
3. The method according to claim 1, wherein the obtaining a plurality of initial key points in the pose image comprises:
and acquiring a plurality of initial key points in the attitude image by using a SimDR algorithm.
4. The method for monitoring the posture for assisting the vestibular rehabilitation training according to claim 1, wherein the method for acquiring the spatial distance group formed by the spatial distances from the initial key point group to the standard key point group formed by any two other standard key points in the standard image comprises:
mapping the initial key points and the standard key points in the standard image to the same image, taking any initial key point in the initial key point group as a first initial key point, and calculating the distance between the first initial key point and any standard key point in the standard image as a first spatial distance; taking another initial key point in the initial key point group as a second initial key point, and calculating the distance between the second initial key point and another standard key point in the standard image as a second spatial distance; the first spatial distance and the second spatial distance constitute a set of spatial distances.
5. The method for monitoring the posture for assisting the vestibular rehabilitation training according to claim 1, wherein the binarizing the initial key points in the multiple cycles to update the pixel values of the initial key points comprises:
when the initial key points in the clusters corresponding to the two initial key points in the initial key point group are uniformly distributed, updating the pixel values of the pixel points corresponding to all the initial key points in the initial key point group to 1;
when the initial key points in the cluster corresponding to at least one of the two initial key points in the initial key point group are non-uniformly distributed, the pixel values of the pixel points corresponding to all the initial key points in the initial key point group are updated to 0.
6. The posture monitoring method for assisting vestibular rehabilitation training according to claim 1, characterized in that after obtaining the motion posture, further comprising:
and comparing the motion posture with a standard posture formed by connecting standard key points in the standard image, and correcting the action of the patient when the motion posture is not completely matched with the standard posture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211243707.6A CN115331153B (en) | 2022-10-12 | 2022-10-12 | Posture monitoring method for assisting vestibule rehabilitation training |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211243707.6A CN115331153B (en) | 2022-10-12 | 2022-10-12 | Posture monitoring method for assisting vestibule rehabilitation training |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115331153A CN115331153A (en) | 2022-11-11 |
| CN115331153B true CN115331153B (en) | 2022-12-23 |
Family
ID=83914653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211243707.6A Active CN115331153B (en) | 2022-10-12 | 2022-10-12 | Posture monitoring method for assisting vestibule rehabilitation training |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115331153B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116469175B (en) * | 2023-06-20 | 2023-08-29 | 青岛黄海学院 | Visual interaction method and system for infant education |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109948590A (en) * | 2019-04-01 | 2019-06-28 | 启霖世纪（北京）教育科技有限公司 | Pose problem detection method and device |
| CN111339903A (en) * | 2020-02-21 | 2020-06-26 | 河北工业大学 | Multi-person human body posture estimation method |
| CN111652076A (en) * | 2020-05-11 | 2020-09-11 | 重庆大学 | Automatic gesture recognition system for AD (analog-digital) scale comprehension capability test |
| CN112084967A (en) * | 2020-09-12 | 2020-12-15 | 周美跃 | Limb rehabilitation training detection method and system based on artificial intelligence and control equipment |
| CN112149494A (en) * | 2020-08-06 | 2020-12-29 | 中国地质大学（武汉） | Multi-person posture recognition method and system |
| CN113386128A (en) * | 2021-05-11 | 2021-09-14 | 华南理工大学 | Body potential interaction method for multi-degree-of-freedom robot |
| CN114067358A (en) * | 2021-11-02 | 2022-02-18 | 南京熊猫电子股份有限公司 | Human body posture recognition method and system based on key point detection technology |
| CN114724185A (en) * | 2022-04-13 | 2022-07-08 | 浙江工业大学 | Light-weight multi-person posture tracking method |
| CN114783061A (en) * | 2022-04-26 | 2022-07-22 | 南京积图网络科技有限公司 | Smoking behavior detection method, device, equipment and medium |
| CN115035037A (en) * | 2022-05-10 | 2022-09-09 | 湘南学院 | Limb rehabilitation training method and system based on image processing and multi-feature fusion |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111191622B (en) * | 2020-01-03 | 2023-05-26 | 华南师范大学 | Gesture recognition method, system and storage medium based on thermodynamic diagram and offset vector |
| CN112861776A (en) * | 2021-03-05 | 2021-05-28 | 罗普特科技集团股份有限公司 | Human body posture analysis method and system based on dense key points |
-
2022
- 2022-10-12 CN CN202211243707.6A patent/CN115331153B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109948590A (en) * | 2019-04-01 | 2019-06-28 | 启霖世纪（北京）教育科技有限公司 | Pose problem detection method and device |
| CN111339903A (en) * | 2020-02-21 | 2020-06-26 | 河北工业大学 | Multi-person human body posture estimation method |
| CN111652076A (en) * | 2020-05-11 | 2020-09-11 | 重庆大学 | Automatic gesture recognition system for AD (analog-digital) scale comprehension capability test |
| CN112149494A (en) * | 2020-08-06 | 2020-12-29 | 中国地质大学（武汉） | Multi-person posture recognition method and system |
| CN112084967A (en) * | 2020-09-12 | 2020-12-15 | 周美跃 | Limb rehabilitation training detection method and system based on artificial intelligence and control equipment |
| CN113386128A (en) * | 2021-05-11 | 2021-09-14 | 华南理工大学 | Body potential interaction method for multi-degree-of-freedom robot |
| CN114067358A (en) * | 2021-11-02 | 2022-02-18 | 南京熊猫电子股份有限公司 | Human body posture recognition method and system based on key point detection technology |
| CN114724185A (en) * | 2022-04-13 | 2022-07-08 | 浙江工业大学 | Light-weight multi-person posture tracking method |
| CN114783061A (en) * | 2022-04-26 | 2022-07-22 | 南京积图网络科技有限公司 | Smoking behavior detection method, device, equipment and medium |
| CN115035037A (en) * | 2022-05-10 | 2022-09-09 | 湘南学院 | Limb rehabilitation training method and system based on image processing and multi-feature fusion |
Non-Patent Citations (3)
| Title |
|---|
| "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression";Zigang Geng 等;《IEEE》;20211102;第14671-14681页 * |
| "Is 2D Heatmap Representation Even Necessary for Human Pose Estimation?";Yanjie Li 等;《arXiv:2107.03332v1 [cs.CV]》;20210707;第1-15页 * |
| "SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation";Yanjie Li 等;《arXiv:2107.03332v3 [cs.CV]》;20220705;第1-18页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115331153A (en) | 2022-11-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110348330B (en) | Face pose virtual view generation method based on VAE-ACGAN | |
| Brown et al. | Comparative study of coarse head pose estimation | |
| CN112766160A (en) | Face replacement method based on multi-stage attribute encoder and attention mechanism | |
| US9480417B2 (en) | Posture estimation device, posture estimation system, and posture estimation method | |
| KR100647322B1 (en) | Apparatus and method of generating shape model of object and apparatus and method of automatically searching feature points of object employing the same | |
| CN109949341B (en) | Pedestrian target tracking method based on human skeleton structural features | |
| JP6368709B2 (en) | Method for generating 3D body data | |
| CN109684969B (en) | Gaze position estimation method, computer device, and storage medium | |
| CN109508700A (en) | A kind of face identification method, system and storage medium | |
| KR20170006355A (en) | Method of motion vector and feature vector based fake face detection and apparatus for the same | |
| MX2012010602A (en) | Face recognizing apparatus, and face recognizing method. | |
| CN110136177B (en) | Image registration method, device and storage medium | |
| CN110555408A (en) | Single-camera real-time three-dimensional human body posture detection method based on self-adaptive mapping relation | |
| CN109048918A (en) | A kind of visual guide method of wheelchair arm robot | |
| CN112232128B (en) | Eye tracking based method for identifying care needs of old disabled people | |
| JP2005339288A (en) | Image processor and its method | |
| CN115331153B (en) | Posture monitoring method for assisting vestibule rehabilitation training | |
| CN113920571A (en) | Micro-expression identification method and device based on multi-motion feature fusion | |
| CN116580169B (en) | Digital man driving method and device, electronic equipment and storage medium | |
| JP2005351814A (en) | Detector and detecting method | |
| CN116469175B (en) | Visual interaction method and system for infant education | |
| US6931145B1 (en) | Method and apparatus for measuring motion of an object surface by multi-resolution analysis using a mesh model | |
| CN114332447B (en) | License plate correction method, license plate correction device and computer readable storage medium | |
| CN115205750B (en) | Motion real-time counting method and system based on deep learning model | |
| CN114419716B (en) | Calibration method for face image face key point calibration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |