CN101894278A - Human motion tracing method based on variable structure multi-model - Google Patents

Abstract

The invention discloses a human motion tracing method based on VSMM, relating to the computer vision field. The invention solves the problems that the existing method can not solve human motion ambiguity, time complexity is high and good three-dimensional human pose estimation can not be obtained by simply adding motion model. The method includes the following steps: (1) a human motion video image is input, so as to obtain a human profile as well as outer contour and skeleton line thereof; (2) position of articulation point is detected; (3) ridge regression method is used for training a motion model, and a total motion model set is divided into groups; (4) a model group M1 is initialized; (5) interactive multi-model algorithm is operated, so as to obtain a human motion pose; (6) the motion model group meeting activation condition is activated and initialized, if no motion model group meets the activation condition, the step (5) is executed; (7) the model group meeting end condition is ended, and the step (5) is executed, otherwise the step is continued. The invention has the advantages of low time complexity and good tracing effect and can be applied to human motion tracing and pose estimation.

Description

Human body motion tracking method based on variable structure multi-model

Technical field

The invention belongs to technical field of computer vision, relate to a kind of human body motion tracking method, can be used for human motion tracking and attitude and estimate.

Background technology

It is an important branch of computer vision field that human motion is followed the tracks of, because it all has potential application in various aspects such as therapeutic treatment, motion-captured, cartoon making, intelligent monitor systems, so received a lot of scholars' concern.Though in reality, can obtain a large amount of unmarked monocular video sequences, but these type of data are the 2D projection of three-dimensional scenic on image or image sequence, lacked depth information, add from block, foreground detection noise or the like, it is difficult to recover the human motion attitude from such video sequence.

Estimating from monocular image and following the tracks of the complicated 3D structure that links object has two kinds of important method: based on the method for model with based on the method for learning.Ankur Agarwal points out, generally all will a clear and definite in advance parameterized manikin based on the method for model, recover human body attitude according to kinematic principle then, perhaps design the likelihood tolerance of a model to image, according to the projection of current time predicted state on image and the likelihood degree of characteristics of image, use optimized method to obtain optimum human body attitude, but be to use the time complexity of optimization method recovery human motion attitude very high, it needs good initialization, and need to solve the local minimum problem in the optimal search procedure, along with the continuous accumulation of error, optimization method can't guarantee correct human body attitude estimation.Consider that one group of typical human body attitude is than similar many of attitude possible on one group of kinematics, by training a model, directly recovering attitude from observed image is measured estimates, method based on study is avoided the 3D modeling problem, it uses the method study movement capturing data and the mapping relations between the characteristics of image or popular of recurrence or dimensionality reduction, recover 3 d pose according to characteristics of image or other forms of input, obtained good effect.

In forefathers' work, Deutscher et al. uses border and silhouette to make up weighting function as characteristics of image, uses annealing particle filter framework and realizes the human motion tracking.Mikic et al. obtains manikin automatically from a plurality of synchronization videos stream, application extension Kalman filtering framework is estimated human body sport parameter according to the measurement information on the voxel data of mark.Urtasun et al. use balance Gaussian process dynamic model to instruct in the monocular video sequence, to follow the tracks of 3D human motion, this dynamic model be from the less training exercise data middle school acquistion that comprises various modes to.XinyuXu et al. uses less training data in the HumanEva database, adopt partial least-square regression method to train the left side of special human motion and definite relation of right side body kinematics, use Rao-Blackwellised particle filter RBPF to follow the tracks of framework at last and carry out the human body motion tracking.Sigal et al. proposes a Bayesian frame, comprises sequential importance sampling and annealing particle filter, has used multiple motion model and likelihood function during tracking; In order to make 3-d recovery meet the anatomical joints restriction more and to reduce the search volume, from training data, learn motion model, use the Euclidean distance difference of virtual tag to measure as error.Ni et al. has proposed a kind of random-tracking framework that combines interactive multi-model and Kalman's particle filter, the 3D human body that uses the reconstruct of vision shell is as input, physical force/the moment of simulation has reduced required number of particles, obtains good three-dimensional tracking effect in conjunction with the interacting multiple model algorithm that comprises a plurality of kinematics models.Farmer et al. in the real-time supervisory system cheaply, not only can accurately follow the tracks of the motion of human body to interactive multi-model Kalman filtering frame application, also can well follow the tracks of the shape of human body.

Tracking based on model uses the method for optimizing when searching for optimal result, the time complexity height, and can't fundamentally solve the ambiguity of human motion, block under the situation of generation, owing to there is not good guidance, the accurate recovery of human motion can't be guaranteed; Based on the method for study, though the motion model that has used training to obtain has increased the accuracy and the stability of following the tracks of, but single motion model can only motor pattern of match, in addition, for improving tracking effect, the application of good descriptor also will spend a large amount of time.Forefathers have attempted using interacting multiple model algorithm IMM to finish human motion and have followed the tracks of, use well-chosen motion model collection to obtain tracking effect preferably to specific human motion pattern, but in the application of reality, complicacy and polytrope that less motion model set can't solve the human motion pattern, for example, when current motor pattern is walking for redirect, and only comprise the motion model that jumps in the motion model set, can't guarantee tracking effect.The simple quantity that increases motion model is infeasible, not only can increase the time complexity of computing, but also can be because the unnecessary competition between the motion model causes the reduction of tracking effect.

Summary of the invention

The objective of the invention is to overcome existing method deficiency, a kind of human body motion tracking method based on variable structure multi-model VSMM has been proposed, to reduce the ambiguousness of human motion pose recovery, improve the accuracy that human motion is followed the tracks of, reduce the time that single frames is followed the tracks of simultaneously.

The technical scheme that realizes the object of the invention is: on the basis of human body articulation point position, by using the motion model of movement capturing data training, solve the human motion tracking problem in conjunction with the VSMM algorithm frame.

One. the present invention is based on the human body tracing method of VSMM, comprising:

Pre-treatment step: input human body video image, obtain the human body silhouette by background subtraction, extract human body silhouette outline, and the human body silhouette is carried out thinning processing;

Articulation point detects step: to pretreated human body video image, carry out following articulation point and detect:

1) uses the concentric circles template to search for, will fall into the point center of circle the most for a long time of annulus as head node along skeleton line;

2) choosing human body silhouette center of gravity position is root node;

3) use the projection on image of 3D human skeleton model, obtain other articulation point positions on the trunk;

4) choose the skeleton line vertex position as hand node and pin node;

5) make the parallel lines of bipod line by the barycenter of lower part of the body silhouette, two intersection points of itself and skeleton line are as knee joint;

6) with equidistant as elbow joint on the skeleton line with hand and shoulder;

7) to because of blocking or can't detected part articulation point because of cutting apart noise, then adopt the Kalman filtering method, one-step prediction obtains;

Motion model training step: from the CMU of CMU motion capture database, choose the seizure data of multiple motor pattern, adopt the state-transition matrix F of Ridge Regression Modeling Method to the motion model equation ⁱTrain, and calculate the noise w of this motion model _kCovariance, the motion model set of acquisition is called total motion model collection M;

Motion model collection covering design step: in total motion model collection M, if two human motion pattern similarities that motion model mated then divide these two kinds of motion models in same model group, otherwise, it is divided into different model groups; Each model group comprises 3 motion models;

Initialization model group's step: the motion model equation that total motion model is concentrated is as the state equation of interactive multi-model wave filter, with interactive multi-model ten cycles of operation, calculate each model group's model group probability, the model group who selects the probability maximum is as initial current model group M ₁

Interactive multi-model mixes estimating step: human joint points is as input constantly with k, and the execution interacting multiple model algorithm obtains the human motion attitude and estimates, upgrades motion model probability and human motion attitude estimation error covariance;

The motion model group activates step: according to the human joint points position, calculate the angle changing value of extremities joint point skeleton line projection on image, if the changing value size satisfies the model group and activates rule, remember that this is k constantly ₀, carry out the following new model group model initialization step that activates, otherwise output human motion attitude is estimated, carries out above-mentioned interactive multi-model and mixes estimating step;

The new model group model initialization step that activates: the probability that will newly activate model is initialized as model probability maximal value among the current model group, and normalization model probability; The predicated error covariance is initialized as the noise covariance of motion model self; Choose in the movement capturing data with the highest state of present mode matching degree as original state; With master mould group M _oCandidate's model group M with new activation _nMerge into new current model group;

The model group stops step: the current model group according to new re-executes above-mentioned interactive multi-model one-period, if model group M _nWith model group M _oModel group likelihood ratio Perhaps model group likelihood ratio

Less than 0.9, then stop model group M _n, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimating step; If

With

All, then stop model group M greater than 1 _o, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimating step; Otherwise output human motion attitude is estimated, and is continued to carry out this step.

Two. a kind of human body tracking system based on VSMM comprises:

Pretreatment unit: be used to import the human body video image, obtain the human body silhouette, extract human body silhouette outline, and the human body silhouette is carried out thinning processing by background subtraction;

Articulation point pick-up unit: be used for pretreated human body video image, carry out following articulation point and detect:

1) uses the concentric circles template to search for, will fall into the point center of circle the most for a long time of annulus as head node along skeleton line;

2) choosing human body silhouette center of gravity position is root node;

3) use the projection on image of 3D human skeleton model, obtain other articulation point positions on the trunk;

4) choose the skeleton line vertex position as hand node and pin node;

5) make the parallel lines of bipod line by the barycenter of lower part of the body silhouette, two intersection points of itself and skeleton line are as knee joint;

6) with equidistant as elbow joint on the skeleton line with hand and shoulder;

7) to because of blocking or can't detected part articulation point because of cutting apart noise, then adopt the Kalman filtering method, one-step prediction obtains;

Motion model trainer: be used for choosing the seizure data of multiple motor pattern, adopt the state-transition matrix F of Ridge Regression Modeling Method to the motion model equation from CMU's CMU motion capture database ⁱTrain, and calculate the noise w of this motion model _kCovariance, the motion model set of acquisition is called total motion model collection M;

Motion model collection covering design device: be used at total motion model collection M, if two human motion pattern similarities that motion model mated then divide these two kinds of motion models in same model group, otherwise, it is divided into different model groups; Each model group comprises 3 motion models;

Initialization model group's device: be used for the state equation of the motion model equation that total motion model is concentrated as the interactive multi-model wave filter, with interactive multi-model ten cycles of operation, calculate each model group's model group probability, the model group who selects the probability maximum is as initial current model group M ₁

Interactive multi-model mixing estimation unit: be used for k constantly human joint points carry out interacting multiple model algorithm as input, obtain the human motion attitude and estimate, upgrade motion model probability and human motion attitude estimation error covariance;

Motion model group active device: be used for according to the human joint points position, calculate the angle changing value of extremities joint point skeleton line projection on image,, remember that this is k constantly if the changing value size satisfies the model group and activates rule ₀, carry out the following new model group model apparatus for initializing that activates, otherwise output human motion attitude is estimated, carries out above-mentioned interactive multi-model mixing estimation unit;

The new model group model apparatus for initializing that activates: the probability that is used for newly activating model is initialized as current model group model probability maximal value, and normalization model probability; The predicated error covariance is initialized as the noise covariance of motion model self; Choose in the movement capturing data with the highest state of present mode matching degree as original state; With master mould group M _oCandidate's model group M with new activation _nMerge into new current model group;

The model group stops device: be used for re-executing above-mentioned interactive multi-model one-period according to new current model group, if model group M _nWith model group M _oModel group likelihood ratio Perhaps model group likelihood ratio

Less than 0.9, then stop model group M _n, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimation unit; If

With

All, then stop model group M greater than 1 _o, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimation unit; Otherwise output human motion attitude is estimated, and is continued to carry out this device.

The present invention has the following advantages compared with prior art:

1, directly uses movement capturing data training motion model, rather than learn the characteristics of image of motion-captured video and the relation between the movement capturing data, eliminated the influence of picture noise, improved the accuracy and the stability of motion model;

2, in the process of implementation, has only the motion model that is complementary with the current motor pattern effect of all rising, rather than each motion model of total motion model collection all in action, reduce the number of incoherent motion model, not only shortened working time, and the malice competition that has alleviated uncorrelated motion model, improved the degree of accuracy that human motion is followed the tracks of;

3, use human joint points as input, algorithm is simple, and time complexity is low.

Description of drawings

Fig. 1 is the human motion tracker block diagram that the present invention is based on VSMM;

Fig. 2 is the human body motion tracking method general flow chart that the present invention is based on VSMM;

Fig. 3 is human motion image pre-service sub-process figure of the present invention;

Fig. 4 is that human joint points of the present invention detects sub-process figure;

Fig. 5 is the human joint points testing result figure of emulation experiment of the present invention;

Fig. 6 is the 3D human skeleton illustraton of model that the present invention tests use;

Fig. 7 is total motion model collection topological diagram of emulation experiment of the present invention;

Fig. 8 is that podomere projection angle of the present invention changes exemplary plot;

Fig. 9 is emulation experiment testing result of the present invention, front projection result and three-dimensional result figure;

Figure 10 is the model probability figure as a result of emulation experiment of the present invention;

Figure 11 is the right elbow of emulation experiment human body of the present invention and right hand 3D projection as a result and the Error Graph that detects articulation point.

Embodiment

With reference to Fig. 1, the human motion tracker that the present invention is based on VSMM comprises: pretreatment unit, the articulation point pick-up unit, the motion model trainer, motion model collection covering design device, initialization model group's device, interactive multi-model mixing estimation unit, motion model group active device, the new model group model apparatus for initializing that activates, the model group stops device, wherein: pretreatment unit, obtain the human motion image, then with background image do poor, obtain the background subtraction image, handle the background subtraction image with morphological method, obtain human body silhouette clearly, adopt border following algorithm to obtain human body silhouette outline, refinement human body silhouette obtains the skeleton line of human body silhouette; The articulation point pick-up unit according to pretreated image, rule of thumb detects the human joint points of each podomere respectively; The motion model trainer, it is right to extract training data from movement capturing data, adopts Ridge Regression Modeling Method training motion model equation state matrix, obtains total motion model collection; Motion model collection covering design device according to transition probability between the motion model and topological structure, divides into groups to total motion model collection, obtains some motion model groups; Initialization model group's device uses interacting multiple model algorithm to obtain the initial model group of variable structure multi-model algorithm; Interactive multi-model mixing estimation unit is input with the human joint points, and current model group's motion model system of equations committee state equation obtains the human motion attitude and estimates, upgrades motion model probability and human motion attitude estimation error covariance; Motion model group active device activates rule if the model group is satisfied in the variation of the projection angle of four limbs, then activates corresponding candidate's model group; The new model group model apparatus for initializing that activates, candidate's model group that motion model group active device is obtained carries out initialization; The model group stops device, if new activation model group and former current model group's model group likelihood ratio or model group likelihood ratio less than 0.9, then stop the new model group who activates, if model group likelihood ratio and model group likelihood ratio then stop former current model group all greater than 1.

With reference to Fig. 2, the present invention is based on the human body motion tracking method of VSMM, the specific implementation process is as follows:

Step 1 is done pre-service to input picture, obtains human body silhouette and outline thereof, skeleton line.

With reference to Fig. 3, this step is implemented as follows:

1.1) before human body did not enter the video camera visual angle, the empty bat background area 3-5 second asked arithmetic mean to each pixel position that sky shoots the video in the image, remembers that final mean value image is a background image;

1.2) obtain the human motion image, to do pixel poor with background image, obtains the background subtraction image;

1.3) adopt the noise of cutting apart in the morphological method removing background subtraction image, obtain human body silhouette clearly;

1.4) adopt border following algorithm to obtain human body silhouette outline; Refinement human body silhouette obtains the skeleton line of human body silhouette.

Step 2:, carry out articulation point and detect to pretreated human body video image.

With reference to Fig. 4, this step is implemented as follows:

2.1) use the concentric circles template to search for along skeleton line, will fall into the human body silhouettes point center of circle the most for a long time of annulus as head node;

2.2) to choose human body silhouette center of gravity position be root node, the arithmetic mean of everyone side shadow point x coordinate figure is as the x coordinate of root node, and the arithmetic mean of y coordinate figure is as the y coordinate of root node;

2.3) with the root node benchmark projection on video image with 3D human skeleton model, obtain trunk central point, clavicle joint point, left and right sides shoulder point and left and right sides buttocks articulation point;

2.4) detect the end points that obtains human body silhouette skeleton line, determine hand node and pin node according to nearest neighbouring rule;

2.5) make the parallel lines of bipod line by the barycenter of lower part of the body silhouette, two intersection points of itself and skeleton line are as knee joint;

2.6) with equidistant as elbow joint with hand and shoulder on the skeleton line;

2.7) to because of blocking or can't detected part articulation point, then adopting the Kalman filtering method because of cutting apart noise, one-step prediction obtains.

The result who human joint points is detected by this step as shown in Figure 5.

Step 3: motion model is trained, obtain total motion model set M.

May be used on multiple motion model in the experiment, motion model is trained, as stiff walking model m ₁, walking model m ₂, stretching hand with arm keeps balance walking model m ₃, Jack mode hopping type m ₄, hopping model m ₅With the model m that squats down ₆, the present invention adopts walking model m ₂, but be not limited to this motion model, its training step is as follows:

3.1) from the CMU of CMU motion capture database, choose the seizure data of walking mode, extract the joint angles that needs, be converted into hypercomplex number and represent, it is right to form training data

3.2) establish

Expression walking model m ₂The motion model equation,

The human body sport parameter of expression walking model, F ²The state-transition matrix of expression motion model equation, The noise of expression motion model;

3.3) F ²Calculate according to following formula:

$F^2=\arg \underset{F^2}{\min }\{\underset{k=1}{\overset{114}{\mathrm{\Σ}}}{\left|\right|F^2{x}_k^2-x_{k+1}^2\left|\right|}^2+R\left(F^2\right)\}---1)$

Wherein, R (F ²)=λ || F ²|| ², λ is the regularization factor, λ in experiment of the present invention=0.15;

All motion models are all pressed the above-mentioned steps training, finally obtain total motion model collection M={m ₁, m ₂, m ₃, m ₄, m ₅, m ₆.

Step 4: total motion model collection is divided into groups.

In total motion model collection M, according to transition probability between the motion model and topological structure, analyze connectedness and redirect possibility between two motion models, if two motion models are not only connection, and model probability again can the between redirect, then claims the human motion pattern similarity of two motion models couplings, and these two kinds of motion models are divided in same model group, otherwise, these two motion model branches are gone into different model groups, each motion model group comprises 3 motion models; In emulation experiment, stiff walking model m ₁, walking model m ₂, stretching hand with arm keeps balance walking model m ₃All with the walking mode coupling, so these three motion model branches are gone into same motion model group; Topological structure between total motion model transporting something containerized movable model as shown in Figure 7, the transition probability between total motion model transporting something containerized movable model is as shown in table 1, total motion model collection group result is as shown in table 2.

Step 5: the current model group of initialization M ₁

The motion model equation that total motion model is concentrated with interactive multi-model ten cycles of operation, calculates each model group's model group probability as the state equation of interactive multi-model wave filter, and the model group who selects the probability maximum is as initial current model group M ₁

Step 6: use interacting multiple model algorithm to calculate the human motion attitude and estimate.

At first design the state equation and the measurement equation of model in the interacting multiple model algorithm, then with k constantly human joint points as input, upgrade and four steps of state estimation fusion through the initialization of model condition, model condition filtering, model probability, the final human motion attitude that obtains estimates that concrete implementation step is as follows:

6.1) select to comprise the interacting multiple model algorithm of 3 models, system state equation and the measurement equation of establishing model i are as follows:

$x_{k+1}^i=F^i{x}_k^i+w_k^i,i=\mathrm{1,2,3}---2)$

$z_k=H\left(x_k^i\right)+v_k---3)$

In the formula, Be the state vector of model i, F ⁱBe state-transition matrix, identical with the motion model state transition equation of being trained in the step 4,

Be state-noise, Q ₁..., Q ₉Be the anglec of rotation of the human joint points represented with hypercomplex number, T ₀, Q ₀..., Q ₉Pairing articulation point position as shown in Figure 6, wherein, T ₀The expression human body is at the whole displacement of global coordinate system, Q ₀The anglec of rotation of expression global coordinate system, Q ₁The anglec of rotation of representing left stern articulation point, Q ₂The anglec of rotation of representing left knee joint point, Q ₃The anglec of rotation of representing right stern articulation point, Q ₄The anglec of rotation of representing right knee joint point, Q ₅The anglec of rotation of representing left shoulder joint node, Q ₆The anglec of rotation of representing left elbow joint point, Q ₇The anglec of rotation of representing right shoulder joint node, Q ₈The anglec of rotation of representing right elbow joint point, Q ₉The anglec of rotation of expression P point; z _kBe k human joint points picture position constantly, totally 34 dimensions; H (x _k) for measuring transition matrix, v _kBe measurement noise;

6.2) initialization of model condition

The wave filter of considering each model all might become current effective system model wave filter, the starting condition of each model filtering device all is each model filtering result's of previous moment a weighted sum, weights are corresponding model probability, calculate respectively and mix probability and mix estimation, and implementation step is as follows:

6.2a) calculate and mix probability

Note k-1 Matching Model constantly is

And k Matching Model constantly is

With k-1 information Z constantly ^K-1For the mixing probability of condition is:

$u_{k-1|k-1}^{(i,j)}=P\left(m_{k-1}^i\right|m_k^j,Z^{k-1})=\frac{1}{{\stackrel{\‾}{c}}_j}{\mathrm{\π}}_{\mathrm{ij}}{u}_{k-1}^i---4)$

Wherein

Be normaliztion constant,

Be k-1 moment Matching Model Probability, π _IjBe Matching Model

To Matching Model

Transition probability, Z ^K-1={ z ₁, z ₂..., z _K-1;

6.2b) calculate to mix and estimate

To k Matching Model constantly

Heavy init state

And error covariance matrix

Mixing estimate to be respectively:

${\stackrel{\widehat{^}}{x}}_{k-1|k-1}^j=E\left(x_{k-1}\right|m_k^j,Z^{k-1})=\underset{i=1}{\overset{3}{\mathrm{\Σ}}}{\hat{x}}_{k-1|k-1}{u}_{k-1|k-1}^{(i,j)}---5)$

${\hat{P}}_{k-1|k-1}^j=\underset{i=1}{\overset{3}{\mathrm{\Σ}}}[P_{k-1|k-1}^i+({\hat{x}}_{k-1|k-1}^i-{\hat{\hat{x}}}_{k-1|k-1}^j){({\hat{x}}_{k-1|k-1}^i-{\hat{\hat{x}}}_{k-1|k-1}^j)}^T]u_{k-1|k-1}^{(i,j)}---6)$

Wherein,

The expression Matching Model To the estimation of human motion attitude,

Expression mixes probability;

6.3) estimation of calculating human body attitude

And error covariance

Residual error

And covariance

Measure z _kWith Matching Model

The likelihood function of coupling

Filter gain

Estimate to upgrade with the human motion attitude

And error covariance matrix

6.3a) will weigh initialized state and covariance matrix and estimate promptly by mixing

With

The substitution Matching Model

Wave filter, obtain state estimation

And error covariance

${\hat{x}}_{k|k-1}^j=F^j{\hat{\hat{x}}}_{k-1|k-1}^j---7)$

$P_{k|k-1}^j=F^j{\hat{P}}_{k-1|k-1}^j{\left(F^j\right)}^T+Q_{k-1}^j---8)$

Wherein,

The expression Matching Model

Noise covariance.

6.3b) will weigh initialized state

Substitution measures transition matrix H (), calculates to measure residual error

And covariance matrix

${\tilde{z}}_k^j=z_k-H\left({\hat{x}}_{k|k-1}^j\right)---9)$

$S_k^j=h_k{P}_{k|k-1}^j{\left(h_k\right)}^T+R_k^j---10)$

Wherein, z _kExpression k measurement constantly, The expression Matching Model

The measurement noise covariance, h _kExpression measures the Jacobian matrix of matrix H.

6.3c) under Gauss's hypothesis, with residual error And covariance matrix

The substitution following formula calculates and measures z _kWith Matching Model

The likelihood function of coupling

${\text{\Λ}}_k^j=p\left(z_k\right|m_k^j,Z^{k-1})$

$\≈p\left[z_k\right|m_k^j,{\hat{\hat{x}}}_{k-1|k-1}^j,S_k^j\left({\hat{P}}_{k-1|k-1}^j\right)]$ $11)$

$\≈2\mathrm{\π}{\left|S_k^j\right|}^{-1/2}\exp \{-\frac{1}{2}{\left({\tilde{z}}_k^j\right)}^T{\left(S_k^j\right)}^{-1}{\tilde{z}}_k^j\}$

Wherein,

The expression Matching Model The measurement prediction residual.

6.3d) with state estimation

And error covariance

Residual error

And covariance matrix

The substitution following formula, the calculation of filtered gain The human motion attitude is estimated to upgrade

And error covariance matrix

$K_k^j=P_{k|k-1}^j{\left(h_k\right)}^T{\left(S_k^j\right)}^{-1}---12)$

${\hat{x}}_{k|k}^j={\hat{x}}_{k|k-1}^j+K_k^j{\tilde{z}}_k^j---13)$

$P_{k|k}^j=P_{k|k-1}^j-K_k^j{S}_k^j{\left(K_k^j\right)}^T---14)$

6.4) the model probability renewal

According to step 6.3c) likelihood function that obtains

Calculate k Matching Model constantly

Probability

$u_k^j=P\left(m_k^j\right|Z^k)=\frac{1}{c}{\mathrm{\Λ}}_k^j{\stackrel{\‾}{c}}_j---15)$

Wherein

Be normaliztion constant, and

6.5) the state estimation fusion

According to step 6.3d) Matching Model that calculates

State estimation

With step 6.4) the Matching Model probability that obtains

Calculating k human motion attitude constantly estimates and human motion attitude estimation error covariance matrix:

6.5a) utilize following formula to k human motion attitude constantly

Estimate:

${\hat{x}}_{k|k}=\underset{j=1}{\overset{3}{\mathrm{\Σ}}}{\hat{x}}_{k|k}^j{u}_k^j---16)$

Wherein, Be k moment Matching Model

The human motion attitude estimate,

Be k moment Matching Model Probability.

6.5b) utilize following formula to calculate k human motion attitude estimation error covariance matrix P constantly _K|k:

$P_{k|k}=\underset{j=1}{\overset{3}{\mathrm{\Σ}}}[P_{k|k}^j+({\hat{x}}_{k|k}-{\hat{x}}_{k|k}^j){({\hat{x}}_{k|k}-{\hat{x}}_{k|k}^j)}^T]u_k^j---17)$

Wherein,

Be k moment Matching Model

Human motion attitude estimation error covariance,

Expression k human motion constantly attitude is estimated.

Step 7: the motion model group activates.

According to the human joint points position, calculate the projection angle changing value of extremities joint point on image, be that example podomere projection angle changes as shown in Figure 8 with lower limb, if the projection angle value satisfies following activation model group rule, then remember k ₀=k carry out the following new model group model initialization step that activates, otherwise output human motion attitude is estimated, carries out above-mentioned steps 6.

A) if the projection angle changing value of buttocks is 2 times of last time, then motion model group 3 is activated;

B) if the projection angle changing value of most of lower limb is 2 times of last time, then motion model group 2 is activated;

C), then motion model group 1 is activated if the projection angle changing value of most of lower limb is 1/2 of the last time.

Step 8: newly activate candidate's model group initialization.

Remember that current model group is M _k, former current model group is M _oWith candidate's model group of new activation be M _n, make M _o=M _k, M _k=M _n∪ M _o

8.1) newly activate motion model m _iProbability be initialized as:

$\widehat{\mathrm{\μ}}\left(m_i\right|M_n,Z^k)=\max \left(\widehat{\mathrm{\μ}}\left(m_j\right|M_o,Z^k)\right)---18)$

The current model group of normalization M _kMiddle model probability

Represent former current model group M _oMiddle model m _jProbability estimate;

8.2) the predicated error covariance is initialized as the noise covariance of motion model;

8.3) choose in the movement capturing data with the highest state of present mode matching degree as original state;

8.4) with master mould group M _oCandidate's model group M with new activation _nMerge into new current model group.

Step 9: the model group stops.

According to new current model group, to model group M _l=M _n, M _o, calculate respectively:

Wherein,

The expression k moment, the model probability of i motion model,

The expression k moment, model group M _lModel group probability and,

The expression k moment, the model likelihood of i motion model,

The expression k moment, model group M _lModel group likelihood and;

The expression k moment, the model probability estimation of i motion model,

The expression k moment, model group M _lModel group probability estimate and, the model group stops carrying out as follows:

9.1) if model group M _nWith model group M _oModel group likelihood ratio Perhaps model group likelihood ratio

Less than 0.9, then stop model group M _n, output human motion attitude is estimated, and is returned and carry out above-mentioned steps 6;

9.2) if model group likelihood ratio

With model group likelihood ratio

All, then stop model group M greater than 1 _o, output human motion attitude is estimated, and is returned and carry out above-mentioned steps 6;

9.3) if step 9.1) and step 9.2) all be not performed, then export the human motion attitude and estimate, and continue to carry out this step.

Effect of the present invention can further specify by following emulation experiment:

1) the used data of emulation experiment

In the emulation experiment, the employed data of training motion model obtain from the CMU motion capture database, and the form of data is ASF+AMC, and data content is the joint anglec of rotation that Eulerian angle are represented, therefrom extract the needed human joint points anglec of rotation in the experiment, be converted to hypercomplex number then and represent.

The human motion video that uses in the experiment is from shooting the video, the video image size is 320 * 240, the empty image of clapping of preceding 100 frames is used for rebuilding background, back 450 frames are used for doing the human body motion tracking, the human motion that comprises in the video sequence has: the marking time of 1-120 frame, the hand of 121-250 frame is waved and is striden, and the Jack of 251-390 frame jumps, and the 391-450 frame is squatted down.

2) emulation content

Adopt the variable structure multi-model method that the human motion sequence is followed the tracks of.Obtain movement capturing data from the CMU motion capture database, the modelling of human body motion of training is respectively: stiff walking model m ₁, walking model m ₂, stretching hand with arm keeps balance walking model m ₃, Jack mode hopping type m ₄, hopping model m ₅With the model m that squats down ₆, as shown in table 1; Topological relation between the motion model as shown in Figure 7, according to table 1 and Fig. 7, analyze connectedness and redirect possibility between the motion model, if two motion models are not only connection, and model probability again can the between redirect, then claims the human motion pattern similarity of two motion models couplings, and these two kinds of motion models are divided in same model group, otherwise, these two motion model branches are gone into different model groups, to total motion model collection M={m ₁, m ₂, m ₃, m ₄, m ₅, m ₆Grouping, group result is as shown in table 2; The human motion attitude merges estimates to adopt the variable structure multi-model algorithm to obtain.

Table 1 motion model transition probability

The total motion model group group result of table 2

3) simulation result and analysis

Adopt the variable structure multi-model algorithm that the human motion in shooting the video is certainly followed the tracks of, final human articulation point testing result, 3D human motion tracking results and 2D projection thereof are as shown in Figure 9, wherein, the human joint points testing result is shown in Fig. 9 (a), human motion is followed the tracks of the result of two-dimensional projection shown in Fig. 9 (b), and human motion is followed the tracks of the 3 d pose estimated result shown in Fig. 9 (c); From Fig. 9 (b) as can be seen: the 2D projection on image of human motion attitude estimated result overlaps with the human skeleton line basically; From Fig. 9 (c) as can be seen: the 3D effect of human motion attitude estimated result is identical with real human motion attitude, and the present invention has effectively solved the ambiguity problem of human motion, has improved accuracy and stability that human motion is followed the tracks of.

In the tracking test, the model probability of each motion model changes as shown in figure 10, wherein, and stiff walking model m ₁Model probability change shown in Figure 10 (a) walking model m ₂Model probability change as Figure 10 (b) shown in stretching hand with arm maintenance balance walking model m ₃Model probability change shown in Figure 10 (c) Jack mode hopping type m ₄Model probability change shown in Figure 10 (d) hopping model m ₅Model probability change shown in Figure 10 (e), model m squats down ₆Model probability change shown in Figure 10 (f); As can be seen from Figure 10: each constantly all has only a motion model to play main effect, when motion model and human motion pattern similarity, the model probability of motion model is bigger, when the human motion pattern changes, the motion model that plays a major role changes thereupon, and motion model group of the present invention activates rule and well finished motion model group activate a task.

The error of projection of 3D articulation point and articulation point physical location as shown in figure 11, wherein, the error of right elbow spot projection of 3D and right elbow physical location is shown in Figure 11 (a), the error of 3D right hand spot projection and right hand physical location is shown in Figure 11 (b), as can be known from Fig. 11, as seen average error uses the variable structure multi-model method less with result's projection error between 2.7cm-5.2cm.

Emulation experiment of the present invention compiles on Matlab and finishes, and execution environment is 1 a second/frame for the HP workstation under the windows framework, human joint points detection speed, and human motion was tracked as for 10 frame/seconds, and time complexity is low.

The present invention uses the variable structure multi-model method to carry out human motion and follows the tracks of, and the motion model that the training of employing movement capturing data obtains makes motion tracking meet the human motion rule more as the state equation of wave filter, has reduced the influence of human motion ambiguousness; The use that total motion model collection covers has not only solved the problem that complicated human motion can't be accurately followed the tracks of in little motion model set, and avoided the unnecessary competition of using a large amount of motion models to bring simultaneously, this competition not only can improve time complexity, also can reduce the accuracy of tracking simultaneously; The characteristics of image that extracts is simple, has reduced single frames working time; When following the tracks of every frame, only use the motion model compatible with current motor pattern, rather than total motion model collection, time complexity reduced.Simulation result shows, this tracking has obtained two-dimensional projection and three-dimensional pose recovery accurately, has reduced the human motion ambiguousness, and time complexity is low.

Claims (4)

1. human body motion tracking method based on VSMM comprises:

Pre-treatment step: input human body video image, obtain the human body silhouette by background subtraction, extract human body silhouette outline, and the human body silhouette is carried out thinning processing;

Articulation point detects step: to pretreated human body video image, carry out following articulation point and detect:

1) uses the concentric circles template to search for, will fall into the point center of circle the most for a long time of annulus as head node along skeleton line;

2) choosing human body silhouette center of gravity position is root node;

3) use the projection on image of 3D human skeleton model, obtain other articulation point positions on the trunk;

4) choose the skeleton line vertex position as hand node and pin node;

5) make the parallel lines of bipod line by the barycenter of lower part of the body silhouette, two intersection points of itself and skeleton line are as knee joint;

6) with equidistant as elbow joint on the skeleton line with hand and shoulder;

7) to because of blocking or can't detected part articulation point because of cutting apart noise, then adopt the Kalman filtering method, one-step prediction obtains;

Motion model training step: from the CMU of CMU motion capture database, choose the seizure data of multiple motor pattern, adopt the state-transition matrix F of Ridge Regression Modeling Method to the motion model equation ⁱTrain, and calculate the noise w of this motion model _kCovariance, the motion model set of acquisition is called total motion model collection M;

Motion model collection covering design step: in total motion model collection M, if two human motion pattern similarities that motion model mated then divide these two kinds of motion models in same model group, otherwise, it is divided into different model groups; Each model group comprises 3 motion models;

Initialization model group's step: the motion model equation that total motion model is concentrated is as the state equation of interactive multi-model wave filter, with interactive multi-model ten cycles of operation, calculate each model group's model group probability, the model group who selects the probability maximum is as initial current model group M ₁

Interactive multi-model mixes estimating step: human joint points is as input constantly with k, and the execution interacting multiple model algorithm obtains the human motion attitude and estimates, upgrades motion model probability and human motion attitude estimation error covariance;

The motion model group activates step: according to the human joint points position, calculate the angle changing value of extremities joint point skeleton line projection on image, if the changing value size satisfies the model group and activates rule, remember that this is k constantly ₀, carry out the following new model group model initialization step that activates, otherwise output human motion attitude is estimated, carries out above-mentioned interactive multi-model and mixes estimating step;

The new model group model initialization step that activates: the probability that will newly activate model is initialized as model probability maximal value among the current model group, and normalization model probability; The predicated error covariance is initialized as the noise covariance of motion model self; Choose in the movement capturing data with the highest state of present mode matching degree as original state; With master mould group M _oCandidate's model group M with new activation _nMerge into new current model group;

The model group stops step: the current model group according to new re-executes above-mentioned interactive multi-model one-period, if model group M _nWith model group M _oModel group likelihood ratio

Perhaps model group likelihood ratio

Less than 0.9, then stop model group M _n, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimating step; If

With

All, then stop model group M greater than 1 _o, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimating step; Otherwise output human motion attitude is estimated, and is continued to carry out this step.

2. human body motion tracking method according to claim 1, wherein the employing Ridge Regression Modeling Method described in the motion model training step is to the state-transition matrix F of motion model equation ⁱTraining, is to carry out as follows:

2a) establish

Represent i the motion model equation that will train, wherein F ⁱThe expression state-transition matrix;

It is right 2b) to obtain data from movement capturing data

Wherein It is the human body sport parameter of representing with hypercomplex number;

2c) F ⁱObtain according to following formula:

$F^i=\arg \underset{F^i}{\min }\{\underset{k=1}{\overset{n-1}{\mathrm{\Σ}}}{\left|\right|F^i{x}_k^i-x_{k+1}^i\left|\right|}^2+R\left(F^i\right)\}$

Wherein, R (F ⁱ)=λ || F ⁱ|| ², λ is the regularization factor, λ in experiment of the present invention=0.15.

May be used on multiple motion model in the experiment, motion model is trained, as stiff walking model m ₁, walking model m ₂, stretching hand with arm keeps balance walking model m ₃, Jack mode hopping type m ₄, hopping model m ₅With the model m that squats down ₆, all motion models are all pressed the above-mentioned steps training, finally obtain total motion model collection M={m ₁, m ₂, m ₃, m ₄, m ₅, m ₆.

3. human body motion tracking method according to claim 1, wherein the motion model group activates the model group described in the step and activates rule, comprising:

3a), then motion model group 3 is activated if the projection angle changing value of buttocks is 2 times of last time;

3b), then motion model group 2 is activated if the projection angle changing value of most of lower limb is 2 times of last time;

3c) if the projection angle changing value of most of lower limb is 1/2 of the last time, then motion model group 1 is activated.

4. human motion tracker based on VSMM comprises:

Pretreatment unit: be used to import the human body video image, obtain the human body silhouette, extract human body silhouette outline, and the human body silhouette is carried out thinning processing by background subtraction;

Articulation point pick-up unit: be used for pretreated human body video image, carry out following articulation point and detect:

1) uses the concentric circles template to search for, will fall into the point center of circle the most for a long time of annulus as head node along skeleton line;

2) choosing human body silhouette center of gravity position is root node;

3) use the projection on image of 3D human skeleton model, obtain other articulation point positions on the trunk;

4) choose the skeleton line vertex position as hand node and pin node;

5) make the parallel lines of bipod line by the barycenter of lower part of the body silhouette, two intersection points of itself and skeleton line are as knee joint;

6) with equidistant as elbow joint on the skeleton line with hand and shoulder;

7) to because of blocking or can't detected part articulation point because of cutting apart noise, then adopt the Kalman filtering method, one-step prediction obtains;

Motion model trainer: be used for choosing the seizure data of multiple motor pattern, adopt the state-transition matrix F of Ridge Regression Modeling Method to the motion model equation from CMU's CMU motion capture database ⁱTrain, and calculate the noise w of this motion model _kCovariance, the motion model set of acquisition is called total motion model collection M;

Motion model collection covering design device: be used at total motion model collection M, if two human motion pattern similarities that motion model mated then divide these two kinds of motion models in same model group, otherwise, it is divided into different model groups; Each model group comprises 3 motion models;

Initialization model group's device: be used for the state equation of the motion model equation that total motion model is concentrated as the interactive multi-model wave filter, with interactive multi-model ten cycles of operation, calculate each model group's model group probability, the model group who selects the probability maximum is as initial current model group M ₁

Interactive multi-model mixing estimation unit: be used for k constantly human joint points carry out interacting multiple model algorithm as input, obtain the human motion attitude and estimate, upgrade motion model probability and human motion attitude estimation error covariance;

Motion model group active device: be used for according to the human joint points position, calculate the angle changing value of extremities joint point skeleton line projection on image,, remember that this is k constantly if the changing value size satisfies the model group and activates rule ₀, carry out the following new model group model apparatus for initializing that activates, otherwise output human motion attitude is estimated, carries out above-mentioned interactive multi-model mixing estimation unit;

The new model group model apparatus for initializing that activates: the probability that is used for newly activating model is initialized as current model group model probability maximal value, and normalization model probability; The predicated error covariance is initialized as the noise covariance of motion model self; Choose in the movement capturing data with the highest state of present mode matching degree as original state; With master mould group M _oCandidate's model group M with new activation _nMerge into new current model group;

The model group stops device: be used for re-executing above-mentioned interactive multi-model one-period according to new current model group, if model group M _nWith model group M _oModel group likelihood ratio

Perhaps model group likelihood ratio Less than 0.9, then stop model group M _n, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimation unit; If

With

All, then stop model group M greater than 1 _o, output human motion attitude is estimated, and is returned and carry out above-mentioned interactive multi-model mixing estimation unit; Otherwise output human motion attitude is estimated, and is continued to carry out this device.

Images