CN102289664B - Method for learning non-linear face movement manifold based on statistical shape theory - Google Patents

Abstract

The invention discloses a method for learning a non-linear face movement manifold based on a statistical shape theory. A method for pre-processing face shape based on the statistical shape theory comprises the following steps of: (1) demeaning, normalizing and pluralizing the shape of each frame in a face movement sequence; (2) removing redundant information in complex representation; and (3) by combining Riemannian geometry tangent space mapping, projecting the face movement sequence of the complex representation into a tangent space of the movement manifold to form a face movement locus. By using a Gaussian process latent variable model, the method for learning the face movement manifold comprises the following steps of: (1) calculating a mean value and a covariance function of a Gaussian process, and determining a probability density function of the constructed Gaussian process; and (2) solving a latent variable by using a scaled conjugate gradient method to obtain a dimension reduction result which corresponds to the face movement locus. In the method, the dimension of face movement data is reduced by using a true manifold distance and using a good dimension reduction method, so that the structure of the face movement manifold is more accurately described.

Description

Non-linear facial movement manifold learning based on the Statistical Shape theory

(1) technical field:

The present invention relates to a kind of non-linear facial movement manifold learning based on the Statistical Shape theory, especially in conjunction with Statistical Shape theoretical (Statistical Shape Theory) and Gaussian process hidden variable model (the Gaussian Processing Latent Variable Models) description to facial motion sequence, belong to image and process, area of pattern recognition.

(2) background technology:

How giving the ability of machine recognition facial movement, make facial movement as the another kind of machine input mode, thereby auxiliary engine is understood the mankind's intention, is better the main task of face analysis now for the mankind serve.Face analysis converts image, video information to machine understandable pattern information take face as research object, is important branch of area of pattern recognition.Face analysis has experienced two stages substantially, that is: still image analysis and dynamic image sequence analysis.The still image information analysis that combining image is processed mainly utilizes static nature (as: point, line etc.), facial model is identified, extracted pattern information, based on the face analysis of the static nature convenience because of its feature extraction, the real-time of processing procedure is subject to early stage researcher's favor.But static information does not comprise the characteristic of facial movement, isolated face-image can not reflect whole motion characteristics, development along with the image sequence analysis, contain in the sequence information of interframe and be introduced in face analysis, expanded the quantity of information of face analysis in conjunction with the dynamic image sequence analysis of inter-frame information.But the dimension of facial motion data is higher, easily cause " dimension disaster ", traditional facial movement sequence analysis method hypothesis facial movement is distributed in a linear manifold, use is based on the method (as: PCA of overall linear hypothesis, MDS etc.) to facial Data Dimensionality Reduction, analyze facial movement, extract inter-frame information.In recent years, along with the development of cognitive psychology and machine vision, it is non-linear that the researcher recognizes that facial movement has, and facial change profile is in the non-linearity manifold of a low-dimensional, and there is birth defect in existing linear analysis method.At present, although some non-linearity manifold study methods are applied in face analysis, mostly do not consider the characteristics of facial movement, according to the needs of face analysis, the method is not revised.Therefore, design a kind of manifold learning based on the facial movement characteristics and analyze facial movement, significant to the development of machine learning and pattern-recognition.

(3) summary of the invention:

the objective of the invention is: 1. the face analysis algorithm does not carry out pre-service according to the face shape characteristics mostly at present, for this deficiency, introduce Statistical Shape theoretical (Statistical Shape Theory) face shape is carried out pre-service, the facial movement sequence is converted to track in facial movement stream shape tangent space (The tangent space offace movement manifold), better (Fig. 1 is the schematic diagram that Euclidean distance and the upper distance of stream shape are compared to the distance between the calculating face shape, (a) hemisphere face in is the distributed flow shape of data, red line is the Euclidean distance of point-to-point transmission, (b) be Euclidean distance and the upward comparison of distance of stream shape, increase along with distance, the upper distance of stream shape is increasing with the Euclidean distance gap), be convenient to follow-up manifold learning arithmetic facial movement is carried out dimensionality reduction.2. be distributed in characteristics in the low-dimensional non-linearity manifold according to facial movement, use Gaussian process hidden variable model (Gaussian Processing Latent Variable Models) that facial movement stream shape is estimated, reach and reduce the facial motion data dimension, the effect that discloses the facial movement rule.

A kind of non-linear facial movement manifold learning based on the Statistical Shape theory of the present invention; Comprise pre-service and two major parts of Gaussian process hidden variable model face motion manifold learning (block diagram of whole system is as shown in Figure 2) based on the Statistical Shape theory, wherein:

Pre-service based on Statistical Shape theoretical (Statistical Shape Theory) is: the facial movement sequence mapping is flowed in the tangent space of shape the Euclidean distance in this space and face shape spacing (Fig. 3 is the processing block diagram of Statistical Shape theory) of equal value to facial movement by going the steps such as average, normalization, plural number mapping, tangent space mapping.

Adopt the facial movement manifold learning of Gaussian process hidden variable model (Gaussian Processing Latent Variable Models) to be: to be distributed in characteristics in non-linearity manifold for facial movement, adopting Method of Nonlinear Dimensionality Reduction to carry out dimension to facial motion data approximately subtracts, with the lower facial motion change of dimension reflection, tranquil and 6 kinds of facial expressions are described: happy, surprised, detest, sad, indignation, frightened, thereby avoid " dimension disaster " that cause because dimension is too high.

A kind of non-linear facial movement manifold learning based on the Statistical Shape theory of the present invention wherein carries out front and continued based on the pre-service of Statistical Shape theory to facial motion sequence and processes, and improves the accuracy of follow-up manifold learning arithmetic, and its concrete steps are:

Suppose Ω={ ω _i| i=1,2...N} are face shape motion sequence, wherein a ω _i={ (x _i1, y _i1), (x _i2, y _i2) ... (x _iM, y _iM) represent to put by M the pattern section's shape (distribution of face shape as shown in Figure 4) that forms.

Step 1: each the frame shape in facial motion sequence Ω is removed average value processing, make

Step 2: each the frame shape in facial motion sequence Ω is made normalized, make

Step 3: each the frame shape in facial motion sequence Ω is done the plural numberization processing, and the horizontal ordinate of each point is real part, and ordinate is imaginary part, and the real-valued face shape vector that 2 * M is tieed up is converted to the complex vector located ω of M dimension _i={ s ₁, s ₂... s _M, s wherein _ij=x _ij+ I * y _ij, I is complex unit, I ²=-1;

Step 4: remove the redundant information in complex representation, the real imaginary part sum of utilizing each shape is zero characteristics (result of step 1), with the facial sequence Ω of particular matrix premultiplication, makes the shape vector dimension of complex representation reduce to M-1;

Step 5: in conjunction with Riemannian tangent space mapping, the facial movement sequence of complex representation is projected to the tangent space of motion stream shape, forms the facial movement track.

A kind of non-linear facial movement manifold learning based on the Statistical Shape theory of the present invention, the facial movement manifold learning that wherein adopts Gaussian process hidden variable model with the facial movement track of front and continued as input, by Gaussian process hidden variable model to the further dimensionality reduction of this track, thereby the structure of portraying facial movement stream shape.

Suppose that ω=f (x) is a real-valued function, the utilization of Gauss's hidden variable model has the Gaussian process of parameter and approaches this function, determine the value of variable x by maximizing posterior probability, because this variable is implied in Gaussian process as parameter, therefore be called Gaussian process hidden variable model (Figure 5 shows that the structure diagram of Gauss's hidden variable model).Wherein Gaussian process refers to the stochastic process of Gaussian distributed, be g (x)～N (m (x), k (x, x)), m (x), k (x, x ') are respectively mean value function and covariance function, and the present invention adopts weights spatial configuration method (Weight-Space) structure Gaussian process implicit function model, wherein covariance function adopts the radial basis function structure, and concrete steps are as follows:

Step 1: calculate average and the covariance function of the Gaussian process of structure, the probability density function of definite Gaussian process of constructing;

Step 2: adopt yardstick method of conjugate gradient (Scaled Conjugate Gradient) to find the solution hidden variable, and then obtain the dimensionality reduction result of corresponding facial movement locus.

A kind of non-linear facial movement manifold learning based on the Statistical Shape theory of the present invention, its advantage and good effect are:

1. because facial movement is distributed in non-linearity manifold, Euclidean distance in the original-shape space differs far away with real shape distance, and the error of estimating the upper shape distance of stream shape with Euclidean distance constantly increases along with the increase of distance, this statistical shape model can project to face shape in the tangent space, and the actual distance between the Euclidean distance in this tangent space and the upper shape of stream shape is of equal value, can overcome well distance and calculate inaccurate shortcoming.

2. although the facial movement sequence through the tangent space mapping has reflected the face shape Changing Pattern, but still there is redundancy, need to further carry out dimension and approximately subtract, this Gaussian process hidden variable model utilizes it to the good approximation capability of non-linearity manifold, facial movement locus to be carried out dimensionality reduction.

3. adopt stochastic process to approach the characteristics that facial stream pictograph closes the facial movement ambiguity, obtained Approximation effect preferably.The facial movement sequence is carried out dimension-reduction treatment, be convenient to follow-up facial movement pattern classification.

Description of drawings:

Fig. 1 flows the upper distance of shape and compares with Euclidean distance

The entire block diagram of Fig. 2 system that the present invention builds

The processing block diagram of Fig. 3 Statistical Shape theory

Fig. 4 face shape point position view

Fig. 5 Gauss hidden variable model structure sketch

(5) embodiment:

A kind of non-linear facial movement manifold learning based on the Statistical Shape theory of the present invention; Comprise based on the pre-service of Statistical Shape theory and two parts of facial movement manifold learning of employing Gaussian process hidden variable model; Wherein:

One, about the pre-service based on the Statistical Shape theory, its step is as follows:

Suppose Ω={ ω _i| i=1,2...N} are face shape motion sequence, wherein a ω _i={ (x _i1, y _i1), (x _i2, y _i2) ... (x _iM, y _iM) expression puts by M the pattern section shape form.

Step 1: each the frame shape in facial motion sequence Ω is removed average value processing, at first obtain the center (x of each pattern section shape ₀, y ₀),

Then remove center information, i.e. x from shape data _ij'=x _ij-x ₀, y _ij'=y _ij-y ₀Make $\underset{j}{\mathrm{\Σ}}{x_{\mathrm{ij}}}^{\′}+\underset{j}{\mathrm{\Σ}}{y_{\mathrm{ij}}}^{\′}=0;$

Step 2: each the frame shape in facial motion sequence Ω is made normalized, at first calculate the quadratic sum of every frame shape (after the past average value processing, the center had been both initial point) distance to the center

Then use L to carry out normalization to shape data, namely ${x_{\mathrm{ij}}}^{\′\′}=\frac{{x_{\mathrm{ij}}}^{\′}}{\sqrt{L}},$ ${y_{\mathrm{ij}}}^{\′\′}=\frac{{y_{\mathrm{ij}}}^{\′}}{\sqrt{L}}$ Make $\underset{j}{\mathrm{\Σ}}{x}_{\mathrm{ij}}^{\′\′2}+\underset{j}{\mathrm{\Σ}}{y}_{\mathrm{ij}}^{\′\′2}=1;$

Step 3: each the frame shape in facial motion sequence Ω is done the plural numberization processing, and the horizontal ordinate of each point is real part, and ordinate is imaginary part, and the real-valued face shape vector that 2 * M is tieed up is converted to the complex vector located ω of M dimension _i={ s ₁, s ₂... s _M, s wherein _ij=x _ij+ I * y _ij, I is complex unit, I ²=-1;

Step 4: remove the redundant information in complex representation.The real imaginary part sum of utilizing each shape is zero characteristics (as step 1), and with the facial sequence Ω of particular matrix premultiplication, this matrix Ω can be with vector

Change

Utilizing each face shape average is zero characteristics, last column of facial movement shape sequence is made zero, remove last column again the shape vector dimension of complex representation is reduced to M-1, the complex number space that forms thus is called " canonical shape space " (Norshape Space);

Step 5: in conjunction with Riemannian tangent space mapping, the facial movement sequence of complex representation is projected to the tangent space of motion stream shape, forms the facial movement track.Process through above-mentioned steps " the canonical shape space " that obtain through above-mentioned steps and have a critical nature: horizontal subspace and the original-shape spatial isomorphism of canonical shape space tangent space, be that Euclidean distance and the manifold distance in the original-shape space in the horizontal subspace of canonical shape space tangent space is of equal value, can be by obtain reflecting the track of facial movement to the tangent space projection.Detailed discussion and concrete theoretical proof about statistical shape model please refer to document A.Kume, I.Dryden, H.Le, Shape-space smoothing splines for planar landmark data, Biometrika 94 (3) (2007) 513-528.

Two, about Gaussian process hidden variable model face motion manifold learning:

Suppose that ω=f (x) is a real-valued function, Gauss's hidden variable model utilizes Gaussian process to approach this function, determine the value of variable x by maximizing posterior probability, because this variable is implied in Gaussian process as parameter, therefore be called Gaussian process hidden variable model.Wherein Gaussian process refers to the stochastic process of Gaussian distributed, i.e. g (x)～N (m (x), δ ²(x)), m (x), δ ²(x) be respectively mean value function and covariance function, the present invention adopts weights spatial configuration method (Weight-Space) structure Gaussian process implicit function model, i.e. hypothesis (0, I), y is Gaussian distribution to w～N.Wherein covariance function adopts the radial basis function structure, and radial basis function is:

Concrete steps are as follows:

Step 1: calculate average and the covariance function of the Gaussian process of structure, the Gaussian process of constructing of probability density function determine to(for) the given facial movement sequence s wherein computing method of average and covariance function is:

Suppose $s=g\left(x\right)=\underset{j=1}{\overset{J}{\mathrm{\Σ}}}{\mathrm{\ω}}_j{\mathrm{\φ}}_j\left(x\right)=w^T\mathrm{\Φ}\left(x\right)$ Be Gaussian distribution,

m(x)＝E(s)＝E[w ^TΦ(x)]＝0

σ ²(x)=E (s _is _j)=E[(w ^TΦ (x _i)) (w ^TΦ (x _j))]=Φ (x _i) ^TΦ (x _j)=K, wherein K _ij=k (x _i, x _j) probability density function of Gaussian process can obtain analytic solution.

Step 2: adopt yardstick method of conjugate gradient (Scaled Conjugate Gradient) to find the solution hidden variable, and then obtain the dimensionality reduction result of corresponding facial movement locus.The objective function of yardstick method of conjugate gradient is:

$\underset{x}{\max \arg }p\left(S\right|x)=\underset{x}{\max \arg }\underset{i=1}{\overset{I}{\mathrm{\Π}}}p\left(s_i\right|x)$

Namely seek the shape sequence S={s that makes output ₁, s ₂... s _NJoint probability maximum, the detailed description of concrete computation process is referring to list of references: Jack M.Wang, David J.Fleet and Aaron Hertzmann, Multifactor Gaussian Process Models for Style-Content Separation, 24th International Conference on Machine learning, 2007,975-982.

Claims (1)

1. non-linear facial movement manifold learning based on the Statistical Shape theory is characterized in that: the method comprises based on the pre-service of Statistical Shape theory and adopts two parts of facial movement manifold learning of Gaussian process hidden variable model; Wherein:

(1) about the pre-service based on the Statistical Shape theory, its step is as follows:

Ω={ ω _i| i=1,2...N} are face shape motion sequence, wherein a ω _i={ (x _i1, y _i1), (x _i2, y _i2) ... (x _iM, y _iM) expression puts by M the pattern section shape form:

Step 1: each the frame shape in facial motion sequence Ω is removed average value processing, at first obtain the center (x of each pattern section shape ₀, y ₀), Then remove center information, i.e. x from shape data _ij'=x _ij-x ₀, y _ij'=y _ij-y ₀Make

Step 2: each the frame shape in facial motion sequence Ω is made normalized, at first calculate the quadratic sum of every frame shape (after the past average value processing, the center had been both initial point) distance to the center

Then use L to carry out normalization to shape data, namely ${x_{\mathrm{ij}}}^{\′\′}=\frac{{x_{\mathrm{ij}}}^{\′}}{\sqrt{L}},$ ${y_{\mathrm{ij}}}^{\′\′}=\frac{{y_{\mathrm{ij}}}^{\′}}{\sqrt{L}}$ Make $\underset{j}{\mathrm{\Σ}}{x^{\′\′}}_{\mathrm{ij}}^2+\underset{j}{\mathrm{\Σ}}{y^{\′\′}}_{\mathrm{ij}}^2=1;$

Step 3: each the frame shape in facial motion sequence Ω is done the plural numberization processing, and the horizontal ordinate of each point is real part, and ordinate is imaginary part, and the real-valued face shape vector that 2 * M is tieed up is converted to the complex vector located ω i={s of M dimension _i1, s _i2... s _iM, s wherein _ij=x _ij+ I * y _ij, j=1,2 ..., M, I are complex unit, I ²=-1;

Step 4: remove the redundant information in complex representation.The real imaginary part sum of utilizing each shape is that (as step 1), with the facial sequence Ω of particular matrix premultiplication, this matrix Ω can be with vector for zero characteristics

Change

Utilizing each face shape average is zero characteristics, last column of facial movement shape sequence is made zero, remove last column again the shape vector dimension of complex representation is reduced to M-1, the complex number space that forms thus is called " canonical shape space " (Norshape Space);

Step 5: in conjunction with Riemannian tangent space mapping, the facial movement sequence of complex representation is projected to the tangent space of motion stream shape, forms the facial movement track.Process through above-mentioned steps " the canonical shape space " that obtain through above-mentioned steps and have a critical nature: horizontal subspace and the original-shape spatial isomorphism of canonical shape space tangent space, be that Euclidean distance and the manifold distance in the original-shape space in the horizontal subspace of canonical shape space tangent space is of equal value, can be by obtain reflecting the track of facial movement to the tangent space projection;

(2) about Gaussian process hidden variable model face motion manifold learning, its step is as follows:

ω=f (x) is a real-valued function, and Gauss's hidden variable model utilizes Gaussian process to approach this function, determines the value of variable x by maximizing posterior probability, and this variable is implied in Gaussian process as parameter, therefore be called Gaussian process hidden variable model.Wherein Gaussian process refers to the stochastic process of Gaussian distributed, i.e. g (x) ~ N (m (x), δ ²(x)), m (x), δ ²(x) be respectively mean value function and covariance function, the present invention adopts weights spatial configuration method (Weight-Space) structure Gaussian process implicit function model, namely

(0, I), y is Gaussian distribution to w ~ N.Wherein covariance function adopts the radial basis function structure, and radial basis function is:

Concrete steps are as follows:

Step 1: calculate average and the covariance function of the Gaussian process of structure, the Gaussian process of constructing of probability density function determine to(for) the given facial movement sequence s wherein computing method of average and covariance function is:

Suppose $s=g\left(x\right)=\underset{j=1}{\overset{J}{\mathrm{\Σ}}}{\mathrm{\ω}}_j{\mathrm{\φ}}_j\left(x\right)=w^T\mathrm{\Φ}\left(x\right)$ Be Gaussian distribution,

m(x)＝E(s)＝E[w ^TΦ(x)]＝0

σ ²(x)=E (s _is _j)=E[(w ^TΦ (x _i)) (w ^TΦ (x _j))]=Φ (x _i) ^TΦ (x _j)=K, wherein K _ij=k (x _i, x _j)

When being known, the probability density function of Gaussian process can obtain analytic solution when mean value function and covariance function.

Step 2: adopt yardstick method of conjugate gradient (Scaled Conjugate Gradient) to find the solution hidden variable, and then obtain the dimensionality reduction result of corresponding facial movement locus.The objective function of yardstick method of conjugate gradient is:

$\underset{x}{\max \arg p}\left(S\right|x)=\underset{x}{\max \arg }\underset{i=1}{\overset{I}{\mathrm{\Π}}}p\left(s_i\right|x)$

Namely seek the shape sequence S={s that makes output ₁, s ₂... s _NJoint probability maximum.

Images