CN106778628A - A kind of facial expression method for catching based on TOF depth cameras - Google Patents
A kind of facial expression method for catching based on TOF depth cameras Download PDFInfo
- Publication number
- CN106778628A CN106778628A CN201611190469.1A CN201611190469A CN106778628A CN 106778628 A CN106778628 A CN 106778628A CN 201611190469 A CN201611190469 A CN 201611190469A CN 106778628 A CN106778628 A CN 106778628A
- Authority
- CN
- China
- Prior art keywords
- facial expression
- image
- model
- face
- depth cameras
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
- G06V40/176—Dynamic expression
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T13/00—Animation
- G06T13/20—3D [Three Dimensional] animation
- G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
The invention provides a kind of facial expression method for catching based on TOF depth cameras, it includes：The first step, facial expression seizure is carried out using TOF depth cameras to real person；Second step, processes the facial expression data for catching, and generate facial expression seizure file；3rd step, builds three-dimensional personage's geometrical model, imports facial expression and catches file, driving model.The method is directly recognized and is recorded the exercise data of facial key point without hand labeled Marker using the method for image recognition.The data of its capture are the real motion data of face, and action is coherent true to nature, and expression is fine and smooth abundant, and the facial expression that gets of the method catches file and can reuse, and has saved cost.
Description
Technical field
Single TOF depths are utilized the present invention relates to computer vision and computer graphics techniques field, more particularly to one kind
Spend the facial expressions and acts of cameras capture face face and be stored as specific data form and caught for the facial expression of cartoon model etc.
Catch method.
Background technology
In recent years, with the development of computer graphics, the facial expression animation of Computer Animated Graph synthesis people is to work as
One of emphasis for preceding animation disciplinary study, it enables animator with the expression direct drive picture image mould of performer
Type, receives animators and more pays close attention to.Facial expression catches the expression that can in real time detect, record performing artist,
Digitized " abstract expression " is converted into, so that animation software is by its " imparting " model, model is had as performing artist
Expression, and generate final expression animation sequence.
At present, facial expression animation can generally be divided into based on geometry, based on image, the research side based on motion tracking
Method.Wherein the research method based on geometry is mainly including keyframe interpolation method, parametric method, muscle model method etc.；Based on figure
The method of picture mainly includes：Anamorphose, dynamic texture mapping, expression of complexion change etc..
In the research method based on geometry, keyframe interpolation method sets up geometrical model at two different moment first,
Then enter row interpolation between the two models and obtain the model of intermediate time, so as to obtain facial expression animation, this is a kind of
Most traditional, most original face cartoon method, while be also a kind of face cartoon method of standard, but its weak point is
Need very big workload.Parametric method is in human face animation (Parke F I.A by Parke first Applications
Parameteric Model for Human Faces [D].Salt Lake City：University of Utah,
1974) action when, lip is spoken is suitable in this way, and the deficiency of this method is the human face expression for producing inadequate certainly
So, and in arrange parameter value substantial amounts of manual setting is needed.Muscle model method is by one group of bullet of composition muscle model
Property muscle contraction control face surface mesh to change and then simulate human face expression.
In the method based on image, anamorphose one kind is that character pair line segment hand-manipulated comes between two images
Realize that 2D deforms；Another kind is to be mapped to the parameter space of 2D by the 3D models for deforming to realize 3D model deformations and pass through
The 3D conversion and 2D deformations of geometrical model are combined, using 3D geometry interpolations, and image are carried out between correspondence texture image
Deformation operation obtains real facial expression animation.In addition, also based on Bezier indicatrixes and based on radial direction base
The anamorphose scheduling algorithm of neutral net.Dynamic texture mapping relies on the texture mapping of viewpoint, and it is allowed using different every time
Texture maps, are dynamically adjusted by drawing repeatedly for model come the mediation weights to current view point, and deficiency is the meter for needing
Calculate and amount of ram is big, and multi-texturing is merged if independently of viewpoint, because there is error and usually make texture in record and sampling
Thicken.
At present, most widely used is that, based on motion tracking method, motion tracking method mainly uses motion capture system
To carry out expression seizure, it mainly uses existing capture data to move on object module the human face expression of source model,
So as to the facial expression for realizing face is captured, facial expression true to nature is obtained.
Existing movement capturing technology is broadly divided into four major classes according to seizure equipment is different with principle：Mechanically, acoustics formula,
Electricity formula and optical motion catch, wherein, data are convenient, sampling precision is high, frequency with obtaining for optical motion capture mode
Rate is high, using scope it is wide the advantages of, the data that it is gathered are sequence identifier point (Marker) the point set data in units of frame,
Face's key point of performing artist sticks Marker, and vision system will recognize and process these Maker, it is possible to realize that expression is caught
Catch.
The present invention uses the method based on motion tracking, different from the method for catching of traditional optical profile type.The present invention
Without hand labeled Marker in capture-process of expressing one's feelings, directly recognize and record facial key point using the method for image recognition
Exercise data.It mainly uses existing capture data to move on object module the human face expression of source model, so that
The facial expression capture of face is realized, facial expression true to nature is obtained.
The content of the invention
For the demand that the industries such as current animation, film are captured to facial expression, the application provides a kind of based on TOF depth
The facial expression method for catching of camera, without hand labeled Marker, is directly recognized and recording surface using the method for image recognition
The exercise data of portion's key point.The data of its capture are the real motion data of face, and action is coherent true to nature, and expression is fine and smooth rich
Richness, and the facial expression that gets of the method catches file and can reuse, and has saved cost.
In order to solve the above technical problems, the invention provides a kind of facial expression method for catching based on TOF depth cameras,
It includes：
The first step, using TOF depth camera sampling depth images and coloured image, using AAM (Active
Appearances Models) algorithm positioning face feature point, and calculate acquisition facial expression data；
Second step, treatment catches facial expression data, and generation facial expression catches file；
3rd step, builds three-dimensional personage's geometrical model, imports facial expression and catches file, driving model.
Wherein, the face feature point of real person is demarcated using AAM algorithms in the first step, in the feelings that head pose is different
Under condition, we also can accurately be positioned to face feature point, so as to carry out facial expression seizure to real person.
Wherein, the first step is further specifically included：
A, using TOF depth cameras obtain face coloured image and depth image；
B, set up for head pose judge random regression forest model；
C, the visual angle model trained and set up AAM algorithms；
D, using AAM algorithms position face feature point；
E, the three-dimensional coordinate for obtaining face feature point.
Wherein, a steps are specially connection TOF depth cameras, switch on power, the data control process list of TOF depth cameras
Unit sends open command, and its body-sensing camera is opened, and it is single with treatment that color data stream and depth data are streamed into data control
Unit, data control changes color data circulation into coloured image with processing unit, changes depth data circulation into depth image.
Wherein, b steps are specially and obtain the corresponding depth image of different head poses using TOF depth cameras, so
The depth image that will be got according to the position of head pose afterwards is divided into multiple different classification based training collection, is approximately put down using face
The normal direction in face represents different head poses, so as to set up the random regression forest model of head pose judgement.
The present invention be extended on the basis of original AAM algorithms, for when portion's characteristic point is positioned over there not
Same head pose, sets up corresponding AAM visual angles model respectively, and each AAM visual angles model is divided into shape and texture again
Model two parts.
Wherein, the d steps are further specially and are input into new image, what the head pose set up using step b was judged
Random regression forest model, the new depth image to being given judges head position therein and face's direction, according to what is obtained
Immediate AAM visual angles model therewith is selected in the AAM visual angles model that head pose information is set up from c steps.Using the AAM for choosing
Model carries out the positioning of face feature point to current input picture.
Wherein, after the e steps are further specially the position for determining face feature point, calculate and obtain face face special
Levy coordinate information a little, the coordinate system of TOF depth cameras is that, with its own as the origin of coordinates, front is Z-direction, to the left for
The positive direction of X-axis, is upwards the positive direction of Y-axis.
Beneficial effects of the present invention：
The application provides a kind of facial expression method for catching based on TOF depth cameras, and the data of the method capture are existing
The real exercise data of real world's face, action is coherent true, and expression is fine and smooth abundant, and the facial table that the method gets
Feelings catch file and can reuse, and have saved cost.
Brief description of the drawings
Fig. 1 is the coloured image that TOF depth cameras get；
Fig. 2 is the depth image that TOF depth cameras get；
Fig. 3 is the training sample of hand labeled；
Fig. 4 is the characteristic point distribution of face and title；
Fig. 5 is that expression catches the design sketch that file imports faceform；
Fig. 6 is to set up the depth image block used by head pose judgment models.
Specific embodiment
The invention provides a kind of facial expression method for catching based on TOF depth cameras, it includes：
The first step, using TOF depth camera sampling depth images and coloured image, using AAM (Active
Appearances Models) algorithm positioning face feature point, and calculate acquisition facial expression data；
Second step, treatment catches facial expression data, and generation facial expression catches file；
3rd step, builds three-dimensional personage's geometrical model, imports facial expression and catches file, driving model.
The data-driven three-dimensional personage's geometrical model captured using TOF depth cameras makes consistent with performing artist
Expression, it is thus necessary to determine that the positional information of face feature point, this is also to realize the key that face facial expression catches.How to determine and
The validity for extracting face facial expression of the characteristic point information to capturing serves conclusive effect.Face feature information is very
Easily influenceed by illumination and attitude, TOF depth cameras can simultaneously provide depth image, RGB color image, thus greatly
Illumination and the influence of attitude are reduced greatly, face features point can be more accurately extracted.The present invention is directed to different heads
Portion's attitude sets up corresponding AAM models respectively, demarcates the face spy of real person using 3D AAM algorithms in the first step
Levy a little, we also can accurately be positioned to face feature point in the case of making head pose different, so as to true people
Thing carries out facial expression seizure.
The first step is further specifically included：
A, using TOF depth cameras obtain face coloured image and depth image；
B, set up head pose judgment models；
C, train and set up AAM algorithm models；
D, using AAM algorithms position face feature point；
E, the three-dimensional coordinate for obtaining face feature point.
A steps are specially connection TOF depth cameras, switch on power, and the data control process unit of TOF depth cameras sends
Open command, its body-sensing camera is opened, and color data stream and depth data are streamed into data control and processing unit, data
Control changes color data circulation into coloured image with processing unit, changes depth data circulation into depth image..
B steps are specially and obtain the corresponding depth image of different head poses using TOF depth cameras, then according to
The depth image that the position of head pose will get is divided into multiple different classification based training collection, uses the method for face's myopia plane
Vector represents different head poses, so as to set up the random regression forest model of head pose judgement.
What the method that the b steps set up head pose judgment models was set up is that a kind of random regression based on statistical method is gloomy
Woods model, it is further specially firstly the need of the head zone and non-head region for identifying depth image, then at this
The image-region of fixed size is chosen respectively as training set in two regions.Whether depth image block includes picture number, belongs to
Head zone, head center position and head pose information.Each tree uses one group in training set respectively in random regression forest
The depth image block for randomly selecting is set up.The foundation of each node classification is two rectangular blocks of selection from image block, meter on tree
The average of the depth value of rectangular block is calculated, the threshold value that this average sets with certain is compared into entrance right subtrees of the more than this threshold value, it is no
Then enter left subtree, the decision tree for carrying out selecting to ultimately generate symmetery to threshold value and rectangular block using Shannon entropy, such as formula
Shown in 1.Another group of depth image block being taken at random and setting up another decision tree, all decision trees constitute random regression forest.
In formula, H (P) is Shannon entropy,For left and right subtree Shannon entropy and.
Wherein, the used AAM algorithms of the c steps and traditional AAM algorithms are otherwise varied.Traditional AAM algorithms enter
The object of row training and identification is all almost close in the case of the visual angle of front, the present invention is on the basis of traditional AAM algorithms
On be improved, for three-dimensional head attitude different when portion's characteristic point is positioned over there, set up respectively corresponding
AAM visual angles model, each AAM visual angles model is divided into shape and texture model two parts again.
The c steps are further specially according to the difference of face's direction, and the direction of wherein face is approximately put down using face
The normal direction in face represented, training set is divided into the sub- training set under multiple different visual angles.One individually is set up to every sub- training set
AAM models, such a single AAM models are called an AAM visual angles model.For each AAM visual angles model, point shape and
Texture two parts are modeled.
It is that training set under visual angle carries out manual characteristic point demarcation, characteristic point to set up shape and be specially first to each
Location sets be referred to as shape vector Si, obtain training set L={ (Ii, Si) | i=1,2 ..., m；Wherein, IiRepresent training sample, SiRepresent the coordinate set of manual markings.i
It is the number of point；X, y are the coordinate of point, (to image removal translation, rotation, scaling) are then normalized to shape and to returning
One shape changed is alignd, and average shape is drawn by PCA transformation calculationsTraining image is deformed to average shape, because
This any face shape S can use linear equationExpression, PsIt is shape principal component that PCA transformation calculations are obtained
The transformation matrix that characteristic component is formed, BsIt is the Statistical Shape parameter for controlling shape conversion, obtains shape Statistics model.
Set up being specially for texture model and obtain corresponding AAM shape eigenvectors SiAnd average shapeSelection byThrough
Shaped grid that Delaunay Triangulation obtains is crossed as the benchmark grid for carrying out texture mapping, by the image root in training set
Delaunay Triangulation is carried out according to correspondence nominal shape characteristic point position, the pixel value in shaped grid envelope after subdivision
Exactly need sampling texture information, secondly, by the affine method of piecewise linearity map that to setting benchmark grid in,
Realize the normalization sampling to texture；Again, PCA dimensionality reductions are carried out to the texture information after normalization and obtains average textureThis
The arbitrary face texture of sample can be usedRepresent, the texture statistics model for finally drawing is：
It is average texture, PgIt is the transformation matrix that the texture main component characteristic component that PCA transformation calculations are obtained is formed,
BgIt is the statistic texture parameter for controlling texture transformation.
Model Fusion
Active apparent model is：Wherein,It is average apparent vector, Q is
The matrix that apparent principal component characteristic vector is formed, c is the apparent parameter of statistics for controlling apparent change.
The d steps are further specially and are input into new image, and the head pose set up using step b judges random regression
Forest model, the new depth image to being given judges head position therein and face's direction, according to the head pose for obtaining
Immediate AAM visual angles model therewith is selected in the AAM visual angles model that information is set up from c steps.Using the AAM models chosen to working as
Preceding input picture carries out the positioning of face feature point.Wherein, the process for carrying out facial characteristics point location is exactly that AAM is matched
The process of iterative calculation, comprises the following steps that：
1. initialization model parameter c；
2. the poor Δ g=g of computation model texture and current textures-gm；
3. using the changes delta c=R of equation of linear regression Prediction Parameters ccΔg；
4. new model parameter c '=c-k Δs c, k=1 is attempted；
5. error function Δ g ' is recalculated；
If 6., | | Δ g ' | | ＜ | | Δs g | |, receives c ' as new parameter；4 are otherwise jumped to, k=1.5 is tasted, 2,
2.5 etc.
7. a two field picture is removed, turns 2.
After the e steps are further specially the position for determining face feature point, face features point is calculated and obtained
Coordinate information, the coordinate system of TOF depth cameras is that, with its own as the origin of coordinates, front is Z-direction, is to the left X-axis
Positive direction, upwards for Y-axis positive direction.The horizontal view angle angle value α and vertical angle of view angle value β of TOF depth cameras are obtained,
By conversion relation RealWorldXtoZ=2tan (α/2), RealWorldYtoZ=2tan (β/2) has obtained depth map
The width value w/ height values h of picture and the ratio apart from d.
NormalizedX=x/512-0.5
NormalizedY=0.5-y/424
X=NormalizedX*Z*RealWorldXtoZ
Y=NormalizedY*Z*RealWorldYtoZ
To sum up, index can be tried to achieve for x, the three-dimensional coordinate of the characteristic point at y is (X, Y, Z).
The second step further drives dummy model to realize face face table specifically by motion capture file
The seizure of feelings.The three-dimensional coordinate information that the first step is obtained, rotation information is calculated by transfer algorithm, and by its according to
The form write activity of BVH is caught in file.
The transfer algorithm is specially：
A, structural information and initial state information to input feature vector point in BVH files；
B, the exercise data for reading face feature point, data form are the three-dimensional coordinate of each point；
C, three-dimensional coordinate is converted into quaternary number；
D, calculating spin matrix；
E, node coordinate is transformed under local coordinate system；
F, Eulerian angles are tried to achieve by formula 3；
θ x=arcsin2 (yz-wx)
Y, z, w are the coordinates of spatial point；
G, Eulerian angles are added in BVH files.
Specific implementation process of the present invention is as follows：
Step 1. obtains face coloured image and depth image
Color data stream and depth data stream are obtained using TOF depth cameras, and changes color data circulation into cromogram
Picture, changes depth data circulation into depth image and changes color data circulation into coloured image, changes depth data circulation into depth
Degree image, as depicted in figs. 1 and 2.
Step 2. sets up head pose judgment models
The data-driven threedimensional model captured using TOF depth cameras makes the expression consistent with performing artist, it is necessary to really
Determine the positional information of face feature point, this is also to realize the key that face facial expression catches.The present invention is directed to different heads
Attitude sets up corresponding AAM models respectively, and we can also enter to face feature point in the case of making head pose different
The accurate positioning of row.
The present invention is a kind of random regression forest mould based on statistical method using head pose judgment models method is set up
Type.Then the method is selected respectively firstly the need of the head zone and non-head region that identify depth image in the two regions
The image-region of fixed size is taken as training set.Whether depth image block includes picture number, belong to head zone, head
Heart position and head pose information.Each tree uses one group of depth map for randomly selecting in training set respectively in random regression forest
As block is set up.The foundation of each node classification is two rectangular blocks of selection from image block on tree, calculates the depth value of rectangular block
Average, by the threshold value that this average sets with certain compare more than this threshold value entrance right subtree, otherwise into left subtree, profit
The decision tree for carrying out selecting to ultimately generate symmetery to threshold value and rectangular block with Shannon entropy, takes another group of depth image at random
Block sets up another decision tree, and all decision trees constitute random regression forest.
In formula, H (P) is Shannon entropy,For left and right subtree Shannon entropy and.
Step 3. is trained and sets up the model of AAM algorithms
Model is divided into two parts of shape and texture model by AAM algorithms, and AAM algorithms are on the basis of ASM algorithms
Upper (will facial image be deformed to average shape) carries out texture analysis to image to position its characteristic point.
Using AAM algorithm locating human face face feature points firstly the need of setting up shape and texture model.In view of head appearance
Influence of the state to the facial positioning feature point degree of accuracy, we set up an AAM model for each head pose, choose every time
The AAM models being best suitable for carry out the matching of face feature point.
Set up shape
Manual characteristic point demarcation, the location sets quilt of characteristic point are carried out to the training sample under each head pose first
Referred to as shape vector Si, obtain training set
L={ (Ii, Si) | i=1,2 ..., m；Then shape is carried out
Normalization (to image removal translation, rotation, scaling) is simultaneously alignd to normalized shape, is drawn by PCA transformation calculations
Average shapeTraining image is deformed to average shape, therefore any face shape S can use linear equation,Expression, PsIt is the transformation matrix of the shape principal component characteristic component formation that PCA transformation calculations are obtained, BsIt is control
The Statistical Shape parameter of shape conversion, obtains shape Statistics model.
Set up texture model
The process for setting up texture model is consistent with the process for setting up shape, and the texture statistics model for finally drawing is：
It is average texture, PgIt is the transformation matrix that the texture main component characteristic component that PCA transformation calculations are obtained is formed,
bgIt is the statistic texture parameter for controlling texture transformation.
Model Fusion
Active apparent model is：Wherein,It is average apparent vector, Q is
The matrix that apparent principal component characteristic vector is formed, c is the apparent parameter of statistics for controlling apparent change.
Step 4. is using AAM algorithms positioning face feature point
The new image of input, the random regression forest model set up using step 2, the new depth image to being given is judged
Head position therein, and one group of estimate of head pose is given, calculate the head center of the depth image for providing
Position and head pose information.Then the 3DAAM models for most matching therewith are selected.Head pose estimation information acquisition head center
Position and head pose, thus calculate the rotation and translation of head model, using end rotation angle calculation spin matrix R,
Using head center position as translation matrix T.The 3D face feature points being calculated in upper section are revolved using R chis and T
Turn and translate, camera internal reference is reused afterwards by the spot projection after conversion to RGB image plane, obtain the feature on RGB image
Point set, in this, as the shape initial value of AAM model instances.
1. initialization model parameter c；
2. the poor Δ g=g of computation model texture and current textures-gm；
3. using the changes delta c=R of equation of linear regression Prediction Parameters ccΔg；
4. new model parameter c '=c-k Δs c, k=1 is attempted；
5. error function Δ g ' is recalculated；
6. c ' is received as new parameter if | | Δ g ' | | ＜ | | Δs g | |；4 are otherwise jumped to, k=1.5 is tasted, 2,
2.5 etc.
7. a two field picture is removed, turns 2.
Step 5. obtains the three-dimensional coordinate of face feature point
After determining the position of face feature point, followed by calculating and obtain the coordinate information of face features point,
The coordinate system of TOF depth cameras is that, with its own as the origin of coordinates, front is Z-direction, is to the left the positive direction of X-axis, to
Upper is the positive direction of Y-axis.TOF depth cameras can get depth image, and in 3D computer graphics, depth image refers to from sight
Examine visual angle to look, image includes a kind of information image related to object scene surface distance or an image channel.So,
Assuming that the visual field direction Z side of the change direction (i.e. video camera shooting direction) of image depth values and the three-dimensional scenic of required description
If identical, then just can easily describe whole three-dimensional scenic.Therefore, depth image is also referred to as range image.
The depth image pixel value that TOF depth cameras get is and TOF depth cameras camera lens is to water actual between object
What flat distance was associated, can be in the hope of the Z values of real space by this incidence relation, the X and Y of real space can be by spies
Levy index value x (row) and y (OK) a little in depth image to try to achieve, computational methods are as follows：
The horizontal view angle angle value α and vertical angle of view angle value β of TOF depth cameras are obtained, by conversion relation
RealWorldXtoZ=2tan (α/2), RealWorldYtoZ=2tan (β/2), that is, obtained the width value w/ of depth image
Height value h and the ratio apart from d.
NormalizedX=x/512-0.5
NormalizedY=0.5-y/424
X=NormalizedX*Z*RealWorldXtoZ
Y=NormalizedY*Z*RealWorldYtoZ
To sum up, index can be tried to achieve for x, the three-dimensional coordinate of the characteristic point at y is (X, Y, Z).
Step 6. generation facial expression catches file
The present invention drives dummy model by motion capture file to realize the seizure of face facial expression.Treatment TOF
Each two field picture of depth camera acquisition simultaneously therefrom determines to extract the three-dimensional coordinate information of characteristic point, is calculated by transfer algorithm
Go out rotation information, and by its form write activity seizure file according to bvh.
The step of realizing of transfer algorithm is：
To the structural information and initial state information of input feature vector point in BVH files；
Exercise data is read to be stored in structure；
Three-dimensional coordinate is converted into quaternary number；
Calculate spin matrix；
Node coordinate is transformed under local coordinate system；
Eulerian angles are input in BVH files.
All above-mentioned this intellectual properties of primarily implementation, the not this new product of implementation of setting limitation other forms
And/or new method.Those skilled in the art will be using this important information, the above modification, to realize similar execution feelings
Condition.But, all modifications or transformation are based on the right that new product of the present invention belongs to reservation.
The above, is only presently preferred embodiments of the present invention, is not the limitation for making other forms to the present invention, is appointed
What those skilled in the art changed possibly also with the technology contents of the disclosure above or be modified as equivalent variations etc.
Effect embodiment.But it is every without departing from technical solution of the present invention content, according to technical spirit of the invention to above example institute
Any simple modification, equivalent variations and the remodeling made, still fall within the protection domain of technical solution of the present invention.
Claims (8)
1. a kind of facial expression method for catching based on TOF depth cameras, it is characterised in that including：
The first step, facial expression seizure is carried out using TOF depth cameras to real person；
Second step, processes the facial expression data for catching, and generate facial expression seizure file；
3rd step, builds three-dimensional personage's geometrical model, imports facial expression and catches file, driving model.
2. the facial expression method for catching of TOF depth cameras is based on as claimed in claim 1, it is characterised in that：In the first step
Middle use AAM algorithms demarcate the face feature point of real person, and we also can be to face in the case of making head pose different
Characteristic point is accurately positioned, so as to carry out facial expression seizure to real person.
3. the facial expression method for catching of TOF depth cameras is based on as claimed in claim 1 or 2, it is characterised in that：Described
One step is further specifically included,
A, using TOF depth cameras obtain face coloured image and depth image；
B, set up head pose judgment models；
C, the model trained and set up AAM algorithms；
D, using AAM algorithms position face feature point；
E, the three-dimensional coordinate for obtaining face feature point.
4. the facial expression method for catching based on TOF depth cameras as described in claims 1 to 3, it is characterised in that：A is walked
The color data stream and depth data stream of three-dimensional face are specially obtained using TOF depth cameras, and color data circulation is changed
Into coloured image, depth data circulation is changed into depth image.
5. the facial expression method for catching based on TOF depth cameras as described in Claims 1-4, it is characterised in that：B is walked
The corresponding depth image of different head poses specially is obtained using TOF depth cameras, then according to the position of head pose
Put the depth image that will be got and be divided into multiple different classification based training collection, represented not using the normal vector of face's myopia plane
Same head pose, so as to set up head pose judgment models.
6. the facial expression method for catching based on TOF depth cameras as described in claim 1 to 5, it is characterised in that：Described
C walks used AAM algorithms and model is divided into two parts of shape and texture model, and AAM algorithms are the bases in ASM algorithms
(will facial image be deformed to average shape) carries out texture analysis to image to position its characteristic point on plinth.
7. the facial expression method for catching based on TOF depth cameras as described in claim 1 to 6, it is characterised in that：Described
D steps are further specially and are input into new image, the random regression forest model set up using step b, to the new depth for being given
Image judges head position therein, and provides one group of estimate of head pose, calculates the head of the depth image for providing
The position at portion center and head pose information, then select the AAM models for most matching therewith, head pose estimation information acquisition head
Portion center and head pose, thus calculate the rotation and translation of head model, are rotated using end rotation angle calculation
Matrix R, using head center position as translation matrix, rotation peace is carried out to the 3D face feature points for obtaining using R and T
Move, recycle camera internal reference by the spot projection after conversion to RGB image plane afterwards, obtain the feature point set on RGB image, with
This as AAM model instances shape initial value.
8. the facial expression method for catching based on TOF depth cameras as described in claim 1 to 7, it is characterised in that：Described
After e steps are further specially the position for determining face feature point, the coordinate information of face features point, TOF are calculated and obtained
The coordinate system of depth camera is that, with its own as the origin of coordinates, front is Z-direction, is to the left the positive direction of X-axis, is upwards
The positive direction of Y-axis, TOF depth cameras can get depth image, finally obtain whole three-dimensional face.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611190469.1A CN106778628A (en) | 2016-12-21 | 2016-12-21 | A kind of facial expression method for catching based on TOF depth cameras |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611190469.1A CN106778628A (en) | 2016-12-21 | 2016-12-21 | A kind of facial expression method for catching based on TOF depth cameras |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106778628A true CN106778628A (en) | 2017-05-31 |
Family
ID=58893609
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611190469.1A Pending CN106778628A (en) | 2016-12-21 | 2016-12-21 | A kind of facial expression method for catching based on TOF depth cameras |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106778628A (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368778A (en) * | 2017-06-02 | 2017-11-21 | 深圳奥比中光科技有限公司 | Method for catching, device and the storage device of human face expression |
| CN107403170A (en) * | 2017-08-08 | 2017-11-28 | 灵然创智（天津）动画科技发展有限公司 | A kind of seizure point positioning system based on facial seizure system |
| CN107657652A (en) * | 2017-09-11 | 2018-02-02 | 广东欧珀移动通信有限公司 | Image processing method and device |
| CN107680034A (en) * | 2017-09-11 | 2018-02-09 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| CN107705276A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| CN107705356A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device |
| CN107886558A (en) * | 2017-11-13 | 2018-04-06 | 电子科技大学 | A kind of human face expression cartoon driving method based on RealSense |
| CN108573526A (en) * | 2018-03-30 | 2018-09-25 | 盎锐（上海）信息科技有限公司 | Face snap device and image generating method |
| CN108648280A (en) * | 2018-04-25 | 2018-10-12 | 深圳市商汤科技有限公司 | virtual role driving method and device, electronic equipment and storage medium |
| CN109299639A (en) * | 2017-07-25 | 2019-02-01 | 虹软(杭州)多媒体信息技术有限公司 | A kind of method and apparatus for Expression Recognition |
| CN109325393A (en) * | 2017-08-01 | 2019-02-12 | 苹果公司 | Using the face detection of single network, Attitude estimation and away from the estimation of camera distance |
| WO2019154231A1 (en) * | 2018-02-12 | 2019-08-15 | 腾讯科技（深圳）有限公司 | Image processing method, electronic device and storage medium |
| CN110378855A (en) * | 2019-07-18 | 2019-10-25 | 博瓦鲸视(深圳)科技有限公司 | A method of personage's statistical model that building is acted comprising personage's figure, facial expression, gesture, posture |
| CN110427917A (en) * | 2019-08-14 | 2019-11-08 | 北京百度网讯科技有限公司 | Method and apparatus for detecting key point |
| CN110738717A (en) * | 2019-10-16 | 2020-01-31 | 网易（杭州）网络有限公司 | Method and device for correcting motion data and electronic equipment |
| CN111290568A (en) * | 2018-12-06 | 2020-06-16 | 阿里巴巴集团控股有限公司 | Interaction method and device and computer equipment |
| CN111416934A (en) * | 2020-03-16 | 2020-07-14 | 南昌航空大学 | Facial expression acquisition method and system based on functional music intervention |
| CN111741285A (en) * | 2020-06-08 | 2020-10-02 | 上海龙旗科技股份有限公司 | Real-time 3D scene implementation method and device |
| CN112241658A (en) * | 2019-07-17 | 2021-01-19 | 青岛大学 | Fatigue driving early warning system and method based on depth camera |
| CN113221600A (en) * | 2020-01-21 | 2021-08-06 | 初速度（苏州）科技有限公司 | Method and device for calibrating image feature points |
| CN114463817A (en) * | 2022-02-10 | 2022-05-10 | 北京中科深智科技有限公司 | Lightweight 2D video-based facial expression driving method and system |
| CN116453384A (en) * | 2023-06-19 | 2023-07-18 | 江西德瑞光电技术有限责任公司 | Immersion type intelligent learning system based on TOF technology and control method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102479388A (en) * | 2010-11-22 | 2012-05-30 | 北京盛开互动科技有限公司 | Expression interaction method based on face tracking and analysis |
| US20130100140A1 (en) * | 2011-10-25 | 2013-04-25 | Cywee Group Limited | Human body and facial animation systems with 3d camera and method thereof |
| CN103093490A (en) * | 2013-02-02 | 2013-05-08 | 浙江大学 | Real-time facial animation method based on single video camera |
| CN103324905A (en) * | 2012-03-21 | 2013-09-25 | 天津生态城动漫园投资开发有限公司 | Next-generation virtual photostudio facial capture system |
| CN103608841A (en) * | 2011-05-17 | 2014-02-26 | 高通股份有限公司 | Head pose estimation using RGBD camera |
| CN103942822A (en) * | 2014-04-11 | 2014-07-23 | 浙江大学 | Facial feature point tracking and facial animation method based on single video vidicon |
| CN104345801A (en) * | 2013-08-09 | 2015-02-11 | 三星电子株式会社 | Hybrid visual communication |
| CN105654537A (en) * | 2015-12-30 | 2016-06-08 | 中国科学院自动化研究所 | Expression cloning method and device capable of realizing real-time interaction with virtual character |
| CN105900144A (en) * | 2013-06-07 | 2016-08-24 | 费斯史福特股份公司 | Online modeling for real-time facial animation |
| CN106228119A (en) * | 2016-07-13 | 2016-12-14 | 天远三维（天津）科技有限公司 | A kind of expression catches and Automatic Generation of Computer Animation system and method |
-
2016
- 2016-12-21 CN CN201611190469.1A patent/CN106778628A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102479388A (en) * | 2010-11-22 | 2012-05-30 | 北京盛开互动科技有限公司 | Expression interaction method based on face tracking and analysis |
| CN103608841A (en) * | 2011-05-17 | 2014-02-26 | 高通股份有限公司 | Head pose estimation using RGBD camera |
| US20130100140A1 (en) * | 2011-10-25 | 2013-04-25 | Cywee Group Limited | Human body and facial animation systems with 3d camera and method thereof |
| CN103324905A (en) * | 2012-03-21 | 2013-09-25 | 天津生态城动漫园投资开发有限公司 | Next-generation virtual photostudio facial capture system |
| CN103093490A (en) * | 2013-02-02 | 2013-05-08 | 浙江大学 | Real-time facial animation method based on single video camera |
| CN105900144A (en) * | 2013-06-07 | 2016-08-24 | 费斯史福特股份公司 | Online modeling for real-time facial animation |
| CN104345801A (en) * | 2013-08-09 | 2015-02-11 | 三星电子株式会社 | Hybrid visual communication |
| CN103942822A (en) * | 2014-04-11 | 2014-07-23 | 浙江大学 | Facial feature point tracking and facial animation method based on single video vidicon |
| CN105654537A (en) * | 2015-12-30 | 2016-06-08 | 中国科学院自动化研究所 | Expression cloning method and device capable of realizing real-time interaction with virtual character |
| CN106228119A (en) * | 2016-07-13 | 2016-12-14 | 天远三维（天津）科技有限公司 | A kind of expression catches and Automatic Generation of Computer Animation system and method |
Non-Patent Citations (1)
| Title |
|---|
| CHENLEE_1: "AAM（Active Appearance Model）算法介绍", 《CSDN》 * |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368778A (en) * | 2017-06-02 | 2017-11-21 | 深圳奥比中光科技有限公司 | Method for catching, device and the storage device of human face expression |
| CN109299639B (en) * | 2017-07-25 | 2021-03-16 | 虹软科技股份有限公司 | Method and device for facial expression recognition |
| CN109299639A (en) * | 2017-07-25 | 2019-02-01 | 虹软(杭州)多媒体信息技术有限公司 | A kind of method and apparatus for Expression Recognition |
| CN109325393A (en) * | 2017-08-01 | 2019-02-12 | 苹果公司 | Using the face detection of single network, Attitude estimation and away from the estimation of camera distance |
| CN109325393B (en) * | 2017-08-01 | 2022-12-09 | 苹果公司 | Face detection, pose estimation, and estimation of distance to camera using a single network |
| CN107403170A (en) * | 2017-08-08 | 2017-11-28 | 灵然创智（天津）动画科技发展有限公司 | A kind of seizure point positioning system based on facial seizure system |
| CN107705276B (en) * | 2017-09-11 | 2021-03-02 | Oppo广东移动通信有限公司 | Image processing method and apparatus, electronic apparatus, and computer-readable storage medium |
| CN107657652A (en) * | 2017-09-11 | 2018-02-02 | 广东欧珀移动通信有限公司 | Image processing method and device |
| CN107680034A (en) * | 2017-09-11 | 2018-02-09 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| CN107705276A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| CN107705356A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device |
| CN107886558A (en) * | 2017-11-13 | 2018-04-06 | 电子科技大学 | A kind of human face expression cartoon driving method based on RealSense |
| US11436779B2 (en) | 2018-02-12 | 2022-09-06 | Tencent Technology (Shenzhen) Company Ltd | Image processing method, electronic device, and storage medium |
| WO2019154231A1 (en) * | 2018-02-12 | 2019-08-15 | 腾讯科技（深圳）有限公司 | Image processing method, electronic device and storage medium |
| CN108573526A (en) * | 2018-03-30 | 2018-09-25 | 盎锐（上海）信息科技有限公司 | Face snap device and image generating method |
| CN108648280B (en) * | 2018-04-25 | 2023-03-31 | 深圳市商汤科技有限公司 | Virtual character driving method and device, electronic device and storage medium |
| CN108648280A (en) * | 2018-04-25 | 2018-10-12 | 深圳市商汤科技有限公司 | virtual role driving method and device, electronic equipment and storage medium |
| CN111290568A (en) * | 2018-12-06 | 2020-06-16 | 阿里巴巴集团控股有限公司 | Interaction method and device and computer equipment |
| CN112241658A (en) * | 2019-07-17 | 2021-01-19 | 青岛大学 | Fatigue driving early warning system and method based on depth camera |
| CN112241658B (en) * | 2019-07-17 | 2023-09-01 | 青岛大学 | Fatigue driving early warning method based on depth camera |
| CN110378855B (en) * | 2019-07-18 | 2023-04-25 | 博瓦鲸视(深圳)科技有限公司 | Method for constructing character statistical model containing character shape, facial expression, gesture and gesture actions |
| CN110378855A (en) * | 2019-07-18 | 2019-10-25 | 博瓦鲸视(深圳)科技有限公司 | A method of personage's statistical model that building is acted comprising personage's figure, facial expression, gesture, posture |
| CN110427917A (en) * | 2019-08-14 | 2019-11-08 | 北京百度网讯科技有限公司 | Method and apparatus for detecting key point |
| CN110427917B (en) * | 2019-08-14 | 2022-03-22 | 北京百度网讯科技有限公司 | Method and device for detecting key points |
| CN110738717A (en) * | 2019-10-16 | 2020-01-31 | 网易（杭州）网络有限公司 | Method and device for correcting motion data and electronic equipment |
| CN110738717B (en) * | 2019-10-16 | 2021-05-11 | 网易（杭州）网络有限公司 | Method and device for correcting motion data and electronic equipment |
| CN113221600B (en) * | 2020-01-21 | 2022-06-21 | 魔门塔(苏州)科技有限公司 | Method and device for calibrating image feature points |
| CN113221600A (en) * | 2020-01-21 | 2021-08-06 | 初速度（苏州）科技有限公司 | Method and device for calibrating image feature points |
| CN111416934A (en) * | 2020-03-16 | 2020-07-14 | 南昌航空大学 | Facial expression acquisition method and system based on functional music intervention |
| CN111741285A (en) * | 2020-06-08 | 2020-10-02 | 上海龙旗科技股份有限公司 | Real-time 3D scene implementation method and device |
| CN114463817A (en) * | 2022-02-10 | 2022-05-10 | 北京中科深智科技有限公司 | Lightweight 2D video-based facial expression driving method and system |
| CN116453384A (en) * | 2023-06-19 | 2023-07-18 | 江西德瑞光电技术有限责任公司 | Immersion type intelligent learning system based on TOF technology and control method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106778628A (en) | A kind of facial expression method for catching based on TOF depth cameras | |
| CN109325437B (en) | Image processing method, device and system | |
| CN109472198B (en) | Gesture robust video smiling face recognition method | |
| Liao et al. | Automatic caricature generation by analyzing facial features | |
| WO2021093453A1 (en) | Method for generating 3d expression base, voice interactive method, apparatus and medium | |
| Min et al. | Kinectfacedb: A kinect database for face recognition | |
| Li et al. | Realtime facial animation with on-the-fly correctives. | |
| US8624901B2 (en) | Apparatus and method for generating facial animation | |
| CN103443826B (en) | mesh animation | |
| CN103473801A (en) | Facial expression editing method based on single camera and motion capturing data | |
| CN102157007A (en) | Performance-driven method and device for producing face animation | |
| CN113449570A (en) | Image processing method and device | |
| CN109034131A (en) | A kind of semi-automatic face key point mask method and storage medium | |
| Wang et al. | Digital twin: Acquiring high-fidelity 3D avatar from a single image | |
| Darujati et al. | Facial motion capture with 3D active appearance models | |
| CN110223368A (en) | A kind of unmarked motion capture method of face based on deep learning | |
| Luo et al. | Facial metamorphosis using geometrical methods for biometric applications | |
| Luo et al. | Alignment and tracking of facial features with component-based active appearance models and optical flow | |
| Pei et al. | Transferring of speech movements from video to 3D face space | |
| Somepalli et al. | Implementation of single camera markerless facial motion capture using blendshapes | |
| Agianpuye et al. | Synthesizing neutral facial expression on 3D faces using Active Shape Models | |
| Tian et al. | Robust facial marker tracking based on a synthetic analysis of optical flows and the YOLO network | |
| Abeysundera et al. | Nearest neighbor weighted average customization for modeling faces | |
| Li et al. | Example-based 3D face reconstruction from uncalibrated frontal and profile images | |
| Liu et al. | Face cartoon synthesis based on the active appearance model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170531 |