CN103065122A - Facial expression recognition method based on facial motion unit combination features - Google Patents

Abstract

The invention discloses a facial expression recognition method based on facial motion unit combination features. The facial expression recognition method based on the facial motion unit combination features is used for solving the technical problem of low recognition rate of a single facial motion unit with an existing facial expression recognition method based on facial motion units. The technical scheme includes that the facial expression recognition method based on the facial motion unit combination features comprises the steps of building a large-scale facial expression data base, carrying out clustering of each category of facial expressions and corresponding training samples by using the affinity propagation (AP) clustering algorithm, judging arbitrary unit (AU) combinations in each sub-category, determining the number of the sub-categories under the same facial expression by combining main AU combinations, generating the number of the categories of the training sample by combining the sub-categories of all facial expressions, and carrying out classification training by using the support vector machine (SVM) method. According to the facial expression recognition method based on the facial motion unit combination features, the recognition rate of the single facial motion unit is improved, namely that average recognition rate of a single AU unit is improved from 87.5% in prior are to 90.1% and by 2.6%.

Description

Facial expression recognizing method based on the Facial action unit assemblage characteristic

Technical field

The present invention relates to a kind of facial expression recognizing method, be specifically related to a kind of facial expression recognizing method based on the Facial action unit assemblage characteristic.

Background technology

Human face expression identification is the important research direction that the fields such as man-machine interaction, machine learning, Based Intelligent Control and image processing relate to, and has become the focus of domestic and international research at present.Based on the facial activity No system on the psychology, by setting up people's face visual signature and facial active unit, and the relation between facial active unit and the six kinds of basic facial expressions is carried out Expression Recognition based on the expression recognition method of facial active unit.The introducing of facial active unit has reduced the wide gap between complicated people's face visual signature and the six kinds of basic facial expressions.

Document 1 " Velusamy Sudha; Kannan Hariprasad; Anand Balasubramanian; Navathe Bilva; Sharma; Anshul Sharma, A Method to Infer Emotions From Facial Action Units, In IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague (2011) " a kind of facial expression recognizing method based on facial active unit disclosed; and the statistical relationship of the method utilization study and suitable matching process have been set up six kinds of facial basic facial expressions and 15 specific Facial action units (Action Unit; the mapping AU), and then carry out Expression Recognition.The prerequisite that the method realizes is accurately to identify 15 kinds of specific AU unit, but research experience shows: generally, expression is the result of some AU unit immixture, therefore, is difficult to accurately detect 15 kinds of single AU unit in facial expression image.For example, the method is used for the JAFFE database, only has 87.5% for the average recognition rate of single AU unit.

Summary of the invention

In order to overcome existing facial expression recognizing method based on facial active unit to the deficiency of single Facial action unit identification rate variance, the invention provides a kind of facial expression recognizing method based on the Facial action unit assemblage characteristic.The method is by setting up extensive Facial expression database, utilize the AP clustering algorithm, training sample corresponding to the facial basic facial expression of each class carried out cluster, judge the AU unit combination of each subclass, and in conjunction with main AU unit combination, determine the subclass number under the same expression; With the subclass classification number that the composing training sample is corresponding altogether of all kinds of expressions, utilize the SVM method to carry out the sorter training, can improve the discrimination to single Facial action unit.

The technical solution adopted for the present invention to solve the technical problems is: a kind of facial expression recognizing method based on the Facial action unit assemblage characteristic is characterized in may further comprise the steps:

(a) make up extensive expression database, to the every width of cloth expression picture in the expression database, carry out people's face and detect and normalized.Select many people that every width of cloth expression picture is carried out manual AU mark, probability is worked the AU combination that is used as this width of cloth picture greater than 50% AU unit combination.

(b) carry out the feature extraction of people's face local binary patterns, people's face is divided into m * m sub-block, each sub-block is calculated its LBP histogram, with the set of histograms of each sub-block altogether, consist of the binary pattern of every width of cloth expression picture at last.

(c) two width of cloth image x, the corresponding LBP feature of y Υ, the visual similarity between Ψ use following kernel function to calculate:

$K(x,y)=e^{-{\mathrm{\χ}}^2(\mathrm{\γ},\mathrm{\ψ})/\mathrm{\θ}}=\underset{j=1}{\overset{n}{\mathrm{\Π}}}\underset{i=1}{\overset{m}{\mathrm{\Π}}}{e}^{-{\mathrm{\χ}}_i^2({\mathrm{\γ}}_j\left(i\right),{\mathrm{\ψ}}_j\left(i\right))/{\mathrm{\θ}}_i---\left(1\right)}$

In the formula, m is the sub-block quantity that every width of cloth image is divided into, and n is the intrinsic dimensionality of each sub-block, θ=[θ ₁..., θ _m] each corresponding blocks χ of expression two width of cloth images ²The mean value of value, χ ²The computing formula of value is:

${\mathrm{\χ}}^2(\mathrm{\α},\mathrm{\ξ})=\underset{i}{\mathrm{\Σ}}\frac{{({\mathrm{\α}}_i-{\mathrm{\ξ}}_i)}^2}{{\mathrm{\α}}_i+{\mathrm{\ξ}}_i}---\left(2\right)$

In the formula, α _iAnd ξ _iTwo width of cloth image x, i component of LBP feature of y.

To the image in the every class expression training set, utilize formula (1) to calculate each other similarity after, utilize the AP clustering algorithm that the image clustering in every class expression training set is different subclasses.

To each subclass of every class expression, add up its corresponding AU combined situation, determine the AU combined situation that this subclass is final according to maximum probability criterion.Probability of erasure makes up and subclass less than 10% AU.

All subclasses of every class expression are combined into classification corresponding to whole expression database.

(d) carry out face normalization and feature extraction, adopt the svm classifier device that trains that every class expression subclass is classified.

The invention has the beneficial effects as follows: because by setting up extensive Facial expression database, utilize the AP clustering algorithm, training sample corresponding to the facial basic facial expression of each class carried out cluster, judge the AU unit combination of each subclass, and in conjunction with main AU unit combination, determine the subclass number under the same expression; With the subclass classification number that the composing training sample is corresponding altogether of all kinds of expressions, utilize the SVM method to carry out the sorter training, improved the discrimination to single Facial action unit.After testing, the inventive method is used for the JAFFE database, brings up to 90.1% for the average recognition rate of single AU unit by 87.5% of background technology, has improved 2.6%.

Below in conjunction with drawings and Examples the present invention is elaborated.

Description of drawings

Fig. 1 is the process flow diagram that the present invention is based on the facial expression recognizing method of Facial action unit assemblage characteristic.

Fig. 2 is the geometry site figure between people's face each several part.

Fig. 3 is human face expression local binary patterns characteristic pattern.

Embodiment

With reference to Fig. 1-3.The inventive method concrete steps are as follows:

(a) express one's feelings the on a large scale foundation of database.

Adopt document 2 " X.Wang; X Feng; and J Peng; A Novel Facial Expression DatabaseConstruction Method based on Web Images; in Proc.Of the Third Intl.Conf.on InternetMultimedia Computing and Service (ICIMCS ' 11) (2011) 124-127 " method, to every kind of expression, download 2000 width of cloth pictures from network, and pass through interactive filter, 500 width of cloth pictures that obtain every kind of expression consist of extensive Facial expression database, and this database is totally 500 * 6=3000 width of cloth expression picture.

To the every width of cloth expression picture in this database, eyes position, manual location is carried out people's face according to the geometric proportion (as shown in Figure 2) at each position of people's face and is detected and normalized.

Select many people that every width of cloth expression picture is carried out manual AU mark, and get majority's annotation results as the AU annotation results of this width of cloth picture.

With " happiness ", " happy ", " laughing at ", " happy ", " happiness ", " smile ", words such as " laugh " is as keyword, and inputted search engine such as ***.com select front 1000 pictures in the lump; Utilize the method for document 2, carry out the rubbish image filtering, obtain the effective picture of 500 width of cloth (also can manually reject the rubbish image).So far, obtain the training sample of 500 width of cloth " happiness " expression.Can obtain the training sample of other 5 kinds of basic facial expressions with similar approach, thereby consist of Sample Storehouse.

Every width of cloth expression picture in the database is carried out Computer display, utilize eyes position, the manual location of mouse, according to the geometry site between eyes, eyes and face in people's face shown in Figure 2, carry out people's face and detect and normalized.

Select 10 to carry out the postgraduates that image is processed, every width of cloth expression picture is carried out manual AU mark, calculate the probability that each AU unit is marked, probability is worked the AU combination that is used as this width of cloth picture greater than 50% AU unit combination.

(b) face characteristic extracts.

Employing document 3 " T.Ahonen,, A.Hadid, and M. Face Description with LocalBinary Patterns:Application to Face Recognition.IEEE Trans.Pattern Analysis andMachine Intelligence.28 (12): 2037-2041,2006 " method; carry out respectively local binary patterns (the Local Binary Patterns; LBP) feature extraction; then people's face is divided into 5 * 5 sub-block; each sub-block is calculated its LBP histogram; with the set of histograms of each sub-block altogether, consist of binary pattern (LBP feature) (see figure 3) of every width of cloth expression picture at last under two different scales of R=1 and R=3.

(c) AP(Affinity Propagation) cluster.

Two width of cloth image x, the corresponding LBP feature of y Υ, the visual similarity between Ψ use following kernel function to calculate:

$K(x,y)=e^{-{\mathrm{\χ}}^2(\mathrm{\γ},\mathrm{\ψ})/\mathrm{\θ}}=\underset{j=1}{\overset{n}{\mathrm{\Π}}}\underset{i=1}{\overset{m}{\mathrm{\Π}}}{e}^{-{\mathrm{\χ}}_i^2({\mathrm{\γ}}_j\left(i\right),{\mathrm{\ψ}}_j\left(i\right))/{\mathrm{\θ}}_i---\left(1\right)}$

Wherein m is the sub-block quantity that every width of cloth image is divided into, and n is the intrinsic dimensionality of each sub-block, θ=[θ ₁..., θ _m] each corresponding blocks χ of expression two width of cloth images ²The mean value of value, χ ²The computing formula of value is:

${\mathrm{\χ}}^2(\mathrm{\α},\mathrm{\ξ})=\underset{i}{\mathrm{\Σ}}\frac{{({\mathrm{\α}}_i-{\mathrm{\ξ}}_i)}^2}{{\mathrm{\α}}_i+{\mathrm{\ξ}}_i}---\left(2\right)$

α wherein _iAnd ξ _iI component of two width of cloth image LBP features.

To the image in the every class expression training set, utilize formula (1) to calculate each other similarity after, utilize the AP clustering algorithm that its cluster is different subclasses.

To each subclass of every class expression, add up its corresponding AU combined situation, according to maximum probability criterion determine this subclass final the AU combined situation.Probability of erasure makes up and subclass less than 10% AU.

All subclasses of every class expression are combined into classification corresponding to whole expression database.

To 500 width of cloth pictures of " happiness " expression, utilize formula (1) to calculate the wherein visual similarity of any two width of cloth images, then utilize the AP clustering algorithm that its cluster is different subclasses, present embodiment is 7 subclasses.

To 7 subclasses of " happiness " expression, add up its corresponding AU combined situation, the combination of maximum probability is defined as the AU combined result of this subclass.If the maximum probability of certain subclass AU combination is deleted this combination and this subclass less than 20%.So far, obtain 4 subclasses and the corresponding combination of " happiness " expression.

By the method, final six kinds of basic facial expressions are divided into 21 subclass, and the expression that every class is corresponding and AU combination see Table 1.

Table 1 basic facial expression and AU combination

So far, set up the rule of correspondence between AU combination and six kinds of basic facial expressions.

(d) svm classifier device training.

Adopt the svm classifier device, carry out the training of 21 expression subclass.

When test sample book is carried out Expression Recognition, adopt the method for above-mentioned steps (a) and step (b), carry out face normalization and feature extraction, then utilize the svm classifier device that trains that test sample book is classified.

After testing, the inventive method is used for the JAFFE database, and average recognition rate is 90.1%.

Claims (1)

1. facial expression recognizing method based on the Facial action unit assemblage characteristic is characterized in that may further comprise the steps:

(a) make up extensive expression database, to the every width of cloth expression picture in the expression database, carry out people's face and detect and normalized; Select many people that every width of cloth expression picture is carried out manual AU mark, probability is worked the AU combination that is used as this width of cloth picture greater than 50% AU unit combination;

(b) carry out the feature extraction of people's face local binary patterns, people's face is divided into m * m sub-block, each sub-block is calculated its LBP histogram, with the set of histograms of each sub-block altogether, consist of the binary pattern of every width of cloth expression picture at last;

(c) two width of cloth image x, the corresponding LBP feature of y Υ, the visual similarity between Ψ use following kernel function to calculate:

$K(x,y)=e^{-{\mathrm{\χ}}^2(\mathrm{\γ},\mathrm{\ψ})/\mathrm{\θ}}=\underset{j=1}{\overset{n}{\mathrm{\Π}}}\underset{i=1}{\overset{m}{\mathrm{\Π}}}{e}^{-{\mathrm{\χ}}_i^2({\mathrm{\γ}}_j\left(i\right),{\mathrm{\ψ}}_j\left(i\right))/{\mathrm{\θ}}_i---\left(1\right)}$

In the formula, m is the sub-block quantity that every width of cloth image is divided into, and n is the intrinsic dimensionality of each sub-block, θ=[θ ₁..., θ _m] each corresponding blocks χ of expression two width of cloth images ²The mean value of value, χ ²The computing formula of value is:

${\mathrm{\χ}}^2(\mathrm{\α},\mathrm{\ξ})=\underset{i}{\mathrm{\Σ}}\frac{{({\mathrm{\α}}_i-{\mathrm{\ξ}}_i)}^2}{{\mathrm{\α}}_i+{\mathrm{\ξ}}_i}---\left(2\right)$

In the formula, α _iAnd ξ _iTwo width of cloth image x, i component of LBP feature of y;

To the image in the every class expression training set, utilize formula (1) to calculate each other similarity after, utilize the AP clustering algorithm that the image clustering in every class expression training set is different subclasses;

To each subclass of every class expression, add up its corresponding AU combined situation, determine the AU combined situation that this subclass is final according to maximum probability criterion; Probability of erasure makes up and subclass less than 10% AU;

All subclasses of every class expression are combined into classification corresponding to whole expression database;

(d) carry out face normalization and feature extraction, adopt the svm classifier device that trains that every class expression subclass is classified.

Images