CN110728193A - Method and device for detecting richness characteristics of face image - Google Patents
Method and device for detecting richness characteristics of face image Download PDFInfo
- Publication number
- CN110728193A CN110728193A CN201910872851.8A CN201910872851A CN110728193A CN 110728193 A CN110728193 A CN 110728193A CN 201910872851 A CN201910872851 A CN 201910872851A CN 110728193 A CN110728193 A CN 110728193A
- Authority
- CN
- China
- Prior art keywords
- facial
- image
- sample image
- face
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The method comprises the steps of carrying out face detection on at least one obtained face sample image to obtain a region to be detected of each image in the at least one face sample image; respectively carrying out facial feature extraction and facial expression recognition on the to-be-detected region of each image in at least one facial sample image to obtain facial feature information and facial expression information of each image in at least one facial sample image; feature fusion is carried out on the facial feature information and facial expression information of each image in at least one facial sample image to obtain the richness features of at least one facial sample image, so that the sample quality of the facial image used for training can be evaluated in the subsequent training of the facial image replacement model, the trained facial image replacement model can be effectively attached to various facial poses, expressions and the like of the face used for replacement, and therefore face replacement experience of a user is improved.
Description
Technical Field
The application relates to the technical field of image processing, in particular to a method and equipment for detecting richness characteristics of a face image.
Background
The face replacement technology is an important research direction in the field of computer vision, and has great influence on business, entertainment and some special industries due to various defects of manual image editing and fusion by software such as photoshop and the like. In the existing deep face recognition technology (such as deep face changing), an effective sample quality evaluation system (such as sample richness) is not established for a batch of face samples provided by a user or a third party, the reliability of a final synthesized sample is difficult to guarantee, various facial gestures, expressions and the like of a target face in a template sample cannot be effectively attached to a training model easily, the final effect is poor, and therefore face changing experience is reduced.
Disclosure of Invention
An object of the present application is to provide a method and an apparatus for detecting richness features of a face image, so as to solve the problem in the prior art that richness features of a face are lacking in a face recognition process.
According to one aspect of the application, a method for detecting richness features of a face image is provided, wherein the method comprises the following steps:
acquiring at least one face sample image;
performing face detection on the at least one face sample image to obtain a region to be detected of each image in the at least one face sample image;
extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain facial feature information of each image in the at least one facial sample image;
performing facial expression recognition on the to-be-detected region of each image in the at least one facial sample image to obtain facial expression information of each image in the at least one facial sample image;
and performing feature fusion on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image.
Further, in the above method, the facial feature information includes the facial feature point information and face angle information, wherein,
the extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain the facial feature information of each image in the at least one facial sample image includes:
extracting facial feature points of a region to be detected of each image in the at least one facial sample image to obtain facial feature point information of each image in the at least one facial sample image;
carrying out face angle recognition on the face characteristic point information of each image in the at least one face sample image to obtain face angle information of each image in the at least one face sample image;
the feature fusion is performed on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image, and the feature fusion includes:
and performing feature fusion on the facial feature point information, the facial angle information and the facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image.
Further, in the above method, the extracting facial feature points of the region to be detected of each image in the at least one facial sample image to obtain facial feature point information of each image in the at least one facial sample image includes:
acquiring a key point positioning model for detecting facial features, wherein the key point positioning model is obtained by training a local binarization feature algorithm and a random forest algorithm;
and extracting facial feature points of the to-be-detected region of each image in the at least one facial sample image through the key point positioning model to obtain facial feature point information of each image in the at least one facial sample image.
Further, in the above method, the performing face angle recognition on the facial feature point information of each image in the at least one face sample image to obtain the face angle information of each image in the at least one face sample image includes:
acquiring a face angle recognition model for recognizing a face angle;
and carrying out face angle recognition on the face feature point information of each image in the at least one face sample image through the face angle recognition model to obtain the face angle information of each image in the at least one face sample image.
Further, in the above method, the performing facial expression recognition on the region to be detected of each image in the at least one face sample image to obtain facial expression information of each image in the at least one face sample image includes:
acquiring a facial expression recognition model for recognizing facial expressions, wherein the facial expression recognition model is obtained by convolutional neural network training based on deep learning;
and carrying out facial expression recognition on the to-be-detected region of each image in the at least one facial sample image through the facial expression recognition model to obtain facial expression information of each image in the at least one facial sample image.
Further, in the above method, the performing feature fusion on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of the at least one facial sample image includes:
fusing facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of each image in the at least one facial sample image;
and obtaining the richness characteristic of the at least one face sample image according to the richness characteristic of each image in the at least one face sample image.
Further, in the above method, the performing feature fusion on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of the at least one facial sample image includes:
obtaining facial feature information of the at least one facial sample image according to the facial feature information of each image in the at least one facial sample image;
obtaining facial expression information of the at least one face sample image according to facial expression information of each image in the at least one face sample image;
and fusing the facial feature information and facial expression information of the at least one facial sample image to obtain the richness feature of the at least one facial sample image.
Further, in the above method, the method further includes:
carrying out validity judgment on the corresponding face sample image based on the to-be-detected region,
and if the face sample image corresponding to the area to be detected is an effective face sample image, performing facial feature extraction and facial expression recognition on the area to be detected of the face sample image.
Further, in the above method, the determining the validity of the corresponding face sample image based on the to-be-detected region includes:
acquiring content information, pixel information and size information of the area to be detected;
and judging the effectiveness of the corresponding face sample image based on the content information, the pixel information and the size information of the region to be detected.
According to another aspect of the present application, there is also provided a face image richness feature detection apparatus, wherein the apparatus includes:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement a method for detecting richness features of facial images as described above.
According to another aspect of the present application, there is also provided a face image richness feature detection apparatus, wherein the apparatus includes:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement a method for detecting richness features of facial images as described above.
Compared with the prior art, the method comprises the steps of obtaining at least one face sample image; performing face detection on the at least one face sample image to obtain a region to be detected of each image in the at least one face sample image; extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain facial feature information of each image in the at least one facial sample image; performing facial expression recognition on the to-be-detected region of each image in the at least one facial sample image to obtain facial expression information of each image in the at least one facial sample image; feature fusion is carried out on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of the at least one facial sample image, detection of the richness feature of the at least one facial sample image is achieved, and therefore the sample quality of the sample facial image used for training the facial image replacement model can be evaluated in the subsequent training process of the facial image replacement model, the trained facial image replacement model can be effectively attached to various facial poses, expressions and the like of a target facial image used for replacement, and therefore face changing experience of a user is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 illustrates a flow diagram of a method for detecting richness features of a facial image in accordance with an aspect of the subject application;
fig. 2 is a schematic diagram illustrating an actual application scenario of a method for detecting richness features of a face image according to an aspect of the present application.
The same or similar reference numbers in the drawings identify the same or similar elements.
Detailed Description
The present application is described in further detail below with reference to the attached figures.
In a typical configuration of the present application, the terminal, the device serving the network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
As shown in fig. 1, a schematic flow chart of a method for detecting richness features of a face image according to an aspect of the present application includes step S11, step S12, step S13, step S14, and step S15, where the method specifically includes:
step S11, acquiring at least one face sample image; here, the face sample image is used for a face image indicating a face sample of which feature information of the face image needs to be trained, and the face sample image includes one or more face samples.
Step S12, carrying out face detection on the at least one face sample image to obtain the to-be-detected region of each image in the at least one face sample image; here, the region to be detected is used to indicate a region of the face image including a human face, which needs to be detected.
Step S13, extracting facial features of the to-be-detected region of each image in the at least one face sample image to obtain facial feature information of each image in the at least one face sample image;
step S14, performing facial expression recognition on the to-be-detected region of each image in the at least one face sample image to obtain facial expression information of each image in the at least one face sample image;
step S15, performing feature fusion on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of the at least one facial sample image.
Through the steps S11 to S15, the detection of the richness characteristics of at least one face sample image (i.e., a group of face sample images) is realized, so that the sample quality of the sample face image used for training the face image replacement model can be evaluated in the subsequent training process of the face image replacement model, and the trained face image replacement model can effectively fit various facial poses, expressions and the like of the target face image used for replacement, thereby improving the face changing experience of the user.
For example, in order to extract and count the features of the face images of a group of face samples in as many dimensions as possible, first, step S11 obtains a group of face sample images, which are respectively face sample image 1, face sample image 2, face sample image 3, … …, and face sample image N, where N is the number of selected face samples that need feature extraction, that is, the number of face sample images; in step S12, performing face detection on the face sample image 1, the face sample image 2, the face sample image 3, … …, and the face sample image N in the N face sample images to obtain a Region to be detected (ROI) 1 of the face sample image 1, a Region to be detected 2 of the face sample image 2, regions to be detected 3, … … of the face sample image 3, and a Region to be detected N of the face sample image N in the N face sample images; in step S13, facial feature extraction is performed on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI (N) of the face sample image N among the N face sample images, so as to obtain facial feature information F1 of the face sample image 1, facial feature information F2 of the face sample image 2, facial feature information F3 and … … of the face sample image 3, and facial feature information F (N) of the face sample image N among the N face sample images; in step S14, performing facial expression recognition on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI N of the face sample image N in the N face sample images to obtain facial expression information E1 of the face sample image 1, facial expression information E2 of the face sample image 2, facial expression information E3 and … … of the face sample image 3, and facial expression information E (N) of the face sample image N in the N face sample images; in step S15, feature fusion is performed on the facial feature information F1 and facial expression information E1 of the face sample image 1, the facial feature information F2 and facial expression information E2 of the face sample image 2, the facial feature information F3 and facial expression information E3, … … of the face sample image 3, the facial feature information F (N) of the face sample image N, and the facial expression information E (N), to obtain the richness feature V of the N face sample images, so as to detect the richness feature of a group of face sample images, and to evaluate the sample quality of the face sample image for training the face image replacement model in the subsequent training process of the face image replacement model, so that the trained face image replacement model can effectively fit various facial poses and expressions of the target face image for replacement, and the like, therefore, the face changing experience of the user is improved.
Next to the above embodiment of the present application, the facial feature information includes facial feature point information and facial angle information, where the step S13 performs facial feature extraction on the to-be-detected region of each image in the at least one face sample image to obtain the facial feature information of each image in the at least one face sample image, and specifically includes:
extracting facial feature points of a region to be detected of each image in the at least one facial sample image to obtain facial feature point information of each image in the at least one facial sample image;
carrying out face angle recognition on the face characteristic point information of each image in the at least one face sample image to obtain face angle information of each image in the at least one face sample image;
in step S15, feature fusion is performed on the facial feature information and facial expression information of each image in the at least one face sample image to obtain the richness feature of the at least one face sample image, which specifically includes:
and performing feature fusion on the facial feature point information, the facial angle information and the facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image.
For example, the facial feature information F includes facial feature point information FP and face angle information FA, and when facial feature extraction is performed on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI (N) of the face sample image N in step S13, the facial feature extraction includes: extracting facial feature points from the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROI3, … … of the face sample image 3, and the ROI (N) of the face sample image N in the N face sample images to obtain facial feature point information FP1 of the face sample image 1, facial feature point information FP2 of the face sample image 2, facial feature point information FP3, … … of the face sample image 3, and facial feature point information FP (N) of the face sample image N, and performing facial angle recognition on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROI3, … … of the face sample image 3, and the ROI (N) of the face sample image N based on the facial feature point information to obtain facial angle information FA1, FA3, FA of the face sample image 1 in the N face sample images, Face angle information FA2 of the face sample image 2, face angle information FA3, … … of the face sample image 3, face angle information FA (N) of the face sample image N; in the step S15, when feature fusion is performed on each of the N face sample images, feature fusion is performed on facial feature point information FP1, facial angle information FA1 and facial expression information E1 of a face sample image 1 of the N face sample images, facial feature information FP2, facial angle information FA2 and facial expression information E2 of a face sample image 2, facial feature information FP3, facial angle information FA3 and facial expression information E3, … … of a face sample image N, facial feature information FP (N), facial angle information FA (N), and facial expression information E (N) of the face sample image 3, so that the degree features V of the N face sample images obtained by feature fusion of the facial feature point information, facial angle information, and facial expression information of each of the N face sample images are enriched, and the subsequent evaluation of the richness degree of the group of face samples provides an effective evaluation basis, so that the effective evaluation of the sample quality of the face images of the face samples is realized.
Next to the above embodiment of the present application, the extracting facial feature points of the to-be-detected region of each image in the at least one face sample image in step S13 to obtain facial feature point information of each image in the at least one face sample image specifically includes:
acquiring a key point positioning model for detecting facial features, wherein the key point positioning model is obtained by training a local binarization feature algorithm and a random forest algorithm;
and extracting facial feature points of the to-be-detected region of each image in the at least one facial sample image through the key point positioning model to obtain facial feature point information of each image in the at least one facial sample image.
It should be noted that, the key point location model may use a local binarization-based feature algorithm to perform key point feature extraction on facial images of a set of face samples used for training the key point location model, and then perform key point regression on key points extracted from the facial images of the set of face samples used for training the key point location model by using the random forest algorithm, so as to train and obtain a key point location model for locating key points of facial features, so that key point feature extraction can be performed on facial images of a face whose facial features need to be extracted based on the key point location model in the following.
For example, when the facial feature point extraction is performed on the region to be detected of each of the N face sample images in step S13, a pre-trained key point location model (1) for detecting facial features is obtained, and then the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3, … … of the face sample image 3, and the ROI (N) of the face sample image N in the N face sample images are subjected to the facial feature point extraction by the key point location model (1), so as to obtain the facial feature point information FP1 of the face sample image 1, the facial feature point information FP2 of the face sample image 2, the facial feature point information FP3, … … of the face sample image 3, and the facial feature point information FP (N) of the face sample image N in the N face sample images, so as to implement the region to be detected of each of the N face sample images by the pre-trained key point location model (1) And extracting facial feature points from the domain.
Next to the foregoing embodiment of the present application, the performing facial angle recognition on the facial feature point information of each image in the at least one face sample image in step S13 to obtain the facial angle information of each image in the at least one face sample image specifically includes:
acquiring a face angle recognition model for recognizing a face angle; the face angle recognition model is obtained by performing face angle recognition training on face images of a group of face samples, so that face angle recognition can be performed on the face image of the face from which the face angle needs to be extracted based on the face angle recognition model trained in advance.
And carrying out face angle recognition on the face feature point information of each image in the at least one face sample image through the face angle recognition model to obtain the face angle information of each image in the at least one face sample image.
For example, when the face angle recognition is performed on the region to be detected of each of the N face sample images in step S13, a face angle recognition model (2) trained in advance and used for recognizing a face angle is obtained, and then the face angle recognition model (2) is used to perform face angle recognition on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3, … … of the face sample image 3, and the ROI (N) of the face sample image N in the N face sample images, so as to obtain the face angle information FA1 of the face sample image 1, the face angle information FA2 of the face sample image 2, the face angle information FA3, … … of the face sample image 3, and the face angle information FA (FA) (N) of the face sample image N in the N face sample images, so as to perform face angle recognition on the region to be detected of each of the N face sample images through the face angle recognition model (2) trained in advance And (5) identifying the face angle.
Next to the above embodiment of the present application, the step S14 performing facial expression recognition on the to-be-detected region of each image in the at least one face sample image to obtain facial expression information of each image in the at least one face sample image includes:
acquiring a facial expression recognition model for recognizing facial expressions, wherein the facial expression recognition model is obtained by convolutional neural network training based on deep learning;
and carrying out facial expression recognition on the to-be-detected region of each image in the at least one facial sample image through the facial expression recognition model to obtain facial expression information of each image in the at least one facial sample image.
It should be noted that the facial expression recognition model may adopt a convolutional neural network based on deep learning and a classification model framework thereof to perform facial expression recognition and training on facial images of a group of face samples used for training the facial expression recognition model, so as to obtain a facial expression recognition model used for recognizing and classifying facial expressions. In the process of training the facial expression recognition model, firstly, dividing facial expressions into: 7 categories of anger (Angry), aversion (dispust), Fear (Fear), Happy (Happy), sadness (Sad), Surprise (Surprise) and normal (Neutral), when a convolutional neural network based on deep learning and a classification model framework thereof are adopted to perform facial expression recognition and training on facial images of a group of face samples used for training the facial expression recognition model, facial expressions of facial images of different face samples can be trained, so that the facial images of faces needing to recognize the facial expressions can be recognized based on the facial expression recognition model.
For example, when the facial expression recognition is performed on the region to be detected of each of the N facial sample images in the step S14, a facial expression recognition model (3) for recognizing facial expressions, which is trained in advance, is obtained, and then facial expression recognition is performed on the ROI1 of the facial sample image 1, the ROI2 of the facial sample image 2, the ROIs 3, … … of the facial sample image 3, and the ROI N of the facial sample image N in the N facial sample images by the facial expression recognition model (3), so as to obtain the facial expression information E1 of the facial sample image 1, the facial expression information E2 of the facial sample image 2, the facial expression information E3, … … of the facial sample image 3, and the facial expression information E (N) of the facial sample image N in the N facial sample images, so as to implement the facial expression recognition on the region to be detected of each of the N facial sample images by the facial expression recognition model trained in advance Otherwise.
Next to the above embodiment of the present application, the step S15 performs feature fusion on the facial feature information and facial expression information of each image in the at least one face sample image to obtain the richness feature of the at least one face sample image, and specifically includes:
fusing facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of each image in the at least one facial sample image;
and obtaining the richness characteristic of the at least one face sample image according to the richness characteristic of each image in the at least one face sample image.
For example, in step S15, feature fusion is performed on facial feature information F1 and facial expression information E1 of the face sample image 1, facial feature information F2 and facial expression information E2 of the face sample image 2, facial feature information F3 and facial expression information E3, … … of the face sample image 3, facial feature information F (N) of the face sample image N, and facial expression information E (N), to obtain richness feature V1 of the face sample image 1, richness feature V2 of the face sample image 2, richness features V3, … … of the face sample image 3, and richness feature V (N) of the face sample image N in the N face sample images, and then richness feature V1, richness feature V2, 2 of the face sample image 2, richness feature V2, and b 2 of the face sample image 1 in the N face sample images, Feature statistics and fusion are carried out on the richness features V3, … … of the face sample image 3 and the richness feature V (N) of the face sample image N, and finally the richness features V of the N face sample images are obtained, so that fusion of the richness features V of the N face sample images is achieved through the richness features of the images in the N face sample images.
Next to the above embodiment of the present application, the step S14 performs feature fusion on the facial feature information and facial expression information of each image in the at least one face sample image to obtain the richness feature of the at least one face sample image, and specifically includes:
obtaining facial feature information of the at least one facial sample image according to the facial feature information of each image in the at least one facial sample image;
obtaining facial expression information of the at least one face sample image according to facial expression information of each image in the at least one face sample image;
and fusing the facial feature information and facial expression information of the at least one facial sample image to obtain the richness feature of the at least one facial sample image.
For example, in step S15, the facial feature information F1 of the face sample image 1, the facial feature information F2 of the face sample image 2, the facial feature information F3, … … of the face sample image 3, and the facial feature information F (N) of the face sample image N are counted and fused to obtain the facial feature information F (integrated) of the N face sample images; counting and fusing facial expression information E1 of a face sample image 1, facial expression information E2 of a face sample image 2, facial expression information E3 and … … of a face sample image 3 and facial expression information E (N) of a face sample image N in the N face sample images to obtain facial expression information E (comprehensive) of the N face sample images; and then, performing feature fusion on facial feature information F (synthesis) and facial expression information E (synthesis) of the N facial sample images to obtain richness features V of the N facial sample images, so as to realize fusion of the richness features of the N facial sample images through the facial feature information F (synthesis) and the facial expression information E (synthesis) of the N facial sample images.
Next to all the above embodiments of the present application, in the method for detecting richness features of a facial image provided in an embodiment of the present application, before performing facial feature extraction and facial expression recognition on a region to be detected of a face sample image, the method further includes:
carrying out validity judgment on the corresponding face sample image based on the to-be-detected region,
and if the face sample image corresponding to the area to be detected is an effective face sample image, performing facial feature extraction and facial expression recognition on the area to be detected of the face sample image.
For example, in the step S12, it is implemented that the N face sample images: after face detection of the face sample image 1, the face sample image 2, the face sample images 3 and … … and the face sample image N, obtaining an ROI1 of the face sample image 1, an ROI2 of the face sample image 2, ROIs 3 and … … of the face sample image 3 and an ROI (N) of the face sample image N in the N face sample images; before facial feature extraction and facial expression recognition are performed on a to-be-detected region of each of the N face sample images, validity judgment needs to be performed on the corresponding face sample image based on the to-be-detected region, for example, validity judgment is performed on the corresponding face sample image 1 based on the ROI1, validity judgment is performed on the corresponding face sample image 2 based on the ROI2, validity judgment is performed on the corresponding face sample image 3 based on the ROI3, and validity judgment is performed on the corresponding face sample image N based on the ROI (N), if there are one or more face sample images corresponding to the to-be-detected regions in the N to-be-detected regions respectively, the face sample image corresponding to each of the one or more to-be-detected regions in the N to-be-detected regions is determined to be a valid face sample image, in the steps S13 and S14, facial feature extraction and facial expression recognition can be performed on the face sample images corresponding to the one or more regions to be detected respectively, so as to ensure the validity of the face sample images for subsequent facial feature extraction and facial expression recognition.
Next, in the above embodiment of the present application, the determining the validity of the corresponding face sample image based on the to-be-detected region includes:
acquiring content information, pixel information and size information of the area to be detected;
and judging the effectiveness of the corresponding face sample image based on the content information, the pixel information and the size information of the region to be detected.
For example, in order to facilitate the determination of the validity of each dimension of the region to be detected of the face sample image, first, the content information, the pixel information and the size information of each region to be detected in the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI (N) of the face sample image N in the N face sample images are obtained; and then judging whether the content information, the pixel information and the size information of each region to be detected meet the image validity conditions, such as whether the content information contains facial features, whether the pixel information meets a preset pixel point threshold value and whether the size information meets a preset size threshold value to be detected, so as to realize validity judgment of each image in the N facial sample images, avoid influencing subsequent facial feature extraction and facial expression recognition due to incomplete content information, wrong pixels, small size and the like, and further guarantee the validity of the facial sample images for subsequently performing facial feature extraction and facial expression recognition.
In an actual application scenario of the face image replacement method provided by the present application, as shown in fig. 2, when a group of face samples needs to be determined by richness features, a group of face samples and face sample images thereof are obtained first: the method comprises the following steps that a face sample image 1, a face sample image 2, a face sample image 3, … … and a face sample image N are obtained, then face detection is carried out on the obtained face sample images of the N face samples in a face detection module, and an ROI1 of the face sample image 1, an ROI2 of the face sample image 2, ROIs 3 and … … of the face sample image 3 and an ROI (N) of the face sample image N in the N face sample images are obtained; then, judging whether each face sample image is a face image corresponding to an effective face or not based on the to-be-detected region, if not, carrying out any processing on the face image corresponding to the ineffective face, and if so, carrying out subsequent face processing on the face image corresponding to the effective face in the N face sample images; if the N face sample images all belong to face images corresponding to valid human faces, extracting facial feature points from the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI (N) of the face sample image N in the N face sample images to obtain facial feature point information FP1 of the face sample image 1, facial feature point information FP2 of the face sample image 2, facial feature point information FP3 and … … of the face sample image 3, and facial feature point information FP (N) of the face sample image N, and identifying the angles of the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the face ROI3 and … … of the face sample image 3, and the ROI (N) of the face sample image N based on the facial feature point information, obtaining face angle information FA1 of a face sample image 1, face angle information FA2 of a face sample image 2, face angle information FA3, … … of a face sample image 3, face angle information FA (N) of a face sample image N of the N face sample images; of course, in order to take the facial expression into consideration, facial expression recognition needs to be performed on the ROI1 of the face sample image 1, the ROI2 of the face sample image 2, the ROIs 3 and … … of the face sample image 3, and the ROI N of the face sample image N in the N face sample images, so as to obtain facial expression information E1 of the face sample image 1, facial expression information E2 of the face sample image 2, facial expression information E3 and … … of the face sample image 3, and facial expression information E (N) of the face sample image N in the N face sample images; then, the richness feature V of the N face sample images may be obtained according to any feature fusion in facial feature point information, facial angle information, and facial expression information, for example, feature fusion is performed on facial feature information F1 and facial expression information E1 of the face sample image 1, facial feature information F2 and facial expression information E2 of the face sample image 2, facial feature information F3 and facial expression information E3, … … of the face sample image 3, facial feature information F (N) and facial expression information E (N) of the face sample image N to obtain the richness feature V of the N face sample images, so as to implement detection of the richness feature of a group of sample face images; for example, feature fusion may be performed on facial feature point information FP1, facial angle information FA1 and facial expression information E1 of the face sample image 1, facial feature information FP2, facial angle information FA2 and facial expression information E2 of the face sample image 2, facial feature information FP3, facial angle information FA3 and facial expression information E3, … … of the face sample image N, facial angle information FA (N) and facial expression information E (N) of the face sample image 3 to obtain richness features V of the N face sample images, so that the richness features V of the N face sample images obtained by feature fusion of the facial feature point information, facial angle information and expression information of each of the N face sample images are richer, and effective evaluation basis is provided for subsequently measuring the richness of the set of face samples, therefore, the effective evaluation of the sample quality of the face image of the face sample is realized.
The application further provides a device for detecting richness characteristics of face images in another embodiment, wherein the device comprises:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement a method for detecting richness features of facial images as described above.
The application further provides a device for detecting richness characteristics of face images in another embodiment, wherein the device comprises:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement a method for detecting richness features of facial images as described above.
Here, the detailed contents of each embodiment of the device for detecting the richness feature of the face image may specifically refer to the corresponding part of the embodiment of the method for detecting the richness feature of the face image provided in the foregoing embodiment, and are not described herein again.
In summary, the present application provides a method for generating a facial image by obtaining at least one facial sample image; performing face detection on the at least one face sample image to obtain a region to be detected of each image in the at least one face sample image; extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain facial feature information of each image in the at least one facial sample image; performing facial expression recognition on the to-be-detected region of each image in the at least one facial sample image to obtain facial expression information of each image in the at least one facial sample image; feature fusion is carried out on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of the at least one facial sample image, detection of the richness feature of the at least one facial sample image is achieved, and therefore the sample quality of the sample facial image used for training the facial image replacement model can be evaluated in the subsequent training process of the facial image replacement model, the trained facial image replacement model can be effectively attached to various facial poses, expressions and the like of a target facial image used for replacement, and therefore face changing experience of a user is improved.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, implemented using Application Specific Integrated Circuits (ASICs), general purpose computers or any other similar hardware devices. In one embodiment, the software programs of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including associated data structures) of the present application may be stored in a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. Additionally, some of the steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
In addition, some of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application through the operation of the computer. Program instructions which invoke the methods of the present application may be stored on a fixed or removable recording medium and/or transmitted via a data stream on a broadcast or other signal-bearing medium and/or stored within a working memory of a computer device operating in accordance with the program instructions. An embodiment according to the present application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to perform a method and/or a solution according to the aforementioned embodiments of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Claims (11)
1. A method for detecting richness characteristics of a face image, wherein the method comprises the following steps:
acquiring at least one face sample image;
performing face detection on the at least one face sample image to obtain a region to be detected of each image in the at least one face sample image;
extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain facial feature information of each image in the at least one facial sample image;
performing facial expression recognition on the to-be-detected region of each image in the at least one facial sample image to obtain facial expression information of each image in the at least one facial sample image;
and performing feature fusion on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image.
2. The method of claim 1, wherein the facial feature information includes the facial feature point information and facial angle information, wherein,
the extracting facial features of the to-be-detected region of each image in the at least one facial sample image to obtain the facial feature information of each image in the at least one facial sample image includes:
extracting facial feature points of a region to be detected of each image in the at least one facial sample image to obtain facial feature point information of each image in the at least one facial sample image;
carrying out face angle recognition on the face characteristic point information of each image in the at least one face sample image to obtain face angle information of each image in the at least one face sample image;
the feature fusion is performed on the facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image, and the feature fusion includes:
and performing feature fusion on the facial feature point information, the facial angle information and the facial expression information of each image in the at least one facial sample image to obtain the richness features of the at least one facial sample image.
3. The method according to claim 2, wherein the extracting facial feature points of the region to be detected of each image in the at least one facial sample image to obtain facial feature point information of each image in the at least one facial sample image comprises:
acquiring a key point positioning model for detecting facial features, wherein the key point positioning model is obtained by training a local binarization feature algorithm and a random forest algorithm;
and extracting facial feature points of the to-be-detected region of each image in the at least one facial sample image through the key point positioning model to obtain facial feature point information of each image in the at least one facial sample image.
4. The method according to claim 3, wherein the performing face angle recognition on the face feature point information of each of the at least one face sample image to obtain the face angle information of each of the at least one face sample image comprises:
acquiring a face angle recognition model for recognizing a face angle;
and carrying out face angle recognition on the face feature point information of each image in the at least one face sample image through the face angle recognition model to obtain the face angle information of each image in the at least one face sample image.
5. The method according to claim 1, wherein the performing facial expression recognition on the region to be detected of each image in the at least one facial sample image to obtain facial expression information of each image in the at least one facial sample image comprises:
acquiring a facial expression recognition model for recognizing facial expressions, wherein the facial expression recognition model is obtained by convolutional neural network training based on deep learning;
and carrying out facial expression recognition on the to-be-detected region of each image in the at least one facial sample image through the facial expression recognition model to obtain facial expression information of each image in the at least one facial sample image.
6. The method of claim 1, wherein the feature fusion of facial feature information and facial expression information of each image in the at least one face sample image to obtain richness features of the at least one face sample image comprises:
fusing facial feature information and facial expression information of each image in the at least one facial sample image to obtain the richness feature of each image in the at least one facial sample image;
and obtaining the richness characteristic of the at least one face sample image according to the richness characteristic of each image in the at least one face sample image.
7. The method of claim 1, wherein the feature fusion of facial feature information and facial expression information of each image in the at least one face sample image to obtain richness features of the at least one face sample image comprises:
obtaining facial feature information of the at least one facial sample image according to the facial feature information of each image in the at least one facial sample image;
obtaining facial expression information of the at least one face sample image according to facial expression information of each image in the at least one face sample image;
and fusing the facial feature information and facial expression information of the at least one facial sample image to obtain the richness feature of the at least one facial sample image.
8. The method of any of claims 1 to 7, wherein the method further comprises:
carrying out validity judgment on the corresponding face sample image based on the to-be-detected region,
and if the face sample image corresponding to the area to be detected is an effective face sample image, performing facial feature extraction and facial expression recognition on the area to be detected of the face sample image.
9. The method according to claim 8, wherein the validity judgment of the corresponding face sample image based on the region to be detected comprises:
acquiring content information, pixel information and size information of the area to be detected;
and judging the effectiveness of the corresponding face sample image based on the content information, the pixel information and the size information of the region to be detected.
10. A face image richness feature detection apparatus, wherein the apparatus comprises:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.
11. A face image richness feature detection apparatus, wherein the apparatus comprises:
one or more processors;
a non-volatile storage medium for storing one or more computer-readable instructions,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910872851.8A CN110728193B (en) | 2019-09-16 | 2019-09-16 | Method and device for detecting richness characteristics of face image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910872851.8A CN110728193B (en) | 2019-09-16 | 2019-09-16 | Method and device for detecting richness characteristics of face image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110728193A true CN110728193A (en) | 2020-01-24 |
| CN110728193B CN110728193B (en) | 2022-10-04 |
Family
ID=69219083
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910872851.8A Active CN110728193B (en) | 2019-09-16 | 2019-09-16 | Method and device for detecting richness characteristics of face image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110728193B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507241A (en) * | 2020-04-14 | 2020-08-07 | 四川聚阳科技集团有限公司 | Lightweight network classroom expression monitoring method |
| CN111553267A (en) * | 2020-04-27 | 2020-08-18 | 腾讯科技（深圳）有限公司 | Image processing method, image processing model training method and device |
| CN112749657A (en) * | 2021-01-07 | 2021-05-04 | 北京码牛科技有限公司 | House renting management method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184249A (en) * | 2015-08-28 | 2015-12-23 | 百度在线网络技术（北京）有限公司 | Method and device for processing face image |
| CN107180234A (en) * | 2017-06-01 | 2017-09-19 | 四川新网银行股份有限公司 | The credit risk forecast method extracted based on expression recognition and face characteristic |
| CN107316020A (en) * | 2017-06-26 | 2017-11-03 | 司马大大（北京）智能***有限公司 | Face replacement method, device and electronic equipment |
| CN108363973A (en) * | 2018-02-07 | 2018-08-03 | 电子科技大学 | A kind of unconfined 3D expressions moving method |
| CN108491835A (en) * | 2018-06-12 | 2018-09-04 | 常州大学 | Binary channels convolutional neural networks towards human facial expression recognition |
| CN109614910A (en) * | 2018-12-04 | 2019-04-12 | 青岛小鸟看看科技有限公司 | A kind of face identification method and device |
| CN110096925A (en) * | 2018-01-30 | 2019-08-06 | 普天信息技术有限公司 | Enhancement Method, acquisition methods and the device of Facial Expression Image |
-
2019
- 2019-09-16 CN CN201910872851.8A patent/CN110728193B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105184249A (en) * | 2015-08-28 | 2015-12-23 | 百度在线网络技术（北京）有限公司 | Method and device for processing face image |
| CN107180234A (en) * | 2017-06-01 | 2017-09-19 | 四川新网银行股份有限公司 | The credit risk forecast method extracted based on expression recognition and face characteristic |
| CN107316020A (en) * | 2017-06-26 | 2017-11-03 | 司马大大（北京）智能***有限公司 | Face replacement method, device and electronic equipment |
| CN110096925A (en) * | 2018-01-30 | 2019-08-06 | 普天信息技术有限公司 | Enhancement Method, acquisition methods and the device of Facial Expression Image |
| CN108363973A (en) * | 2018-02-07 | 2018-08-03 | 电子科技大学 | A kind of unconfined 3D expressions moving method |
| CN108491835A (en) * | 2018-06-12 | 2018-09-04 | 常州大学 | Binary channels convolutional neural networks towards human facial expression recognition |
| CN109614910A (en) * | 2018-12-04 | 2019-04-12 | 青岛小鸟看看科技有限公司 | A kind of face identification method and device |
Non-Patent Citations (1)
| Title |
|---|
| 段培聪: "基于特征点的人脸姿态估计与识别***研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507241A (en) * | 2020-04-14 | 2020-08-07 | 四川聚阳科技集团有限公司 | Lightweight network classroom expression monitoring method |
| CN111553267A (en) * | 2020-04-27 | 2020-08-18 | 腾讯科技（深圳）有限公司 | Image processing method, image processing model training method and device |
| CN111553267B (en) * | 2020-04-27 | 2023-12-01 | 腾讯科技（深圳）有限公司 | Image processing method, image processing model training method and device |
| CN112749657A (en) * | 2021-01-07 | 2021-05-04 | 北京码牛科技有限公司 | House renting management method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110728193B (en) | 2022-10-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110827247B (en) | Label identification method and device | |
| CN107239666B (en) | Method and system for desensitizing medical image data | |
| Diem et al. | cBAD: ICDAR2017 competition on baseline detection | |
| Yang et al. | Lecture video indexing and analysis using video ocr technology | |
| CN110728193B (en) | Method and device for detecting richness characteristics of face image | |
| CN110175609B (en) | Interface element detection method, device and equipment | |
| CN111291661B (en) | Method and equipment for identifying text content of icon in screen | |
| CN111738041A (en) | Video segmentation method, device, equipment and medium | |
| US20110026764A1 (en) | Detection of objects using range information | |
| CN110348392B (en) | Vehicle matching method and device | |
| CN113591746B (en) | Document table structure detection method and device | |
| CN112487848A (en) | Character recognition method and terminal equipment | |
| CN110827246A (en) | Electronic equipment frame appearance flaw detection method and equipment | |
| CN113920085A (en) | Automatic auditing method and system for product display video | |
| CN111507332A (en) | Vehicle VIN code detection method and equipment | |
| CN111652144A (en) | Topic segmentation method, device, equipment and medium based on target region fusion | |
| CN111062262B (en) | Invoice recognition method and invoice recognition device | |
| CN112597909A (en) | Method and equipment for evaluating quality of face picture | |
| CN112434555A (en) | Key value pair region identification method and device, storage medium and electronic equipment | |
| CN112749696A (en) | Text detection method and device | |
| CN116189063B (en) | Key frame optimization method and device for intelligent video monitoring | |
| CN117351505A (en) | Information code identification method, device, equipment and storage medium | |
| CN111985400A (en) | Face living body identification method, device, equipment and storage medium | |
| CN111860122A (en) | Method and system for recognizing reading comprehensive behaviors in real scene | |
| CN112348112B (en) | Training method and training device for image recognition model and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |