CN113112596A - Face geometric model extraction and 3D face reconstruction method, device and storage medium - Google Patents
Face geometric model extraction and 3D face reconstruction method, device and storage medium Download PDFInfo
- Publication number
- CN113112596A CN113112596A CN202110520122.3A CN202110520122A CN113112596A CN 113112596 A CN113112596 A CN 113112596A CN 202110520122 A CN202110520122 A CN 202110520122A CN 113112596 A CN113112596 A CN 113112596A
- Authority
- CN
- China
- Prior art keywords
- face
- geometric model
- image
- model
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000000605 extraction Methods 0.000 title claims description 20
- 238000005457 optimization Methods 0.000 claims abstract description 43
- 230000011218 segmentation Effects 0.000 claims description 36
- 230000015654 memory Effects 0.000 claims description 23
- 210000004709 eyebrow Anatomy 0.000 claims description 21
- 230000006870 function Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 13
- 230000014509 gene expression Effects 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 4
- 238000003062 neural network model Methods 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract description 6
- 230000001953 sensory effect Effects 0.000 abstract description 3
- 210000001508 eye Anatomy 0.000 description 23
- 230000008569 process Effects 0.000 description 12
- 238000013461 design Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000000903 blocking effect Effects 0.000 description 5
- 238000013179 statistical model Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 230000008921 facial expression Effects 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 210000000697 sensory organ Anatomy 0.000 description 2
- 239000000758 substrate Substances 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Geometry (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a method, a device and a storage medium for extracting a geometric model of a human face and reconstructing a 3D human face, wherein the method comprises the following steps: acquiring an initial face geometric model of at least one face image and converting the at least one face image into a first silhouette image; acquiring a projected image of the face geometric model, and converting the projected image into a second silhouette image; and adjusting the relevant parameters of the face geometric model and the vertex coordinates of the corresponding optimization area of the face geometric model to ensure that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value, and acquiring the face geometric model corresponding to the second silhouette image at the moment as a final face geometric model. According to the method, the human face shape estimation accuracy is improved, the construction reality degree of the 3D human face model is improved, and sensory difference between a rendered image and an actual image is reduced by optimizing relevant parameters of a human face geometric model.
Description
Technical Field
The invention belongs to the technical field of 3D face modeling, and particularly relates to a face geometric model extraction and 3D face reconstruction method, a device and a storage medium.
Background
Image-based 3D face reconstruction is the reconstruction of a face from a single or limited number of images, which may be taken by any device in any scene. The 3D face reconstruction based on images is currently and commonly based on a scheme of a deformable 3D face Model (3D deformable Model, 3DMM), and a 3D face Model is obtained by solving appropriate deformable 3D face Model parameters from an image.
The 3D face model is usually represented in mesh format, consisting of vertices vertex and triangular patch faces. Each vertex has 3D coordinates x, y, z, and the triangular patches define the connections between the vertices. Usually, the connection relation of the triangular patches is not changed, and the coordinates of the vertexes are adjusted to obtain the face models with different forms. 3DMM as a kind of parameterized statistical model, the vertex 3D coordinates of the model can be expressed by the following formula:
wherein S is the 3D coordinate of the vertex (vertex) of the final face 3D model,
is the mean face 3D model (mean geometry) in statistical sense, BidA group of mutually orthogonal linear substrates are obtained according to a statistical model, and the geometrical shapes of the human face under neutral expression, such as facial shapes, five sense organs shapes and the like, are influenced; b isexpIs a set of linear bases orthogonal to each other obtained from a statistical model, corresponding to the compilation of expressions, also called blendshape.
Bid、BexpIs 3DMThe 3D geometric shape of a corresponding face can be determined by giving a fixed value given by the M model and any group of legal coefficients alpha and beta. And the recovery of 3D geometric information from a face image means that a suitable set of coefficients α and β is solved so that the rendered face shape represented by S corresponds to the given face image.
Some 3DMM models also include a statistical model T for texture:
wherein,
is the average texture, BtDifferent textures can be obtained by solving the appropriate coefficient delta for a fixed value given by the 3DMM model, but the textures have low reality sense.
In the existing scheme for estimating deformable 3D face model parameters from images, model parameters are regressed from face images by using a neural network, a face geometric model and textures are constructed according to the model parameters, and rendering is carried out. However, in the prior art, the estimation of the human face shape is inaccurate, so that the sensory difference between the rendered image and the actual image is large, and the calculation of the visual difference is not reasonable. How to reduce the sensory difference between the rendered image and the actual image and improve the construction reality degree of the 3D face model is worthy of study.
Disclosure of Invention
The invention provides a method, equipment and a storage medium for extracting a geometric face model and reconstructing a 3D face, aiming at solving the problem of poor construction fidelity of a 3D face model caused by inaccurate estimation of the existing face morphology.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a face geometric model extraction method in a first aspect, which comprises the following steps:
acquiring an initial face geometric model of at least one face image and converting the at least one face image into a first silhouette image;
acquiring a projected image of the face geometric model, and converting the projected image into a second silhouette image;
and adjusting relevant parameters of the face geometric model and vertex coordinates of a corresponding optimization area of the face geometric model to enable the difference between the second silhouette image and the first silhouette image to be smaller than a threshold value, and acquiring the face geometric model corresponding to the second silhouette image at the moment as a final face geometric model, wherein the relevant parameters of the face geometric model comprise a parameter alpha relevant to the shape of the face, a parameter beta relevant to the expression of the face, a scale parameter s, a rotation parameter R and a translation parameter t.
According to the scheme, the initial face geometric model is converted into the second silhouette image, the difference between the second silhouette image and the first silhouette image obtained by converting the face image is minimized by adjusting the relevant parameters of the face geometric model, namely the initial face geometric model is adjusted to an optimal model, the estimation accuracy of the face form is improved, and the construction reality degree of the subsequent 3D face model is improved.
In one possible design, the method further includes:
performing semantic segmentation on a face image to obtain at least one semantic segmentation region, wherein the semantic segmentation region comprises an eye region, an eyebrow region, a nose region and a mouth region;
determining vertex coordinates respectively corresponding to each semantic segmentation area in the at least one semantic segmentation area in the face geometric model;
the adjusting of the relevant parameters of the face geometric model and the vertex coordinates of the corresponding area of the face geometric model comprises:
and under the condition of keeping the vertex coordinates corresponding to other semantic segmentation areas unchanged, adjusting the related parameters of the face geometric model and the vertex coordinates corresponding to one semantic segmentation area to ensure that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value.
According to the scheme, at least one semantic segmentation area is optimized and adjusted to achieve local optimization.
In a possible design, the adjusting the relevant parameters of the geometric face model and the vertex coordinates of the corresponding region of the geometric face model includes:
and sequentially carrying out the following iterative operations on each divided semantic division area: and under the condition of keeping the vertex coordinates corresponding to other semantic segmentation areas unchanged, adjusting the related parameters of the face geometric model and the vertex coordinates corresponding to one semantic segmentation area to ensure that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value.
According to the scheme, the segmented semantic segmentation areas are optimized and adjusted in sequence, and local optimization of the eye area, the eyebrow area, the nose area and the mouth area is achieved.
In a possible design, the adjusting the relevant parameters of the geometric face model and the vertex coordinates of the corresponding region of the geometric face model further includes:
and adjusting the relevant parameters of the face geometric model and the vertex coordinates of the face geometric model corresponding to all the semantic segmentation areas to ensure that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value.
According to the scheme, the eye area, the eyebrow area, the nose area and the mouth area are locally optimized, and then the whole end-to-end optimization is carried out, so that the face shape estimation accuracy is further improved.
In a possible design, the adjusting the relevant parameters of the geometric face model and the vertex coordinates of the corresponding optimized region of the geometric face model so that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value includes:
performing mask processing on an unoccluded area in the semantic segmentation area to obtain a mask image of the unoccluded area;
constructing a loss function according to the vertex coordinates of the corresponding optimized region of the face geometric model, the first silhouette image, the second silhouette image and the mask image of the unoccluded region;
calculating a derivative of a loss function to a relevant parameter of the face geometric model;
and adjusting the relevant parameters of the face geometric model to the minimum loss function according to the gradient by using a gradient descent algorithm.
In one possible design, the obtaining the initial face geometric model of the at least one face image includes:
extracting 2D key points of the face image;
aligning the face of the image to a standard reference coordinate system according to the key point to obtain a normalized face image;
inputting the normalized human face image into a trained parameter regression model to obtain a 3D human face model coefficient corresponding to the human face image and an external parameter of the human face model relative to a camera coordinate system;
and obtaining an initial face geometric model according to the 3D face model coefficient and the external parameters of the face model relative to the camera coordinate system.
According to the scheme, the parameter regression model is used for regressing the relevant parameters of the face geometric model, and the accuracy is high.
In one possible design, the method further includes: and performing vertex interpolation and/or mesh smoothing on the initial geometric model.
The second aspect of the present invention provides a 3D face reconstruction method, including the following steps:
obtaining a face geometric model by adopting any one of the face geometric model extraction methods in the first aspect;
extracting a face texture image from the face image based on the face geometric model;
converting a human face geometric model into information in a 2D mode, wherein the information in the 2D mode is a depth map or a normal vector map;
and inputting the information of the 2D mode and the face texture map into the trained neural network model to obtain a rendered image of the 3D face.
A third aspect of the present invention provides an apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements a face geometric model extraction method as described in any one of the first aspect or a 3D face reconstruction method as described in the second aspect when executing the computer program.
A third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements a face geometric model extraction method as described in any one of the first aspect or a 3D face reconstruction method as described in the second aspect.
Compared with the prior art, the invention at least has the following advantages and beneficial effects:
1. according to the scheme, the difference between the silhouette image obtained by projection of the initial face geometric model and the silhouette image corresponding to the face image is reduced by adjusting the relevant parameters of the face geometric model and the vertex coordinates of the optimized area, iterative optimization is performed on the relevant parameters of the face geometric model and the parameters are adjusted to be optimal, the estimation accuracy of the face form is improved, and the 3D face reconstruction accuracy can be further improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a process diagram of preprocessing a face image.
Fig. 2 is a flowchart of a method for extracting geometric information of a human face.
Fig. 3 is a silhouette image of a geometric model of a face.
Fig. 4 is a flow chart of a 3D face reconstruction method.
Detailed Description
The invention is further described with reference to the following figures and specific embodiments. It should be noted that the description of the embodiments is provided to help understanding of the present invention, but the present invention is not limited thereto. Specific structural and functional details disclosed herein are merely illustrative of example embodiments of the invention. This invention may, however, be embodied in many alternate forms and should not be construed as limited to the embodiments set forth herein.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments of the present invention.
It should be understood that, for the term "and/or" as may appear herein, it is merely an associative relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, B exists alone, and A and B exist at the same time; in addition, for the character "/" that may appear herein, it generally means that the former and latter associated objects are in an "or" relationship.
It should be understood that specific details are provided in the following description to facilitate a thorough understanding of example embodiments. However, it will be understood by those of ordinary skill in the art that the example embodiments may be practiced without these specific details. For example, systems may be shown in block diagrams in order not to obscure the examples in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail in order to avoid obscuring example embodiments.
Examples
As shown in fig. 2, a first aspect of this embodiment provides a method for extracting geometric information of a human face, where the method may be executed on an information extraction device, where the information extraction device may be a computer, a server, a portable intelligent device, or other intelligent devices, and may be applied in the fields of facial expression editing, facial beautification and face thinning applications, and the like. The method for extracting the geometric information of the human face comprises the following steps S101 to S103.
Step S101, obtaining at leastA face image IfaceInitial face geometric model S0And converting the at least one face image into a first silhouette image Isilhouette. Specifically, an initial face geometric model S of the face image is obtained0This can be realized by steps S1011 to S1016.
In step S1011, referring to fig. a and b in fig. 1, for a given face image, 2D key points of the face image are extracted.
Step S1012, aligning the face correction of the image to a standard reference coordinate system according to the key point, so as to obtain a normalized face image. Further, the normalized face image can be further processed, including but not limited to pre-processing operations such as deblurring, super resolution, illumination equalization and the like, so that the image quality is improved, and the acquisition precision of the 3D face model coefficient and the external parameters relative to the camera coordinate system is improved.
Step S1013, errors may exist in the normalization processing process, and at this time, the normalized face image is input into the trained parameter regression model to obtain a 3D face model coefficient corresponding to the face image and an external parameter of the face model relative to the camera coordinate system, where the 3D face model coefficient includes a parameter α related to the face shape and a parameter β related to the face expression, and the external parameter of the face model relative to the camera coordinate system includes a scale parameter S, a rotation parameter R, and a translation parameter t.
Step S1014, according to the estimated 3D face model coefficient and the external parameter of the face model relative to the camera coordinate system, constructing a face geometric model, and obtaining a corresponding position through similarity transformation and alignment to obtain an initial face geometric model S0，
Wherein,
is the mean face 3D model (mean geometry) in statistical sense, BidIs obtained according to a statistical modelThe mutually orthogonal linear substrates influence the geometric shape of the face under neutral expression, such as the face shape, the five sense organs shape and the like; b isexpIs a set of linear bases orthogonal to each other obtained from a statistical model, corresponding to the compilation of expressions, also called blendshape.
If the number of vertices of the geometric model is small, step S1015 may be performed: for the initial face geometric model S0And performing vertex interpolation or mesh smoothing processing, or performing vertex interpolation and mesh smoothing processing simultaneously. By adopting the steps, the number of vertexes can be increased, the continuity of the face mesh is maintained, the sampling rate is enhanced, the definition of the texture image is improved, the adjustment freedom degree of fine adjustment of the face geometric model is enhanced, and the shape of the face image can be better fitted.
Step S1016, performing semantic segmentation on the face image to obtain at least one semantic segmentation region and a region of a blocking object, where the semantic segmentation region includes an eye region, an eyebrow region, a nose region, and a mouth region, any region in the semantic segmentation region may also be a region blocked by the blocking object, or may also be a region not blocked by the blocking object, that is, an effective face region, and the blocking object includes glasses, a mask, or other blocking objects. As shown in a c diagram in fig. 1, each different color represents a specific semantic region of the face, wherein, since the c diagram is adjusted to be black and white, a specific semantic region of the face represented at different depths in the c diagram, an effective face region mask image of the diagram obtained by semantic segmentation is marked as Imask。
Step S102, obtaining a projection image of the geometric model of the human face, and converting the projection image into a second silhouette image ISsilhouette. In this step, the projection image of the geometric model of the human face can be obtained by performing perspective projection operation on the geometric model of the human face.
Step S103, relevant parameters of the face geometric model and vertex coordinates of a corresponding optimization area of the face geometric model are adjusted, so that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value, and the face geometric model corresponding to the second silhouette image at the moment is obtained and used as a final face geometric model.
The difference between the first and second silhouette images may be achieved by: performing mask processing on an unoccluded area in the semantic segmentation area to obtain a mask image of the unoccluded area; and constructing a loss function L, solving the derivative of the loss function L to the five parameters of alpha, beta, s, R and t, and adjusting the parameter value according to the gradient by using a gradient descent algorithm. And repeating the iteration for a plurality of times to minimize the loss function L, namely adjusting the parameters to the optimal parameters.
Step S103 is a process of performing iterative optimization on the initial geometric model, and the iterative optimization may be performed by local optimization, global optimization, and first local optimization and then global optimization. The first and second images are further processed when a certain area is optimized for optimization adjustment, that is, the corresponding optimized areas of the first and second images are simultaneously adjusted to be the same color, which may be black or white, and the areas are opposite colors.
Now, a detailed process of this embodiment will be described in a manner of performing only global optimization, where the eye region, the eyebrow region, the nose region, and the mouth region of the first and second silhouette images are first turned to white, and the face region is black, as shown in fig. 3.
Constructing a loss function L:
wherein, IsilhouetteFor converting a face image into a first silhouette image, ISsilhouette(alpha, beta, s, R, t) is a second silhouette image converted from the geometric model of the human face; i ismaskThe mask image is a mask image of an unoccluded region of the human face obtained by semantic segmentation; lambda [ alpha ]iRepresents ViAnd VjThe weight of the change of the relative displacement between the two points, Vi represents the ith vertex in the optimized area, namely the face image, and VjRepresentation and vertex ViAdjacent vertex, NiRepresentation and vertex ViAdjacent vertex VjSet of sequence numbers of SallFor the eyebrow area, eye area, nose area and mouth area of the human faceiA set of sequence numbers of.
And solving the derivatives of the loss function L to the five parameters of alpha, beta, s, R and t, and adjusting the parameter values according to the gradient by using a gradient descent algorithm. And repeating the iteration for multiple times to minimize the loss function L, namely adjusting the parameters to the optimal parameters even if the difference between the second silhouette image and the first silhouette image is smaller than a threshold value, and acquiring the optimal human face geometric model corresponding to the second silhouette image as the final human face geometric model S.
In the optimization process, alpha, beta, s, R and t are regarded as adjustable variables, and the geometric information is regarded as adjustable variables, so that the geometric information is allowed to deform in a certain range on the basis of the constraint of alpha, beta, s, R and t, and the expression range of the geometric information is enlarged, and the accuracy of constructing the face geometric model is improved.
The detailed process of this embodiment will now be described in a manner of performing a local optimization:
taking mouth optimization as an example, the mouth regions of the first and second silhouette images are adjusted to be white or black, and correspondingly, the eye region, the eyebrow region, the nose region and the face region are adjusted to be black or white. Determining vertex V corresponding to mouth region in human face geometric modeliKeeping the vertex coordinates corresponding to other semantic segmentation areas unchanged, such as an eye area, an eyebrow area and a nose area, only adjusting the vertex coordinates of the mouth area and related parameters of the geometric model of the face, and specifically, constructing a loss function L according to the vertex coordinates of the corresponding optimized area of the geometric model of the face, namely the vertex coordinates of the mouth area, the first silhouette image and the second silhouette image:
wherein, IsilhouetteFor converting a face image into a first silhouette image, ISsilhouette(alpha, beta, s, R, t) is a second silhouette transformed from the geometric model of the human faceAn image; i ismaskObtaining a mask image of an obtained area where the face is not shielded according to semantic segmentation; lambda [ alpha ]iRepresents ViAnd VjWeight between which the relative displacement changes, ViI-th vertex, V, representing the optimization region, i.e. the mouth regionjRepresentation and vertex ViAdjacent vertex, NiRepresentation and vertex ViAdjacent vertex VjSet of sequence numbers of SmouthIs a set of sequence numbers for the vertices of the optimization region, i.e., the mouth region.
And solving the derivatives of the loss function L to the five parameters of alpha, beta, s, R and t, and adjusting the parameter values according to the gradient by using a gradient descent algorithm. And repeating the iteration for multiple times to minimize the loss function L, namely adjusting the parameters to the optimal parameters even if the difference between the second silhouette image and the first silhouette image is smaller than a threshold value, and acquiring the optimal human face geometric model corresponding to the second silhouette image as the final human face geometric model S.
In the optimization process, alpha, beta, s, R and t are taken as adjustable variables, and the geometric information is taken as the adjustable variables, so that the geometric information is allowed to deform in a certain range on the basis of the constraint of alpha, beta, s, R and t, and the expression range of the geometric information is enlarged, and the accuracy of constructing the face geometric model is improved.
If only the eye area, the eyebrow area and the nose area are optimized separately, the processing mode of the mouth can be referred to.
Optimally, the detailed process of the embodiment is now described in a manner of local optimization and then global optimization:
the description will be given by taking an example of a sequence of optimizing the mouth region, the eye region, the eyebrow region, and the nose region, and then performing global end-to-end optimization, but the local optimization may also adopt other ordering modes.
And obtaining an optimal second silhouette image under local optimization by adopting the mouth optimization mode, and determining a human face geometric model S1 corresponding to the optimal second silhouette image. And adjusting the eye areas of the first and second silhouette images to be white or black, and adjusting the mouth areas, the eyebrow areas, the nose areas and the face areas to be black or white. Keeping vertex coordinates corresponding to other semantic segmentation areas unchanged, such as a mouth area, an eyebrow area and a nose area, only adjusting vertex coordinates of an eye area and related parameters of a face geometric model, and specifically, constructing a loss function construction loss function L according to the vertex coordinates of a corresponding optimization area of the face geometric model, namely the vertex coordinates of the eye area, a first silhouette image and a second silhouette image, and continuously performing iterative optimization on the eye area:
wherein, IsilhouetteFor converting a face image into a first silhouette image, ISsilhouette(alpha, beta, s, R, t) is a second silhouette image converted from the geometric model of the human face; i ismaskObtaining a mask image of an obtained area where the face is not shielded according to semantic segmentation; lambda [ alpha ]iRepresents ViAnd VjWeight between which the relative displacement changes, ViI-th vertex, V, representing an optimized region, i.e. an eye regionjRepresentation and vertex ViAdjacent vertex, NiRepresentation and vertex ViAdjacent vertex VjSet of sequence numbers of SeyeIs a set of sequence numbers for the vertices of the optimization area, i.e. the eye area.
And solving the derivative of the loss function L to the relevant parameters of the five human face geometric models of alpha, beta, s, R and t, and adjusting the parameter value according to the gradient by using a gradient descent algorithm. And repeating the iteration for multiple times to minimize the loss function L, namely adjusting the parameters to the optimal parameters even if the difference between the second silhouette image and the first silhouette image is smaller than a threshold value, and acquiring the optimal human face geometric model corresponding to the second silhouette image as a final human face geometric model S2, namely optimizing the human face geometric model S1 to the human face geometric model S2.
The optimization of the eyebrow area and the nose area is also carried out by adopting the method, when the eyebrow area is optimized, the eyebrow areas of the first and second silhouette images are adjusted to be white or black, and the mouth area, the eye area, the nose area and the face area are adjusted to be black or white; and keeping the vertex coordinates of the eye region, the nose region and the mouth region unchanged, and only adjusting the vertex coordinates of the eyebrow region and the related parameters of the face geometric model to obtain the optimal face geometric model S3. When the nose area is optimized, adjusting the nose areas of the first and second silhouette images to be white or black, and adjusting the mouth area, the eye area, the eyebrow area and the face area to be black or white; and keeping the vertex coordinates of the eye region, the eyebrow region and the mouth region unchanged, and only adjusting the vertex coordinates of the nose region and the related parameters of the human face geometric model to obtain the optimal human face geometric model S4.
Finally, performing end-to-end global optimization, and adjusting the nose area, the mouth area, the eye area and the eyebrow area of the first and second silhouette images to be white or black, and the other face areas to be black or white; and adjusting relevant parameters of the face geometric model and vertex coordinates of an eye region, an eyebrow region, a nose region and a mouth region in the face geometric model, and solving the face geometric model corresponding to the second silhouette image as a final face geometric model S by adopting the method for constructing the loss function.
The scheme is based on initial geometric information S0According to semantic division of geometric vertexes of the human face, local human face geometries such as eyes, a nose, a mouth and a face are sequentially and independently optimized and fine-tuned through local 3D space transformation, the vertexes of corresponding semantics are allowed to change along with changes of geometric parameters, each vertex can freely move in the space, and smooth constraint is added between adjacent vertexes. When the corresponding position is optimized, other vertex parts are not moved, the change is limited to be local, and smooth continuity is realized. After each semantic segmentation area is optimized, end-to-end optimization of geometric coefficients and vertexes is carried out. With the scheme of the embodiment, whether the optimization is local optimization or end-to-end optimization, the method considers alpha, beta, s, R and t as adjustable variables, and simultaneously takes the geometric information as the adjustable variables, so that the geometric information is allowed to be usedOn the basis of the constraint of alpha, beta, s, R and t, some deformation is carried out in a certain range, so that the expression range of geometric information is enlarged, and the accuracy of the construction of a face geometric model is improved.
Based on the method of the first aspect, a plurality of face images of the same person can be jointly optimized, the face images share the same alpha coefficient in the optimization process, specifically, the face images are respectively processed in the steps S101 and S102, the face images share the same alpha coefficient in each optimization process, the respective beta, S, R and t and the vertex coordinates of the corresponding optimization area of the geometric face model corresponding to each image are respectively adjusted, and the optimization areas are the same in each optimization process.
A second aspect of the present disclosure provides a 3D face reconstruction method, as shown in fig. 4, the method includes the following steps S201 to S204.
Step S201, obtaining a face geometric model by adopting the face geometric model extraction method in the first aspect;
step S202, extracting a face texture image from a face image based on a face geometric model;
step S203, converting the face geometric model into information of a 2D mode, wherein the information of the 2D mode is a depth map or a normal vector map, in the step, parameters of the face geometric model can be fixed, and the face geometric model in a grid mode is converted into the information of the 2D mode through a differentiable renderer.
And S204, rendering based on the 2D mode information and the face texture map to obtain a 3D face image. In this step, a neural network model may be used to receive the depth map, normal vector map, and face texture map, and generate an image of the 3D face. In the method, the neural network module replaces a differentiable renderer to render the image, so that the rendering effect is further improved.
According to the scheme, two graphical elements of a face geometric model and a face texture image are estimated from a face image, the face image is rendered according to the estimated elements, and the similarity between the face image and a face in a real image is improved.
A third aspect of the present invention provides a face geometric model extraction method, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the face geometric model extraction method according to the first aspect when executing the computer program. By way of specific example, the Memory may include, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Flash Memory (Flash Memory), a first-in-first-out Memory (FIFO), a first-in-last-out Memory (FILO), and/or the like; the processor is not limited to a processor integrated with a NPU (neutral-network processing unit) by using a microprocessor of model STM32F105 series, an ARM, an X86, and the like.
A fourth aspect of the present disclosure provides a 3D face reconstruction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the 3D face reconstruction method in the second aspect when executing the computer program. As in the third aspect, the Memory may include, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Flash Memory (Flash Memory), a first-in-first-out Memory (FIFO), and/or a first-in-last-out Memory (FILO), etc.; the processor is not limited to a processor integrated with a NPU (neutral-network processing unit) by using a microprocessor of model STM32F105 series, an ARM, an X86, and the like.
A fifth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for extracting a geometric face model according to the first aspect or the method for reconstructing a 3D face according to the second aspect. The computer-readable storage medium refers to a carrier for storing data, and may include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, flash disks and/or Memory sticks (Memory sticks), etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
Finally, it should be noted that: the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A face geometric model extraction method is characterized by comprising the following steps:
acquiring an initial face geometric model of at least one face image and converting the at least one face image into a first silhouette image;
acquiring a projected image of the face geometric model, and converting the projected image into a second silhouette image;
and adjusting relevant parameters of the face geometric model and vertex coordinates of a corresponding optimization area of the face geometric model to enable the difference between the second silhouette image and the first silhouette image to be smaller than a threshold value, and acquiring the face geometric model corresponding to the second silhouette image at the moment as a final face geometric model, wherein the relevant parameters of the face geometric model comprise a parameter alpha relevant to the shape of the face, a parameter beta relevant to the expression of the face, a scale parameter s, a rotation parameter R and a translation parameter t.
2. The method for extracting the geometric model of the human face according to claim 1, wherein the method further comprises the following steps:
performing semantic segmentation on a face image to obtain at least one semantic segmentation region, wherein the semantic segmentation region comprises an eye region, an eyebrow region, a nose region and a mouth region;
determining vertex coordinates respectively corresponding to each semantic segmentation area in the at least one semantic segmentation area in the face geometric model;
the adjusting of the relevant parameters of the face geometric model and the vertex coordinates of the corresponding area of the face geometric model comprises:
and under the condition of keeping the vertex coordinates corresponding to other semantic segmentation areas unchanged, adjusting the related parameters of the face geometric model and the vertex coordinates corresponding to one semantic segmentation area to ensure that the difference between the first and second silhouette images is smaller than a threshold value.
3. The extraction method of the geometric model of the human face as claimed in claim 2, wherein: the adjusting of the relevant parameters of the face geometric model and the vertex coordinates of the corresponding area of the face geometric model comprises:
and sequentially carrying out the following iterative operations on each divided semantic division area: and under the condition of keeping the vertex coordinates corresponding to other semantic segmentation areas unchanged, adjusting the related parameters of the face geometric model and the vertex coordinates corresponding to one semantic segmentation area to ensure that the difference between the first and second silhouette images is smaller than a threshold value.
4. The extraction method of the geometric model of the human face according to the claim 2 or 3, characterized in that: the adjusting of the relevant parameters of the face geometric model and the vertex coordinates of the corresponding region of the face geometric model further comprises:
and adjusting the relevant parameters of the face geometric model and the vertex coordinates of the face geometric model corresponding to all the semantic segmentation areas to ensure that the difference between the second silhouette image and the first silhouette image is smaller than a threshold value.
5. The extraction method of the geometric model of the human face as claimed in claim 2, wherein: the adjusting the relevant parameters of the face geometric model and the vertex coordinates of the corresponding optimization area of the face geometric model to make the difference between the second silhouette image and the first silhouette image smaller than a threshold value comprises:
performing mask processing on an unoccluded area in the semantic segmentation area to obtain a mask image of the unoccluded area;
constructing a loss function according to the vertex coordinates of the corresponding optimized region of the face geometric model, the first silhouette image, the second silhouette image and the mask image of the unoccluded region;
calculating a derivative of a loss function to a relevant parameter of the face geometric model;
and adjusting the relevant parameters of the face geometric model to the minimum loss function according to the gradient by using a gradient descent algorithm.
6. The extraction method of the geometric model of the human face according to claim 1, characterized in that: the acquiring of the initial face geometric model of at least one face image comprises:
extracting 2D key points of the face image;
aligning the face of the image to a standard reference coordinate system according to the key point to obtain a normalized face image;
inputting the normalized human face image into a trained parameter regression model to obtain a 3D human face model coefficient corresponding to the human face image and an external parameter of the human face model relative to a camera coordinate system;
and obtaining an initial face geometric model according to the 3D face model coefficient and the external parameters of the face model relative to the camera coordinate system.
7. The method for extracting a geometric model of a human face according to claim 6, wherein the method further comprises: and performing vertex interpolation and/or mesh smoothing on the initial geometric model.
8. A3D face reconstruction method is characterized by comprising the following steps:
obtaining a human face geometric model by adopting the human face geometric model extraction method of any one of the claims 1 to 7;
extracting a face texture image from the face image based on the face geometric model;
converting a human face geometric model into information in a 2D mode, wherein the information in the 2D mode is a depth map or a normal vector map;
and inputting the information of the 2D mode and the face texture map into the trained neural network model to obtain a rendered image of the 3D face.
9. An apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements a face geometry model extraction method as claimed in any one of claims 1 to 7 or a 3D face reconstruction method as claimed in claim 8 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements a face geometry model extraction method according to any one of claims 1 to 7 or a 3D face reconstruction method according to claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110520122.3A CN113112596B (en) | 2021-05-12 | 2021-05-12 | Face geometric model extraction and 3D face reconstruction method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110520122.3A CN113112596B (en) | 2021-05-12 | 2021-05-12 | Face geometric model extraction and 3D face reconstruction method, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113112596A true CN113112596A (en) | 2021-07-13 |
| CN113112596B CN113112596B (en) | 2023-10-24 |
Family
ID=76722080
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110520122.3A Active CN113112596B (en) | 2021-05-12 | 2021-05-12 | Face geometric model extraction and 3D face reconstruction method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113112596B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114783022A (en) * | 2022-04-08 | 2022-07-22 | 马上消费金融股份有限公司 | Information processing method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530900A (en) * | 2012-07-05 | 2014-01-22 | 北京三星通信技术研究有限公司 | Three-dimensional face model modeling method, face tracking method and equipment |
| CN109508678A (en) * | 2018-11-16 | 2019-03-22 | 广州市百果园信息技术有限公司 | Training method, the detection method and device of face key point of Face datection model |
| CN111563959A (en) * | 2020-05-06 | 2020-08-21 | 厦门美图之家科技有限公司 | Updating method, device, equipment and medium of three-dimensional deformable model of human face |
| CN112215050A (en) * | 2019-06-24 | 2021-01-12 | 北京眼神智能科技有限公司 | Nonlinear 3DMM face reconstruction and posture normalization method, device, medium and equipment |
-
2021
- 2021-05-12 CN CN202110520122.3A patent/CN113112596B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530900A (en) * | 2012-07-05 | 2014-01-22 | 北京三星通信技术研究有限公司 | Three-dimensional face model modeling method, face tracking method and equipment |
| CN109508678A (en) * | 2018-11-16 | 2019-03-22 | 广州市百果园信息技术有限公司 | Training method, the detection method and device of face key point of Face datection model |
| CN112215050A (en) * | 2019-06-24 | 2021-01-12 | 北京眼神智能科技有限公司 | Nonlinear 3DMM face reconstruction and posture normalization method, device, medium and equipment |
| CN111563959A (en) * | 2020-05-06 | 2020-08-21 | 厦门美图之家科技有限公司 | Updating method, device, equipment and medium of three-dimensional deformable model of human face |
Non-Patent Citations (4)
| Title |
|---|
| LI, FM等: "3D face reconstruction based on convolutional neural network", 《2017 10TH INTERNATIONAL CONFERENCE ON INTELLIGENT COMPUTATION TECHNOLOGY AND AUTOMATION》, pages 71 - 74 * |
| 彭伟龙: "面向多源数据的B样条人脸重建研究", 《中国知网》, no. 9, pages 1 - 102 * |
| 旷世FACEREC组: "3D人脸技术漫游指南", pages 1 - 10, Retrieved from the Internet <URL:http：//zhuanlan.zhihu.com/p/76010796> * |
| 王兴 等: "基于人脸关键点和三维重建的算法研究与实现", 《中国知网》, no. 3, pages 1 - 54 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114783022A (en) * | 2022-04-08 | 2022-07-22 | 马上消费金融股份有限公司 | Information processing method and device, computer equipment and storage medium |
| CN114783022B (en) * | 2022-04-08 | 2023-07-21 | 马上消费金融股份有限公司 | Information processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113112596B (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113111861A (en) | Face texture feature extraction method, 3D face reconstruction method, device and storage medium | |
| US9679192B2 (en) | 3-dimensional portrait reconstruction from a single photo | |
| CN110503680B (en) | Unsupervised convolutional neural network-based monocular scene depth estimation method | |
| US9317970B2 (en) | Coupled reconstruction of hair and skin | |
| CN113269862B (en) | Scene self-adaptive fine three-dimensional face reconstruction method, system and electronic equipment | |
| Lozes et al. | Partial difference operators on weighted graphs for image processing on surfaces and point clouds | |
| US8335403B2 (en) | Soft edge smoothness prior and application on alpha channel super resolution | |
| CN111445582A (en) | Single-image human face three-dimensional reconstruction method based on illumination prior | |
| Zhang et al. | Critical regularizations for neural surface reconstruction in the wild | |
| CN113298936B (en) | Multi-RGB-D full-face material recovery method based on deep learning | |
| WO2020108304A1 (en) | Method for reconstructing face mesh model, device, apparatus and storage medium | |
| CN109766866B (en) | Face characteristic point real-time detection method and detection system based on three-dimensional reconstruction | |
| CN112862807B (en) | Hair image-based data processing method and device | |
| CN114842136A (en) | Single-image three-dimensional face reconstruction method based on differentiable renderer | |
| CN111951383A (en) | Face reconstruction method | |
| CN110909778A (en) | Image semantic feature matching method based on geometric consistency | |
| Lawonn et al. | Stylized image triangulation | |
| CN107590858A (en) | Medical sample methods of exhibiting and computer equipment, storage medium based on AR technologies | |
| CN110717978A (en) | Three-dimensional head reconstruction method based on single image | |
| CN114429518A (en) | Face model reconstruction method, device, equipment and storage medium | |
| CN113112596A (en) | Face geometric model extraction and 3D face reconstruction method, device and storage medium | |
| CN112862684A (en) | Data processing method for depth map super-resolution reconstruction and denoising neural network | |
| CN116342377A (en) | Self-adaptive generation method and system for camouflage target image in degraded scene | |
| CN115082640A (en) | Single image-based 3D face model texture reconstruction method and equipment | |
| CN115115860A (en) | Image feature point detection matching network based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240618 Address after: Unit 202, 2nd Floor, Unit 1, Building 13, Ganluyuan Zhongli, Chaoyang District, Beijing, 100000 Patentee after: Beijing Huimei Future Technology Co.,Ltd. Country or region after: China Address before: Room 601, 5th floor, 8 Haidian North 2nd Street, Haidian District, Beijing Patentee before: Beijing Shenshang Technology Co.,Ltd. Country or region before: China |
|
| TR01 | Transfer of patent right |