CN115409932A

CN115409932A - Texture mapping and completion method of three-dimensional human head and face model

Info

Publication number: CN115409932A
Application number: CN202211041713.3A
Authority: CN
Inventors: 刘诗雅; 樊养余; 马浩悦; 刘洋
Original assignee: Star Shark Information Technology Shanghai Co ltd
Current assignee: Star Shark Information Technology Shanghai Co ltd
Priority date: 2022-08-29
Filing date: 2022-08-29
Publication date: 2022-11-29

Abstract

A texture mapping and completion method of a three-dimensional human face model comprises the following steps of firstly, constructing a three-dimensional human face mesh model; secondly, carrying out visible face discrimination on the human head-face grid model; then obtaining a texture mapping result with smaller distortion; then, based on a weighted fusion algorithm of an included angle between a patch normal vector and an image plane normal vector, realizing smooth transition of multi-image texture fusion; then, aiming at the problem of poor ear texture effect, the fusion weight of the ear region is calculated in a targeted manner, and an ear boundary fusion band is set, so that the texture optimization of the ear region is realized; and finally, performing texture missing detection by using the mask image, and respectively realizing the texture completion of a large area and a small area of the model to obtain a three-dimensional human head and face model with complete, real and natural texture. The invention can automatically generate the human head and face model with complete and natural texture under the conditions of no dependence on expensive hardware equipment and low data acquisition cost.

Description

Texture mapping and completion method of three-dimensional human head and face model

Technical Field

The invention belongs to the technical field of image processing and computer graphics, and particularly relates to a texture mapping and completion method of a three-dimensional human head and face model.

Background

The texture is used as an important component of the realistic virtual human face, so that the appearance detail characteristics of the virtual human head and face are effectively expressed, and the realistic sense of the virtual human head and face model is enhanced. With the development of technology, people have higher and higher requirements on an automatic human head and face texture mapping system, but in practice, there are many problems that are difficult to solve or the effect is difficult to satisfy:

(1) The three-dimensional face reconstruction method based on deep learning can directly reconstruct a three-dimensional face model with textures, the effect is vivid, most methods depend on a training data set, the generation of a face related data set is still performed by a traditional method, and the deep learning method has high requirements on computer hardware;

(2) In the three-dimensional face reconstruction based on the traditional method, most researches on face texture mapping focus on developing UV on a face mesh model, then pasting UV, and wrapping UV with textures on the three-dimensional model, however, the face is an inextensible surface, and the texture mapping result obtained by the method is easy to generate larger distortion;

(3) When the texture of the human face model is generated by a plurality of images of the same individual, due to factors such as illumination change of the images and geometric reconstruction errors caused by camera parameters, obvious unnatural 'traces' can appear on the surface of the model, and the model needs to be subjected to texture fusion.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention discloses a texture mapping and completion method of a three-dimensional face model, which can automatically generate the face model with complete and natural texture under the conditions of no dependence on expensive hardware equipment and low data acquisition cost.

In order to achieve the purpose, the technical scheme of the invention is as follows:

a texture mapping and completion method for a three-dimensional human head and face model comprises the following steps:

step 1: acquiring front, right, left and back views of a real face of a target person and binary mask images corresponding to the four images;

step 2: detecting the characteristic points of the face image:

predicting the human face characteristic points of the front side, the right side and the left side of the human face by utilizing a cascading regression model based on a random forest, and respectively obtaining two-dimensional human face characteristic points in the front side, the right side and the left side;

and step 3: reconstructing a three-dimensional head and face geometric model:

according to two-dimensional face characteristic points in front, right side and left side views, a face low-resolution reconstruction algorithm based on a random image set is utilized to obtain a three-dimensional face model, face posture parameters and camera parameters corresponding to each view;

the three-dimensional face geometric model is a triangular mesh patch model, and the vertex coordinates of the three-dimensional face geometric model are expressed under a world coordinate system;

and 4, step 4: and (3) judging the visible surface of the face mesh model:

visible mesh surface discrimination is carried out on the face mesh models under the front, right and left side visual angles by using a face mesh model visible surface discrimination method based on light projection, and invisible surface blanking results under three visual angles are respectively obtained;

and 5: calculating the texture coordinates of the mesh vertexes of the model:

projecting the mesh model with the invisible surfaces being blanked under the three visual angles to a pixel plane coordinate system by using weak perspective projection transformation, and respectively obtaining vertex texture coordinates of the mesh model under the three visual angles;

step 6: calculating a mesh vertex color matrix of the model:

interpolating and calculating the vertex color value of the grid model in the image of the corresponding visual angle based on the vertex texture coordinates of the grid model at the visual angles of the front side, the right side and the left side by using a nearest neighbor interpolation method, and respectively obtaining texture mapping results of the human face grid model at the three visual angles;

and 7: calculating a normal vector of a patch of the face mesh model to obtain a normal vector matrix of the patch of the face mesh;

and 8: calculating a plane normal vector of the visual angle image:

respectively calculating image plane normal vectors of the three views under a camera coordinate system by using the face pose parameters corresponding to the front, the right side and the left side views;

and step 9: effective texture image screening and texture fusion:

according to whether included angles between normal vectors of the face mesh patches and normal vectors of image planes of the front side view, the right side view and the left side view are larger than 80 degrees or not, screening effective texture images of each patch of the mesh model to obtain the number of effective images and the image sequence number of each patch of the model; then, determining fusion weight distribution basis according to cosine values of included angles between each patch normal vector of the face mesh model and the plane normal vector of the effective texture image, and performing weighted fusion on mesh vertex texture values of the face model at corresponding visual angles according to the fusion weight to obtain mesh vertex texture values after the face mesh model is fused;

the effective texture image is a view image of which the included angle between the normal vector of an image plane and the normal vector of a certain surface patch of the grid model is less than 80 degrees;

step 10: optimizing ear textures of the model;

dividing left and right ear regions of a human head and face model according to human face characteristic points, resetting fusion weights of the front side view images in the ear regions, optimizing ear textures, performing two-way continuation on boundary lines of the ear regions to obtain fusion band regions, distributing certain weights to ear mapping result texture fusion results of the side images to obtain textures of the fusion regions, achieving natural connection of the ear regions and other regions, and finally obtaining the human head and face model after the ear region textures are optimized;

step 11: carrying out binarization processing on mask images corresponding to the front view, the right view, the left view and the back view, and acquiring binary mask images of the four side views;

step 12: judging the non-texture region of the model:

mapping the binary mask images of the front view, the right view, the left view and the back view onto a face mesh model to obtain a texture missing detection result of the face mesh model;

wherein, the vertex with the color value of 0 is the vertex of the model mesh with missing texture;

step 13: and (3) completing the texture of the large area of the model:

calculating a scaling coefficient and a translation matrix according to the binary mask images of the front view and the back view, scaling and translating the back view to obtain a back image matched with the shape of the front view, mapping the matched back image onto a face mesh model according to the texture mapping method, and acquiring a mesh vertex texture value after the back texture of the face mesh model is completed;

step 14: and (3) completing the small-area texture of the model:

determining a model nose region according to coordinates of human face feature points in space, calculating a cosine value of an included angle between a model surface patch of a texture to be compensated and a surface patch normal vector of the model nose region, selecting a surface patch top texture with the largest included angle cosine value as a model surface patch top texture of the texture to be compensated, and obtaining a mesh top texture value after the small region texture of the human face mesh model is compensated;

the texture patch to be compensated is a texture missing patch after the texture on the back of the face grid model is compensated.

Preferably, in step 2, the adopted face feature point definition standard is a 73-point standard, and the method includes: 16 eye characteristic points, 12 eyebrow characteristic points, 15 face outline characteristic points, 12 outer lip characteristic points, 6 inner lip characteristic points and 12 nose characteristic points.

Preferably, in step 3, the three-dimensional human head and face model is represented in a world coordinate system as follows: f _w The three-dimensional head and face model is respectively viewed on the front, the right side and the left sideThe representation under the face pose corresponding to the graph is:

F _i ＝R _i ·F _w +T _i (1)

in the formula, F _i I =1,2,3; respectively representing a three-dimensional human head-face model under the visual angles of the front face, the right side face and the left side face; r is _i I =1,2,3 and T _i I =1,2,3; respectively a rotation matrix and a translation matrix under front, right side and left side views; the camera parameters corresponding to the front, right and left side views are respectively f _i ,i＝1,2,3。

Preferably, in step 4, the algorithm flow for distinguishing the visible face of the face mesh model is as follows:

(1) Acquiring a three-dimensional human head and face grid model under a certain visual angle in the front surface, the right side surface and the left side surface as a human head and face grid model under the current visual angle;

(2) Traversing each vertex on the current human head and face mesh model, taking the model vertex as an incident point to serve as a ray, enabling the direction of the ray to be parallel to the normal vector direction of the current visual angle image plane, and judging the intersection condition of the ray and a mesh model triangular surface patch;

if the ray is not intersected with the triangular patch on the model except the incident point, no processing is carried out on the model; if the ray has intersection points with other triangular surfaces on the model except the incident point, the triangular surface patch where other intersection points are located except the incident point is judged to be an invisible surface patch at the current visual angle;

(3) Hiding all invisible triangular patches on the current head and face mesh model;

(4) Repeating the operations (1) to (3) until the visible surface blanking of the three-dimensional human head-face mesh model under the front, right side and left side visual angles is completed, and respectively recording the model after the invisible surface blanking as F _i ',i＝1,2,3。

Preferably, in the step 5, the invisible surface blanking model F under the front, right and left side viewing angles is respectively formed _i ', i =1,2,3 to the pixel plane coordinate system, and the calculation flow for obtaining the model vertex texture coordinates of the human head-face mesh model in the corresponding view angle image is as follows:

(1) Obtaining an invisible surface blanking model F under a certain visual angle in the front surface, the right side surface and the left side surface _i ', and camera parameters f corresponding to the angle-of-view image _i ；

(2) Obtaining each vertex coordinate (x) on the current invisible surface blanking model _{C_vintex} ,y _{C_vintex} ,z _{C_vintex} )；

(3) Calculate each vertex (x) _{C_vintex} ,y _{C_vintex} ,z _{C_vintex} ) Two-dimensional projection point coordinate (x) after weak perspective projection transformation _{I_vintex} ,y _{I_vintex} ) The calculation process is as follows:

(4) Calculating the projection point (x) _{I_vintex} ,y _{I_vintex} ) Texture coordinates (u) _vintex ,v _vintex ) The calculation process is as follows:

where dx and dy denote respectively how many millimeters, u, are represented respectively for each row and each column in the pixel coordinate system ₀ And v ₀ Pixel coordinates respectively representing the center of the image plane;

(5) And (4) repeating the steps (1) and (4) until the texture coordinates of the model vertexes of the human head-face mesh model in the front view, the right side view and the left side view are calculated.

Preferably, in step 6, a nearest neighbor interpolation algorithm is used, based on the texture coordinates of the grid vertices of the human head-face grid model after the invisible surface is blanked in the front, right and left side views, the grid vertex color matrix of the model is respectively interpolated in the corresponding view-angle images, and the grid vertex color matrix ColorMatrix _ I of the human head-face grid model at three view angles is obtained _i (I =1,2,3), wherein, colorMatrix _ I _i (i =1,2,3) represents the mode corresponding to the front, right and left side view angle images, respectivelyThe type mesh vertex color matrix.

Preferably, in step 7, the calculation flow of the normal vector matrix of the face mesh model is as follows:

(1) Determining the index number faceIntex of the current patch from the first triangular patch on the mesh model;

(2) Acquiring three vertex indexes of VerIntex1, verIntex2 and VerIntex3 in a plane information matrix of the grid model according to the FaceIntex;

(3) Obtaining the space coordinates P of three vertexes in the vertex coordinate matrix of the grid model according to VerIntex1, verIntex2 and VerIntex3 ₀ (x ₀ ,y ₀ ,z ₀ ),P ₁ (x ₁ ,y ₁ ,z ₁ ),P ₂ (x ₂ ,y ₂ ,z ₂ )；

(4) Computing normal vectors for patches

Wherein the vertex coordinate P of the triangular patch is known ₀ (x ₀ ,y ₀ ,z ₀ ),P ₁ (x ₁ ,y ₁ ,z ₁ ),P ₂ (x ₂ ,y ₂ ,z ₂ ) Normal plane vector of its triangular surface

The calculation process of (2) is as follows:

first, utilize

And

the cross product of (a) to construct a plane normal vector is shown as follows:

then there are:

in the formula (5), i, j, k are unit vectors of three coordinate axes perpendicular to each other in space, and there are:

(5) Storing the normal vector obtained by current calculation into a FaceNorm matrix, and updating a patch index number faceIntex to the next patch;

(6) Repeating the operations (1) - (5) until all the triangular patches on the grid model are traversed, namely FaceIntex =22800;

the FaceNorm of the matrix of the mesh model patches obtained by calculation is a matrix of 22800 multiplied by 3;

the FaceNorm is a normal vector matrix of the triangular patch and is used for storing a normal vector of each triangular patch on the model, faceNum =22800 is the total number of the patches of the model, and FaceIntex is the index number of the current triangular patch.

Preferably, in step 8, the face pose parameters R corresponding to the front, right and left side perspective images obtained in step 3 are used _i I =1,2,3, calculating image plane normal vectors of three view images in a camera coordinate system respectively

i =1,2,3, wherein,

the calculation formula for i =1,2,3 is as follows:

preferably, in the step 9, the cosine values of the included angle are used as weight distribution basis, and texture values obtained by the vertexes of the triangular surface patch on different texture images are fused to obtain fused vertex texture values;

the effective texture image screening and texture fusion process is as follows:

(1) The calculation formula of the included angle between the normal vector of the triangular patch on the mesh model and the plane normal vector of the three visual angle images is as follows:

wherein, theta _{i_FaceIntex} Denotes an angle between a triangle patch with a patch index FaceIntex and the ith view image, i =1,2,3 denotes front, right side and left side view images, respectively,

a triangular patch normal vector representing a patch index of FaceIntex,

an image plane normal vector representing the ith perspective image;

(2) Setting a Threshold value, when theta _{i_FaceIntex} When the value is more than or equal to Threshold, the view angle image i is an invalid texture image for the FaceIntex;

(3) Calculating the fusion weight of the vertex on the triangular patch on different texture images by taking the cosine value of the included angle between the normal vector of the triangular patch on the grid model and the plane normal vector of the three visual angle images as the weight distribution basis;

the weight of the texture image is

Wherein Δ FaceIntex denotes the current triangular patch, I _K For a valid image, K is the image number,

the weight of the image with the sequence number K under the delta FaceIntex patch is set;

firstly, judging the number of effective texture images of a face mask FaceIntex, and dividing the number into the following three conditions: only one effective texture image is available; two effective texture images are available; there are three effective texture images;

when the number of valid texture images is only 1, the weight of the image is the same

When the number of the effective texture images is 2, the effective texture images are respectively marked as I _K1 ，I _K2 The included angles between the normal vector of the patch and the image plane are respectively

Then there are:

when the number of the effective texture images is 3, the effective texture images are respectively marked as I _K1 ，I _K2 ，I _K3 The included angles between the normal vector of the patch and the image plane are respectively

Then there are:

(4) According to the fusion weight, fusing texture values of vertexes on the triangular surface patch on different texture images to obtain fused vertex texture values;

obtaining vertex index VerIntex on the patch according to patch index FaceIntex of the mesh model _{ΔFaceIntex(p)} (p =1,2,3); and according to the vertex texture color matrix ColorMatrix I of the vertex of the grid model on different texture images _K And calculating the texture value after the top points on the patch are fused, wherein the calculation formula is as follows:

in the formula (I), the compound is shown in the specification,

as vertex VerIntex _{ΔFaceIntex(p)} The final texture value of (a) is,

as vertex VerIntex _{ΔFaceIntex(p)} In ColorMatrix _ I _K The texture value of (1).

Preferably, in step 10, the method for optimizing ear texture includes:

(1) Determining the boundary line of the right ear and the left ear of the model by using the human face characteristic points, thereby dividing the left ear region and the right ear region of the human face model;

the method for determining the two boundary lines of the right ear and the left ear comprises the following steps: firstly, coordinates of 6 face contour feature points are obtained and recorded as R _ Point1, R _ Point2, R _ Point3, L _ Point1, L _ Point2 and L _ Point3, and right ear and left ear boundary lines R _ earLine and L _ earLine are obtained according to the following formulas respectively:

judging three vertexes of the model triangular patch, recording all triangular patches of which the three vertexes are right from the right ear boundary line as a model right ear region, and obtaining a model left ear region in the same way;

(2) Resetting the fusion weight of the left and right visual angle images in the ear region of the human face model to be 1, and obtaining the human face model after resetting the fusion weight of the ear region textures;

(3) Performing two-way continuation on the boundary line of the ear region to obtain a fusion zone region, and then respectively allocating weight omega to the side view image mapping result and the texture fusion result obtained in the step 9 _e1 And ω _e2 Therefore, the texture of the fusion area is obtained, and the ear area is smoothly connected in the human head-face model;

assuming that the vertex color matrices of the texture blend result in the blend zone in the side view image and step 9 are ColorMatrix _ Band _ fromside i and ColorMatrix _ Band _ fromFusion, respectively, colorMatrix _ Band being the vertex color matrix of the final blend zone, the final vertex color matrix of the blend zone is represented by the following equation:

wherein, through test verification, the fusion bandwidth is selected to be 12, and the weight omega is selected _e1 ＝0.5，ω _e2 ＝0.5。

Preferably, in step 11, a threshold is determined by using a graythresh function in MATLAB, the graythresh function finds the threshold by using a maximum inter-class variance method, and then the mask image is binarized according to the im2bw function.

Preferably, in step 12, texture mapping is performed by using the methods in steps 5 to 6, a corresponding relationship between the vertices of the face model and the pixels of the binary mask image is established, a vertex color matrix of the model is obtained, the vertices with vertex color values of 0 are determined as model mesh vertices lacking correct textures, and the binary mask image is mapped onto the three-dimensional model.

Preferably, the process of step 13 is:

(1) Obtaining a front-surface binary mask image and a back-surface binary mask image, drawing a horizontal boundary line at the neck of the binary mask image, wherein the horizontal boundary line in the boundary line diagram 17 (a) shows, and at the moment, a white area in the image is a head area and a neck area of a person;

(2) In the two images, white region boundary points B on the boundary line are acquired _f0 ，B _f1 And B _b0 ，B _b1 And obtaining the pixel coordinates (x) of the boundary points _f0 ,y _f0 ),(x _f1 ,y _f1 ),(x _b0 ,y _b0 ),(x _b1 ,y _b1 ) For the pixel coordinates of the boundary points, there is y _f0 ＝y _f1 ，y _b0 ＝y _b1 ；

(3) Calculating the pixels of the white area above the boundary line according to the boundary lines of the two imagesThe number of dots, i.e. the area of the head and neck region, is denoted S _f And S _b ，S _f And S _b ；

(4) Construct an Image matrix B _ Image of size (W, H, 3), with each pixel initialized to [0,0 ] by default](ii) a Zooming the back Image B _ Image0 of the person by a zooming coefficient step (scale) to obtain an Image B _ Image1, and calculating the zooming coefficient under B _b0 ，B _b1 Update value B of _b0 '(x _b0 ',y _b0 ')，B _b1 '(x _b1 ',y _b1 ') to a host; the calculation formula of the scaling factor is as follows:

then, a translation matrix tras is constructed, and the Image B _ Image1 is translated according to the tras to obtain an Image B _ Image2, wherein the translation matrix is as follows:

wherein alpha is _x And alpha _y The calculation formula of (c) is as follows:

(5) Judging and deleting border-crossing pixel points in the Image matrix B _ Image2 to obtain a final Image B _ Image;

(6) And performing texture completion on the back of the human head and face model according to the matched back image.

Preferably, the process of step 14 is:

(1) According to the characteristic points of the human face, dividing a human head and face model nose area:

the set of triangular surface slices in the nose region is recorded as delta _nose ＝{Δ _ni |1≤ni≤Num_nose}；

(2) Calculating patch delta of texture to be compensated _{add_j} Normal vector of (1)

Normal vector of all patches in nose region

The cosine value of the included angle is calculated according to the following formula:

wherein cos θ _{add_j,ni} Patch delta representing texture to be compensated _{add_j} Triangular patch delta with nose region _ni The cosine value of the included angle;

(3) Screening out Delta _{add_j} Lower make cos theta _{add_j,ni} Maximum triangular patch Δ _ni And is recorded as Δ _ni ^maxSi ；

(4) A triangular patch Δ _ni ^maxSi The texture values of three vertices are assigned to Δ _{add_j} Three vertices of (2);

(5) And (4) repeating the steps (2) to (4) until the vertex texture value of each patch in the texture region to be compensated is calculated.

The texture mapping and completion method of the three-dimensional human head and face model has the beneficial effects that:

the invention designs a texture mapping and completion method of a three-dimensional human head and face model, which comprises the steps of firstly, constructing a three-dimensional human face mesh model with higher similarity to a target person through front, right and left side images, aiming at the problems that a texture mapping method of a UV map is distorted and needs manual operation and the problems that the texture on the surface of the model has a brightness seam and the texture is incomplete when texture fusion and completion are carried out; secondly, carrying out visible face discrimination on the human head-face grid model; then, projection point coordinates of the model mesh vertex under a pixel coordinate system are obtained by weak perspective projection transformation, texture values of the mesh vertex are calculated by a nearest neighbor interpolation method, and a texture mapping result with small distortion is automatically obtained; then, based on a weighted fusion algorithm of an included angle between a patch normal vector and an image plane normal vector, smooth transition of multi-image texture fusion is realized; then, aiming at the problem of poor ear texture effect, the fusion weight of the ear region is calculated in a targeted manner, and an ear boundary fusion band is set, so that the texture optimization of the ear region is realized; and finally, performing texture missing detection by using the mask image, and respectively realizing the texture completion of a large area and a small area of the model to obtain a three-dimensional human head and face model with complete, real and natural texture. The invention can automatically generate the human head and face model with complete and natural texture under the conditions of no dependence on expensive hardware equipment and low data acquisition cost.

Drawings

FIG. 1 is a schematic flow diagram of the present invention;

FIG. 2, front, right, left, and back perspective images of a target person and corresponding mask images;

FIG. 3 (a) is a schematic diagram of the facial feature point definition standard employed by the present invention, and (b) is a diagram of the facial feature point detection result of the front, right, and left side views of the target person;

FIG. 4, three-dimensional human head and face models at front, right side and left side perspectives;

FIG. 5 is a diagram of invisible plane blanking results at three viewing angles;

FIG. 6 is a projection of a human head-face mesh model after invisible plane blanking in a corresponding view angle image;

fig. 7 is a texture mapping result of the human head-face mesh model after the invisible surface is blanked at the three view angles in the image at the corresponding view angle;

FIG. 8 (a) is a schematic diagram of a point whose pixel coordinate is a non-integer, and (b) is a schematic diagram of nearest neighbor interpolation;

FIG. 9 is a flowchart of a computation of a normal vector matrix of the human head-face mesh model;

FIG. 10, texture mapping effect of images on models with thresholds set at 60, 70 and 80;

FIG. 11 is a comparison graph of the human head and face model effect before and after texture fusion;

FIG. 12, (a) is the left and right ear demarcations of the model, (b) is the ear region of the model, and (c) is the fusion zone region of the ear of the model;

FIG. 13, ear boundary fusion bands at different bandwidths;

FIG. 14 is a graph showing the engagement effect of the ear with other regions of the model under different weighting values;

FIG. 15 is a comparison graph of the head and face model effects before and after ear optimization;

FIG. 16, texture miss detection results for the back, left and right of the model;

fig. 17 (a) is a schematic diagram of the boundary points and the head region areas defined in the binary mask image, (b) is a two-dimensional projection of the back of the model, and (c) is the back texture completion effect of the model;

FIG. 18, (a) is a small region texture missing in the human face model, (b) and (c) are model nose regions divided based on feature points;

FIG. 19 is a comparison graph of the head and face model effects before and after the texture completion;

fig. 20 is a diagram showing the positive effects of the head and face models of 10 groups of characters generated by the method of the present invention.

Detailed Description

The following description is only exemplary of the invention and should not be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents, and improvements falling within the spirit and scope of the invention.

Step 1: acquiring front, right side, left side and back views of a real human face of a person to be mapped and mask images of the four images;

wherein the person image and the mask image are shown in fig. 2.

Step 2: detecting characteristic points of the face image;

and predicting the human face characteristic points of the front, the right side and the left side views of the human face by utilizing a cascading regression model based on a random forest, and respectively obtaining two-dimensional human face characteristic points in the front, the right side and the left side views. The face feature point definition standard used is a 73-point standard, as shown in fig. 3 (a). In the figure, 16 eye feature points, 12 eyebrow feature points, 15 face contour feature points, 12 outer lip feature points, 6 inner lip feature points, and 12 nose feature points are shown. The obtained face feature point detection is shown in fig. 3 (b).

And step 3: reconstructing a three-dimensional human head and face geometric model;

and (3) according to the two-dimensional characteristic point detection results of the front side view, the right side view and the left side view obtained in the step (2), obtaining a three-dimensional human head and face grid model, and human face posture parameters (a rotation matrix and a translation matrix) and camera parameters corresponding to each view by using a human face low-resolution reconstruction algorithm based on a random image set.

Wherein, the expression of the three-dimensional human head and face model in the world coordinate system is as follows: f _w The expression of the three-dimensional human head and face model under the human face postures corresponding to the front view, the right side view and the left side view is respectively as follows:

F _i ＝R _i ·F _w +T _i (1)

in the formula, F _i I =1,2,3 represents a three-dimensional human head-face model at front, right and left side viewing angles, respectively, as shown in fig. 4, R _i I =1,2,3 and T _i I =1,2,3 is respectively a rotation matrix and a translation matrix under the front, right side and left side views, and the camera parameters corresponding to the front, right side and left side views are respectively f _i ,i＝1,2,3。

And 4, step 4: judging the visible surface of the face grid model;

according to a face mesh model visible surface discrimination algorithm based on light projection, visible surface discrimination is respectively performed on three-dimensional face mesh models under front, right and left side view angles, and invisible surface blanking results under three view angles are respectively obtained, as shown in fig. 5.

The visible face discrimination algorithm process of the face mesh model based on light projection designed by the invention comprises the following steps:

(1) Acquiring a three-dimensional human head and face grid model at a certain visual angle in the front, the right side and the left side as a human head and face grid model at the current visual angle;

if the ray is not intersected with the triangular patch on the model except the incident point, no processing is carried out on the model; if the ray has intersection points with other triangular surfaces on the model except the incident point, the triangular surface patch where other intersection points are located except the incident point is judged to be an invisible surface patch at the current visual angle.

And 5: calculating the texture coordinates of the grid vertex of the model;

respectively blanking the invisible surface under the front, right and left side view angles by using weak perspective projection transformation _i ', i =1,2,3 is projected to a pixel plane coordinate system, and model vertex texture coordinates of the human head-face mesh model in the corresponding view angle image are obtained. Fig. 6 is a projection of the human head-face mesh model after invisible plane blanking in the corresponding view angle image. The specific calculation flow is as follows:

(1) Obtaining an invisible surface blanking model F under a certain viewing angle in the front surface, the right side surface and the left side surface _i ', and camera parameters f corresponding to the view angle image _i ；

wherein dx and dy denote respectively how many millimeters u are represented respectively for each row and each column in the pixel coordinate system ₀ And v ₀ Respectively, representing the pixel coordinates of the center of the image plane.

Step 6: calculating a mesh vertex color matrix of the model;

utilizing a nearest neighbor interpolation algorithm, respectively interpolating the grid vertex color matrix of the calculation model in the corresponding view angle images based on the grid vertex texture coordinates of the human head connected grid model after the invisible surface is blanked in the front, right side and left side views, and obtaining the grid vertex color matrix ColorMatrix I of the human head connected grid model under three view angles _i (I =1,2,3), wherein, colorMatrix _ I _i And (i =1,2, 3) represent model mesh vertex color matrices corresponding to the front, right side, and left side perspective images, respectively. The principle of nearest neighbor interpolation is as follows:

and in four adjacent pixel points of the pixel to be solved, giving the value of the pixel closest to the pixel to be solved. Let (i + Δ u, j + Δ v) be the pixel coordinate to be calculated, where i and j are positive integers, and Δ u and Δ v are decimals greater than 0 and smaller than 1, as shown in fig. 8 (a), then the pixel value f (i + Δ u, j + Δ v) to be calculated is shown in fig. 8 (b). In FIG. 8 (b), if (i + Δ u, j + Δ v) falls in the region A, i.e., Δ u < 0.5, Δ v < 0.5, the value of the top left pixel is assigned to the pixel to be solved; if (i + delta u, j + delta v) falls in the B area, namely delta u is more than 0.5 and delta v is less than 0.5, the value of the pixel at the upper right corner is given to the pixel to be solved; if (i + delta u, j + delta v) falls in the C area, namely delta u is less than 0.5, and delta v is more than 0.5, the value of the lower left corner pixel is given to the pixel to be solved; if (i + Δ u, j + Δ v) falls in the D region, i.e., Δ u > 0.5 and Δ v > 0.5, the value of the bottom-right pixel is assigned to the pixel to be solved.

Fig. 7 is a texture mapping result of the human head and face mesh model after invisible plane blanking in images of corresponding view angles;

and 7: calculating a normal vector matrix of the human head and face grid model;

fig. 9 shows a calculation flow of a normal vector matrix of a human head-face mesh model, where FaceNorm is a normal vector matrix of a triangular patch, and is used to store a normal vector of each triangular patch on the model, faceNum =22800 is the total number of patches of the model, and FaceIntex is the index number of the current triangular patch. The specific calculation flow is as follows:

(4) Computing normal vectors for patches

Wherein the vertex coordinates P of the triangular patch are known ₀ (x ₀ ,y ₀ ,z ₀ ),P ₁ (x ₁ ,y ₁ ,z ₁ ),P ₂ (x ₂ ,y ₂ ,z ₂ ) Normal plane vector of its triangular surface

The calculation process of (c) is as follows:

first, utilize

And

then there are:

wherein:

(6) Repeating operations (1) - (5) until all triangular patches on the mesh model are traversed, i.e., faceIntex =22800.

The calculated mesh model patch matrix FaceNorm is a matrix of 22800 x 3.

And 8: calculating a plane normal vector of the visual angle image;

utilizing the face pose parameters R corresponding to the front, right and left side view images obtained in the step 3 _i I =1,2,3, calculating image plane normal vectors of three view angle images in a camera coordinate system respectively

i =1,2,3, wherein,

the calculation formula for i =1,2,3 is as follows:

and step 9: screening effective texture images and fusing textures;

and calculating included angles between normal vectors of the triangular surface patches on the grid model and plane normal vectors of the three visual angle images, screening effective texture images of each triangular surface patch on the model according to the included angles, and fusing texture values of vertexes on the triangular surface patches on different texture images by taking cosine values of the included angles as weight distribution bases to obtain fused vertex texture values.

The effective texture image screening and texture fusion process designed by the invention comprises the following steps:

a triangular patch normal vector representing a patch index of FaceIntex,

an image plane normal vector representing the ith perspective image.

(2) Setting a Threshold value Threshold when theta _{i_FaceIntex} When the Threshold is more than or equal to, the view image i is an invalid texture image for the facelet FaceIntex;

fig. 10 shows the texture mapping effect of an image on a model when the Threshold values are set to 60, 70, and 80, and tests prove that when Threshold =80, not only most of the less effective textures in texture mapping can be screened out, but also the more effective textures can be retained as much as possible, so the Threshold value Threshold =80 is set in the present invention.

(3) Calculating the fusion weight of the vertex on the triangular patch on different texture images by taking the cosine value of an included angle between the normal vector of the triangular patch on the grid model and the plane normal vector of the three visual angle images as a weight distribution basis;

the weight of the texture image is defined according to the included angle between the normal vector of the triangular patch and the normal vector of the image surface, the weight is inversely proportional to the included angle between the normal vector and the normal vector of the image surface, and the larger the included angle is, the smaller the weight is. The weight of the texture image is

is the weight of the image with sequence number K under the delta faceIntex patch.

Firstly, judging the number of effective texture images of a face mask FaceIntex, and dividing the number into the following three conditions: only one effective texture image is available; two effective texture images are provided; there are three effective texture images.

When there are only 1 valid texture image, the weight of the image is at this time

Then there are:

Then there are:

according to the patch index FaceIntex of the grid model, a vertex index VerIntex on the patch can be obtained _{ΔFaceIntex(p)} (p =1,2,3) and a vertex texture color matrix ColorMatrix _ I on different texture images according to the mesh model vertices _K And calculating the texture value after the top points on the patch are fused, wherein the calculation formula is as follows:

wherein the content of the first and second substances,

as vertex VerIntex _{ΔFaceIntex(p)} The final texture value of (a) is,

Fig. 11 is a comparison graph of the effect of the human head and face model before and after texture fusion, in the figure, the first column is the human head and face model with obvious brightness seam, and the second column is the human head and face model obtained by the texture fusion method of the present invention. As can be seen from the figure, the fused human head and face model obtained by the method of the invention has smooth color transition in most areas, and the problem of seam caused by brightness difference is improved.

Step 10: optimizing ear textures of the model;

the model ear texture optimization method designed by the invention comprises the following steps: the method comprises the steps of firstly dividing left and right ear regions of a human head-face model according to human face characteristic points, resetting fusion weights of front and side view images in the ear regions, optimizing ear textures, performing bidirectional continuation on boundary lines of the ear regions to obtain fusion band regions, distributing certain weights to ear mapping result texture fusion results of the side images to obtain textures of the fusion regions, achieving natural connection of the ear regions and other regions, and finally obtaining the human head-face model after the ear region textures are optimized. The specific process comprises the following steps:

(1) Determining boundary lines of the right ear and the left ear of the model by using the human face characteristic points so as to divide the left ear area and the right ear area of the human head and face model;

the definition of the human face feature points is given in fig. 3 (a), there are 15 face contour feature points, which are assigned with

numbers

1,2,3 and 13, 14, 15, and the coordinates of these feature points in space can be used to determine the boundary lines of the right ear and the left ear as shown in fig. 12 (a), respectively, and the two boundary lines are determined as follows: firstly, coordinates of the 6 feature points are obtained and recorded as R _ Point1, R _ Point2, R _ Point3, L _ Point1, L _ Point2 and L _ Point3, and right ear and left ear boundary lines R _ earLine and L _ earLine are obtained according to the following formulas respectively:

judging three vertexes of the model triangular patch, and recording all triangular patches of which the three vertexes are right from the right ear boundary line as a model right ear region, and obtaining a model left ear region in the same way, as shown in fig. 12 (b).

(3)since obvious connecting traces inevitably appear at the ear boundary lines, the invention carries out bidirectional continuation on the ear region boundary lines to obtain fusion band regions, and the fusion band regions are shown in (c) of FIG. 12, and then assigns weights omega to the side view image mapping results and the texture fusion results obtained in step 9 respectively _e1 And omega _e2 Therefore, the texture of the fusion area is obtained, and the ear area is smoothly connected in the human head and face model.

Assuming that the vertex color matrices of the blend zone of the side view image and the texture blend result in step 9 are ColorMatrix _ Band _ fromSideI and ColorMatrix _ Band _ fromFusion, respectively, colorMatrix _ Band is the vertex color matrix of the final blend zone, the final vertex color matrix of the blend zone can be represented by the following formula:

the test verifies that the fusion bandwidth is selected to be 12 according to the following conditions:

figure 13 shows ear border fusion bands at different bandwidths. In the figure, the red area is a fusion belt area under different bandwidths, the fusion belt area is used for realizing natural connection of the ear area texture and other area textures of the model, if the bandwidth value is too small, smooth transition at the connection part is difficult to realize, if the bandwidth value is too large, the texture effect of other areas can be influenced, especially the eye corner area of the model, therefore, the fusion belt area under different bandwidths is compared, and the fusion belt area is found to be 12, so that the fusion belt value can be ensured not to be too small, and the fusion belt can not extend to the eye corner area of the model;

wherein, the weight omega is selected according to test verification _e1 ＝0.5，ω _e2 =0.5, selected according to:

the texture value in the fused band is determined by the side image and the texture fusion result in step 9, and the weights of the two are ω _e1 And ω _e2 And has ω _e1 +ω _e2 And =1. Texture due to step 9The fusion result is calculated according to the included angle between the normal vector of the patch and the image plane, and the side image is considered. Thus, the weight ω _e1 Too large will make the ratio of the side image effect in the final texture fusion result large, resulting in further reduction of the ratio of other side images and distortion. And from testing, it can be derived that when ω is _e1 After reaching a certain value, the texture effect of the blend zone does not change significantly, as shown in fig. 14. FIG. 14 shows different weight values ω _e1 The effect of the engagement of the lower ear with the other regions of the model, the red box shows the fusion zone region of the right ear of the model. When the weight ω is _e1 When =0, namely before the fusion zone is processed, a joint gap exists at the ear boundary line; when the weight ω is _e1 When the value is not less than 0.5, the connecting gap is obviously improved; when the weight ω is _e1 In the case of =0.7, the joint gap is also improved obviously, but the joint gap is also improved by omega _e1 Compared with the effect of 0.5, the effect is not obviously changed; when omega _e1 If =0.9, a gap is again generated at the joint due to the excessive texture occupation of the side image. Therefore, based on the above analysis, the present invention will select the weight value ω _e1 ＝0.5，ω _e2 ＝0.5；

Fig. 15 is a comparison graph of the human head and face model effects before and after ear optimization, wherein the first column and the second column respectively represent the human head and face model before and after ear region texture optimization, and the ear region of the model is shown in a dotted line frame.

Step 11: carrying out binarization processing on the mask images of the front side, the right side, the left side and the back side views of the real human face of the figure to be mapped in the step 1 to obtain binary mask images of four images;

the method comprises the steps of determining a threshold value by utilizing a graythresh function in MATLAB, searching the threshold value by utilizing the maximum inter-class variance method through the graythresh function, and then carrying out binarization processing on a mask image according to an im2bw function.

Step 12: judging a non-texture region of the model;

by using the texture mapping method described in steps 5-6 of the present invention, a corresponding relationship between the vertices of the face model and the pixels of the binary mask image is established, a vertex color matrix of the model is obtained, and the vertex with a vertex color value of 0 is determined as the vertex lacking the correct texture, and the binary mask image is mapped onto the three-dimensional model, with the result shown in fig. 16. The back, left and right sides of the model are shown in fig. 16, on which black areas are mapped from black areas on the binary mask image, i.e. texture missing areas in the model. Most of the texture missing regions are concentrated on the back of the model, and the small texture missing regions are on the two sides of the neck and the auricle of the model.

Step 13: completing the texture of the large area of the model;

the model large-area texture completion algorithm designed by the invention carries out shape adaptation on the front and back images, thereby establishing the relationship between the human face geometric model and the back image and further completing the back large-area texture completion of the model. The specific process is as follows:

(2) In the two images, white region boundary points B on the boundary line are respectively obtained _f0 ，B _f1 And B _b0 ，B _b1 And obtaining the pixel coordinates (x) of the boundary points _f0 ,y _f0 ),(x _f1 ,y _f1 ),(x _b0 ,y _b0 ),(x _b1 ,y _b1 ) As shown in FIG. 17 (a), there is y for the pixel coordinates of the boundary points _f0 ＝y _f1 ，y _b0 ＝y _b1 ；

(3) According to the boundary lines of the two images, the number of pixel points of the white area above the boundary line, namely the area of the head and the neck area, is respectively calculated and is marked as S _f And S _b ，S _f And S _b As shown by the hatched area in fig. 17 (a);

(4) Construct an Image matrix B _ Image of size (W, H, 3), with each pixel initialized to [0,0 ] by default]. Zooming the back Image B _ Image0 of the person by a zooming coefficient step (scale) to obtain an Image B _ Image1, and calculating the zooming coefficient under B _b0 ，B _b1 Update value B of _b0 '(x _b0 ',y _b0 ')，B _b1 '(x _b1 ',y _b1 '). The scaling factor is calculated as follows:

wherein alpha is _x And alpha _y The calculation formula of (a) is as follows:

(5) And judging out-of-range pixel points in the Image matrix B _ Image2 and deleting the out-of-range pixel points to obtain a final Image B _ Image.

The visible surface of the back face human head face model is a complement of the visible surface of the front face model, and the projection of the back face model is shown in fig. 17 (b).

Fig. 17 (c) shows the effect of complementing the back texture of the human head-face model, and the result shows that the texture of the back region, the neck region, and the pinna region of the model can be complemented using the shape-adapted back image, and the complemented texture performs well in most regions.

Step 14: completing the small-area texture of the model;

after the texture is completed on the back surface of the model of the human head and face, there is also a small region missing texture in the model, as shown in fig. 18 (a).

The model small region texture completion algorithm designed by the invention calculates the normal vector of the triangular patch in the step 7, calculates the cosine value of the included angle between the patch of the texture to be completed and the normal vector of the patch of the nose region, and selects the patch texture with the largest cosine value of the included angle as the triangular patch texture of the texture to be completed, thereby realizing the small region texture completion of the human head and face model. The specific process comprises the following steps:

(1) And dividing the nose region of the face model according to the face characteristic points.

Triangular patches for nose region As shown in FIG. 18 (c), the set of triangular patches for recording nose region is Δ _nose ＝{Δ _ni Ni is not less than 1 but not more than Num _ lose }. Fig. 18 (b) shows the boundary of the nose region based on the feature point connection, and the right figure shows the divided nose model. The boundary of the nose region is a rectangular side line drawn by feature points of

serial numbers

28, 30, 32 and 36 as defined in fig. 3 (a);

Normal vector of all patches in nose region

wherein cos θ _{add_j,ni} Patch delta representing texture to be compensated _{add_j} Triangular patch delta with nose region _ni The cosine value of the included angle.

(5) And (5) repeating the steps (2) to (4) until the vertex texture value of each patch in the texture region to be compensated is calculated.

Fig. 19 is a comparison diagram of the effect of the face model before and after the texture completion, in which the first line shows the face model before the texture completion at three angles, and the second line shows the face model after the texture completion at three angles. As shown in the figure, the back surface of the face model before texture completion has a large-area texture missing phenomenon, and the neck area and the auricle area have a small-area texture missing phenomenon. The model after the texture completion generated by the method completes the texture on the back side of the model, recovers the texture of the neck and the auricle area, and finally generates the human head and face model with complete texture and good reality.

Claims

1. A texture mapping and completion method of a three-dimensional human head and face model is characterized by comprising the following steps: the method comprises the following steps:

and 2, step: detecting the characteristic points of the face image:

predicting the characteristic points of the human face on the front, the right side and the left side of the human face by utilizing a cascade regression model based on a random forest, and respectively obtaining two-dimensional characteristic points of the human face in the front, the right side and the left side;

and step 3: reconstructing a three-dimensional human head and face geometric model:

and 4, step 4: and (3) judging the visible surface of the face mesh model:

and 5: calculating the texture coordinates of the grid vertexes of the model:

step 6: calculating a mesh vertex color matrix of the model:

interpolating and calculating the vertex color value of the mesh model in the corresponding view angle image based on the vertex texture coordinates of the mesh model under the front view angle, the right side view angle and the left side view angle by using a nearest neighbor point interpolation method, and respectively obtaining texture mapping results of the face mesh model under the three view angles;

and step 8: calculating a plane normal vector of the visual angle image:

respectively calculating image plane normal vectors of the three views under a camera coordinate system by using the face posture parameters corresponding to the front side view, the right side view and the left side view;

and step 9: effective texture image screening and texture fusion:

according to whether included angles between normal vectors of face mesh patches and normal vectors of image planes of the front, right and left side views are larger than 80 degrees or not, screening effective texture images of each patch of the mesh model to obtain the number of effective images and image sequence numbers of each patch of the model; then, determining fusion weight distribution basis according to cosine values of included angles between each patch normal vector of the face mesh model and the plane normal vector of the effective texture image, and performing weighted fusion on mesh vertex texture values of the face model at corresponding visual angles according to the fusion weight to obtain mesh vertex texture values after the face mesh model is fused;

the effective texture image is a view image of a certain patch of the grid model, wherein the included angle between the image plane normal vector and the patch normal vector is less than 80 degrees;

step 10: optimizing ear textures of the model;

dividing left and right ear regions of a human head and face model according to human face characteristic points, resetting fusion weights of the front side view images in the ear regions, optimizing ear textures, performing bidirectional continuation on boundary lines of the ear regions to obtain fusion band regions, distributing certain weights to ear mapping result texture fusion results of the side images to obtain textures of the fusion regions, achieving natural connection of the ear regions and other regions, and finally obtaining the human head and face model after the ear region texture optimization;

step 11: carrying out binarization processing on mask images corresponding to the front, right, left and back views, and acquiring a binary mask image of the four side views;

step 12: judging the non-texture region of the model:

mapping the binary mask images of the front view, the right view, the left view and the back view onto a face mesh model to obtain a texture loss detection result of the face mesh model;

step 13: and (3) completing the texture of the large area of the model:

step 14: and (3) completing the small-area texture of the model:

2. A texture mapping and completion method of a three-dimensional human face model as claimed in claim 1, characterized by: in step 2, the adopted human face feature point definition standard is a 73-point standard, which includes: 16 eye characteristic points, 12 eyebrow characteristic points, 15 face contour characteristic points, 12 outer lip characteristic points, 6 inner lip characteristic points and 12 nose characteristic points;

in the step 3, the representation of the three-dimensional human head and face model in the world coordinate system is as follows: f _w The three-dimensional human head and face model is represented under the human face postures corresponding to the front, the right side and the left side view respectively as follows:

F _i ＝R _i ·F _w +T _i (1)

in the formula, F _i I =1,2,3; respectively representing a three-dimensional human head-face model under the visual angles of the front face, the right side face and the left side face; r is _i I =1,2,3 and T _i I =1,2,3; respectively a rotation matrix and a translation matrix under front, right side and left side views; the camera parameters corresponding to the front, right and left side views are respectively f _i ,i＝1,2,3；

In the step 4, the algorithm flow for distinguishing the visible face of the face mesh model is as follows:

(2) Traversing each vertex on the current head and face grid model, taking the model vertex as an incident point to serve as a ray, enabling the direction of the ray to be parallel to the normal vector direction of the current visual angle image plane, and judging the intersection condition of the ray and a triangular surface patch of the grid model;

if the ray is not intersected with the triangular patch on the model except the incident point, no processing is carried out on the model; if the ray has intersection points with other triangular surfaces on the model except the incident point, the triangular surface patch where the other intersection points are positioned except the incident point is judged to be an invisible surface patch under the current visual angle;

(3) Hiding all invisible triangular surface patches on the current head and face grid model;

3. A texture mapping and completion method of a three-dimensional human face model as claimed in claim 2, characterized by: in the step 5, the invisible surface blanking models F under the front, right and left side view angles are respectively processed _i ', i =1,2,3 to a pixel plane coordinate system, and the calculation flow for obtaining model vertex texture coordinates of the human head and face mesh model in the corresponding view angle image is as follows:

(4) Calculating a projection point (x) _{I_vintex} ,y _{I_vintex} ) Texture coordinates (u) _vintex ,v _vintex ) The calculation process is as follows:

(5) Repeating the steps (1) and (4) until the texture coordinates of the model vertexes of the human head-face mesh model in the front view, the right side view and the left side view are all calculated;

in the step 6, based on the mesh vertex texture coordinates of the invisible face blanked human head and face mesh model in the front, right side and left side views, the mesh vertex color matrix of the interpolation calculation model is respectively interpolated in the corresponding view angle images by using a nearest neighbor interpolation algorithm, and the mesh vertex color matrix ColorMatrix _ I of the human head and face mesh model under three view angles is obtained _i (I =1,2,3), wherein, colorMatrix _ I _i (i =1,2, 3) representing model mesh vertex color matrices corresponding to the front, right side, and left side perspective images, respectively;

in step 7, the normal vector matrix of the face mesh model is calculated as follows:

(4) Computing normal vectors for patches

The calculation process of (2) is as follows:

first, utilize

And with

then there are:

4. A texture mapping and completion method of a three-dimensional human face model as claimed in claim 3, characterized by: in step 8, the human face posture parameters R corresponding to the front, right side and left side visual angle images obtained in step 3 are utilized _i I =1,2,3, calculating image plane normal vectors of three view images in a camera coordinate system respectively

i =1,2,3, wherein,

the calculation formula of i =1,2,3 is as follows:

5. a texture mapping and completion method of a three-dimensional human face model according to claim 4, characterized by: in the step 9, the cosine values of the included angles are taken as weight distribution basis, texture values of vertexes on the triangular surface patches obtained on different texture images are fused, and fused vertex texture values are obtained;

the effective texture image screening and texture fusion process is as follows:

(1) The calculation formula of the included angle between the normal vector of the triangular patch on the grid model and the normal vector of the plane of the three visual angle images is as follows:

wherein, theta _{i_FaceIntex} Representing a patch index as FaceIntThe angle between the ex triangular patch and the ith view image, i =1,2,3 represents the frontal, right and left view images respectively,

a triangular patch normal vector representing a patch index of FaceIntex,

an image plane normal vector representing the ith perspective image;

(2) Setting a Threshold value, when theta _{i_FaceIntex} When the Threshold is more than or equal to, the view image i is an invalid texture image for the facelet FaceIntex;

the weight of the texture image is

the weight of the image with the sequence number K under the delta faceIntex patch;

firstly, judging the number of effective texture images of a face mask FaceIntex, and dividing the number into the following three conditions: only one effective texture image is available; two effective texture images are provided; there are three effective texture images;

When the number of the effective texture images is 2, the effective texture images are respectively marked as I _K1 ，I _K2 Of normal vector of patch and image planeRespectively having an included angle of

Then there are:

Then there are:

(4) According to the fusion weight, fusing texture values of vertexes on the triangular patch obtained on different texture images to obtain a fused vertex texture value;

obtaining a vertex index VerIntex on a patch according to the patch index FaceIntex of the grid model _{ΔFaceIntex(p)} (p =1,2,3); and according to the vertex texture color matrix ColorMatrix I of the vertex of the grid model on different texture images _K And calculating the texture value after the top points on the patch are fused, wherein the calculation formula is as follows:

in the formula (I), the compound is shown in the specification,

as vertex VerIntex _{ΔFaceIntex(p)} Is determined by the final texture value of (a),

is vertex VerIntex _{ΔFaceIntex(p)} In ColorMatrix _ I _K The texture value of (1).

6. A texture mapping and completion method of a three-dimensional human face model according to claim 5, characterized by: in the step 10, the ear texture optimization method includes:

(2) Resetting the fusion weight of the left and right visual angle images in the ear region of the human face model to be 1, and obtaining the human head and face model after the fusion weight of the ear region texture is reset;

(3) Performing two-way continuation on the boundary line of the ear region to obtain a fusion zone region, and then respectively allocating weight omega to the side view image mapping result and the texture fusion result obtained in the step 9 _e1 And omega _e2 Therefore, the texture of the fusion area is obtained, and the ear area is smoothly connected in the human head and face model;

7. A texture mapping and completion method of a three-dimensional human face model according to claim 6, characterized by: in step 11, a threshold is determined by using a graythresh function in the MATLAB, the graythresh function finds the threshold by using a maximum inter-class variance method, and then binarization processing is performed on the mask image according to an im2bw function.

8. A texture mapping and completion method of a three-dimensional human face model according to claim 7, characterized by: in the step 12, texture mapping is performed by using the methods in the steps 5 to 6, a corresponding relation between the vertex of the face model and the pixel point of the binary mask image is established, a vertex color matrix of the model is obtained, the vertex with the vertex color value of 0 is judged to be the vertex of the model mesh lacking correct texture, and the binary mask image is mapped to the three-dimensional model.

9. A texture mapping and completion method of a three-dimensional human face model according to claim 8, characterized by: the process of step 13 is:

(3) According to the boundary lines of the two images, the number of pixel points of the white area above the boundary line, namely the area of the head and the neck area, is respectively calculated and is marked as S _f And S _b ，S _f And S _b ；

(4) Construct an Image matrix B _ Image of size (W, H, 3), with each pixel initialized to [0,0 ] by default](ii) a Zooming the back Image B _ Image0 of the person by a zooming coefficient step (scale) to obtain an Image B _ Image1, and calculating the zooming coefficient under B _b0 ，B _b1 Update value B of _b0 '(x _b0 ',y _b0 ')，B _b1 '(x _b1 ',y _b1 ') to a host; the scaling factor is calculated as follows:

wherein alpha is _x And alpha _y The calculation formula of (c) is as follows:

10. A texture mapping and completion method of a three-dimensional human face model as claimed in claim 9, wherein: the process of step 14 is as follows:

(1) According to the face characteristic points, dividing a face model nose area:

the set of triangular patches in the nasal region is recorded as delta _nose ＝{Δ _ni |1≤ni≤Num_nose}；

Normal vector of all patches in nose region