CN112541966A

CN112541966A - Face replacement method based on reconstruction and network generation

Info

Publication number: CN112541966A
Application number: CN202011425921.4A
Authority: CN
Inventors: 谭晓阳; 蒋珂
Original assignee: Nanjing University of Aeronautics and Astronautics
Current assignee: Nanjing University of Aeronautics and Astronautics
Priority date: 2020-12-09
Filing date: 2020-12-09
Publication date: 2021-03-23

Abstract

The invention discloses a face replacement method based on reconstruction and network generation, which comprises the following steps: inputting the source face image and the target face image into an azimuth mapping network to carry out 3D face reconstruction, and adjusting the posture of the source face to the posture of the target face; generating human mouth details by using the mouth detail generating network; and removing the whole artifacts and noise of the human face by using the global sharpening generation network, thereby obtaining a human face replacement image result which meets the requirements that human eyes cannot easily identify and an algorithm cannot easily identify. The method can generate a vivid face replacement result on the basis of only one pair of the source face image and the target face image, improves the efficiency of the face replacement method, is suitable for generating a large number of face replacement results with different identities, and provides a large number of antagonistic samples for a face counterfeiting detection algorithm.

Description

Face replacement method based on reconstruction and network generation

Technical Field

The invention belongs to the field of automatic control, and particularly relates to a face replacement method based on reconstruction and network generation.

Background

Face replacement (face swap) is a popular research topic in face editing. This document focuses primarily on high quality face replacement for identity replacement. The high-quality face replacement means that human eyes and a computer discriminator algorithm cannot easily recognize a broken face replacement image, and the face replacement for identity replacement means that the identity of a person in a source portrait is transferred to a person in a target portrait and other characteristics of the target portrait are maintained unchanged.

The mainstream face replacement method at present is a deep fakes method, which mainly has the following ideas: the face replacement based on the convolutional network is a face replacement method based on deep learning, a series of convolutional networks are designed to solve various problems required to be faced when the face is replaced, such as posture, skin color, illumination and the like, but the expression of a face replacement result depends on the task design of the networks, and a forged trace is obvious due to poor design. The other DeepFakes method mainly replaces the face based on the self-coding model with a main one, firstly designs two pairs of self-coder structures to respectively perform self-supervision training on a source portrait set and a target portrait set, and then tests the two decoders in an exchange mode, and has larger improvement in performance compared with the prior method, particularly a DeepFaceLab method introduced with a GAN network can generate more vivid counterfeiting results, but the identity retentivity can be seen from some experimental samples, and the DeepFakes method needs a large amount of source portraits and target portraits for training due to the algorithm design, and needs to respectively train different source identities.

Another face replacement method is face replacement based on 3D face reconstruction, but its forged trace is still more obvious due to its lack of processing for post-3D reconstruction mapping operation.

The method of the deep fakes class is inefficient in generating a face replacement result, and the method of the 3D face reconstruction class lacks effective processing of a map.

Disclosure of Invention

The purpose of the invention is as follows: aiming at the problems in the prior art, the face replacement method based on the reconstruction and generation network is high in quality and efficiency and is suitable for generating a large-scale confrontation sample set.

The technical scheme is as follows: in order to solve the above technical problem, the present invention provides a face replacement method based on reconstruction and network generation, comprising the following steps:

(1) and inputting the source face image and the target face image into a position mapping network to reconstruct the 3D face.

(2) And (3) adjusting the pose of the source face model in the step (1) to the pose of the target face.

(3) And (3) generating the human mouth details by using the mouth details generating network according to the result of the step (2).

(4) And (4) removing the whole artifacts and noise of the human face by using the global sharpening generation network on the result of the step (3), thereby obtaining a human face replacement image result which meets the requirements that human eyes cannot easily identify and an algorithm cannot easily identify.

Further, the face replacement method based on reconstruction and network generation is characterized in that: the training step of generating the network through the mouth details in the step (3) is as follows:

and (3.1) establishing a least square countermeasure generation network to extract the discrimination error.

And (3.2) establishing a VGG network to extract the reconstruction error.

And (3.3) extracting the mouth posture features of the face image output in the step (2).

(3.3) during training, the generator inputs the mouth pose feature of (3.3), the discriminator inputs the output of the generator and the mouth image of the target face image, and the VGG structure inputs the output of the generator and the mouth image of the target face image. Training the network by adopting the discrimination error in (3.1) and the reconstruction error in (3.2), wherein the loss function is as follows:

where LD is the discriminator Loss, LG is the generator Loss, and the V (x) function represents the output of x through the VGG structure, with the resulting Loss called the Content Loss (Content Loss), and the gamma parameter representing the coefficient of the countervailing Loss (additive Loss).

Further, the face replacement method based on reconstruction and network generation is characterized in that: the training step of generating the network through the global sharpening in the step (4) is as follows:

and (4.1) establishing a two-times countermeasure generation network to extract the discrimination error.

And (4.2) establishing a VGG network to extract the reconstruction error.

And (4.3) establishing a translator network for translating the face image into a corresponding semantic image for extracting semantic errors.

And (4.4) carrying out 3D face reconstruction on the original image, and carrying out the operations of the steps (1), (2) and (3) on the original image to obtain an original image.

During training, the generator input is the original image in (4.4), the discriminator input is the generator output and the original image in (4.4), the translator network input is the generator output and the original image in (4.4), and the VGG network input is the generator output and the original image in (4.4). Training the network by adopting the discrimination error in (4.1), the reconstruction error in (4.2) and the semantic error in (4.3), wherein the loss function is as follows:

where LD is the discriminator penalty, LG is the generator penalty, and V (x) represents the content penalty of x, P (x) represents the result of x going through PR-Net, and T (x) represents the result of the translator output, the penalty of which is referred to herein as the semantic penalty. The coefficients α, β are coefficients of the countermeasure loss and the reconstruction loss, respectively.

The method is high in efficiency and quality and based on a 3D face Reconstruction method, and organically combines a position mapping network in a 3D face Reconstruction algorithm with a generation countermeasure network for processing the details of a map, and is called a Reconstruction-generation network (EGNet). The new structure can not only generate a face replacement result with the quality equivalent to that of a deep fakes method, but also has the high efficiency characteristic based on a 3D face reconstruction method: high quality face replacement results can be generated with only one pair of source and target face images, and without retraining.

Compared with the prior art, the invention has the advantages that:

1. the deep faces type face replacement method, for example, deep faces lab needs retraining aiming at different source face images, but the method does not need retraining aiming at different source face images, so that the efficiency is greatly improved, and the method is more suitable for generating diversified large-scale data.

2. A3D face reconstruction type face replacement method, such as a Nirkin et al method, is poor in processing of a mapping, and a reconstruction-generation network mode of the method inherits the high efficiency of the 3D face reconstruction type method and simultaneously makes up the defects of mapping processing, particularly loss of details of a mouth.

3. The prior method mainly meets the requirement of human vision, and the method also considers the requirement of algorithm identification, so that the mainstream human face replacement detection method is difficult to identify the result of the method.

Drawings

FIG. 1 is a general flow diagram of a reconstruct-generate network schema in an exemplary embodiment;

FIG. 2 is a schematic diagram of a network for generating details of a mouth in an embodiment;

fig. 3 is a schematic structural diagram of a global sharpening generation network in a specific embodiment.

Detailed Description

The invention is further elucidated with reference to the drawings and the detailed description. The described embodiments of the present invention are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by a person of ordinary skill in the art without any creative effort belong to the protection scope of the present invention.

As shown in fig. 1-3, the human face replacement method based on reconstruction and generation network according to the present invention is characterized in that: a simple and efficient reconstruction-generation network mode is designed, and the new method has the advantages of no need of multiple training, low sample size requirement, high generated result quality and the like.

The reestablishing-generating network pattern comprises the steps of:

(1) and inputting the source face image and the target face image into a PR-Net (PRN) orientation mapping network to carry out 3D face reconstruction, and adjusting the posture of the source face model to the posture of the target face.

(2) And (2) generating human Mouth details on the basis of the result of the step (1) by using a Mouth detail generating network (Mouth-GAN), so as to solve the problem of loss of the Mouth details in the 3D face reconstruction method. And (3) training the network by adopting the discrimination error and the reconstruction error in the step (3.1), wherein the loss function is as follows:

(3) And (3) removing the whole human face artifact and noise by using a global sharpening generation network (Sharpen-GAN) on the result of the step (2), thereby obtaining a human face replacement image result which meets the requirements that human eyes cannot be easily identified and an algorithm cannot be easily identified. During training, the generator input is the original image, the discriminator input is the generator output and the original image, the translator network input is the generator output and the original image, and the VGG network input is the generator output and the original image. And training the network by adopting the discrimination error, the reconstruction error and the semantic error, wherein the loss function is as follows:

The trained model of each module only needs to be tested by using a generator, and the test indexes are as follows: structural Similarity (SSIM), Posi Error (PE), Skin color Error (Skin Error, SE), and Identity Distance (Identity Distance). Next, the evaluation index will be explained. Four indexes which can measure the quality of the face replacement image are summarized: the SSIM, the attitude error, the skin color error and the identity gap comprehensively evaluate the counterfeiting quality of the forged image. SSIM (structural similarity) is a common indicator used to measure image quality. The SSIM index evaluates whether the forged image has distortion or not and the occurrence of noise compared with a target portrait; and the other three are evaluation indexes provided by the text: the attitude error evaluates whether the attitudes of the foreground image and the background image in the forged image are consistent, the skin color error evaluates whether the skin colors of the foreground image and the background image in the forged image are consistent and whether an obvious forged boundary exists, and the identity difference evaluates whether the forged image successfully transfers the identity of the source portrait so as to achieve the aim of replacing the identity. These indices will be used in subsequent experiments to measure the performance of the face replacement algorithm.

If the Pose of the person in the face replacement result is different from the Pose of the target portrait, a visual hole is caused. Therefore, the attitude error is provided to measure the result of the attitude adjustment of the algorithm. Defining the distance D between two figures as the pose error is shown below,

wherein I₁,I₂Two human face images to be measured are respectively, wherein gamma is used for controlling the coefficient in the horizontal direction or the vertical direction, and delta (x) represents that difference calculation is carried out on the x indexes of the two images.

Skin color Error (SE) source and target portraits produce poor results if the Skin color is different without processing. A large number of samples indicate that the difference in skin tone is mainly concentrated in the forehead portion. The pixel F of the sampled face and the pixel T of the forehead portion are compared to obtain a skin color error θ, which is shown below,

wherein

And

the mean value of the face pixel F and the forehead portion pixel T is expressed, and the norm function expresses that they are normalized.

Identity Distance (ID) since the primary purpose of the face replacement method described herein is Identity replacement, the method is considered invalid if the Identity of a person cannot be effectively replaced. An index is therefore needed to measure the retention of the Identity, i.e. the Identity Distance (Identity Distance) between the source portrait and the replacement result. Inspired by a Face-Net method of a Face recognition method, the measurement method carries out regression on each Face image through a depth network, maps the regression to an Euclidean space, and meets the following properties:

if the identity of two figures is the same, then their corresponding points will be as close as possible to

If the identities of two figures are different, then their corresponding points will be as far away as possible

The objective function of the deep network is as follows,

wherein theta is a target parameter of the depth network, D is a distribution formed by all face images, D represents the distribution of a certain identity face image, x and y are samples of the face images, and gamma is a parameter for adjusting two emphasis. Therefore, the trained deep network can measure the face replacement result and the identity distance of the source face. In an experiment, the network is a model trained on VGGFace2, and 1.1 is selected as a threshold value through a large number of tests, namely if the identity distance exceeds 1.1, the face replacement algorithm is not considered to be poor in effect, and if the identity distance is smaller than 1.1, the face replacement algorithm is considered to be good in effect.

A comparison of the present process with the mainstream process is shown in table 1.

Table 1 comparison of performance indicators for the method herein and the deep fake method and Nirkin et al; wherein the index labeled "↓" indicates the higher the better, and the index labeled "↓" indicates the lower the better

From the above results, the method has a great advantage over the mainstream method in terms of the four indexes of face replacement. Especially SSIM for measuring image quality, the method uses global sharpening to generate network, so that the image quality is improved more. Besides, the posture accuracy, the identity retention and the replacement efficiency of the 3D face reconstruction method are inherited, and the generated image quality is equivalent to that of the deep fakes method. Therefore, the method has certain application value and prospect.

Claims

1. A face replacement method based on reconstruction and network generation is characterized by comprising the following steps:

(1) inputting the source face image and the target face image into a position mapping network to carry out 3D face reconstruction;

(2) adjusting the pose of the source face model in the step (1) to the pose of the target face;

(3) generating human mouth details on the result of the step (2) by using the mouth detail generating network;

2. The face replacement method based on reconstruction and generation network as claimed in claim 1, wherein: the specific steps for generating the details of the human mouth in the step (3) are as follows:

(3.1) establishing a least square countermeasure generation network for extracting a discrimination error;

(3.2) establishing a VGG network for extracting reconstruction errors;

(3.3) extracting mouth posture features of the face image output in the step (2);

(3.3) during training, the generator inputs the mouth pose feature of (3.3), the discriminator inputs the output of the generator and the mouth image of the target face image, and the VGG structure inputs the output of the generator and the mouth image of the target face image; training the network by adopting the discrimination error in (3.1) and the reconstruction error in (3.2), wherein the loss function is as follows:

where LD is the discriminator Loss, LG is the generator Loss, and the V (x) function represents the output of x through the VGG structure, with the resulting Loss called the Content Loss, and the gamma parameter representing the coefficient to combat the additive Loss.

3. The face replacement method based on reconstruction and generation network as claimed in claim 1, wherein: the specific steps of generating the network by using the global sharpening in the step (4) are as follows:

(4.1) establishing a second-product confrontation generation network to extract the discrimination error;

(4.2) establishing a VGG network to extract reconstruction errors;

(4.3) establishing a translator network for translating the face image into a corresponding semantic image for extracting semantic errors;

(4.4) carrying out 3D face reconstruction on the original image to obtain an original image;

wherein LD is discriminant loss, LG is generator loss, V (x) function represents content loss of x, P (x) function represents result of x passing through PR-Net, T (x) function represents result of translator output, and its generated loss is referred to as semantic loss; the coefficients α, β are coefficients of the countermeasure loss and the reconstruction loss, respectively.