CN114663274A - Portrait image hair removing method and device based on GAN network - Google Patents

Portrait image hair removing method and device based on GAN network Download PDF

Info

Publication number
CN114663274A
CN114663274A CN202210172409.6A CN202210172409A CN114663274A CN 114663274 A CN114663274 A CN 114663274A CN 202210172409 A CN202210172409 A CN 202210172409A CN 114663274 A CN114663274 A CN 114663274A
Authority
CN
China
Prior art keywords
hair
hidden
portrait image
male
codes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210172409.6A
Other languages
Chinese (zh)
Inventor
吴奕谦
金小刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202210172409.6A priority Critical patent/CN114663274A/en
Publication of CN114663274A publication Critical patent/CN114663274A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a Portrait image hair removing method based on a GAN network, which comprises the following steps: step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hair hidden code-fraction data set; step 2, training through a support vector machine to obtain a hair separation boundary and a gender separation boundary; step 3, obtaining male optical head hidden codes through hair separation boundary editing; step 4, training to obtain a Male HairMapper model for editing the hidden codes of the Male with the hair based on the data in the step 3; step 5, editing the gender separation boundary and a Male HairMapper model to obtain a female optical head hidden code; step 6, training and obtaining a hair removal model for generating a high-quality optical head portrait image based on the data; and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image of the hair to be removed after calculation. The invention also provides a portrait image hair removing device. The method provided by the invention can generate a high-quality optical head portrait image.

Description

Portrait image hair removing method and device based on GAN network
Technical Field
The invention relates to the technical field of portrait editing, in particular to a method and a device for removing portrait image hairs based on a GAN network.
Background
Hair is not only an important component of the human body, but also a key element representing individuality and fashion. However, the presence of hair in portrait images poses significant challenges for digital hair styling and three-dimensional reconstruction of human faces. In the aspect of digital hair style design, the new hair style is easily mixed with the old hair by directly overlaying the new hair style on the original drawing, so that problems are caused; replacing an old hair style with a new one requires the use of matting and image completion techniques, both of which are very error prone. For the reconstruction of a three-dimensional face, most of the existing methods cannot process hairs blocking the face, and the hairs still remain in the texture, so that obvious artifacts can be generated in the reconstructed face. This prompted us to develop a hair treatment method that naturally removes hair from the portrait to facilitate practical use.
The traditional method for removing hair generates a training set by modifying and calibrating actual male and female images, but the method is time-consuming and labor-consuming, and the volume of the obtained training set is not large, so that the model of the final training is not good. At the present stage, a scientific researcher adopts hidden codes in a StyleGAN hidden space to represent image information, and then finishes the calibration of an image through a computer, but the StyleGAN hidden space has a problem: the semantic information of the combination of female-optical head is lacked in the hidden space, so that the method can only obtain the hidden code pair of male hair and optical head.
Academic literature interrelating the patent Space of GANs for Semantic Face Editing (In Advances In Neural Information Processing Systems 33(2020): pages 12104 and 12114,2020) discloses a method for establishing separation boundaries In a StyleGAN hidden Space, thereby Editing Face semantics by simple linear combination, but the method cannot ensure that the Face features before and after Editing are unchanged.
Academic documents Text-drive management of Text image (In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pages 2085-2094,2021) disclose a CILP language-image pre-training model for implementing Text-based semantic image operations by dynamically finding the direction of a given Text prompt In a StyleGAN hidden space.
Patent document CN111598762A discloses a generating robust image steganography method, which includes: constructing an image data set, and pre-training the image data set; constructing and initializing a deep learning network architecture; training a deep learning network architecture by adopting a combined-fine tuning method to obtain a network architecture model; and generating a secret-carrying pseudo-graph by using a network architecture model, and carrying out secret communication to finish the image steganography process. The method adopts an image steganography method, and integrates the embedding process of secret information into the image generation process by utilizing a generated antagonistic network StyleGAN, so that the obtained generation type image steganography method has the advantages of large embedding capacity, good generated image quality, strong undetectable secret image statistics, high practicability and the like, and overcomes the problems of poor quality of the secret image generated by the existing generation type image steganography, low embedding capacity, low information extraction accuracy and the like, but the method cannot well process the hair removal problem because the StyleGAN does not have semantic information of a female-optical head combination, and meanwhile, the invention content does not mention how to solve the problem.
Disclosure of Invention
In order to solve the problems, the invention provides a method for removing portrait image hair based on a GAN network, which solves the problem that semantic information of a combination of female and optical heads does not exist in a StyleGAN hidden space through a gender boundary; furthermore, a model is constructed based on the hair hidden code set of the StyleGAN hidden space, so that hair can be removed under the condition that other characteristics of the face are kept unchanged, and the hair in the portrait image can be removed quickly and automatically.
A portrait image hair removing method based on a GAN network comprises the following steps:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the label set into a hair hidden code-fraction data set;
step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit a hidden code of male hair and cannot ensure that the facial features of a portrait image output by the hidden code are not changed, and the gender separation boundary is used for editing the gender corresponding to the hidden code and ensuring that the facial features of the portrait image output by the hidden code are not changed;
step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, and obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images;
step 4, combining the Male bareheaded hidden code obtained in the step 3 and the corresponding Male haired hidden code into a training set, inputting the training set into a pre-constructed Male HairMapper model, and after iterative training, obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged;
step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair;
step 6, the male bald hidden code generated in the step 3 and the corresponding male haired hidden code form a data set, the data set is input into a pre-constructed hair removal model, and after iterative training, a hair removal model for generating a high-quality bald portrait image is obtained;
and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image after the hair is removed after the portrait image is subjected to editing calculation and fusion splicing with the image.
Specifically, the hidden code set in step 1 includes a female hidden code with hair, a male hidden code with hair and a corresponding male hidden code with hair.
Specifically, the hair separation boundary and the gender separation boundary are obtained through training of a support vector machine in the step 2, and the specific steps are as follows:
step 2.1, based on the male hidden codes with hair and the corresponding male hidden codes with optical head in the hair hidden code-fraction data set, training by a support vector machine to obtain hair separation boundaries;
and 2.2, editing the randomly generated hidden codes through StyleFlow software to obtain a gender hidden code-score data set with opposite gender hidden codes of the portrait image, and training through a support vector machine based on the gender hidden code-score data set to obtain a gender separation boundary.
Preferably, the semantic diffusion refinement optimization is to diffuse the facial features in the original portrait image into the edited hidden codes, perform iterative optimization on the target hidden codes, and output pairs of hidden codes with hairs and an optical head when the total loss function reaches the minimum.
Specifically, when the target hidden code is subjected to iterative optimization, the total loss function adopted is specifically:
Ldiffuse＝λrecLrec+λperLper
wherein L isrecFor pixel level reconstruction loss, LperFor structure level reconstruction loss, λrecWeight lost for pixel level reconstruction, λperThe lost weights are reconstructed for the structure level.
Preferably, the hair removal model in step 6 is obtained by using the following formula of the target loss function during iterative training:
L＝λlLlatent+λhLhair+λfLface+λiLid
wherein L is the total loss, LlatentFor implicit code loss, λlWeights lost for implicit codes, LhairIs the pixel-level loss, λ, of the hair regionhWeight lost for pixels of the hair region, LfaceIs the pixel level loss, λ, of the face regionfWeight lost at pixel level for face region, LidFor loss of facial features, λiThe weight of face feature loss is added with the calculation parameters of hidden code loss and pixel loss on the basis of the traditional loss function formula, so that the quality of the portrait image output by the final model is improved.
Preferably, the hair removal model in step 6 further includes an encoder and an image generator, the encoder is configured to encode the input portrait image to obtain a corresponding hidden code, and obtain a corresponding optical head hidden code after calculation and editing, and the image generator generates the portrait image with hair removed based on the obtained optical head hidden code.
Preferably, the fusing and splicing in step 7 is to perform seamless fusion on the face features of the portrait image to be hair-removed through Poisson editing operation and the portrait image output by the optical head hidden code obtained through editing calculation, so as to obtain the portrait image with hair removed.
The invention also provides a portrait image hair removing device for rapidly removing portrait image hair, comprising:
a computer memory, a computer processor and a computer program stored in and executable on the computer memory, wherein the computer processor performs the portrait image hair removal method described above; the computer processor, when executing the computer program, performs the steps of: inputting the portrait image of the hair to be removed into the portrait image hair removal device, and outputting the portrait image after the hair is removed through calculation.
Compared with the prior art, the invention has the beneficial effects that:
(1) the invention provides a 'female-male-bald' process, thereby solving the problem that semantic information of a female-bald combination does not exist in a StyleGAN hidden space.
(2) Based on the hair hidden code of the StyleGAN hidden space, a portrait image hair removal model is constructed, so that various facial forms with different expressions, postures, ages and sexes can be processed, and a corresponding optical head portrait image with high quality is generated.
Drawings
FIG. 1 is a schematic flow diagram of a portrait image hair removal method according to the present invention;
FIG. 2 is a flow chart of the training process of the Male HairMapper model and the hair removal model in the present embodiment;
FIG. 3 is a flow chart of the hair removing model for removing hair according to the present embodiment;
FIG. 4 is a portrait image of the present embodiment with hair removed;
fig. 5 is a portrait image after hair is removed in this embodiment.
Detailed Description
As shown in fig. 1, a portrait image hair removal method based on GAN network includes:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the labeling label into a hair hidden code-score data set (W, S)hair：
Step 1.1, randomly sampling to obtain 2NwInvisible code
Figure BDA0003518856660000071
Wherein N iswA hidden code constitutes a data set D0：
Figure BDA0003518856660000072
Calculating NwStandard deviation w of implicit codestdAnd calculating a noise ratio value according to the formula (II):
scalenoise＝(0.5wstd)2 (II)
in [0,1 ]]Are generated randomlyMaking noise and using scalenoiseControlling the weight of the noise to obtain a final noise data set Dnoise：
Figure BDA0003518856660000073
Wherein n isi0,……,ni17Noise added to each layer;
step 1.2, D0And DnoiseThe hidden codes in the system are input into a generator of StyleGAN2-ada to obtain corresponding random sampling portrait images, a hair classifier is trained to score the random sampling portrait images, and the corresponding score of each hidden code is obtained according to the formula (IV):
s＝C(g(w+)) (IV)
wherein s is hair score, C is hair classifier, g is generator of StyleGAN2-ada, g (w)+) A portrait image output by the generator, wherein the hair classifier adopts a ResNeXt-50 (32x4D) structure, the portrait image containing hair is graded as 1, the portrait image of the optical head is graded as 0, and a hidden code set D is completed0And DnoiseLabeling with a label with or without hair;
step 1.3, mixing the above D0Hidden code of data set and corresponding hair score constitute a hair hidden code-score data set (W, S)hair：
Figure BDA0003518856660000081
Step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit the hidden code of male hair, but cannot ensure that the facial features of the hidden code output portrait image are not changed, the gender separation boundary is used for editing the gender corresponding to the hidden code, and simultaneously ensures that the facial features of the hidden code output portrait image are not changed:
step 2.1, based on the hair hidden code-score data set (W, S)hairTraining and outputting a coarse hair by using a support vector machineSeparation boundary bhNormal vector n ofh；
Step 2.2 Generation of N Using StyleFlow stochastic editing according to formula (VII)genderHidden code
Figure BDA0003518856660000082
Obtaining a gender-converted covert code
Figure BDA0003518856660000083
Figure BDA0003518856660000084
The method of using the formula is disclosed In academic literature including Attribute-conditioned expansion of quantized-generated images using conditioned connecting normal communicating flows (In ACM Transactions On Graphics (TOG),2021,40(3): 1-21, 2021), and will not be described In detail;
step 2.3, mixing
Figure BDA0003518856660000091
And w+Setting the hidden code number of all males to be 1 and setting the hidden code number of all females to be 0 to obtain the hidden code-score data set of gender (W, S)gender：
Figure BDA0003518856660000092
Wherein, the first and the second end of the pipe are connected with each other,
Figure BDA0003518856660000093
to mean
Figure BDA0003518856660000094
Or w+；
Step 2.4, hidden code-score data set based on gender (W, S)genderTraining and outputting a sex separation boundary b by using a support vector machinegNormal vector n ofg。
Step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images:
step 3.1, editing hidden codes by using hair separation boundaries
Figure BDA0003518856660000095
The
Figure BDA0003518856660000096
Is D0And DnoiseThe hidden code of the male portrait image corresponding to the middle is obtained according to the formula (IX)
Figure BDA0003518856660000097
Figure BDA0003518856660000098
Wherein n ishSeparating the boundary b for the hairhThe normal vector of (a);
step 3.2, adopting FaceParsing extraction
Figure BDA0003518856660000099
Corresponding original portrait image
Figure BDA00035188566600000910
Mask m of hair regionh；
Step 3.3, based on the original portrait image
Figure BDA00035188566600000911
And an intermediate portrait image
Figure BDA00035188566600000912
Calculating and outputting prior information according to formula (X)
Figure BDA00035188566600000913
Figure BDA00035188566600000914
Wherein an element level multiplication is indicated;
step 3.4, mixing
Figure BDA0003518856660000101
As hidden codes to be optimized
Figure BDA0003518856660000102
An initial value of (1);
step 3.5, calculating according to the formula (XI) to obtain the pixel level reconstruction loss Lrec：
Figure BDA0003518856660000103
Step 3.6, calculating and obtaining the structural level reconstruction loss L according to the formula (XII)per：
Figure BDA0003518856660000104
Wherein φ represents the VGG16 model after training;
step 3.7 Total loss L calculated according to formula (XIII)diffuse：
Ldiffuse＝λrecLrec+λperLper(XIII) in which λrecWeight lost for pixel level reconstruction, λperWeights lost for structure level reconstruction;
step 3.8, optimization by continuous iteration
Figure BDA0003518856660000105
So that the total loss LdiffuseMinimum, get the finalSemantic diffusion results
Figure BDA0003518856660000106
Hidden code for obtaining portrait image without hair and keeping facial features unchanged
Figure BDA0003518856660000107
Step 4, combining the Male hidden code obtained in the step 3 and the corresponding Male hidden code with hair to form a training set, inputting the training set into a pre-constructed Male HairMapper model, and after iterative training, obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged:
step 4.1, based on the pair of hair-and-no-hair male crypts obtained in step 3
Figure BDA0003518856660000108
And with
Figure BDA0003518856660000109
Obtaining a corresponding hidden code-fraction data set Hm：
Figure BDA00035188566600001010
Wherein n ismAs a data set HmThe number of crypto-code pairs for middle males;
step 4.2, training a full-connection network Male HairMapper, and recording as Mm，MmEditing the hidden code according to equation (XV):
Figure BDA0003518856660000111
wherein β is a hyperparameter controlling the weight;
step 4.3, calculating according to the formula (XVI) to obtain the hidden code loss Llatent：
Figure BDA0003518856660000112
Step 4.4, S44 the pixel level loss L of the hair region is calculated according to formula (XVII)hair：
Figure BDA0003518856660000113
Step 4.5, S45 obtains the pixel level loss L of the face area by calculation according to the formula (XVIII)face：
Figure BDA0003518856660000114
Step 4.6, calculating by ArcFace according to the formula (XIX) to obtain the loss L of the facial featuresid：
Figure BDA0003518856660000115
Wherein, R is an ArcFace network, calculating cosine value;
and 4.7, calculating the total loss L according to the formula (XX):
L＝λlLlatent+λhLhair+λfLface+λiLid (XX)
wherein λ islWeight lost for implicit codes, λhWeight of pixel-level loss, λ, for hair regionfWeight lost at pixel level, λ, for face regioniWeights lost for facial features;
step 4.8, optimizing M by continuous iterationmReducing the total loss L to obtain a trained Male HairMapper model; the Male HairMapper model is used to directly edit Male cryptic code to remove hairs in a portrait image and leave facial features in the portrait image unchanged.
Step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair:
step 5.1, editing hidden codes by adopting gender separation boundaries
Figure BDA0003518856660000121
The above-mentioned
Figure BDA0003518856660000122
Is D0And DnoiseThe corresponding female portrait image hidden code is obtained according to the formula (XXX) and the male intermediate hidden code with the facial pose and the skin color basically unchanged
Figure BDA0003518856660000123
Figure BDA0003518856660000124
Wherein n isgSeparating the boundary for sex bgThe normal vector of (a);
step 5.2, hiding the male hidden code with hair
Figure BDA0003518856660000125
Inputting the codes into a Male HairMapper model, and calculating to obtain corresponding Male hidden codes with hair removed
Figure BDA0003518856660000126
Figure BDA0003518856660000127
Wherein the hidden code
Figure BDA0003518856660000128
And
Figure BDA0003518856660000129
in contrast, hair was removed, but facial features were changed;
step 5.3, adopting FaceParsing extraction
Figure BDA00035188566600001210
Corresponding original female portrait image
Figure BDA00035188566600001211
Mask m of hair regionh；
Step 5.4, based on the original female portrait image
Figure BDA00035188566600001212
And a middle male anophelifuge portrait image
Figure BDA00035188566600001213
Calculating output prior information according to formula (L)
Figure BDA00035188566600001214
Figure BDA00035188566600001215
Wherein an element level multiplication is indicated;
step 5.5, mixing
Figure BDA00035188566600001216
As hidden codes to be optimized
Figure BDA00035188566600001217
An initial value of (1);
step 5.6, calculating according to the formula (LX) to obtain the pixel level reconstruction loss Lrec：
Figure BDA00035188566600001218
Step 5.7, calculating according to the formula (LXX) to obtain the structure level reconstruction loss Lper：
Figure BDA0003518856660000131
Where φ represents the trained VGG16 model;
step 5.8, calculating according to the formula (LXXX) to obtain the total loss Ldiffuse：
Ldiffuse＝λrecLrec+λperLper(LXXX) wherein λrecWeight lost for pixel level reconstruction, λperReconstructing the lost weights for the structure level;
step 5.9, optimizing by utilizing continuous iteration
Figure BDA0003518856660000132
So that the total loss LdiffuseMinimum, obtain the final semantic diffusion result
Figure BDA0003518856660000133
Figure BDA0003518856660000134
And
Figure BDA0003518856660000135
form a pair of female crypts with and without hair.
Step 6, hiding the male bald hidden code generated in the step 3 and the corresponding male haired hidden code, and forming a data set by the female bald hidden code generated in the step 5 and the corresponding female haired hidden code, inputting the data set into a pre-constructed hair removal model, and obtaining a hair removal model for generating a high-quality bald portrait image after iterative training:
step 6.1, based on the male covert codes with and without hair in pairs obtained in step 3
Figure BDA0003518856660000136
And
Figure BDA0003518856660000137
the female covert code obtained in the step 5 and paired with hair and without hair
Figure BDA0003518856660000138
And
Figure BDA0003518856660000139
the composition data set H:
Figure BDA00035188566600001310
wherein n ismNumber of male covert code pairs, n, in dataset HfNumber of covert code pairs for females;
step 6.2, constructing a hair removal model, which comprises an identification module, a hair removal module, an imaging module and a fusion module:
the identification module comprises an encoder, the encoder is used for encoding an input portrait image to obtain a corresponding hidden code in a StyleGAN hidden space and inputting the hidden code to the hair removal module, wherein the encoder selects a projector provided by design an encoder for Style GAN image management, the projector is the prior art, and the specific operation process is not repeated;
the hair removing module is a fully connected network and is used for editing the hidden hair codes in the portrait image hidden codes into hidden optical head codes and inputting the edited portrait image hidden codes into the imaging module;
the imaging module comprises an image generator, the image generator generates a corresponding optical head portrait image according to an input optical head hidden code, and obtains the portrait image without hair through an image fusion method.
The image fusion method specifically comprises the following steps:
first, a mask m of a hair region of a portrait image X input into a hair removal model is extracted using FaceParsingtest；
Then, tomtestPerforming dilation and blur operation to obtain a blurred edge mask
Figure BDA0003518856660000141
Finally, seamlessly fusing the new portrait image with the portrait image needing hair removal through Poisson editing, and calculating to obtain the portrait image X with the hair removedres：
Figure BDA0003518856660000142
Wherein, P is Poisson edition operation;
and 6.3, performing iterative training on the hair removal model based on the data set H to finally obtain the hair removal model for generating the high-quality optical head portrait image, wherein the training method is consistent with the training method of the Male HairMapper model in the step 4, namely, the detailed description is omitted.
As shown in FIG. 2, a flow chart of the training of the Male HairMapper model and the hair removal model is shown.
As shown in fig. 3, a specific process for removing hair for the hair removal model is as follows: the method comprises the steps of transferring the facial features of an original portrait image to a generated optical head portrait image to finish the hair removal work of the original portrait image, wherein the movement of the facial features in a StyleGAN hidden space is linear work, so that the method is simpler and quicker compared with the movement of the hair features, and meanwhile, the problem that the hair features are lost during movement is avoided, so that a high-quality fused portrait image is obtained.
Step 7, inputting the portrait image of the hair to be removed shown in fig. 4 into a hair removal model, and encoding the input portrait image through a projector provided by a Designing an encoder for style gan image manipulation to obtain a corresponding hidden code
Figure BDA0003518856660000151
According to formula (XCIX)
Figure BDA0003518856660000152
Editing and calculating to obtain corresponding hidden codes for removing hairs
Figure BDA0003518856660000153
Figure BDA0003518856660000154
Invisible code for removing hair
Figure BDA0003518856660000155
Input to the image generator, a new portrait image is obtained with no hairs and with other facial features unchanged:
Figure BDA0003518856660000156
and finally, seamlessly fusing the new portrait image and the portrait image without the hair through Poisson editing on the mask of the portrait image with the hair to be removed, and calculating to obtain the portrait image without the hair as shown in the figure 5.

Claims (9)

1. A portrait image hair removing method based on a GAN network is characterized by comprising the following steps:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the label set into a hair hidden code-fraction data set;
step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit a hidden code of male hair and cannot ensure that the facial features of a portrait image output by the hidden code are not changed, and the gender separation boundary is used for editing the gender corresponding to the hidden code and ensuring that the facial features of the portrait image output by the hidden code are not changed;
step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, and obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images;
step 4, combining the Male hidden code obtained in the step 3 and the corresponding Male hidden code with hair to form a training set, inputting the training set into a pre-constructed Male HairMapper model, and obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged after iterative training;
step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair;
step 6, the male bald hidden code generated in the step 3 and the corresponding male haired hidden code form a data set, the data set is input into a pre-constructed hair removal model, and after iterative training, a hair removal model for generating a high-quality bald portrait image is obtained;
and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image of the removed hair after editing calculation and image fusion splicing.
2. The method for removing portrait image hair based on GAN network of claim 1, wherein the hidden code set in step 1 includes hidden codes with hair for female, hidden codes with hair for male and corresponding hidden codes with hair for male.
3. The method for removing portrait images based on GAN network as claimed in claim 1, wherein the step 2 obtains the hair separation boundary and the gender separation boundary through the training of a support vector machine, and the specific steps are as follows:
step 2.1, based on the male hidden codes with hair and the corresponding male hidden codes with optical head in the hair hidden code-fraction data set, training by a support vector machine to obtain hair separation boundaries;
and 2.2, editing the randomly generated hidden codes through StyleFlow software to obtain a gender hidden code-score data set with opposite gender hidden codes of the portrait image, and training through a support vector machine based on the gender hidden code-score data set to obtain a gender separation boundary.
4. The method for removing the hairs of the portrait image based on the GAN network as claimed in claim 1, wherein the semantic diffusion refining optimization is to diffuse the facial features in the original portrait image into the edited hidden codes, perform iterative optimization on the target hidden codes, and output pairs of hidden codes with hairs and optical heads when the total loss function reaches the minimum.
5. The method for removing portrait image hair based on GAN network as claimed in claim 4, wherein the total loss function adopted when iteratively optimizing the target hidden code is specifically:
Ldiffuse＝λrecLrec+λperLper
wherein L isrecFor pixel level reconstruction loss, LperFor structure level reconstruction loss, λrecWeight lost for pixel level reconstruction, λperThe lost weights are reconstructed for the structure level.
6. The method for removing the portrait image hair based on the GAN network as claimed in claim 1, wherein the hair removal model in the step 6 is iteratively trained by using a target loss function formula as follows:
L＝λlLlatent+λhLhair+λfLface+λiLid
wherein L is the total loss, LlatentFor implicit code loss, λlWeights lost for implicit codes, LhairIs the pixel-level loss, λ, of the hair regionhWeight lost for pixels of the hair region, LfaceIs the pixel level loss, λ, of the face regionfWeight lost at pixel level for face region, LidFor loss of facial features, λiIs the weight lost to the facial features.
7. The method for removing the hair of the portrait image based on the GAN network as claimed in claim 1, wherein the hair removal model in step 6 further comprises an encoder and an image generator, the encoder is configured to encode the input portrait image to obtain a corresponding hidden code, and obtain a corresponding hidden code after calculation and editing, and the image generator generates the portrait image with the hair removed based on the obtained hidden code.
8. The method for removing the hairs of the portrait image based on the GAN network as claimed in claim 1, wherein the fusing and splicing in the step 7 is to seamlessly fuse the facial features of the portrait image to be hair-removed with the portrait image output by the optical head hidden code obtained by editing and calculating through Poisson editing operation, so as to obtain the portrait image with the hairs removed.
9. A portrait image hair removal device comprising a computer memory, a computer processor and a computer program stored in and executable on the computer memory, wherein the computer processor performs the GAN network-based portrait image hair removal method of any of claims 1-8; the computer processor, when executing the computer program, performs the steps of: inputting the portrait image of the hair to be removed into the portrait image hair removal device, and outputting the portrait image after the hair is removed through calculation.
CN202210172409.6A 2022-02-24 2022-02-24 Portrait image hair removing method and device based on GAN network Pending CN114663274A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210172409.6A CN114663274A (en) 2022-02-24 2022-02-24 Portrait image hair removing method and device based on GAN network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210172409.6A CN114663274A (en) 2022-02-24 2022-02-24 Portrait image hair removing method and device based on GAN network

Publications (1)

Publication Number Publication Date
CN114663274A true CN114663274A (en) 2022-06-24

Family

ID=82027718

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210172409.6A Pending CN114663274A (en) 2022-02-24 2022-02-24 Portrait image hair removing method and device based on GAN network

Country Status (1)

Country Link
CN (1) CN114663274A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115810215A (en) * 2023-02-08 2023-03-17 科大讯飞股份有限公司 Face image generation method, device, equipment and storage medium
CN117115295A (en) * 2023-09-28 2023-11-24 北京数字力场科技有限公司 Face texture generation method, electronic equipment and computer storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115810215A (en) * 2023-02-08 2023-03-17 科大讯飞股份有限公司 Face image generation method, device, equipment and storage medium
CN117115295A (en) * 2023-09-28 2023-11-24 北京数字力场科技有限公司 Face texture generation method, electronic equipment and computer storage medium

Similar Documents

Publication Publication Date Title
Song et al. Geometry-aware face completion and editing
Guo et al. Image inpainting via conditional texture and structure dual generation
US11880766B2 (en) Techniques for domain to domain projection using a generative model
CN114663274A (en) Portrait image hair removing method and device based on GAN network
CN112541864A (en) Image restoration method based on multi-scale generation type confrontation network model
CN111814566A (en) Image editing method, image editing device, electronic equipment and storage medium
Ren et al. Two-stage sketch colorization with color parsing
CN111402394B (en) Three-dimensional exaggerated cartoon face generation method and device
CN113034355B (en) Portrait image double-chin removing method based on deep learning
CN112686816A (en) Image completion method based on content attention mechanism and mask code prior
CN112801914A (en) Two-stage image restoration method based on texture structure perception
CN111161405A (en) Three-dimensional reconstruction method for animal hair
Li et al. Uphdr-gan: Generative adversarial network for high dynamic range imaging with unpaired data
CN115049556A (en) StyleGAN-based face image restoration method
CN116188912A (en) Training method, device, medium and equipment for image synthesis model of theme image
Peng et al. Difffacesketch: High-fidelity face image synthesis with sketch-guided latent diffusion model
Mei et al. Deep supervised image retargeting
CN113516604B (en) Image restoration method
DE102021124537A1 (en) ENERGY-BASED VARIATIONAL AUTOENCODER
Tan et al. Style2talker: High-resolution talking head generation with emotion style and art style
CN117593178A (en) Virtual fitting method based on feature guidance
CN112241708A (en) Method and apparatus for generating new person image from original person image
CN117557683A (en) Novel image generation method driven by text and semantic segmentation map together
CN116740281A (en) Three-dimensional head model generation method, three-dimensional head model generation device, electronic equipment and storage medium
CN116152060A (en) Double-feature fusion guided depth image super-resolution reconstruction method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination