CN114663274A - Portrait image hair removing method and device based on GAN network - Google Patents
Portrait image hair removing method and device based on GAN network Download PDFInfo
- Publication number
- CN114663274A CN114663274A CN202210172409.6A CN202210172409A CN114663274A CN 114663274 A CN114663274 A CN 114663274A CN 202210172409 A CN202210172409 A CN 202210172409A CN 114663274 A CN114663274 A CN 114663274A
- Authority
- CN
- China
- Prior art keywords
- hair
- hidden
- portrait image
- male
- codes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000004209 hair Anatomy 0.000 title claims abstract description 194
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012549 training Methods 0.000 claims abstract description 40
- 238000000926 separation method Methods 0.000 claims abstract description 34
- 230000003287 optical effect Effects 0.000 claims abstract description 24
- 238000012706 support-vector machine Methods 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 10
- 238000005070 sampling Methods 0.000 claims abstract description 7
- 230000001815 facial effect Effects 0.000 claims description 31
- 238000009792 diffusion process Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000007526 fusion splicing Methods 0.000 claims description 2
- 238000007670 refining Methods 0.000 claims 1
- 208000009119 Giant Axonal Neuropathy Diseases 0.000 description 6
- 201000003382 giant axonal neuropathy 1 Diseases 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 238000002156 mixing Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 102100026622 Cartilage intermediate layer protein 1 Human genes 0.000 description 1
- 101000913767 Homo sapiens Cartilage intermediate layer protein 1 Proteins 0.000 description 1
- 230000003042 antagnostic effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a Portrait image hair removing method based on a GAN network, which comprises the following steps: step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hair hidden code-fraction data set; step 2, training through a support vector machine to obtain a hair separation boundary and a gender separation boundary; step 3, obtaining male optical head hidden codes through hair separation boundary editing; step 4, training to obtain a Male HairMapper model for editing the hidden codes of the Male with the hair based on the data in the step 3; step 5, editing the gender separation boundary and a Male HairMapper model to obtain a female optical head hidden code; step 6, training and obtaining a hair removal model for generating a high-quality optical head portrait image based on the data; and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image of the hair to be removed after calculation. The invention also provides a portrait image hair removing device. The method provided by the invention can generate a high-quality optical head portrait image.
Description
Technical Field
The invention relates to the technical field of portrait editing, in particular to a method and a device for removing portrait image hairs based on a GAN network.
Background
Hair is not only an important component of the human body, but also a key element representing individuality and fashion. However, the presence of hair in portrait images poses significant challenges for digital hair styling and three-dimensional reconstruction of human faces. In the aspect of digital hair style design, the new hair style is easily mixed with the old hair by directly overlaying the new hair style on the original drawing, so that problems are caused; replacing an old hair style with a new one requires the use of matting and image completion techniques, both of which are very error prone. For the reconstruction of a three-dimensional face, most of the existing methods cannot process hairs blocking the face, and the hairs still remain in the texture, so that obvious artifacts can be generated in the reconstructed face. This prompted us to develop a hair treatment method that naturally removes hair from the portrait to facilitate practical use.
The traditional method for removing hair generates a training set by modifying and calibrating actual male and female images, but the method is time-consuming and labor-consuming, and the volume of the obtained training set is not large, so that the model of the final training is not good. At the present stage, a scientific researcher adopts hidden codes in a StyleGAN hidden space to represent image information, and then finishes the calibration of an image through a computer, but the StyleGAN hidden space has a problem: the semantic information of the combination of female-optical head is lacked in the hidden space, so that the method can only obtain the hidden code pair of male hair and optical head.
Academic literature interrelating the patent Space of GANs for Semantic Face Editing (In Advances In Neural Information Processing Systems 33(2020): pages 12104 and 12114,2020) discloses a method for establishing separation boundaries In a StyleGAN hidden Space, thereby Editing Face semantics by simple linear combination, but the method cannot ensure that the Face features before and after Editing are unchanged.
Academic documents Text-drive management of Text image (In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pages 2085-2094,2021) disclose a CILP language-image pre-training model for implementing Text-based semantic image operations by dynamically finding the direction of a given Text prompt In a StyleGAN hidden space.
Patent document CN111598762A discloses a generating robust image steganography method, which includes: constructing an image data set, and pre-training the image data set; constructing and initializing a deep learning network architecture; training a deep learning network architecture by adopting a combined-fine tuning method to obtain a network architecture model; and generating a secret-carrying pseudo-graph by using a network architecture model, and carrying out secret communication to finish the image steganography process. The method adopts an image steganography method, and integrates the embedding process of secret information into the image generation process by utilizing a generated antagonistic network StyleGAN, so that the obtained generation type image steganography method has the advantages of large embedding capacity, good generated image quality, strong undetectable secret image statistics, high practicability and the like, and overcomes the problems of poor quality of the secret image generated by the existing generation type image steganography, low embedding capacity, low information extraction accuracy and the like, but the method cannot well process the hair removal problem because the StyleGAN does not have semantic information of a female-optical head combination, and meanwhile, the invention content does not mention how to solve the problem.
Disclosure of Invention
In order to solve the problems, the invention provides a method for removing portrait image hair based on a GAN network, which solves the problem that semantic information of a combination of female and optical heads does not exist in a StyleGAN hidden space through a gender boundary; furthermore, a model is constructed based on the hair hidden code set of the StyleGAN hidden space, so that hair can be removed under the condition that other characteristics of the face are kept unchanged, and the hair in the portrait image can be removed quickly and automatically.
A portrait image hair removing method based on a GAN network comprises the following steps:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the label set into a hair hidden code-fraction data set;
step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit a hidden code of male hair and cannot ensure that the facial features of a portrait image output by the hidden code are not changed, and the gender separation boundary is used for editing the gender corresponding to the hidden code and ensuring that the facial features of the portrait image output by the hidden code are not changed;
step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, and obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images;
step 4, combining the Male bareheaded hidden code obtained in the step 3 and the corresponding Male haired hidden code into a training set, inputting the training set into a pre-constructed Male HairMapper model, and after iterative training, obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged;
step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair;
step 6, the male bald hidden code generated in the step 3 and the corresponding male haired hidden code form a data set, the data set is input into a pre-constructed hair removal model, and after iterative training, a hair removal model for generating a high-quality bald portrait image is obtained;
and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image after the hair is removed after the portrait image is subjected to editing calculation and fusion splicing with the image.
Specifically, the hidden code set in step 1 includes a female hidden code with hair, a male hidden code with hair and a corresponding male hidden code with hair.
Specifically, the hair separation boundary and the gender separation boundary are obtained through training of a support vector machine in the step 2, and the specific steps are as follows:
step 2.1, based on the male hidden codes with hair and the corresponding male hidden codes with optical head in the hair hidden code-fraction data set, training by a support vector machine to obtain hair separation boundaries;
and 2.2, editing the randomly generated hidden codes through StyleFlow software to obtain a gender hidden code-score data set with opposite gender hidden codes of the portrait image, and training through a support vector machine based on the gender hidden code-score data set to obtain a gender separation boundary.
Preferably, the semantic diffusion refinement optimization is to diffuse the facial features in the original portrait image into the edited hidden codes, perform iterative optimization on the target hidden codes, and output pairs of hidden codes with hairs and an optical head when the total loss function reaches the minimum.
Specifically, when the target hidden code is subjected to iterative optimization, the total loss function adopted is specifically:
Ldiffuse=λrecLrec+λperLper
wherein L isrecFor pixel level reconstruction loss, LperFor structure level reconstruction loss, λrecWeight lost for pixel level reconstruction, λperThe lost weights are reconstructed for the structure level.
Preferably, the hair removal model in step 6 is obtained by using the following formula of the target loss function during iterative training:
L=λlLlatent+λhLhair+λfLface+λiLid
wherein L is the total loss, LlatentFor implicit code loss, λlWeights lost for implicit codes, LhairIs the pixel-level loss, λ, of the hair regionhWeight lost for pixels of the hair region, LfaceIs the pixel level loss, λ, of the face regionfWeight lost at pixel level for face region, LidFor loss of facial features, λiThe weight of face feature loss is added with the calculation parameters of hidden code loss and pixel loss on the basis of the traditional loss function formula, so that the quality of the portrait image output by the final model is improved.
Preferably, the hair removal model in step 6 further includes an encoder and an image generator, the encoder is configured to encode the input portrait image to obtain a corresponding hidden code, and obtain a corresponding optical head hidden code after calculation and editing, and the image generator generates the portrait image with hair removed based on the obtained optical head hidden code.
Preferably, the fusing and splicing in step 7 is to perform seamless fusion on the face features of the portrait image to be hair-removed through Poisson editing operation and the portrait image output by the optical head hidden code obtained through editing calculation, so as to obtain the portrait image with hair removed.
The invention also provides a portrait image hair removing device for rapidly removing portrait image hair, comprising:
a computer memory, a computer processor and a computer program stored in and executable on the computer memory, wherein the computer processor performs the portrait image hair removal method described above; the computer processor, when executing the computer program, performs the steps of: inputting the portrait image of the hair to be removed into the portrait image hair removal device, and outputting the portrait image after the hair is removed through calculation.
Compared with the prior art, the invention has the beneficial effects that:
(1) the invention provides a 'female-male-bald' process, thereby solving the problem that semantic information of a female-bald combination does not exist in a StyleGAN hidden space.
(2) Based on the hair hidden code of the StyleGAN hidden space, a portrait image hair removal model is constructed, so that various facial forms with different expressions, postures, ages and sexes can be processed, and a corresponding optical head portrait image with high quality is generated.
Drawings
FIG. 1 is a schematic flow diagram of a portrait image hair removal method according to the present invention;
FIG. 2 is a flow chart of the training process of the Male HairMapper model and the hair removal model in the present embodiment;
FIG. 3 is a flow chart of the hair removing model for removing hair according to the present embodiment;
FIG. 4 is a portrait image of the present embodiment with hair removed;
fig. 5 is a portrait image after hair is removed in this embodiment.
Detailed Description
As shown in fig. 1, a portrait image hair removal method based on GAN network includes:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the labeling label into a hair hidden code-score data set (W, S)hair:
Step 1.1, randomly sampling to obtain 2NwInvisible codeWherein N iswA hidden code constitutes a data set D0:
Calculating NwStandard deviation w of implicit codestdAnd calculating a noise ratio value according to the formula (II):
scalenoise=(0.5wstd)2 (II)
in [0,1 ]]Are generated randomlyMaking noise and using scalenoiseControlling the weight of the noise to obtain a final noise data set Dnoise:
Wherein n isi0,……,ni17Noise added to each layer;
step 1.2, D0And DnoiseThe hidden codes in the system are input into a generator of StyleGAN2-ada to obtain corresponding random sampling portrait images, a hair classifier is trained to score the random sampling portrait images, and the corresponding score of each hidden code is obtained according to the formula (IV):
s=C(g(w+)) (IV)
wherein s is hair score, C is hair classifier, g is generator of StyleGAN2-ada, g (w)+) A portrait image output by the generator, wherein the hair classifier adopts a ResNeXt-50 (32x4D) structure, the portrait image containing hair is graded as 1, the portrait image of the optical head is graded as 0, and a hidden code set D is completed0And DnoiseLabeling with a label with or without hair;
step 1.3, mixing the above D0Hidden code of data set and corresponding hair score constitute a hair hidden code-score data set (W, S)hair:
Step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit the hidden code of male hair, but cannot ensure that the facial features of the hidden code output portrait image are not changed, the gender separation boundary is used for editing the gender corresponding to the hidden code, and simultaneously ensures that the facial features of the hidden code output portrait image are not changed:
step 2.1, based on the hair hidden code-score data set (W, S)hairTraining and outputting a coarse hair by using a support vector machineSeparation boundary bhNormal vector n ofh;
Step 2.2 Generation of N Using StyleFlow stochastic editing according to formula (VII)genderHidden codeObtaining a gender-converted covert code
The method of using the formula is disclosed In academic literature including Attribute-conditioned expansion of quantized-generated images using conditioned connecting normal communicating flows (In ACM Transactions On Graphics (TOG),2021,40(3): 1-21, 2021), and will not be described In detail;
step 2.3, mixingAnd w+Setting the hidden code number of all males to be 1 and setting the hidden code number of all females to be 0 to obtain the hidden code-score data set of gender (W, S)gender:
Step 2.4, hidden code-score data set based on gender (W, S)genderTraining and outputting a sex separation boundary b by using a support vector machinegNormal vector n ofg。
Step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images:
step 3.1, editing hidden codes by using hair separation boundariesTheIs D0And DnoiseThe hidden code of the male portrait image corresponding to the middle is obtained according to the formula (IX)
Wherein n ishSeparating the boundary b for the hairhThe normal vector of (a);
step 3.2, adopting FaceParsing extractionCorresponding original portrait imageMask m of hair regionh;
Step 3.3, based on the original portrait imageAnd an intermediate portrait imageCalculating and outputting prior information according to formula (X)
Wherein an element level multiplication is indicated;
step 3.5, calculating according to the formula (XI) to obtain the pixel level reconstruction loss Lrec:
Step 3.6, calculating and obtaining the structural level reconstruction loss L according to the formula (XII)per:
Wherein φ represents the VGG16 model after training;
step 3.7 Total loss L calculated according to formula (XIII)diffuse:
Ldiffuse=λrecLrec+λperLper(XIII) in which λrecWeight lost for pixel level reconstruction, λperWeights lost for structure level reconstruction;
step 3.8, optimization by continuous iterationSo that the total loss LdiffuseMinimum, get the finalSemantic diffusion resultsHidden code for obtaining portrait image without hair and keeping facial features unchanged
Step 4, combining the Male hidden code obtained in the step 3 and the corresponding Male hidden code with hair to form a training set, inputting the training set into a pre-constructed Male HairMapper model, and after iterative training, obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged:
step 4.1, based on the pair of hair-and-no-hair male crypts obtained in step 3And withObtaining a corresponding hidden code-fraction data set Hm:
Wherein n ismAs a data set HmThe number of crypto-code pairs for middle males;
step 4.2, training a full-connection network Male HairMapper, and recording as Mm,MmEditing the hidden code according to equation (XV):
wherein β is a hyperparameter controlling the weight;
step 4.3, calculating according to the formula (XVI) to obtain the hidden code loss Llatent:
Step 4.4, S44 the pixel level loss L of the hair region is calculated according to formula (XVII)hair:
Step 4.5, S45 obtains the pixel level loss L of the face area by calculation according to the formula (XVIII)face:
Step 4.6, calculating by ArcFace according to the formula (XIX) to obtain the loss L of the facial featuresid:
Wherein, R is an ArcFace network, calculating cosine value;
and 4.7, calculating the total loss L according to the formula (XX):
L=λlLlatent+λhLhair+λfLface+λiLid (XX)
wherein λ islWeight lost for implicit codes, λhWeight of pixel-level loss, λ, for hair regionfWeight lost at pixel level, λ, for face regioniWeights lost for facial features;
step 4.8, optimizing M by continuous iterationmReducing the total loss L to obtain a trained Male HairMapper model; the Male HairMapper model is used to directly edit Male cryptic code to remove hairs in a portrait image and leave facial features in the portrait image unchanged.
Step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair:
step 5.1, editing hidden codes by adopting gender separation boundariesThe above-mentionedIs D0And DnoiseThe corresponding female portrait image hidden code is obtained according to the formula (XXX) and the male intermediate hidden code with the facial pose and the skin color basically unchanged
Wherein n isgSeparating the boundary for sex bgThe normal vector of (a);
step 5.2, hiding the male hidden code with hairInputting the codes into a Male HairMapper model, and calculating to obtain corresponding Male hidden codes with hair removed
step 5.3, adopting FaceParsing extractionCorresponding original female portrait imageMask m of hair regionh;
Step 5.4, based on the original female portrait imageAnd a middle male anophelifuge portrait imageCalculating output prior information according to formula (L)
Wherein an element level multiplication is indicated;
step 5.6, calculating according to the formula (LX) to obtain the pixel level reconstruction loss Lrec:
Step 5.7, calculating according to the formula (LXX) to obtain the structure level reconstruction loss Lper:
Where φ represents the trained VGG16 model;
step 5.8, calculating according to the formula (LXXX) to obtain the total loss Ldiffuse:
Ldiffuse=λrecLrec+λperLper(LXXX) wherein λrecWeight lost for pixel level reconstruction, λperReconstructing the lost weights for the structure level;
step 5.9, optimizing by utilizing continuous iterationSo that the total loss LdiffuseMinimum, obtain the final semantic diffusion result Andform a pair of female crypts with and without hair.
Step 6, hiding the male bald hidden code generated in the step 3 and the corresponding male haired hidden code, and forming a data set by the female bald hidden code generated in the step 5 and the corresponding female haired hidden code, inputting the data set into a pre-constructed hair removal model, and obtaining a hair removal model for generating a high-quality bald portrait image after iterative training:
step 6.1, based on the male covert codes with and without hair in pairs obtained in step 3Andthe female covert code obtained in the step 5 and paired with hair and without hairAndthe composition data set H:
wherein n ismNumber of male covert code pairs, n, in dataset HfNumber of covert code pairs for females;
step 6.2, constructing a hair removal model, which comprises an identification module, a hair removal module, an imaging module and a fusion module:
the identification module comprises an encoder, the encoder is used for encoding an input portrait image to obtain a corresponding hidden code in a StyleGAN hidden space and inputting the hidden code to the hair removal module, wherein the encoder selects a projector provided by design an encoder for Style GAN image management, the projector is the prior art, and the specific operation process is not repeated;
the hair removing module is a fully connected network and is used for editing the hidden hair codes in the portrait image hidden codes into hidden optical head codes and inputting the edited portrait image hidden codes into the imaging module;
the imaging module comprises an image generator, the image generator generates a corresponding optical head portrait image according to an input optical head hidden code, and obtains the portrait image without hair through an image fusion method.
The image fusion method specifically comprises the following steps:
first, a mask m of a hair region of a portrait image X input into a hair removal model is extracted using FaceParsingtest;
Finally, seamlessly fusing the new portrait image with the portrait image needing hair removal through Poisson editing, and calculating to obtain the portrait image X with the hair removedres:
Wherein, P is Poisson edition operation;
and 6.3, performing iterative training on the hair removal model based on the data set H to finally obtain the hair removal model for generating the high-quality optical head portrait image, wherein the training method is consistent with the training method of the Male HairMapper model in the step 4, namely, the detailed description is omitted.
As shown in FIG. 2, a flow chart of the training of the Male HairMapper model and the hair removal model is shown.
As shown in fig. 3, a specific process for removing hair for the hair removal model is as follows: the method comprises the steps of transferring the facial features of an original portrait image to a generated optical head portrait image to finish the hair removal work of the original portrait image, wherein the movement of the facial features in a StyleGAN hidden space is linear work, so that the method is simpler and quicker compared with the movement of the hair features, and meanwhile, the problem that the hair features are lost during movement is avoided, so that a high-quality fused portrait image is obtained.
Step 7, inputting the portrait image of the hair to be removed shown in fig. 4 into a hair removal model, and encoding the input portrait image through a projector provided by a Designing an encoder for style gan image manipulation to obtain a corresponding hidden codeAccording to formula (XCIX)Editing and calculating to obtain corresponding hidden codes for removing hairs
Invisible code for removing hairInput to the image generator, a new portrait image is obtained with no hairs and with other facial features unchanged:
and finally, seamlessly fusing the new portrait image and the portrait image without the hair through Poisson editing on the mask of the portrait image with the hair to be removed, and calculating to obtain the portrait image without the hair as shown in the figure 5.
Claims (9)
1. A portrait image hair removing method based on a GAN network is characterized by comprising the following steps:
step 1, randomly sampling hidden codes in a StyleGAN hidden space to obtain a hidden code set, labeling the hidden codes in the hidden code set to obtain a hair label and an optical head label, and combining the hidden code set and the label set into a hair hidden code-fraction data set;
step 2, obtaining a hair separation boundary and a gender separation boundary through training of a support vector machine, wherein the hair separation boundary can only edit a hidden code of male hair and cannot ensure that the facial features of a portrait image output by the hidden code are not changed, and the gender separation boundary is used for editing the gender corresponding to the hidden code and ensuring that the facial features of the portrait image output by the hidden code are not changed;
step 3, editing the male hidden codes in the head-hair hidden code-fraction data set through a hair separation boundary, and obtaining male hidden codes corresponding to the male hidden codes in the head-hair hidden code-fraction data set by adopting semantic diffusion refinement optimization, wherein the facial features of the male hidden codes output the portrait images are consistent with the facial features of the male hidden codes output the portrait images;
step 4, combining the Male hidden code obtained in the step 3 and the corresponding Male hidden code with hair to form a training set, inputting the training set into a pre-constructed Male HairMapper model, and obtaining a Male HairMapper model which is used for editing the Male hidden code to remove the hair in the portrait image and keeping the facial features of the portrait image unchanged after iterative training;
step 5, converting the female hidden codes with hair in the hair hidden code-fraction data set into Male hidden codes with hair through a gender separation boundary, inputting the converted Male hidden codes with hair into a trained Male HairMapper model to obtain Male hidden codes with hair, and optimizing the Male hidden codes with hair through semantic diffusion refinement to obtain corresponding female hidden codes with hair;
step 6, the male bald hidden code generated in the step 3 and the corresponding male haired hidden code form a data set, the data set is input into a pre-constructed hair removal model, and after iterative training, a hair removal model for generating a high-quality bald portrait image is obtained;
and 7, inputting the portrait image of the hair to be removed into the hair removal model, and outputting the portrait image of the removed hair after editing calculation and image fusion splicing.
2. The method for removing portrait image hair based on GAN network of claim 1, wherein the hidden code set in step 1 includes hidden codes with hair for female, hidden codes with hair for male and corresponding hidden codes with hair for male.
3. The method for removing portrait images based on GAN network as claimed in claim 1, wherein the step 2 obtains the hair separation boundary and the gender separation boundary through the training of a support vector machine, and the specific steps are as follows:
step 2.1, based on the male hidden codes with hair and the corresponding male hidden codes with optical head in the hair hidden code-fraction data set, training by a support vector machine to obtain hair separation boundaries;
and 2.2, editing the randomly generated hidden codes through StyleFlow software to obtain a gender hidden code-score data set with opposite gender hidden codes of the portrait image, and training through a support vector machine based on the gender hidden code-score data set to obtain a gender separation boundary.
4. The method for removing the hairs of the portrait image based on the GAN network as claimed in claim 1, wherein the semantic diffusion refining optimization is to diffuse the facial features in the original portrait image into the edited hidden codes, perform iterative optimization on the target hidden codes, and output pairs of hidden codes with hairs and optical heads when the total loss function reaches the minimum.
5. The method for removing portrait image hair based on GAN network as claimed in claim 4, wherein the total loss function adopted when iteratively optimizing the target hidden code is specifically:
Ldiffuse=λrecLrec+λperLper
wherein L isrecFor pixel level reconstruction loss, LperFor structure level reconstruction loss, λrecWeight lost for pixel level reconstruction, λperThe lost weights are reconstructed for the structure level.
6. The method for removing the portrait image hair based on the GAN network as claimed in claim 1, wherein the hair removal model in the step 6 is iteratively trained by using a target loss function formula as follows:
L=λlLlatent+λhLhair+λfLface+λiLid
wherein L is the total loss, LlatentFor implicit code loss, λlWeights lost for implicit codes, LhairIs the pixel-level loss, λ, of the hair regionhWeight lost for pixels of the hair region, LfaceIs the pixel level loss, λ, of the face regionfWeight lost at pixel level for face region, LidFor loss of facial features, λiIs the weight lost to the facial features.
7. The method for removing the hair of the portrait image based on the GAN network as claimed in claim 1, wherein the hair removal model in step 6 further comprises an encoder and an image generator, the encoder is configured to encode the input portrait image to obtain a corresponding hidden code, and obtain a corresponding hidden code after calculation and editing, and the image generator generates the portrait image with the hair removed based on the obtained hidden code.
8. The method for removing the hairs of the portrait image based on the GAN network as claimed in claim 1, wherein the fusing and splicing in the step 7 is to seamlessly fuse the facial features of the portrait image to be hair-removed with the portrait image output by the optical head hidden code obtained by editing and calculating through Poisson editing operation, so as to obtain the portrait image with the hairs removed.
9. A portrait image hair removal device comprising a computer memory, a computer processor and a computer program stored in and executable on the computer memory, wherein the computer processor performs the GAN network-based portrait image hair removal method of any of claims 1-8; the computer processor, when executing the computer program, performs the steps of: inputting the portrait image of the hair to be removed into the portrait image hair removal device, and outputting the portrait image after the hair is removed through calculation.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210172409.6A CN114663274A (en) | 2022-02-24 | 2022-02-24 | Portrait image hair removing method and device based on GAN network |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210172409.6A CN114663274A (en) | 2022-02-24 | 2022-02-24 | Portrait image hair removing method and device based on GAN network |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114663274A true CN114663274A (en) | 2022-06-24 |
Family
ID=82027718
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210172409.6A Pending CN114663274A (en) | 2022-02-24 | 2022-02-24 | Portrait image hair removing method and device based on GAN network |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114663274A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115810215A (en) * | 2023-02-08 | 2023-03-17 | 科大讯飞股份有限公司 | Face image generation method, device, equipment and storage medium |
CN117115295A (en) * | 2023-09-28 | 2023-11-24 | 北京数字力场科技有限公司 | Face texture generation method, electronic equipment and computer storage medium |
-
2022
- 2022-02-24 CN CN202210172409.6A patent/CN114663274A/en active Pending
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115810215A (en) * | 2023-02-08 | 2023-03-17 | 科大讯飞股份有限公司 | Face image generation method, device, equipment and storage medium |
CN117115295A (en) * | 2023-09-28 | 2023-11-24 | 北京数字力场科技有限公司 | Face texture generation method, electronic equipment and computer storage medium |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Song et al. | Geometry-aware face completion and editing | |
Guo et al. | Image inpainting via conditional texture and structure dual generation | |
US11880766B2 (en) | Techniques for domain to domain projection using a generative model | |
CN114663274A (en) | Portrait image hair removing method and device based on GAN network | |
CN112541864A (en) | Image restoration method based on multi-scale generation type confrontation network model | |
CN111814566A (en) | Image editing method, image editing device, electronic equipment and storage medium | |
Ren et al. | Two-stage sketch colorization with color parsing | |
CN111402394B (en) | Three-dimensional exaggerated cartoon face generation method and device | |
CN113034355B (en) | Portrait image double-chin removing method based on deep learning | |
CN112686816A (en) | Image completion method based on content attention mechanism and mask code prior | |
CN112801914A (en) | Two-stage image restoration method based on texture structure perception | |
CN111161405A (en) | Three-dimensional reconstruction method for animal hair | |
Li et al. | Uphdr-gan: Generative adversarial network for high dynamic range imaging with unpaired data | |
CN115049556A (en) | StyleGAN-based face image restoration method | |
CN116188912A (en) | Training method, device, medium and equipment for image synthesis model of theme image | |
Peng et al. | Difffacesketch: High-fidelity face image synthesis with sketch-guided latent diffusion model | |
Mei et al. | Deep supervised image retargeting | |
CN113516604B (en) | Image restoration method | |
DE102021124537A1 (en) | ENERGY-BASED VARIATIONAL AUTOENCODER | |
Tan et al. | Style2talker: High-resolution talking head generation with emotion style and art style | |
CN117593178A (en) | Virtual fitting method based on feature guidance | |
CN112241708A (en) | Method and apparatus for generating new person image from original person image | |
CN117557683A (en) | Novel image generation method driven by text and semantic segmentation map together | |
CN116740281A (en) | Three-dimensional head model generation method, three-dimensional head model generation device, electronic equipment and storage medium | |
CN116152060A (en) | Double-feature fusion guided depth image super-resolution reconstruction method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |