CN114385883A - Contour enhancement method for approximately simulating wrinkle method in style conversion - Google Patents

Contour enhancement method for approximately simulating wrinkle method in style conversion Download PDF

Info

Publication number
CN114385883A
CN114385883A CN202111483200.3A CN202111483200A CN114385883A CN 114385883 A CN114385883 A CN 114385883A CN 202111483200 A CN202111483200 A CN 202111483200A CN 114385883 A CN114385883 A CN 114385883A
Authority
CN
China
Prior art keywords
edge
landscape
gradient
translation
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111483200.3A
Other languages
Chinese (zh)
Other versions
CN114385883B (en
Inventor
彭先霖
彭盛霖
胡琦瑶
彭进业
王佳欣
刘鑫煜
范建平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwest University
Original Assignee
Northwest University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwest University filed Critical Northwest University
Priority to CN202111483200.3A priority Critical patent/CN114385883B/en
Publication of CN114385883A publication Critical patent/CN114385883A/en
Application granted granted Critical
Publication of CN114385883B publication Critical patent/CN114385883B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration using histogram techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a contour enhancement method for approximately simulating a wrinkle method in style migration. The method develops an unsupervised and contour-enhanced image style migration frame based on CycleGAN, pays attention to lines, sets constraint conditions and highlights edge characteristics according to the characteristics of landscape paintings, and realizes style conversion from landscape photos to landscape paintings. A gradient conversion branch is added in a CycleGAN network, and a gradient guiding method is designed to effectively guide the style conversion from a photo to gradient information in drawing. According to the visual characteristics of human eyes, it is necessary to selectively enhance the edge information in the image, that is, to retain the strong edge with a relatively large contrast ratio, rather than the weak edge with a small contrast ratio.

Description

Contour enhancement method for approximately simulating wrinkle method in style conversion
Technical Field
The invention belongs to the technical field of computer vision, relates to style migration, and particularly relates to a contour enhancement method for approximately simulating a wrinkle method in style migration.
Background
Recently, AI (artificial intelligence) applications, which convert landscape photos into landscape pictures, have become a popular point of interest for many artistic enthusiasts and cultural relic protection. The artificial intelligence technology has the effects of extracting typical characteristics reflecting the style of the artistic works and simulating painting, and is beneficial to training painting skills, innovating the artistic works and inheriting and protecting the traditional art.
The landscape photo is converted into a landscape painting, and is essentially the migration of the image style. GAN is proposed by Goodfellow et al as an effective method for implementing image style migration, and is gradually becoming the mainstream method for image style migration. GAN is powerful and has many practical applications, such as generating high quality images, generating images from text, converting images from one style to another, etc. Many widely popular GAN-based architectures have emerged, such as DCGAN, BigGAN, StarGAN, and the like.
Although these methods can be applied to the conversion of landscape photos into landscape paintings, there are still some unique problems to be solved.
Due to the particularity of the painting materials and the extensive use of various drawing methods, penmanship and ink methods, the creation process of landscape paintings is complicated. The main contents of landscape paintings are mountains, stones and trees, and painters often use a chap method to express the contents. This makes "wrinkle" become the important expression language of landscape painting, becomes the artistic form of painter's aesthetic experience and aesthetic expression. These "wrinkles" are mainly represented by lines, which morphologically mimic the rough contours of objects and scenery. Therefore, when automatically generating landscape paintings from landscape photographs, strengthening the outlines of various objects to imitate the "chap" effect is a challenging subject to be studied in depth.
Disclosure of Invention
In view of the defects in the prior art, an object of the present invention is to provide a contour enhancement method for approximating a wrinkle method in style migration, so as to solve the technical problem in the prior art that the style conversion effect from landscape photos to landscape photos needs to be further improved.
In order to solve the technical problems, the invention adopts the following technical scheme:
a contour enhancement method for approximately simulating a wrinkle method in style migration adopts a contour enhancement model for approximately simulating the wrinkle method in style migration, wherein the contour enhancement model for approximately simulating the wrinkle method in style migration is constructed based on a CycleGAN model; the method for constructing the contour enhancement model for approximately simulating the wrinkle method in the style migration comprises the following steps of:
respectively using keywords of landscape painting and landscape photo, capturing and downloading related pictures in a search engine through a crawler program, and obtaining landscape photos of a source domain and landscape painting of a target domain;
step two, irrelevant images are eliminated through screening, and screened landscape photos and landscape paintings are reserved; storing a part of landscape photos in the source domain as training samples in a tranA folder, and storing the rest of landscape photos in the source domain as test samples in a testA folder; storing a part of landscape paintings in the target domain as training samples in a tranB folder, and storing the rest landscape paintings in the target domain as test samples in a testB folder;
step three, all the images are adjusted to be 256 multiplied by 256 resolution, the step four is carried out, and training iteration is carried out on the cycleGAN model;
defining the landscape picture as a domain A and the landscape picture as a domain B; by utilizing the characteristic of cycle consistency, the network is designed into forward translation and backward translation, and the forward translation and the backward translation both comprise two branches: a drawing translation branch and a gradient translation branch;
step five, forward translation is carried out, and landscape photos in the domain A are converted into landscape pictures, namely GABA → B; for GAB: a → B mapping, this process can be described as expression 1 and expression 2:
recB_edge＝GAB_edge(fakeA_edge) Formula 1;
recB＝Fuse(GAB(fakeA)，recB_edge) Formula 2;
in the formula:
recB_edgeis an image gradient map;
recBthe method comprises the following steps of (1) obtaining an original landscape painting;
GAB_edgea generator for the gradient translation branch in the forward translation process;
GABto convert landscape photos into landscape pictures;
fuse is fusion;
fakeA_edgeenhancing a gradient map for the edge of the landscape photo;
fakeAthe final edge enhancement map is a landscape photo;
drawing translation branch real scene photo realAAs a generator GABThe generated image enhancement features are used as the input of a fusion module; the gradient translation branch converts the real image gradient map realA_edgeAs a generator GBAThe generated edge enhanced gradient map fakeB_edgeAnd true edge-enhanced gradient map real in reverse translationB_edgeIs input to a discriminator DB_edgeJudging the authenticity; simultaneous fakeB_edgeSplicing with the output characteristics of the image translation branch, sending the output characteristics into a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fakeB(ii) a Generated fakeBWill match the true edge enhancement graph realBReverse translating the two together and inputting the two into the certificatePin DBPerforming authenticity identification; the process is described by equations 3 and 4:
fakeB_edge＝(GAB_edge(realA_edge) Formula 3);
fakeB＝Fuse(GAB(realA)，fakeB_edge) Formula 4;
in the formula:
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeBdrawing a final edge enhancement image for the landscape;
GAB_edgea generator for the gradient translation branch in the forward translation process;
realA_edgeenhancing a gradient map for the real edge of the landscape photo;
fakeB_eagedrawing an edge enhancement gradient map for the mountains and waters;
the corresponding generative opposition loss also comes from the gradient panning branch and the image panning branch, which are represented as equations 5 and 6;
Figure BDA0003396264710000041
Figure BDA0003396264710000042
in the formula:
LGAN_B_edgegenerating a penalty function for the gradient translation branch;
LGAN_Bgenerating a penalty function for the image translation branch;
DBa landscape painting discriminator which is an image translation branch;
DB_edgea landscape painting discriminator which is a gradient translation branch;
e is a coefficient;
Figure BDA0003396264710000043
a discriminator loss function that is an image translation branch;
Figure BDA0003396264710000044
a discriminator loss function that is a gradient panning branch;
Figure BDA0003396264710000045
a generator loss function that is an image translation branch;
Figure BDA0003396264710000046
a generator loss function that is a gradient translation branch;
realBdrawing a real picture for the mountains and the waters;
realB_edgedrawing a real edge enhancement gradient map for the mountains and waters;
step six, the original landscape picture needs to be converted back from the landscape painting to realize the cycle consistency, namely GBAMapping of (2); generated edge enhancement map fakeBGenerator G input to a drawing translation branchBAPerforming the following steps; obtaining a group of repaired image characteristics as input of a fusion module for repairing; then the generated edge enhancement gradient map fakeB_edgeGenerator G input to gradient translation branchBA_edgeIn (1), an image gradient map rec for restoration is generatedA_edge(ii) a Finally, recA_edgeThe color drawing translation branches spliced to the restored image features are sent to a fusion module to guide gradient information to restore the original scene photo recA(ii) a The process can be described as formula 7 and formula 8;
recA_edge＝GBA_edge(fakeB_edge) Formula 7;
recA＝Fuse(GBA(fakeB)，recA_edge) Formula 8;
in the formula:
recA_edgeis an image gradient map;
recAis the original windA scene photo;
GBA_edgea generator for the gradient translation branch in the reverse translation process;
GBAto convert landscape pictures into landscape pictures;
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeABdrawing a final edge enhancement image for the landscape;
the cycle consistency loss of the restored image gradient map and the real image gradient map is set, so that the gradient conversion branch can provide more accurate gradient information for the image conversion branch, and the image is restored; meanwhile, the consistency of the restored scene photo and the real scene photo needs to be ensured; the L1 distance loss defines the cyclic consistency loss of these two parts as equation 9 and equation 10;
Lcyc_A_edge(GAB_edge，GBA_edge)＝E[||recA_edge-realA_edge||1]formula 9;
Lcyc_A(GAB，GBA)＝E[||recA-realA||1]formula 10;
in the formula:
Lcyc_A_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Athe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
e is the difference between the input value and the forward prediction;
to this end, the forward translation process has been completed;
step seven, reverse translation is the reverse process of forward translation; converting the landscape picture back to the original landscape picture by reverse translation, i.e. GBA: b → A; reverse translation is through GBAConverting landscape painting into landscape photo, and then passing through GABConverting the landscape picture into an original landscape painting; therefore, the invention realizes the conversion from the scene photo to the scene painting; the invention then requires a switch from landscape to original, to achieve cyclic consistency, i.e. GBAMapping of (2); for GBA: b → A mapping, which can be described as equation 11 and equation 12;
fakeA_edge＝(GBA_edge(realB_edge) Formula 11;
fakeA＝Fuse(BBA(realB)，fakeA_edge) Formula 12;
the corresponding generative antagonism loss functions are defined as formula 13 and formula 14;
Figure BDA0003396264710000061
Figure BDA0003396264710000062
in the formula:
LGAN_A_edgegenerating a penalty function for the gradient translation branch;
LGAN_Agenerating a penalty function for the image translation branch;
DAa landscape discriminator which is an image translation branch;
DA_edgea landscape discriminator being a gradient translation branch;
Figure BDA0003396264710000063
a discriminator loss function that is an image translation branch;
Figure BDA0003396264710000064
a discriminator loss function that is a gradient panning branch;
Figure BDA0003396264710000065
a generator loss function that is an image translation branch;
Figure BDA0003396264710000066
a generator loss function that is a gradient translation branch;
realAthe landscape is photographed with a real picture;
realA_edgeenhancing a gradient map for the landscape photo of the real edge;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
Lcyc_B_edge(GBA_edge，GAB_edge)＝E[||recB_edge-realB_edge||1]formula 15;
Lcyc_B(GBA，GAB)＝E[||recB-realB||1]formula 16;
in the formula:
Lcyc_B_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Bthe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
the overall objective loss function of the model is equation 17:
Figure BDA0003396264710000071
in the formula:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weight of the generator versus resistance loss and cyclic consistency loss.
Compared with the traditional cycleGAN method in the prior art, the method has the following technical effects:
(I) the method develops an unsupervised and contour-enhanced image style migration frame based on CycleGAN, pays attention to lines, sets constraint conditions and highlights edge characteristics according to the characteristics of landscape paintings, and realizes style conversion from landscape photos to landscape paintings.
(II) adding a gradient conversion branch in a CycleGAN network, and designing a gradient guiding method to effectively guide the style conversion of the gradient information from the photo to the drawing. According to the visual characteristics of human eyes, it is necessary to selectively enhance the edge information in the image, that is, to retain the strong edge with a relatively large contrast ratio, rather than the weak edge with a small contrast ratio.
(III) in order to enhance the style of the landscape, the invention uses an edge detection operator to extract the strong edges of the gray level image. Sobel operator is used as the edge detection operator. The operator introduces a similar local averaging operation which smoothes out the noise and removes its effect, which has a good detection effect on rough edges. The characteristic meets the requirement of the method on the simulation effect of the landscape painting.
Drawings
FIG. 1 is a schematic diagram of a CE-cycleGAN network structure.
FIG. 2 is a photograph of the landscape after screening in the source domain.
FIG. 3 is a landscape picture after screening in the target domain.
Figure 4 is a NIMA score histogram of 195 test images.
Fig. 5 is the first five images that outperform the CycleGAN method.
Fig. 6 is five worst images compared to the CycleGAN method.
FIG. 7 is the first five images that are superior to the U-GAT-IT method.
FIG. 8 is the five worst images compared to the U-GAT-IT method.
The present invention will be explained in further detail with reference to examples.
Detailed Description
In the invention, the landscape painting is a Chinese landscape painting.
CycleGAN refers to an image style transformation framework.
CE-cycleGAN, i.e., Contour-Enhanced cycleGAN, unsupervised and Contour-Enhanced image style transformation framework.
The entire network with gradient translation branches is called CE-CycleGAN.
The CE-CycleGAN model is an unsupervised and outline-enhanced image style conversion frame model, namely an outline enhancement model which approximates wrinkle in style migration and is described in the invention.
U-GAT-IT refers to that on the basis of CycleGAN, an end-to-end weak supervision image cross-domain conversion model is constructed by proposing a new attention mechanism and combining a new regularization mode.
NIMA refers to a neural network based on the latest depth object recognition, which can predict the distribution of human evaluation opinions on images from the point of direct sense (technical point of view) and the degree of attraction (aesthetic point of view). NIMA may score an image by generating a score histogram for any image-i.e., scoring the image 1-10 points and comparing to images of the same subject.
FID refers to the mean and covariance matrix of the 2048-dimensional feature vector set output from the two image sets of Incepton Net-V3.
Preferably, NIMA is cited from Talebi, H.and P.Milanfar, NIMA: Neural Image Association, IEEE Transactions on Image Processing,2017: p.1-1.
As a preference, U-GAT-IT is cited from Kim, J., et al, U-GAT-IT: unused general Adaptive Networks with Adaptive Layer-Instance Normalization for Image-to-Image transformation.2019.
As a preference, the FID is cited from DC Dowson, Landau B V.the Frechet Distance between Multivariant Normal Distributions [ J ]. Journal of Multivariant Analysis,1982,12(3): 450-.
The present invention is not limited to the following embodiments, and all equivalent changes based on the technical solutions of the present invention fall within the protection scope of the present invention.
Example (b):
the embodiment provides a contour enhancement method for approximately simulating a wrinkle method in style migration, which adopts a contour enhancement model for approximately simulating a wrinkle method in style migration, as shown in fig. 1, wherein the contour enhancement model for approximately simulating a wrinkle method in style migration is constructed and obtained based on a CycleGAN model; the method for constructing the contour enhancement model for approximately simulating the wrinkle method in the style migration comprises the following steps of:
step one, respectively using keywords of landscape pictures and landscape pictures, capturing and downloading related pictures in a search engine through a crawler program, and obtaining 2562 landscape pictures (source domain) and 2363 landscape pictures (target domain).
And step two, eliminating irrelevant images by using a manual screening method. In total 1956 landscape pictures and 1884 landscape paintings were kept. 1761 landscape photos in the source domain are used as training samples to be stored in a trainA folder, and 195 landscape photos are used as testing samples to be stored in a testA folder; 1696 landscape paintings in the target domain are stored in the trainB folder as training samples, and 188 landscape paintings are stored in the testB folder as test samples. Some example pictures are shown in fig. 2, 3.
Step three, preparing a simulation environment: the simulation used a workstation with NVIDIA RTX3090 graphics card. The operating system was Ubuntu 20.04, PyTorch 1.7.1. The CUDA version was 11.2 and the CuDNN version was 8.1.
For convenience of training and evaluation, all images were adjusted to a resolution of 256 × 256. And step four, training and iterating the CycleGAN model. Specifically, 400,000 iterations were trained with a learning rate of 0.0002 for the first 200 durations, and then the learning rate decayed linearly to zero (800,000 iterations) for the subsequent 400 durations. An Adam optimizer [22] was applied. The batch size is set to 1.
Defining the landscape picture as a domain A and the landscape picture as a domain B; by utilizing the characteristic of cycle consistency, the network is designed into forward translation and backward translation, and the forward translation and the backward translation both comprise two branches: a drawing translation branch and a gradient translation branch;
step five, forward translation is carried out, and landscape photos in the domain A are converted into landscape pictures, namely GAB：A→B；
For GAB: a → B mapping, this process can be described as expression 1 and expression 2:
recB_edge＝GAB_edge(fakeA_edge) Formula 1;
recB＝Fuse(GAB(fakeA)，recB_edge) Formula 2;
in the formula:
recB_edgeis an image gradient map;
recBthe method comprises the following steps of (1) obtaining an original landscape painting;
GAB_edgea generator for the gradient translation branch in the forward translation process;
GABto convert landscape photos into landscape pictures;
fuse is fusion;
fakeA_edgeenhancing a gradient map for the edge of the landscape photo;
fakeAthe final edge enhancement map is a landscape photo;
drawing translation branch real scene photo realAAs a generator GABThe generated image enhancement features are used as the input of a fusion module; the gradient translation branch converts the real image gradient map realA_edgeAs a generator GBAThe generated edge enhanced gradient map fakeB_edgeAnd true edge-enhanced gradient map real in reverse translationB_edgeIs input to a discriminator DB_edgeJudging the authenticity; simultaneous fakeB_edgeSplicing with the output characteristics of the image translation branch, sending the output characteristics into a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fakeB(ii) a Generated fakeBWill match the true edge enhancement graph realBAre reversely translated together and input to a discriminator DBPerforming authenticity identification; the process is described by equations 3 and 4:
fakeB_edge＝(GAB_edge(realA_edge) Formula 3);
fakeB＝Fuse(GAB(realA)，fakeB_edge) Formula 4;
in the formula:
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeBdrawing a final edge enhancement image for the landscape;
GAB_edgea generator for the gradient translation branch in the forward translation process;
realA_edgeenhancing a gradient map for the real edge of the landscape photo;
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
the corresponding generative opposition loss also comes from the gradient panning branch and the image panning branch, which are represented as equations 5 and 6;
Figure BDA0003396264710000111
Figure BDA0003396264710000112
in the formula:
LGAN_B_edgegenerating a penalty function for the gradient translation branch;
LGAN_Bgenerating a penalty function for the image translation branch;
DBa landscape painting discriminator which is an image translation branch;
DB_edgea landscape painting discriminator which is a gradient translation branch;
e is a coefficient;
Figure BDA0003396264710000121
a discriminator loss function that is an image translation branch;
Figure BDA0003396264710000122
a discriminator loss function that is a gradient panning branch;
Figure BDA0003396264710000123
for shifting branches of the imageA generator loss function;
Figure BDA0003396264710000124
a generator loss function that is a gradient translation branch;
realBdrawing a real picture for the mountains and the waters;
realB_edgedrawing a real edge enhancement gradient map for the mountains and waters;
step six, the original landscape picture needs to be converted back from the landscape painting to realize the cycle consistency, namely GBAMapping of (2); generated edge enhancement map fakeBGenerator G input to a drawing translation branchBAPerforming the following steps; obtaining a group of repaired image characteristics as input of a fusion module for repairing; then the generated edge enhancement gradient map fakeB_edgeGenerator G input to gradient translation branchBA_edgeIn (1), an image gradient map rec for restoration is generatedA_edge(ii) a Finally, recA_edgeThe color drawing translation branches spliced to the restored image features are sent to a fusion module to guide gradient information to restore the original scene photo recA(ii) a The process can be described as formula 7 and formula 8;
recA_edge＝GBA_edge(fakeB_edge) Formula 7;
recA＝Fuse(GBA(fakeB)，recA_edge) Formula 8;
in the formula:
recA_edgeis an image gradient map;
recAthe original landscape picture is taken;
GBA_edgea generator for the gradient translation branch in the reverse translation process;
GBAto convert landscape pictures into landscape pictures;
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeABdrawing a final edge enhancement image for the landscape;
the cycle consistency loss of the restored image gradient map and the real image gradient map is set, so that the gradient conversion branch can provide more accurate gradient information for the image conversion branch, and the image is restored; meanwhile, the consistency of the restored scene photo and the real scene photo needs to be ensured; the L1 distance loss defines the cyclic consistency loss of these two parts as equation 9 and equation 10;
Lcyc_A_edge(GAB_edge，GBA_edge)＝E[||recA_edge-realA_edge||1]formula 9;
Lcyc_A(GAB，GBA)＝E[||recA-realA||1]formula 10;
in the formula:
Lcyc_A_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Athe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
e is the difference between the input value and the forward prediction;
to this end, the forward translation process has been completed;
step seven, reverse translation is the reverse process of forward translation; converting the landscape picture back to the original landscape picture by reverse translation, i.e. GBA: b → A; reverse translation is through GBAConverting landscape painting into landscape photo, and then passing through GABConverting the landscape picture into an original landscape painting; therefore, the invention realizes the conversion from the scene photo to the scene painting; the invention then requires a switch from landscape to original, to achieve cyclic consistency, i.e. GBAMapping of (2); for GBA: b → A mapping, which can be described as equation 11 and equation 12;
fakeA_edge＝(GBA_edge(realB_edge) Formula 11;
fakeA＝Fuse(BBA(realB)，fakeA_edge) Formula 12;
the corresponding generative antagonism loss functions are defined as formula 13 and formula 14;
Figure BDA0003396264710000141
Figure BDA0003396264710000142
in the formula:
LGAN_A_edgegenerating a penalty function for the gradient translation branch;
LGAN_Agenerating a penalty function for the image translation branch;
DAa landscape discriminator which is an image translation branch;
DA_edgea landscape discriminator being a gradient translation branch;
Figure BDA0003396264710000143
a discriminator loss function that is an image translation branch;
Figure BDA0003396264710000144
a discriminator loss function that is a gradient panning branch;
Figure BDA0003396264710000145
a generator loss function that is an image translation branch;
Figure BDA0003396264710000146
a generator loss function that is a gradient translation branch;
realAthe landscape is photographed with a real picture;
realA_edgeenhancing a gradient map for the landscape photo of the real edge;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
Lcyc_B_edge(GBA_edge，GAB_edge)＝E[||recB_edge-realB_edge||1]formula 15;
Lcyc_B(GBA，GAB)＝E[||recB-realB||1]formula 16;
in the formula:
Lcyc_B_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Bthe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
the overall objective loss function of the model is equation 17:
L(G，D)＝LGAN_B_edge+LGAN_B+LGAN_A_edge+LGAN_A+λ(Lcyc_A+Lcyc_A_edge+Lcyc_B+Lcyc_B_edge) Formula 17;
in the formula:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weight of the generator versus resistance loss and cyclic consistency loss.
And (3) effect testing:
firstly, the invention can better highlight the outlines of objects such as rocks, trees and the like and imitate the special texture effect of a chap method.
To assess the difference between the true image distribution and the generator-generated image distribution, the present invention employs a FID score [20 ]]. The FID scores for the different models are shown in table 1. FIDAllIs the FID score, FID, between the generated image set and the training setCunIs the FID score between the generated image set and the "chap" set. IT can be seen that the set of images generated by the U-GAT-IT model is closest to the training set. The image set generated by the CE-CycleGAN model is closest to the "chap method" set. Considering that the chap method focuses only on the stroke method, and is insensitive to color information, FID scores of the real image distribution and the image distribution generated after the graying are also calculated. The FID score is obviously improved, especially
Figure BDA0003396264710000151
From FIDCun80.50 is reduced to 61.14. IT can be concluded that the U-GAT-IT model is more sensitive to color information, while the focus on silhouette information is relatively less. This indicates that the model may be more advantageous in simulating image texture, which is a desirable feature for wrinkle simulation. These results indicate that introducing edge-enhanced translation branches can make the images generated by the neural network closer to the effect of the wrinkle.
TABLE 1 FID scores for different models
Score FIDAll FIDGray All FIDCun FIDGray Cun
U-GAT-IT 91.40 80.58 86.61 84.19
CycleGAN 94.70 85.21 81.03 67.71
Our 115.17 94.97 80.50 61.14
Secondly, in order to evaluate the artistry of the landscape painting generated, the invention introduces an aesthetic evaluation index NIMA, so that the performance of the method is evaluated from an artistic perspective, and the improvement of the aesthetic performance of the landscape painting generated after the Sobel operator is introduced can be effectively evaluated.
In addition to the closeness of the generated picture to the wrinkle, the present invention also focuses on the aesthetic quality of the generated picture. NIMA can predict human opinion on image evaluation from the aspects of direct perception and attractiveness, with the advantage similar to human subjective scoring, so the present invention selects it as an image quality evaluation index. NIMA generates a fractional histogram for any image. The images are scored 1-10 points and the images of the same subject are directly compared. The design is consistent with the histogram generated by the human scoring system in form, and the evaluation effect is closer to the evaluation result of human.
FIG. 4 shows a comparison of the NIMA score histograms of the 195 test images for the method of the invention, the cycleGAN method and the U-GAT-IT method. As can be seen from FIG. 4, the method of the present invention is superior to the cycleGAN method and the U-GAT-IT method mainly. The average score of the images generated by the method of the present invention was 4.74 points, the CycleGAN method was 4.05 points, and the U-GAT-IT method was 3.96 points. From the NIMA evaluation index, the performance of the method of the invention is improved by about 17 percent on average relative to the CycleGAN and improved by about 20 percent on average relative to the U-GAT-IT.
Third, to further analyze the effectiveness of the method of the present invention, the first five pictures of the present invention that performed better than the CycleGAN method and the U-GAT-IT method, respectively, were selected, as shown in fig. 5 and 7. Meanwhile, the worst five pictures are shown in FIGS. 6 and 8, compared to the cycleGAN and U-GAT-IT methods. It can be seen that the original photo content in fig. 5 and 7 is relatively rich, and its NIMA score is also high. In contrast, the original photo content in fig. 6 and 8 is relatively monotonous, with a generally low NIMA score. This result is consistent with the expectations of the invention. The method of the present invention highlights the boundaries of rocks and trees in the photograph. These boundaries will stand out when the image content is rich. When the image is monotonous, the boundary information is not abundant. Therefore, the method of the invention is mainly suitable for the situation of photos with abundant details.

Claims (1)

1. A contour enhancement method for approximately simulating wrinkle in style migration is characterized in that a contour enhancement model for approximately simulating wrinkle in style migration is adopted, and the contour enhancement model for approximately simulating wrinkle in style migration is constructed and obtained on the basis of a CycleGAN model; the method for constructing the contour enhancement model for approximately simulating the wrinkle method in the style migration comprises the following steps of:
respectively using keywords of landscape painting and landscape photo, capturing and downloading related pictures in a search engine through a crawler program, and obtaining landscape photos of a source domain and landscape painting of a target domain;
step two, irrelevant images are eliminated through screening, and screened landscape photos and landscape paintings are reserved; storing a part of landscape photos in the source domain as training samples in a tranA folder, and storing the rest of landscape photos in the source domain as test samples in a testA folder; storing a part of landscape paintings in the target domain as training samples in a tranB folder, and storing the rest landscape paintings in the target domain as test samples in a testB folder;
step three, all the images are adjusted to be 256 multiplied by 256 resolution, the step four is carried out, and training iteration is carried out on the cycleGAN model;
defining the landscape picture as a domain A and the landscape picture as a domain B; by utilizing the characteristic of cycle consistency, the network is designed into forward translation and backward translation, and the forward translation and the backward translation both comprise two branches: a drawing translation branch and a gradient translation branch;
step five, forward translation is carried out, and landscape photos in the domain A are converted into landscape pictures, namely GAB：A→B；
For GAB: a → B mapping, this process can be described as expression 1 and expression 2:
recB_edge＝GAB_edge(fakeA_edge) Formula 1;
recB＝Fuse(GAB(fakeA)，recB_edge) Formula 2;
in the formula:
recB_edgeis an image gradient map;
recBthe method comprises the following steps of (1) obtaining an original landscape painting;
GAB_edgea generator for the gradient translation branch in the forward translation process;
GABto convert landscape photos into landscape pictures;
fuse is fusion;
fakeA_edgeenhancing a gradient map for the edge of the landscape photo;
fakeAthe final edge enhancement map is a landscape photo;
drawing translation branch real scene photo realAAs a generator GABThe generated image enhancement features are used as the input of a fusion module; the gradient translation branch converts the real image gradient map realA_edgeAs a generator GBAThe generated edge enhanced gradient map fakeB_edgeAnd true edge-enhanced gradient map real in reverse translationB_edgeIs input to a discriminator DB_edgeJudging the authenticity; simultaneous fakeB_eageSplicing with the output characteristics of the image translation branch, sending the output characteristics into a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fakeB(ii) a Generated fakeBWill match the true edge enhancement graph realBAre reversely translated together and input to a discriminator DBPerforming authenticity identification; the process is described by equations 3 and 4:
fakeB_edge＝(GAB_edge(realA_edge) Formula 3);
fakeB＝Fuse(GAB(realA)，fakeB_edge) Formula 4;
in the formula:
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeBdrawing a final edge enhancement image for the landscape;
GAB_edgea generator for the gradient translation branch in the forward translation process;
realA_edgeenhancing a gradient map for the real edge of the landscape photo;
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
the corresponding generative opposition loss also comes from the gradient panning branch and the image panning branch, which are represented as equations 5 and 6;
Figure FDA0003396264700000031
Figure FDA0003396264700000032
in the formula:
LGAN_B_edgegenerating a penalty function for the gradient translation branch;
LGAN_Bgenerating a penalty function for the image translation branch;
DBa landscape painting discriminator which is an image translation branch;
DB_edgea landscape painting discriminator which is a gradient translation branch;
e is a coefficient;
Figure FDA0003396264700000035
a discriminator loss function that is an image translation branch;
Figure FDA0003396264700000033
a discriminator loss function that is a gradient panning branch;
Figure FDA0003396264700000036
a generator loss function that is an image translation branch;
Figure FDA0003396264700000034
a generator loss function that is a gradient translation branch;
realBdrawing a real picture for the mountains and the waters;
realB_edgedrawing a real edge enhancement gradient map for the mountains and waters;
step six, the original landscape picture needs to be converted back from the landscape painting to realize the cycle consistency, namely GBAMapping of (2); generated edge enhancement map fakeBGenerator G input to a drawing translation branchBAPerforming the following steps; obtaining a group of repaired image characteristics as input of a fusion module for repairing; then the generated edge enhancement gradient map fakeB_edgeGenerator G input to gradient translation branchBA_edgeIn (1), an image gradient map rec for restoration is generatedA_edge(ii) a Finally, recA_edgeThe color drawing translation branches spliced to the restored image features are sent to a fusion module to guide gradient information to restore the original scene photo recA(ii) a The process can be described as formula 7 and formula 8;
recA_edge＝GBA_edge(fakeB_edge) Formula 7;
recA＝Fuse(GBA(fakeB)，recA_edge) Formula 8;
in the formula:
recA_edgeis an image gradient map;
recAthe original landscape picture is taken;
GBA_edgea generator for the gradient translation branch in the reverse translation process;
GBAto draw the landscapeIs a landscape photo;
fakeB_edgedrawing an edge enhancement gradient map for the mountains and waters;
fakeABdrawing a final edge enhancement image for the landscape;
the cycle consistency loss of the restored image gradient map and the real image gradient map is set, so that the gradient conversion branch can provide more accurate gradient information for the image conversion branch, and the image is restored; meanwhile, the consistency of the restored scene photo and the real scene photo needs to be ensured; the L1 distance loss defines the cyclic consistency loss of these two parts as equation 9 and equation 10;
Lcyc_A_edge(GAB_edge，GBA_edge)＝E[||recA_edge-realA_edge||1]formula 9;
Lcyc_A(GAB，GBA)＝E[||recA-realA||1]formula 10;
in the formula:
Lcyc_A_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Athe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
e is the difference between the input value and the forward prediction;
to this end, the forward translation process has been completed;
step seven, reverse translation is the reverse process of forward translation; converting the landscape picture back to the original landscape picture by reverse translation, i.e. GBA: b → A; reverse translation is through GBAConverting landscape painting into landscape photo, and then passing through GABConverting the landscape picture into an original landscape painting; therefore, the invention realizes the conversion from the scene photo to the scene painting; the invention then requires a switch from landscape to original, to achieve cyclic consistency, i.e. GBAMapping of (2); for GBA: b → A mapping, which can be described as equation 11 and equation 12;
fakeA_edge＝(GBA_edge(realB_edge) Formula 11;
fakeA＝Fuse(BBA(realB)，fakeA_edge) Formula 12;
the corresponding generative antagonism loss functions are defined as formula 13 and formula 14;
Figure FDA0003396264700000051
Figure FDA0003396264700000052
in the formula:
LGAN_A_edgegenerating a penalty function for the gradient translation branch;
LGAN_Agenerating a penalty function for the image translation branch;
DAa landscape discriminator which is an image translation branch;
DA_edgea landscape discriminator being a gradient translation branch;
Figure FDA0003396264700000053
a discriminator loss function that is an image translation branch;
Figure FDA0003396264700000054
a discriminator loss function that is a gradient panning branch;
Figure FDA0003396264700000055
a generator loss function that is an image translation branch;
Figure FDA0003396264700000056
generator loss for gradient shifted branchesA loss function;
realAthe landscape is photographed with a real picture;
realA_edgeenhancing a gradient map for the landscape photo of the real edge;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
Lcyc_B_edge(GBA_edge，GAB_edge)＝E[||recB_edge-realB_edge||1]formula 15;
Lcyc_B(GBA，GAB)＝E[||recB-realB||1]formula 16;
in the formula:
Lcyc_B_edgeloss of cyclic consistency for the restored image gradient map and the true image gradient map;
Lcyc_Bthe cyclic consistency loss of the restored scene photo and the real scene photo is realized;
the overall objective loss function of the model is equation 17:
L(G，D)＝LGAN_B_edge+LGAN_B+LGAN_A_edge+LGAN_A+λ(Lcyc_A+Lcyc_A_edge+Lcyc_B+Lcyc_B_edge) Formula 17;
in the formula:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weight of the generator versus resistance loss and cyclic consistency loss.
CN202111483200.3A 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion Active CN114385883B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111483200.3A CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111483200.3A CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Publications (2)

Publication Number Publication Date
CN114385883A true CN114385883A (en) 2022-04-22
CN114385883B CN114385883B (en) 2024-03-15

Family

ID=81196050

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111483200.3A Active CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Country Status (1)

Country Link
CN (1) CN114385883B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503598A (en) * 2019-07-30 2019-11-26 西安理工大学 The font style moving method of confrontation network is generated based on condition circulation consistency
CN110675335A (en) * 2019-08-31 2020-01-10 南京理工大学 Superficial vein enhancement method based on multi-resolution residual error fusion network
US20200051565A1 (en) * 2018-08-13 2020-02-13 Carnegie Mellon University Processing speech signals of a user to generate a visual representation of the user
CN111696028A (en) * 2020-05-22 2020-09-22 华南理工大学 Method and device for processing cartoon of real scene image, computer equipment and storage medium
US20200356839A1 (en) * 2019-05-09 2020-11-12 ClimateAI, Inc. Systems and methods for selecting global climate simulation models for training neural network climate forecasting models
CN112487999A (en) * 2020-12-02 2021-03-12 西安邮电大学 Remote sensing image robust feature extraction method based on cycleGAN
CN112561782A (en) * 2020-12-15 2021-03-26 哈尔滨工程大学 Method for improving reality degree of simulation picture of offshore scene

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200051565A1 (en) * 2018-08-13 2020-02-13 Carnegie Mellon University Processing speech signals of a user to generate a visual representation of the user
US20200356839A1 (en) * 2019-05-09 2020-11-12 ClimateAI, Inc. Systems and methods for selecting global climate simulation models for training neural network climate forecasting models
CN110503598A (en) * 2019-07-30 2019-11-26 西安理工大学 The font style moving method of confrontation network is generated based on condition circulation consistency
CN110675335A (en) * 2019-08-31 2020-01-10 南京理工大学 Superficial vein enhancement method based on multi-resolution residual error fusion network
CN111696028A (en) * 2020-05-22 2020-09-22 华南理工大学 Method and device for processing cartoon of real scene image, computer equipment and storage medium
CN112487999A (en) * 2020-12-02 2021-03-12 西安邮电大学 Remote sensing image robust feature extraction method based on cycleGAN
CN112561782A (en) * 2020-12-15 2021-03-26 哈尔滨工程大学 Method for improving reality degree of simulation picture of offshore scene

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
王诣天;石文;: "基于M-CycleGAN实现口罩图像风格迁移", 电脑编程技巧与维护, no. 03 *
许哲豪;陈玮;: "基于生成对抗网络的图片风格迁移", 软件导刊, no. 06 *

Also Published As

Publication number Publication date
CN114385883B (en) 2024-03-15

Similar Documents

Publication Publication Date Title
Hao et al. Low-light image enhancement with semi-decoupled decomposition
Song et al. Geometry-aware face completion and editing
Li et al. A closed-form solution to photorealistic image stylization
CN111161137B (en) Multi-style Chinese painting flower generation method based on neural network
CN110378985B (en) Animation drawing auxiliary creation method based on GAN
Kumar et al. A comprehensive survey on non-photorealistic rendering and benchmark developments for image abstraction and stylization
Yang et al. Training with augmented data: Gan-based flame-burning image synthesis for fire segmentation in warehouse
Peng et al. Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings
Li et al. Everyone is a cartoonist: Selfie cartoonization with attentive adversarial networks
Xing et al. Diffsketcher: Text guided vector sketch synthesis through latent diffusion models
Zang et al. Artistic preprocessing for painterly rendering and image stylization
He Application of local color simulation method of landscape painting based on deep learning generative adversarial networks
Fu et al. Multi‐style Chinese art painting generation of flowers
Wei et al. Facial image inpainting with deep generative model and patch search using region weight
CN117557683A (en) Novel image generation method driven by text and semantic segmentation map together
Zhang et al. Reducing bias in AI-based analysis of visual artworks
CN114385883B (en) Contour enhancement method for approximately simulating chapping method in style conversion
Guan et al. DiffWater: Underwater image enhancement based on conditional denoising diffusion probabilistic model
An et al. Fast universal style transfer for artistic and photorealistic rendering
CN106570928A (en) Image-based re-lighting method
Wang et al. Damage sensitive and original restoration driven thanka mural inpainting
Li et al. Chinese painting style transfer system based on machine learning
Chen et al. Parallel pencil drawing stylization via structure-aware optimization
de Arruda et al. Non-photorealistic neural sketching: A case study on frontal-face images
Liang et al. Image-based rendering for ink painting

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant