CN114385883B - Contour enhancement method for approximately simulating chapping method in style conversion - Google Patents

Contour enhancement method for approximately simulating chapping method in style conversion Download PDF

Info

Publication number
CN114385883B
CN114385883B CN202111483200.3A CN202111483200A CN114385883B CN 114385883 B CN114385883 B CN 114385883B CN 202111483200 A CN202111483200 A CN 202111483200A CN 114385883 B CN114385883 B CN 114385883B
Authority
CN
China
Prior art keywords
edge
landscape
image
gradient
real
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202111483200.3A
Other languages
Chinese (zh)
Other versions
CN114385883A (en
Inventor
彭先霖
彭盛霖
胡琦瑶
彭进业
王佳欣
刘鑫煜
范建平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NORTHWEST UNIVERSITY
Original Assignee
NORTHWEST UNIVERSITY
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NORTHWEST UNIVERSITY filed Critical NORTHWEST UNIVERSITY
Priority to CN202111483200.3A priority Critical patent/CN114385883B/en
Publication of CN114385883A publication Critical patent/CN114385883A/en
Application granted granted Critical
Publication of CN114385883B publication Critical patent/CN114385883B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration using histogram techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a contour enhancement method of an approximation simulation chap method in style migration, which adopts a contour enhancement model of the approximation simulation chap method in style migration, wherein the contour enhancement model of the approximation simulation chap method in style migration is constructed and obtained based on a CE-cycleGAN model. The method develops an unsupervised and contour enhanced image style migration framework based on the CycleGAN, pays attention to lines according to the characteristics of landscape painting, sets constraint conditions, highlights edge characteristics, and realizes style conversion from landscape photos to landscape painting. Gradient conversion branches are added into a CycleGAN network, and a gradient guiding method is designed to effectively guide the style conversion of gradient information from a photo to a drawing. It is necessary to selectively enhance edge information in an image based on the visual characteristics of the human eye, that is, to preserve strong edges with relatively large contrast, rather than weak edges with small contrast.

Description

Contour enhancement method for approximately simulating chapping method in style conversion
Technical Field
The invention belongs to the technical field of computer vision, relates to style migration, and in particular relates to a contour enhancement method of an approximate simulation chapping method in style migration.
Background
Recently, AI (artificial intelligence) application of converting landscape photos into landscape paintings has become a popular point of interest for many artistic lovers and cultural relics protection. The artificial intelligence technology plays roles in extracting typical characteristics reflecting the style of the art work and simulating painting, and is helpful for training painting skills, innovating the art work, inheriting and protecting the traditional art.
The landscape photograph is converted into landscape painting, essentially a migration of the image style. GAN is proposed by Goodfellow et al as an effective method for implementing image style migration, and gradually becomes a mainstream method for image style migration. GAN is powerful and has many practical applications, such as generating high quality images, generating images from text, converting images from one style to another, etc. Many widely popular GAN-based architectures have emerged, such as DCGAN, bigGAN, starGAN and the like.
Although these methods may be suitable for landscape photo to landscape painting style conversion, there are still some unique problems to be solved.
The creation process of the landscape painting is complicated due to the specificity of the painting material and the large amount of use of various painting methods, pen methods, and ink methods. The main contents of landscape painting are mountains, stones and trees, and painters often use a "chapping method" to express the contents. This makes "chapping" an important expression language for landscape painting, an artistic form of aesthetic experience and aesthetic expression for painters. These "chapping" are primarily represented by lines, morphologically mimicking the rough contours of objects and scenes. Therefore, strengthening the contours of various objects to simulate the "chapping" effect is a challenging task worth intensive research when automatically generating landscape painting from landscape photos.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a contour enhancement method similar to a simulation chapping method in style migration so as to solve the technical problem that the style conversion effect from landscape photos to landscape paintings is to be further improved in the prior art.
In order to solve the technical problems, the invention adopts the following technical scheme:
the contour enhancement method of the similar simulation chap method in the style migration adopts a contour enhancement model of the similar simulation chap method in the style migration, and the contour enhancement model of the similar simulation chap method in the style migration is constructed and obtained based on a CycleGAN model; the construction method of the contour enhancement model for the similar simulation chapping method in style migration comprises the following steps:
step one, respectively using keywords of 'landscape painting' and 'landscape photo', capturing related pictures in a search engine through a crawler program, and downloading to obtain a source domain landscape photo and a target domain landscape painting;
step two, the irrelevant images are removed through screening, and the screened landscape photos and landscape paintings are reserved; storing a part of scenery photos in the source domain as training samples in a trainA folder, and storing the rest part of scenery photos in the source domain as test samples in a testA folder; storing a part of the landscape painting in the target domain as a training sample in a trainB folder, and storing the rest of the landscape painting in the target domain as a test sample in a testB folder;
step three, all images are adjusted to 256 multiplied by 256 resolution, the step four is carried out, and training iteration is carried out on the CycleGAN model;
step four, defining a landscape photo as a domain A and defining a landscape painting as a domain B; with the feature of loop consistency, the network is designed to translate forward and reverse, both of which contain two branches: drawing translation branches and gradient translation branches;
step five, forward translation is carried out, and the mountain-water photo in the domain A is converted into a mountain-water picture, namely G AB A, B; for G AB : a→b mapping, a process that can be described as expression 1 and expression 2:
rec B_edge ＝G AB_edge (fake A_edge ) Formula 1;
rec B ＝Fuse(G AB (fake A )，rec B_edge ) Formula 2;
wherein:
rec B_edge is an image gradient map;
rec B is an original landscape painting;
G AB_edge a generator for gradient translation branches in the forward translation process;
G AB the landscape photo is converted into landscape painting;
fuse is fusion;
fake A_edge enhancing a gradient map for the edge of the landscape photograph;
fake A a final edge enhancement map for the landscape photograph;
the drawing translation branch makes real scenery photo real A As generator G AB Taking the generated image enhancement features as the input of a fusion module; the gradient translation branch divides the real image gradient map real A_edge As generator G BA Is input into the generated edge enhancement gradient map fake B_edge And real edge enhancement gradient map real in reverse translation B_edge Is input to discriminator D B_edge Performing true and false judgment; simultaneous fake B_edge Splicing with the output characteristics of the image translation branches, sending the spliced image to a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fake B The method comprises the steps of carrying out a first treatment on the surface of the Generated fake B Real of the edge enhancement graph B Is input to the discriminator D after being translated reversely B In the process, the authenticity is identified; this process is described by formulas 3 and 4:
fake B_edge ＝(G AB_edge (real A_edge ) Formula 3;
fake B ＝Fuse(G AB (real A )，fake B_edge ) Formula 4;
wherein:
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake B a final edge enhancement map is drawn for the mountain and water;
G AB_edge a generator for gradient translation branches in the forward translation process;
real A_edge enhancing a gradient map for the real edge of the landscape photograph;
fake B_eage a gradient map is enhanced for the edge of the landscape painting;
the corresponding generation fight loss also comes from the gradient shift branch and the image shift branch, which are represented as equations 5 and 6;
wherein:
L GAN_B_edge generating a contrast loss function for the gradient translation branch;
L GAN_B generating a contrast loss function for the image translation branch;
D B a landscape painting discriminator for shifting branches of the image;
D B_edge a landscape painting discriminator for gradient translation branches;
e is a coefficient;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real B drawing a real image for mountain and water;
real B_edge a real edge enhancement gradient map is drawn for mountain and water;
step six, the original landscape photograph needs to be converted from landscape painting to realize the cyclical consistency, namely G BA Is mapped to; generated edge enhancement graph fake B Generator G input to drawing translation branch BA In (a) and (b); obtaining a group of repaired image features, and repairing the image features as the input of a fusion module; the generated edge enhancement gradient map fake is then processed B_edge Generator G input to gradient translation branch BA_edge In generating an image gradient map rec for restoration A_edge The method comprises the steps of carrying out a first treatment on the surface of the Finally, rec A_edge The color drawing translation branches spliced to the restored image features are sent to a fusion module, and gradient information is guided to restore the original scene photo rec A The method comprises the steps of carrying out a first treatment on the surface of the The process can be described as formula 7 and formula 8;
rec A_edge ＝G BA_edge (fake B_edge ) Formula 7;
rec A ＝Fuse(G BA (fake B )，rec A_edge ) Formula 8;
wherein:
rec A_edge is an image gradient map;
rec A is an original landscape photograph;
G BA_edge a generator for gradient translation branches in the reverse translation process;
G BA the landscape painting is converted into a landscape photo;
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake AB a final edge enhancement map is drawn for the mountain and water;
the cyclic consistency loss of the restored image gradient map and the real image gradient map is set, which is beneficial to the gradient conversion branch to provide more accurate gradient information for the image conversion branch, so that the image is restored; meanwhile, consistency of restored scene photos and real scene photos is required to be ensured; l1 distance loss defines the cyclic consistency loss of the two parts as formula 9 and formula 10;
L cyc_A_edge (G AB_edge ，G BA_edge )＝E[||rec A_edge -real A_edge || 1 ]formula 9;
L cyc_A (G AB ，G BA )＝E[||rec A -real A || 1 ]formula 10;
wherein:
L cyc_A_edge the cyclic consistency loss of the restored image gradient map and the true image gradient map;
L cyc_A a loss of cyclical consistency for restored scene shots and real scene shots;
e is the difference between the input value and the forward prediction;
so far, the forward translation process has been completed;
step seven, reverse translation is the opposite process to forward translation; converting landscapes back to original landscapes by reverse translation, i.e. G BA : B-A; reverse translation is performed by G BA Converting landscape painting into landscape photo, and then passing through G AB Converting the landscape photo into an original landscape painting; thus, the present invention enables the conversion of a photograph of a scene to a landscape; the invention then requires a switch from landscape painting back to the original landscape photograph to achieve a consistent cycle, i.e., G BA Is mapped to; for G BA : b→a mapping, a process that can be described as equation 11 and equation 12;
fake A_edge ＝(G BA_edge (real B_edge ) Formula 11;
fake A ＝Fuse(B BA (real B )，fake A_edge ) Formula 12;
the corresponding generated formula contrast loss function is defined as formula 13 and formula 14;
wherein:
L GAN_A_edge generating a contrast loss function for the gradient translation branch;
L GAN_A generating a contrast loss function for the image translation branch;
D A a landscape discriminator for shifting branches for the image;
D A_edge a landscape discriminator for gradient panning branches;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real A the real image is photographed by the landscape;
real A_edge enhancing the gradient map for the real edge of the landscape;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
L cyc_B_edge (G BA_edge ，G AB_edge )＝E[||rec B_edge -real B_edge || 1 ]formula 15;
L cyc_B (G BA ，G AB )＝E[||rec B -real B || 1 ]formula 16;
wherein:
L cyc_B_edge the cyclic consistency loss of the restored image gradient map and the true image gradient map;
L cyc_B a loss of cyclical consistency for restored scene shots and real scene shots;
the overall objective loss function of the model is equation 17:
wherein:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weights of the generated formula against the loss and the cyclic consistency loss.
Compared with the traditional CycleGAN method in the prior art, the method has the following technical effects:
(I) The method develops an unsupervised and contour enhanced image style migration framework based on the CycleGAN, pays attention to lines according to the characteristics of landscape painting, sets constraint conditions, highlights edge characteristics, and realizes style conversion from landscape photos to landscape painting.
And (II) adding a gradient conversion branch into the CycleGAN network, and designing a gradient guiding method to effectively guide the style conversion of gradient information from the photo to the drawing. It is necessary to selectively enhance edge information in an image based on the visual characteristics of the human eye, that is, to preserve strong edges with relatively large contrast, rather than weak edges with small contrast.
(III) in order to enhance the landscape style, the present invention uses an edge detection operator to extract strong edges of the gray image. The Sobel operator is used as an edge detection operator. This operator introduces a similar local averaging operation, which smoothes and removes the noise and has a good detection effect on rough edges. The characteristic meets the requirement of the method on the landscape painting simulation effect.
Drawings
FIG. 1 is a schematic diagram of a CE-CycleGAN network architecture.
Fig. 2 is a photograph of mountain water after screening in the source domain.
Fig. 3 is a landscape painting after screening in the target domain.
Fig. 4 is a NIMA score histogram of 195 test images.
Fig. 5 is the first five images superior to the CycleGAN method.
Fig. 6 is the five worst images compared to the CycleGAN method.
Fig. 7 is the first five images superior to the U-GAT-IT method.
FIG. 8 is the five worst images compared to the U-GAT-IT method.
The following examples illustrate the invention in further detail.
Detailed Description
In the invention, the landscape painting is a Chinese landscape painting.
The CycleGAN refers to an image style conversion framework.
CE-CycleGAN, contourlet-Enhanced CycleGAN, unsupervised and Contour enhanced image style conversion framework.
The entire network with gradient translation branches is called CE-CycleGAN.
The CE-CycleGAN model is an unsupervised and contour enhanced image style conversion frame model, namely a contour enhancement model which approximates a chapping method in style migration.
U-GAT-IT refers to building an end-to-end weak supervision image cross-domain conversion model by proposing a new attention mechanism and combining a new regularization mode on the basis of CycleGAN.
NIMA refers to the distribution of human assessment opinions on images that can be predicted from direct sensory (technical) and attraction (aesthetic) perspective, based on the latest deep object recognition neural network. NIMA may generate a scoring histogram for any image, i.e., scoring 1-10 for an image, and comparing with an image of the same subject.
FID refers to the calculation of the mean and covariance matrix of the 2048-dimensional feature vector set output by the two image sets of the concept Net-V3.
As a preferred, NIMA is referenced from Talebi, H.and P.Milanfar, NIMA:Nereal Image Assembly.IEEE Transactions on Image Processing,2017:p.1-1.
As a preferred, U-GAT-IT is cited from Kim, J.et al, U-GAT-IT Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image transfer.2019.
As a preferred variant, FID is cited from DC Dowson, landau B V.the Fre chet Distance between Multivariate Normal Distributions [ J ]. Journal of Multivariate Analysis,1982,12 (3): 450-455.
The following specific embodiments of the present invention are given according to the above technical solutions, and it should be noted that the present invention is not limited to the following specific embodiments, and all equivalent changes made on the basis of the technical solutions of the present application fall within the protection scope of the present invention.
Examples:
the embodiment provides a contour enhancement method for simulating a chap method in style migration, which adopts a contour enhancement model for simulating the chap method in style migration, as shown in fig. 1, wherein the contour enhancement model for simulating the chap method in style migration is constructed and obtained based on a CycleGAN model; the construction method of the contour enhancement model for the similar simulation chapping method in style migration comprises the following steps:
step one, using the keywords of "landscape painting" and "landscape photo", respectively, capturing related pictures in a search engine by a crawler program and downloading, and obtaining 2562 landscape photos (source domain) and 2363 Zhang Shanshui pictures (target domain) in total.
And step two, eliminating irrelevant images by using a manual screening method. A total of 1956 scenery photos and 1884 landscape paintings are reserved. 1761 scenery photos in the source domain are stored in the trainA folder as training samples, and 195 scenery photos are stored in the testA folder as test samples; 1696 mountain and water drawings in the target domain are stored as training samples in the trainB folder, and 188 mountain and water drawings are stored as test samples in the testB folder. Some example pictures are shown in fig. 2 and 3.
Step three, preparing a simulation environment: the simulation used a workstation with an NVIDIA RTX3090 graphics card. The operating system is Ubuntu 20.04,PyTorch 1.7.1.CUDA version 11.2 and cudnn version 8.1.
For ease of training and evaluation, all images were adjusted to 256×256 resolution. Turning to step four, training and iterating the CycleGAN model. Specifically, 400,000 iterations were trained at a learning rate of 0.0002 for the first 200 durations, and then the learning rate decayed linearly to zero (800,000 iterations) for the next 400 durations. An Adam optimizer is applied [22]. The batch size is set to 1.
Step four, defining a landscape photo as a domain A and defining a landscape painting as a domain B; with the feature of loop consistency, the network is designed to translate forward and reverse, both of which contain two branches: drawing translation branches and gradient translation branches;
step five, forward translation is carried out, and the mountain-water photo in the domain A is converted into a mountain-water picture, namely G AB ：A→B；
For G AB : a→b mapping, a process that can be described as expression 1 and expression 2:
rec B_edge ＝G AB_edge (fake A_edge ) Formula 1;
rec B ＝Fuse(G AB (fake A )，rec B_edge ) Formula 2;
wherein:
rec B_edge is an image gradient map;
rec B is an original landscape painting;
G AB_edge a generator for gradient translation branches in the forward translation process;
G AB the landscape photo is converted into landscape painting;
fuse is fusion;
fake A_edge enhancing a gradient map for the edge of the landscape photograph;
fake A a final edge enhancement map for the landscape photograph;
the drawing translation branch makes real scenery photo real A As generator G AB Taking the generated image enhancement features as the input of a fusion module; the gradient translation branch divides the real image gradient map real A_edge As generator G BA Is input into the generated edge enhancement gradient map fake B_edge And real edge enhancement gradient map real in reverse translation B_edge Is input to discriminator D B_edge Performing true and false judgment; simultaneous fake B_edge Splicing with the output characteristics of the image translation branches, sending the spliced image to a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fake B The method comprises the steps of carrying out a first treatment on the surface of the Generated fake B Real of the edge enhancement graph B Is input to the discriminator D after being translated reversely B In the process, the authenticity is identified; this process is described by formulas 3 and 4:
fake B_edge ＝(G AB_edge (real A_edge ) Formula 3;
fake B ＝Fuse(G AB (real A )，fake B_edge ) Formula 4;
wherein:
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake B a final edge enhancement map is drawn for the mountain and water;
G AB_edge a generator for gradient translation branches in the forward translation process;
real A_edge enhancing a gradient map for the real edge of the landscape photograph;
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
the corresponding generation fight loss also comes from the gradient shift branch and the image shift branch, which are represented as equations 5 and 6;
wherein:
L GAN_B_edge generating a contrast loss function for the gradient translation branch;
L GAN_B generating a contrast loss function for the image translation branch;
D B a landscape painting discriminator for shifting branches of the image;
D B_edge a landscape painting discriminator for gradient translation branches;
e is a coefficient;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real B drawing a real image for mountain and water;
real B_edge a real edge enhancement gradient map is drawn for mountain and water;
step six, the original landscape photograph needs to be converted from landscape painting to realize the cyclical consistency, namely G BA Is mapped to; generated edge enhancement graph fake B Generator G input to drawing translation branch BA In (a) and (b); obtaining a group of repaired image features, and repairing the image features as the input of a fusion module; the resulting edge enhancement gradient is then usedGraph fake B_edge Generator G input to gradient translation branch BA_edge In generating an image gradient map rec for restoration A_edge The method comprises the steps of carrying out a first treatment on the surface of the Finally, rec A_edge The color drawing translation branches spliced to the restored image features are sent to a fusion module, and gradient information is guided to restore the original scene photo rec A The method comprises the steps of carrying out a first treatment on the surface of the The process can be described as formula 7 and formula 8;
rec A_edge ＝G BA_edge (fake B_edge ) Formula 7;
rec A ＝Fuse(G BA (fake B )，rec A_edge ) Formula 8;
wherein:
rec A_edge is an image gradient map;
rec A is an original landscape photograph;
G BA_edge a generator for gradient translation branches in the reverse translation process;
G BA the landscape painting is converted into a landscape photo;
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake AB a final edge enhancement map is drawn for the mountain and water;
the cyclic consistency loss of the restored image gradient map and the real image gradient map is set, which is beneficial to the gradient conversion branch to provide more accurate gradient information for the image conversion branch, so that the image is restored; meanwhile, consistency of restored scene photos and real scene photos is required to be ensured; l1 distance loss defines the cyclic consistency loss of the two parts as formula 9 and formula 10;
L cyc_A_edge (G AB_edge ，G BA_edge )＝E[||rec A_edge -real A_edge || 1 ]formula 9;
L cyc_A (G AB ，G BA )＝E[||rec A -real A || 1 ]formula 10;
wherein:
L cyc_A_edge for cyclic coincidence of restored image gradient map and true image gradient mapLoss of sex;
L cyc_A a loss of cyclical consistency for restored scene shots and real scene shots;
e is the difference between the input value and the forward prediction;
so far, the forward translation process has been completed;
step seven, reverse translation is the opposite process to forward translation; converting landscapes back to original landscapes by reverse translation, i.e. G BA : B-A; reverse translation is performed by G BA Converting landscape painting into landscape photo, and then passing through G AB Converting the landscape photo into an original landscape painting; thus, the present invention enables the conversion of a photograph of a scene to a landscape; the invention then requires a switch from landscape painting back to the original landscape photograph to achieve a consistent cycle, i.e., G BA Is mapped to; for G BA : b→a mapping, a process that can be described as equation 11 and equation 12;
fake A_edge ＝(G BA_edge (real B_edge ) Formula 11;
fake A ＝Fuse(B BA (real B )，fake A_edge ) Formula 12;
the corresponding generated formula contrast loss function is defined as formula 13 and formula 14;
wherein:
L GAN_A_edge generating a contrast loss function for the gradient translation branch;
L GAN_A generating a contrast loss function for the image translation branch;
D A a landscape discriminator for shifting branches for the image;
D A_edge a landscape discriminator for gradient panning branches;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real A the real image is photographed by the landscape;
real A_edge enhancing the gradient map for the real edge of the landscape;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
L cyc_B_edge (G BA_edge ，G AB_edge )＝E[||rec B_edge -real B_edge || 1 ]formula 15;
L cyc_B (G BA ，G AB )＝E[||rec B -real B || 1 ]formula 16;
wherein:
L cyc_B_edge the cyclic consistency loss of the restored image gradient map and the true image gradient map;
L cyc_B a loss of cyclical consistency for restored scene shots and real scene shots;
the overall objective loss function of the model is equation 17:
L(G，D)＝L GAN_B_edge +L GAN_B +L GAN_A_edge +L GAN_A +λ(L cyc_A +L cyc_A_edge +L cyc_B +L cyc_B_edge ) Formula 17;
wherein:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weights of the generated formula against the loss and the cyclic consistency loss.
And (3) effect test:
first, the invention can better highlight the outline of objects such as rocks, trees and the like, and imitate the special texture effect of the chapping method.
In order to evaluate the difference between the true image distribution and the generator-generated image distribution, the present invention employs FID scoring [20]. FID scores for the different models are shown in table 1. FID (FID) All Is the FID score between the generated image set and the training set, FID Cun Is the FID score between the generated image set and the "chapped" set. IT can be seen that the image set generated by the U-GAT-IT model is closest to the training set. The image set generated by the CE-CycleGAN model is closest to the "chapped" set. Considering that the chapping method only focuses on the stroke method and is insensitive to color information, the FID scores of the real image distribution and the image distribution generated after ashing are also calculated. FID score is significantly improved, especiallyFrom FID Cun Is reduced to 61.14 at 80.50. IT can be inferred that the U-GAT-IT model is more sensitive to color information and has relatively less focus on silhouette information. This suggests that the model may be more advantageous in modeling image texture, which is the desired feature for modeling chapping. These results indicate that introducing edge-enhanced translation branches can make the neural network generated image more closely resemble the effect of the chapping method.
TABLE 1 FID score for different models
Score FID All FID Gray All FID Cun FID Gray Cun
U-GAT-IT 91.40 80.58 86.61 84.19
CycleGAN 94.70 85.21 81.03 67.71
Our 115.17 94.97 80.50 61.14
Second, in order to evaluate the artistry of the produced landscape painting, the invention introduces an aesthetic evaluation index NIMA, and from the viewpoint of the performance of the evaluation method, the improvement of the aesthetic performance of the produced landscape painting after the Sobel operator is introduced can be effectively evaluated.
In addition to the closeness of the generated picture to the chapping, the invention focuses on the aesthetic quality of the generated picture. NIMA can predict human opinion of image evaluation from direct perception and appeal, and has advantages similar to human subjective scoring, so the present invention selects it as an image quality evaluation index. NIMA generates a fractional histogram for any image. The images are scored 1-10 and the images of the same subject are compared directly. The design is consistent in form with the histogram generated by the human scoring system, and the evaluation effect is closer to that of human beings.
Figure 4 shows a comparison of NIMA score histograms over 195 test images for the method, cycleGAN method and U-GAT-IT method of the present invention. As can be seen from FIG. 4, the method of the present invention is mainly superior to the cycleGAN method and the U-GAT-IT method. The average score of the images generated by the method of the present invention was 4.74 points, the CycleGAN method was 4.05 points, and the U-GAT-IT method was 3.96 points. From NIMA evaluation index, the performance of the method is improved by about 17% compared with the CycleGAN average, and is improved by about 20% compared with the U-GAT-IT average.
Third, to further analyze the effectiveness of the method of the present invention, the first five pictures of the present invention were selected to perform better than the CycleGAN method and the U-GAT-IT method, respectively, as shown in fig. 5 and 7. Meanwhile, the worst five pictures are shown in fig. 6 and 8, compared to the CycleGAN and U-GAT-IT methods. It can be seen that the original photos in fig. 5 and 7 are relatively rich in content, and the NIMA score is also high. In contrast, the original photo content in fig. 6 and 8 is relatively monotonous, with a generally lower NIMA score. This result is consistent with the expectations of the present invention. The method of the invention is to highlight the boundaries of rocks and trees in the photograph. These boundaries are highlighted when the image content is rich. When the image is monotonous, the boundary information is not rich. Thus, the method of the present invention is mainly applicable to the case of photographs with rich details.

Claims (1)

1. The contour enhancement method is characterized in that the contour enhancement model which is approximately simulated by the chap method in the style migration is adopted and is constructed and obtained based on a CycleGAN model; the construction method of the contour enhancement model for the similar simulation chapping method in style migration comprises the following steps:
step one, respectively using keywords of 'landscape painting' and 'landscape photo', capturing related pictures in a search engine through a crawler program, and downloading to obtain a source domain landscape photo and a target domain landscape painting;
step two, the irrelevant images are removed through screening, and the screened landscape photos and landscape paintings are reserved; storing a part of scenery photos in the source domain as training samples in a trainA folder, and storing the rest part of scenery photos in the source domain as test samples in a testA folder; storing a part of the landscape painting in the target domain as a training sample in a trainB folder, and storing the rest of the landscape painting in the target domain as a test sample in a testB folder;
step three, all images are adjusted to 256 multiplied by 256 resolution, the step four is carried out, and training iteration is carried out on the CycleGAN model;
step four, defining a landscape photo as a domain A and defining a landscape painting as a domain B; with the feature of loop consistency, the network is designed to translate forward and reverse, both of which contain two branches: drawing translation branches and gradient translation branches;
step five, forward translation is carried out, and the mountain-water photo in the domain A is converted into a mountain-water picture, namely G AB ：A→B；
For G AB : a→b mapping, a process that can be described as expression 1 and expression 2:
rec B_edge ＝G AB_edge (fake A_edge ) Formula 1;
rec B ＝Fuse(G AB (fake A )，rec B_edge ) Formula 2;
wherein:
rec B_edge is an image gradient map;
rec B is an original landscape painting;
G AB_edge a generator for gradient translation branches in the forward translation process;
G AB the landscape photo is converted into landscape painting;
fuse is fusion;
fake A_edge enhancing a gradient map for the edge of the landscape photograph;
fake A a final edge enhancement map for the landscape photograph;
the drawing translation branch makes real scenery photo real A As generator G AB Taking the generated image enhancement features as the input of a fusion module; the gradient translation branch divides the real image gradient map real A_edge As generator G BA Is input into the generated edge enhancement gradient map fake B_edge And real edge enhancement gradient map real in reverse translation B_edge Is input to discriminator D B_edge Performing true and false judgment; simultaneous fake B_eage Splicing with the output characteristics of the image translation branches, sending the spliced image to a fusion module, providing guidance of gradient information for the fusion module, and generating a final edge enhancement image fake B The method comprises the steps of carrying out a first treatment on the surface of the Generated fake B Real of the edge enhancement graph B Is input to the discriminator D after being translated reversely B In the process, the authenticity is identified; this process is described by formulas 3 and 4:
fake B_edge ＝(G AB_edge (real A_edge ) Formula 3;
fake B ＝Fuse(G AB (real A )，fake B_edge ) Formula 4;
wherein:
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake B a final edge enhancement map is drawn for the mountain and water;
G AB_edge a generator for gradient translation branches in the forward translation process;
real A_edge enhancing a gradient map for the real edge of the landscape photograph;
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
the corresponding generation fight loss also comes from the gradient shift branch and the image shift branch, which are represented as equations 5 and 6;
wherein:
L GAN_B_edge generating a contrast loss function for the gradient translation branch;
L GAN_B generating a contrast loss function for the image translation branch;
D B a landscape painting discriminator for shifting branches of the image;
D B_edge a landscape painting discriminator for gradient translation branches;
e is a coefficient;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real B drawing a real image for mountain and water;
real B_edge a real edge enhancement gradient map is drawn for mountain and water;
step six, the scenery is converted back to the original airScenery, to achieve cyclic consistency, i.e. G BA Is mapped to; generated edge enhancement graph fake B Generator G input to drawing translation branch BA In (a) and (b); obtaining a group of repaired image features, and repairing the image features as the input of a fusion module; the generated edge enhancement gradient map fake is then processed B_edge Generator G input to gradient translation branch BA_edge In generating an image gradient map rec for restoration A_edge The method comprises the steps of carrying out a first treatment on the surface of the Finally, rec A_edge The color drawing translation branches spliced to the restored image features are sent to a fusion module, and gradient information is guided to restore the original scene photo rec A The method comprises the steps of carrying out a first treatment on the surface of the The process can be described as formula 7 and formula 8;
rec A_edge ＝G BA_edge (fake B_edge ) Formula 7;
rec A ＝Fuse(G BA (fake B )，rec A_edge ) Formula 8;
wherein:
rec A_edge is an image gradient map;
rec A is an original landscape photograph;
G BA_edge a generator for gradient translation branches in the reverse translation process;
G BA the landscape painting is converted into a landscape photo;
fake B_edge a gradient map is enhanced for the edge of the landscape painting;
fake AB a final edge enhancement map is drawn for the mountain and water;
the cyclic consistency loss of the restored image gradient map and the real image gradient map is set, which is beneficial to the gradient conversion branch to provide more accurate gradient information for the image conversion branch, so that the image is restored; meanwhile, consistency of restored scene photos and real scene photos is required to be ensured; l1 distance loss defines the cyclic consistency loss of the two parts as formula 9 and formula 10;
L cyc_A_edge (G AB_edge ，G BA_edge )＝E[||rec A_edge -real A_edge || 1 ]formula 9;
L cyc_A (G AB ，G BA )＝E[||rec A -real A || 1 ]formula 10;
wherein:
L cyc_A_edge the cyclic consistency loss of the restored image gradient map and the true image gradient map;
L cyc_A a loss of cyclical consistency for restored scene shots and real scene shots;
e is the difference between the input value and the forward prediction;
so far, the forward translation process has been completed;
step seven, reverse translation is the opposite process to forward translation; converting landscapes back to original landscapes by reverse translation, i.e. G BA : B-A; reverse translation is performed by G BA Converting landscape painting into landscape photo, and then passing through G AB Converting the landscape photo into an original landscape painting; thus, the present invention enables the conversion of a photograph of a scene to a landscape; the invention then requires a switch from landscape painting back to the original landscape photograph to achieve a consistent cycle, i.e., G BA Is mapped to; for G BA : b→a mapping, a process that can be described as equation 11 and equation 12;
fake A_edge ＝(G BA_edge (real B_edge ) Formula 11;
fake A ＝Fuse(B BA (real B )，fake A_edge ) Formula 12;
the corresponding generated formula contrast loss function is defined as formula 13 and formula 14;
wherein:
L GAN_A_edge is flat in gradientGenerating a counterloss function of the shift branch;
L GAN_A generating a contrast loss function for the image translation branch;
D A a landscape discriminator for shifting branches for the image;
D A_edge a landscape discriminator for gradient panning branches;
a discriminator loss function that translates branches for the image;
a discriminator loss function for the gradient shift branch;
generator loss functions for image translation branches;
generator loss function for gradient shift branches;
real A the real image is photographed by the landscape;
real A_edge enhancing the gradient map for the real edge of the landscape;
the corresponding cycle consistency loss is defined as equation 15 and equation 16;
L cyc_B_edge (G BA_edge ，G AB_edge )＝E[||rec B_edge -real B_edge || 1 ]formula 15;
L cyc_B (G BA ，G AB )＝E[||rec B -real B || 1 ]formula 16;
wherein:
L cyc_B_edge loss of cyclical uniformity for restored image gradient maps and true image gradient mapsLoss of function;
L cyc_B a loss of cyclical consistency for restored scene shots and real scene shots;
the overall objective loss function of the model is equation 17:
L(G，D)＝L GAN_B_edge +L GAN_B +L GAN_A_edge +L GAN_A +λ(L cyc_A +L cyc_A_edge +L cyc_B +L cyc_B_edge ) Formula 17;
wherein:
l (G, D) is the overall objective loss function of the model;
λ represents the relative weights of the generated formula against the loss and the cyclic consistency loss.
CN202111483200.3A 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion Active CN114385883B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111483200.3A CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111483200.3A CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Publications (2)

Publication Number Publication Date
CN114385883A CN114385883A (en) 2022-04-22
CN114385883B true CN114385883B (en) 2024-03-15

Family

ID=81196050

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111483200.3A Active CN114385883B (en) 2021-12-07 2021-12-07 Contour enhancement method for approximately simulating chapping method in style conversion

Country Status (1)

Country Link
CN (1) CN114385883B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503598A (en) * 2019-07-30 2019-11-26 西安理工大学 The font style moving method of confrontation network is generated based on condition circulation consistency
CN110675335A (en) * 2019-08-31 2020-01-10 南京理工大学 Superficial vein enhancement method based on multi-resolution residual error fusion network
CN111696028A (en) * 2020-05-22 2020-09-22 华南理工大学 Method and device for processing cartoon of real scene image, computer equipment and storage medium
CN112487999A (en) * 2020-12-02 2021-03-12 西安邮电大学 Remote sensing image robust feature extraction method based on cycleGAN
CN112561782A (en) * 2020-12-15 2021-03-26 哈尔滨工程大学 Method for improving reality degree of simulation picture of offshore scene

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11568864B2 (en) * 2018-08-13 2023-01-31 Carnegie Mellon University Processing speech signals of a user to generate a visual representation of the user
US10909446B2 (en) * 2019-05-09 2021-02-02 ClimateAI, Inc. Systems and methods for selecting global climate simulation models for training neural network climate forecasting models

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110503598A (en) * 2019-07-30 2019-11-26 西安理工大学 The font style moving method of confrontation network is generated based on condition circulation consistency
CN110675335A (en) * 2019-08-31 2020-01-10 南京理工大学 Superficial vein enhancement method based on multi-resolution residual error fusion network
CN111696028A (en) * 2020-05-22 2020-09-22 华南理工大学 Method and device for processing cartoon of real scene image, computer equipment and storage medium
CN112487999A (en) * 2020-12-02 2021-03-12 西安邮电大学 Remote sensing image robust feature extraction method based on cycleGAN
CN112561782A (en) * 2020-12-15 2021-03-26 哈尔滨工程大学 Method for improving reality degree of simulation picture of offshore scene

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
基于M-CycleGAN实现口罩图像风格迁移;王诣天;石文;;电脑编程技巧与维护(03);全文 *
基于生成对抗网络的图片风格迁移;许哲豪;陈玮;;软件导刊(06);全文 *

Also Published As

Publication number Publication date
CN114385883A (en) 2022-04-22

Similar Documents

Publication Publication Date Title
He et al. Chipgan: A generative adversarial network for chinese ink wash painting style transfer
CN111161137B (en) Multi-style Chinese painting flower generation method based on neural network
CN102881011B (en) Region-segmentation-based portrait illumination transfer method
Mechrez et al. Photorealistic style transfer with screened poisson equation
Tang et al. Animated construction of Chinese brush paintings
Yang et al. Training with augmented data: Gan-based flame-burning image synthesis for fire segmentation in warehouse
Gupta et al. Image style transfer using convolutional neural networks based on transfer learning
Peng et al. Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings
Li et al. Everyone is a cartoonist: Selfie cartoonization with attentive adversarial networks
Xing et al. Diffsketcher: Text guided vector sketch synthesis through latent diffusion models
CN106570928B (en) A kind of heavy illumination method based on image
Fu et al. Multi‐style Chinese art painting generation of flowers
CN114385883B (en) Contour enhancement method for approximately simulating chapping method in style conversion
Zhang et al. Reducing bias in AI-based analysis of visual artworks
CN116777738A (en) Authenticity virtual fitting method based on clothing region alignment and style retention modulation
Guan et al. DiffWater: Underwater image enhancement based on conditional denoising diffusion probabilistic model
CN116342377A (en) Self-adaptive generation method and system for camouflage target image in degraded scene
Chen et al. Parallel pencil drawing stylization via structure-aware optimization
Liang et al. Image-based rendering for ink painting
Li et al. Chinese painting style transfer system based on machine learning
Lian et al. Transformation of portraits to Picasso’s cubism style
Rosin et al. Watercolour rendering of portraits
Fu et al. See clearly on rainy days: Hybrid multiscale loss guided multi-feature fusion network for single image rain removal
Zeng et al. 3D plants reconstruction based on point cloud
Park et al. StyleBoost: A Study of Personalizing Text-to-Image Generation in Any Style using DreamBooth

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant