CN114219701A - Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium - Google Patents
Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114219701A CN114219701A CN202111324175.4A CN202111324175A CN114219701A CN 114219701 A CN114219701 A CN 114219701A CN 202111324175 A CN202111324175 A CN 202111324175A CN 114219701 A CN114219701 A CN 114219701A
- Authority
- CN
- China
- Prior art keywords
- dunhuang
- image
- fresco
- data set
- style conversion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 85
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 52
- 238000007781 pre-processing Methods 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 28
- 238000004364 calculation method Methods 0.000 claims description 20
- 238000005070 sampling Methods 0.000 claims description 9
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 6
- 230000003042 antagnostic effect Effects 0.000 claims description 3
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 239000013598 vector Substances 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 abstract description 3
- 238000000605 extraction Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000000739 chaotic effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010422 painting Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 241000283070 Equus zebra Species 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000005562 fading Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010428 oil painting Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a Dunhuang fresco artistic style conversion method, a system, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a data set, wherein the data set comprises a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set; preprocessing a data set and dividing a training set; constructing a normalized contrast learning generation countermeasure network based on the CUT framework; training the confrontation network generated by the normalized comparison learning by using a training set, and generating the confrontation network generated by the trained normalized comparison learning as a Dunhuang mural artistic style conversion model; acquiring an image to be processed; inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image. The invention overcomes the problem that the existing algorithm can not realize vivid Dunhuang mural artistic style conversion, and realizes high-quality and vivid Dunhuang mural artistic style image conversion.
Description
Technical Field
The invention belongs to the field of image processing, computer vision and image conversion, and particularly relates to a Dunhuang mural artistic style conversion method, a system, computer equipment and a storage medium.
Background
Dunhuang fresco is one of the big cultural artistic magnifications of China and also the largest treasury of world art, and is famous for a long history, rich and profound drawing contents and extremely high art value, however, due to the special form of the Dunhuang fresco and the high difficulty of the drawing process, the Dunhuang fresco cannot be circulated in the life of people like other arts such as oil painting, Dunhuang fresco and the like. In addition, with the lapse of time, various natural and artificial factors cause damages such as the falling-off of murals, the color fading and the like, and the existence of the onlynhuang mural art is threatened. Therefore, the realization of the automatic creation of the artistic style of the Dunhuang fresco is of great practical significance for protecting, inheriting and developing the art.
In recent years, although a CNN-based style migration technique, such as gantys, can migrate the style of one image to the style of a reference image, the style generation effect of this technique depends heavily on the reference style image, and it is difficult to express a style of art completely using only one reference style image. An image conversion technology based on a Generation countermeasure network (GAN) which is proposed by taking a CycleGAN (an article, "unknown image-to-image conversion using cycle-dependent adaptive Networks") as a representative can better overcome the problem, can realize the style conversion of two domain image sets (for example, converting an image with a content of a night scene into an image with a content of a day scene), and can fully learn style characteristics of a class of arts.But instead of the other end of the tubeThe cyclic consistency structure adopted by CycleGAN has limited ability to implement conversion tasks with large style differences (style of natural images and dunhuang mural images).
In 2020, CUT (from the article "contrast Learning for unamplified Image-to-Image transformation") proposed by Taesung Park et al first introduces contrast Learning into the Image transformation domain, ensures content consistency by maximizing mutual information (based on contrast loss of tiles) between corresponding tiles of input and output images, and simultaneously learns the style characteristics of the target Image domain under the constraint of the countermeasures against loss. The method realizes excellent conversion performance in a plurality of image style conversion tasks (such as converting an image with horse content into an image with zebra content and converting the style of a natural image into western artistic style) by using a simple model, and can also show excellent conversion performance for image conversion tasks with large style difference. Nevertheless, the task of directly applying the method to the conversion of natural images into the style of Dunhuang fresco cannot achieve a realistic style conversion effect, and even the problem of image degradation occurs, mainly because the Dunhuang fresco art is greatly different from the visual art studied in the past in style characteristics. The Dunhuang fresco is a fusion product of eastern and western culture exchange, and forms a unique artistic style under the influence of the Chinese and western painting styles. The Dunhuang fresco artistic style is a product organically combined with several aspects of line drawing modeling, decorative picture composition, art painting and calligraphy.
Disclosure of Invention
In view of the above, the present invention provides a method, a system, a computer device and a storage medium for converting the artistic style of Dunhuang fresco. The invention proposes the normalized contrast loss based on the image block on the basis of the CUT model, thereby constructing the normalized contrast learning generation confrontation network, overcoming the problem caused by the inter-domain style difference existing when the same encoder is used for extracting the characteristics, simultaneously introducing the sky semantic loss, the line loss and the color loss to constrain the network, leading the network to fully learn various style characteristics of the Dunhuang mural, and further generating the Dunhuang mural style conversion image with better effect and higher quality.
The first purpose of the invention is to provide a method for converting the artistic style of Dunhuang fresco.
The second purpose of the invention is to provide a system for converting the artistic style of Dunhuang fresco.
It is a third object of the invention to provide a computer apparatus.
It is a fourth object of the present invention to provide a storage medium.
The first purpose of the invention can be achieved by adopting the following technical scheme:
a Dunhuang fresco artistic style conversion method comprises the following steps:
acquiring a data set, wherein the data set comprises a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
preprocessing a data set and dividing a training set;
constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
training the confrontation network generated by the normalized comparison learning by using a training set, and generating the confrontation network generated by the trained normalized comparison learning as a Dunhuang mural artistic style conversion model;
acquiring an image to be processed;
inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image.
Further, the normalized contrast learning generation countermeasure network comprises a coding and decoding generator taking a residual block as a core and a discriminator taking PatchGAN as a structure on the basis of a CUT framework; the loss function of the countermeasure network generated by the normalized contrast learning comprises a normalized contrast loss function based on a picture block, a countermeasure loss function, a sky semantic loss function, a line loss function and a color loss function;
the generator is formed by cascading three down-sampling layers, nine conversion network layers based on a residual block and three up-sampling layers, wherein the first half part is an encoder, and the second half part is a decoder;
the discriminator consists of four down-sampling layers.
Further, the normalized contrast loss based on the image block is obtained by computing and maximizing mutual information loss through the encoder of the generator and features extracted by a two-layer MLP network, and the computation formula is as follows:
wherein the content of the first and second substances,
to maximize the mutual information loss, the formula is calculated as follows:
further, the countermeasure loss is obtained by calculating the true/false probability of the generated image and the dunhuang fresco image inputted to the discriminator, and the calculation formula is:
further, the color loss is obtained by calculating the cosine similarity of the hue color histogram vectors extracted from the generated image and the dunghuang fresco image, and the calculation formula is:
further, the training of the confrontation network generated by the normalized comparison learning by using the training set and the generation of the confrontation network generated by the trained normalized comparison learning as the dunhuang mural artistic style conversion model specifically include:
initializing the parameters of the normalized contrast learning generated confrontation network by adopting Gaussian distribution, and setting the parameters of the normalized contrast learning generated confrontation network and the weight parameters of the loss function;
inputting the training set into a normalized comparison learning generation countermeasure network for training to obtain the calculation of each loss function;
and updating network parameters by adopting an optimization strategy of a random gradient descent method, so that the loss is gradually reduced to convergence, thereby obtaining a trained confrontation network generated by the normalized comparison learning, and using the trained confrontation network generated by the normalized comparison learning as a Dunhuang mural artistic style conversion model.
Further, the preprocessing the data set specifically includes:
unifying all picture formats in the Dunhuang fresco image domain data set and the natural image domain data set, wherein the picture formats comprise picture file types and picture sizes.
The second purpose of the invention can be achieved by adopting the following technical scheme:
a dunhuang artistic style conversion system, the system comprising:
a first acquisition unit for acquiring a data set including a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
the preprocessing unit is used for preprocessing the data set and dividing a training set;
the construction unit is used for constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
the training unit is used for training the confrontation network generated by the normalized comparison learning by utilizing a training set and taking the trained confrontation network generated by the normalized comparison learning as a Dunhuang mural artistic style conversion model;
the second acquisition unit is used for acquiring an image to be processed;
and the style conversion unit is used for inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image and realize the Dunhuang fresco artistic style conversion of the natural image.
The third purpose of the invention can be achieved by adopting the following technical scheme:
a computer device comprising a processor and a memory for storing a processor executable program, the processor implementing the aforementioned method of converting the artistic style of the dunghuang fresco when executing the program stored in the memory.
The fourth purpose of the invention can be achieved by adopting the following technical scheme:
a storage medium storing a program which, when executed by a processor, implements the aforementioned method of converting the artistic style of dunhuang fresco.
Compared with the prior art, the invention has the following beneficial effects:
1. the normalized contrast learning generation countermeasure network provided by the invention comprises a generator (a generator for converting a corresponding natural image into a Dunhuang mural style image) taking a residual block as a core and a discriminator taking PatchGAN as a structure in the aspect of structure, and comprises a normalized contrast loss, an countermeasure loss, a sky semantic loss, a line loss and a color loss function based on a graphic block in the aspect of optimizing a loss function. Based on the structure foundation of the generated countermeasure network and the combination optimization of a plurality of loss functions, the natural image and the Dunhuang fresco image can be combined to realize the high-quality Dunhuang fresco style conversion effect on the basis of not increasing the training parameters of the model and improving the training model speed.
2. The graph block-based normalized contrast loss provided by the invention aims at the problem that the graph block-based contrast loss adopted by the baseline model CUT cannot effectively overcome the problems of image degradation and difficulty in model optimization caused by domain style difference, is used for improving the stability and efficiency of model training and simultaneously improving the quality of generated images.
3. The invention provides sky semantic loss, which aims at the problem that the model has a chaotic sky region style in the process of realizing Dunhuang fresco style conversion and is used for improving the visual quality of images generated by the Dunhuang fresco style.
4. The line loss and the color loss provided by the invention aim at the problem that the model cannot effectively learn the vivid Dunhuang mural style when realizing the Dunhuang mural style conversion, and are used for promoting the model to generate more vivid and more vivid images with the Dunhuang mural style.
5. The invention can realize the style conversion of various natural images of Dunhuang fresco, greatly simplifies and innovatively obtains and creates the mode of Dunhuang fresco art, and is beneficial to the protection, the inheritance and the promotion of the Dunhuang fresco art.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
Fig. 1 is a flowchart of the dunghuang artistic style conversion method in embodiment 1 of the present invention.
Fig. 2 is a structural diagram of the normalized contrast learning generation countermeasure network of embodiment 1 of the present invention.
Fig. 3a to 3b are conversion diagrams of the natural image and the corresponding dunhuang fresco style in embodiment 1 of the present invention.
Fig. 4 is a flowchart of the dunghuang artistic style conversion system in embodiment 2 of the present invention.
Fig. 5 is a block diagram of a computer device according to embodiment 3 of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the protection scope of the present invention.
Example 1:
as shown in fig. 1, the present embodiment provides a method for converting artistic style of dunhuang fresco, which includes the following steps:
s101, acquiring a data set;
the data sets of the present embodiment include the dunghuang fresco image domain data set and the natural image domain data set corresponding to the dunghuang fresco image domain data set.
Acquiring Dunhuang mural image domain data set: dunhuang fresco image data is collected from each picture website and museum, and the captured Dunhuang fresco image data is screened and classified according to the content and quality of the Dunhuang fresco images, wherein the Dunhuang fresco data comprises building fresco and landscape fresco, and therefore a Dunhuang fresco image domain data set is created.
Acquiring a natural image domain dataset: and acquiring natural images corresponding to the content types of the Dunhuang mural image domain data sets on websites such as Baidu pictures, Google pictures and the like by utilizing a crawler technology, wherein the natural image data comprises building images and landscape images, and thus, the natural image domain data sets are created.
And S102, preprocessing the data set and dividing a training set.
Preprocessing of the data set: unifying all picture formats in the Dunhuang fresco image domain data set and the natural image domain data set, wherein the picture formats comprise picture file types and picture sizes; the preprocessed data sets are divided into two types of data sets according to the building type and the landscape type, and one type of data set is selected in each subsequent model training.
The data set selected in this example includes 1723 natural images and 985 Dunhuang fresco; and unifying the picture file type and the picture size of the preliminarily obtained building data set, wherein the unified picture type is a 'png' file, and the unified picture size is '256 x 256'.
In order to keep the training and testing data as same as possible, the data set is divided by adopting layered sampling, 80% of pictures are randomly selected from the natural image domain data set (picture A domain) and the Dunhuang mural image domain data set (picture B domain) in the building class data set to form the training set, and the rest 20% of the pictures are used as the testing set. Finally, 4 training and testing data folders are obtained, namely, tranA, tranB, testA and testB.
S103, constructing a normalized contrast learning generation countermeasure network based on the CUT framework.
The generation countermeasure network of this embodiment adopts an LSGAN structure, and on the basis of the CUT framework, a codec generator G taking a residual block as a core and a discriminator D taking a PatchGAN as a structure construct a normalized contrast learning generation countermeasure network, that is, an untrained normalized contrast learning generation countermeasure network, as shown in fig. 2.
Specifically, the generator G is composed of three down-sampling layers, nine conversion network layers based on a residual block, and three up-sampling layers in cascade, the first half is an encoder Genc, the second half is a decoder Gdec, and the output of the generator composed of the encoder and the decoder is a generated image with the same size as the input; the discriminator D is a full convolution network consisting of 4 downsampling layers, which outputs the true-false probability condition of one generated image.
The loss function of the normalized contrast learning generation countermeasure network in the embodiment is a composite loss function consisting of the normalized contrast loss, the countermeasure loss, the sky semantic loss, the line loss and the color loss based on the image blocks; the method comprises the steps of comparing the normalized contrast loss constraint model input and output images based on image blocks, learning the conversion capability of the Dunhuang fresco style by the antagonistic loss constraint model, overcoming the style chaos problem of the Dunhuang fresco style conversion process in the sky area by the sky semantic loss, generating the Dunhuang style characteristic of the image line drawing modeling by the line loss prominently, and learning the capability of simulating the Dunhuang fresco color style characteristic by the color loss constraint model.
1) Based on the normalized contrast loss of the image blocks, the normalized contrast loss is obtained by calculating mutual information between the corresponding image blocks of the input image and the output image, and the normalized contrast loss is mainly used for ensuring the content consistency of the input image and the output image; further, the normalized contrast loss based on the tile is calculated by the encoder Genc in the generator G and the features extracted by the two-layer MLP network H to maximize the mutual information loss, specifically: inputting the input natural image and the corresponding generated image distribution into a cascade network of an encoder Genc and an MLP network H, carrying out normalization processing of mean value removal and variance on the obtained features, and then carrying out maximum mutual information loss calculation by utilizing the normalized features, wherein the calculation formula is as follows:
wherein the content of the first and second substances,
to maximize mutual information loss, the calculation formula is:
2) the sky semantic loss is calculated by calculating a sky region output image extracted by a sky semantic extraction module additionally arranged between the generated Dunhuang fresco style image and the corresponding natural image, and a constraint model can overcome the problem of chaotic sky region style in the process of realizing Dunhuang fresco style conversion; furthermore, the sky semantic loss is obtained by calculating the SSIM loss of the sky region extracted by the trained segmentation network as a sky semantic region extraction module from the input natural image and the corresponding generated image, and specifically comprises the following steps: respectively inputting the input natural image and the corresponding generated image into a sky semantic extraction module, and performing SSIM (structural similarity model) similarity loss calculation on the obtained sky semantic region, wherein the calculation formula is as follows:
3) the line loss is calculated by an edge output image extracted by an edge extraction module additionally arranged between the Dunhuang fresco style image generated by calculation and the corresponding natural image, and the Dunhuang fresco style characteristic of the line modeling generated by a network can be restrained; further, the line loss is obtained by calculating the balance cross entropy loss of the edge extraction graph of the input natural image and the corresponding generated image extracted by the trained edge extraction module HED network, specifically: respectively inputting the input natural image and the corresponding generated image into an edge extraction module, and carrying out balance cross entropy loss calculation on the obtained edge image, wherein the calculation formula is as follows:
4) the countermeasures against the loss are obtained by calculating the true and false degrees of the generated image and the reference image, and are mainly used for ensuring that the generated image has the style of Dunhuang fresco; further, the countermeasure loss is obtained by calculating the probability of authenticity of the generated image and the dunhuang fresco image inputted to the discriminator D, and the calculation formula is:
5) the color loss is calculated by the similarity of the hue histogram extracted by the hue histogram extraction module added between the Dunhuang fresco style image generated by calculation and the real Dunhuang fresco image, and the network learning can be restricted to obtain the style with the color characteristics of the Dunhuang fresco; further, the color loss respectively inputs the input natural image and the reference Dunhuang fresco image into a hue histogram extraction module, and cosine similarity loss calculation is carried out on the obtained hue histogram vector, and the calculation formula is as follows:
and S104, training the confrontation network generated by the normalized comparison learning by utilizing the training set to obtain the Dunhuang mural artistic style conversion model.
Step S104 is a model training stage, and includes the following specific steps:
1) network initialization: all parameters to be trained are initialized with a gaussian distribution with a mean of 0 and a variance of 0.02.
2) Setting network parameters: optimizing by adopting an Adam algorithm, wherein the parameter beta is (0.5, 0.999); the learning factor was 0.0002 for the first 100 epochs, gradually decaying to 0 at a rate of 0.01 for the last 100 epochs, and the minimum batch data was 1.
3) Setting a loss function weight parameter: graph block based normalized contrast loss, countermeasure loss, sky semantic loss, line lossThe weight parameters of color loss and color loss are respectively lambdaN_PatchNCE,λGAN,λSky,λLine,λColor。
4) And loading the training data trainA and trainB to the normalized contrast learning to generate an antagonistic network.
5) Training the model: performing iterative training on the normalized contrast learning generation countermeasure network shown in fig. 2, inputting a natural image X randomly selected from the raina training set into the G generator, and then inputting a generated G (X) picture into the D discriminator for discrimination, and simultaneously, randomly selecting a dunhuang mural image Y from the rainb training set into the D discriminator for discrimination; and then calculating each obtained loss, then performing gradient back transmission under an optimization strategy of a random gradient descent method, and updating network parameters to gradually reduce the loss to be convergent.
In summary, the overall loss function is:
L(G,D,X,Y)＝λN_PatchNCE LN_PatchNCE(G,H,X,Y)+λGANLGAN(G,D,X,Y)+λSkyLSky(G,X)
+λLine LLine(G,X)+λColorLColor(G,X,Y)
wherein, each weight value is as follows: lambda [ alpha ]N_PatchNCE＝10,λGAN＝1,λSky＝2,λLine＝2,λColor1. And optimizing the neural network according to the obtained loss function until the trained normalized comparison learning is obtained and the confrontation network is generated, and using the trained normalized comparison learning and the confrontation network as a final Dunhuang mural artistic style conversion model.
And S105, acquiring an image to be processed.
S106, inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image.
Reading test data testA and preprocessing the data, wherein the preprocessing only converts the image size into 256 × 256 in the test process; the image to be processed is sent to the Dunhuang fresco artistic style conversion model for calculation, the image to be processed is as shown in figure 3a, in the testing process, the image to be processed only obtains a corresponding output image through a generator G (a generator for converting a corresponding natural image into a Dunhuang fresco image), and as shown in figure 3b, the Dunhuang fresco style conversion of the natural image is realized.
And (3) carrying out performance test on the Dunhuang fresco artistic style conversion model: the Dunhuang fresco artistic style conversion model is used for carrying out Dunhuang fresco style conversion on the testA picture set, and then the effect of generating the image is quantitatively measured through two assessment methods, namely FID and KID. The FID can calculate the distance between the real image domain and the generated image domain in the feature space, and a lower FID means that the generated image has higher picture quality and style characteristics. KID measures the difference between the true image domain and the generated image domain by calculating the square of the maximum mean difference between the Incep representations, with lower KID meaning that the generated image has higher picture quality and style characteristics. As can be seen from table 1, the method of this embodiment performs better in comparison to other advanced algorithms. Table 1 demonstrates the better performance of the method of the present embodiment in the dunghuang mural style conversion task from the perspective of evaluating the effect of generating images and from the perspective of training time.
Table 1 evaluation of generated images
| KID | FID | Training time | |
| CycleGAN | 0.0762±0.0017 | 216.9905 | 36.5h |
| CUT | 0.0591±0.0020 | 198.9915 | 54.4h |
| Method of the present embodiment | 0.0479±0.0015 | 192.5047 | 27.2h |
It should be noted that although the method operations of the above-described embodiments are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Rather, the depicted steps may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Example 2:
as shown in fig. 4, the present embodiment provides a system for converting the artistic style of the dunghuang fresco, which includes a first obtaining unit 401, a preprocessing unit 402, a constructing unit 403, a training unit 404, a second obtaining unit 405, and a style converting unit 406, and the specific functions of each unit are as follows:
a first acquiring unit 401 for acquiring a data set including a dunghuang fresco image domain data set and a natural image data set corresponding to the dunghuang fresco image domain data set;
a preprocessing unit 402, configured to preprocess the data set and partition a training set;
a constructing unit 403, configured to construct a first normalized contrast learning generation countermeasure network based on the CUT framework;
a training unit 404, configured to utilize a training set to train the confrontation network generated by the normalized comparison learning, and use the trained confrontation network generated by the normalized comparison learning as a dunhuang mural artistic style conversion model;
a second acquiring unit 405 for acquiring an image to be processed;
and the style conversion unit 406 is used for inputting the image to be processed into the Dunhuang fresco artistic style conversion model to realize the Dunhuang fresco artistic style conversion of the image to be processed.
The specific implementation of each unit in this embodiment may refer to embodiment 1, which is not described herein any more; it should be noted that the system provided in this embodiment is only illustrated by the division of the functional units, and in practical applications, the above function distribution may be completed by different functional units according to needs, that is, the internal structure is divided into different functional units to complete all or part of the functions described above.
Example 3:
as shown in fig. 5, the present embodiment provides a terminal device, which includes a processor 502, a memory, an input device 503, a display device 504 and a network interface 505 connected by a system bus 501, the processor is used for providing computing and control capability, the memory includes a nonvolatile storage medium 506 and an internal memory 507, the nonvolatile storage medium 506 stores an operating system, a computer program and a database, the internal memory 507 provides an environment for the operating system and the computer program in the nonvolatile storage medium to run, when the processor 502 executes the computer program stored in the memory, the dunhuang mural style conversion method of the above embodiment 1 is implemented, as follows:
acquiring a data set, wherein the data set comprises a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
preprocessing a data set and dividing a training set;
constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
training the confrontation network generated by the normalized comparison learning by using a training set, and generating the confrontation network generated by the trained normalized comparison learning as a Dunhuang mural artistic style conversion model;
acquiring an image to be processed;
inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image.
Example 4:
the present embodiment provides a storage medium, which is a computer-readable storage medium, and stores a computer program, which, when executed by a processor, implements the method for converting the artistic style of the dunghuang mural of the above embodiment 1, as follows:
acquiring a data set, wherein the data set comprises a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
preprocessing a data set and dividing a training set;
constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
training the confrontation network generated by the normalized comparison learning by using a training set, and generating the confrontation network generated by the trained normalized comparison learning as a Dunhuang mural artistic style conversion model;
acquiring an image to be processed;
inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image.
It should be noted that the computer readable storage medium of the present embodiment may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In the present embodiment, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this embodiment, however, a computer readable signal medium may include a propagated data signal with a computer readable program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer-readable storage medium may be written with a computer program for performing the present embodiments in one or more programming languages, including an object oriented programming language such as Java, Python, C + +, and conventional procedural programming languages, such as C, or similar programming languages, or combinations thereof. The program may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
In summary, the invention includes, on the basis of the CUT framework, a codec generator using a residual block as a core and a discriminator using a patch gan as a structure, and a plurality of loss functions based on the normalized contrast loss, the countermeasure loss, the sky semantic loss, the line loss, the color loss, and the like of the image block, wherein on the basis of constraining the consistency of the contents of the input and output images, the problems of image degradation and difficulty in model optimization caused by the difference of domain styles are effectively overcome; sky semantic loss, line loss and color loss effectively improve the generation quality of the conversion style image and the Dunhuang fresco style effect.
The above description is only for the preferred embodiments of the present invention, but the protection scope of the present invention is not limited thereto, and any person skilled in the art can substitute or change the present invention and its inventive concept within the scope of the present invention disclosed in the present patent.
Claims (10)
1. A Dunhuang fresco artistic style conversion method is characterized by comprising the following steps:
acquiring a data set, wherein the data set comprises a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
preprocessing a data set and dividing a training set;
constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
training the confrontation network generated by the normalized comparison learning by using a training set, and generating the confrontation network generated by the trained normalized comparison learning as a Dunhuang mural artistic style conversion model;
acquiring an image to be processed;
inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image, and realizing the Dunhuang fresco artistic style conversion of the natural image.
2. The Dunhuang fresco artistic style conversion method according to claim 1, wherein said normalized contrast learning generation confrontation network comprises a coding and decoding generator with residual block as core and a discriminator with PatchGAN as structure based on CUT skeleton; the loss function of the countermeasure network generated by the normalized contrast learning comprises a normalized contrast loss function based on a picture block, a countermeasure loss function, a sky semantic loss function, a line loss function and a color loss function;
the generator is formed by cascading three down-sampling layers, nine conversion network layers based on a residual block and three up-sampling layers, wherein the first half part is an encoder, and the second half part is a decoder;
the discriminator consists of four down-sampling layers.
3. The Dunhuang fresco artistic style conversion method, according to claim 2, wherein said tile-based normalized contrast loss is calculated by maximizing mutual information loss through the encoder of the generator and a feature calculation extracted by a two-layer MLP network, the calculation formula is as follows:
wherein, l () is the maximum mutual information loss, and the calculation formula is as follows:
4. the Dunhuang artistic style conversion method according to claim 2, wherein said antagonistic loss is obtained by calculating the true or false probability of the generated image and Dunhuang fresco image inputted to the discriminator by the formula:
5. the Dunhuang artistic style conversion method according to claim 2, wherein said color loss is obtained by calculating cosine similarity of hue color histogram vectors extracted from the generated image and the Dunhuang fresco image, by the formula:
6. the Dunhuang artistic style conversion method according to claim 1, wherein said training set is used to train the generation of the confrontation network for the normalized contrast learning, and the trained normalization contrast learning is used as the Dunhuang fresco artistic style conversion model, which specifically comprises:
initializing the parameters of the normalized contrast learning generated confrontation network by adopting Gaussian distribution, and setting the parameters of the normalized contrast learning generated confrontation network and the weight parameters of the loss function;
inputting the training set into a normalized comparison learning generation countermeasure network for training to obtain the calculation of each loss function;
and updating network parameters by adopting an optimization strategy of a random gradient descent method, so that the loss is gradually reduced to convergence, thereby obtaining a trained confrontation network generated by the normalized comparison learning, and using the trained confrontation network generated by the normalized comparison learning as a Dunhuang mural artistic style conversion model.
7. The dunghuang artistic style conversion method according to claims 1-6, characterized in that the data set is preprocessed, specifically:
unifying all picture formats in the Dunhuang fresco image domain data set and the natural image domain data set, wherein the picture formats comprise picture file types and picture sizes.
8. A Dunhuang artistic style conversion system, the system comprising:
a first acquisition unit for acquiring a data set including a Dunhuang fresco image domain data set and a natural image domain data set corresponding to the Dunhuang fresco image domain data set;
the preprocessing unit is used for preprocessing the data set and dividing a training set;
the construction unit is used for constructing a normalized contrast learning generation countermeasure network based on the CUT framework;
the training unit is used for training the confrontation network generated by the normalized comparison learning by utilizing a training set and taking the trained confrontation network generated by the normalized comparison learning as a Dunhuang mural artistic style conversion model;
the second acquisition unit is used for acquiring an image to be processed;
and the style conversion unit is used for inputting the image to be processed into the Dunhuang fresco artistic style conversion model to obtain the Dunhuang fresco image and realize the Dunhuang fresco artistic style conversion of the natural image.
9. A computer apparatus comprising a processor and a memory for storing a processor executable program, wherein the processor, when executing the program stored in the memory, implements the method of converting the artistic style of the dunghuang fresco of any one of claims 1 to 7.
10. A storage medium storing a program which, when executed by a processor, realizes the dunghuang fresco artistic style conversion method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111324175.4A CN114219701A (en) | 2021-11-10 | 2021-11-10 | Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111324175.4A CN114219701A (en) | 2021-11-10 | 2021-11-10 | Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114219701A true CN114219701A (en) | 2022-03-22 |
Family
ID=80696803
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111324175.4A Pending CN114219701A (en) | 2021-11-10 | 2021-11-10 | Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114219701A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115375601A (en) * | 2022-10-25 | 2022-11-22 | 四川大学 | Decoupling expression traditional Chinese painting generation method based on attention mechanism |
| CN118036555A (en) * | 2024-04-11 | 2024-05-14 | 江西师范大学 | Low-sample font generation method based on skeleton transfer and structure contrast learning |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10373023B1 (en) * | 2019-01-28 | 2019-08-06 | StradVision, Inc. | Learning method and learning device for runtime input transformation of real image on real world into virtual image on virtual world, to be used for object detection on real images, by using cycle GAN capable of being applied to domain adaptation |
| CN113313625A (en) * | 2021-05-13 | 2021-08-27 | 华南理工大学 | Ink and wash painting artistic style conversion method, system, computer equipment and storage medium |
-
2021
- 2021-11-10 CN CN202111324175.4A patent/CN114219701A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10373023B1 (en) * | 2019-01-28 | 2019-08-06 | StradVision, Inc. | Learning method and learning device for runtime input transformation of real image on real world into virtual image on virtual world, to be used for object detection on real images, by using cycle GAN capable of being applied to domain adaptation |
| CN113313625A (en) * | 2021-05-13 | 2021-08-27 | 华南理工大学 | Ink and wash painting artistic style conversion method, system, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| TAESUNG PARK, ET AL.: "Contrastive Learning for Unpaired Image-to-Image Translation", 《CONTRASTIVE LEARNING FOR UNPAIRED IMAGE-TO-IMAGE TRANSLATION》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115375601A (en) * | 2022-10-25 | 2022-11-22 | 四川大学 | Decoupling expression traditional Chinese painting generation method based on attention mechanism |
| CN115375601B (en) * | 2022-10-25 | 2023-02-28 | 四川大学 | Decoupling expression traditional Chinese painting generation method based on attention mechanism |
| CN118036555A (en) * | 2024-04-11 | 2024-05-14 | 江西师范大学 | Low-sample font generation method based on skeleton transfer and structure contrast learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113240580B (en) | Lightweight image super-resolution reconstruction method based on multi-dimensional knowledge distillation | |
| Van den Oord et al. | Conditional image generation with pixelcnn decoders | |
| JP2022548712A (en) | Image Haze Removal Method by Adversarial Generation Network Fusing Feature Pyramids | |
| CN111832570A (en) | Image semantic segmentation model training method and system | |
| CN105512723A (en) | Artificial neural network calculating device and method for sparse connection | |
| CN110223234A (en) | Depth residual error network image super resolution ratio reconstruction method based on cascade shrinkage expansion | |
| CN114219701A (en) | Dunhuang fresco artistic style conversion method, system, computer equipment and storage medium | |
| CN113343775B (en) | Deep learning generalization method for remote sensing image ground surface coverage classification | |
| CN111476249A (en) | Construction method of multi-scale large-receptive-field convolutional neural network | |
| CN112884758B (en) | Defect insulator sample generation method and system based on style migration method | |
| CN110717953A (en) | Black-white picture coloring method and system based on CNN-LSTM combined model | |
| CN109598676A (en) | A kind of single image super-resolution method based on Hadamard transform | |
| CN113313625B (en) | Ink and wash painting artistic style conversion method, system, computer equipment and storage medium | |
| CN112712033A (en) | Automatic division method for catchment areas of municipal drainage pipe network | |
| CN111768326B (en) | High-capacity data protection method based on GAN (gas-insulated gate bipolar transistor) amplified image foreground object | |
| CN116682021A (en) | High-resolution remote sensing image building vector outline data extraction method | |
| CN114581789A (en) | Hyperspectral image classification method and system | |
| CN113888505A (en) | Natural scene text detection method based on semantic segmentation | |
| CN113554655B (en) | Optical remote sensing image segmentation method and device based on multi-feature enhancement | |
| CN116152263A (en) | CM-MLP network-based medical image segmentation method | |
| CN115660979A (en) | Attention mechanism-based double-discriminator image restoration method | |
| CN114331894A (en) | Face image restoration method based on potential feature reconstruction and mask perception | |
| CN112396598B (en) | Portrait matting method and system based on single-stage multitask collaborative learning | |
| CN114140317A (en) | Image animation method based on cascade generation confrontation network | |
| CN113744152A (en) | Tide water image denoising processing method, terminal and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220322 |
|
| RJ01 | Rejection of invention patent application after publication |