CN113837953A - Image restoration method based on generation countermeasure network - Google Patents

Image restoration method based on generation countermeasure network Download PDF

Info

Publication number
CN113837953A
CN113837953A CN202110653410.6A CN202110653410A CN113837953A CN 113837953 A CN113837953 A CN 113837953A CN 202110653410 A CN202110653410 A CN 202110653410A CN 113837953 A CN113837953 A CN 113837953A
Authority
CN
China
Prior art keywords
image
generator
loss
discriminator
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110653410.6A
Other languages
Chinese (zh)
Other versions
CN113837953B (en
Inventor
赵莉
赵瑞霞
焦焱
陈非凡
许鹤馨
史嘉琪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xian Technological University
Original Assignee
Xian Technological University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xian Technological University filed Critical Xian Technological University
Priority to CN202110653410.6A priority Critical patent/CN113837953B/en
Publication of CN113837953A publication Critical patent/CN113837953A/en
Application granted granted Critical
Publication of CN113837953B publication Critical patent/CN113837953B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an image restoration method based on a generation countermeasure network, which comprises the following steps: firstly, training a network model according to a set iteration termination number, and saving parameters of a generator and a discriminator, training configuration, a structure, weight and the state of an optimizer of the network model after each 1000 times of training; secondly, after the training of the network model is finished, repairing the image by using the generated network model; finally, comparing the repaired image with an original image; the image repaired by the image repairing method based on the generated countermeasure network is clearer, natural and continuous, and accords with the visual effect of human, so that the image repairing result realized by using the text algorithm is better than other two algorithms, and the effectiveness of the text algorithm is proved.

Description

Image restoration method based on generation countermeasure network
Technical Field
The invention belongs to the technical field of image restoration, and particularly relates to an image restoration system and method based on a generation countermeasure network.
Background
Image restoration is an ancient image processing technique, originating from the renaissance of literature. The current artwork has the phenomena of blurring, partial deletion and the like due to time, artists directly carry out manual repair on the artwork, so that the repair technology of the artists has high requirements, and once a small error occurs, the artwork can be completely damaged, and irreparable loss is caused. The manual repairing method is not only inefficient, but also requires a large amount of manpower and material resources, so people begin to explore a rapid automatic image repairing mode to replace manual repairing.
With the rapid development of computer science technology, Bertalmio et al proposed a partial differential equation-based digital image restoration algorithm at the 2000 Siggraph conference. The image restoration algorithm simulates a manual restoration mode, and enables a computer to estimate information of a missing area by using information of a good area of an image according to an isophote direction of pixel points in the image and fill the information to finish image restoration. Because of the popularization of computer electronic products and stable and quick network, network social contact becomes a frequent communication mode for people, and the image is used as a main carrier of social communication, thereby greatly improving the importance of the digital image in the network society. The digital image restoration technology is not only applied to daily life of people, but also widely applied to historical research, medical imaging and entertainment movie and television.
At present, digital image restoration technologies can be divided into two categories, namely a traditional digital image restoration technology and an image restoration technology based on deep learning, wherein the traditional image restoration technology mainly repairs missing areas of images according to mathematical formulas or thermal diffusion modes, and only images with simple structures and small damaged areas can be repaired. With the expansion of image areas, the problems of unclear structure, discontinuous textures, slow restoration speed and the like of images restored by using the traditional restoration technology can occur, and the method is difficult to be applied to the daily life of people. The advent of deep learning techniques has led to a great development in image restoration techniques, and researchers have begun to restore missing images using convolutional neural networks for encoding-decoding. Although the result of the image restored by the convolutional neural network is greatly improved, the problems of unclear restored image, incomplete semantics and the like still exist.
Disclosure of Invention
The invention aims to solve the technical problems of unclear restored image, incomplete semantics and the like in the conventional image restoration technology.
In order to achieve the above object, the present invention provides an image restoration method based on a generative countermeasure network, which is characterized by comprising the following steps:
firstly, training a network model according to a set iteration termination number, and saving parameters of a generator and a discriminator, training configuration, a structure, weight and the state of an optimizer of the network model after each 1000 times of training;
secondly, after the training of the network model is finished, repairing the image by using the generated network model;
finally, the restored image is compared with the original image.
Further, the network model training process includes the following steps:
step one, fixing a generator G1、G2Discriminator D2The weight parameter of (2); will supplement the image IcInput to the generator G1Generator G1Outputting reconstructed image I after encoding and decoding stagerecReconstructed image to be restored IrecInput to discriminator D1(ii) a Calculating discriminator D from countermeasure loss1Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training1The network parameter of (1);
step two, fixing the generator G1、G2Discriminator D1(ii) a Will occlude the image ImInput to the generator G2Generator G2Extracting characteristic information of image through coding stage and decodingOutputting generated image I after decoding in code stagegenGenerated image I to be restoredgenInput to discriminator D2(ii) a Calculating discriminator D from countermeasure loss2Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training2The network parameter of (1);
step three, fixing the discriminator D1、D2Generator G1(ii) a Will occlude the image ImInput to the generator G2Generator G2The image I is output and generated after the characteristic information of the image is extracted in the coding stage and the decoding stage is carried outgenGenerated image I to be restoredgenInput to discriminator D2(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss2Updating the generator G with an inverse gradient propagation algorithm2The network parameter of (2);
step four, fixing the discriminator D1、D2Generator G2(ii) a Will supplement the image IcInput to the generator G1Generator G1Outputting reconstructed image I after encoding and decoding stagerecReconstructed image to be restored IrecInput to discriminator D1(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss1Updating the generator G with an inverse gradient propagation algorithm1The network parameter of (2);
step five, calculating a total loss function to reduce the total function value;
and step six, repeating the step one to the step five until the preset iteration times are reached.
Further, the generators G1, G2 first extract feature information of the input image by the encoding section and then input the extracted image features to the decoding section to generate an image.
Further, the generators G1 and G2 have the same encoding portion, and the encoding portion includes 1 residual network R1 residual module and 6 residual network R2 residual modules.
Further, the generators G1, G2 have the same decoding portion, which includes 2 residual network R2 residual modules, 1 layer attention mechanism and 5 residual network R3 residual modules.
Furthermore, the network structures of the discriminators D1 and D2 are the same, and the discriminators comprise five layers of convolution, wherein the convolution kernel size used by the first three layers of convolution layers is 4 multiplied by 4, the sliding step size is 1, and the padding is 1; the convolution kernel size used by the last two convolutional layers is 4 × 4, the sliding step is 2, and the padding is 1.
Further, the loss function consists of three parts of appearance matching loss, KL divergence loss and confrontation loss; the formula for calculating the loss function is shown in equation 5:
Figure RE-BDA0003360258600000041
wherein, the alpha KL, the alpha app and the alpha ad represent hyper-parameters for adjusting the proportion of the KL loss, the appearance matching loss and the confrontation loss in the total loss function respectively.
Further, the KL divergence loss function is shown in formulas 1-a and 1-b
Figure RE-BDA0003360258600000042
Figure RE-BDA0003360258600000043
Wherein I represents the ith sample, IcRepresentation output generator G1Complementary image of (1)mRepresentation output generator G2The occlusion image of (a) is displayed,
Figure RE-BDA0003360258600000044
indicating a loss of KL of the reconstructed path,
Figure RE-BDA0003360258600000045
representing KL penalty of a generated path, z representing a potential vector, qψ(. l.) represents an importance sampling functionNumber, Nm(0，σ2(n) I) denotes obeying to a normal distribution, pφ(. l.) represents a conditional prior.
Further, the appearance matching loss function is shown in formulas 2-a and 2-b
Figure RE-BDA0003360258600000046
Figure RE-BDA0003360258600000047
Where i represents the ith sample, Igen represents the image generated by generator G2, Irec represents the image generated by generator G1, Ig represents the real image, and M represents the binary mask of visible pixels.
Further, the countering loss function consists of two parts of loss of the generator and loss of the discriminator;
the generator loss function is
LG＝E[D(G(z))] (3)
Wherein E (×) represents an expected value of the loss function, z represents random noise input to the generative model, g (z) represents a sample generated by the generative model, and D (g (z)) represents a result output when the input to the discriminant model is the generative sample;
the discriminator has a loss function of
Figure RE-BDA0003360258600000051
Figure RE-BDA0003360258600000052
Figure RE-BDA0003360258600000053
Where E (, denotes the expected value of the loss function and x denotes the data setRandomly selected samples, D (x) represents the output result when the input of the discriminant model is a real sample,
Figure RE-BDA0003360258600000054
the loss function corresponding to the WGAN discriminator is shown, Lgp is shown as a newly added gradient penalty loss function in the WGAN-GP, alpha is shown as a learning rate or a step factor of the adam optimization algorithm, and lambda is shown as a penalty coefficient.
The invention has the advantages that: the image repaired by the image repairing method based on the generated countermeasure network is clearer, natural and continuous, accords with the visual effect of human, shows that the image repairing result realized by the algorithm is better than other two algorithms, and proves the effectiveness of the algorithm.
The present invention will be described in detail below with reference to the accompanying drawings and examples.
Drawings
Fig. 1 is a network architecture diagram of a training phase.
FIG. 2 is a residual network R1The network architecture of (1).
FIG. 3 is a residual network R2The network architecture of (1).
FIG. 4 is a residual network R3The network architecture of (1).
Fig. 5 is a network configuration diagram of an encoder.
Fig. 6 is a network configuration diagram of a decoder.
Fig. 7 is a diagram of the structure of the discriminator.
FIG. 8a is an example of a broken image for intermediate occlusion image repair.
FIG. 8b is a repaired image example one of intermediate occlusion image repair.
Fig. 8c is a real image example one of intermediate occlusion image inpainting.
FIG. 9a is an example of a second broken image for intermediate occlusion image repair.
FIG. 9b is a repaired image example two of intermediate occlusion image repair.
FIG. 9c is a second example of a real image with intermediate occlusion image inpainting.
FIG. 10a is an example of a broken image of intermediate occlusion image repair III.
FIG. 10b is a repaired image example three of intermediate occlusion image repair.
FIG. 10c is a third example of a real image for intermediate occlusion image inpainting.
FIG. 11a is an example four of a broken image for intermediate occlusion image repair.
FIG. 11b is a repaired image example four of intermediate occlusion image repair.
FIG. 11c is a real image example four of intermediate occlusion image inpainting.
FIG. 12a is an example five of a broken image for intermediate occlusion image repair.
FIG. 12b is a repaired image example five of middle occlusion image repair.
FIG. 12c is an example of a real image of a middle occlusion image fix.
FIG. 13a is an example of a broken image for random occlusion image repair.
FIG. 13b is a repaired image example one of random occlusion image repair.
FIG. 13c is an example of a real image for random occlusion image inpainting.
FIG. 14a is an example of a second broken image for random occlusion image repair.
FIG. 14b is a repaired image example two of random occlusion image repair.
FIG. 14c is an example two of a real image for random occlusion image inpainting.
FIG. 15a is an example three of a broken image for random occlusion image repair.
FIG. 15b is example three of a repaired image of random occlusion image repair.
FIG. 15c is an example of a real image of random occlusion image inpainting three.
FIG. 16a is an example four of a broken image for random occlusion image repair.
FIG. 16b is a repaired image example four of random occlusion image repair.
FIG. 16c is an example of a real image of random occlusion image inpainting four.
FIG. 17a is an example five of a broken image for random occlusion image repair.
FIG. 17b is example five of a repaired image of random occlusion image repair.
FIG. 17c is an example of a real image for random occlusion image inpainting five.
FIG. 18a is an example of a patch discriminator fix-up image.
FIG. 18b is an example of a patch discriminator + WGAN-GP loss function inpainting image one.
Fig. 18c is a first example of a repaired image according to the method of the present application.
Fig. 18d is an original image example one.
FIG. 19a is a patch discriminator repaired image example two.
FIG. 19b is an example two of a patch discriminator + WGAN-GP loss function inpainting image.
Fig. 19c is a second example of a repaired image according to the method of the present application.
Fig. 19d is image example two.
FIG. 20a is a patch discriminator repaired image example three.
FIG. 20b is a patch discriminator + WGAN-GP loss function inpainting image example three.
Fig. 20c is a third example of a repaired image according to the method of the present application.
Fig. 20d is image example three.
FIG. 21a is a patch discriminator repaired image example four.
FIG. 21b is an example four of a patch discriminator + WGAN-GP loss function inpainting image.
Fig. 21c is an example four of a repaired image according to the method of the present application.
Fig. 21d is image example four.
Detailed Description
To further explain the technical means and effects of the present invention adopted to achieve the intended purpose, the following detailed description of the embodiments, structural features and effects of the present invention will be made with reference to the accompanying drawings and examples.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "aligned", "overlapping", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience of description and simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention.
The terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of the present invention, "a plurality" means two or more unless otherwise specified.
Example 1
The embodiment provides an image restoration method based on a generative countermeasure network as shown in fig. 1 to 7, which includes the following steps:
firstly, training a network model according to a set iteration termination number, and saving parameters of a generator and a discriminator, training configuration, a structure, weight and the state of an optimizer of the network model after each 1000 times of training;
secondly, after the training of the network model is finished, repairing the image by using the generated network model;
finally, the restored image is compared with the original image.
Further, the network model training process includes the following steps:
further, the network model training process includes the following steps:
step one, fixing a generator G1、G2Discriminator D2The weight parameter of (2); will supplement the image IcInput to the generator G1Generator G1Outputting reconstructed image I after encoding and decoding stagerecReconstructed image to be restored IrecInput to discriminator D1(ii) a Calculating discriminator D from countermeasure loss1Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training1The network parameter of (1);
step two, fixing the generator G1、G2Discriminator D1(ii) a Will occlude the image ImInput to the generator G2Generator G2The image I is output and generated after the characteristic information of the image is extracted in the coding stage and the decoding stage is carried outgenGenerated image I to be restoredgenInput to discriminator D2(ii) a Calculating discriminator D from countermeasure loss2Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training2The network parameter of (1);
step three, fixing the discriminator D1、D2Generator G1(ii) a Will occlude the image ImInput to the generator G2Generator G2The image I is output and generated after the characteristic information of the image is extracted in the coding stage and the decoding stage is carried outgenGenerated image I to be restoredgenInput to discriminator D2(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss2Updating the generator G with an inverse gradient propagation algorithm2The network parameter of (2);
step four, fixing the discriminator D1、D2Generator G2(ii) a Will supplement the image IcInput to the generator G1Generator G1Outputting reconstructed image I after encoding and decoding stagerecReconstructed image to be restored IrecInput deviceTo discriminator D1(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss1Updating the generator G with an inverse gradient propagation algorithm1The network parameter of (2);
step five, calculating a total loss function to reduce the total function value;
and step six, repeating the step one to the step five until the preset iteration times are reached.
Further, the generator G1、G2The feature information of the input image is extracted by the encoding section, and then the extracted image features are input to the decoding section to generate an image.
Further, the generator G1、G2Is identical, the coding part comprises 1 residual error network R1Residual module and 6 residual networks R2And a residual error module. The network structure of the encoder is shown in fig. 5.
Further, the generator G1、G2Is identical, said decoding portion comprises 2 residual networks R2Residual module, 1-layer attention mechanism and 5 residual networks R3And a residual error module. The network structure of the decoder is shown in fig. 6.
The long-short term Attention mechanism layer is developed by a Self-Attention (Self-Attention) mechanism, and not only the Self-Attention is used by a decoder layer to acquire finer texture features of an image, but also context features between an encoder and a decoder are further acquired, so that the capability of generating the texture of an image restoration model is improved. Finally, the short-term and long-term attention features are combined and input to the next decoding layer.
Residual error network R1The residual part of (2) consists of 2 convolutional layers, 1 activation function layer and 1 average pooling layer, and the jump connection part consists of 1 convolutional layer and 1 average pooling layer. Residual error network R1The network structure of (2) is shown in fig. 2.
Residual error network R1The convolution layer of the middle residual part uses a convolution kernel of 3 × 3 size, the step size of sliding is 1, and the padding size is 1. Activation function layer Using LeakyReLU performs the activation process. The average pooling layers of the residual part and the jump connection part both use convolution kernels of size 2 × 2, the step size of the sliding is 2, and there is no padding. The convolutional layer of the skip-join uses a convolution kernel of size 1 × 1, with a sliding step of 1 and no padding. Residual error network R1The detailed parameters of (a) are shown in table 1.
TABLE 1 residual error network R1Detailed parameter table of
Figure RE-BDA0003360258600000111
Residual error network R2The residual part of (2) consists of 2 convolutional layers and 2 activation functions, and the jump connection part consists of 1 convolutional layer. Residual error network R2The network structure of (2) is shown in fig. 3.
Residual error network R2The convolution layer of the middle residual part uses a convolution kernel of 3 × 3 size, the step size of sliding is 1, and the padding size is 1. The activation function layer uses LeakyReLU for activation processing. The convolutional layer of the skip-join uses a convolution kernel of size 1 × 1, with a sliding step of 1 and no padding. Residual error network R2The detailed parameters of (a) are shown in table 2.
TABLE 2 residual error network R2Detailed parameter table of
Figure RE-BDA0003360258600000112
Residual error network R3Including both residual and jump join portions. The residual part is composed of 2 normalization layers, 1 convolution layer, 2 activation functions and 1 deconvolution layer, and the jump link part is composed of 1 deconvolution layer. Residual error network R3The network structure of (2) is shown in fig. 4.
Residual error network R3The convolution layer of the middle residual part uses convolution kernels with the size of 3 multiplied by 3, the sliding step length is 1, and the filling size is 1. The deconvolution layers of the residual section and the jump junction section use transposed convolution of size 3 × 3, with sliding steps of 2 each and a padding size of 1. Activation function layer activation using LeakyReLUAnd (4) processing, namely performing normalization processing by using a spectrum normalization method. Residual error network R3The detailed parameters of (a) are shown in table 3.
TABLE 3 residual error network R3Detailed parameter table of
Figure RE-BDA0003360258600000121
Further, the discriminator D1，D2The network structure of the convolution layer is the same and comprises five layers of convolution, the convolution kernel size used by the first three layers of convolution layers is 4 multiplied by 4, the sliding step length is 1, and the filling is 1; the convolution kernel size used by the last two convolutional layers is 4 × 4, the sliding step is 2, and the padding is 1. The discriminator firstly extracts the characteristics of the input image, then analyzes and compares the extracted characteristics and outputs a discrimination result. The network structure of the arbiter is shown in fig. 7.
Furthermore, the loss function is used for guiding the learning of the model by evaluating the inconsistency degree of the generated image of the image restoration model and the real image, and the smaller the loss function is, the better the robustness of the representative model is. Because the network structure of the technical scheme of the patent application is dual-channel, each loss item is also dual-channel. Appearance matching loss is used for restricting the generation of an image structure, KL divergence loss mainly enables the distribution of a high-dimensional space to be close, and countermeasures loss is used for reducing the difference between a generated image and a real image.
The loss function consists of appearance matching loss, KL divergence loss and confrontation loss; the formula for calculating the loss function is shown in equation 5:
Figure RE-BDA0003360258600000131
wherein, the alpha KL, the alpha app and the alpha ad represent hyper-parameters for adjusting the proportion of the KL loss, the appearance matching loss and the confrontation loss in the total loss function respectively.
Furthermore, the technical scheme uses the KL loss to enable the generated path to be close to the high-dimensional space standard normal distribution of the reconstruction path, and two KL loss functions are correspondingly arranged due to the fact that the two paths are a dual-path framework.
The KL divergence loss function is
Figure RE-BDA0003360258600000132
Figure RE-BDA0003360258600000133
Wherein I represents the ith sample, IcRepresentation output generator G1Complementary image of (1)mRepresentation output generator G2The occlusion image of (a) is displayed,
Figure RE-BDA0003360258600000134
indicating a loss of KL of the reconstructed path,
Figure RE-BDA0003360258600000135
representing KL penalty of a generated path, z representing a potential vector, qψ(. l.) represents an importance sampling function, Nm(0，σ2(n) I) denotes obeying to a normal distribution, pφ(. l.) represents a conditional prior.
Further, in order to increase the ability of the generator to generate images, the image features extracted during the encoding stage are input to a decoder to generate a reconstructed image, where the appearance matching loss is used to constrain the generation of the image structure.
The appearance matching loss function is
Figure RE-BDA0003360258600000136
Figure RE-BDA0003360258600000137
Where i represents the ith sample, Igen represents the image generated by generator G2, Irec represents the image generated by generator G1, Ig represents the real image, and M represents the binary mask of visible pixels.
Further, the technical scheme uses a loss function of WGAN-GP to optimize the network structure, and the countermeasure loss function is composed of the loss of the generator and the loss of the discriminator;
the generator loss function is
LG＝E[D(G(z))] (3)
Wherein E (×) represents an expected value of the loss function, z represents random noise input to the generative model, g (z) represents a sample generated by the generative model, and D (g (z)) represents a result output when the input to the discriminant model is the generative sample;
the discriminator has a loss function of
Figure RE-BDA0003360258600000144
Figure RE-BDA0003360258600000141
Figure RE-BDA0003360258600000142
Where E (×) represents the expected value of the loss function, x represents a randomly chosen sample in the data set, d (x) represents the result output when the input to the discriminant model is a true sample,
Figure RE-BDA0003360258600000143
representing the corresponding loss function, L, of the WGAN discriminatorgpAnd a newly added gradient penalty loss function in the WGAN-GP is represented, alpha represents the learning rate or step factor of the adam optimization algorithm, and lambda represents a penalty coefficient.
The invention has the advantages that: the image repaired by the image repairing method based on the generated countermeasure network is clearer, natural and continuous, accords with the visual effect of human, shows that the image repairing result realized by the algorithm is better than other two algorithms, and proves the effectiveness of the algorithm.
Fig. 8 to 12 are images of middle blocks, and images restored by an image restoration method based on a spanning tree, where fig. 8(a), 9(a), 10(a), 11(a), and 12 (a) each show a damaged image, fig. 8(b), 9(b), 10(b), 11(b), and 12(b) each show an image after restoration, and fig. 8(c), 9(c), 10(c), 11(c), and 12 (c) show a real image. After the technical scheme of the patent application is used for image restoration algorithm restoration, the structure is reasonable, the texture is continuous, and no obvious difference exists between the texture and a real image.
Fig. 13 to 17 are images which are randomly blocked and which are restored by an image restoration method based on a spanning tree, in which fig. 13(a), 14(a), 15(a), 16(a), and 17(a) each show a damaged image, fig. 13(b), 14(b), 15(b), 16(b), and 17(b) each show an image after restoration, and fig. 13(c), 14(c), 15(c), 16(c), and 17(c) show a real image. After the technical scheme of the patent application is used for image restoration algorithm restoration, the image restoration algorithm is natural and clear, and the texture is continuous, so that restoration is more vivid and is close to a real image.
The image restoration result achieved by the method of the present embodiment is compared with the image restoration result achieved using the patch discriminator and the image restoration result achieved using the patch discriminator + WGAN-GP loss function. The experimental pair of the Patch discriminator, the Patch discriminator + WGAN-GP loss function and the image restoration part realized by the algorithm in the invention is shown in FIGS. 18 to 21.
Fig. 18(a), 19(a), 20(a), and 21(a) are graphs of results of image restoration part experiments implemented using the patch discriminator algorithm, and fig. 18(b), 19(b), 20(b), and 21(b) are graphs of results of image restoration part experiments implemented using the patch discriminator + WGAN loss function algorithm. Fig. 18(c), fig. 19(c), fig. 20(c), fig. 21(c) are all graphs of experimental results of the image restoration portion achieved using the algorithm herein. Fig. 18(d), 19(d), 20(d), and 21(d) are real images.
The qualitative and quantitative evaluation results are integrated to obtain, and the method for restoring the deep learning image provided by the invention uses a patch discriminator to replace an original residual discriminator, so that the discrimination network can more accurately identify whether the input image is a generated image or a real image. The impedance loss uses WAGN-GP loss to optimize the network structure, so that the network parameter convergence is faster. The image repaired by the algorithm is clearer, natural and continuous, and accords with the visual effect of human, so that the image repairing result realized by the method is better than that of other two algorithms, and the effectiveness of the method is proved.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.

Claims (10)

1. An image restoration method based on a generation countermeasure network is characterized by comprising the following steps:
firstly, training a network model according to a set iteration termination number, and saving parameters of a generator and a discriminator, training configuration, a structure, weight and the state of an optimizer of the network model after each 1000 times of training;
secondly, after the training of the network model is finished, repairing the image by using the generated network model;
finally, the restored image is compared with the original image.
2. The image inpainting method based on the generation of the countermeasure network as claimed in claim 1, wherein the network model training process comprises the following steps:
step one, fixing a generator G1、G2Discriminator D2The weight parameter of (2); will supplement the image IcInput to the generator G1Generator G1Output after encoding and decoding stageReconstructed image IrecReconstructed image to be restored IrecInput to discriminator D1(ii) a Calculating discriminator D from countermeasure loss1Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training1The network parameter of (1);
step two, fixing the generator G1、G2Discriminator D1(ii) a Will occlude the image ImInput to the generator G2Generator G2The image I is output and generated after the characteristic information of the image is extracted in the coding stage and the decoding stage is carried outgenGenerated image I to be restoredgenInput to discriminator D2(ii) a Calculating discriminator D from countermeasure loss2Updating discriminator D with inverse gradient propagation algorithm in loss function value during network training2The network parameter of (1);
step three, fixing the discriminator D1、D2Generator G1(ii) a Will occlude the image ImInput to the generator G2Generator G2The image I is output and generated after the characteristic information of the image is extracted in the coding stage and the decoding stage is carried outgenGenerated image I to be restoredgenInput to discriminator D2(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss2Updating the generator G with an inverse gradient propagation algorithm2The network parameter of (2);
step four, fixing the discriminator D1、D2Generator G2(ii) a Will supplement the image IcInput to the generator G1Generator G1Outputting reconstructed image I after encoding and decoding stagerecReconstructed image to be restored IrecInput to discriminator D1(ii) a Generator G is calculated from image appearance matching loss, countermeasure loss, and KL loss1Updating the generator G with an inverse gradient propagation algorithm1The network parameter of (2);
step five, calculating a total loss function to reduce the total function value;
and step six, repeating the step one to the step five until the preset iteration times are reached.
3. The image restoration method based on the generation countermeasure network as claimed in claim 2, wherein: institute generator G1、G2The feature information of the input image is extracted by the encoding section, and then the extracted image features are input to the decoding section to generate an image.
4. The image restoration method based on the generation countermeasure network as claimed in claim 3, wherein: the generator G1、G2Is identical, the coding part comprises 1 residual error network R1Residual module and 6 residual networks R2And a residual error module.
5. An image restoration method based on a generation countermeasure network as claimed in claim 3, characterized in that: the generator G1、G2Is identical, said decoding portion comprises 2 residual networks R2Residual module, 1-layer attention mechanism and 5 residual networks R3And a residual error module.
6. The image restoration method based on the generation countermeasure network as claimed in claim 2, wherein: the discriminator D1，D2The network structure of the convolution layer is the same and comprises five layers of convolution, the convolution kernel size used by the first three layers of convolution layers is 4 multiplied by 4, the sliding step length is 1, and the filling is 1; the convolution kernel size used by the last two convolutional layers is 4 × 4, the sliding step is 2, and the padding is 1.
7. The image restoration method based on the generation countermeasure network as claimed in claim 2, wherein: the loss function consists of appearance matching loss, KL divergence loss and confrontation loss; the formula for calculating the loss function is shown in equation 5:
Figure FDA0003112742050000031
wherein, the alpha KL, the alpha app and the alpha ad represent hyper-parameters for adjusting the proportion of the KL loss, the appearance matching loss and the confrontation loss in the total loss function respectively.
8. The image inpainting method based on the generation countermeasure network as claimed in claim 7, wherein: the KL divergence loss function is shown in formulas 1-a and 1-b
Figure FDA0003112742050000032
Figure FDA0003112742050000033
Wherein I represents the ith sample, IcRepresentation output generator G1Complementary image of (1)mRepresentation output generator G2The occlusion image of (a) is displayed,
Figure FDA0003112742050000034
indicating a loss of KL of the reconstructed path,
Figure FDA0003112742050000035
representing KL penalty of a generated path, z representing a potential vector, qψ(. l.) represents an importance sampling function, Nm(0，σ2(n) I) denotes obeying to a normal distribution, pφ(. l.) represents a conditional prior.
9. The image inpainting method based on the generation countermeasure network as claimed in claim 7, wherein: the appearance matching loss function is shown in formulas 2-a and 2-b
Figure FDA0003112742050000036
Figure FDA0003112742050000037
Wherein I denotes the ith sample, IgenRepresentation generator G2Generated image, IrecRepresentation generator G1Generated image, IgRepresenting a real image and M representing a binary mask of visible pixels.
10. The image inpainting method based on the generation countermeasure network as claimed in claim 7, wherein: the loss resisting function consists of two parts of loss of a generator and loss of a discriminator;
the generator loss function is
LG＝E[D(G(z))] (3)
Wherein E (×) represents an expected value of the loss function, z represents random noise input to the generative model, g (z) represents a sample generated by the generative model, and D (g (z)) represents a result output when the input to the discriminant model is the generative sample;
the discriminator has a loss function of
Figure FDA0003112742050000041
Figure FDA0003112742050000044
Figure FDA0003112742050000042
Where E (×) represents the expected value of the loss function, x represents a randomly chosen sample in the data set, d (x) represents the result output when the input to the discriminant model is a true sample,
Figure FDA0003112742050000043
representing the corresponding loss function, L, of the WGAN discriminatorgpAnd a newly added gradient penalty loss function in the WGAN-GP is represented, alpha represents the learning rate or step factor of the adam optimization algorithm, and lambda represents a penalty coefficient.
CN202110653410.6A 2021-06-11 2021-06-11 Image restoration method based on generation countermeasure network Active CN113837953B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110653410.6A CN113837953B (en) 2021-06-11 2021-06-11 Image restoration method based on generation countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110653410.6A CN113837953B (en) 2021-06-11 2021-06-11 Image restoration method based on generation countermeasure network

Publications (2)

Publication Number Publication Date
CN113837953A true CN113837953A (en) 2021-12-24
CN113837953B CN113837953B (en) 2024-04-12

Family

ID=78962651

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110653410.6A Active CN113837953B (en) 2021-06-11 2021-06-11 Image restoration method based on generation countermeasure network

Country Status (1)

Country Link
CN (1) CN113837953B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131453A (en) * 2022-05-17 2022-09-30 广西北投信创科技投资集团有限公司 Color filling model training method, color filling device, and electronic equipment
CN116523985A (en) * 2023-05-06 2023-08-01 兰州交通大学 Structure and texture feature guided double-encoder image restoration method

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559287A (en) * 2018-11-20 2019-04-02 北京工业大学 A kind of semantic image restorative procedure generating confrontation network based on DenseNet
WO2020168731A1 (en) * 2019-02-19 2020-08-27 华南理工大学 Generative adversarial mechanism and attention mechanism-based standard face generation method
US20210073630A1 (en) * 2019-09-10 2021-03-11 Robert Bosch Gmbh Training a class-conditional generative adversarial network
CN112541864A (en) * 2020-09-25 2021-03-23 中国石油大学(华东) Image restoration method based on multi-scale generation type confrontation network model

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109559287A (en) * 2018-11-20 2019-04-02 北京工业大学 A kind of semantic image restorative procedure generating confrontation network based on DenseNet
WO2020168731A1 (en) * 2019-02-19 2020-08-27 华南理工大学 Generative adversarial mechanism and attention mechanism-based standard face generation method
US20210073630A1 (en) * 2019-09-10 2021-03-11 Robert Bosch Gmbh Training a class-conditional generative adversarial network
CN112541864A (en) * 2020-09-25 2021-03-23 中国石油大学(华东) Image restoration method based on multi-scale generation type confrontation network model

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
孙全;曾晓勤;: "基于生成对抗网络的图像修复", 计算机科学, no. 12 *
李天成;何嘉;: "一种基于生成对抗网络的图像修复算法", 计算机应用与软件, no. 12 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131453A (en) * 2022-05-17 2022-09-30 广西北投信创科技投资集团有限公司 Color filling model training method, color filling device, and electronic equipment
CN116523985A (en) * 2023-05-06 2023-08-01 兰州交通大学 Structure and texture feature guided double-encoder image restoration method
CN116523985B (en) * 2023-05-06 2024-01-02 兰州交通大学 Structure and texture feature guided double-encoder image restoration method

Also Published As

Publication number Publication date
CN113837953B (en) 2024-04-12

Similar Documents

Publication Publication Date Title
CN109785258B (en) Face image restoration method based on multi-discriminator generated countermeasure network
CN109377452B (en) Face image restoration method based on VAE and generation type countermeasure network
CN108460746B (en) Image restoration method based on structure and texture layered prediction
Bertalmio et al. Simultaneous structure and texture image inpainting
CN111783566B (en) Video synthesis method based on lip synchronization and enhancement of mental adaptation effect
CN111815523A (en) Image restoration method based on generation countermeasure network
CN113837953B (en) Image restoration method based on generation countermeasure network
CN110706303B (en) Face image generation method based on GANs
CN110610464A (en) Face image super-resolution method based on dense residual error neural network
CN112686817B (en) Image completion method based on uncertainty estimation
CN111583135A (en) Nuclear prediction neural network Monte Carlo rendering image denoising method
CN113298734B (en) Image restoration method and system based on mixed hole convolution
CN112381716B (en) Image enhancement method based on generation type countermeasure network
CN112686816A (en) Image completion method based on content attention mechanism and mask code prior
CN110689495A (en) Image restoration method for deep learning
CN113723174B (en) Face image super-resolution restoration and reconstruction method and system based on generation countermeasure network
CN111787187A (en) Method, system and terminal for repairing video by utilizing deep convolutional neural network
CN111768326A (en) High-capacity data protection method based on GAN amplification image foreground object
CN114092354A (en) Face image restoration method based on generation countermeasure network
Liu et al. Facial image inpainting using multi-level generative network
CN114694176A (en) Lightweight human body posture estimation method based on deep learning
Liu Literature review on image restoration
CN113487512A (en) Digital image restoration method and device based on edge information guidance
CN116523985B (en) Structure and texture feature guided double-encoder image restoration method
CN116051407A (en) Image restoration method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant