CN113409216A - Image restoration method based on frequency band self-adaptive restoration model - Google Patents
Image restoration method based on frequency band self-adaptive restoration model Download PDFInfo
- Publication number
- CN113409216A CN113409216A CN202110707809.8A CN202110707809A CN113409216A CN 113409216 A CN113409216 A CN 113409216A CN 202110707809 A CN202110707809 A CN 202110707809A CN 113409216 A CN113409216 A CN 113409216A
- Authority
- CN
- China
- Prior art keywords
- image
- frequency
- band
- frequency sub
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 230000009466 transformation Effects 0.000 claims abstract description 7
- 230000006870 function Effects 0.000 claims description 24
- 238000012549 training Methods 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 12
- 238000009826 distribution Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 230000008439 repair process Effects 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 230000008485 antagonism Effects 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 238000005315 distribution function Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000003044 adaptive effect Effects 0.000 claims 4
- 206010061619 Deformity Diseases 0.000 claims 1
- 238000010276 construction Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 10
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003252 repetitive effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20064—Wavelet transform [DWT]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention relates to an image restoration method based on a frequency band self-adaptive restoration model, which is used for solving the problem that the prior art cannot simultaneously reconstruct the reasonable structure and the fine texture of a damaged image. The invention takes an encoder-decoder network structure as a baseline network of a generator on the basis of generating a confrontation network model. To extract deep information from known regions, the corrupted image features are decomposed into low-frequency and high-frequency subbands by Discrete Wavelet Transform (DWT), and the multi-frequency subbands are then passed through a convolutional network to extract features. And finally, reconstructing each horizontal sub-band image through wavelet inverse transformation, and complementing the missing region information to generate an image. The idea of processing low frequency information separately from high frequency information is advantageous for processing the structure and texture information of the image in a targeted manner. The image generated by the method has a clearer structure and finer texture, and the repaired image is more real and has no obvious boundary.
Description
The technical field is as follows:
the invention relates to the field of computer image processing, in particular to an image restoration method based on a frequency band self-adaptive restoration model.
Background art:
image inpainting is a basic task in multimedia applications and computer vision, with the goal of generating alternative global semantic structures and local detail textures for missing regions, and ultimately producing visually realistic results. It is widely applied in the multimedia fields of image editing, restoration, synthesis and the like. The conventional image block-based image inpainting method is to search and copy the best matching image block from a known area to the missing area. The traditional image restoration method has a good processing effect on static textures, but has a limited processing effect on textures of complex or non-repetitive structures such as human faces and the like, and is not suitable for capturing high-level semantic information.
In recent years, modeling image inpainting as a conditional generation problem based on a learning approach, Pathak et al first train a deep neural network with a penalty-fighting function to predict missing regions, which is advantageous for capturing the edges and global structure of large-area missing regions. Ishikawa et al improve it by combining global and local penalty functions to produce finer textures. The deep features are extracted and transmitted through the convolutional neural network, so that the defects of the traditional image restoration algorithm are overcome better, and visually real and reasonable restoration results are obtained by the methods. However, since these methods treat and process the structure and texture information of the input image equally, an edge over-smoothing or texture phenomenon often occurs.
To address this problem, Liu et al propose a two-stage network, recovering the coarse structure of the missing region in the first stage, and generating the final result using the reconstruction information of the first stage in the second stage. However, the second-stage network depends greatly on the correctness of the reconstruction structure of the first-stage network, and the two-stage training also brings additional computational burden. Meanwhile, the data distribution of the low-frequency characteristic and the high-frequency characteristic in the input image are completely different. If the feature distributions of different frequencies are computed indiscriminately, reconstruction of the structure or generation of texture may be misled.
In summary, the existing image restoration algorithm often cannot reconstruct a reasonable structure and a fine texture at the same time, and has limitations.
Disclosure of Invention
In order to solve the problem that the prior art cannot reconstruct the reasonable structure and fine texture of a damaged image at the same time, the invention provides an image restoration method based on a frequency band self-adaptive restoration model.
The image restoration method based on the frequency band self-adaptive restoration model of the invention takes a coder-decoder network structure as a baseline network of a generator on the basis of generating a confrontation network model. To extract deep information from known regions, we decompose features into low-frequency and high-frequency subbands by Discrete Wavelet Transform (DWT), and then extract features from the multi-frequency subbands by a convolutional network. And finally, reconstructing each horizontal sub-band image through wavelet inverse transformation, and complementing the missing region information to generate an image. The idea of processing low frequency information separately from high frequency information is advantageous for processing the structure and texture information of the image in a targeted manner.
The following is a description of the belief in both training and actual measurements:
the training phase comprises the following steps:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: inputting the multi-frequency sub-band feature representation extracted by the encoder in the step two into a decoder to obtain multi-frequency sub-band feature representation of a generated image, and finally obtaining a repaired image through inverse discrete wavelet transform;
step four: inputting the generated image in the third step into a discriminator network, iteratively adjusting generator parameters through an overall loss function until the loss function is converged and the generator parameters reach an optimal value, and stopping training;
the specific operation of each step is as follows:
the specific operation of the step one is as follows:
aiming at the damaged image x, the image is divided into multiple frequencies by adopting discrete wavelet transformSub-bands, we use in the present invention a Haar wavelet transform, using 4 convolution filters, including the low frequency filterAnd 3 high-pass filtersAnditeratively decomposing an image into 4 multi-frequency subband images x with a convolution step size of 2sub={xLL,xHL,xLH,xHH}:
In the formula,for convolution operation, ↓2represents a standard down-sampling operator with a factor of 2, and {. is a collocation operation.
Low frequency sub-band: x is the number ofL=xLLAnd, high-frequency sub-band: x is the number ofH={xHL,xLH,xHH}。
The specific operation of the second step:
because the low-frequency sub-band frequency transformation is slow, in the low-frequency sub-band feature extraction, the model uses an encoder consisting of a first convolution block, a second convolution block, a first residual convolution block, a second residual convolution block, a third convolution block and a fourth residual convolution block to extract features; on the contrary, the high-frequency sub-band frequency conversion is fast, the model uses the encoder consisting of the first residual convolution block, the second residual convolution block, the third residual convolution block, the fourth residual convolution block, the fifth residual convolution block, the first residual convolution block and the sixth residual convolution block to extract the characteristics, and different encoder structures are adopted to be beneficial to the extraction of the characteristics of the high-frequency sub-band and the low-frequency sub-band.
The third step of concrete operation:
the decoder is composed of six serial attention-embedded devicesResidual convolution blocks of modules with an embedded attention module embedded behind each residual convolution block, x 'for each residual convolution block embedded with an attention module'subRepresenting the output of the residual volume block, the working process of the attention module is as follows:
first, an attention matrix M is generated:
where σ represents sigmoid activation function, ki(i ═ 1,2) represents different convolution operations, with a filter size of 1 × 1;
then, output x 'of the residual convolution block is passed through attention matrix M'subThe matrix multiplication of (a) yields a weighted representation of the feature x ″)subThe global attention model is expressed as follows:
x″sub=M*x′sub (2)
wherein denotes multiplication of corresponding elements.
Step four, constructing an integral loss function:
the training process for the entire network can be written as:
Iout=IDWT[Ggen(xL,xH)] (3)
wherein, IoutIndicating the repair result of the network, IDWT [ ·]Representing an inverse discrete wavelet transform, xL,xHHigh and low frequency subbands representing a damaged image, GgenRepresenting a generating network.
First, a reconstruction loss function is defined as a prediction result IoutAnd a real image IgtL1 distance between:
Lr=‖Iout-Igt‖1 (4)
to make the features of the real image and the reconstructed image closer in the discriminator, we use the antagonism constraint, the antagonism loss function LadvThe definition is as follows:
in the formula,expressing the expected value, p, of the distribution functiondata(Igt) Representing the true sample distribution, prec(Iout) Indicating the distribution of the generated image, D indicating the discriminator, G being Ggen。
Finally, in order to keep the consistency of the content and style of the generated image and the real image, a VGG-16 network trained in advance by ImageNet is used for extracting a high-level feature space, and the texture loss L is reducedtIs defined as follows:
Lt=‖Gram{Φ(Iout)}-Gram{Φ(Igt)}‖2+λ‖Φ(Iout)-Φ(Igt)‖2 (6)
wherein phi represents a feature representation extracted by a VGG-16 network trained in advance by ImageNet, and Gram represents a Gram matrix operation.
Considering reconstruction loss, countermeasure loss, and texture loss, the overall loss function that defines the generated network is defined as:
Ltotal=λrLr+λadvLadv+λtLt (7)
wherein λ isr、λadv、λtRepresenting the weight coefficients.
In the actual measurement stage, the repaired image is obtained by using the generated network, and the steps are as follows:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: and inputting the multi-frequency sub-band feature representation extracted by the encoder in the step two into a decoder, obtaining the multi-frequency sub-band feature representation of the generated image through residual convolution operation of a plurality of embedded attention modules, and finally obtaining the repaired image through inverse discrete wavelet transformation, wherein the obtained image has strong reality and clear texture.
Compared with the prior art, the method decomposes the characteristics into high-frequency and low-frequency sub-bands through frequency division processing, and extracts and transmits the depth information in the high-frequency sub-band region and the low-frequency sub-band region respectively. The idea of processing the low-frequency information and the high-frequency information separately is beneficial to processing the structure and texture information of the image in a targeted manner, and has the beneficial effects that: the model not only can synthesize a clear image structure, but also can generate fine textures in a missing area, and is obviously superior to the most advanced method.
Description of the drawings:
FIG. 1 is a technical framework diagram of a band-adaptive repair model;
FIG. 2 is an exemplary graph of repair results on a data set;
FIG. 3 compares visual results with different algorithms;
the specific implementation mode is as follows:
in order to more clearly describe the technical contents of the present invention, the following is further described with reference to specific examples:
in the image restoration technology frame diagram based on the frequency band self-adaptive restoration model in fig. 1, firstly, a discrete wavelet transform is used for dividing a damaged image into a high-frequency sub-band and a low-frequency sub-band, different convolution blocks are adopted to respectively carry out deep feature extraction on the high-frequency sub-band and the low-frequency sub-band, then, the extracted high-frequency sub-band and the extracted low-frequency sub-band are spliced and input into a decoder, in the decoder part, a structure combining a residual block and the convolution blocks is adopted for feature restoration, an attention mechanism is utilized to further extract and transmit key features, a final multi-frequency representation of a restoration result is obtained, and a restored image is obtained after inverse wavelet transform.
The following is a description of the belief in both training and actual measurements:
the training phase comprises the following steps:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: inputting the multi-frequency sub-band feature representation extracted by the encoder in the step two into a decoder to obtain multi-frequency sub-band feature representation of a generated image, and finally obtaining a repaired image through inverse discrete wavelet transform;
step four: inputting the generated image in the third step into a discriminator network, iteratively adjusting generator parameters through an overall loss function until the loss function is converged and the generator parameters reach an optimal value, and stopping training;
the specific operation of each step is as follows:
the specific operation of the step one is as follows:
for damaged image x, the image is divided into multi-frequency sub-bands by discrete wavelet transform, and in the invention we use Haar wavelet transform, using 4 convolution filters, including low-frequency filterAnd 3 high-pass filtersAnditeratively decomposing an image into 4 multi-frequency subband images x with a convolution step size of 2sub={xLL,xHL,xLH,xHH}:
In the formula,for convolution operation, ↓2represents a standard down-sampling operator with a factor of 2, and {. is a collocation operation.
Low frequency sub-band: x is the number ofL=xLLAnd, high-frequency sub-band: x is the number ofH={xHL,xLH,xHH}。
The specific operation of the second step:
because the low-frequency sub-band frequency transformation is slow, in the low-frequency sub-band feature extraction, the model uses an encoder consisting of a first convolution block, a second convolution block, a first residual convolution block, a second residual convolution block, a third convolution block and a fourth residual convolution block to extract features; on the contrary, the high-frequency sub-band frequency conversion is fast, the model uses the encoder consisting of the first residual convolution block, the second residual convolution block, the third residual convolution block, the fourth residual convolution block, the fifth residual convolution block, the first residual convolution block and the sixth residual convolution block to extract the characteristics, and different encoder structures are adopted to be beneficial to the extraction of the characteristics of the high-frequency sub-band and the low-frequency sub-band.
The third step of concrete operation:
the decoder consists of six concatenated residual convolution blocks with embedded attention module, embedded attention module behind each residual convolution block, x 'for each residual convolution block embedded attention module'subRepresenting the output of the residual volume block, the working process of the attention module is as follows:
first, an attention matrix M is generated:
where σ represents sigmoid activation function, ki(i ═ 1,2) represents different convolution operations, with a filter size of 1 × 1;
then, output x 'of the residual convolution block is passed through attention matrix M'subThe matrix multiplication of (a) yields a weighted representation of the feature x ″)subIntegral attention modelThe type is represented as follows:
x″sub=M*x′sub (2)
wherein denotes multiplication of corresponding elements.
Step four, constructing an integral loss function:
the training process for the entire network can be written as:
Iout=IDWT[Ggen(xL,xH)] (3)
wherein, IoutIndicating the repair result of the network, IDWT [ ·]Representing an inverse discrete wavelet transform, xL,xHHigh and low frequency subbands representing a damaged image, GgenRepresenting a generating network.
First, a reconstruction loss function is defined as a prediction result IoutAnd a real image IgtL1 distance between:
Lr=‖Iout-Igt‖1 (4)
to make the features of the real image and the reconstructed image closer in the discriminator, we use the antagonism constraint, the antagonism loss function LadvThe definition is as follows:
in the formula,expressing the expected value, p, of the distribution functiondata(Igt) Representing the true sample distribution, prec(Iout) Indicating the distribution of the generated image, D indicating the discriminator, G being Ggen。
Finally, in order to keep the consistency of the content and style of the generated image and the real image, a VGG-16 network trained in advance by ImageNet is used for extracting a high-level feature space, and the texture loss L is reducedtIs defined as follows:
Lt=‖Gram{Φ(Iout)}-Gram{Φ(Igt)}‖2+λ‖Φ(Iout)-Φ(Igt)‖2 (6)
wherein phi represents a feature representation extracted by a VGG-16 network trained in advance by ImageNet, and Gram represents a Gram matrix operation.
Considering reconstruction loss, countermeasure loss, and texture loss, the overall loss function that defines the generated network is defined as:
Ltotal=λrLr+λadvLadv+λtLt (7)
wherein λ isr、λadv、λtRepresenting the weight coefficients.
In the actual measurement stage, the repaired image is obtained by using the generated network, and the steps are as follows:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: and inputting the multi-frequency sub-band feature representation extracted by the encoder in the step two into a decoder, obtaining the multi-frequency sub-band feature representation of the generated image through residual convolution operation of a plurality of embedded attention modules, and finally obtaining the repaired image through inverse discrete wavelet transformation, wherein the obtained image has strong reality and clear texture.
An exemplary result of the present invention is shown in fig. 2, where (a) represents an input image and (b) represents a repair result of the present invention and (c) represents a group Truth image.
And (3) evaluating the image quality:
FIG. 3 is a comparison of the visual results of the present invention and Context Encoder, Context attribute, and PICNet on the repaired image of the central region. Where Context Encoders (CEs) produce warped structures and blurred results. The Contextual Attention (CA) exhibits a warped structure. PICNet is intended to produce a wide variety of authentic images, but sometimes produces repetitive and structurally distorted images. The present invention deals better with these problems and produces more intuitive and realistic results than these methods. We also performed quantitative comparisons using common evaluation indices. Table 1 shows that our process achieves the best performance.
TABLE 1 Objective quality comparison of different algorithms
Claims (5)
1. An image restoration method based on a frequency band self-adaptive restoration model is characterized in that a baseline network with an encoder-decoder network structure as a generator is used on the basis of generating a confrontation network model, the image restoration method comprises a training phase and a prediction restoration phase,
the training phase comprises the following steps:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: inputting the multi-frequency sub-band feature representation extracted by the second encoder into a decoder to obtain the multi-frequency sub-band feature representation of the generated image, and finally obtaining the repaired image through inverse discrete wavelet transform, namely a prediction result Iout;
Step four: inputting the generated image in the third step into a discriminator network, iteratively adjusting generator parameters through an overall loss function until the loss function is converged and the generator parameters reach an optimal value, and stopping training;
the predicted repair phase comprises the following steps:
the method comprises the following steps: decomposing the damaged image into a low-frequency sub-band and a high-frequency sub-band through Discrete Wavelet Transform (DWT);
step two: directly inputting the low-frequency sub-band obtained in the step one into a low-frequency feature encoder, splicing the high-frequency sub-band in the channel direction and inputting the high-frequency sub-band into a high-frequency feature encoder, and finally cascading the features extracted from the high-frequency encoder and the low-frequency encoder together in the channel direction to obtain multi-frequency sub-band feature representation of the image;
step three: and inputting the multi-frequency sub-band feature representation extracted by the encoder in the step two into a decoder, obtaining the multi-frequency sub-band feature representation of the generated image through residual convolution operation of a plurality of embedded attention modules, and finally obtaining the repaired image through inverse discrete wavelet transformation, wherein the obtained image has strong reality and clear texture.
2. The image restoration method based on the band adaptive restoration model according to claim 1, wherein:
the specific operation of the first training phase is as follows,
for damaged image x, discrete wavelet transform is used to divide the image into multiple frequency sub-bands, specifically Haar wavelet transform is used, and 4 convolution filters are used, including low-frequency filterAnd 3 high-pass filtersAnditeratively decomposing an image into 4 multi-frequency subband images x with a convolution step size of 2sub={xLL,xHL,xLH,xHH}:
In the formula,for convolution operation, ↓2represents a standard down-sampling operator with a factor of 2, and {. is collocation operation;
low frequency sub-band: x is the number ofL=xLLAnd, high-frequency sub-band: x is the number ofH={xHL,xLH,xHH}。
3. The image restoration method based on the band adaptive restoration model according to claim 1, wherein:
the low-frequency feature encoder in the step two of the training stage consists of a first convolution block, a second convolution block, a first residual convolution block, a second residual convolution block, a third convolution block and a fourth residual convolution block in sequence; the high-frequency characteristic encoder is sequentially composed of a first residual convolution block, a second residual convolution block, a third residual convolution block, a fourth residual convolution block, a fifth residual convolution block, a sixth residual convolution block, and different encoder structures are adopted to facilitate extraction of high-frequency and low-frequency sub-band characteristics.
4. The image restoration method based on the band adaptive restoration model according to claim 1, wherein:
the decoder consists of six concatenated residual convolution blocks with embedded attention module, embedded attention module behind each residual convolution block, x 'for each residual convolution block embedded attention module'subRepresenting the output of the residual volume block, the working process of the attention module is as follows:
first, an attention matrix M is generated:
where σ represents sigmoid activation function, ki(i ═ 1,2) represents different convolution operations, with a filter size of 1 × 1;
then, the user can use the device to perform the operation,output x 'of the residual convolution block by attention matrix M'subThe matrix multiplication of (a) yields a weighted representation of the feature x ″)subThe global attention model is expressed as follows:
x″sub=M*x′sub (2)
wherein denotes multiplication of corresponding elements.
5. The image restoration method based on the band adaptive restoration model according to claim 1, wherein:
the overall loss function in the fourth step is constructed as follows:
first, a reconstruction loss function is constructed, which is defined as a prediction result IoutAnd a real image IgtL1 distance between:
Lr=||Iout-Igt||1 (4)
wherein,
Iout=IDWT[Ggen(xL,xH)] (3)
wherein, IoutIndicating the repair result of the network, IDWT [ ·]Representing an inverse discrete wavelet transform, xL,xHHigh and low frequency subbands representing a damaged image, GgenThe representation is generated as a network of networks,
construction of the antagonism loss function LadvFor approximating the characteristics of the real image and the reconstructed image in the discriminator, a disfigurement function LadvThe definition is as follows:
in the formula,expressing the expected value, p, of the distribution functiondata(Igt) Representing the true sample distribution, prec(Iout) Representing the generated image distribution, D-tableSignal discriminator, G is Ggen;
Constructing texture loss LtFor maintaining the consistency of the content and style of the generated image and the real image, the loss of texture LtIs defined as follows:
Lt=||Gram{Φ(Iout)}-Gram{Φ(Igt)}||2+λ||Φ(Iout)-Φ(Igt)||2 (6)
wherein phi represents that a VGG-16 network pre-trained by ImageNet is utilized to extract a high-level feature space, and Gram represents the operation of a Gram matrix.
And finally, constructing a total loss function of the generated network, which is specifically as follows:
Ltotal=λrLr+λadvLadv+λtLt (7)
wherein λ isr、λadv、λtRepresenting the weight coefficients.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110707809.8A CN113409216A (en) | 2021-06-24 | 2021-06-24 | Image restoration method based on frequency band self-adaptive restoration model |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110707809.8A CN113409216A (en) | 2021-06-24 | 2021-06-24 | Image restoration method based on frequency band self-adaptive restoration model |
Publications (1)
Publication Number | Publication Date |
---|---|
CN113409216A true CN113409216A (en) | 2021-09-17 |
Family
ID=77683159
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110707809.8A Pending CN113409216A (en) | 2021-06-24 | 2021-06-24 | Image restoration method based on frequency band self-adaptive restoration model |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113409216A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113808123A (en) * | 2021-09-27 | 2021-12-17 | 杭州跨视科技有限公司 | Machine vision-based dynamic detection method for liquid medicine bag |
WO2023197219A1 (en) * | 2022-04-13 | 2023-10-19 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Cnn-based post-processing filter for video compression with multi-scale feature representation |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110443775A (en) * | 2019-06-20 | 2019-11-12 | 吉林大学 | Wavelet transform domain multi-focus image fusing method based on convolutional neural networks |
CN111047541A (en) * | 2019-12-30 | 2020-04-21 | 北京工业大学 | Image restoration method based on wavelet transformation attention model |
CN111275640A (en) * | 2020-01-17 | 2020-06-12 | 天津大学 | Image enhancement method for fusing two-dimensional discrete wavelet transform and generating countermeasure network |
US20210183022A1 (en) * | 2018-11-29 | 2021-06-17 | Tencent Technology (Shenzhen) Company Limited | Image inpainting method and apparatus, computer device, and storage medium |
-
2021
- 2021-06-24 CN CN202110707809.8A patent/CN113409216A/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210183022A1 (en) * | 2018-11-29 | 2021-06-17 | Tencent Technology (Shenzhen) Company Limited | Image inpainting method and apparatus, computer device, and storage medium |
CN110443775A (en) * | 2019-06-20 | 2019-11-12 | 吉林大学 | Wavelet transform domain multi-focus image fusing method based on convolutional neural networks |
CN111047541A (en) * | 2019-12-30 | 2020-04-21 | 北京工业大学 | Image restoration method based on wavelet transformation attention model |
CN111275640A (en) * | 2020-01-17 | 2020-06-12 | 天津大学 | Image enhancement method for fusing two-dimensional discrete wavelet transform and generating countermeasure network |
Non-Patent Citations (1)
Title |
---|
肖锋等: "结合感知注意力机制的生成式对抗网络图像修复", 《西安工业大学学报》, vol. 41, no. 2, 25 April 2021 (2021-04-25), pages 198 - 205 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113808123A (en) * | 2021-09-27 | 2021-12-17 | 杭州跨视科技有限公司 | Machine vision-based dynamic detection method for liquid medicine bag |
CN113808123B (en) * | 2021-09-27 | 2024-03-29 | 杭州跨视科技有限公司 | Dynamic detection method for liquid medicine bag based on machine vision |
WO2023197219A1 (en) * | 2022-04-13 | 2023-10-19 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Cnn-based post-processing filter for video compression with multi-scale feature representation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111047541B (en) | Image restoration method based on wavelet transformation attention model | |
Yoo et al. | Photorealistic style transfer via wavelet transforms | |
CN110276726B (en) | Image deblurring method based on multichannel network prior information guidance | |
CN103810755B (en) | Compressed sensing spectrum picture method for reconstructing based on documents structured Cluster rarefaction representation | |
CN103279933B (en) | A kind of single image super resolution ratio reconstruction method based on bilayer model | |
CN113409216A (en) | Image restoration method based on frequency band self-adaptive restoration model | |
CN112686817B (en) | Image completion method based on uncertainty estimation | |
CN105590296B (en) | A kind of single-frame images Super-Resolution method based on doubledictionary study | |
Du et al. | Blind image denoising via dynamic dual learning | |
TW202217742A (en) | Image quality improvement method and image processing apparatus using the same | |
Agrawal et al. | Image resolution enhancement using lifting wavelet and stationary wavelet transform | |
Li et al. | Detail-enhanced image inpainting based on discrete wavelet transforms | |
CN106056640B (en) | The method for compressing image for combining compressed sensing is decomposed based on anatomic element | |
CN117726540A (en) | Image denoising method for enhanced gate control converter | |
Cetinkaya et al. | Image denoising using deep convolutional autoencoder with feature pyramids | |
CN115131226B (en) | Image restoration method based on wavelet tensor low-rank regularization | |
Wang | Single image super-resolution with u-net generative adversarial networks | |
CN114331853B (en) | Single image restoration iteration framework based on target vector updating module | |
You et al. | Indigo: An inn-guided probabilistic diffusion algorithm for inverse problems | |
Adeyemi et al. | Sparse representations of images using overcomplete complex wavelets | |
Yin et al. | Deep Convolutional Dictionary Learning Denoising Method Based on Distributed Image Patches | |
CN113409217A (en) | Image restoration method based on multi-frequency sub-band probabilistic inference model | |
CN117274064B (en) | Image super-resolution method | |
Huang et al. | DF3Net: Dual frequency feature fusion network with hierarchical transformer for image inpainting | |
Fatimah et al. | Multiscale modeling using signal adapted filter bank |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |