CN112927137A - Method, device and storage medium for acquiring blind super-resolution image - Google Patents
Method, device and storage medium for acquiring blind super-resolution image Download PDFInfo
- Publication number
- CN112927137A CN112927137A CN202110251126.6A CN202110251126A CN112927137A CN 112927137 A CN112927137 A CN 112927137A CN 202110251126 A CN202110251126 A CN 202110251126A CN 112927137 A CN112927137 A CN 112927137A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- paired
- generator
- resolution image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000006731 degradation reaction Methods 0.000 claims abstract description 55
- 230000015556 catabolic process Effects 0.000 claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 21
- 230000005540 biological transmission Effects 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 16
- 238000009826 distribution Methods 0.000 claims description 13
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 3
- 238000004088 simulation Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 238000013507 mapping Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- OAICVXFJPJFONN-UHFFFAOYSA-N Phosphorus Chemical compound [P] OAICVXFJPJFONN-UHFFFAOYSA-N 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 208000032370 Secondary transmission Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The present disclosure relates to a method, apparatus, and storage medium for acquiring a blind super-resolution image. Estimating image degradation model parameters in a natural image degradation process, wherein the image degradation model parameters comprise a convolution kernel pool and a noise pool; establishing a low-resolution-high-resolution paired image data set according to the image degradation model parameters; training and optimizing an image generator using the paired image dataset; and training and optimizing an image discriminator based on the paired image dataset and the image generator to obtain a blind super-resolution image. According to the method, model parameters are estimated through a degradation model of the natural image so as to establish a paired data set, and the paired data set is further used for training and optimizing an image generator and a discriminator, so that a blind super-resolution image can be obtained, and the quality of the super-resolution image is greatly improved.
Description
Technical Field
The present disclosure relates generally to the field of image processing technology. More particularly, the present disclosure relates to a method, apparatus, and computer-readable storage medium for acquiring a blind super-resolution image.
Background
This section is intended to provide a background or context to the embodiments of the disclosure recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Thus, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.
With the development of information technology, people have higher and higher requirements on digital image quality, and particularly in the computer vision fields of medicine, automatic driving, astronomy, monitoring and the like, high-resolution and high-detail high-definition images are required to be obtained. However, in the actual image acquisition, the spatial resolution of the obtained image is not high due to the influence of factors such as the spatial resolution of the imaging system, the light or ray intensity, the spatial distance, and the system noise. The traditional method for improving the resolution of the image is usually designed based on system hardware equipment or software algorithms, and improving the resolution of the image based on the system hardware equipment is a direct method, but the method is high in cost and difficult to implement. In addition, system hardware devices are also susceptible to limitations from the physics of electromagnetic waves and the physics of components, resulting in limited improved spatial resolution.
Single Image Super Resolution ("SISR") reconstruction techniques are mainly based on software algorithm level designs, which can achieve the enhancement from low Resolution ("LR") observation images to high Resolution ("HR") observation images. Recently, since deep convolutional neural networks have shown a powerful function in processing image data having a euclidean structure, many convolutional neural network-based super-resolution techniques have solved the task of single-image super-resolution analysis from some specific perspectives. However, these methods are usually designed by a priori information to find a balance between detail recovery and noise suppression, which often results in loss of high frequency detail information. In addition, there are many methods for obtaining and training HR-LR images based on bicubic downsampling, and therefore, when a problem of natural image degradation due to distance, electromagnetic wave signal intensity, and system noise is faced, the image degradation process does not conform to these natural degradation processes, and the performance of the super-resolution model is limited.
Disclosure of Invention
In order to solve at least the above problems, the present disclosure proposes a scheme for acquiring a blind super-resolution image by estimating a degraded model parameter of a natural image to obtain a pair image data set based on the model parameter, so that an image generator and a discriminator can be trained and optimized to acquire a high-quality blind super-resolution image. In view of this, the present disclosure provides corresponding solutions in the following aspects.
In a first aspect, the present disclosure provides a method for acquiring a blind super-resolution image, comprising: estimating image degradation model parameters in a natural image degradation process, wherein the image degradation model parameters comprise a convolution kernel pool and a noise pool; establishing a low-resolution-high-resolution paired image data set according to the image degradation model parameters; training and optimizing an image generator using the paired image dataset; and training and optimizing an image discriminator based on the paired image dataset and the image generator to obtain a blind super-resolution image.
In one embodiment, estimating image degradation model parameters in a natural image degradation process includes: constructing an image degradation model degraded from a high-resolution image to a low-resolution image in the natural image; learning a blur kernel for each of the natural images in the image degradation model using a linear convolution network to estimate a pool of blur kernels; and extracting a noise distribution of the natural image in the image degradation model according to relative variance selection and sub-block division to estimate a noise pool.
In another embodiment, establishing the low-resolution-high-resolution paired image dataset comprises: respectively randomly sampling from the fuzzy kernel pool and the noise pool according to a random principle to obtain a target fuzzy kernel and a target noise block; and performing a simulation of a degradation process for a known high resolution image based on the target blur kernel, the target noise block, and the image degradation model to create a low resolution image paired therewith.
In yet another embodiment, training and optimizing an image generator using the paired image dataset comprises: inputting a low-resolution image into the image generator to generate a pseudo high-resolution image under the constraint of a quadratic Wassertein distance to obtain a loss function of the image generator; and training an image generator based on the loss function.
In yet another embodiment, training and optimizing an image generator using the paired image dataset further comprises: adding a plurality of target losses to the loss function to optimize the image generator.
In yet another embodiment, the plurality of target losses includes a countermeasure loss, a boundary loss, a pixel loss, and/or a perceptual loss.
In yet another embodiment, training and optimizing an image discriminator includes: inputting the pseudo high-resolution image and the true high-resolution image into the image discriminator so as to obtain a discrimination score; determining a numerical solution of the discrimination score by using linear programming and a deep neural network; and training and optimizing the image discriminator based on the optimal transmission principle and the quadratic Wasserstein distance determined by the numerical solution.
In yet another embodiment, the method further comprises: adding an optimal transmission regularization term into an image discriminator so as to optimize the image discriminator.
In a second aspect, the present disclosure provides an apparatus for acquiring a blind super-resolution image, comprising: a processor; and a memory connected to the processor, the memory having stored therein computer program code which, when executed by the processor, causes the apparatus to perform the method according to the above and its various embodiments.
In a third aspect, the present disclosure provides a computer-readable storage medium having stored thereon computer-readable instructions for acquiring a blind super-resolution image, which computer-readable instructions, when executed by one or more processors, implement a method according to the above and its various embodiments.
According to the scheme of the disclosure, the degradation model parameters of the natural images are estimated so as to obtain the paired image data sets based on the model parameters, so that the image generator and the discriminator can be trained and optimized to obtain high-quality blind super-resolution images. Further, the disclosed embodiments are based on an optimal transmission principle and utilize a quadratic Wasserstein distance as a metric to facilitate mutual optimization between the image generator and the discriminator, so that the image passes through the optimized generator to generate a high-resolution image closer to reality. In addition, in the embodiment of the disclosure, the image generator is trained by adding various target losses, so that the generator is optimal in the aspects of edge high-frequency detail recovery and noise elimination, and a blind super-resolution image with high-frequency detail fidelity and remarkable noise reduction effect can be obtained.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar or corresponding parts and in which:
fig. 1 is a simplified flow diagram illustrating a method for acquiring a blind super-resolution image according to an embodiment of the present disclosure;
fig. 2 is a detailed flowchart illustrating a method for acquiring a blind super-resolution image according to an embodiment of the present disclosure;
3-6 are exemplary diagrams illustrating the generation of a blind super-resolution image according to embodiments of the present disclosure; and
fig. 7 is an exemplary schematic diagram illustrating a generated blind super-resolution image and an otherwise generated image according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the embodiments described in this specification are only some of the embodiments provided by the present disclosure to facilitate a clear understanding of the aspects and to comply with legal requirements, and not all embodiments in which the present invention may be practiced. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed in the specification without making any creative effort, shall fall within the protection scope of the present disclosure.
Fig. 1 is a simplified flow diagram illustrating a method 100 for acquiring a blind super-resolution image according to an embodiment of the present disclosure.
As shown, at step S102, image degradation model parameters in the natural image degradation process are estimated. In one embodiment, the aforementioned image degradation model parameters may include a convolution kernel pool and a noise pool. This step S102 can be understood as being capable of estimating a convolution kernel and a noise distribution conforming to a true degradation process by learning (e.g., machine learning or deep learning) on a low-quality image data set in a natural scene. An image degradation model degraded from a high resolution ("HR") image to a low resolution ("LR") image in a natural image may be first constructed before estimating image degradation model parameters. In one implementation scenario, the image degradation model from a high resolution image to a low resolution image may be expressed as the following equation:
where LR represents a low resolution image, HR represents a high resolution image, k and n represent a convolution kernel and noise, respectively,
denotes convolution operation, ↓denotesdownsampling operation, and s denotes downsampling multiple.
Based on the image degradation model constructed as described above, image degradation model parameters (including a convolution kernel pool and a noise pool) can be estimated. In one embodiment, a linear convolution network (e.g., a Kernelgan network) may be used to learn the blur kernel for each natural image in the image degradation model, where all blur kernels may form a blur pool. In another embodiment, a noise pool may be formed by extracting a noise distribution of a natural image in an image degradation model according to relative variance selection and sub-block division. Specifically, the aforementioned noise distribution can be extracted using the following formula:
wherein p isiWhich represents an image block or blocks of an image,
representing a selection from image blocks piThe local image sub-blocks of (1), mean (-) and var (-) represent the mean and variance operations, respectively, mu and gamma may represent empirical parameters, and v ismaxRepresenting the maximum variance threshold. In some embodiments, the aforementioned μ and γ are parameters that are typically entered by one skilled in the art based on empirical data. For example, in aspects of the present disclosure, μ and γ may be set to 0.1 and 0.25, respectively.
It can be understood that, for an image, when the variance of a local image sub-block is small, it can indicate that the local image sub-block in the image block belongs to a gray-level flat area, i.e. has no noise distribution or has little noise. When the variance of the local image sub-block is large, it indicates that the local image sub-block in the image block belongs to an edge or high-frequency region (i.e., the region contains noise or is very noisy). Thus, combining the above equation (2) can make the local image sub-block variance larger than the maximum variance threshold vmaxAs a noise distribution to form a noise pool. In this disclosureIn an embodiment, the maximum variance threshold v ismaxMay be provided as 40.
After obtaining the image degradation model parameters (including the blur kernel pool and the noise pool), at step S104, a low-resolution-high-resolution paired image dataset is built according to the image degradation model parameters. With respect to establishing the aforementioned paired image data set, in one embodiment, the present disclosure proposes that first random sampling may be performed from the blur kernel pool and the noise pool, respectively, according to a random principle, so as to obtain a target blur kernel and a target noise block. Then, based on the extracted target fuzzy kernel and the extracted target noise block, the target fuzzy kernel and the target noise block are substituted into the image degradation model represented by the formula (1), so that the simulation of the degradation process can be performed on the known high-resolution image, and the low-resolution image which accords with the natural degradation process can be obtained. Thus, the present disclosure establishes a low-resolution-high-resolution paired image dataset.
After the above-described paired image data set is established, the flow advances to step S106. At this step, the image generator is trained and optimized with the paired image dataset. In one implementation scenario, this step may be implemented according to an optimal transmission principle. As known to those skilled in the art, Optimal Transport (Optimal Transport) can be used to solve the mapping problem between two distributions. For example, assume that two variables X, Y are given, and that two metrics X, Y each correspond to a spatial distribution of μ, ν. Under this assumption, a transmission transformation T X → Y is sought which is capable of transforming the random variable X corresponding to the distribution-obeying μ into a Y random variable obeying the distribution v, thereby achieving the expectation of minimizing the transmission cost c (X, T (X)). Thereby, a mapping between the aforementioned two variables can be obtained. It should be understood that, in the optimal transmission principle, the main transmission cost basis may be the second Wasserstein distance.
Specifically, under the constraint of the quadratic Wasserstein distance, the low-resolution image may be input to the image generator to generate the pseudo high-resolution image. At this time, since there is a difference between the input low resolution image and the generated pseudo high resolution image, a loss function of the image generator can be obtained. Further, the image generator may be trained based on the obtained loss function. For example, the image generator may be trained by adjusting the weighting coefficients of the loss function. Based on the trained image generator, a pseudo high resolution image can be obtained for subsequent training and optimization of the image discriminator to form an optimal mapping of low resolution images to high resolution images. In one embodiment, the aforementioned quadratic Wasserstein distance may be specifically expressed as the following formula:
wherein G denotes an image generator, D denotes an image discriminator, ziAnd yiRespectively representing the low resolution image and the high resolution image in the paired image dataset, and m represents the number of low resolution images and high resolution images in each iteration.
As can be seen from the foregoing description, image noise is generally distributed in an edge or a high-frequency region, and therefore, in order to obtain an image with high fidelity of high-frequency details and obtain more paired image information, the embodiments of the present disclosure may further add various target losses to a loss function so as to optimize the foregoing image generator. In various implementation scenarios, the aforementioned multiple target losses may be, for example, a countering loss, a boundary loss, a pixel loss, and/or a perceptual loss, and the countering loss may be the aforementioned second-order Wasserstein distance. The boundary loss, the pixel loss, and the perceptual loss among the aforementioned various target losses will be described in detail below, respectively.
In one embodiment, when an image with high fidelity of high frequency details is to be acquired, boundary loss can be added to the loss function, and an attention mechanism is introduced through the boundary loss, so that the image generator can put more attention on edge high frequency detail recovery. The boundary loss LedgeSpecifically, the following formula can be expressed:
wherein the content of the first and second substances,
representing the boundary operator and G the image generator. Additionally, in embodiments of the present disclosure, the foregoing operations may be implemented using, for example, an RRDB model in a deep neural network.
In one embodiment, in order to be able to obtain more paired image information to optimize the image generator, pixel loss representing the image texture content may be added to the loss function. In particular, the aforementioned pixel loss may be, for example, the pixel mean square error L1And the pixel mean square error L1Can be expressed as follows:
wherein G represents an image generator, | · |. non-woven phosphor1Representing a 1-norm, and h and w represent the number of rows and columns, respectively, of the image.
In one embodiment, perceptual loss may also be added to the loss function in order to generate an image with better visual effect. The perception loss LperSpecifically, the following formula can be expressed:
wherein G represents an image generator, | · |. non-woven phosphor2Representing the 2-norm, ψ represents the feature extractor, and h, w, and c represent the number of rows, columns, and channels of the image, respectively. In some implementation scenarios, the aforementioned feature extractor may employ a neural network model, such as VGG-19, to achieve feature extraction.
Based on the various target losses added to the loss function, a final optimization objective may be obtained for optimizing the image generator, for example, by using a weighted sum of the various target losses. Record the optimization goal asLGThen, it can be expressed as the following formula:
LG＝λadvLadv+λedgeLedge+λ1L1+λperLper (7)
wherein L isadv、Ledge、L1And LperRespectively representing the contrast loss, boundary loss, pixel loss and perceptual loss, lambdaadv、λedge、λ1And λperRespectively representing the weight coefficients set corresponding to the various losses. In the disclosed embodiment, the weighting factor λ may beadv、λedge、λ1And λperSet to 0.05, 0.01 and 0.7, respectively.
From the paired image dataset and image generator obtained as described above, the flow of method 100 proceeds to step S108. At step S108, an image discriminator is trained and optimized based on the paired image dataset and the image generator to obtain a blind super-resolution image. As described above, the low-resolution image and the high-resolution image are included in the paired image data set, and the low-resolution image in the paired image data set is input to the image generator to generate the pseudo high-resolution image. Further, the true high resolution image and the pseudo high resolution image in the paired image data set may be simultaneously input to the image discriminator to train and optimize the image discriminator. More specifically, a linear programming and a deep neural network such as VGG-128 can be utilized to solve the problem of the Monge-Kantorovich duality under the discrete condition, so that a numerical solution of a discrimination score after passing through an image discriminator can be obtained, and the image discriminator is trained and optimized.
In one implementation scenario, the above-mentioned monte-Kantorovich dual problem in discrete case can be represented by the following formula:
wherein SR and HR represent a pseudo-high resolution image set and a true high resolution image set, respectivelyAnd m represents the number of pseudo high resolution images and true high resolution images in each iteration. c (y)i,xj) A quadratic transmission cost function is represented, the definition of which is expressed as follows:
in formula (9), | · non-woven phosphor2Denotes the 2-norm, xjAnd yiThe distributions represent a pseudo high resolution image and a true high resolution image. By combining the above formula (8) and formula (9), the discriminant score phi (y) can be obtained by linear programmingi) And τ (x)j) The numerical solutions of (A) can be expressed, for example, as
And
based on the obtained numerical solution of the discrimination score, the image discriminator is trained and optimized again according to the optimal transmission principle. Similar to the training and optimization image generator described above, the quadratic Wasserstein distance based on the quadratic transmission cost can be used as the optimization target in training and optimizing the image discriminator. The quadratic Wassertein distance may further be determined via a numerical solution of the discriminant score to obtain an optimal image discriminator. At this time, the secondary Wasserstein distance is decorrelated with respect to the numerical value of the discrimination score, and its specific expression is as follows:
in one embodiment, in order to improve the stability of the image discriminator, the embodiment of the present disclosure further adds an optimal transmission regularization term. The optimal transmission regularization term LOTRCan be expressed by the following formula:
in addition, for the secondary Wasserstein distance LwassAnd an optimal transmission regularization term LOTRObtaining a final optimization objective L of the image discriminator using, for example, weighted summationDTherefore, a stable optimal image discriminator can be obtained. The aforementioned optimization target LDSpecifically, the following formula can be expressed:
LD＝1×Lwass+0.41×LOTR (12)
it is to be understood that L in the formula (12)wassCoefficients 1 and L ofOTRThe coefficient of 0.41 is merely exemplary and not limiting, and other suitable coefficients may be selected by one skilled in the art to achieve the optimization goal in light of the teachings of the present disclosure.
As can be seen from the above description, in the embodiment of the present disclosure, the real degradation process of the image in the natural scene and the corresponding parameters are estimated first, and further, the low-resolution image conforming to the natural degradation process is obtained according to the parameters, so that the low-resolution-high-resolution paired image dataset is established. Respectively training and optimizing the image generator and the image discriminator based on the obtained pairing data set and the optimal transmission principle, for example, taking a secondary Wasserstein distance based on secondary transmission cost as the measurement of the distance between a pseudo high-resolution image space and a true high-resolution image space, and solving a Monge-Kantorovich dual problem by utilizing linear programming to continuously optimize the secondary Wasserstein distance so that the generated pseudo high-resolution image is closer to the true high-resolution image. In addition, the embodiment of the disclosure adds various target losses in the optimization process, so that a blind super-resolution image with high fidelity of high-frequency details and obvious noise reduction effect can be obtained. By adopting the scheme disclosed by the invention, the high-frequency detail recovery and noise elimination of the image can be optimized simultaneously.
Fig. 2 is a detailed flowchart illustrating a method for acquiring a blind super-resolution image according to an embodiment of the present disclosure. As can be seen from the above description of fig. 1, the method 200 is a specific implementation of the method 100 shown in fig. 1, and therefore the description of the method 100 also applies to the method 200.
As shown, at step S202, an unpaired natural image dataset is input into the constructed image degradation model. In one embodiment, the image degradation model may be constructed based on equation (1) above. Next, at step S204, a natural image degradation process is estimated from the input natural image data set, thereby obtaining image degradation model parameters, such as a convolution pool K and a noise pool N. Wherein, the blur pool may be obtained by learning a convolution kernel of the natural image through a linear convolution network, and the noise pool N may be obtained based on the above equation (2). As mentioned above, the image distribution with local image sub-block variance larger than the maximum variance threshold is usually selected as the noise distribution, thereby forming the noise pool.
Based on the obtained image degradation model parameters (including the convolution pool K and the noise pool N), at step S206, a convolution kernel K and a noise N can be randomly extracted from K and N according to a random allocation principle and applied to the high resolution image, so as to obtain a low resolution image conforming to the true degradation process. This step S206 is repeatedly executed until the low resolution-high resolution (LR-HR) paired image dataset of the present disclosure is obtained. Further, at step S208, the low resolution image in the paired image data set obtained at step S206 is input to the image generator, and the image generator is continuously trained and optimized under the constraint of twice Wasserstein distance, so that the image generator can gradually approach an optimal mapping from low resolution to high resolution. In other words, the input low resolution image may generate a pseudo high resolution image based on the aforementioned image generator. At step S210, the pseudo high resolution image and the true high resolution image are input to an image discriminator, and the ability to distinguish the pseudo high resolution image from the true high resolution image by the image discriminator (e.g., using the discrimination scores described above) causes the image discriminator to approach the optimal discriminator by continually optimizing the quadratic Wasserstein distance. Finally, by repeatedly performing the above steps S208 and S210, the image generator and the image discriminator can mutually promote and optimize, and finally generate a high-quality blind super-resolution image, such as shown in fig. 3-6.
Fig. 3-6 are exemplary diagrams illustrating generation of a blind super-resolution image according to an embodiment of the present disclosure. As shown in the figures, the left images in fig. 3 to 6 all represent natural images before processing, and as can be seen from fig. 3 to 6, the natural images all present the problem of blurring, i.e., low resolution. The right images in fig. 3-6 all represent images processed by the disclosed scheme, and it can be seen that the processed images are clearer and have better display effect.
Fig. 7 is an exemplary schematic diagram illustrating a generated blind super-resolution image and an otherwise generated image according to an embodiment of the present disclosure. As shown, the first column on the left side in the figure shows a plurality of images, and each is a low-fraction ("LR") image, and a plurality of columns immediately behind the first column are images generated for partial images (shown by rectangular boxes in the figure) in the first column image on the left side based on different processing methods, respectively. The second to last column on the left is further shown to be the resulting image generated based on the scheme of the present disclosure, the paired data set realSR, DAN algorithm, IKC algorithm, DPSR algorithm, ESRGAN algorithm, and zsr algorithm processing. By comparing results of different algorithm processing, it can be seen that the result image obtained by the method of the embodiment of the present disclosure is clearer (i.e., higher in resolution).
From the above description in conjunction with the accompanying drawings, those skilled in the art will also appreciate that embodiments of the present disclosure may also be implemented by software programs. The present disclosure thus also provides a computer program product. The computer program product may be used to implement the method for acquiring a blind super-resolution image as described in the present disclosure in connection with fig. l and 2.
It should be noted that while the operations of the disclosed methods are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
It should be understood that the terms "first," "second," "third," and "fourth," etc. used in the claims, the specification, and the drawings of the present disclosure are only used for distinguishing between different objects, and are not used to describe a particular order. The terms "comprises" and "comprising," when used in the specification and claims of this disclosure, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the disclosure herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the disclosure. As used in the specification and claims of this disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the term "and/or" as used in the specification and claims of this disclosure refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
Although the embodiments of the present invention are described above, the descriptions are only examples for facilitating understanding of the present invention, and are not intended to limit the scope and application scenarios of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method for acquiring a blind super-resolution image, comprising:
estimating image degradation model parameters in a natural image degradation process, wherein the image degradation model parameters comprise a convolution kernel pool and a noise pool;
establishing a low-resolution-high-resolution paired image data set according to the image degradation model parameters;
training and optimizing an image generator using the paired image dataset; and
training and optimizing an image discriminator based on the paired image dataset and the image generator to obtain a blind super-resolution image.
2. The method of claim 1, wherein estimating image degradation model parameters in a natural image degradation process comprises:
constructing an image degradation model degraded from a high-resolution image to a low-resolution image in the natural image;
learning a blur kernel for each of the natural images in the image degradation model using a linear convolution network to estimate a pool of blur kernels; and
extracting a noise distribution of the natural image in the image degradation model according to relative variance selection and sub-block division to estimate a noise pool.
3. The method of claim 2, wherein creating a low-resolution-high-resolution paired image dataset comprises:
respectively randomly sampling from the fuzzy kernel pool and the noise pool according to a random principle to obtain a target fuzzy kernel and a target noise block; and
performing a simulation of a degradation process for a known high resolution image based on the target blur kernel, the target noise block, and the image degradation model to create a low resolution image paired therewith.
4. The method of claim 1, wherein training and optimizing an image generator using the paired image dataset comprises:
inputting a low-resolution image into the image generator to generate a pseudo high-resolution image under the constraint of a quadratic Wassertein distance to obtain a loss function of the image generator; and
training an image generator based on the loss function.
5. The method of claim 4, wherein training and optimizing an image generator using the paired image dataset further comprises:
adding a plurality of target losses to the loss function to optimize the image generator.
6. The method of claim 5, wherein the plurality of target losses includes a countermeasure loss, a boundary loss, a pixel loss, and/or a perceptual loss.
7. The method of claim 4, wherein training and optimizing an image discriminator comprises:
inputting the pseudo high-resolution image and the true high-resolution image into the image discriminator so as to obtain a discrimination score;
determining a numerical solution of the discrimination score by using linear programming and a deep neural network; and
the image discriminator is trained and optimized based on the optimal transmission principle and the quadratic Wasserstein distance determined by the numerical solution.
8. The method of claim 7, further comprising: adding an optimal transmission regularization term into an image discriminator so as to optimize the image discriminator.
9. An apparatus for acquiring a blind super-resolution image, comprising:
a processor; and
a memory connected to the processor, the memory having stored therein computer program code which, when executed by the processor, causes the apparatus to perform the method of any of claims 1-8.
10. A computer-readable storage medium having stored thereon computer-readable instructions for acquiring a blind super-resolution image, the computer-readable instructions, when executed by one or more processors, implementing the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110251126.6A CN112927137A (en) | 2021-03-08 | 2021-03-08 | Method, device and storage medium for acquiring blind super-resolution image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110251126.6A CN112927137A (en) | 2021-03-08 | 2021-03-08 | Method, device and storage medium for acquiring blind super-resolution image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112927137A true CN112927137A (en) | 2021-06-08 |
Family
ID=76171880
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110251126.6A Pending CN112927137A (en) | 2021-03-08 | 2021-03-08 | Method, device and storage medium for acquiring blind super-resolution image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112927137A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113469884A (en) * | 2021-07-15 | 2021-10-01 | 长视科技股份有限公司 | Video super-resolution method, system, equipment and storage medium based on data simulation |
| CN113538245A (en) * | 2021-08-03 | 2021-10-22 | 四川启睿克科技有限公司 | Degradation model-based super-resolution image reconstruction method and system |
| CN113724134A (en) * | 2021-08-20 | 2021-11-30 | 广东工业大学 | Aerial image blind super-resolution reconstruction method based on residual distillation network |
-
2021
- 2021-03-08 CN CN202110251126.6A patent/CN112927137A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113469884A (en) * | 2021-07-15 | 2021-10-01 | 长视科技股份有限公司 | Video super-resolution method, system, equipment and storage medium based on data simulation |
| CN113538245A (en) * | 2021-08-03 | 2021-10-22 | 四川启睿克科技有限公司 | Degradation model-based super-resolution image reconstruction method and system |
| CN113724134A (en) * | 2021-08-20 | 2021-11-30 | 广东工业大学 | Aerial image blind super-resolution reconstruction method based on residual distillation network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114140353B (en) | Swin-Transformer image denoising method and system based on channel attention | |
| Sánchez et al. | Brain MRI super-resolution using 3D generative adversarial networks | |
| EP4109392A1 (en) | Image processing method and image processing device | |
| CN112927137A (en) | Method, device and storage medium for acquiring blind super-resolution image | |
| CN113284051B (en) | Face super-resolution method based on frequency decomposition multi-attention machine system | |
| CN112819910B (en) | Hyperspectral image reconstruction method based on double-ghost attention machine mechanism network | |
| CN112837224A (en) | Super-resolution image reconstruction method based on convolutional neural network | |
| Rivadeneira et al. | Thermal image super-resolution challenge-pbvs 2021 | |
| KR20200132682A (en) | Image optimization method, apparatus, device and storage medium | |
| Huang et al. | Two-step approach for the restoration of images corrupted by multiplicative noise | |
| Ahmed et al. | PIQI: perceptual image quality index based on ensemble of Gaussian process regression | |
| Zheng et al. | T-net: Deep stacked scale-iteration network for image dehazing | |
| CN115345866A (en) | Method for extracting buildings from remote sensing images, electronic equipment and storage medium | |
| CN115063318A (en) | Adaptive frequency-resolved low-illumination image enhancement method and related equipment | |
| CN116309178A (en) | Visible light image denoising method based on self-adaptive attention mechanism network | |
| CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
| CN115700731A (en) | Underwater image enhancement method based on dual-channel convolutional neural network | |
| CN115082336A (en) | SAR image speckle suppression method based on machine learning | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| CN112819731B (en) | Gray scale image enhancement method, device, computer equipment and storage medium | |
| CN111986102B (en) | Digital pathological image deblurring method | |
| US20240054605A1 (en) | Methods and systems for wavelet domain-based normalizing flow super-resolution image reconstruction | |
| Khandelwal et al. | W-VDSR: Wavelet-based secure image transmission using machine learning VDSR neural network | |
| Peng et al. | MND-GAN: A research on image deblurring algorithm based on generative adversarial network | |
| US20220398696A1 (en) | Image processing method and device, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |