CN111754403B - Image super-resolution reconstruction method based on residual learning - Google Patents
Image super-resolution reconstruction method based on residual learning Download PDFInfo
- Publication number
- CN111754403B CN111754403B CN202010542676.9A CN202010542676A CN111754403B CN 111754403 B CN111754403 B CN 111754403B CN 202010542676 A CN202010542676 A CN 202010542676A CN 111754403 B CN111754403 B CN 111754403B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- layer
- reconstruction
- convolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000000605 extraction Methods 0.000 claims abstract description 18
- 238000013528 artificial neural network Methods 0.000 claims abstract description 17
- 238000012360 testing method Methods 0.000 claims abstract description 5
- 238000010586 diagram Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 11
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 238000011084 recovery Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image super-resolution reconstruction method based on residual error learning, which comprises the following steps: acquiring a common image data set for a training system, and processing to obtain a training set and a test set; acquiring a low-resolution image to be processed; building and fusing all modules of a system core neural network to form a deep convolutional network based on residual learning; calculating network prediction and label loss, and adjusting network parameters according to the loss; training a depth residual error network and inputting a low-resolution image to be processed to the depth residual error network; and outputting the reconstructed high-resolution image, evaluating the reconstruction precision of the image, and finishing the reconstruction of the super-resolution image. The method increases the extraction capability of the image characteristics, improves the generalization and the high efficiency of the image reconstruction system, and ensures that the image reconstruction precision is more accurate.
Description
Technical Field
The invention relates to an image processing method, in particular to an image super-resolution reconstruction method based on residual error learning, and belongs to the field of deep learning.
Background
Image super-resolution reconstruction is a classic problem in the field of computer vision, aiming at converting one or more low-resolution images into high-resolution images by means of algorithms. As a criterion for evaluating the image quality, the higher the resolution, the higher the pixel density of the picture, and the richer the detail information contained, the higher the picture quality. In reality, under the influence of a plurality of objective factors such as acquisition equipment, noise, shooting environment and the like, the acquired picture often cannot meet the requirement, so that the resolution of the image is improved, the quality of the image is improved, and the problem is always a hot spot concerned by people. The most direct method for improving the image resolution is to improve the optical hardware in the acquisition system, but this method is constrained by the difficulty in greatly improving the manufacturing process, the high manufacturing cost, and the like, so the technology for realizing the super-resolution image reconstruction from the viewpoint of software and algorithm becomes a hot research topic in a plurality of fields such as image processing and computer vision.
At present, image super-resolution reconstruction algorithms are mainly classified into 3 types, that is, interpolation-based image reconstruction algorithms, reconstruction-based image reconstruction algorithms, and learning-based image reconstruction algorithms.
The super-resolution reconstruction algorithm based on interpolation is a reconstruction technology starting to be widely researched in the early stage, and the algorithm mainly utilizes adjacent pixel points to calculate the gray value of the pixel point to be inserted. The interpolation method is a non-iterative spatial domain algorithm, has the characteristic of high calculation efficiency, and is the simplest image super-resolution reconstruction algorithm. But at the same time, the reconstructed image may have blocking and blurring effects due to the simplistic interpolation method.
The reconstruction-based super-resolution reconstruction algorithm reflects the relationship between the low-resolution image and the high-resolution image by establishing a degradation model of the image, and based on some constraint conditions, the optimal estimation value of the high-resolution image is solved from the model by using some optimization algorithms. The reconstruction effect of such methods depends to a large extent on the choice of a priori conditions, and the generated image may lack important detailed information or be too smooth.
The method learns the mapping relation of the super-resolution image based on the learning image super-resolution reconstruction algorithm by utilizing the low-resolution image and the high-resolution image, and then reconstructs the high-resolution image. The existing mainstream methods include a sparse coding method and super-resolution reconstruction based on deep learning. With the emergence of massive data and the development of artificial intelligence technology, a deep learning-based method becomes a main trend of current research. In 2014, Dong et al introduced the image super-resolution field through a convolutional network first, and originally proposed an SRCNN method, the algorithm takes a low-resolution image subjected to bicubic interpolation as input, utilizes three convolutional layers to nonlinearly map the image from a low-resolution space to a high-resolution space, completes the processes of image extraction and feature representation, nonlinear mapping and image reconstruction, directly reconstructs the high-resolution image, realizes end-to-end image reconstruction, has a reconstruction effect remarkably superior to that of a traditional super-resolution algorithm, and proves the excellent performance of deep learning in the image super-resolution field. Then Dong et al propose FSRCNN to improve SRCNN, propose training with low resolution images as input to the network, and enlarge the size and reconstruct high resolution images at the end of the network through a deconvolution layer. In 2016, Kim et al propose a 20-layer deep convolutional neural network model VDSR to perform deep extraction on image features, and accelerate the convergence rate of the network by introducing a residual error model, thereby improving the generalization capability of the model.
The image super-resolution reconstruction technology based on deep learning utilizes a convolutional neural network framework, and obtains hierarchical feature representation through automatic learning of an image data set, so that a good effect is achieved. However, in the training process, because the parameters are too many and the convergence speed is slow, designing a network with high image reconstruction quality and fast training convergence becomes the key of the problem.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the problems of weak network learning capability, long training time, pending improvement of reconstructed image quality and the like of most networks, the invention provides an image super-resolution method based on residual learning, which takes an original low-resolution image as the input of a model, cascades a deep convolutional network by using a small convolutional kernel, enlarges the receptive field and fully utilizes context information; by introducing the residual block, high-frequency information of the image is learned, the calculation complexity is reduced, the convergence rate of network training is improved, and high-precision real-time reconstruction of the image is realized.
The technical scheme is as follows: in order to achieve the purpose, the invention adopts the technical scheme that:
an image super-resolution reconstruction method based on residual learning comprises the following steps:
step 1), a common data set for training the system is obtained.
The data sets include DIV2K, Set5, and Set14, with DIV2K as the training data Set and Set5 and Set14 as the testing data Set.
And 2) acquiring a low-resolution image to be processed.
And 3) constructing a neural network unit based on the residual error structure to form an image super-resolution network based on residual error learning.
The neural network unit comprises a feature extraction layer of a low-resolution image, a nonlinear mapping layer for detail recovery and an image reconstruction layer.
The feature extraction layer comprises a 3 x 3 convolutional layer, the number of channels is 64, and the feature extraction layer is used for extracting shallow features of the image.
The nonlinear mapping layer is composed of two residual error modules, the extracted shallow layer features are used as input to learn high-frequency information of the image, and the two residual error modules are marked as a first residual error convolution module and a second residual error convolution module. The first residual convolution module inputs the feature diagram with the shape of (n, c, x, y) into 8 convolution layers in sequence, wherein n is the batch size, c is the number of channels of the feature diagram, x and y represent the dimensions of the feature diagram, and then adds the output result to the original feature diagram, and the 8 convolution layers are all 3 × 3 convolution layers with the number of channels being 64. The second residual module inputs 8 convolutional layers in the input sequence with the feature diagram output by the first residual module as the input sequence, the shape of the convolutional layers is (n, c, x, y), and the output result is added with the original feature diagram.
The convolution operation of the feature extraction layer and the nonlinear mapping layer is expressed as the following formula:
wherein f is l Represents the output of the current layer, W l Representing the filter weight of each layer, b l Representing the offset vector of each layer, f l-1 Represents the output of the previous layer, represents the convolution operation of the current layer,
and expressing a nonlinear activation function, wherein the activation function is selected from a corrected Linear Unit (ReLU).
The image reconstruction layer consists of a 3 x 3 deconvolution layer. Inputting the feature image into a deconvolution layer so that the 2-, 3-, or 4-fold reduced image is up-sampled to a high resolution image size of the target, inputting a relationship between the image size and the target size:
wherein, W 1 And H 1 Respectively representing the width and height, W, of the input image 2 And H 2 Respectively representing the width and height of the output image, S being the step size of the convolution kernel movement, F being the size of the convolution kernel, and P representing 0 fill.
Step 4), calculating the loss of the network prediction and the label, adjusting the network parameters according to the loss, and calculating the loss by using a Root Mean Square Error (RMSE) loss function, wherein the loss function formula is as follows:
wherein, H and W respectively represent the height and width of the image, and X (i, j) and Y (i, j) respectively represent pixel points corresponding to the reconstructed image and the original image.
The loss function is calculated by using a mean square and error function, so that the deviation between an observed value and a true value can be measured, the relation between a reconstructed image and a true value is directly displayed, and the prediction effect of the model can be directly obtained.
And during training, comparing the reconstructed image with a corresponding high-resolution image in a training set, and making convergence judgment. If the convergence is reached, outputting a reconstructed image, otherwise, updating the parameters and training until the convergence is reached.
And 5) inputting the data set into a neural network unit for training to obtain a trained neural network unit, and inputting the low-resolution image to be processed obtained in the step 2) into the trained neural network unit.
And 6), outputting the reconstructed high-resolution image, and evaluating the image reconstruction precision.
And (3) measuring the difference between the generated image and the original high-resolution image by using two judgment standards of peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM).
The peak signal-to-noise ratio PSNR measures the image reconstruction quality by calculating the error between corresponding pixel points, and the calculation method is as the formula:
where n represents the number of bits per pixel and MSE refers to the mean square error. A larger peak signal-to-noise ratio PSNR value indicates less distortion.
The structural similarity SSIM comprehensively measures the similarity of images from three angles of brightness, contrast and structure, x and y respectively represent an original high-resolution image and a restored high-resolution image, and the calculation method is as shown in the formula:
SSIM(x,y)＝l(x,y)*c(x,y)*s(x,y)
in the above calculation formula, l (x, y) represents luminance comparison, c (x, y) represents contrast comparison, and s (x, y) represents structural comparison. Wherein, mu x And mu y Representing the mean value of the pixels, delta, of the two images, respectively x And delta y Representing the standard deviation, delta, of two images xy Representing the covariance of the blocks of pixels in the two images, C 1 、C 2 And C 3 Is a constant. SSIM value range is [0,1 ]]A closer result to 1 indicates less distortion.
Preferably: each convolutional layer consists of 64 channels, and the convolutional kernel size is 3 × 3.
Preferably: zero padding of the feature image is required prior to each convolution operation.
Preferably: the number of bits per pixel, n, is 8.
Compared with the prior art, the invention has the following beneficial effects:
the invention increases the extraction capability of the image characteristics, reduces the calculation complexity of a training system, and improves the accuracy of edge prediction, so that the image reconstruction precision is increased.
Drawings
FIG. 1 is a flow chart of the present invention.
Fig. 2 is a diagram comparing the structure of the conventional residual block and the residual block of the present invention.
Fig. 3 is a diagram of a super-resolution network structure.
Detailed Description
The present invention is further illustrated by the following description in conjunction with the accompanying drawings and the specific embodiments, it is to be understood that these examples are given solely for the purpose of illustration and are not intended as a definition of the limits of the invention, since various equivalent modifications will occur to those skilled in the art upon reading the present invention and fall within the limits of the appended claims.
An image super-resolution reconstruction method based on residual learning is disclosed, as shown in fig. 1, and comprises the following steps:
step 1), a common data set for training the system is obtained. The data sets include DIV2K, Set5, and Set14, where DIV2K contains 800 training images, Set5 contains 100 verification images, Set14 contains 100 test data sets, 800 training data sets in DIV2K are used, and Set5 and Set14 are used as test data sets, each instance of data contains a high resolution image and a low resolution image that has been down-sampled by 2, 3, or 4 times.
And 2) acquiring a low-resolution image to be processed.
Step 3), a neural network unit is constructed on the basis of a residual error structure to form an image super-resolution network based on residual error learning, as shown in FIG. 3;
the neural network unit comprises a feature extraction layer of a low-resolution image, a non-linear mapping layer based on a residual block for detail recovery and an image reconstruction layer.
The feature extraction layer comprises a 3 x 3 convolutional layer, the number of channels is 64, and the feature extraction layer is used for extracting shallow features of the image.
The nonlinear mapping layer is composed of two residual error modules, and is marked as a first residual error convolution module and a second residual error convolution module.
Considering that the batch regularization in the conventional residual block regularizes the output characteristics of the convolutional layer, which results in partial loss of information, and because the batch regularization parameters are the same as the convolution operation parameters, a large amount of memory is consumed, and the conventional residual structure is shown in the left part of fig. 2. Therefore, the optimized residual error module is used, compared with the traditional residual error module, the batch normalization layer is removed, the network parameters are reduced, and the network performance is improved. The present invention uses the residual structure shown in the right part of fig. 2.
And taking the extracted shallow features as input to learn high-frequency information of the image. The first residual convolution module inputs the feature map with the shape of (n, c, x, y) into 8 convolution layers in sequence, wherein n is the batch size, c is the number of channels of the feature map, x and y represent the size of the feature map, the output result is added with the image features after feature extraction, and the 8 convolution layers are 3 x 3 convolution layers with the number of channels of 64. The second residual module inputs 8 convolutional layers in the input sequence by taking the (n, c, x, y) feature diagram output by the first residual module as an input sequence, and then adds the output result with the image features after feature extraction. In addition, zero filling needs to be carried out on the characteristic image before convolution operation every time, so that the size of the image after convolution is not changed, and the accuracy of edge prediction can be improved.
The convolution operation of the feature extraction layer and the nonlinear mapping layer can be simply expressed as the following formula:
wherein, W l Representing the filter weight of each layer, b l Representing the offset vector of each layer, f l-1 Represents the output of the previous layer, represents the convolution operation of the current layer,
representing a nonlinear activation function, the activation function of the invention selects a modified Linear Unit (ReLU), f l Representing the output of the current layer. Downsampling the convolved results in view of the pooling layer, resulting in refinementsSection information is lost and the goal of the super resolution algorithm is to increase the detail information of the image, so the present invention does not use a pooling layer.
The image reconstruction layer consists of a 3 x 3 deconvolution layer. The deconvolution layer is used to learn the upsampling kernel of the image features and can be considered as the inverse of the convolution. Deconvolution is performed by transposing the convolution kernel and convolving it with the convolved result. Inputting the feature image into the deconvolution layer such that the 2-, 3-, or 4-fold reduced image is up-sampled to a high resolution image size of the target, the relationship between the input image size and the target size can be expressed as:
wherein, W 1 And H 1 Respectively representing the width and height, W, of the input image 2 And H 2 Respectively representing the width and height of the output image, S being the step size of the convolution kernel movement, F being the size of the convolution kernel, and P representing 0 fill.
Step 4), calculating the loss of the network prediction and the label, adjusting the network parameters according to the loss, and calculating the loss by using an MSE loss function, wherein the loss function formula is as follows:
wherein, H and W respectively represent the height and width of the image, and X (i, j) and Y (i, j) respectively represent pixel points corresponding to the reconstructed image and the original image. And during training, comparing the reconstructed image with a corresponding high-resolution image in a training set, and making convergence judgment. If the convergence is achieved, outputting a reconstructed image; otherwise, the information is reversely propagated from bottom to top, and the parameters are updated to carry out training and learning again until convergence.
Network parameters are adjusted according to the loss, and MSE is minimized by Adam optimization method (using random gradient descent). Initial learning rate of 10 -4 The learning rate attenuation period is 200 steps, and the number of training iteration steps is 4000 steps.
Each convolution layer in the neural network unit is composed of 64 channels, and the size of the convolution kernel is 3 x 3. In addition, zero filling needs to be carried out on the characteristic image before convolution operation every time, so that the size of the image is not changed after convolution, and the accuracy of edge prediction can be improved.
And 5) inputting the data set into a neural network unit for training to obtain a trained neural network unit, and inputting the low-resolution image to be processed obtained in the step 2) into the trained neural network unit.
And 6), outputting the reconstructed high-resolution image, and evaluating the image reconstruction precision.
The evaluation standard of the reconstructed image uses two judgment standards of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) to measure the difference between the generated image and the original high-resolution image.
PSNR measures image reconstruction quality by calculating errors between corresponding pixel points, and the calculation method is as the formula (4):
where n represents the number of bits per pixel, typically 8, and MSE refers to the mean square error. PSNR is in dB, with larger values indicating less distortion.
The similarity of the images by the SSIM is comprehensively measured from three aspects of brightness, contrast and structure. Assuming that x and y represent the original high-resolution image and the restored high-resolution image, respectively, the calculation method is as shown in equations (5) to (8):
SSIM(x,y)＝l(x,y)*c(x,y)*s(x,y) (5)
in the above calculation formula, l (x, y) represents luminance comparison, c (x, y) represents contrast comparison, and s (x, y) represents structural comparison; wherein, mu x And mu y Representing the mean value of the pixels, delta, of the two images, respectively x And delta y Representing the standard deviation, delta, of two images xy Representing the covariance of the blocks of pixels in the two images, C 1 、C 2 And C 3 Is constant and is used for avoiding system errors caused by the denominator being 0. SSIM value range is [0,1 ]]A closer result to 1 indicates less distortion.
The invention provides an image super-resolution method based on residual error learning, aiming at the problems of weak network learning capability, long training time, pending improvement of reconstructed image quality and the like of most of current networks. Through verification, the system has excellent performance on image reconstruction accuracy, and achieves better super-resolution reconstruction performance.
The above description is only of the preferred embodiments of the present invention, and it should be noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.
Claims (5)
1. An image super-resolution reconstruction method based on residual learning is characterized by comprising the following steps:
step 1), acquiring a public data set for training a system;
step 2), acquiring a low-resolution image to be processed;
step 3), constructing a neural network unit on the basis of a residual error structure to form an image super-resolution network based on residual error learning;
the neural network unit comprises a feature extraction layer of a low-resolution image, a nonlinear mapping layer for detail recovery and an image reconstruction layer;
the feature extraction layer comprises a 3 x 3 convolution layer, the number of channels is 64, and the feature extraction layer is used for extracting shallow features of the image;
the nonlinear mapping layer is composed of two residual error modules, the extracted shallow layer characteristics are used as input to learn high-frequency information of the image, and the two residual error modules are marked as a first residual error convolution module and a second residual error convolution module; the first residual convolution module inputs the feature diagram with the shape of (n, c, x, y) into 8 convolution layers in sequence, wherein n is the batch size, c is the number of channels of the feature diagram, x and y represent the dimensions of the feature diagram, the output result is added with the original feature diagram, and the 8 convolution layers are all 3 x 3 convolution layers with the number of channels of 64; the second residual error module takes the characteristic diagram output by the first residual error module as an input sequence to input 8 convolutional layers, the shape of the convolutional layers is (n, c, x and y), and then the output result is added with the original characteristic diagram;
the convolution operation of the feature extraction layer and the nonlinear mapping layer is expressed as the following formula:
wherein f is l Represents the output of the current layer, W l Representing the filter weight of each layer, b l Representing the offset vector of each layer, f l-1 Represents the output of the previous layer, represents the convolution operation of the current layer,
expressing a nonlinear activation function, wherein the activation function selects a modified linear unit ReLU;
the image reconstruction layer is composed of a 3 x 3 deconvolution layer; inputting the feature image into a deconvolution layer so that the 2-, 3-, or 4-fold reduced image is up-sampled to a high resolution image size of the target, inputting a relationship between the image size and the target size:
wherein, W 1 And H 1 Respectively representing the width and height, W, of the input image 2 And H 2 Respectively representing the width and the height of an output image, wherein S is the step length of the movement of a convolution kernel, F is the size of the convolution kernel, and P represents 0 filling;
step 4), calculating the loss of the network prediction and the label, adjusting the network parameters according to the loss, and calculating the loss by using a root mean square error loss function, wherein the loss function formula is as follows:
h and W respectively represent the height and width of the image, and X (i, j) and Y (i, j) respectively represent pixel points corresponding to the reconstructed image and the original image;
during training, the reconstructed image is compared with a high-resolution image corresponding to a training set, and convergence judgment is made; if the convergence is reached, outputting a reconstructed image, otherwise, updating the parameters and training until the convergence is reached;
step 5), inputting the data set into a neural network unit for training to obtain a trained neural network unit, and inputting the low-resolution image to be processed obtained in the step 2) into the trained neural network unit;
step 6), outputting the reconstructed high-resolution image and evaluating the image reconstruction precision;
measuring the difference between the generated image and the original high-resolution image by using two judgment standards of peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM);
the peak signal-to-noise ratio PSNR measures the image reconstruction quality by calculating the error between corresponding pixel points, and the calculation method is as the formula:
wherein n represents the number of bits per pixel, and MSE refers to the mean square error; the larger the peak signal-to-noise ratio PSNR value is, the smaller the distortion is;
the structural similarity SSIM comprehensively measures the similarity of images from three angles of brightness, contrast and structure, x and y respectively represent an original high-resolution image and a restored high-resolution image, and the calculation method is as shown in the formula:
SSIM(x,y)＝l(x,y)*c(x,y)*s(x,y)
in the above calculation formula, l (x, y) represents luminance comparison, c (x, y) represents contrast comparison, and s (x, y) represents structural comparison; wherein, mu x And mu y Representing the mean value of the pixels, delta, of the two images, respectively x And delta y Representing the standard deviation, delta, of two images xy Representing the covariance of the blocks of pixels in the two images, C 1 、C 2 And C 3 Is a constant; SSIM value range is [0,1 ]]A closer result to 1 indicates less distortion.
2. The image super-resolution reconstruction method based on residual learning of claim 1, wherein: each convolutional layer consists of 64 channels, and the convolutional kernel size is 3 × 3.
3. The image super-resolution reconstruction method based on residual learning of claim 1, wherein: zero padding of the feature image is required prior to each convolution operation.
4. The image super-resolution reconstruction method based on residual learning of claim 1, wherein: the number of bits per pixel, n, is 8.
5. The image super-resolution reconstruction method based on residual learning of claim 1, wherein: the data sets include DIV2K, Set5, and Set14, with DIV2K as the training data Set and Set5 and Set14 as the testing data Set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010542676.9A CN111754403B (en) | 2020-06-15 | 2020-06-15 | Image super-resolution reconstruction method based on residual learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010542676.9A CN111754403B (en) | 2020-06-15 | 2020-06-15 | Image super-resolution reconstruction method based on residual learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111754403A CN111754403A (en) | 2020-10-09 |
| CN111754403B true CN111754403B (en) | 2022-08-12 |
Family
ID=72676098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010542676.9A Active CN111754403B (en) | 2020-06-15 | 2020-06-15 | Image super-resolution reconstruction method based on residual learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111754403B (en) |
Families Citing this family (50)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112261415B (en) * | 2020-10-23 | 2022-04-08 | 青海民族大学 | Image compression coding method based on overfitting convolution self-coding network |
| CN112288630A (en) * | 2020-10-27 | 2021-01-29 | 武汉大学 | Super-resolution image reconstruction method and system based on improved wide-depth neural network |
| CN112184559B (en) * | 2020-11-09 | 2022-07-22 | 西北工业大学 | Super-resolution image abnormal target detection method and system of photoelectric navigation system |
| CN112288658B (en) * | 2020-11-23 | 2023-11-28 | 杭州师范大学 | Underwater image enhancement method based on multi-residual joint learning |
| CN112348745B (en) * | 2020-11-26 | 2022-10-14 | 河海大学 | Video super-resolution reconstruction method based on residual convolutional network |
| CN112581366B (en) * | 2020-11-30 | 2022-05-20 | 黑龙江大学 | Portable image super-resolution system and system construction method |
| CN112508792A (en) * | 2020-12-22 | 2021-03-16 | 北京航空航天大学杭州创新研究院 | Single-image super-resolution method and system of deep neural network integration model based on online knowledge migration |
| CN112907441B (en) * | 2020-12-29 | 2023-05-30 | 中央财经大学 | Space downscaling method based on super-resolution of ground water satellite image |
| CN112734643A (en) * | 2021-01-15 | 2021-04-30 | 重庆邮电大学 | Lightweight image super-resolution reconstruction method based on cascade network |
| CN112785501B (en) * | 2021-01-20 | 2023-09-01 | 北京百度网讯科技有限公司 | Text image processing method, text image processing device, text image processing equipment and storage medium |
| CN112767251B (en) * | 2021-01-20 | 2023-04-07 | 重庆邮电大学 | Image super-resolution method based on multi-scale detail feature fusion neural network |
| CN112907444B (en) * | 2021-02-07 | 2024-03-22 | 中国科学院沈阳自动化研究所 | Terahertz image super-resolution reconstruction method based on complex domain zero sample learning |
| CN112819823B (en) * | 2021-03-02 | 2023-10-20 | 中山大学 | Round hole detection method, system and device for furniture plate |
| CN112927136B (en) * | 2021-03-05 | 2022-05-10 | 江苏实达迪美数据处理有限公司 | Image reduction method and system based on convolutional neural network domain adaptation |
| CN112926598B (en) * | 2021-03-08 | 2021-12-07 | 南京信息工程大学 | Image copy detection method based on residual error domain deep learning characteristics |
| CN112862690B (en) * | 2021-03-09 | 2022-08-30 | 湖北工业大学 | Transformers-based low-resolution image super-resolution method and system |
| CN113096207B (en) * | 2021-03-16 | 2023-01-13 | 天津大学 | Rapid magnetic resonance imaging method and system based on deep learning and edge assistance |
| CN112837224A (en) * | 2021-03-30 | 2021-05-25 | 哈尔滨理工大学 | Super-resolution image reconstruction method based on convolutional neural network |
| CN113077382B (en) * | 2021-04-27 | 2024-01-12 | 东南大学 | Face beautifying image restoration method based on BEMD and deep learning |
| CN113298714B (en) * | 2021-05-24 | 2024-04-26 | 西北工业大学 | Image cross-scale super-resolution method based on deep learning |
| CN113222825B (en) * | 2021-06-03 | 2023-04-18 | 北京理工大学 | Infrared image super-resolution reconstruction method based on visible light image training and application |
| CN113269818B (en) * | 2021-06-09 | 2023-07-25 | 河北工业大学 | Deep learning-based seismic data texture feature reconstruction method |
| CN113538231B (en) * | 2021-06-17 | 2024-04-02 | 杭州电子科技大学 | Single image super-resolution reconstruction system and method based on pixel distribution estimation |
| CN113538307B (en) * | 2021-06-21 | 2023-06-20 | 陕西师范大学 | Synthetic aperture imaging method based on multi-view super-resolution depth network |
| CN113506215B (en) * | 2021-06-22 | 2023-07-04 | 中国公路工程咨询集团有限公司 | Super-resolution image reconstruction method and device based on wide activation and electronic equipment |
| CN113495291A (en) * | 2021-07-27 | 2021-10-12 | 成都爱为贝思科技有限公司 | Method for intelligently and quantitatively evaluating amplitude preservation of forward gather based on deep learning |
| CN113674185B (en) * | 2021-07-29 | 2023-12-08 | 昆明理工大学 | Weighted average image generation method based on fusion of multiple image generation technologies |
| CN113610912B (en) * | 2021-08-13 | 2024-02-02 | 中国矿业大学 | System and method for estimating monocular depth of low-resolution image in three-dimensional scene reconstruction |
| CN113947642B (en) * | 2021-10-18 | 2024-06-04 | 北京航空航天大学 | X-space magnetic particle imaging deconvolution method |
| CN114049254B (en) * | 2021-10-29 | 2022-11-29 | 华南农业大学 | Low-pixel ox-head image reconstruction and identification method, system, equipment and storage medium |
| CN114022360B (en) * | 2021-11-05 | 2024-05-03 | 长春理工大学 | Rendered image super-resolution system based on deep learning |
| WO2023082162A1 (en) * | 2021-11-12 | 2023-05-19 | 华为技术有限公司 | Image processing method and apparatus |
| CN114092330B (en) * | 2021-11-19 | 2024-04-30 | 长春理工大学 | Light-weight multi-scale infrared image super-resolution reconstruction method |
| CN114463176B (en) * | 2022-01-25 | 2024-03-01 | 河南大学 | Image super-resolution reconstruction method based on improved ESRGAN |
| WO2023155032A1 (en) * | 2022-02-15 | 2023-08-24 | 华为技术有限公司 | Image processing method and image processing apparatus |
| CN114972040B (en) * | 2022-07-15 | 2023-04-18 | 南京林业大学 | Speckle image super-resolution reconstruction method for laminated veneer lumber |
| CN115564808B (en) * | 2022-09-01 | 2023-08-25 | 宁波大学 | Multi-resolution hyperspectral/SAR image registration method based on public space-spectrum subspace |
| CN115272084B (en) * | 2022-09-27 | 2022-12-16 | 成都信息工程大学 | High-resolution image reconstruction method and device |
| CN115760655B (en) * | 2022-11-07 | 2023-06-23 | 贵州大学 | Interference array imaging method based on big data technology |
| CN115994893B (en) * | 2022-12-05 | 2023-08-15 | 中国科学院合肥物质科学研究院 | Sealing element weak and small leakage target detection method based on low-resolution infrared image |
| CN115619646B (en) * | 2022-12-09 | 2023-04-18 | 浙江大学 | Deep learning optical illumination super-resolution imaging method for sub-fifty nano-structure |
| CN116402682B (en) * | 2023-03-29 | 2024-02-09 | 辽宁工业大学 | Image reconstruction method and system based on differential value dense residual super-resolution |
| CN116664677B (en) * | 2023-05-24 | 2024-06-14 | 南通大学 | Sight estimation method based on super-resolution reconstruction |
| CN117132468B (en) * | 2023-07-11 | 2024-05-24 | 汕头大学 | Curvelet coefficient prediction-based super-resolution reconstruction method for precise measurement image |
| CN116934618B (en) * | 2023-07-13 | 2024-06-11 | 江南大学 | Image halftone method, system and medium based on improved residual error network |
| CN117111013B (en) * | 2023-08-22 | 2024-04-30 | 南京慧尔视智能科技有限公司 | Radar target tracking track starting method, device, equipment and medium |
| CN117291802A (en) * | 2023-09-27 | 2023-12-26 | 华北电力大学（保定） | Image super-resolution reconstruction method and system based on composite network structure |
| CN117611954B (en) * | 2024-01-19 | 2024-04-12 | 湖北大学 | Method, device and storage device for evaluating effectiveness of infrared video image |
| CN117575916B (en) * | 2024-01-19 | 2024-04-30 | 青岛漫斯特数字科技有限公司 | Image quality optimization method, system, equipment and medium based on deep learning |
| CN117788292B (en) * | 2024-01-24 | 2024-06-11 | 合肥工业大学 | Sub-pixel displacement-based super-resolution reconstruction system for sequence image |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675321A (en) * | 2019-09-26 | 2020-01-10 | 兰州理工大学 | Super-resolution image reconstruction method based on progressive depth residual error network |
-
2020
- 2020-06-15 CN CN202010542676.9A patent/CN111754403B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110675321A (en) * | 2019-09-26 | 2020-01-10 | 兰州理工大学 | Super-resolution image reconstruction method based on progressive depth residual error network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111754403A (en) | 2020-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111754403B (en) | Image super-resolution reconstruction method based on residual learning | |
| CN109118432B (en) | Image super-resolution reconstruction method based on rapid cyclic convolution network | |
| CN110675321A (en) | Super-resolution image reconstruction method based on progressive depth residual error network | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| CN111861884B (en) | Satellite cloud image super-resolution reconstruction method based on deep learning | |
| CN107784628B (en) | Super-resolution implementation method based on reconstruction optimization and deep neural network | |
| CN109949224B (en) | Deep learning-based cascade super-resolution reconstruction method and device | |
| CN109214989A (en) | Single image super resolution ratio reconstruction method based on Orientation Features prediction priori | |
| CN111861886B (en) | Image super-resolution reconstruction method based on multi-scale feedback network | |
| CN111369466B (en) | Image distortion correction enhancement method of convolutional neural network based on deformable convolution | |
| CN112801904B (en) | Hybrid degraded image enhancement method based on convolutional neural network | |
| CN112819705B (en) | Real image denoising method based on mesh structure and long-distance correlation | |
| CN113538234A (en) | Remote sensing image super-resolution reconstruction method based on lightweight generation model | |
| CN112699844A (en) | Image super-resolution method based on multi-scale residual error level dense connection network | |
| CN109064407A (en) | Intensive connection network image super-resolution method based on multi-layer perception (MLP) layer | |
| CN115880158A (en) | Blind image super-resolution reconstruction method and system based on variational self-coding | |
| CN110288529B (en) | Single image super-resolution reconstruction method based on recursive local synthesis network | |
| CN113487482B (en) | Self-adaptive super-resolution method based on meta-shift learning | |
| CN113160057B (en) | RPGAN image super-resolution reconstruction method based on generation countermeasure network | |
| CN117132472B (en) | Forward-backward separable self-attention-based image super-resolution reconstruction method | |
| CN112184552B (en) | Sub-pixel convolution image super-resolution method based on high-frequency feature learning | |
| CN115511705A (en) | Image super-resolution reconstruction method based on deformable residual convolution neural network | |
| CN111008930B (en) | Fabric image super-resolution reconstruction method | |
| CN115293983A (en) | Self-adaptive image super-resolution restoration method fusing multi-level complementary features | |
| CN113298719B (en) | Feature separation learning-based super-resolution reconstruction method for low-resolution fuzzy face image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |