CN111161150A - Image super-resolution reconstruction method based on multi-scale attention cascade network - Google Patents

Image super-resolution reconstruction method based on multi-scale attention cascade network Download PDF

Info

Publication number
CN111161150A
CN111161150A CN201911392155.3A CN201911392155A CN111161150A CN 111161150 A CN111161150 A CN 111161150A CN 201911392155 A CN201911392155 A CN 201911392155A CN 111161150 A CN111161150 A CN 111161150A
Authority
CN
China
Prior art keywords
convolution
image
scale attention
output
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911392155.3A
Other languages
Chinese (zh)
Other versions
CN111161150B (en
Inventor
付利华
李宗刚
张博
陈辉
赵茹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN201911392155.3A priority Critical patent/CN111161150B/en
Publication of CN111161150A publication Critical patent/CN111161150A/en
Application granted granted Critical
Publication of CN111161150B publication Critical patent/CN111161150B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network, which comprises the steps of extracting shallow features of a low-resolution image by using convolution operation; then, inputting the shallow layer characteristics into a characteristic extraction subnet to obtain cascade characteristics; further, the cascade characteristic passes through a convolution layer with convolution kernel of 1 to obtain an optimized characteristic; inputting the optimized features into an image depth learning up-sampling module to obtain a reconstructed image
Figure DDA0002345297760000011
At the same time, for low resolution images ILRObtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Figure DDA0002345297760000012
Finally, the image is reconstructed
Figure DDA0002345297760000013
And
Figure DDA0002345297760000014
fusing to obtain the final high-resolution reconstructed image ISR. The method is suitable for super-resolution reconstruction of the image, and the obtained reconstructed image has high definition, more real texture and good perception effect.

Description

Image super-resolution reconstruction method based on multi-scale attention cascade network
Technical Field
The invention belongs to the field of image restoration, relates to an image super-resolution reconstruction method, and particularly relates to an image super-resolution reconstruction method based on a multi-scale attention cascade network.
Background
Single image super-resolution reconstruction (SISR) has recently received a great deal of attention. In general, the purpose of SISR is to produce a visually High Resolution (HR) output from a Low Resolution (LR) input. However, the whole process is completely irreversible, since there are multiple solutions to the mapping between LR and HR. Therefore, a number of image super-resolution reconstruction (SR) methods have been proposed, ranging from earlier interpolation-based methods and model-based methods to more recently deep learning-based methods.
Interpolation-based methods are simple and fast, but cannot be applied more widely because of poor image quality. For more flexible SR methods, more advanced model-based methods and sparse matrix methods have been proposed by exploiting strong image priors, such as non-local similarities, and although such model-based methods can flexibly produce relatively high quality HR images, they still have some drawbacks: 1) such methods often involve a time-consuming optimization process; 2) when the image statistics are biased from the image, the reconstruction performance may degrade rapidly.
At present, Convolutional Neural Networks (CNNs) have been shown to provide significant performance in SISR problems. However, the existing SR model has the following problems: 1) the method has the characteristics of insufficient utilization: most approaches increase the depth of the network blindly to improve the performance of the network, but neglect to fully exploit the image feature characteristics of the LR. As the depth of the network increases, the information gradually disappears during transmission. How to fully exploit these features is crucial for the network to reconstruct high quality images. 2) SR image detail loss: using the interpolated LR image as input will increase the computational complexity while not facilitating the learning of the final image details. Therefore, recent methods focus more on enlarging LR images. However, the effect of the SR image cannot be improved only by the single network structure.
In order to solve the problems, the invention provides a novel image super-resolution reconstruction method based on deep learning.
Disclosure of Invention
The invention aims to solve the problems that: in the existing super-resolution reconstruction method based on deep learning, most methods increase the depth of the network blindly to improve the performance of the network, but neglect to fully utilize the characteristics of LR images; moreover, as the depth of the network increases, the characteristic information gradually disappears in the transmission process; the LR image amplified by interpolation is used as the input of the network, which increases the computational complexity and is not favorable for the network to learn the image details. A new super-resolution reconstruction method based on deep learning is required to be provided, so that the impression and robustness after image super-resolution reconstruction are improved.
In order to solve the above problems, the present invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network, which extracts the features of an LR image by using a U-shaped multi-scale attention block group and performs super-resolution reconstruction on the LR image by combining interpolation reconstruction and deep learning-based network reconstruction, and comprises the following steps:
1) low resolution image ILRAs input to the multiscale attention cascade network, pair ILRPerforming convolution operation to extract shallow feature F0；
2) Shallow feature F0Inputting a feature extraction subnet formed by n multi-scale attention blocks, cascading features output by each multi-scale attention block in the subnet to obtain cascading features Fc；
3) Will cascade feature FcReducing the number of parameters by a convolution layer with convolution kernel of 1 to obtain optimized characteristic vector FdOptimized feature vector FdThe training of data and the feature extraction can be more effectively and intuitively carried out;
4) feature vector F to be optimizeddIn an input image deep learning up-sampling module, a reconstructed image is obtainedImage
Figure BDA0002345297740000021
5) To ILRObtaining a reconstructed image using an interpolation algorithm
Figure BDA0002345297740000022
Will be provided with
Figure BDA0002345297740000023
And
Figure BDA0002345297740000024
fusing to obtain the final reconstructed image ISR。
As a further preferable mode, the obtaining of the cascade characteristic in the step 2) specifically includes:
2.1) shallow feature F0Inputting the feature extraction sub-network composed of n multi-scale attention blocks to respectively obtain n features Fi，i＝1,2,3,…,n。
For the ith multi-scale attention block, the input of the ith multi-scale attention block is the characteristic output F of the previous multi-scale attention blocki-1The output characteristic is Fi。
Each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module.
2.1.1) for the U-shaped structure module in the ith multi-scale attention block, the U-shaped structure module is formed by connecting non-local means, 3 × 3 convolution, 5 × 5 convolution, 7 × 7 convolution, attention mechanism, 5 × 5 convolution, 3 × 3 convolution and non-local means in series; furthermore, there is one Concat layer between the two 3 × 3 convolutions; there is one Concat layer between the two 5 x 5 convolution convolutions. The characteristic input of which is the output characteristic F of the previous multi-scale attention blocki-1After processing by the U-shaped structure module, the characteristic F is obtainedi,0；
For the feature input F of non-local mean, firstly, the input features F are respectively input into three convolution layers in parallel to obtain three features Fx,Fy,FzThen, these three features are operated in through ConcatRow feature fusion and input of fused features into subsequent convolutional layers to obtain feature FwAnd finally, the feature FwAnd adding the input characteristic F point by point to obtain the output characteristic of the non-local mean value.
For the input feature F of the attention mechanism, first, the feature F is input to the global pooling layer extraction channel information descriptor MavgThen, the channel information descriptor M is describedavgInputting the data into the subsequent two convolution layers for further processing to obtain M, and finally multiplying the M and the characteristic F channel by channel to obtain the output characteristic of the attention mechanism.
2.1.2) for the bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is composed of two bottleneck layers in series. The input of which is the output characteristic F of the U-shaped structure module in the multi-scale attention blocki,0Obtaining a characteristic F after the processing of the bottleneck layer structure modulei,2；
2.1.3) for the residual module in the ith multi-scale attention block, the residual module is the point-by-point addition of the output characteristics of the previous multi-scale attention block and the output characteristics of the bottleneck layer structure. The input of which is the output characteristic F of the previous multi-scale attention blocki-1And the output characteristics F of the bottleneck layer structure module in the multi-scale attention blocki,2Obtaining a characteristic F after processing by a residual error modulei；
2.2) feature F output for n multiscale attention blocksiI-1, 2,3, …, n using Concat ligation, yields cascade characteristic Fc：
Fc＝Concat(F1,F2,...,Fn)
Where Concat (. cndot.) represents the operation of concatenating the features of the n multi-scale attention block outputs.
As a further preferable mode, the step 3) is specifically:
3.1) cascading feature FcInputting the convolution layer with convolution kernel of 1, reducing the number of parameters and obtaining optimized characteristic Fd:
Fd＝Conv1×1(Fc)
Wherein，Conv1×1(. cndot.) represents a convolution operation with a convolution kernel of 1.
As a further preferred mode, the step 4) of obtaining the reconstructed image through the image deep learning upsampling module
Figure BDA0002345297740000041
The image deep learning up-sampling module consists of a convolution layer with convolution kernel of 3 and a sub-pixel convolution layer. Optimized feature FdObtaining a reconstructed image through an image deep learning up-sampling module
Figure BDA0002345297740000042
The specific process comprises the following steps:
4.1) optimized feature F using a convolution layer pair with convolution kernel 3dRearranging to obtain a characteristic Fe：
Fe＝Conv3×3(Fd)
Wherein, Conv3×3(. cndot.) represents a convolution operation with a convolution kernel of 3.
4.2) feature F after rearrangementeInputting into a sub-pixel convolution layer, enlarging it to corresponding scale, and obtaining reconstructed image
Figure BDA0002345297740000043
Figure BDA0002345297740000044
Wherein HSp(. cndot.) denotes a sub-pixel convolution operation.
As a further preferred mode, the step 5) of obtaining the final reconstruction image ISRThe method comprises the following specific steps:
5.1) for low resolution images ILRObtaining a reconstructed image after interpolation by using a bicubic linear interpolation algorithm
Figure BDA0002345297740000045
5.2) reconstructing the image obtained by the image deep learning up-sampling module
Figure BDA0002345297740000046
And the reconstructed image after the interpolation of the bicubic linear interpolation algorithm
Figure BDA0002345297740000047
Fusing to obtain the final reconstructed image ISR：
Figure BDA0002345297740000048
Although the super-resolution reconstruction method using the interpolation algorithm is high in reconstruction speed, redundant information is added, and the super-resolution reconstruction effect is poor; the image super-resolution reconstruction method based on the deep learning lacks a reasonable guide in the reconstruction process, so that partial detail information in the reconstructed image is lost. According to the method, the reconstruction result of the interpolation algorithm is used as the guidance of the reconstruction process of the image super-resolution reconstruction method based on the deep learning, so that the image super-resolution reconstruction effect can be improved, and the redundant information generated by the interpolation algorithm can be removed.
The invention provides an image super-resolution reconstruction method of a multi-scale attention cascade network, which comprises the steps of extracting shallow features of a low-resolution image by using convolution operation; then, inputting the shallow layer characteristics into a characteristic extraction subnet to obtain cascade characteristics; further, the cascade characteristic passes through a convolution layer with convolution kernel of 1 to obtain an optimized characteristic vector; inputting the optimized feature vector into an image deep learning up-sampling module to obtain a reconstructed image
Figure BDA0002345297740000051
At the same time, for low resolution images ILRObtaining a reconstructed image using an interpolation algorithm
Figure BDA0002345297740000052
Finally, the image is reconstructed
Figure BDA0002345297740000053
And
Figure BDA0002345297740000054
fusing to obtain the final high-resolution reconstructed image ISR. By applying the method, the problem of low detail definition of the reconstructed image of the existing image super-resolution reconstruction method is solved, and the impression is improved; the method also solves the problem that the existing image super-resolution reconstruction algorithm based on deep learning cannot fully extract the low-resolution image features. The method is suitable for super-resolution reconstruction of the image, and the reconstructed image obtained by using the method for super-resolution reconstruction of the image has high definition, more real texture and good perception effect.
Advantageous effects
Firstly, the invention adopts a group of multi-scale attention blocks to extract the characteristics of the low-resolution image, and can fully utilize the detail information of the low-resolution image; secondly, super-resolution reconstruction is realized by combining interpolation reconstruction and deep learning-based network reconstruction, and the reconstruction effect of the reconstructed image is improved.
Drawings
FIG. 1 is a flow chart of an image super-resolution reconstruction method based on a multi-scale attention cascade network according to the present invention;
FIG. 2 is a network structure diagram of the image super-resolution method based on the multi-scale attention cascade network of the present invention;
FIG. 3 is a block diagram of a multi-scale attention module designed by the present invention;
Detailed Description
The invention provides an image super-resolution reconstruction method of a multi-scale attention cascade network, which comprises the steps of extracting shallow features of a low-resolution image by using convolution operation; then, inputting the shallow layer characteristics into a characteristic extraction subnet to obtain cascade characteristics; further, the cascade characteristic passes through a convolution layer with convolution kernel of 1 to obtain an optimized characteristic vector; inputting the optimized feature vector into an image deep learning up-sampling module to obtain a reconstructed image
Figure BDA0002345297740000061
At the same time, for low resolution images ILRObtaining a reconstructed image using an interpolation algorithm
Figure BDA0002345297740000062
Finally, the image is reconstructed
Figure BDA0002345297740000063
And
Figure BDA0002345297740000064
fusing to obtain the final high-resolution reconstructed image ISR. The method is suitable for super-resolution reconstruction of images, and the super-resolution reconstruction is carried out by using the method, so that the obtained high-resolution images are high in definition, more real in texture and good in perception effect.
As shown in fig. 1, the present invention comprises the steps of:
1) low resolution image ILRAs input to the multi-scale attention cascade network, a convolution operation is used to extract the low-resolution image ILRMiddle extracted shallow feature F0：
F0＝Hsf(ILR)
Wherein Hsf() Representing a convolution operation.
2) Shallow feature F0Inputting a feature extraction subnet formed by a group of multi-scale attention blocks, cascading features output by each multi-scale attention block in the subnet by using Cancat operation to obtain cascading features Fc；
2.1) shallow feature F0Inputting the feature extraction sub-network composed of n multi-scale attention blocks to respectively obtain n features Fi，i＝1,2,3,…,n。
For the ith multi-scale attention block, the input of the ith multi-scale attention block is the characteristic output F of the previous multi-scale attention blocki-1The output characteristic is Fi。
Each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module.
2.1.1) for the U-shaped structure module in the ith multi-scale attention block, the module is composed of non-local mean, 3 × 3 convolution, 5 × 5 convolution, 7 × 7 convolution, attention mechanism, 5 × 5 convolution, 3 × 3 convolution and non-local mean in series connection; furthermore, there is one Concat layer between the two 3 × 3 convolutions; there is one Concat layer between the two 5 x 5 convolution convolutions. The input of which is the output characteristic F of the previous multi-scale attention blocki-1The output of the U-shaped structural module is characteristic Fi,0；
Will be characterized by Fi-1Input to a U-shaped building block to obtain a feature Fi,0：
Fi,0＝Hu(Fi-1)
Wherein HuRepresenting the extraction of features using a U-shaped structure module.
2.1.2) for the bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is composed of two bottleneck layers in series. The input of which is the output characteristic F of the U-shaped structure module in the multi-scale attention blocki,0The output of the bottleneck layer structure module is a characteristic Fi,2；
Each bottleneck layer structure module consists of two bottleneck layers. The feature input of the first bottleneck layer is the feature output F of the previous multi-scale attention blocki-1The characteristic output is Fi,1：
Fi,1＝Hb(Fi-1)
Wherein Hb(. cndot.) represents the first bottleneck layer operation.
Further extracting the characteristics F of the U-shaped structural modulei,0And the output F of the first bottleneck layeri,1Inputting the data into a second bottleneck layer for detail fusion to obtain a characteristic Fi,2：
Fi,2＝Hc(Fi,0,Fi,1)
Wherein Hc(. cndot.) represents a second bottleneck layer operation.
2.1.3) for the residual module in the ith multi-scale attention block, the module is the output characteristic of the previous multi-scale attention block and the output characteristic of the bottleneck layer structureAdding point by point. The input of which is the output characteristic F of the previous multi-scale attention blocki-1And the output characteristics F of the bottleneck layer structure module in the multi-scale attention blocki,2The output of the residual block is a feature Fi；
Output characteristics F of a previous multi-scale attention blocki-1And feature Fi,2Inputting the residual error into a residual error module to obtain a characteristic Fi：
Fi＝Fi-1+Fi,2
2.2) feature F output for n multiscale attention blocksiI-1, 2,3, …, n using Concat ligation, yields cascade characteristic Fc：
Fc＝Concat(F1,F2,...,Fn)
Where Concat (. cndot.) represents the operation of concatenating the features of the n multi-scale attention block outputs.
3) Will cascade feature FcReducing the number of parameters by a convolution layer with convolution kernel of 1 to obtain optimized characteristic vector FdOptimized feature vector FdThe training of data and the feature extraction can be more effectively and intuitively carried out;
3.1) cascading feature FcInputting the convolution layer with convolution kernel of 1, reducing the number of parameters and obtaining optimized characteristic Fd:
Fd＝Conv1×1(Fc)
Wherein, Conv1×1(. cndot.) represents a convolution operation with a convolution kernel of 1.
4) Feature vector F to be optimizeddObtaining a reconstructed image in an input image deep learning up-sampling module
Figure BDA0002345297740000081
4.1) optimized feature F using a convolution layer pair with convolution kernel 3dRearranging to obtain a characteristic Fe：
Fe＝Conv3×3(Fd)
Wherein, Conv3×3(. cndot.) represents a convolution operation with a convolution kernel of 3.
4.2) feature F after rearrangementeInputting into a sub-pixel convolution layer, enlarging it to corresponding scale, and obtaining reconstructed image
Figure BDA0002345297740000082
Figure BDA0002345297740000083
Wherein HSp(. cndot.) represents a subpixel convolution layer operation.
5) To ILRObtaining a reconstructed image using an interpolation algorithm
Figure BDA0002345297740000084
Will be provided with
Figure BDA0002345297740000085
And
Figure BDA0002345297740000086
fusing to obtain the final reconstructed image ISR。
5.1) for low resolution images ILRObtaining a reconstructed image after interpolation by using a bicubic linear interpolation algorithm
Figure BDA0002345297740000087
5.2) reconstructing the image obtained by the image deep learning up-sampling module
Figure BDA0002345297740000088
And the reconstructed image after the interpolation of the bicubic linear interpolation algorithm
Figure BDA0002345297740000089
Fusing to obtain the final reconstructed image ISR：
Figure BDA00023452977400000810
Wherein, ISRAnd obtaining a final image super-resolution reconstruction result.
The invention has wide application in the field of image restoration, such as putting large-size photo billboards, reducing image transmission pressure, enlarging thumbnails and the like. The present invention will now be described in detail with reference to the accompanying drawings.
1) Low resolution image ILRAs input to the multiscale attention cascade network, pair ILRPerforming convolution operation to extract shallow feature F0；
2) Shallow feature F0Inputting a feature extraction subnet formed by a group of multi-scale attention blocks, cascading features output by each multi-scale attention block in the subnet to obtain cascading features Fc；
3) Will cascade feature FcReducing the number of parameters by a convolution layer with convolution kernel of 1 to obtain optimized characteristic vector Fd；
4) Feature vector F to be optimizeddObtaining a reconstructed image in an input image deep learning up-sampling module
Figure BDA0002345297740000091
5) For low resolution image ILRObtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Figure BDA0002345297740000092
Will be provided with
Figure BDA0002345297740000093
And
Figure BDA0002345297740000094
fusing to obtain the final reconstructed image ISR。
The method is realized based on a PyTorch deep learning framework under NVIDIA GeForce GTX 1080Ti and Ubuntu 16.0464 bit operating systems.
The invention provides an image super-resolution reconstruction method based on a multi-scale attention cascade network. The method is suitable for super-resolution reconstruction of images, and the reconstructed images obtained by using the method for super-resolution reconstruction are high in definition, more real in texture and good in perception effect.

Claims (8)

1. An image super-resolution reconstruction method based on a multi-scale attention cascade network is characterized by comprising the following steps:
step 1) low resolution image ILRAs input to the multiscale attention cascade network, pair ILRPerforming a convolution operation to extract shallow feature F0；
Step 2) shallow feature F0Inputting a feature extraction subnet formed by n multi-scale attention blocks, cascading features output by each multi-scale attention block in the subnet to obtain cascading features Fc；
Step 3) cascading characteristics FcObtaining optimized characteristic vector F by convolution layer with convolution kernel of 1dOptimized feature vector FdThe data training and feature extraction can be more effectively and intuitively carried out, and the specific expression is as follows:
Fd＝Conv1×1(Fc)
wherein, Conv1×1(. -) represents a convolution operation with a convolution kernel of 1;
step 4) optimizing the feature vector FdObtaining a reconstructed image in an input image deep learning up-sampling module
Figure FDA0002345297730000011
Step 5) for the low-resolution image ILRObtaining a reconstructed image by adopting a bicubic linear interpolation algorithm
Figure FDA0002345297730000012
Will be provided with
Figure FDA0002345297730000013
And
Figure FDA0002345297730000014
fusing to obtain the final reconstructed image ISRThe method comprises the following steps:
Figure FDA0002345297730000015
2. the image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 1, wherein: the feature extraction subnet described in step 2 is composed of n multi-scale attention blocks, wherein the input of the ith multi-scale attention block is the feature output F of the previous multi-scale attention blocki-1The output characteristic is Fi；
Furthermore, each multi-scale attention block consists of a U-shaped structure module, a bottleneck layer structure module and a residual error module;
for the U-shaped structure module in the ith multi-scale attention block, the input of the U-shaped structure module is the output characteristic F of the previous multi-scale attention blocki-1After processing by the U-shaped structure module, the characteristic F is obtainedi,0；
For a bottleneck layer structure module in the ith multi-scale attention block, the bottleneck layer structure module is formed by connecting two bottleneck layers in series; output characteristic F of previous multi-scale attention blocki-1Inputting a first bottleneck layer, inputting the output of the first bottleneck layer into a second bottleneck layer, and simultaneously receiving the output characteristic F of the U-shaped structure module in the multi-scale attention block by the second bottleneck layeri,0Obtaining the output characteristic F of the bottleneck layer structure module through the processi,2；
For the residual module in the ith multi-scale attention block, specifically: output characteristics F of previous multi-scale attention blocki-1And output characteristics F of the bottleneck layer structurei,2Adding point by point to obtain characteristic Fi。
3. The image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 2, wherein:
the U-shaped structure module is formed by connecting non-local mean values, 3 multiplied by 3 convolution, 5 multiplied by 5 convolution, 7 multiplied by 7 convolution, attention mechanism, 5 multiplied by 5 convolution, 3 multiplied by 3 convolution and non-local mean values in series; wherein, a Concat layer is contained between two 3 × 3 convolutions; two 5 × 5 convolution convolutions contain one Concat layer in between; the input to the first 5 x 5 convolution is the sum of the output of the first non-local mean and the output of the first 3 x 3 convolution, the input to the 7 x 7 convolution is the sum of the output of the first 5 x 5 convolution and the output of the first non-local mean, the input to the second 5 x 5 convolution is the Concat of the output of the first 5 x 5 convolution and the attention mechanism output, and the input to the second 3 x 3 convolution is the Concat of the output of the first 3 x 3 convolution and the output of the second 5 x 5 convolution.
4. The image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 2, wherein: the non-local mean value is formed by connecting three convolutions with convolution kernel 1 in parallel and a convolution for feature fusion in series;
for the feature input F of non-local mean, firstly, the input features F are respectively input into three convolution layers in parallel to obtain three features Fx,Fy,FzThen, the three features are subjected to feature fusion through Concat operation, and the fused features are input into a subsequent convolution layer to obtain features FwAnd finally, the feature FwAnd adding the input characteristic F point by point to obtain the output characteristic of the non-local mean value.
5. The image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 2, wherein: the attention mechanism is formed by sequentially connecting a global pooling layer and two convolutions;
for the input feature F of the attention mechanism, first, the feature F is input to the global pooling layer extraction channel information descriptor MavgThen, the channel information descriptor M is describedavgInput into the subsequent two convolution layersAnd further processing to obtain M, and finally multiplying the M and the characteristic F channel by channel to obtain the output characteristic of the attention mechanism.
6. The image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 2, wherein: the bottleneck layer is formed by connecting two convolution layers in series.
7. The image super-resolution reconstruction method based on the multi-scale attention cascade network as claimed in claim 1, wherein: the cascade operation described in step 2 is specifically represented as follows:
Fc＝Concat(F1,F2,...,Fn)
where Concat (. cndot.) represents an operation of concatenating features of n multi-scale attention block outputs, FiI 1,2,3, n denotes the characteristics of the n multi-scale attention block outputs.
8. The method for reconstructing image super resolution based on multi-scale attention cascade network as claimed in claim 1, wherein the step 4) of obtaining the reconstructed image in the image deep learning up-sampling module
Figure FDA0002345297730000031
The method specifically comprises the following steps:
4.1) optimized feature F using a convolution layer pair with convolution kernel 3dRearranging to obtain a characteristic Fe：
Fe＝Conv3×3(Fd)
Wherein, Conv3×3(. h) represents a convolution operation with a convolution kernel of 3;
4.2) feature F after rearrangementeInputting into a sub-pixel convolution layer, enlarging it to corresponding scale, and obtaining reconstructed image
Figure FDA0002345297730000032
Figure FDA0002345297730000033
Wherein HSp(. cndot.) denotes a sub-pixel convolution operation.
CN201911392155.3A 2019-12-30 2019-12-30 Image super-resolution reconstruction method based on multi-scale attention cascade network Active CN111161150B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911392155.3A CN111161150B (en) 2019-12-30 2019-12-30 Image super-resolution reconstruction method based on multi-scale attention cascade network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911392155.3A CN111161150B (en) 2019-12-30 2019-12-30 Image super-resolution reconstruction method based on multi-scale attention cascade network

Publications (2)

Publication Number Publication Date
CN111161150A true CN111161150A (en) 2020-05-15
CN111161150B CN111161150B (en) 2023-06-23

Family

ID=70558951

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911392155.3A Active CN111161150B (en) 2019-12-30 2019-12-30 Image super-resolution reconstruction method based on multi-scale attention cascade network

Country Status (1)

Country Link
CN (1) CN111161150B (en)

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111368849A (en) * 2020-05-28 2020-07-03 腾讯科技（深圳）有限公司 Image processing method, image processing device, electronic equipment and storage medium
CN111724308A (en) * 2020-06-28 2020-09-29 深圳壹账通智能科技有限公司 Blurred image processing method and system
CN111915481A (en) * 2020-06-08 2020-11-10 北京大米未来科技有限公司 Image processing method, image processing apparatus, electronic device, and medium
CN111951164A (en) * 2020-08-11 2020-11-17 哈尔滨理工大学 Image super-resolution reconstruction network structure and image reconstruction effect analysis method
CN111970513A (en) * 2020-08-14 2020-11-20 成都数字天空科技有限公司 Image processing method and device, electronic equipment and storage medium
CN112070669A (en) * 2020-08-28 2020-12-11 西安科技大学 Super-resolution image reconstruction method for any fuzzy kernel
CN112116527A (en) * 2020-09-09 2020-12-22 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network framework and cascade network
CN112215755A (en) * 2020-10-28 2021-01-12 南京信息工程大学 Image super-resolution reconstruction method based on back projection attention network
CN112381839A (en) * 2020-11-14 2021-02-19 四川大学华西医院 Breast cancer pathological image HE cancer nest segmentation method based on deep learning
CN112580473A (en) * 2020-12-11 2021-03-30 北京工业大学 Motion feature fused video super-resolution reconstruction method
CN112767247A (en) * 2021-01-13 2021-05-07 京东方科技集团股份有限公司 Image super-resolution reconstruction method, model distillation method, device and storage medium
CN112801868A (en) * 2021-01-04 2021-05-14 青岛信芯微电子科技股份有限公司 Method for image super-resolution reconstruction, electronic device and storage medium
CN113012046A (en) * 2021-03-22 2021-06-22 华南理工大学 Image super-resolution reconstruction method based on dynamic packet convolution
CN113052016A (en) * 2021-03-09 2021-06-29 北京工业大学 Face super-resolution method based on multi-scale attention residual error and equal-variation mapping
CN113160047A (en) * 2020-11-23 2021-07-23 南京邮电大学 Single image super-resolution method based on multi-scale channel attention mechanism
CN113222822A (en) * 2021-06-02 2021-08-06 西安电子科技大学 Hyperspectral image super-resolution reconstruction method based on multi-scale transformation
CN113538250A (en) * 2021-09-14 2021-10-22 苏州微清医疗器械有限公司 Fundus shooting system with function of rapidly processing images
CN113674156A (en) * 2021-09-06 2021-11-19 苏州大学 Method and system for reconstructing image super-resolution
CN113688783A (en) * 2021-09-10 2021-11-23 柚皮(重庆)科技有限公司 Face feature extraction method, low-resolution face recognition method and device
CN113763251A (en) * 2021-09-14 2021-12-07 浙江师范大学 Image super-resolution amplification model and method thereof
CN113837946A (en) * 2021-10-13 2021-12-24 中国电子技术标准化研究院 Lightweight image super-resolution reconstruction method based on progressive distillation network
CN114025200A (en) * 2021-09-15 2022-02-08 湖南广播影视集团有限公司 Ultra-high definition post-production solution based on cloud technology
CN115546032A (en) * 2022-12-01 2022-12-30 泉州市蓝领物联科技有限公司 Single-frame image super-resolution method based on feature fusion and attention mechanism
CN115564653A (en) * 2022-09-30 2023-01-03 江苏济远医疗科技有限公司 Multi-factor fusion image super-resolution method
CN116797456A (en) * 2023-05-12 2023-09-22 苏州大学 Image super-resolution reconstruction method, system, device and storage medium
WO2024007160A1 (en) * 2022-07-05 2024-01-11 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Convolutional neural network (cnn) filter for super-resolution with reference picture resampling (rpr) functionality

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106709875A (en) * 2016-12-30 2017-05-24 北京工业大学 Compressed low-resolution image restoration method based on combined deep network
CN107240066A (en) * 2017-04-28 2017-10-10 天津大学 Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks
CN109886871A (en) * 2019-01-07 2019-06-14 国家新闻出版广电总局广播科学研究院 The image super-resolution method merged based on channel attention mechanism and multilayer feature
CN109903228A (en) * 2019-02-28 2019-06-18 合肥工业大学 A kind of image super-resolution rebuilding method based on convolutional neural networks
CN110136060A (en) * 2019-04-24 2019-08-16 西安电子科技大学 The image super-resolution rebuilding method of network is intensively connected based on shallow-layer
CN110276721A (en) * 2019-04-28 2019-09-24 天津大学 Image super-resolution rebuilding method based on cascade residual error convolutional neural networks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106709875A (en) * 2016-12-30 2017-05-24 北京工业大学 Compressed low-resolution image restoration method based on combined deep network
CN107240066A (en) * 2017-04-28 2017-10-10 天津大学 Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks
CN109886871A (en) * 2019-01-07 2019-06-14 国家新闻出版广电总局广播科学研究院 The image super-resolution method merged based on channel attention mechanism and multilayer feature
CN109903228A (en) * 2019-02-28 2019-06-18 合肥工业大学 A kind of image super-resolution rebuilding method based on convolutional neural networks
CN110136060A (en) * 2019-04-24 2019-08-16 西安电子科技大学 The image super-resolution rebuilding method of network is intensively connected based on shallow-layer
CN110276721A (en) * 2019-04-28 2019-09-24 天津大学 Image super-resolution rebuilding method based on cascade residual error convolutional neural networks

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
YULUN ZHANG ET AL.: "Image Super-Resolution Using Very Deep Residual Channel Attention Networks", ECCV 2018 *
付利华 等: "基于运动特征融合的快速视频超分辨率重构方法", 模式识别与人工智能 *

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111368849A (en) * 2020-05-28 2020-07-03 腾讯科技（深圳）有限公司 Image processing method, image processing device, electronic equipment and storage medium
CN111368849B (en) * 2020-05-28 2020-08-28 腾讯科技（深圳）有限公司 Image processing method, image processing device, electronic equipment and storage medium
CN111915481A (en) * 2020-06-08 2020-11-10 北京大米未来科技有限公司 Image processing method, image processing apparatus, electronic device, and medium
CN111915481B (en) * 2020-06-08 2024-03-29 北京大米未来科技有限公司 Image processing method, device, electronic equipment and medium
CN111724308A (en) * 2020-06-28 2020-09-29 深圳壹账通智能科技有限公司 Blurred image processing method and system
CN111951164A (en) * 2020-08-11 2020-11-17 哈尔滨理工大学 Image super-resolution reconstruction network structure and image reconstruction effect analysis method
CN111970513A (en) * 2020-08-14 2020-11-20 成都数字天空科技有限公司 Image processing method and device, electronic equipment and storage medium
CN112070669A (en) * 2020-08-28 2020-12-11 西安科技大学 Super-resolution image reconstruction method for any fuzzy kernel
CN112070669B (en) * 2020-08-28 2024-04-16 西安科技大学 Super-resolution image reconstruction method for arbitrary fuzzy core
CN112116527B (en) * 2020-09-09 2024-02-23 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network frame and cascade network
CN112116527A (en) * 2020-09-09 2020-12-22 北京航空航天大学杭州创新研究院 Image super-resolution method based on cascade network framework and cascade network
CN112215755A (en) * 2020-10-28 2021-01-12 南京信息工程大学 Image super-resolution reconstruction method based on back projection attention network
CN112381839A (en) * 2020-11-14 2021-02-19 四川大学华西医院 Breast cancer pathological image HE cancer nest segmentation method based on deep learning
CN113160047B (en) * 2020-11-23 2023-05-23 南京邮电大学 Single image super-resolution method based on multi-scale channel attention mechanism
CN113160047A (en) * 2020-11-23 2021-07-23 南京邮电大学 Single image super-resolution method based on multi-scale channel attention mechanism
CN112580473B (en) * 2020-12-11 2024-05-28 北京工业大学 Video super-resolution reconstruction method integrating motion characteristics
CN112580473A (en) * 2020-12-11 2021-03-30 北京工业大学 Motion feature fused video super-resolution reconstruction method
CN112801868B (en) * 2021-01-04 2022-11-11 青岛信芯微电子科技股份有限公司 Method for image super-resolution reconstruction, electronic device and storage medium
CN112801868A (en) * 2021-01-04 2021-05-14 青岛信芯微电子科技股份有限公司 Method for image super-resolution reconstruction, electronic device and storage medium
CN112767247A (en) * 2021-01-13 2021-05-07 京东方科技集团股份有限公司 Image super-resolution reconstruction method, model distillation method, device and storage medium
CN113052016A (en) * 2021-03-09 2021-06-29 北京工业大学 Face super-resolution method based on multi-scale attention residual error and equal-variation mapping
CN113012046A (en) * 2021-03-22 2021-06-22 华南理工大学 Image super-resolution reconstruction method based on dynamic packet convolution
CN113012046B (en) * 2021-03-22 2022-12-16 华南理工大学 Image super-resolution reconstruction method based on dynamic packet convolution
CN113222822A (en) * 2021-06-02 2021-08-06 西安电子科技大学 Hyperspectral image super-resolution reconstruction method based on multi-scale transformation
CN113222822B (en) * 2021-06-02 2023-01-24 西安电子科技大学 Hyperspectral image super-resolution reconstruction method based on multi-scale transformation
CN113674156A (en) * 2021-09-06 2021-11-19 苏州大学 Method and system for reconstructing image super-resolution
CN113688783B (en) * 2021-09-10 2022-06-28 一脉通(深圳)智能科技有限公司 Face feature extraction method, low-resolution face recognition method and equipment
CN113688783A (en) * 2021-09-10 2021-11-23 柚皮(重庆)科技有限公司 Face feature extraction method, low-resolution face recognition method and device
CN113763251A (en) * 2021-09-14 2021-12-07 浙江师范大学 Image super-resolution amplification model and method thereof
CN113538250A (en) * 2021-09-14 2021-10-22 苏州微清医疗器械有限公司 Fundus shooting system with function of rapidly processing images
CN114025200B (en) * 2021-09-15 2022-09-16 湖南广播影视集团有限公司 Ultra-high definition post-production solution based on cloud technology
CN114025200A (en) * 2021-09-15 2022-02-08 湖南广播影视集团有限公司 Ultra-high definition post-production solution based on cloud technology
CN113837946A (en) * 2021-10-13 2021-12-24 中国电子技术标准化研究院 Lightweight image super-resolution reconstruction method based on progressive distillation network
WO2024007160A1 (en) * 2022-07-05 2024-01-11 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Convolutional neural network (cnn) filter for super-resolution with reference picture resampling (rpr) functionality
CN115564653A (en) * 2022-09-30 2023-01-03 江苏济远医疗科技有限公司 Multi-factor fusion image super-resolution method
CN115546032A (en) * 2022-12-01 2022-12-30 泉州市蓝领物联科技有限公司 Single-frame image super-resolution method based on feature fusion and attention mechanism
CN115546032B (en) * 2022-12-01 2023-04-21 泉州市蓝领物联科技有限公司 Single-frame image super-resolution method based on feature fusion and attention mechanism
CN116797456A (en) * 2023-05-12 2023-09-22 苏州大学 Image super-resolution reconstruction method, system, device and storage medium

Also Published As

Publication number Publication date
CN111161150B (en) 2023-06-23

Similar Documents

Publication Publication Date Title
CN111161150B (en) Image super-resolution reconstruction method based on multi-scale attention cascade network
CN109903228B (en) Image super-resolution reconstruction method based on convolutional neural network
CN108537733B (en) Super-resolution reconstruction method based on multi-path deep convolutional neural network
CN106910161B (en) Single image super-resolution reconstruction method based on deep convolutional neural network
CN110136062B (en) Super-resolution reconstruction method combining semantic segmentation
CN110163801B (en) Image super-resolution and coloring method, system and electronic equipment
CN113888744A (en) Image semantic segmentation method based on Transformer visual upsampling module
CN111861961A (en) Multi-scale residual error fusion model for single image super-resolution and restoration method thereof
CN109035146B (en) Low-quality image super-resolution method based on deep learning
CN110349087B (en) RGB-D image high-quality grid generation method based on adaptive convolution
CN116664400A (en) Video high space-time resolution signal processing method
CN111932461A (en) Convolutional neural network-based self-learning image super-resolution reconstruction method and system
CN115358932B (en) Multi-scale feature fusion face super-resolution reconstruction method and system
CN111833261A (en) Image super-resolution restoration method for generating countermeasure network based on attention
CN113222818A (en) Method for reconstructing super-resolution image by using lightweight multi-channel aggregation network
CN113837946A (en) Lightweight image super-resolution reconstruction method based on progressive distillation network
WO2023185284A1 (en) Video processing method and apparatuses
CN115797176A (en) Image super-resolution reconstruction method
CN116468605A (en) Video super-resolution reconstruction method based on time-space layered mask attention fusion
CN114841859A (en) Single-image super-resolution reconstruction method based on lightweight neural network and Transformer
Wu et al. Lightweight asymmetric convolutional distillation network for single image super-resolution
CN112102388B (en) Method and device for obtaining depth image based on inspection robot monocular image
CN116434039B (en) Target detection method based on multiscale split attention mechanism
CN116797456A (en) Image super-resolution reconstruction method, system, device and storage medium
CN116485654A (en) Lightweight single-image super-resolution reconstruction method combining convolutional neural network and transducer

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant