CN115293968A - Super-light-weight high-efficiency single-image super-resolution method - Google Patents
Super-light-weight high-efficiency single-image super-resolution method Download PDFInfo
- Publication number
- CN115293968A CN115293968A CN202210861170.3A CN202210861170A CN115293968A CN 115293968 A CN115293968 A CN 115293968A CN 202210861170 A CN202210861170 A CN 202210861170A CN 115293968 A CN115293968 A CN 115293968A
- Authority
- CN
- China
- Prior art keywords
- feature
- information
- fusion
- branch
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000004927 fusion Effects 0.000 claims abstract description 80
- 238000013507 mapping Methods 0.000 claims abstract description 15
- 238000004821 distillation Methods 0.000 claims abstract description 12
- 230000007246 mechanism Effects 0.000 claims abstract description 11
- 230000006835 compression Effects 0.000 claims abstract description 9
- 238000007906 compression Methods 0.000 claims abstract description 9
- 238000000605 extraction Methods 0.000 claims description 27
- 230000002087 whitening effect Effects 0.000 claims description 21
- 230000008569 process Effects 0.000 claims description 19
- 230000009466 transformation Effects 0.000 claims description 12
- 230000002787 reinforcement Effects 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 10
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 2
- 230000009467 reduction Effects 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 abstract description 11
- 230000002708 enhancing effect Effects 0.000 abstract description 4
- 230000005540 biological transmission Effects 0.000 abstract description 3
- 238000001514 detection method Methods 0.000 abstract description 3
- 238000002059 diagnostic imaging Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 14
- 238000004590 computer program Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention relates to an ultra-light-weight high-efficiency single-image super-resolution method, which takes a low-resolution image as input, so that an algorithm can be deployed on ultra-light-weight equipment or high-speed equipment. The method mainly comprises the following steps: a more lightweight super-resolution backbone network is provided, so that the parameter quantity of the network is greatly reduced while the performance of the network is maintained; the double-branch multi-scale reception field fusion module is provided, so that the characteristic diversity among distillation information and the overall network reception field are increased, the information overlapping is removed, and the network performance is improved; the light attention mechanism based on the learnable Hash mapping is provided, and is used for decoupling input information and enhancing regional boundary reconstruction and regional relation learning; a second-order coordinate attention mechanism based on global information supervision is provided, and accuracy of network feature expression is improved. The strategy provided by the invention can be used in a plurality of fields such as public security, medical imaging, remote sensing detection, video compression transmission and the like.
Description
Technical Field
The invention relates to the field of computer vision, in particular to an ultra-light high-efficiency single-image super-resolution method.
Background
The single image super resolution task aims at reconstructing detail information from a low resolution image to obtain a high resolution image. However, the same low resolution image can be generated by different high resolution image degradations, which leads to task inadaptation and high difficulty.
In order to solve the problem, a plurality of single-image super-resolution algorithms based on deep learning are proposed and achieve the effect far beyond that of the traditional algorithm, and become the current mainstream method. At present, in many scenes, for example, in various mobile devices and wearable devices with weak computing power, the accuracy of the image super-resolution algorithm based on the traditional method is too low, while the complexity of the image super-resolution algorithm based on the deep learning is too high, and the computation amount is too large, so that the model cannot be deployed normally.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides an ultra-light-weight and high-efficiency single-image super-resolution method, so as to solve the technical problem that the existing deep learning method is difficult to deploy in various mobile devices and wearable devices with weak computing power.
According to a first aspect of the present invention, there is provided an ultra-lightweight and efficient single-image super-resolution method, comprising:
Step 2, the shallow layer characteristic F is treated 0 Extracting deep layer characteristics to obtain deep layer characteristics F 6 ；
Step 4, utilizing a 3 x 3 convolution to convert the residual error characteristic F r Conversion into feature F u The feature F is then processed by the Sub-pixel module u Upsampling to high resolution pictures I h 。
On the basis of the technical scheme, the invention can be improved as follows.
Optionally, the process of performing deep feature extraction in step 2 includes: the shallow feature F 0 Inputting a double-branch multi-scale receptive field fusion module for extraction and fusion;
the double-branch multi-scale perception fusion module comprises an information distillation branch and a feature extraction branch;
the feature extraction branch respectively extracts features by convolution of 1 × 1, 3 × 3, 5 × 5 and 7 × 7; and after the features are extracted by adopting 1 × 1, 3 × 3, 5 × 5 and 7 × 7 depth-by-depth convolution in the information distillation branch, fusion features are obtained.
Optionally, the process of performing deep feature extraction in step 2 includes: inputting the fusion features into a second-order coordinate attention module based on global information supervision;
the second order coordinate attention module includes: a global mean information coding branch and a global variance information coding branch;
inputting the fusion characteristics into the global mean information coding branch to carry out learnable weight coding to obtain global mean information after weight adjustment;
inputting the fusion characteristics into the global variance information coding branch to carry out learnable weight coding to obtain global variance information after weight adjustment;
and adding the global mean information and the global variance information after the weighting adjustment, and performing nonlinear mapping on the global mean information and the global variance information by using a Sigmoid function to obtain comprehensive coding global information.
Optionally, the process of performing deep feature extraction in step 2 includes: the shallow layer features are repeatedly input into the double-branch multi-scale receptive field fusion module and the second-order coordinate attention module for three times to obtain deep layer features { F } 1 ，F 2 ，F 3 }；
The deep layer characteristic F 1 And F 3 Inputting adjacent feature fusion module to perform feature fusion to obtainA deep fusion feature, the adjacent feature fusion module comprising a 1 x 1 reduced dimension convolution and activation function.
Optionally, the process of performing deep feature extraction in step 2 further includes: inputting the deep fusion features into a reinforcement learning module for reinforcement learning, the reinforcement learning module reinforcement learning the deep fusion features using a decoupling-enhanced, learnable hash-mapping based lightweight Non-local attention mechanism.
Optionally, the reinforcement learning module includes a feature transformation branch and a global semantic information coding branch;
the process of the feature transformation branch processing the deep fusion features comprises the following steps: adding a whitening module to the right side of the 3 x 3 convolution with the channel compression ratio of r, and inputting the deep fusion features into the whitening module to obtain whitening features F B ；
For the whitening characteristic F B Obtaining the similarity information F between the whitening points in barrel-shaped distribution through similar map calculation bs ；
The similarity information F between the whitening points bs And the whitening feature F B The self-similar characteristic correction operation in the barrel is executed together to obtain the corrected barrel-shaped distribution characteristic F BR ；
According to the inverse mapping H -1 The barrel-shaped distribution feature F is formed BR Reduced to a self-similar corrected feature F of size C/2 XH W P ；
The process of the global semantic information coding branch for processing the deep fusion features comprises the following steps: convolving the deep fusion characteristic input channel by 1 multiplied by 1 with the compression ratio of C to obtain pixel-by-pixel significance information F with the size of 1 multiplied by H multiplied by W i ；
The pixel-by-pixel saliency information F i The input Softmax module carries out nonlinear mapping to obtain pixel-by-pixel relative significance information F 'distributed in the range of 0-1' i ；
Comparing the pixel-by-pixel relative saliency information F' i Performing pixel-by-pixel feature weighted summation operation on the transformation features obtained by the feature transformation branch as weights to obtain a global feature with the size of C/2 multiplied by 1Semantic information F G ；
Using two 1 x 1 convolutions on the global semantic information F G Channel coding is carried out to obtain channel-sensitive global semantic information F' G 。
Optionally, the process of performing deep feature extraction in step 2 further includes: correcting the self-similarity of pixel point form by using the characteristic F P And channel-sensitive global semantic information F' G Simply adding to obtain the fusion feature F with the size of C/2 XHXW f The fusion feature F is convolved by 1 x 1 liter dimension f Reduced to C H W dimension size feature F o 。
Optionally, the process of performing deep feature extraction in step 2 further includes:
the feature F is measured o Obtaining a characteristic F after passing through the double-branch multi-scale receptive field fusion module and the second-order coordinate attention module 4 The feature F is measured 4 Obtaining a depth feature F through the dual-branch multi-scale receptive field fusion module and the second-order coordinate attention module 5 ；
Using the neighboring feature fusion module to pair depth features F o And the depth feature F 5 After feature fusion, the depth feature F is obtained through the dual-branch multi-scale receptive field fusion module 6 。
The embodiment of the invention provides an ultra-light-weight high-efficiency single-image super-resolution method, which has obvious advantages for being deployed on mobile equipment with limited computing power, and the advantages mainly come from the following aspects: firstly, a super-resolution backbone network with lighter weight is obtained by removing and replacing a large number of redundant structures of the previous super-resolution algorithm, so that the parameter quantity of the network is greatly reduced while the performance of the network is maintained. On the basis, the double-branch multi-scale reception field fusion module is further utilized to increase the characteristic diversity among distillation information and the overall network reception field, and information overlapping is removed, so that the network performance is improved. And then decoupling the input information by using a decoupling enhanced lightweight attention mechanism based on learnable HashMap, and enhancing the region boundary reconstruction and the region internal relation learning by using the two types of decoupling information. And finally, the effective global information is extracted to supervise and correct the network, so that the accuracy of network feature expression is improved, and a high-quality high-resolution image is obtained. And obtaining a high-resolution image by reconstructing the detail information. The method ensures the precision and minimizes the model, and solves the problem that the previous super-resolution algorithm cannot be deployed on ultra-light equipment or high-speed equipment. The strategy provided by the invention can be used in a plurality of fields such as public security, medical imaging, remote sensing detection, video compression transmission and the like.
Drawings
FIG. 1 is a flow chart of a super-resolution method for single image with ultra-light weight and high efficiency provided by the present invention;
fig. 2 is a schematic overall architecture diagram of an embodiment of an ultra-lightweight and efficient single-image super-resolution method provided by the present invention:
fig. 3 is a schematic diagram of a dual-branch multi-scale receptive field fusion module architecture according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a lightweight attention mechanism architecture based on learnable hash mapping with decoupling enhancement according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth to illustrate, but are not to be construed to limit the scope of the invention.
In order to overcome the defects and problems in the background art, the invention provides a light super-resolution backbone network, which removes and replaces a large number of redundant structures of the previous super-resolution algorithm, so that the parameter quantity of the network is greatly reduced while the performance of the network is maintained. Fig. 1 is a flowchart of an ultra-lightweight high-efficiency single-image super-resolution method provided by the present invention, and as shown in fig. 1, the ultra-lightweight high-efficiency single-image super-resolution method includes:
Step 2, shallow layer characteristic F 0 Extracting deep layer characteristics to obtain deep layer characteristics F 6 。
Step 4, utilizing a 3 x 3 convolution to convert the residual error characteristic F r Conversion to features F more suitable for upsampling u And then the Sub-pixel module is used to convert the low-resolution feature F u Upsampling to high resolution pictures I h 。
The super-resolution method for the single image, provided by the invention, has the advantages that the accuracy is ensured, the parameter quantity of the model is minimized, and the deployment and the use on various mobile devices are realized.
Example 1
In order to overcome the defects and problems in the background art, the embodiment 1 provided by the present invention provides a super-resolution backbone network with lighter weight, which removes and replaces a large number of redundant structures of the previous super-resolution algorithm, so that the parameter amount of the network is greatly reduced while the performance of the network is maintained. Meanwhile, on the basis, a double-branch multi-scale reception field fusion module is provided, and the module further increases the characteristic diversity among distillation information and the overall network reception field, removes information overlapping and improves the network performance. The light attention mechanism based on the learnable Hash mapping is provided, and decoupling is carried out on input information, and area boundary reconstruction and area relation learning are respectively enhanced by utilizing two kinds of decoupling information. A second-order coordinate attention mechanism based on global information supervision is provided, and the network is supervised and corrected by extracting effective global information, so that the accuracy of network feature expression is improved.
Fig. 2 is a schematic overall architecture diagram of an embodiment of an ultra-lightweight and efficient single-image super-resolution method provided by the present invention, and as can be seen from fig. 1 and fig. 2, the embodiment of the ultra-lightweight and efficient single-image super-resolution method includes:
Step 2, shallow layer characteristic F 0 Extracting deep layer characteristics to obtain deep layer characteristics F 6 。
In a possible embodiment, the deep feature extraction in step 2 includes:
step 201, shallow layer characteristic F 0 Inputting the two-branch multi-scale receptive field fusion module for extraction and fusion.
Fig. 3 is a schematic diagram of a dual-branch multi-scale receptive field fusion module architecture according to an embodiment of the present invention, and it can be seen from fig. 3 that the dual-branch multi-scale receptive field fusion module includes an information distillation branch and a feature extraction branch.
Extracting features in the feature extraction branch by convolution with 1 × 1, 3 × 3, 5 × 5 and 7 × 7 respectively; and extracting features by adopting 1 × 1, 3 × 3, 5 × 5 and 7 × 7 depth-by-depth convolution in the information distillation branch to obtain fusion features.
The characteristic of each time of entering the information distillation branch is ensured to be the characteristic before residual connection, so that the two branches not only effectively enlarge the network receptive field, but also reduce the information redundancy among the distillation characteristics.
Step 202, the process of deep layer feature extraction in step 2 includes: and inputting the fusion features into a second-order coordinate attention module based on global information supervision, and enhancing the overall control of the network on the image.
The second order coordinate attention module includes: a global mean information coding branch and a global variance information coding branch.
And inputting the fusion characteristics into a global mean information coding branch to carry out learnable weight coding to obtain the global mean information after weight adjustment.
And inputting the fusion characteristics into a global variance information coding branch to carry out learnable weight coding to obtain the global variance information after weight adjustment.
The method enables the features to adaptively adjust the weight distribution of the one-dimensional feature vector according to the task in the network training process, thereby obtaining more robust and reliable global information.
And at the end of the module, adding the global mean information and the global variance information after the weighting adjustment, and carrying out nonlinear mapping on the global mean information and the global variance information by using a Sigmoid function to obtain comprehensive coding global information.
Step 203, inputting the shallow layer feature into the double-branch multi-scale receptive field fusion module and the second-order coordinate attention module for three times to obtain the deep layer feature { F } 1 ，F 2 ，F 3 }。
Deep layer characteristics F 1 And F 3 Inputting an adjacent feature fusion module to perform feature fusion to obtain a deep fusion feature, wherein the adjacent feature fusion module comprises a 1 x 1 dimensionality reduction convolution and an activation function to enhance the liquidity of shallower information in the network.
And 204, inputting the deep fusion features into a reinforced learning module for reinforced learning, wherein the reinforced learning module uses a decoupling reinforced lightweight Non-local attention mechanism based on learnable Hash mapping to carry out reinforced learning on the deep fusion features.
As shown in fig. 4, a schematic diagram of a decoupling enhanced lightweight attention mechanism architecture based on learnable hash mapping according to an embodiment of the present invention, and as can be seen in fig. 4, in a possible embodiment, the reinforcement learning module includes a feature transformation branch and a global semantic information coding branch.
The process of processing the deep fusion features by the feature transformation branch comprises the following steps: adding the 3 x 3 convolution right side with the channel compression ratio of r into a whitening module, and inputting the deep fusion features into the whitening module to obtain whitening features F B 。
Whitening characteristic F B Obtaining the similarity information F between the whitening points in barrel-shaped distribution through similar map calculation bs 。
The similarity information F between the whitening points bs And whitening feature F B The self-similar characteristic correction operation in the barrel is executed together to obtain the corrected barrel-shaped distribution characteristic F BR 。
According to the inverse mapping H -1 Barrel-shaped distribution of features F BR Reduced to a self-similar corrected feature F of size C/2 XH W P 。
The process of processing the deep fusion features by the global semantic information coding branch comprises the following steps: the deep fusion characteristics are input into a 1 multiplied by 1 convolution with a channel compression ratio of C to obtain pixel-by-pixel significance information F with the size of 1 multiplied by H multiplied by W i 。
Pixel-by-pixel saliency information F i The input Softmax module carries out nonlinear mapping to obtain pixel-by-pixel relative significance information F 'distributed in the range of 0-1' i 。
Pixel-by-pixel relative saliency information F' i Performing pixel-by-pixel feature weighted summation operation on the transformation features obtained by the feature transformation branch as weights to obtain global semantic information F with the size of C/2 multiplied by 1 G 。
Using two 1 x 1 convolutions on global semantic information F G Channel coding is carried out to obtain channel-sensitive global semantic information F' G . The global semantic information F' G The region boundaries can be well modeled.
Step 205, correcting self-similarity of pixel point form by using characteristic F P And channel sensitive global semantic information F' G Simply adding to obtain the fusion feature F with the size of C/2 XHXW f Fusing features F by 1 × 1L dimensional convolution f Reduced to C H W dimension size feature F o 。
Step 206, the feature F o Obtaining a characteristic F after passing through a double-branch multi-scale receptive field fusion module and a second-order coordinate attention module 4 Will feature F 4 Obtaining a depth characteristic F after passing through a double-branch multi-scale receptive field fusion module and a second-order coordinate attention module 5 。
Step 207, using the neighboring feature fusion module to pair depth features F o And depth feature F 5 After feature fusion, a depth feature F is obtained through a double-branch multi-scale receptive field fusion module 6 。
Step 4, utilizing a 3 x 3 convolution to convert the residual error characteristic F r Conversion to features F more suitable for upsampling u The Sub-pixel module then converts the low-resolution feature F u Upsampled to high resolution image I h 。
The embodiment of the invention provides an ultra-light-weight high-efficiency single-image super-resolution method, which has obvious advantages for being deployed on mobile equipment with limited computing power, and the advantages mainly come from the following aspects: firstly, a super-resolution backbone network with lighter weight is obtained by removing and replacing a large number of redundant structures of the previous super-resolution algorithm, so that the parameter quantity of the network is greatly reduced while the performance of the network is maintained. On the basis, the double-branch multi-scale reception field fusion module is further utilized to increase the characteristic diversity among distillation information and the overall network reception field, and information overlapping is removed, so that the network performance is improved. And then decoupling the input information by using a decoupling enhanced light attention mechanism based on learnable HashMap, and respectively enhancing the region boundary reconstruction and the region interior relation learning by using the two kinds of decoupling information. And finally, the effective global information is extracted to supervise and correct the network, so that the accuracy of network feature expression is improved, and a high-quality high-resolution image is obtained. And obtaining a high-resolution image by reconstructing the detail information. The method ensures the precision and minimizes the model, and solves the problem that the previous super-resolution algorithm cannot be deployed on ultra-light equipment or high-speed equipment. The strategy provided by the invention can be used in a plurality of fields such as public security, medical imaging, remote sensing detection, video compression transmission and the like.
It should be noted that, in the foregoing embodiments, the description of each embodiment has an emphasis, and reference may be made to the related description of other embodiments for a part that is not described in detail in a certain embodiment.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (8)
1. An ultra-lightweight and efficient single-image super-resolution method comprises the following steps:
step 1, inputting a low-resolution image I to be processed l For said low-resolution image I, using a 3 x 3 convolution l Shallow feature F containing coarse grain information is obtained by shallow feature extraction 0 ；
Step 2, for the shallow layer characteristic F 0 Extracting deep layer characteristics to obtain deep layer characteristics F 6 ；
Step 3, building the shallow layer characteristic F 0 And the deep layer characteristic F 6 The residual error connection between the two groups obtains the residual error characteristic of F r ；
Step 4, utilizing a 3 x 3 convolution to convert the residual error characteristic F r Conversion into feature F u The feature F is then processed by the Sub-pixel module u Upsampling to high resolution pictures I h 。
2. The single-image super-resolution method of claim 1, wherein the deep feature extraction in step 2 comprises: the shallow feature F 0 Inputting a double-branch multi-scale receptive field fusion module for extraction and fusion;
the double-branch multi-scale perception fusion module comprises an information distillation branch and a feature extraction branch;
extracting features in the feature extraction branch by convolution with 1 × 1, 3 × 3, 5 × 5 and 7 × 7 respectively; and after the features are extracted by adopting 1 × 1, 3 × 3, 5 × 5 and 7 × 7 depth-by-depth convolution in the information distillation branch, fusion features are obtained.
3. The single-image super-resolution method of claim 2, wherein the deep feature extraction in step 2 comprises: inputting the fusion features into a second-order coordinate attention module based on global information supervision;
the second order coordinate attention module includes: a global mean information coding branch and a global variance information coding branch;
inputting the fusion characteristics into the global mean information coding branch to carry out learnable weight coding to obtain global mean information after weight adjustment;
inputting the fusion characteristics into the global variance information coding branch to carry out learnable weight coding to obtain global variance information after weight adjustment;
and adding the global mean information and the global variance information after the weighting adjustment, and performing nonlinear mapping on the global mean information and the global variance information by using a Sigmoid function to obtain comprehensive coding global information.
4. The single-image super-resolution method according to claim 3, wherein the process of deep feature extraction in step 2 comprises: the shallow layer features are repeatedly input into the double-branch multi-scale receptive field fusion module and the second-order coordinate attention module for three times to obtain deep layer features { F } 1 ,F 2 ,F 3 }；
The deep layer characteristic F 1 And F 3 Inputting an adjacent feature fusion module to perform feature fusion to obtain a deep fusion feature, wherein the adjacent feature fusion module comprises a 1 x 1 dimensionality reduction convolution and an activation function.
5. The single-image super-resolution method of claim 4, wherein the process of deep feature extraction in step 2 further comprises: inputting the deep fusion features into a reinforcement learning module for reinforcement learning, the reinforcement learning module using a decoupled enhanced lightweight Non-local attention mechanism based on learnable HashMaps for reinforcement learning of the deep fusion features.
6. The single image super resolution method according to claim 5, wherein the reinforcement learning module comprises a feature transformation branch and a global semantic information coding branch;
the process of the feature transformation branch for processing the deep fusion features comprises the following steps: adding a whitening module to the right side of the 3 x 3 convolution with the channel compression ratio of r, and inputting the deep fusion features into the whitening module to obtain whitening features F B ；
For the whitening characteristic F B Obtaining the similarity information F between the whitening points in barrel-shaped distribution through similar map calculation bs ；
The similarity information F between the whitening points bs And the whitening feature F B The self-similar characteristic correction operation in the barrel is executed together to obtain the corrected barrel-shaped distribution characteristic F BR ；
According to the inverse mapping H -1 The barrel-shaped distribution feature F is formed BR Reduced to a self-similar corrected feature F of size C/2 XH W P ；
The process of the global semantic information coding branch processing the deep fusion features comprises the following steps: convolving the deep fusion characteristic input channel by 1 multiplied by 1 with the compression ratio of C to obtain pixel-by-pixel significance information F with the size of 1 multiplied by H multiplied by W i ；
The pixel-by-pixel saliency information F i The input Softmax module carries out nonlinear mapping to obtain pixel-by-pixel relative significance information F 'distributed in the range of 0-1' i ；
Comparing the pixel-by-pixel relative saliency information F' i Performing pixel-by-pixel feature weighted summation operation on the transformation features obtained by the feature transformation branch as weights to obtain global semantic information F with the size of C/2 multiplied by 1 G ；
Using two 1 x 1 convolutions on the global semantic information F G Channel coding is carried out to obtain global semantic information F 'sensitive to channels' G 。
7. According to claim6, the process of deep feature extraction in the step 2 further includes: correcting the self-similarity of pixel point form by using the characteristic F P And channel-sensitive global semantic information F' G Simply adding to obtain the fusion feature F with the size of C/2 XHXW f The fusion feature F is convolved by 1 x 1 liter dimension f Feature F reduced to CxHxW dimension size o 。
8. The single-image super-resolution method of claim 7, wherein the process of deep feature extraction in step 2 further comprises:
the feature F o Obtaining a characteristic F after passing through the double-branch multi-scale receptive field fusion module and the second-order coordinate attention module 4 The feature F is 4 Obtaining a depth feature F through the dual-branch multi-scale receptive field fusion module and the second-order coordinate attention module 5 ；
Using the adjacent feature fusion module to pair depth features F o And the depth feature F 5 After feature fusion, the depth feature F is obtained through the dual-branch multi-scale receptive field fusion module 6 。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210861170.3A CN115293968A (en) | 2022-07-19 | 2022-07-19 | Super-light-weight high-efficiency single-image super-resolution method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210861170.3A CN115293968A (en) | 2022-07-19 | 2022-07-19 | Super-light-weight high-efficiency single-image super-resolution method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115293968A true CN115293968A (en) | 2022-11-04 |
Family
ID=83823913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210861170.3A Pending CN115293968A (en) | 2022-07-19 | 2022-07-19 | Super-light-weight high-efficiency single-image super-resolution method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115293968A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115994857A (en) * | 2023-01-09 | 2023-04-21 | 深圳大学 | Video super-resolution method, device, equipment and storage medium |
| CN116934598A (en) * | 2023-09-19 | 2023-10-24 | 湖南大学 | Multi-scale feature fusion light-weight remote sensing image superdivision method and system |
-
2022
- 2022-07-19 CN CN202210861170.3A patent/CN115293968A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115994857A (en) * | 2023-01-09 | 2023-04-21 | 深圳大学 | Video super-resolution method, device, equipment and storage medium |
| CN115994857B (en) * | 2023-01-09 | 2023-10-13 | 深圳大学 | Video super-resolution method, device, equipment and storage medium |
| CN116934598A (en) * | 2023-09-19 | 2023-10-24 | 湖南大学 | Multi-scale feature fusion light-weight remote sensing image superdivision method and system |
| CN116934598B (en) * | 2023-09-19 | 2023-12-01 | 湖南大学 | Multi-scale feature fusion light-weight remote sensing image superdivision method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113658051B (en) | Image defogging method and system based on cyclic generation countermeasure network | |
| CN109087273B (en) | Image restoration method, storage medium and system based on enhanced neural network | |
| CN112801901A (en) | Image deblurring algorithm based on block multi-scale convolution neural network | |
| CN115293968A (en) | Super-light-weight high-efficiency single-image super-resolution method | |
| CN110363068B (en) | High-resolution pedestrian image generation method based on multiscale circulation generation type countermeasure network | |
| CN112184585B (en) | Image completion method and system based on semantic edge fusion | |
| CN114511798B (en) | Driver distraction detection method and device based on transformer | |
| CN112419191B (en) | Image motion blur removing method based on convolution neural network | |
| CN113076957A (en) | RGB-D image saliency target detection method based on cross-modal feature fusion | |
| CN111696110A (en) | Scene segmentation method and system | |
| Rivadeneira et al. | Thermal image super-resolution challenge-pbvs 2021 | |
| Zheng et al. | T-net: Deep stacked scale-iteration network for image dehazing | |
| Zhong et al. | Deep attentional guided image filtering | |
| CN116468605A (en) | Video super-resolution reconstruction method based on time-space layered mask attention fusion | |
| CN115511708A (en) | Depth map super-resolution method and system based on uncertainty perception feature transmission | |
| CN113962878B (en) | Low-visibility image defogging model method | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| CN117237623B (en) | Semantic segmentation method and system for remote sensing image of unmanned aerial vehicle | |
| CN117952883A (en) | Backlight image enhancement method based on bilateral grid and significance guidance | |
| CN117333398A (en) | Multi-scale image denoising method and device based on self-supervision | |
| CN113096032A (en) | Non-uniform blur removing method based on image area division | |
| CN113033616B (en) | High-quality video reconstruction method, device, equipment and storage medium | |
| CN114943655A (en) | Image restoration system for generating confrontation network structure based on cyclic depth convolution | |
| CN114331894A (en) | Face image restoration method based on potential feature reconstruction and mask perception | |
| Wei et al. | Diffusion models for spatio-temporal-spectral fusion of homogeneous Gaofen-1 satellite platforms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: No. 548, 5th Floor, Building 10, No. 28 Linping Avenue, Donghu Street, Linping District, Hangzhou City, Zhejiang Province Applicant after: Hangzhou Tuke Intelligent Information Technology Co.,Ltd. Address before: 430000 B033, No. 05, 4th floor, building 2, international enterprise center, No. 1, Guanggu Avenue, Donghu New Technology Development Zone, Wuhan, Hubei (Wuhan area of free trade zone) Applicant before: Wuhan Tuke Intelligent Technology Co.,Ltd. |
|
| CB02 | Change of applicant information |