CN117934286B - Lightweight image super-resolution method and device and electronic equipment thereof - Google Patents
Lightweight image super-resolution method and device and electronic equipment thereof Download PDFInfo
- Publication number
- CN117934286B CN117934286B CN202410324546.6A CN202410324546A CN117934286B CN 117934286 B CN117934286 B CN 117934286B CN 202410324546 A CN202410324546 A CN 202410324546A CN 117934286 B CN117934286 B CN 117934286B
- Authority
- CN
- China
- Prior art keywords
- representing
- resolution
- level
- feature
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000000605 extraction Methods 0.000 claims abstract description 32
- 230000004927 fusion Effects 0.000 claims abstract description 26
- 230000003993 interaction Effects 0.000 claims abstract description 17
- 238000013135 deep learning Methods 0.000 claims abstract description 16
- 238000005516 engineering process Methods 0.000 claims abstract description 4
- 238000002156 mixing Methods 0.000 claims description 40
- 210000002569 neuron Anatomy 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 11
- 230000000946 synaptic effect Effects 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 210000005036 nerve Anatomy 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 101100365548 Caenorhabditis elegans set-14 gene Proteins 0.000 claims description 4
- 241000556720 Manga Species 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000001537 neural effect Effects 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 2
- 238000012886 linear function Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/52—Scale-space analysis, e.g. wavelet analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Neurology (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the field of computer vision, in particular to a lightweight image super-resolution method, a device and electronic equipment thereof, wherein the method comprises the following steps: s1: constructing a super-resolution method of the whole lightweight image by using NSNP-type Conv units; s2: constructing a local feature enhancement module by using a global self-attention and NSNP-type Conv unit, and constructing a multi-scale feature extraction unit by using a NSNP-type Conv unit; s3: using NSNP-type Conv units to construct a multi-level feature self-adaptive fusion module, and self-adaptively fusing the feature information of different levels through the level attention weight generated by the multi-level feature self-adaptive fusion module; s4: the method comprises the steps of constructing an image super-resolution method of multi-level characteristic information interaction, learning the super-resolution of the whole image in an end-to-end mode through a deep learning technology, and outputting a high-resolution reconstructed image of an image to be reconstructed. The method has better reconstruction performance, and solves the problems of large parameter quantity and large calculation burden in the existing algorithm to a great extent.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a lightweight image super-resolution method and device and electronic equipment thereof.
Background
With the development of the field of computer vision, people pay more attention to the field of limited image resolution in the task of computer vision, and how to efficiently recover a clearer and more detailed high-resolution image from a low-resolution image becomes the direction of research. The existing image super-resolution method can be roughly divided into: traditional interpolation-based methods and current deep learning-based methods.
The interpolation-based method is to increase the resolution of the image by estimating and filling the details among pixels, and common methods include bicubic interpolation, bilinear interpolation, linear interpolation and the like. Interpolation-based methods are generally unable to recover the missing high frequency details and thus deep learning-based methods have evolved greatly.
The deep learning-based method can learn complex image features and details from a large amount of data, common deep learning-based methods include SRCNN（Super-Resolution Convolutional Neural Network）、VDSR（Very Deep Super-Resolution Network）、EDSR（Enhanced Deep Super-Resolution） and the like, and can generally better keep details, textures and structures of images, so that a low-resolution image can be converted into a high-quality high-resolution image. These methods have found widespread use in many fields of application, such as image processing, computer vision, medical imaging, and satellite image processing.
Since "C. Dong, C.C. Loy, K. He, et al, Learning a deep convolutional network for image super-resolution, in: Proceedings of the European Conference Computer Vision (ECCV), 2014, pp. 184-199." first proposes a SRCNN model with only 3 layers of convolution, SISR based on deep learning is greatly developed, the image super-resolution model is increased from 3 layers to 20 layers, and the MemNet layers of the image super-resolution model are up to 80 layers for better image recovery.
In recent years, more and more image super-resolution models begin to pay attention to the balance between performance and memory consumption, and more lightweight models are proposed. How to effectively fuse the extracted feature information without increasing excessive parameters and calculation load, and to obtain excellent recovery results is one of the main problems of current image super-resolution concerns.
Part of the prior art is based on a deep learning method, a more complex and deeper neural network is constructed, and a better reconstruction result is obtained, however, the parameters of the methods are large, the methods are not suitable for being deployed on small electronic equipment, and the lightweight image super-resolution method has a greatly reduced parameter, but does not ensure an excellent reconstruction effect; some light-weight image super-resolution methods are also proposed in part of the prior art, and although there is a substantial reduction in the number of parameters, an excellent reconstruction effect is not ensured.
The image reconstruction method described in the chinese patent CN113674156B, which is a method and system for reconstructing an image with super resolution, relies on stacking a plurality of convolutions with different sizes to extract feature information, which results in that the feature information provided by the feature extraction module is not comprehensive enough during image reconstruction, while ensuring a lower parameter amount, but the reconstruction effect is not excellent.
Disclosure of Invention
In order to solve the problems, the invention provides a lightweight image super-resolution method, a lightweight image super-resolution device and electronic equipment thereof, which are used for efficiently extracting characteristic information of different layers and carrying out self-adaptive interactive fusion by using a small quantity of parameters, thereby providing effective characteristic information for image reconstruction. The relationship between the effect of network super-resolution and the network parameter quantity is balanced, so that the network super-resolution system can be better deployed on corresponding equipment.
In order to achieve the above object, the technical scheme of the present invention is as follows: in one aspect, a method for super resolution of a lightweight image is provided, including the steps of:
s1: constructing a super-resolution method of the whole lightweight image by using NSNP-type Conv units;
S2: in the multi-level feature mixing module, a local feature enhancement module is built by using a global self-attention and NSNP-type Conv unit, and a multi-scale feature extraction unit is built by using a NSNP-type Conv unit;
S3: using NSNP-type Conv units to construct a multi-level feature self-adaptive fusion module, and self-adaptively fusing the feature information of different levels through the level attention weight generated by the multi-level feature self-adaptive fusion module;
S4: the image super-resolution method for constructing multi-level characteristic information interaction is based on NSNP-type Conv units, a local characteristic enhancement module, a multi-scale characteristic extraction unit and a multi-level characteristic self-adaptive fusion module, an image super-resolution model is built through a deep learning technology, the whole image super-resolution model is learned in an end-to-end mode, and a high-resolution reconstructed image of an image to be reconstructed is output.
Further, NSNP-type Conv unit is constructed by the following steps:
firstly, according to a nonlinear pulse mechanism, an update equation of a nonlinear pulse nerve P system is obtained:
In which, in the process, Representing a synaptic connection between neuron i and neuron j; /(I)And/>Representing the state of neuron i at times t and t-1,/>The state of neuron j at time t-1; /(I)Indicating that the pulse threshold value is set,Representing a linear or nonlinear function,/>Representing a non-linear function,/>A pulse representing neuronal i depletion; /(I)A pulse representing the generation of neuron j;
then, setting t= - ≡ensures that the neuron is always in an activated state, and then setting And introducing synaptic weights among neurons to obtain a state equation: Wherein m represents the number of neurons; /(I) Representing synaptic weights between neuron i and neuron j; finally, NSNP-type Conv units are constructed by state equations.
Further, the local feature enhancement module in S2 consists of a layer normalization, a global self-attention module and NSNP-type Conv units; the operation is as follows: wherein, the method comprises the steps of, wherein, Representing local feature enhancement Module,/>Input features representing local feature enhancement modules,/>Representation layer normalization module,/>Representing a global self-attention module,/>Representing an element-by-element addition operation,The NSNP-type Conv unit containing two 1×1, 3×3 units is shown.
Further, in the multi-level feature mixing module, a plurality of multi-scale feature extraction units are constructed by using NSNP-type Conv units with different sizes; the operation is as follows: Wherein/> Multi-scale feature extraction units of size k x k representing convolution kernels of 1x 1, 3 x 3, 5x 5 and 7 x 7 respectively, each using a NSNP-type Conv unit of size 1x 1 to reduce the number of channels,/>NSNP-type Conv units representing a convolution kernel size of kXk, k being the convolution kernel size in NSNP-type Conv units, 1X 1, 3X 3, 5X 5 and 7X 7,/>, respectivelyNSNP-type Conv units of convolution kernel size 1×1 are shown.
Further, the multi-level feature mixing module comprises 3 local feature enhancement modules and 4 multi-scale feature extraction units, and features extracted by each local feature enhancement module pass through the scale feature extraction units with different sizes, and the specific representation is as follows: Wherein/> Representing a scale feature extraction unit comprising convolution kernels of sizes 1x 1, 3 x3, 5 x 5 and 7 x 7, respectively; /(I)Representing input features of the scale feature extraction unit; /(I)Representing feature information extracted by scale feature extraction units of convolution kernel sizes 1×1, 3×3,5×5, and 7×7, respectively,/>Respectively representing the characteristic information extracted by the 1 st, 2 nd and 3 rd local characteristic enhancement modules;
The characteristics extracted by the 4 scale characteristic extraction units are spliced according to the channel dimension, the extracted multi-level characteristics are simply fused through a NSNP-type Conv unit of 1 multiplied by 1 and 3 multiplied by 3, then the multi-level characteristics are connected through residual errors, the multi-level characteristics are added with the original input characteristics F, the obtained characteristics are used as the output of the multi-level characteristic mixing module, and the operation is as follows:
Wherein/> Representing the output characteristics of the multi-level characteristic mixing module; /(I)Representing a per channel splice operation,/>Input features representing a multi-level feature blending module,/>A NSNP-type Conv unit with a convolution kernel size of 3 x3 is shown.
Further, in S3, the hierarchical attention weight generated by the multi-level feature adaptive fusion module adaptively fuses feature information of different levels, where the operation is as follows:
wherein, the method comprises the steps of, wherein, The sizes are all 1 multiplied by 1,/>Representation pooling,/>Representing the generated hierarchical attention weights; /(I)Representing input features of an ith layer in the multi-layer feature blending module; /(I)Respectively representing input features of layers 1,2 and 3 in the multi-layer feature mixing module,/>Representing the passing of different input featuresGlobal feature information obtained after the operation of the ith layer in MLFH; /(I)The global feature information obtained after the input features are subjected to the 1 st, 2 nd and 3 rd layer operation in MLFH th layer operation is respectively represented; /(I)Representing an activation function; /(I)Respectively representing the obtained level attention weights after global feature information interaction of the 1 st level, the 2 nd level and the 3 rd level in the multi-layer feature mixing module; /(I)Representation pass-by-pass concatenation/>Convolution operation/>The characteristic information obtained afterwards; /(I)Representing input characteristics of the multi-level characteristic self-adaptive fusion module; /(I)Representing an element-wise multiplication operation.
Further, the image super-resolution method of multi-level characteristic information interaction in S4 comprises the following steps:
performing preliminary feature processing on the input image by using a NSNP-type Conv unit with the size of 3 multiplied by 3, and inputting the processed features into a multi-level feature interaction stage;
In the multi-level interaction stage, the input features sequentially pass through 3 cascaded multi-level feature mixing modules, and the multi-level features extracted by each multi-level feature mixing module are input into a multi-level feature self-adaptive fusion module for self-adaptive fusion and then are input into a reconstruction stage;
in the reconstruction stage, the features after upsampling are obtained, and then added with the image after bilinear interpolation to output a finally reconstructed high-resolution image; the operation is as follows:
In the above, the ratio of/> Representing the reconstructed high resolution image; /(I)Representing the input low resolution image; /(I)Representing a bilinear interpolation operation; /(I)Respectively representing 1 st, 2 nd and 3 rd multi-level characteristic mixing modules; respectively representing the characteristic information extracted by the 1 st and 2 nd multi-level characteristic mixing modules; /(I) Representing a reconstruction operation,/>NSNP-type Conv units of convolution kernel size 1×1 are shown.
Further, the low-resolution image-high-resolution image pair in the existing DIV2K dataset is used as a training dataset, a trained image super-resolution model weight file is saved, and the dataset of Set5, set14, B100, urban100 and Manga is used for testing the image super-resolution model performance.
In another aspect, there is provided a lightweight image super-resolution apparatus, comprising:
An acquisition module for acquiring a low resolution and high resolution image pair;
The training module is used for constructing an image super-resolution model by utilizing Pytorch deep learning frames on the deep learning equipment, and learning the weight of the image super-resolution model of the image super-resolution algorithm in the lightweight image super-resolution method in any one of the above modes in a training mode;
The generation module directly carries out reconstruction operation on the low-resolution image to be restored by utilizing the trained image super-resolution model weight parameter file, and outputs a corresponding reconstructed module.
In yet another aspect, an electronic device is provided that includes a computer-readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the lightweight image super resolution method of any of the above.
The adoption of the scheme has the following beneficial effects:
The invention not only has less parameter quantity, but also obtains better reconstruction performance, and obtains good balance between model parameters and performance. Compared with a lightweight image super-resolution algorithm, the method has better reconstruction performance, and the problems of large parameter quantity and large calculation load in the existing algorithm are solved to a great extent. According to the invention, a novel neuron NSNP-type Conv is constructed, and local information can be further enhanced in global information and context information is introduced by constructing a local feature enhancement module. Compared with the existing image super-resolution algorithm which simply cascades the features of different layers before the reconstruction stage, the gating fusion module in the invention can adaptively fuse the features of different layers by generating the attention weights of adjacent layers, and can better perform context interaction.
In addition, by using the trained model weight parameters, the image reconstruction work can be rapidly realized, and the method is simple and easy to deploy.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flow chart of an embodiment of a lightweight image super-resolution method of the present invention;
FIG. 2 is a schematic diagram of a local feature enhancement module according to an embodiment of the light-weight image super-resolution method of the present invention;
FIG. 3 is a schematic diagram of a multi-level feature blending module structure according to an embodiment of the light-weight image super-resolution method of the present invention;
FIG. 4 is a schematic diagram of a hierarchical feature adaptive fusion module according to an embodiment of the light-weight image super-resolution method of the present invention;
FIG. 5 is a schematic diagram of a light-weight image super-resolution device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an embodiment of the lightweight image super-resolution device of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The following is a further detailed description of the embodiments:
example 1: a light-weight image super-resolution method comprises the following steps:
S1: the entire lightweight image super-resolution method is constructed using NSNP-type Conv units.
The NSNP-type Conv unit is a novel neuron abstracted by a nonlinear pulse mechanism in a nonlinear pulse nerve P system. A nonlinear impulse nerve P system comprises m neurons each including a state unitAnd pulse nerve rule/>The impulse neural rule may be expressed specifically as/>Where T represents the pulse threshold, a represents the pulse,/>Is a linear or nonlinear function,/>Is a nonlinear function. Pulse condition is;/>Representing synaptic connections between neurons.
The NSNP-type Conv unit is constructed by the following steps: firstly, according to a nonlinear pulse mechanism, an update equation of a nonlinear pulse nerve P system is obtained:
In the above, the ratio of/> Representing a synaptic connection between neuron i and neuron j; /(I)And/>Representing the state of neuron i at times t and t-1,/>The state of neuron j at time t-1; /(I)Representing a set pulse threshold; /(I)Representing a linear or nonlinear function,/>Representing a non-linear function,/>A pulse representing neuronal i depletion; /(I)A pulse representing the generation of neuron j; then, to further construct NSNP-type Conv, set t= - ≡to ensure that the neuron is always in an activated state, and then setAnd introducing synaptic weights among neurons to obtain a state equation:
Wherein m represents the number of neurons; /(I) Representing synaptic weights between neuron i and neuron j; finally, NSNP-type Conv units are novel neurons constructed according to the above state equation and the requirements of image processing.
The light-weight image super-resolution method comprises a preliminary feature processing stage, a multi-level feature interaction stage and a reconstruction stage, and is particularly shown in the figure 1.
S2: in the multi-level feature blending module, a local feature enhancement module is constructed using the global self-attention and NSNP-type Conv unit, and a multi-scale feature extraction unit is constructed using the NSNP-type Conv unit.
In S2, in order to extract local feature information, a local feature enhancement module (hereinafter referred to as TsFE) is constructed in cooperation with the global self-attention and NSNP-type Conv, tsFE combines different global self-attention blocks to form a global self-attention group (hereinafter referred to as OSAG), global information is extracted, and nonlinear change is introduced by using NSNP-type Conv, so that local information in the global information is further enhanced, and the network expression capability is improved.
TsFE consists of layer normalization (LN below), OSAG, NSNP-type Conv units, as shown in FIG. 2; the operation is as follows:
Wherein/> Representing local feature enhancement Module,/>Input features representing local feature enhancement modules,/>Representation layer normalization module,/>Representing a global self-attention module,/>Representing an element-by-element addition operation,The NSNP-type Conv unit containing two 1×1, 3×3 units is shown.
In a multi-level feature mixing Module (MLFH), a plurality of multi-scale feature extraction units are constructed by using NSNP-type Conv units with different sizes; the operation is as follows:
Wherein/> Multi-scale feature extraction units of size k x k representing convolution kernels of 1x 1, 3 x 3, 5x 5 and 7 x 7 respectively, each using a NSNP-type Conv unit of size 1x 1 to reduce the number of channels,/>NSNP-type Conv units representing a convolution kernel size of kXk, k being the convolution kernel size in NSNP-type Conv units, 1X 1, 3X 3, 5X 5 and 7X 7,/>, respectivelyNSNP-type Conv units of convolution kernel size 1×1 are shown.
The multi-level feature mixing module comprises 3 local feature enhancement modules and 4 multi-scale feature extraction units, as shown in fig. 3, features extracted by each local feature enhancement module pass through scale feature extraction units with different sizes, and the specific representation is as follows:
Wherein/> Representing a scale feature extraction unit (MSEU) comprising convolution kernels of sizes 1x 1, 3 x 3,5 x 5 and 7 x 7, respectively; /(I)Representing input features of the scale feature extraction unit; /(I)Representing feature information extracted by a scale feature extraction unit (MSEU) having convolution kernel sizes of 1×1,3×3, 5×5, and 7×7, respectively,/>The feature information extracted by the 1 st, 2 nd and 3 rd local feature enhancement modules (TsFE) are respectively represented.
The characteristics extracted by the 4 scale characteristic extraction units are spliced according to the channel dimension, the extracted multi-level characteristics are simply fused through a NSNP-type Conv unit of 1 multiplied by 1 and 3 multiplied by 3, then the multi-level characteristics are connected through residual errors, the multi-level characteristics are added with the original input characteristics F, the obtained characteristics are used as the output of the multi-level characteristic mixing module, and the operation is as follows:
Wherein/> Representing the output characteristics of a multi-level feature blending Module (MLFH); /(I)Representing a channel-by-channel splice operation; Feature information extracted by a scale feature extraction unit (MSEU) having convolution kernel sizes of 1×1,3×3, 5×5, and 7×7, respectively; /(I) Representing the input features of a multi-level feature blending Module (MLFH),NSNP-type Conv units representing a convolution kernel size of 3×3 size,/>NSNP-type Conv units of convolution kernel size 1×1 are shown.
S3: and a NSNP-type Conv unit is used for constructing a multi-level characteristic self-adaptive fusion module, and as shown in figure 4, the hierarchical attention weight generated by the multi-level characteristic self-adaptive fusion module is used for self-adaptively fusing the characteristic information of different layers.
Specifically, in S3, the hierarchical attention weight generated by the multi-level feature adaptive fusion module adaptively fuses feature information of different levels, where the operation is as follows: Wherein/> The sizes are all 1 multiplied by 1,/>Representation pooling,/>Representing the generated hierarchical attention weights; /(I)Representing input features of an i-th layer in a multi-layer feature blending Module (MLFH); /(I)Input features of layers 1, 2 and 3 in the multilayer feature mixing Module (MLFH) are respectively represented,/>Representing the passing of different input featuresGlobal feature information obtained after the operation of the ith layer in MLFH; The global feature information obtained after the input features are subjected to the 1 st, 2 nd and 3 rd layer operation in MLFH th layer operation is respectively represented; representing an activation function; /(I) Respectively representing the level attention weights obtained after the global feature information interaction of the 1 st level, the 2 nd level and the 3 rd level in the multi-layer feature mixing Module (MLFH); /(I)Representation pass-by-pass concatenation/>Convolution operation/>The characteristic information obtained afterwards; /(I)Representing input features of a multi-level feature adaptive fusion Module (MFAF); /(I)Representing an element-wise multiplication operation.
S4: the image super-resolution method for constructing multi-level characteristic information interaction specifically comprises the following steps:
performing preliminary feature processing on the input image by using a NSNP-type Conv unit with the size of 3 multiplied by 3, and inputting the processed features into a multi-level feature interaction stage;
In the multi-level interaction stage, the input features sequentially pass through 3 cascaded multi-level feature mixing modules, and the multi-level features extracted by each multi-level feature mixing module are input into a multi-level feature self-adaptive fusion module for self-adaptive fusion and then are input into a reconstruction stage;
In the reconstruction stage, the features after upsampling are obtained, and then added with the image after bilinear interpolation to output a finally reconstructed high-resolution image; wherein the operation is that . Wherein,Representing the reconstructed high resolution image; /(I)Representing the input low resolution image; /(I)Representing a bilinear (Biliner) interpolation operation; /(I)Respectively representing 1 st, 2 nd and 3 rd multi-level characteristic mixing Modules (MLFH); /(I)Respectively representing the characteristic information extracted by the 1 st and 2 nd multi-level characteristic mixing Modules (MLFH); Representing a reconstruction operation,/> NSNP-type Conv units of convolution kernel size 1×1 are shown.
Through a deep learning technology, an image super-resolution network is learned in an end-to-end mode, a low-resolution image to be reconstructed is obtained through the learned image super-resolution network, and then a high-resolution reconstructed image is output.
Specifically, the learning process of constructing the lightweight image super-resolution network uses a low-resolution image-high-resolution image pair in the DIV2K dataset as a training dataset, stores a trained image super-resolution model weight file, and uses the datasets of Set5, set14, B100, urman 100 and Manga to test the performance of the image super-resolution model.
The image super-resolution model is trained for better training and the effectiveness of the image super-resolution model is evaluated. Thus, the image super-resolution model is trained on the public dataset DIV2K and evaluated on the other 5 reference datasets (Set 5, set14, B100, urban100, manga, 109). And comparing the image super-resolution model with a lightweight image super-resolution method to verify the performance of the image super-resolution model.
From the evaluation result, the image super-resolution model in the proposed lightweight image super-resolution method has good balance between performance and parameter quantity. The evaluation results are shown in the following table:
table 1: comparison result table (scale factor X2) of 5 reference data sets and other image super-resolution methods
Table 2: comparison result table (scale factor X3) of 5 reference data sets and other image super-resolution methods
Table 3: comparison result table (scale factor X4) of 5 reference data sets and other image super-resolution methods
Example 2: as shown in fig. 5, a lightweight image super-resolution device, based on the lightweight image super-resolution method of embodiment 1, comprises: an acquisition module 501, a training module 502 and a generation module 503.
The acquisition module 501 is configured to acquire a low resolution and high resolution image pair.
The training module 502 builds an image super-resolution model on the deep learning device by using Pytorch deep learning framework, and learns the weights of the image super-resolution model of the image super-resolution algorithm in the lightweight image super-resolution method of embodiment 1 in a training manner.
The generating module 503 uses the trained image super-resolution model weight parameter file to directly reconstruct the low-resolution image to be restored, and outputs a corresponding reconstructed module.
Example 3: as shown in fig. 6, an electronic device 600 includes an input device 610, a processing device 620, a storage device 630, and an output device 640, where the storage device 630 is provided with an image super-resolution processing module 631.
The input device 610 may use a camera, touch screen display, or the like. The processing device 620 may use a processor such as an image processing unit (GPU), a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA), if a trained image super resolution model weight file (the lightweight image super resolution network in embodiment 1) is used in the storage device 630, if a small-scale image reconstruction is performed, it is recommended to use a configuration of CPU intel i6 series with 8GB RAM or more, if a large-scale image reconstruction needs to be processed, it is recommended to use CPU intel xen platform 8255C with 32GB DDR4 and NVIDIA TESLA T and above configurations. The storage device 630 may be a Solid state disk (Solid STATE DRIVE, SSD), a USB flash drive, a read-only memory (ROM), a Random Access Memory (RAM), or other media capable of storing codes and weight files, the image super-resolution processing module 631 in the storage device 630 is implemented in a program form, and the output device 640 may be a computer display screen or a projector, or other devices.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (7)
1. The light-weight image super-resolution method is characterized by comprising the following steps of:
s1: constructing a super-resolution method of the whole lightweight image by using NSNP-type Conv units;
S2: in the multi-level feature mixing module, a local feature enhancement module is built by using a global self-attention and NSNP-type Conv unit, and a multi-scale feature extraction unit is built by using a NSNP-type Conv unit;
S3: using NSNP-type Conv units to construct a multi-level feature self-adaptive fusion module, and self-adaptively fusing the feature information of different levels through the level attention weight generated by the multi-level feature self-adaptive fusion module;
S4: constructing an image super-resolution method of multi-level characteristic information interaction, constructing an image super-resolution model through a deep learning technology based on NSNP-type Conv units, a local characteristic enhancement module, a multi-scale characteristic extraction unit and a multi-level characteristic self-adaptive fusion module, learning the whole image super-resolution model in an end-to-end mode, and outputting a high-resolution reconstructed image of an image to be reconstructed;
wherein NSNP-type Conv unit is constructed by the following steps:
firstly, according to a nonlinear pulse mechanism, an update equation of a nonlinear pulse nerve P system is obtained: In the method, in the process of the invention, Representing a synaptic connection between neuron i and neuron j; /(I)And/>Representing the state of neuron i at times t and t-1,/>The state of neuron j at time t-1; /(I)Representing the set pulse threshold,/>Representing a linear or nonlinear function,/>Representing a nonlinear function; /(I)A pulse representing neuronal i depletion; /(I)A pulse representing the generation of neuron j;
then, setting t= - ≡ensures that the neuron is always in an activated state, and then setting And introducing synaptic weights among neurons to obtain a state equation: Wherein m represents the number of neurons; /(I) Representing synaptic weights between neuron i and neuron j;
finally, a NSNP-type Conv unit is constructed through a state equation;
and S3, the hierarchical attention weight self-adaptive fusion module generates hierarchical attention weight self-adaptive fusion of the feature information of different layers, wherein the operation is as follows: Wherein/> The sizes are all 1 multiplied by 1,/>Representation pooling,/>Representing the generated hierarchical attention weights; /(I)Representing input features of an ith layer in the multi-layer feature blending module; /(I)、/>、/>Respectively representing input features of layers 1,2 and 3 in the multi-layer feature mixing module,/>Representing the passing of different input features/>、/>、Global feature information obtained after the operation of the ith layer in MLFH; /(I)、/>、/>The global feature information obtained after the input features are subjected to the 1 st, 2 nd and 3 rd layer operation in MLFH th layer operation is respectively represented; /(I)Representing an activation function; /(I)、Respectively representing the obtained level attention weights after global feature information interaction of the 1 st level, the 2 nd level and the 3 rd level in the multi-layer feature mixing module; /(I)Representation pass-by-pass concatenation/>Convolution operationThe characteristic information obtained afterwards; /(I)Representing input characteristics of the multi-level characteristic self-adaptive fusion module; /(I)Representing an element-wise multiplication operation;
s4, an image super-resolution method for multi-level characteristic information interaction comprises the following steps:
performing preliminary feature processing on the input image by using a NSNP-type Conv unit with the size of 3 multiplied by 3, and inputting the processed features into a multi-level feature interaction stage;
In the multi-level interaction stage, the input features sequentially pass through 3 cascaded multi-level feature mixing modules, and the multi-level features extracted by each multi-level feature mixing module are input into a multi-level feature self-adaptive fusion module for self-adaptive fusion and then are input into a reconstruction stage;
in the reconstruction stage, the features after upsampling are obtained, and then added with the image after bilinear interpolation to output a finally reconstructed high-resolution image; the operation is as follows: in the/> Representing the reconstructed high resolution image; /(I)Representing the input low resolution image; /(I)Representing a bilinear interpolation operation;、/>、/> Respectively representing 1 st, 2 nd and 3 rd multi-level characteristic mixing modules; /(I) 、/>Respectively representing the characteristic information extracted by the 1 st and 2 nd multi-level characteristic mixing modules; /(I)Representing a reconstruction operation; NSNP-type Conv units of convolution kernel size 1×1 are shown.
2. The method of claim 1, wherein the local feature enhancement module in S2 consists of a layer normalization, global self-attention module, NSNP-type Conv unit; the operation is as follows: Wherein/> Representing local feature enhancement Module,/>Input features representing local feature enhancement modules,/>The layer normalization module is represented by a layer,Representing a global self-attention module,/>Representing element-by-element addition operations,/>The NSNP-type Conv unit containing two 1×1, 3×3 units is shown.
3. The method according to claim 2, wherein in the multi-level feature mixing module, a plurality of multi-scale feature extraction units are constructed by using NSNP-type Conv units with different sizes; the operation is as follows: Wherein/> Multi-scale feature extraction units of size k x k representing convolution kernels of 1x 1, 3 x 3, 5x 5 and 7 x 7 respectively, each using a NSNP-type Conv unit of size 1x 1 to reduce the number of channels,/>NSNP-type Conv units representing a convolution kernel size of kXk, k being the convolution kernel size in NSNP-type Conv units, 1X 1, 3X 3, 5X 5 and 7X 7,/>, respectivelyNSNP-type Conv units of convolution kernel size 1×1 are shown.
4. The method of super-resolution of a lightweight image according to claim 3, wherein the multi-level feature blending module comprises 3 local feature enhancement modules and 4 multi-scale feature extraction units, and features extracted by each local feature enhancement module pass through the scale feature extraction units with different sizes, specifically expressed as follows: Wherein, 、/>、/>、/>Representing a scale feature extraction unit comprising convolution kernels of sizes 1x 1, 3 x3, 5 x 5 and 7 x 7, respectively; /(I)Representing input features of the scale feature extraction unit; /(I)、/>、/>、Representing feature information extracted by scale feature extraction units of convolution kernel sizes 1×1, 3×3,5×5, and 7×7, respectively,/>、/>、/>Respectively representing the characteristic information extracted by the 1 st, 2 nd and 3 rd local characteristic enhancement modules;
The characteristics extracted by the 4 scale characteristic extraction units are spliced according to the channel dimension, the extracted multi-level characteristics are simply fused through a NSNP-type Conv unit of 1 multiplied by 1 and 3 multiplied by 3, then the multi-level characteristics are connected through residual errors, the multi-level characteristics are added with the original input characteristics F, the obtained characteristics are used as the output of the multi-level characteristic mixing module, and the operation is as follows: Wherein/> Representing the output characteristics of the multi-level characteristic mixing module; /(I)Representing a channel-by-channel splice operation; /(I)Representing input features of the multi-level feature mixing module; /(I)NSNP-type Conv units of convolution kernel size 1×1 are shown.
5. The method of claim 4, wherein the training dataset is a low-resolution image-high-resolution image pair in the existing DIV2K dataset, the trained image super-resolution model weight file is saved, and the image super-resolution model performance is tested using the Set5, set14, B100, urban100, and Manga datasets.
6. A lightweight image super-resolution device, comprising:
An acquisition module for acquiring a low resolution and high resolution image pair;
The training module is used for constructing an image super-resolution model by utilizing Pytorch deep learning frames on the deep learning equipment, and learning the weight of the image super-resolution model in the lightweight image super-resolution method according to any one of claims 1-5 in a training mode;
The generation module directly carries out reconstruction operation on the low-resolution image to be restored by utilizing the trained image super-resolution model weight parameter file, and outputs a corresponding reconstructed module.
7. An electronic device comprising a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the lightweight image super resolution method as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410324546.6A CN117934286B (en) | 2024-03-21 | 2024-03-21 | Lightweight image super-resolution method and device and electronic equipment thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410324546.6A CN117934286B (en) | 2024-03-21 | 2024-03-21 | Lightweight image super-resolution method and device and electronic equipment thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117934286A CN117934286A (en) | 2024-04-26 |
| CN117934286B true CN117934286B (en) | 2024-06-04 |
Family
ID=90750886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410324546.6A Active CN117934286B (en) | 2024-03-21 | 2024-03-21 | Lightweight image super-resolution method and device and electronic equipment thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117934286B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109118430A (en) * | 2018-08-24 | 2019-01-01 | 深圳市商汤科技有限公司 | Super-resolution image reconstruction method and device, electronic equipment and storage medium |
| CN109886871A (en) * | 2019-01-07 | 2019-06-14 | 国家新闻出版广电总局广播科学研究院 | The image super-resolution method merged based on channel attention mechanism and multilayer feature |
| CN110602403A (en) * | 2019-09-23 | 2019-12-20 | 华为技术有限公司 | Method for taking pictures under dark light and electronic equipment |
| CN112446828A (en) * | 2021-01-29 | 2021-03-05 | 成都东方天呈智能科技有限公司 | Thermal imaging super-resolution reconstruction method fusing visible image gradient information |
| CN112991167A (en) * | 2021-01-27 | 2021-06-18 | 广东工业大学 | Aerial image super-resolution reconstruction method based on layered feature fusion network |
| CN114049251A (en) * | 2021-09-01 | 2022-02-15 | 中国矿业大学 | Fuzzy image super-resolution reconstruction method and device for AI video analysis |
| CN114596488A (en) * | 2022-02-25 | 2022-06-07 | 河南大学 | Lightweight remote sensing target detection method based on dense feature fusion network |
| CN115496658A (en) * | 2022-09-25 | 2022-12-20 | 桂林理工大学 | Lightweight image super-resolution reconstruction method based on double attention mechanism |
| CN117291803A (en) * | 2023-09-28 | 2023-12-26 | 太原理工大学 | PAMGAN lightweight facial super-resolution reconstruction method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10852838B2 (en) * | 2014-06-14 | 2020-12-01 | Magic Leap, Inc. | Methods and systems for creating virtual and augmented reality |
| CN109461495B (en) * | 2018-11-01 | 2023-04-14 | 腾讯科技（深圳）有限公司 | Medical image recognition method, model training method and server |
-
2024
- 2024-03-21 CN CN202410324546.6A patent/CN117934286B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109118430A (en) * | 2018-08-24 | 2019-01-01 | 深圳市商汤科技有限公司 | Super-resolution image reconstruction method and device, electronic equipment and storage medium |
| CN109886871A (en) * | 2019-01-07 | 2019-06-14 | 国家新闻出版广电总局广播科学研究院 | The image super-resolution method merged based on channel attention mechanism and multilayer feature |
| CN110602403A (en) * | 2019-09-23 | 2019-12-20 | 华为技术有限公司 | Method for taking pictures under dark light and electronic equipment |
| CN112991167A (en) * | 2021-01-27 | 2021-06-18 | 广东工业大学 | Aerial image super-resolution reconstruction method based on layered feature fusion network |
| CN112446828A (en) * | 2021-01-29 | 2021-03-05 | 成都东方天呈智能科技有限公司 | Thermal imaging super-resolution reconstruction method fusing visible image gradient information |
| CN114049251A (en) * | 2021-09-01 | 2022-02-15 | 中国矿业大学 | Fuzzy image super-resolution reconstruction method and device for AI video analysis |
| CN114596488A (en) * | 2022-02-25 | 2022-06-07 | 河南大学 | Lightweight remote sensing target detection method based on dense feature fusion network |
| CN115496658A (en) * | 2022-09-25 | 2022-12-20 | 桂林理工大学 | Lightweight image super-resolution reconstruction method based on double attention mechanism |
| CN117291803A (en) * | 2023-09-28 | 2023-12-26 | 太原理工大学 | PAMGAN lightweight facial super-resolution reconstruction method |
Non-Patent Citations (1)
| Title |
|---|
| 基于自适应注意力融合特征提取网络的图像超分辨率;王拓然等;《计算机应用研究》;20231208;第40卷(第11期);第3472-3477页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117934286A (en) | 2024-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lan et al. | MADNet: A fast and lightweight network for single-image super resolution | |
| CN111369440B (en) | Model training and image super-resolution processing method, device, terminal and storage medium | |
| CN113240580B (en) | Lightweight image super-resolution reconstruction method based on multi-dimensional knowledge distillation | |
| Ahn et al. | Image super-resolution via progressive cascading residual network | |
| CN111242846B (en) | Fine-grained scale image super-resolution method based on non-local enhancement network | |
| CN110163801B (en) | Image super-resolution and coloring method, system and electronic equipment | |
| CN112862689B (en) | Image super-resolution reconstruction method and system | |
| Zhang et al. | Gated fusion network for degraded image super resolution | |
| CN109949224B (en) | Deep learning-based cascade super-resolution reconstruction method and device | |
| CN111932461A (en) | Convolutional neural network-based self-learning image super-resolution reconstruction method and system | |
| CN115564649B (en) | Image super-resolution reconstruction method, device and equipment | |
| Jin et al. | Single image super-resolution with multi-level feature fusion recursive network | |
| CN111862251A (en) | Method, apparatus, storage medium and electronic device for medical image reconstruction technology | |
| Rivadeneira et al. | Thermal image super-resolution challenge-pbvs 2021 | |
| Conde et al. | Efficient deep models for real-time 4k image super-resolution. NTIRE 2023 benchmark and report | |
| CN111932480A (en) | Deblurred video recovery method and device, terminal equipment and storage medium | |
| CN115358932A (en) | Multi-scale feature fusion face super-resolution reconstruction method and system | |
| CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
| Wang et al. | Learning a degradation-adaptive network for light field image super-resolution | |
| CN111553861A (en) | Image super-resolution reconstruction method, device and equipment and readable storage medium | |
| Liu et al. | Facial image inpainting using multi-level generative network | |
| Wen et al. | Progressive representation recalibration for lightweight super-resolution | |
| CN106981046A (en) | Single image super resolution ratio reconstruction method based on multi-gradient constrained regression | |
| CN113989122A (en) | Super-resolution recovery method and system for image, electronic device and storage medium | |
| CN115760670B (en) | Unsupervised hyperspectral fusion method and device based on network implicit priori |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |