CN114972043B - Image super-resolution reconstruction method and system based on combined trilateral feature filtering - Google Patents
Image super-resolution reconstruction method and system based on combined trilateral feature filtering Download PDFInfo
- Publication number
- CN114972043B CN114972043B CN202210924242.4A CN202210924242A CN114972043B CN 114972043 B CN114972043 B CN 114972043B CN 202210924242 A CN202210924242 A CN 202210924242A CN 114972043 B CN114972043 B CN 114972043B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- feature
- gradient
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001914 filtration Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000004927 fusion Effects 0.000 claims abstract description 38
- 238000007670 refining Methods 0.000 claims abstract description 16
- 238000012549 training Methods 0.000 claims description 35
- 230000006870 function Effects 0.000 claims description 34
- 238000000605 extraction Methods 0.000 claims description 22
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 14
- 238000005070 sampling Methods 0.000 claims description 13
- 230000002457 bidirectional effect Effects 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 238000012360 testing method Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 5
- 238000011156 evaluation Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image super-resolution reconstruction method and system based on combined trilateral feature filtering, wherein the method comprises the following steps: downsampling the high-resolution image to obtain a low-resolution image, and constructing a single-image super-resolution reconstruction model; extracting and obtaining multi-level image features through a feature thinning module in the image reconstruction branch; extracting and obtaining multi-level gradient characteristics through a gradient refining module in the gradient prediction branch; based on the image reconstruction branch and the gradient prediction branch, fusion guidance is carried out through a combined trilateral feature filtering module so as to realize self-adaptive adjustment of a convolution kernel of a target domain, and reconstruction of a high-resolution image is carried out from coarse to fine; and when the single image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image. The invention deeply excavates the mutual guidance between the gradient domain and the pixel domain, and finally realizes good reconstruction effect.
Description
Technical Field
The invention relates to the technical field of computer image processing, in particular to an image super-resolution reconstruction method and system based on combined trilateral feature filtering.
Background
The image super-resolution reconstruction means giving a point low-resolution image and then restoring a corresponding high-resolution image, and belongs to the problem of low-level computer vision. The image super-resolution reconstruction technology can significantly improve the performance of other advanced vision tasks, such as classification and identification.
At present, the image super-resolution reconstruction technology makes great progress, and particularly, with the rise of a deep convolutional neural network, a relevant method learns corresponding prior information through an end-to-end model characterized by hierarchical features. In early studies in this direction, researchers were working to improve model performance by designing specific network architectures. For example: residual learning and dense concatenation, etc. These deep convolutional neural network based methods implicitly learn the mapping function between the low resolution input and the high resolution output by means of black box training. Therefore, the designed model guarantees the orderly training while pursuing the high characterization capability of deeper networks.
Although deep networks trained by means of black boxes can significantly improve performance, this model has two drawbacks: (1), the first drawback is the lack of explanatory model; (2) The second drawback is the lack of generalization of the model, the parameters in the test phase are fixed and independent of the input content. To address the first drawback, researchers embed a priori into the network, such as a feature pyramid, discrete cosine transform. However, this solution is not complete. In particular, the design trend of modern networks requires efficient and effective fusion of multi-scale features in the same domain. Furthermore, cross-domain feature fusion is also necessary under the multitask learning framework. In these methods, feature fusion is usually achieved by simple channel merging, followed by a convolutional layer. Although sharing convolution weights at spatial locations with translational invariance can improve feature extraction, this simple and undirected approach still fails to model content-dependent feature fusion. For the second drawback, existing approaches mostly improve generalization capability through a mechanism of attention that scales feature representations in spatial and channel dimensions at the element level. However, this approach does not explicitly model the fusion of image features and gradient features.
Based on this, it is necessary to provide a new image super-resolution reconstruction method to solve the above technical problems.
Disclosure of Invention
In view of the above-mentioned situation, the main object of the present invention is to provide a method for reconstructing super-resolution images based on joint trilateral feature filtering, which is used to solve the above-mentioned technical problems.
The embodiment of the invention provides an image super-resolution reconstruction method based on combined trilateral feature filtering, wherein the method comprises the following steps:
the method comprises the steps of firstly, down-sampling a high-resolution image to obtain a low-resolution image, constructing a single-image super-resolution reconstruction model, and inputting the low-resolution image into the single-image super-resolution reconstruction model, wherein the single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, the image reconstruction branch at least comprises a feature refining module, and the gradient prediction branch at least comprises a gradient refining module;
secondly, extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch;
step three, extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch;
performing fusion guidance by combining a trilateral feature filtering module based on the image reconstruction branch and the gradient prediction branch to realize self-adaptive adjustment of a convolution kernel of a target domain, and reconstructing a high-resolution image from coarse to fine;
and fifthly, when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image.
The invention provides an image super-resolution reconstruction method based on combined trilateral feature filtering, which comprises the steps of firstly, extending a filtering definition domain from a pixel domain to a high-dimensional feature domain, and simulating a combined trilateral feature filter defined on an image feature domain and a gradient feature domain by constructing a neural network structure; by sensing corresponding features on an image feature domain and a gradient feature domain and combining with the trilateral feature filtering module, the convolution kernel of the features of the target domain is adaptively adjusted, and the generalization performance of the feature fusion module is improved;
and secondly, based on a combined trilateral feature filtering module, alternately taking the image features and the gradient features as guide domain features, deeply excavating mutual guide effect between the gradient domain and the image domain, and realizing bidirectional fusion of the feature domains. Compared with the most advanced method at present, the image super-resolution reconstruction method based on the combined trilateral feature filtering achieves the best effect of subjective evaluation and objective evaluation.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that a plurality of high-resolution images form a high-resolution image data set, and the method for processing the high-resolution image data set comprises the following steps:
dividing the high-resolution image data set into a training set, a verification set and a test set;
down-sampling a high resolution image in the high resolution image dataset to generate a corresponding low resolution image;
and correspondingly cutting the high-resolution image and the low-resolution image into paired sub image blocks according to a preset image size, selecting a specific sub image block, and performing random overturning and rotation to perform data enhancement, thereby finally obtaining the training set.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that the image reconstruction branch comprises a first shallow feature extraction module, a feature refinement module and an image up-sampling module which are sequentially connected;
the first shallow feature extraction module consists of two convolution layers and is used for extracting and obtaining the first shallow image features
；
The feature refinement module comprises
A plurality of residual dense connecting blocks which are connected in sequence and used for extracting and obtaining multi-level image characteristics
，
Wherein
representing the maximum number of residual dense connection blocks, each comprising two basic units, wherein the current residual dense connection block is used for the multi-level image features obtained by extraction
Updating through bidirectional feature fusion before inputting to the next residual dense connecting block;
the image up-sampling module is used for converting the image into a digital image
In a residual dense connecting block
The outputs of the elementary cells are stitched in channel dimensions as input to the image upsampling module.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that the gradient prediction branch comprises a second shallow feature extraction module, a gradient refinement module and a gradient reconstruction module;
the second shallow feature extraction module consists of two convolution layers and is used for extracting and obtaining second shallow image features
；
The ladderThe degree refining module comprises
A plurality of sequentially connected residual blocks for extracting multi-level gradient features
，
Wherein
representing the maximum number of residual blocks, each of which comprises two elementary units, wherein the current residual block is used for multi-level gradient features to be extracted
Updating through bidirectional feature fusion before inputting to the next residual block;
the gradient reconstruction module is used for carrying out multi-level gradient feature after refinement
As input, a high resolution gradient map is then output.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that a loss function corresponding to the single-image super-resolution reconstruction model is expressed as follows:
wherein,
the corresponding loss value of the loss function is represented,
presentation pairThe operation of taking the minimum value of the loss function,
represents the maximum number of training images and,
a sequence number representing the training image is shown,
is shown as
A high resolution original image in a training image,
is shown as
A low resolution original image in a training image,
is shown as
The bicubic interpolation result corresponding to the training image,
is shown as
Parameters in sheet training images at low resolution
The corresponding model is generated according to the model,
a value representing the weight of the balance weight,
is shown as
A high resolution gradient map in a training image,
representing the low resolution gradient map in the first training image,
is shown as
Parameters under high-resolution gradient map in training image
And generating a corresponding model.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that bidirectional feature fusion represents image features at multiple levels
And multi-level gradient feature
And the combined trilateral feature filtering module performs feature fusion alternately to realize image-guided gradient feature enhancement and gradient-guided image feature enhancement.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that in the combined trilateral feature filtering module, an expression corresponding to the fusion feature of the target feature obtained through output is as follows:
wherein,
indicating pre-fusion channels
In space of
The characteristics of (a) to (b),
represents the weights of the convolution kernels corresponding to the features of the target domain,
representing the convolution kernel weights corresponding to the guiding domain features,
a local window is represented that is a window of a scene,
the representation being located in a local window
A fused feature of the central target feature.
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that in the combined trilateral feature filtering module, target domain features are used for convolution operation
In that
Dimensionally in the order of
A traversal is made for the window size to obtain a plurality of three-dimensional image blocks,
wherein, in the process,
respectively representing the number, height and width of channels;
wherein the processed target domain has a characteristic size of
，
In order to perform the batch processing number,
the representation of the real number field is performed,
representing a coordinate dimension of
A high-dimensional tensor space over a real domain of (a).
The image super-resolution reconstruction method based on the combined trilateral feature filtering is characterized in that a kernel function of an image feature and a kernel function of a gradient feature are learned through two sub-networks with the same structure in the combined trilateral feature filtering module respectively;
sub-network learning from image features
Kernel function to image features
The mapping of (a), wherein,
，
；
sub-network learning from gradient features
Kernel function to gradient features
The mapping of (a), wherein,
，
；
by kernel functions of image features
Kernel function of gradient feature
Multiplying the corresponding elements to obtain a combined trilateral feature filtering kernel
；
Wherein,
representing a coordinate dimension of
A high-dimensional tensor space over a real number domain of,
represents a coordinate dimension of
A high-dimensional tensor space over a real number domain.
The invention also provides an image super-resolution reconstruction system based on the combined trilateral feature filtering, wherein the system comprises:
a model building module to:
the method comprises the steps of downsampling a high-resolution image to obtain a low-resolution image, constructing a single-image super-resolution reconstruction model, and inputting the low-resolution image into the single-image super-resolution reconstruction model, wherein the single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, the image reconstruction branch at least comprises a feature thinning module, and the gradient prediction branch at least comprises a gradient thinning module;
a first extraction module to:
extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch;
a second extraction module to:
extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch;
a feature fusion module to:
based on the image reconstruction branch and the gradient prediction branch, fusion guidance is carried out through a combined trilateral feature filtering module to realize self-adaptive adjustment of a convolution kernel of a target domain, and reconstruction of a high-resolution image is carried out from coarse to fine;
an iterative convergence module to:
and when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a flow chart of an image super-resolution reconstruction method based on combined trilateral feature filtering according to the present invention;
FIG. 2 is a network topology diagram of an image reconstruction branch and a gradient prediction branch in the image super-resolution reconstruction method based on combined trilateral feature filtering provided by the invention;
fig. 3 is a structural diagram of an image super-resolution reconstruction system based on combined trilateral feature filtering according to the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
These and other aspects of embodiments of the invention will be apparent with reference to the following description and attached drawings. In the description and drawings, particular embodiments of the invention have been disclosed in detail as being indicative of some of the ways in which the principles of the embodiments of the invention may be practiced, but it is understood that the scope of the embodiments of the invention is not limited correspondingly. On the contrary, the embodiments of the invention include all changes, modifications and equivalents coming within the spirit and terms of the claims appended hereto.
Referring to fig. 1 and fig. 2, the present invention provides an image super-resolution reconstruction method based on a combined trilateral feature filtering, wherein the method includes the following steps:
s101, down-sampling is carried out on the high-resolution image to obtain a low-resolution image, a single-image super-resolution reconstruction model is constructed, and the low-resolution image is input into the single-image super-resolution reconstruction model.
The single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, wherein the image reconstruction branch at least comprises a characteristic thinning module, and the gradient prediction branch at least comprises a gradient thinning module.
It should be noted here that the neural network model (image super-resolution reconstruction model based on joint trilateral feature filtering) in the present invention is only used for predicting the result of bicubic interpolation corresponding to an image
And high resolution original image
The residual error between.
In step S101, a plurality of high resolution images constitute a high resolution image data set, and the method of processing the high resolution image data set includes the steps of:
s1011, dividing the high-resolution image data set into a training set, a verification set and a test set;
s1012, down-sampling the high resolution image in the high resolution image dataset to generate a corresponding low resolution image;
and S1013, correspondingly cutting the high-resolution image and the low-resolution image into paired sub-image blocks according to a preset image size, and selecting a specific sub-image block to perform data enhancement through random overturning and rotation, thereby finally obtaining the training set.
For the single-image super-resolution reconstruction model, the loss function corresponding to the single-image super-resolution reconstruction model is represented as:
wherein,
the corresponding loss value of the loss function is represented,
denotes an operation of taking the minimum value of the loss function,
represents the maximum number of training images and,
the sequence number of the training image is represented,
denotes the first
A high resolution original image in a training image,
denotes the first
A low resolution original image in a training image,
is shown as
The result of the bicubic interpolation corresponding to a training image,
is shown as
Parameters in a training image at low resolution
The corresponding model is generated according to the model,
a value representing the weight of the balance weight,
is shown as
A high resolution gradient map in a training image,
is shown as
A low resolution gradient map in a training image,
is shown as
Parameters under high-resolution gradient map in training image
And generating a corresponding model.
S102, extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch.
Specifically, the image reconstruction branch comprises a first shallow feature extraction module, a feature thinning module and an image upsampling module which are sequentially connected.
The first shallow layer feature extraction module consists of two convolution layers and is used for extracting and obtaining the first shallow layer image features
. To further extract features, a feature refinement module includes
A plurality of residual dense connecting blocks which are connected in sequence and used for extracting and obtaining the multi-level image characteristics
，
。
Wherein,
representing the maximum number of residual dense connected blocks, each comprising two elementary units. It should be noted that the current residual dense connection block is used forAt the multi-level image features obtained by extraction
Before inputting to the next residual dense connection block, updating is carried out through bidirectional feature fusion.
Meanwhile, the image up-sampling module is used for converting the image into a digital image
In a residual dense connecting block
The outputs of the elementary cells are stitched in channel dimensions as input to the image upsampling module. Furthermore, unlike the feature refinement module, the image upsampling module only performs gradient-guided image feature enhancement.
S103, extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch.
The gradient prediction branch comprises a second shallow layer feature extraction module, a gradient refinement module and a gradient reconstruction module.
The second shallow layer feature extraction module consists of two convolution layers and is used for extracting and obtaining the second shallow layer image features
. To further extract features, the gradient refinement module includes
A plurality of sequentially connected residual blocks for extracting to obtain multi-level gradient characteristics
，
。
Wherein,
representing the maximum number of residual blocks, each residual block comprising two elementary units. It should be noted that the current residual block is used for multi-level gradient features obtained by extraction
Before inputting to the next residual block, updating is carried out through bidirectional feature fusion. The gradient reconstruction module is used for carrying out multi-level gradient feature after refinement
As input, a high resolution gradient map is then output. Furthermore, the low resolution gradient image of the gradient prediction branch input is generated by a Sobel operator.
And S104, performing fusion guidance by combining a trilateral feature filtering module based on the image reconstruction branch and the gradient prediction branch to realize self-adaptive adjustment of convolution kernels of a target domain, and reconstructing a high-resolution image from coarse to fine.
It should be noted that bidirectional feature fusion represents image features at multiple levels
And multi-level gradient feature
And the combined trilateral feature filtering module performs feature fusion alternately to realize image-guided gradient feature enhancement and gradient-guided image feature enhancement. Since the quality of the gradient features of the low-resolution domain is low, the image features are firstly used for refining the corresponding gradient features, and then the direction of feature fusion is changed alternately along a preset sequence, and in the process, the image features and the gradient features are strengthened from coarse to fine.
In addition, for the above combined trilateral feature filtering module, in the combined trilateral feature filtering module, the expression corresponding to the fusion feature of the target feature obtained by output is as follows:
wherein,
indicating pre-fusion channels
In space of
The above-mentioned features of the present invention,
represents the weights of the convolution kernels corresponding to the features of the target domain,
representing the convolution kernel weights corresponding to the guiding domain features,
a local window is represented that is a window of a scene,
the representation being located in a local window
A fused feature of the central target feature.
Further, in the combined trilateral feature filtering module, the target domain feature is used for carrying out convolution operation
In that
Dimensionally in the order of
A traversal is made for the window size to obtain a plurality of three-dimensional image blocks,
wherein
respectively representing the number of channels, height and width.
Thus, the processed target domain feature size is
，
In order to perform the batch processing number,
the representation of the real number field is performed,
representing a coordinate dimension of
A high-dimensional tensor space over a real domain of (a).
At the same time, the learned combined trilateral filtering kernel size is
. A Joint Trilateral Filtering (JTF) convolution operation in
Is dimensionally passed through
And completing inner product independently in dimensions.
In this embodiment, in the above combined trilateral feature filtering module, a kernel function of an image feature and a kernel function of a gradient feature are learned through two subnetworks with the same structure, respectively;
sub-network learning from image features
Kernel function to image features
The mapping of (a), wherein,
，
；
sub-network learning from gradient features
Kernel function to gradient features
The mapping of (a), wherein,
，
；
by kernel functions of image features
Kernel function of gradient feature
The corresponding elements are multiplied to obtain a combined trilateral feature filtering kernel
；
Wherein,
represents a coordinate dimension of
A high-dimensional tensor space over a real number domain of,
representing a coordinate dimension of
A high-dimensional tensor space over a real number domain. Target Domain characteristics (dimension of
) Each position of (A) has
Generating individual kernel weights corresponding to kernel sizes
. Inspired by Involution, channels are divided into multiple groups, and the groups are internal
The channels share spatially different cores to further compress the model, i.e. the number of channels of a core
. Hence, a joint trilateral feature filtering kernel
Is measured by
Is updated to
；
After updating the kernel function of the image feature and the kernel function of the gradient feature, in the combined trilateral feature filtering module, the expression corresponding to the fusion feature of the target feature is updated as follows:
wherein,
representing the spatial position of a target domain in a set of channels
The kernel function learned in the above-mentioned manner,
representing the spatial position of the guiding domain in the set of channels
The kernel function learned above.
The number of residual dense connection blocks and residual blocks in the feature refinement module and the gradient refinement module of the present invention is set to 5, the kernel size of all bottleneck layers is 1 × 1, and the kernel size of other convolutional layers is set to 3 × 3. The number of channels of the first convolutional layer of the shallow feature extraction modules of the image reconstruction branch and the gradient prediction branch is set to be 128, and the number of channels of other features is 64. Number of channels for design of combined trilateral filtering module
Number of shared channels in a sum group
Set to 32 and 2, respectively.
And S105, when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image.
In this step, after the single-image super-resolution reconstruction model iterates to converge, the residual error output by the single-image super-resolution reconstruction model is added to the bicubic up-sampled image of the initial low-resolution image to obtain the predicted super-resolution reconstructed image.
The invention provides an image super-resolution reconstruction method based on combined trilateral feature filtering, which comprises the steps of firstly, extending a filtering definition domain from a pixel domain to a high-dimensional feature domain, and simulating a combined trilateral feature filter defined on an image feature domain and a gradient feature domain by constructing a neural network structure; by sensing corresponding features on an image feature domain and a gradient feature domain and combining with the trilateral feature filtering module, the convolution kernel of the features of the target domain is adaptively adjusted, and the generalization performance of the feature fusion module is improved;
and secondly, based on a combined trilateral feature filtering module, alternately taking the image features and the gradient features as guide domain features, deeply excavating mutual guide effect between the gradient domain and the image domain, and realizing bidirectional fusion of the feature domains. Compared with the most advanced method at present, the image super-resolution reconstruction method based on the combined trilateral feature filtering achieves the best effect of subjective evaluation and objective evaluation.
Referring to fig. 3, the present invention further provides an image super-resolution reconstruction system based on joint trilateral feature filtering, wherein the system includes:
a model building module to:
the method comprises the steps of downsampling a high-resolution image to obtain a low-resolution image, constructing a single-image super-resolution reconstruction model, and inputting the low-resolution image into the single-image super-resolution reconstruction model, wherein the single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, the image reconstruction branch at least comprises a feature thinning module, and the gradient prediction branch at least comprises a gradient thinning module;
a first extraction module to:
extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch;
a second extraction module to:
extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch;
a feature fusion module to:
based on the image reconstruction branch and the gradient prediction branch, fusion guidance is carried out through a combined trilateral feature filtering module so as to realize self-adaptive adjustment of convolution kernels of a target domain, and high-resolution image reconstruction is carried out from coarse to fine;
an iterative convergence module to:
and when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that various changes and modifications can be made by those skilled in the art without departing from the spirit of the invention, and these changes and modifications are all within the scope of the invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (5)
1. An image super-resolution reconstruction method based on combined trilateral feature filtering is characterized by comprising the following steps:
the method comprises the steps of firstly, down-sampling a high-resolution image to obtain a low-resolution image, constructing a single-image super-resolution reconstruction model, and inputting the low-resolution image into the single-image super-resolution reconstruction model, wherein the single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, the image reconstruction branch at least comprises a feature refining module, and the gradient prediction branch at least comprises a gradient refining module;
secondly, extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch;
step three, extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch;
performing fusion guidance by combining a trilateral feature filtering module based on the image reconstruction branch and the gradient prediction branch to realize self-adaptive adjustment of a convolution kernel of a target domain, and reconstructing a high-resolution image from coarse to fine;
fifthly, when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image;
the image reconstruction branch comprises a first shallow layer feature extraction module, a feature thinning module and an image up-sampling module which are sequentially connected;
the first shallow feature extraction module consists of two convolution layers and is used for extracting and obtaining a first shallow graphImage characteristics
；
The feature refinement module comprises
A plurality of residual dense connecting blocks which are connected in sequence and used for extracting and obtaining multi-level image characteristics
，
Wherein
representing the maximum number of residual dense connecting blocks, each of which comprises two basic units, wherein the current residual dense connecting block is used for extracting the obtained multi-level image features
Updating through bidirectional feature fusion before inputting to the next residual dense connecting block;
the image up-sampling module is used for converting the image into a digital image
In a residual dense connecting block
The outputs of the basic units are spliced in the channel dimension to be used as the input of the image up-sampling module;
the gradient prediction branch comprises a second shallow layer feature extraction module, a gradient refinement module and a gradient reconstruction module;
the second shallow feature extractorThe fetching module consists of two convolution layers and is used for extracting and obtaining the characteristics of the second shallow image
；
The gradient refining module comprises
A plurality of sequentially connected residual blocks for extracting multi-level gradient features
，
Wherein
representing the maximum number of residual blocks, each of which comprises two elementary units, wherein the current residual block is used for multi-level gradient features to be extracted
Updating through bidirectional feature fusion before inputting to the next residual block;
the gradient reconstruction module is used for carrying out multi-level gradient feature after refinement
As input, then outputting a high resolution gradient map;
the loss function corresponding to the single-image super-resolution reconstruction model is expressed as follows:
wherein,
the corresponding loss value of the loss function is represented,
representing the operation of taking the minimum value of the loss function,
represents the maximum number of training images and,
the sequence number of the training image is represented,
is shown as
A high resolution original image in a training image,
is shown as
A low resolution original image in a training image,
is shown as
The result of the bicubic interpolation corresponding to a training image,
denotes the first
Stretch training pictureParameters in low resolution images
The corresponding model is generated according to the model,
a value representing the weight of the balance weight,
is shown as
A high resolution gradient map in a training image,
is shown as
A low resolution gradient map in a training image,
is shown as
Parameters under high-resolution gradient map in training image
Generating a corresponding model;
bi-directional feature fusion representing image features at multiple levels
And multi-level gradient feature
By the combined trilateral feature filteringThe modules alternate to implement image-guided gradient feature enhancement and gradient-guided image feature enhancement;
in the combined trilateral feature filtering module, the expression corresponding to the fusion feature of the target feature obtained by output is as follows:
wherein,
indicating pre-fusion channels
In space of
The above-mentioned features of the present invention,
represents the weights of the convolution kernels corresponding to the features of the target domain,
representing the convolution kernel weights corresponding to the guiding domain features,
a local window is represented that is a window of a scene,
the representation being located in a local window
A fused feature of the central target feature.
2. The image super-resolution reconstruction method based on combined trilateral feature filtering according to claim 1, wherein a plurality of the high-resolution images form a high-resolution image data set, and the method for processing the high-resolution image data set comprises the following steps:
dividing the high-resolution image data set into a training set, a verification set and a test set;
down-sampling a high resolution image in the high resolution image data set to generate a corresponding low resolution image;
and correspondingly cutting the high-resolution image and the low-resolution image into paired sub image blocks according to a preset image size, selecting a specific sub image block, and performing random overturning and rotation to perform data enhancement, thereby finally obtaining the training set.
3. The image super-resolution reconstruction method based on the joint trilateral feature filtering of claim 1, wherein in the joint trilateral feature filtering module, target domain features are used for convolution operation
In that
Dimensionally in the order of
A traversal is made for the window size to obtain a plurality of three-dimensional image blocks,
wherein
respectively representing the number, height and width of channels;
wherein the processed target domain has a characteristic size of
，
In order to perform the batch processing number,
the representation of the real number field is performed,
representing a coordinate dimension of
A high-dimensional tensor space over a real domain of (a).
4. The image super-resolution reconstruction method based on joint trilateral feature filtering of claim 3, wherein in the joint trilateral feature filtering module, a kernel function of an image feature and a kernel function of a gradient feature are learned through two sub-networks with the same structure respectively;
sub-network learning from image features
Kernel function to image features
The mapping of (a), wherein,
，
；
sub-network learning from gradient features
Kernel function to gradient features
The mapping of (a), wherein,
，
；
by kernel functions characterizing images
Kernel function of gradient feature
The corresponding elements are multiplied to obtain a combined trilateral feature filtering kernel
；
Wherein,
representing a coordinate dimension of
A high-dimensional tensor space over a real number domain of,
represents a coordinate dimension of
A high-dimensional tensor space over a real domain of (a).
5. A combined trilateral feature filtering-based image super-resolution reconstruction system, which applies the combined trilateral feature filtering-based image super-resolution reconstruction method of any one of the above claims 1 to 4, the system comprising:
a model building module to:
the method comprises the steps of down-sampling a high-resolution image to obtain a low-resolution image, constructing a single-image super-resolution reconstruction model, and inputting the low-resolution image into the single-image super-resolution reconstruction model, wherein the single-image super-resolution reconstruction model comprises an image reconstruction branch and a gradient prediction branch, the image reconstruction branch at least comprises a feature refining module, and the gradient prediction branch at least comprises a gradient refining module;
a first extraction module to:
extracting and obtaining multi-level image features through the feature thinning module in the image reconstruction branch;
a second extraction module to:
extracting and obtaining multi-level gradient characteristics through the gradient refining module in the gradient prediction branch;
a feature fusion module to:
based on the image reconstruction branch and the gradient prediction branch, fusion guidance is carried out through a combined trilateral feature filtering module so as to realize self-adaptive adjustment of convolution kernels of a target domain, and high-resolution image reconstruction is carried out from coarse to fine;
an iterative convergence module to:
and when the single-image super-resolution reconstruction model is iterated to be converged, carrying out forward inference on the single-image super-resolution reconstruction model to finally obtain a super-resolution reconstructed image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210924242.4A CN114972043B (en) | 2022-08-03 | 2022-08-03 | Image super-resolution reconstruction method and system based on combined trilateral feature filtering |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210924242.4A CN114972043B (en) | 2022-08-03 | 2022-08-03 | Image super-resolution reconstruction method and system based on combined trilateral feature filtering |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114972043A CN114972043A (en) | 2022-08-30 |
| CN114972043B true CN114972043B (en) | 2022-10-25 |
Family
ID=82969635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210924242.4A Active CN114972043B (en) | 2022-08-03 | 2022-08-03 | Image super-resolution reconstruction method and system based on combined trilateral feature filtering |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114972043B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116228786B (en) * | 2023-05-10 | 2023-08-08 | 青岛市中心医院 | Prostate MRI image enhancement segmentation method, device, electronic equipment and storage medium |
| CN116402692B (en) * | 2023-06-07 | 2023-08-18 | 江西财经大学 | Depth map super-resolution reconstruction method and system based on asymmetric cross attention |
| CN116523759B (en) * | 2023-07-04 | 2023-09-05 | 江西财经大学 | Image super-resolution reconstruction method and system based on frequency decomposition and restarting mechanism |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017075768A1 (en) * | 2015-11-04 | 2017-05-11 | 北京大学深圳研究生院 | Super-resolution image reconstruction method and device based on dictionary matching |
| CN112200720A (en) * | 2020-09-29 | 2021-01-08 | 中科方寸知微(南京)科技有限公司 | Super-resolution image reconstruction method and system based on filter fusion |
| EP3816928A1 (en) * | 2019-11-04 | 2021-05-05 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Image super-resolution reconstruction method, image super-resolution reconstruction apparatus, and computer-readable storage medium |
| WO2021258529A1 (en) * | 2020-06-22 | 2021-12-30 | 北京大学深圳研究生院 | Image resolution reduction and restoration method, device, and readable storage medium |
| CN113920014A (en) * | 2021-10-25 | 2022-01-11 | 江西财经大学 | Neural-networking-based combined trilateral filter depth map super-resolution reconstruction method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9064476B2 (en) * | 2008-10-04 | 2015-06-23 | Microsoft Technology Licensing, Llc | Image super-resolution using gradient profile prior |
| CN109345449B (en) * | 2018-07-17 | 2020-11-10 | 西安交通大学 | Image super-resolution and non-uniform blur removing method based on fusion network |
| CN111192200A (en) * | 2020-01-02 | 2020-05-22 | 南京邮电大学 | Image super-resolution reconstruction method based on fusion attention mechanism residual error network |
-
2022
- 2022-08-03 CN CN202210924242.4A patent/CN114972043B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017075768A1 (en) * | 2015-11-04 | 2017-05-11 | 北京大学深圳研究生院 | Super-resolution image reconstruction method and device based on dictionary matching |
| EP3816928A1 (en) * | 2019-11-04 | 2021-05-05 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Image super-resolution reconstruction method, image super-resolution reconstruction apparatus, and computer-readable storage medium |
| WO2021258529A1 (en) * | 2020-06-22 | 2021-12-30 | 北京大学深圳研究生院 | Image resolution reduction and restoration method, device, and readable storage medium |
| CN112200720A (en) * | 2020-09-29 | 2021-01-08 | 中科方寸知微(南京)科技有限公司 | Super-resolution image reconstruction method and system based on filter fusion |
| CN113920014A (en) * | 2021-10-25 | 2022-01-11 | 江西财经大学 | Neural-networking-based combined trilateral filter depth map super-resolution reconstruction method |
Non-Patent Citations (3)
| Title |
|---|
| Image Super-Resolution reconstruction based on adaptive gradient field sharpening;T Li 等;《Digital Signal Processing (DSP)》;20130731;全文 * |
| 利用双通道卷积神经网络的图像超分辨率算法;徐冉等;《中国图象图形学报》;20160516(第05期);全文 * |
| 基于像素及梯度域双层深度卷积神经网络的页岩图像超分辨率重建;占文枢等;《科学技术与工程》;20180128(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114972043A (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114972043B (en) | Image super-resolution reconstruction method and system based on combined trilateral feature filtering | |
| CN109389556B (en) | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device | |
| CN111311518B (en) | Image denoising method and device based on multi-scale mixed attention residual error network | |
| CN110136066B (en) | Video-oriented super-resolution method, device, equipment and storage medium | |
| CN114549731B (en) | Method and device for generating visual angle image, electronic equipment and storage medium | |
| CN110136062B (en) | Super-resolution reconstruction method combining semantic segmentation | |
| Cheng et al. | Zero-shot image super-resolution with depth guided internal degradation learning | |
| Singh et al. | Survey on single image based super-resolution—implementation challenges and solutions | |
| CN109447897B (en) | Real scene image synthesis method and system | |
| Couturier et al. | Image denoising using a deep encoder-decoder network with skip connections | |
| CN113837946B (en) | Lightweight image super-resolution reconstruction method based on progressive distillation network | |
| CN114049420B (en) | Model training method, image rendering method, device and electronic equipment | |
| CN111179196B (en) | Multi-resolution depth network image highlight removing method based on divide-and-conquer | |
| CN113096239B (en) | Three-dimensional point cloud reconstruction method based on deep learning | |
| Bastanfard et al. | Toward image super-resolution based on local regression and nonlocal means | |
| CN116343052B (en) | Attention and multiscale-based dual-temporal remote sensing image change detection network | |
| CN112200720A (en) | Super-resolution image reconstruction method and system based on filter fusion | |
| CN112184587A (en) | Edge data enhancement model, and efficient edge data enhancement method and system based on model | |
| Wei et al. | A-ESRGAN: Training real-world blind super-resolution with attention U-Net Discriminators | |
| CN112184547A (en) | Super-resolution method of infrared image and computer readable storage medium | |
| CN113920014A (en) | Neural-networking-based combined trilateral filter depth map super-resolution reconstruction method | |
| CN113506305B (en) | Image enhancement method, semantic segmentation method and device for three-dimensional point cloud data | |
| CN113902611A (en) | Image beautifying processing method and device, storage medium and electronic equipment | |
| Sun et al. | A rapid and accurate infrared image super-resolution method based on zoom mechanism | |
| Zheng et al. | Joint residual pyramid for joint image super-resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |