CN114708178A - Remote sensing image fusion method based on guided filtering and sparse representation - Google Patents
Remote sensing image fusion method based on guided filtering and sparse representation Download PDFInfo
- Publication number
- CN114708178A CN114708178A CN202210356388.3A CN202210356388A CN114708178A CN 114708178 A CN114708178 A CN 114708178A CN 202210356388 A CN202210356388 A CN 202210356388A CN 114708178 A CN114708178 A CN 114708178A
- Authority
- CN
- China
- Prior art keywords
- image
- panchromatic
- multispectral
- low
- dictionary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001914 filtration Methods 0.000 title claims abstract description 62
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 19
- 230000004927 fusion Effects 0.000 claims abstract description 41
- 238000000034 method Methods 0.000 claims abstract description 26
- 238000012549 training Methods 0.000 claims abstract description 25
- 230000009466 transformation Effects 0.000 claims abstract description 17
- 238000003384 imaging method Methods 0.000 claims abstract description 11
- 238000006243 chemical reaction Methods 0.000 claims description 32
- 239000011159 matrix material Substances 0.000 claims description 26
- 238000000605 extraction Methods 0.000 claims description 18
- 230000003595 spectral effect Effects 0.000 claims description 18
- 239000013598 vector Substances 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000008447 perception Effects 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 6
- 238000005316 response function Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000003044 adaptive effect Effects 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2136—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on sparsity criteria, e.g. with an overcomplete basis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/28—Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10036—Multispectral image; Hyperspectral image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10041—Panchromatic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a remote sensing image fusion method based on guided filtering and sparse representation, and belongs to the technical field of remote sensing image processing. The implementation method of the invention comprises the following steps: carrying out nonlinear HSV (hue, saturation and value) transformation on the multispectral image to reduce image aliasing and reduce calculated amount; extracting the detail information of the panchromatic image and the multispectral image for multiple times by using a guided filtering means to obtain a detail image for effectively extracting structural feature information; fusing low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing energy of the low-frequency information of the image; according to the extracted high-frequency component structural feature information and low-frequency component information, a new brightness image V' is obtained through fusion, namely remote sensing image fusion is achieved based on guided filtering and sparse representation, and the resolution of the multispectral image is remarkably improved; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image. The invention utilizes the panchromatic image to directly reconstruct the low-frequency dictionary, thereby reducing the uncertainty of the result caused by the lack of an ideal image training dictionary.
Description
Technical Field
The invention relates to a remote sensing image fusion method based on guided filtering and sparse representation, and belongs to the technical field of remote sensing image processing.
Background
In the field of remote sensing, due to the limitation of related technologies of sensors, a remote sensing satellite is difficult to acquire remote sensing images with high spatial resolution and high spectral resolution simultaneously. Panchromatic images (PAN) acquired by the optical remote sensing satellite have high spatial resolution, can reflect ground feature detail information, but lack ground feature spectrum and color information, multispectral images contain spectral characteristics of ground features to different wave bands, can provide effective color information for identification and classification of ground feature targets, but the spatial resolution is lower than that of panchromatic images, so that the multispectral images need to be subjected to super-resolution fusion processing by utilizing the panchromatic images to realize improvement of the resolution of the multispectral images, and the classification and identification capability of ground feature targets based on high-resolution multispectral ground remote sensing images is improved
The fusion of multispectral images and panchromatic images can be regarded as multi-source image fusion, and generally three methods are adopted: 1) methods that perform fusion directly on image pixels or in other transform domains; this is achieved by replacing the intensity information or first principal component of the multispectral image with a panchromatic image. 2) A multi-scale decomposition based approach; this is achieved by injecting spatial details obtained by multiresolution decomposition of the panchromatic image into the multispectral phase of resampling. 3) Sparse representation-based methods; it is achieved by solving the NP method of super-resolution image reconstruction. The disadvantages of method 1) are: local differences between the spectral responses of the panchromatic image and the multispectral image are not taken into account and may therefore cause color distortions in the fused image. The disadvantages of method 2) are: since this method involves signal filtering and pixel level fusion, the fusion result is prone to aliasing effects, which can cause shifting and blurring of contours or textures. Although various space results are extracted in different scales by multi-scale transformation, low-frequency components cannot be sparsely represented, and because the low-frequency components contain a large amount of energy, if the low-frequency coefficients are directly selected by means of average or maximum values, the fusion effect is reduced. The disadvantages of method 3) are: firstly, the high-resolution multispectral image is lacked, and the dictionary is difficult to establish. Secondly, the limitation of the number of atoms in the dictionary makes the small-scale detail information (detail information) difficult to reconstruct.
In summary, it is necessary to design a multispectral image super-resolution algorithm with short running time, high spectral fidelity, and effectively improved spatial resolution. How to better extract the space details of the panchromatic image, and reduce the phenomena of color aliasing and the like caused by different spectral responses of the sensor and the like in the injection process of the space details, and the key problem which is worthy of being solved is to generate the remote sensing image which is beneficial to the human to interpret and classify.
Disclosure of Invention
The invention mainly aims to provide a remote sensing image fusion method based on guided filtering and sparse representation, which is used for carrying out nonlinear HSV (hexagonal cone) transformation on a multispectral image so as to reduce image aliasing and reduce the calculated amount; extracting the detail information of the panchromatic image and the multispectral image for multiple times by using a guided filtering means to obtain a detail image for effectively extracting structural feature information; fusing low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing energy of the low-frequency information of the image; according to the extracted high-frequency component structural feature information and low-frequency component information, fusing to obtain a brightness image V', namely realizing remote sensing image fusion based on guided filtering and sparse representation, and remarkably improving the resolution of the multispectral image; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
The purpose of the invention is realized by the following technical scheme.
The invention discloses a remote sensing image fusion method based on guided filtering and sparse representation. Secondly, edge detail information of the panchromatic image and the multispectral image brightness component V is extracted for multiple times by using a guide filtering method, and low-frequency information of the panchromatic image and the multispectral image extracted by using the guide filtering is fused through sparse representation. And finally, fusing the extracted high-frequency component structural feature information and the extracted low-frequency component information to obtain a brightness image V ', and performing HSV (hue, saturation and value) spatial inverse transformation on the brightness image V', so as to realize super-resolution imaging of the multispectral image. Compared with algorithms such as HIS and the like, the method has short operation time and high spectral fidelity, and can effectively improve the spatial resolution.
The invention discloses a remote sensing image fusion method based on guided filtering and sparse representation, which comprises the following steps:
firstly, image interpolation operation is carried out on the multispectral image (remote sensing image) to expand the multispectral image to the same scale as the panchromatic image (remote sensing image), so that subsequent fusion operation between the multispectral image and the panchromatic image is facilitated.
The image interpolation operation comprises nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. Because the nearest neighbor interpolation does not change the radiometric values of the image pixels, and when the data are aligned even, the nearest neighbor interpolation can compensate for 1.5 pixel displacement between the interpolated multispectral image and panchromatic image, reduce systematic offset between the expanded multispectral image and panchromatic image, and help to ensure the quality of the final fusion result. Preferably, a nearest neighbor interpolation operation is selected.
And secondly, performing nonlinear HSV conversion on the multispectral image to obtain a luminance image after the nonlinear HSV conversion, reducing image aliasing caused by subsequent operation, reducing the calculation amount of the subsequent operation and improving the fusion efficiency of the remote sensing image.
Since the subjective perception of the luminance component V of the multispectral image is strong compared to the subjective perception of the color component, the color component is downsampled on the digital representation of the picture and video without any degradation of the luminance component. According to the HSV conversion model shown in the formulas (1), (2) and (3), the multispectral image is subjected to nonlinear HSV conversion, compared with other conversions, the fused image can be sharper, image aliasing caused by subsequent operation is reduced, the calculation amount of the subsequent operation is reduced, the remote sensing image fusion efficiency is improved, and compared with the conversions such as HIS, the nonlinear HSV conversion introduces more noise.
V＝max(R,G,B) (1)
Where R denotes the red channel of the image, G denotes the green channel of the image, B denotes the blue channel of the image, H denotes hue, S denotes saturation, and V denotes brightness.
And step three, performing histogram matching between the V component extracted from the multispectral image and the panchromatic image in the step two, and reducing spectral distortion generated by a fusion result due to inconsistency of spectral response functions of the multispectral image and the panchromatic image.
And (4) performing histogram matching between the V component extracted from the multispectral image and the panchromatic image according to a formula (4), and reducing spectral distortion generated by a fusion result due to inconsistency of spectral response functions of the multispectral image and the panchromatic image.
Wherein PAN is a full color image, μPANAnd σPANIs the mean and variance, μ, of the panchromatic imageVAnd σVIs the mean and variance of the luminance components, and P is the full-color image after histogram matching.
And step four, performing guided filtering on the brightness component of the multispectral image by taking the panchromatic image (remote sensing image) as a guide image by utilizing the structural transfer characteristic of the guided filtering, and removing the noise generated by the multispectral image in the HSV conversion stage.
And (3) performing guide filtering on the brightness components of the multispectral image according to formulas (5), (6) and (7) by taking the panchromatic image subjected to histogram matching in the step three as a guide image according to the structural transfer characteristic of the guide filtering on the multispectral image subjected to nonlinear HSV conversion in the step two, and removing noise generated by the multispectral image in the HSV conversion stage.
q1＝a1I+b1 (5)
Where p is the input image, I is the guide image, ε2Is the regularization parameter, q is the output image, a1And b1Is a hyperparameter of formula (5).
And step five, extracting edge detail information of the panchromatic image and the multispectral image luminance component V by utilizing the edge-preserving smoothness characteristic of the guide filtering and using a guide filtering means, and fusing the edge detail information of the panchromatic image and the multispectral image luminance component V, wherein the extraction process does not depend on the size of a sliding block window, and the extraction speed is improved.
And (3) extracting edge detail information of the panchromatic image and the multispectral image brightness component V according to the guiding filtering formulas shown in formulas (8), (9) and (10) by utilizing the edge-preserving smoothing characteristic of guiding filtering on the multispectral image after nonlinear HSV conversion in the second step and the panchromatic image after histogram matching in the third step, and fusing the edge detail information of the panchromatic image and the multispectral image brightness component V, wherein the extraction process does not depend on the size of a sliding block window, and the extraction speed is improved.
q2＝a2p+b2 (8)
Where p is the input image, ε2Is the regularization parameter, q is the output image, a2And b2Is a hyperparameter of formula (8).
Step six: and repeating the iteration step five aiming at the multispectral image brightness component and the panchromatic image guided and filtered high-frequency component, and extracting the detail information of the panchromatic image and the multispectral image for multiple times until the high-frequency component structural feature information of the panchromatic image is effectively extracted.
And seventhly, aiming at the multispectral image brightness component and the low-frequency component of the panchromatic image after the panchromatic image is guided and filtered, fusing the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and effectively extracting the low-frequency component information of the panchromatic image and the multispectral image by fully utilizing the energy of the low-frequency information of the image.
The sparse representation method is implemented as follows: acquiring a low-frequency information training image, converting the image into a feature matrix, solving a dictionary through a dictionary learning algorithm, solving the sparse coefficient of the panchromatic image and the brightness component under the dictionary, and fusing and reconstructing the low-frequency component according to the sparse coefficient.
And 7.1, selecting a plurality of panchromatic images containing ground object information along the subway as training images, performing low-pass filtering on the training images, and performing down-sampling on the training images to the size of the multi-spectral image to obtain corresponding low-resolution training images. In order to make the dimensions before and after fusion the low resolution training image down-sampled to the size of the multi-spectral image is up-sampled to the size of the panchromatic image to obtain the high resolution low frequency information training image.
Step 7.2, dividing the image of the low-frequency information obtained in the step 7.1 into images by using a sliding window
The image blocks are changed into column vectors of n × 1 size, and the obtained column vectors are combined into a feature matrix.
And 7.3, solving the dictionary by using a formula (11) through a dictionary learning algorithm.
Wherein β ═ α1，α2，...，αm) Is the sparse coefficient of the image, DLFor training the resulting low resolution dictionary, XLFor the image matrix obtained in step 7.2, T is the sparsity of the sparse coefficient (the number of non-zero values in the sparse coefficient).
The dictionary learning algorithm generally includes 2 classes, one class is composed of a group of parameters and a set of selected functions containing parameters, which are used to approximate the expression signal. The dictionary does not need to store the whole dictionary, only the related parameter information needs to be stored, so that the storage capacity is greatly reduced, but the related dictionary is irrelevant to the original signal, so that the method has no adaptability; another dictionary learning method is an adaptive dictionary obtained by training and learning according to the characteristics of signals or images. Including K-SVD algorithms, MOD dictionary learning algorithms, etc. The K-SVD algorithm can be better suitable for various images with different characteristics, and has the characteristic of strong adaptability.
Preferably, a K-SVD algorithm learning dictionary is selected. The step 7.3 is realized by the following concrete method:
7.3.1, dictionary initialization: and randomly selecting K column vectors from the feature matrix obtained in the step 7.2 as atoms of the initial dictionary.
7.3.2, sparse coding: and (4) carrying out sparse coding by using a formula (12) to obtain a sparse coefficient.
x＝Dα (12)
Where D represents a dictionary obtained from 7.3.1 and x is the feature matrix of the image;
7.3.3, dictionary updating: and updating the dictionary D column by utilizing an OMP algorithm, repeating the step 7.3.2, meanwhile, calculating an error matrix by utilizing the formula (13), and stopping updating when a specified error is converged to obtain the dictionary D capable of representing low-frequency component information.
Wherein EkRepresenting the error matrix, djRepresentation dictionary D (D)1，d2....dk) Column j of atoms.
Step 7.4, aiming at the multispectral image brightness component and the panchromatic image guided and filtered low-frequency component, converting the multispectral image brightness component and the panchromatic image into a characteristic matrix by utilizing the step 7.2, solving the formula (14) to obtain the multispectral image brightness component and the panchromatic image sparse coefficient,
wherein DLIs the low resolution dictionary obtained in step 7.3, ε is the tolerance error, yLFIs the feature matrix of the image and alpha is the sparse coefficient to be solved.
And 7.5, reconstructing the multispectral image brightness component and the panchromatic image sparse coefficient obtained in the step 7.4 according to a fusion rule shown in the formula (15) that the sparse coefficient absolute value is the maximum to obtain a new sparse coefficient. And reconstructing a new characteristic matrix according to the formula (12) to realize the fusion of the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing the low-frequency characteristic information of the multispectral image and the panchromatic image to realize the effective extraction of the low-frequency component information of the panchromatic image and the multispectral image.
α＝max(α1,α2) (15)
Wherein alpha is1And alpha2Representing the panchromatic image and multispectral image luminance components obtained in step 7.4 in dictionary DLAnd obtaining the sparse coefficient.
Step eight: according to the high-frequency component structural feature information extracted in the sixth step and the low-frequency component information extracted in the seventh step, a brightness image V' is obtained through fusion, namely remote sensing image fusion is achieved based on guide filtering and sparse representation, and the resolution ratio of the multispectral image is remarkably improved; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
And fusing the high-frequency component structural feature information extracted in the sixth step and the low-frequency component information extracted in the seventh step to obtain a brightness image V ', and performing HSV (hue, saturation, value) spatial inverse transformation on the brightness image V' according to formulas (16), (17), (18), (19), (20) and (21) to realize super-resolution imaging of the multispectral image.
h＝floor(H/60) (16)
f＝H/60-h (17)
p＝V*(1-S) (18)
q＝V*(1-f*S) (19)
t＝V*(1-(1-f)*S (20)
Wherein: floor denotes rounding down.
Has the advantages that:
1. the invention discloses a remote sensing image fusion method based on guided filtering and sparse representation, which is used for carrying out nonlinear HSV (hue, saturation, value) transformation on a multispectral image to reduce image aliasing and reduce the calculated amount; extracting the detail information of the panchromatic image and the multispectral image for multiple times by using a guided filtering means to obtain a detail image for effectively extracting structural feature information; the low-frequency information of the panchromatic image and the multispectral image is fused through sparse representation, the energy of the low-frequency information of the image is fully utilized, and the maintenance of spectral details is facilitated. (ii) a According to the extracted high-frequency component structural feature information and low-frequency component information, fusing to obtain a brightness image V', namely realizing remote sensing image fusion based on guided filtering and sparse representation, and remarkably improving the resolution of the multispectral image; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
2. The invention discloses a remote sensing image fusion method based on guided filtering and sparse representation, which utilizes a method for directly reconstructing a low-frequency dictionary by using a panchromatic image, avoids a large amount of guessing assumptions caused by lack of a high-resolution multispectral image, and reduces the uncertainty of a result.
Drawings
Fig. 1 is a flow chart of a remote sensing image fusion method based on guided filtering and sparse representation disclosed by the invention.
FIG. 2 is a diagram of the V component extracted by the non-linear HSV conversion of the multi-spectral image according to the present invention.
Fig. 3 is an image after histogram transformation of a full-color image according to the present invention.
Fig. 4 shows the low-frequency and high-frequency images obtained by applying the panchromatic image to the self-guided filtering according to the present invention.
FIG. 5 is a low resolution dictionary built by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to specific embodiments and the accompanying drawings.
As shown in fig. 1, the flow chart of the remote sensing image fusion method based on guided filtering and sparse representation disclosed in this embodiment specifically includes the following steps:
firstly, performing image nearest neighbor interpolation operation on the multispectral image (remote sensing image) to expand the multispectral image to the same scale as the panchromatic image (remote sensing image), so that the fusion operation between the multispectral image and the panchromatic image is facilitated.
And secondly, performing nonlinear HSV conversion on the multispectral image to obtain a luminance image after the nonlinear HSV conversion as shown in figure 2, reducing image aliasing caused by subsequent operation, reducing the calculation amount of the subsequent operation and improving the fusion efficiency of the remote sensing image.
Since the subjective perception of the luminance component V of the multispectral image is strong compared to the subjective perception of the color component, the color component is downsampled on the digital representation of the picture and video without any degradation of the luminance component. According to the HSV conversion model shown in the formulas (1), (2) and (3), the multispectral image is subjected to nonlinear HSV conversion, compared with other conversions, the fused image can be sharper, image aliasing brought by subsequent operation is reduced, the calculation amount of the subsequent operation is reduced, and the remote sensing image fusion efficiency is improved.
V＝max(R,G,B) (1)
Where R denotes the red channel of the image, G denotes the green channel of the image, B denotes the blue channel of the image, H denotes hue, S denotes saturation, and V denotes brightness.
And step three, performing histogram matching between the V component extracted from the multispectral image in the step two and the panchromatic image according to a formula (4) to obtain the panchromatic image subjected to histogram matching as shown in the figure 3. Spectral distortion generated by fusion results due to the fact that the spectral response functions of the multispectral image and the panchromatic image are inconsistent is reduced.
Wherein PAN is a full color image, μPANAnd σPANIs the mean and square of the full color imageDifference, muVAnd σVIs the mean and variance of the luminance components, and P is the full-color image after histogram matching.
And step four, performing guided filtering on the brightness component of the multispectral image by taking the panchromatic image (remote sensing image) as a guide image by utilizing the structural transfer characteristic of the guided filtering, and removing the noise generated by the multispectral image in the HSV conversion stage.
And (4) performing guide filtering on the multispectral image subjected to the nonlinear HSV conversion in the second step by using the structure transfer characteristic of the guide filtering and taking the panchromatic image subjected to the histogram matching in the third step as a guide image according to formulas (5), (6) and (7), so as to remove noise generated by the multispectral image in the HSV conversion stage.
q1＝a1I+b1 (5)
Where p is the input image, I is the guide image, ε2Is the regularization parameter, q is the output image, a1And b1Is a hyperparameter of formula (5).
And step five, extracting edge detail information of the brightness components V of the panchromatic image and the multispectral image by using the edge-preserving smooth characteristic of the guide filtering and a means of the guide filtering to obtain a high-frequency detail image and a low-frequency image shown in the figure 4, fusing the edge detail information of the brightness components V of the panchromatic image and the multispectral image, and improving the extraction speed because the extraction process does not depend on the size of a sliding block window.
And (3) extracting edge detail information of the panchromatic image and the multispectral image brightness component V according to the guiding filtering formulas shown in formulas (8), (9) and (10) by utilizing the edge-preserving smoothing characteristic of guiding filtering on the multispectral image after nonlinear HSV conversion in the second step and the panchromatic image after histogram matching in the third step, and fusing the edge detail information of the panchromatic image and the multispectral image brightness component V, wherein the extraction process does not depend on the size of a sliding block window, and the extraction speed is improved.
q2＝a2p+b2 (8)
Where p is the input image, ε2Is the regularization parameter, q is the output image, a2And b2Is a hyperparameter of formula (8).
Step six: and repeating the iteration step five aiming at the multispectral image brightness component and the panchromatic image guided and filtered high-frequency component, and extracting the detail information of the panchromatic image and the multispectral image for multiple times until the effective extraction of the high-frequency component structural feature information of the panchromatic image is realized.
And seventhly, aiming at the multispectral image brightness component and the low-frequency component of the panchromatic image after the panchromatic image is guided and filtered, fusing the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and effectively extracting the low-frequency component information of the panchromatic image and the multispectral image by fully utilizing the energy of the low-frequency information of the image.
The sparse representation method is implemented as follows: acquiring a low-frequency information training image, converting the image into a feature matrix, solving the dictionary through a dictionary learning algorithm, solving the sparse coefficients of the panchromatic image and the brightness component under the dictionary, and fusing and reconstructing the low-frequency component according to the sparse coefficients.
And 7.1, selecting a plurality of panchromatic images containing ground object information along the subway as training images, performing low-pass filtering on the training images, and performing down-sampling on the training images to the size of the multi-spectral image to obtain corresponding low-resolution training images. In order to make the dimensions before and after fusion the low resolution training image down-sampled to the size of the multi-spectral image is up-sampled to the size of the panchromatic image to obtain the high resolution low frequency information training image.
Step 7.2, dividing the image of the low-frequency information obtained in the step 7.1 into images by using a sliding window
The image blocks are changed into column vectors of n × 1 size, and the obtained column vectors are combined into a feature matrix.
And 7.3, solving the formula (11) by using a KSVD dictionary learning algorithm to obtain the dictionary shown in the figure 5.
Wherein β ═ α1，α2，...，αm) Is the sparse coefficient of the image, DLFor training the resulting low resolution dictionary, XLFor the image matrix obtained in step 7.2, T is the sparsity of the sparse coefficient (the number of non-zero values in the sparse coefficient).
Step 7.4, aiming at the multispectral image brightness component and the panchromatic image guided and filtered low-frequency component, converting the multispectral image brightness component and the panchromatic image into a characteristic matrix by utilizing the step 7.2, solving the formula (14) to obtain the multispectral image brightness component and the panchromatic image sparse coefficient,
wherein DLIs the low resolution dictionary obtained in step 7.3, ε is the tolerance error, yLFIs the feature matrix of the image and alpha is the sparse coefficient to be solved.
And 7.5, reconstructing the multispectral image brightness component and the panchromatic image sparse coefficient obtained in the step 7.4 according to a fusion rule shown in the formula (15) that the sparse coefficient absolute value is the maximum to obtain a new sparse coefficient. And reconstructing a new characteristic matrix according to the formula (12) to realize the fusion of the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing the low-frequency characteristic information of the multispectral image and the panchromatic image to realize the effective extraction of the low-frequency component information of the panchromatic image and the multispectral image.
α＝max(α1,α2) (15)
Wherein alpha is1And alpha2Representing the panchromatic image and multispectral image luminance components obtained using step 7.4 in dictionary DLAnd obtaining the sparse coefficient.
Step eight: according to the high-frequency component structural feature information extracted in the sixth step and the low-frequency component information extracted in the seventh step, a brightness image V' is obtained through fusion, namely remote sensing image fusion is achieved based on guide filtering and sparse representation, and the resolution ratio of the multispectral image is remarkably improved; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
And fusing the high-frequency component structural feature information extracted in the sixth step and the low-frequency component information extracted in the seventh step to obtain a brightness image V ', and performing HSV (hue, saturation, value) spatial inverse transformation on the brightness image V' according to formulas (16), (17), (18), (19), (20) and (21) to realize super-resolution imaging of the multispectral image.
h＝floor(H/60) (16)
f＝H/60-h (17)
p＝V*(1-S) (18)
q＝V*(1-f*S) (19)
t＝V*(1-(1-f)*S (20)
Wherein: floor means rounding down.
It should be noted that the above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A remote sensing image fusion method based on guide filtering and sparse representation is characterized in that: comprises the following steps of (a) preparing a solution,
firstly, performing image interpolation operation on the multispectral image to expand the multispectral image to the same size as a panchromatic image, so as to facilitate the subsequent fusion operation between the multispectral image and the panchromatic image;
step two, performing nonlinear HSV conversion on the multispectral image to obtain a luminance image after the nonlinear HSV conversion, reducing image aliasing brought by subsequent operation, reducing the calculation amount of the subsequent operation and improving the fusion efficiency of the remote sensing image;
thirdly, histogram matching is carried out between the V component extracted from the multispectral image and the panchromatic image in the second step, and spectral distortion generated by a fusion result due to inconsistency of spectral response functions of the multispectral image and the panchromatic image is reduced;
fourthly, conducting guide filtering on the brightness component of the multispectral image by taking the panchromatic image as a guide image by utilizing the structure transfer characteristic of the guide filtering, and removing noise generated by the multispectral image in an HSV conversion stage;
fifthly, by utilizing the edge-preserving smoothness characteristic of the guide filtering, extracting edge detail information of the brightness components V of the panchromatic image and the multispectral image by using a guide filtering means, and fusing the edge detail information of the brightness components V of the panchromatic image and the multispectral image, wherein the extraction process does not depend on the size of a sliding block window, so that the extraction speed is increased;
step six, aiming at the multispectral image brightness component and the panchromatic image guided and filtered high-frequency component, repeating the iteration step four, and extracting the detail information of the panchromatic image and the multispectral image for multiple times until the effective extraction of the high-frequency component structural feature information of the panchromatic image is realized;
step seven, aiming at the multispectral image brightness component and the low-frequency component of the panchromatic image after the guidance and filtration, fusing the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing the energy of the low-frequency information of the image to realize the effective extraction of the low-frequency component information of the panchromatic image and the multispectral image;
step eight: according to the high-frequency component structural feature information extracted in the sixth step and the low-frequency component information extracted in the seventh step, a new brightness image V' is obtained through fusion, namely remote sensing image fusion is achieved based on guided filtering and sparse representation, and the resolution of the multispectral image is remarkably improved; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
2. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the second step is realized by the method that,
since the subjective perception of the luminance component V of the multispectral image is strong compared with the subjective perception of the color component, the color component is down-sampled on the digital representation of the picture and video, and the luminance component is not degraded; according to the HSV conversion model shown in the formulas (1), (2) and (3), the multispectral image is subjected to nonlinear HSV conversion, compared with other conversions, the fused image can be sharper, image aliasing caused by subsequent operation is reduced, the calculation amount of the subsequent operation is reduced, the remote sensing image fusion efficiency is improved, and compared with the conversions such as HIS (hue, saturation and value), more noise is introduced by the nonlinear HSV conversion;
V＝max(R,G,B) (1)
(3) where R denotes the red channel of the image, G denotes the green channel of the image, B denotes the blue channel of the image, H denotes hue, S denotes saturation, and V denotes brightness.
3. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the third step is to realize the method as follows,
performing histogram matching between the V component extracted from the multispectral image and the panchromatic image according to a formula (4), and reducing spectral distortion generated by a fusion result due to inconsistency of spectral response functions of the multispectral image and the panchromatic image;
wherein PAN is a full color image, μPANAnd σPANIs the mean and variance, μ, of the panchromatic imageVAnd σVIs the mean and variance of the luminance components, and P is the full-color image after histogram matching.
4. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the implementation method of the fourth step is that,
performing guiding filtering on the multispectral image subjected to the nonlinear HSV conversion in the second step by using the structure transfer characteristic of the guiding filtering and taking the panchromatic image subjected to the histogram matching in the third step as a guiding image according to formulas (5), (6) and (7) to remove noise generated by the multispectral image in the HSV conversion stage;
q1＝a1I+b1 (5)
where p is the input image, I is the guide image, ε2Is the regularization parameter, q is the output image, a1And b1Is a hyperparameter of formula (5).
5. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the fifth step is to realize that the method is that,
extracting edge detail information of the panchromatic image and the multispectral image luminance component V according to a guiding filtering formula shown in formulas (8), (9) and (10) by utilizing the edge-protecting smooth characteristic of guiding filtering on the multispectral image after nonlinear HSV conversion in the second step and the panchromatic image after histogram matching in the third step, and fusing the edge detail information of the panchromatic image and the multispectral image luminance component V, wherein the extraction process does not depend on the size of a sliding block window, and the extraction speed is improved;
q2＝a2p+b2 (8)
where p is the input image, ε2Is the regularization parameter, q is the output image, a2And b2Is a hyperparameter of formula (8).
6. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the sparse representation method is implemented as follows: acquiring a low-frequency information training image, converting the image into a feature matrix, solving a dictionary through a dictionary learning algorithm, solving the full-color image and the sparse coefficient of the brightness component under the dictionary, fusing and reconstructing a low-frequency component according to the sparse coefficient;
6.1, selecting a plurality of panchromatic images containing ground object information along the subway as training images, performing low-pass filtering on the panchromatic images and performing down-sampling on the panchromatic images to obtain corresponding low-resolution training images; in order to enable the dimensionality before and after fusion to be the same, the low-resolution training image which is down-sampled to the size of the multi-spectral image is up-sampled to the size of a full-color image to obtain a high-resolution low-frequency information training image;
step 6.2, dividing the image of the low-frequency information obtained in the step 6.1 into images by using a sliding window
The image blocks are changed into column vectors with the size of n multiplied by 1, and the obtained column vectors form a characteristic matrix;
6.3, solving the dictionary by using a formula (11) through a dictionary learning algorithm;
wherein β ═ α1,α2,...,αm) Is the sparse coefficient of the image, DLFor training the resulting low resolution dictionary, XLFor the image matrix obtained in step 6.2, T is the sparsity of the sparse coefficient (the number of non-zero values in the sparse coefficient);
the dictionary learning algorithm generally comprises 2 classes, one class is composed of a group of parameters and a set of selected functions containing the parameters, and the functions are used for approximately expressing signals; the dictionary does not need to store the whole dictionary, only needs to store related parameter information, thereby greatly reducing the storage capacity, but has no adaptability because the related dictionary is irrelevant to the original signal; the other dictionary learning method is an adaptive dictionary obtained by training and learning according to the characteristics of signals or images, and comprises a K-SVD algorithm and an MOD dictionary learning algorithm; the K-SVD algorithm can be better suitable for various images with different characteristics, and has the characteristic of strong adaptability; selecting a K-SVD algorithm learning dictionary; step 6.3 the specific implementation method is as follows:
6.3.1, dictionary initialization: randomly selecting K column vectors from the feature matrix obtained in the step 6.2 as atoms of the initial dictionary;
6.3.2, sparse coding: carrying out sparse coding by using a formula (12) to obtain a sparse coefficient;
x＝Dα (12)
where D represents a dictionary obtained from 6.3.1 and x is the feature matrix of the image;
6.3.3, dictionary update: updating the dictionary D column by utilizing an OMP algorithm, repeating the step 2, meanwhile, calculating an error matrix by utilizing a formula (13), and stopping updating when a specified error is converged to obtain the dictionary D capable of representing low-frequency component information;
wherein EkRepresents an error matrix, djRepresentation dictionary D (D)1,d2....dk) J column atom of (1);
step 6.4, aiming at the multispectral image brightness component and the panchromatic image guided and filtered low-frequency component, converting the multispectral image brightness component and the panchromatic image into a characteristic matrix by using the step 6.2, solving the formula (14) to obtain the multispectral image brightness component and the panchromatic image sparse coefficient,
wherein DLIs the low resolution dictionary obtained in step 6.3, ε is the tolerance error, yLFIs a feature matrix of the image, alpha is a sparse coefficient to be solved;
step 6.5, reconstructing the multispectral image brightness component and the panchromatic image sparse coefficient obtained in the step 6.4 according to a fusion rule that the sparse coefficient absolute value is the maximum and shown in the formula (15) to obtain a new sparse coefficient; reconstructing a new characteristic matrix according to the formula (12) to realize the fusion of the low-frequency information of the panchromatic image and the multispectral image through sparse representation, and fully utilizing the low-frequency characteristic information of the multispectral image and the panchromatic image to realize the effective extraction of the low-frequency component information of the panchromatic image and the multispectral image;
α＝max(α1,α2) (15)
wherein alpha is1And alpha2Representing the panchromatic image and multispectral image luminance components obtained in step 6.4 in dictionary DLAnd obtaining the sparse coefficient.
7. The remote sensing image fusion method based on the guided filtering and the sparse representation as claimed in claim 1, wherein: the eight steps of the implementation method are as follows,
fusing the high-frequency component structural feature information extracted in the fourth step and the low-frequency component information extracted in the fifth step to obtain a brightness image V ', and performing HSV (hue, saturation, value) space inverse transformation on the brightness image V' according to formulas (16), (17), (18), (19), (20) and (21) to realize super-resolution imaging of the multispectral image;
h＝floor(H/60) (16)
f＝H/60-h (17)
p＝V*(1-S) (18)
q＝V*(1-f*S) (19)
t＝V*(1-(1-f)*S (20)
wherein: floor denotes rounding down.
8. The remote sensing image fusion method based on the guided filtering and sparse representation as claimed in claims 1, 2, 3, 4, 5, 6 and 7, characterized by: carrying out nonlinear HSV (hue, saturation and value) transformation on the multispectral image to reduce image aliasing and reduce the calculated amount; extracting the detail information of the panchromatic image and the multispectral image for multiple times by using a guided filtering means to obtain a detail image for effectively extracting structural feature information; the low-frequency information of the panchromatic image and the multispectral image is fused through sparse representation, the energy of the low-frequency information of the image is fully utilized, and the maintenance of spectral details is facilitated. (ii) a According to the extracted high-frequency component structural feature information and low-frequency component information, fusing to obtain a brightness image V', namely realizing remote sensing image fusion based on guided filtering and sparse representation, and remarkably improving the resolution of the multispectral image; and performing HSV space inverse transformation on the brightness image V' to realize super-resolution imaging of the multispectral image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210356388.3A CN114708178B (en) | 2022-03-30 | 2022-03-30 | Remote sensing image fusion method based on guided filtering and sparse representation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210356388.3A CN114708178B (en) | 2022-03-30 | 2022-03-30 | Remote sensing image fusion method based on guided filtering and sparse representation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114708178A true CN114708178A (en) | 2022-07-05 |
| CN114708178B CN114708178B (en) | 2024-06-14 |
Family
ID=82172990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210356388.3A Active CN114708178B (en) | 2022-03-30 | 2022-03-30 | Remote sensing image fusion method based on guided filtering and sparse representation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114708178B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105761234A (en) * | 2016-01-28 | 2016-07-13 | 华南农业大学 | Structure sparse representation-based remote sensing image fusion method |
| CN105893971A (en) * | 2016-04-01 | 2016-08-24 | 上海理工大学 | Traffic signal lamp recognition method based on Gabor and sparse representation |
| CN109064437A (en) * | 2018-07-11 | 2018-12-21 | 中国人民解放军国防科技大学 | Image fusion method based on guided filtering and online dictionary learning |
| AU2020100179A4 (en) * | 2020-02-04 | 2020-03-19 | Huang, Shuying DR | Optimization Details-Based Injection Model for Remote Sensing Image Fusion |
| CN111680752A (en) * | 2020-06-09 | 2020-09-18 | 重庆工商大学 | Infrared and visible light image fusion method based on Framelet framework |
| CN112950518A (en) * | 2021-03-19 | 2021-06-11 | 中国科学院长春光学精密机械与物理研究所 | Image fusion method based on potential low-rank representation nested rolling guide image filtering |
| CN114066786A (en) * | 2020-08-03 | 2022-02-18 | 四川大学 | Infrared and visible light image fusion method based on sparsity and filter |
-
2022
- 2022-03-30 CN CN202210356388.3A patent/CN114708178B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105761234A (en) * | 2016-01-28 | 2016-07-13 | 华南农业大学 | Structure sparse representation-based remote sensing image fusion method |
| CN105893971A (en) * | 2016-04-01 | 2016-08-24 | 上海理工大学 | Traffic signal lamp recognition method based on Gabor and sparse representation |
| CN109064437A (en) * | 2018-07-11 | 2018-12-21 | 中国人民解放军国防科技大学 | Image fusion method based on guided filtering and online dictionary learning |
| AU2020100179A4 (en) * | 2020-02-04 | 2020-03-19 | Huang, Shuying DR | Optimization Details-Based Injection Model for Remote Sensing Image Fusion |
| CN111680752A (en) * | 2020-06-09 | 2020-09-18 | 重庆工商大学 | Infrared and visible light image fusion method based on Framelet framework |
| CN114066786A (en) * | 2020-08-03 | 2022-02-18 | 四川大学 | Infrared and visible light image fusion method based on sparsity and filter |
| CN112950518A (en) * | 2021-03-19 | 2021-06-11 | 中国科学院长春光学精密机械与物理研究所 | Image fusion method based on potential low-rank representation nested rolling guide image filtering |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114708178B (en) | 2024-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110119780B (en) | Hyper-spectral image super-resolution reconstruction method based on generation countermeasure network | |
| CN106920214B (en) | Super-resolution reconstruction method for space target image | |
| CN109741256B (en) | Image super-resolution reconstruction method based on sparse representation and deep learning | |
| CN107123089B (en) | Remote sensing image super-resolution reconstruction method and system based on depth convolution network | |
| CN104112263B (en) | The method of full-colour image and Multispectral Image Fusion based on deep neural network | |
| CN103455988B (en) | The super-resolution image reconstruction method of structure based self-similarity and rarefaction representation | |
| CN103871041B (en) | The image super-resolution reconstructing method built based on cognitive regularization parameter | |
| CN106709875A (en) | Compressed low-resolution image restoration method based on combined deep network | |
| CN109146787B (en) | Real-time reconstruction method of dual-camera spectral imaging system based on interpolation | |
| JP2003018398A (en) | Method for generating a super-resolution image from pixel image | |
| CN112529776B (en) | Training method of image processing model, image processing method and device | |
| CN112669214B (en) | Fuzzy image super-resolution reconstruction method based on alternating direction multiplier algorithm | |
| CN111696043A (en) | Hyperspectral image super-resolution reconstruction algorithm of three-dimensional FSRCNN | |
| CN114862731B (en) | Multi-hyperspectral image fusion method guided by low-rank priori and spatial spectrum information | |
| CN113658057A (en) | Swin transform low-light-level image enhancement method | |
| Zhou et al. | PAN-guided band-aware multi-spectral feature enhancement for pan-sharpening | |
| Zhong et al. | Attention_FPNet: Two-branch remote sensing image pansharpening network based on attention feature fusion | |
| CN104200439B (en) | Image super-resolution method based on adaptive filtering and regularization constraint | |
| CN109859153B (en) | Multispectral image fusion method based on adaptive spectrum-spatial gradient sparse regularization | |
| CN111275620B (en) | Image super-resolution method based on Stacking integrated learning | |
| CN114708178B (en) | Remote sensing image fusion method based on guided filtering and sparse representation | |
| CN108492264B (en) | Single-frame image fast super-resolution method based on sigmoid transformation | |
| CN116612009A (en) | Multi-scale connection generation countermeasure network medical image super-resolution reconstruction method | |
| CN108921785B (en) | Super-resolution reconstruction method based on wavelet packet | |
| CN113674154B (en) | Single image super-resolution reconstruction method and system based on generation countermeasure network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |