CN117726916B - Implicit fusion method for enhancing image resolution fusion - Google Patents
Implicit fusion method for enhancing image resolution fusion Download PDFInfo
- Publication number
- CN117726916B CN117726916B CN202410181120.XA CN202410181120A CN117726916B CN 117726916 B CN117726916 B CN 117726916B CN 202410181120 A CN202410181120 A CN 202410181120A CN 117726916 B CN117726916 B CN 117726916B
- Authority
- CN
- China
- Prior art keywords
- fusion
- image
- feature
- resolution
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 75
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 9
- 230000002708 enhancing effect Effects 0.000 title claims abstract description 7
- 238000000034 method Methods 0.000 claims abstract description 39
- 238000001228 spectrum Methods 0.000 claims abstract description 36
- 230000003595 spectral effect Effects 0.000 claims abstract description 27
- 230000001537 neural effect Effects 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims abstract description 5
- 230000004913 activation Effects 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 14
- 238000012360 testing method Methods 0.000 claims description 13
- 238000010586 diagram Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000002452 interceptive effect Effects 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 claims description 2
- 230000004044 response Effects 0.000 claims description 2
- 238000005316 response function Methods 0.000 claims description 2
- 238000013528 artificial neural network Methods 0.000 abstract description 2
- 238000013461 design Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 238000002474 experimental method Methods 0.000 abstract description 2
- 238000007499 fusion processing Methods 0.000 abstract description 2
- 230000003993 interaction Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 31
- 238000012544 monitoring process Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an implicit fusion method for enhancing image resolution fusion, belonging to the field of remote sensing image fusion design; the method adopts a network structure of an encoder-decoder, and is carefully designed in two stages. In the first stage, coordinate information, spatial information and spectral information are integrated by an implicit neural fusion function. In the second stage, a Fourier unit is introduced to extract the spectrum information, and the effective interaction of the spectrum information and the fusion information is realized through the double-branch encoder. Compared with other neural network methods, the method realizes continuous representation of the image in image processing, successfully maps the space coordinates to the pixel information, fully utilizes multi-modal information in the fusion process, and obtains a superior fusion effect. Experiments prove that the method obtains advanced performance on two different data sets and provides a universal model for other multi-mode fusion tasks.
Description
Technical Field
The invention belongs to the field of remote sensing image fusion design, and particularly relates to an implicit fusion method for enhancing image resolution fusion.
Background
The hyperspectral image (HSI) carries a large amount of spectrum wave bands with rich spectrum information, is beneficial to comprehensively grasp the characteristics of substances under different wavelengths, can effectively describe the spectrum distribution of each scene point, and provides inherent and identifying spectrum information of the scene. The collected dense spectrum band data can be applied to various fields including object identification and segmentation, anomaly detection, medical image analysis, remote sensing and the like, so that the data is increasingly applied to real life. But the hyperspectral images obtained are generally of lower spatial resolution due to the inherent mechanism of physical imaging today. While the spatial resolution of the multispectral image (MSI) is higher, this is accompanied by a reduction in spectral resolution. Therefore, the combination of a multispectral image and a hyperspectral pattern, which is expected to simultaneously obtain the hyperspectral resolution of the hyperspectral image and the high spatial resolution of the multispectral image, is attracting more and more attention.
Hyperspectral image fusion is an efficient and economical method of enhancing the spatial resolution of hyperspectral images (HSI) by combining hyperspectral images (HSI) with higher spatial resolution multispectral images (MSI) of the same scene. The multispectral image provides information of a limited wave band, and the hyperspectral image collects spectrum information in more wave bands, so that the information richness can be improved by fusion. Multispectral images typically have higher spatial resolution, while hyperspectral images provide more detailed spectral information. Fusion can achieve richer spectral details at high resolution. In the fields of agriculture, environmental monitoring and the like, the fusion task is helpful for more accurately identifying ground objects, monitoring vegetation, detecting environmental changes and the like. For the remote sensing field, the fusion task is helpful for more comprehensively understanding the surface characteristics, and the application value of the remote sensing data in resource management and environment monitoring is improved.
Although the prior art studies have achieved significant results in the task of hyperspectral fusion super-resolution, there are inevitably some problems and drawbacks. Current research has demonstrated that designing a dual-branched fusion network for high-resolution multispectral images (HR-MSI) and low-resolution hyperspectral images (LR-HSI) is limited by image features. Most existing networks are based on a generic CNN framework that lacks interpretability of hyperspectral image super-resolution fusion tasks and is mostly based on explicit representation of images. In the current research, we are urgent to pay close attention to the rapid development of technology to find new research break-outs and methods.
Disclosure of Invention
The invention aims to provide a high-efficiency and accurate multispectral and hyperspectral image fusion super-resolution method, provides a hyperspectral image fusion super-resolution method based on an implicit fusion function, provides a more innovative solution of an image fusion task, and provides more powerful and reasonable technical support for application in the field.
In order to achieve the above object, the technical scheme of the present invention is as follows: the hyperspectral image fusion super-resolution method based on the implicit fusion function comprises the following steps:
step 1: acquiring and processing a data set; acquiring an image dataset, preprocessing an image in the dataset to obtain a low-resolution hyperspectral image High resolution multispectral image/>Corresponding real pixel value image/>Dividing a training data set and a test data set;
Step 2: coding and multi-mode information generation;
First, a low resolution hyperspectral image Obtaining an up-sampled low-resolution hyperspectral image/>, through bicubic interpolation；
Then the up-sampled low resolution hyperspectral image is processedAnd high resolution multispectral image/>Splice together in the channel dimension to obtain splice characteristics/>；
Will low resolution hyperspectral imageAnd splice features/>Respectively inputting a spectrum coding function and a space coding function, and calculating the spectrum modal characteristic/>And spatial modality characteristics/>；
The center position of the pixel is expressed as a coordinate point of the pixel, and the coordinate graph is scaled into a square grid with the size of 2 multiplied by 2, so that a normalized two-dimensional coordinate graph with high resolution domain is obtainedThereby obtaining the query coordinates/>And nearest coordinates/>Obtain the coordinate modal information/>；
Step 3: the implicit neural image interpolation is fused in the first stage; the spectrum modal characteristics obtained in the step2 are characterizedSpatial modality characteristics/>And coordinate modality feature/>Splicing in the channel dimension, and obtaining the interpolation pixel value/>, of the pixel by using an implicit characteristic fusion functionSimilarity weight/>Then, similarity feature integration is carried out to obtain a fusion feature map/>；
Step 4: a decoding stage; will be represented by high resolution multispectral imagesSpectral response information obtained via a fourier unit/>Fusion of feature map with code output/>Simultaneously inputting a Decoder of the dual-branch Decoder for fusion, and uploading the Decoder by fusion characteristics/>For input, the lower branch takes the spectral features/>For input, the upper branch uses ReLU activation function to activate nonlinearity, and the lower branch uses complex Gabor wavelet activation function to activate nonlinearity, to obtain output/>, of decoding stage；
Step 5: obtaining output; adopting a long jumper wire connection mode to output the output obtained in the step 4And upsampled low resolution hyperspectral image/>Adding elements by elements to obtain a fusion result/>; And obtaining a final fusion result after training and testing.
The beneficial effects of the invention are as follows: the method provided by the invention is an efficient multispectral and hyperspectral image fusion method, a network structure of an encoder-decoder is designed to perform two-stage fusion, coordinate information, spatial information and spectrum information are fused in a first stage through an implicit nerve fusion function, a Fourier unit is designed to extract the spectrum information, and interaction between the spectrum information and the fusion information in a second stage is realized by using a double-branch encoder. The effective utilization of the frequency domain information by the framework is verified by fusing the multi-modal information such as coordinates, space, frequency spectrum data and the like for a plurality of times. Compared with other neural network methods, the method can continuously represent the images, realize continuous mapping from the space coordinates to the pixel information, and utilize multi-mode information in the fusion process so as to obtain a better fusion effect, and the implicit image interpolation brings more reasonable interpretation to MHIF tasks. Experiments prove that the method achieves the most advanced performance on two different data sets, and provides a universal example for other multi-mode fusion tasks.
Drawings
FIG. 1 is a main flow chart of an embodiment of the present invention.
Fig. 2 is a schematic diagram of the relative coordinates of step 2 and step 3 according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a fourier cell of step 4 in accordance with an embodiment of the invention.
Fig. 4 is a schematic diagram of a decoder according to step 4 of the embodiment of the present invention.
Detailed Description
The invention will be further described with reference to the drawings and the specific examples. As shown in fig. 1, an implicit fusion method for enhancing image resolution fusion includes the following steps:
Step1, acquiring and processing an image data set: 20 images were randomly selected from the CAVE dataset for training, with the remaining 11 images of the dataset comprising the test dataset. 20 images were selected from the Harvard dataset and a 1000 x 1000 section was cut in the upper left corner, 10 of which were used for training and 10 for testing.
For the CAVE dataset, we cut 3920 overlapping patches from the 20 training images selected, each patch (patch) being 64×64×31 in size, and take it as a real pixel value image (group-truth, hereinafter GT image). To generate a suitable low resolution hyperspectral image (LR-HSI), standard deviation is usedIs blurred to the original high resolution multispectral image by a3 x3 gaussian kernel and then uses the value/>Downsampling the blurred block by the scale factor of/>, in this embodimentIs 0.5,/>4. Furthermore, we created a high resolution multispectral image patch using the common spectral response function of the nikon D7003 camera and the high resolution multispectral image (HR-HSI). Thus, we generate 3920 LR-HSIs of size 16×16×31 and HR-MSIs of size 64×64×3 from the input pairs (LR-HSIs, HR-MSIs). The input pairs and corresponding GT images were then randomly split into training data (80%) and test data (20%).
The above procedure is equally applicable to the generation of Harvard datasets. The two data sets are applied to obtain the low-resolution hyperspectral imageAnd high resolution multispectral image/>Corresponding GT image/>。
Step2, generating spectral modal characteristicsSpatial modality characteristics/>And coordinate modality feature/>：
Step 2.1: first, a low resolution hyperspectral image is imagedObtaining an up-sampled low-resolution hyperspectral image (/ >) through bicubic interpolation) The process can be expressed as: /(I)
Wherein,Up-sampling function for bicubic interpolation,/>Is a low resolution hyperspectral image with a spatial size/>And the number of channels is/>，/>Upsampled image for LR-HSI bicubic interpolation with spatial size/>And the number of channels is/>And/>Is a high-resolution multispectral image with a spatial size ofAnd the number of channels is s.
Up-sampled low resolution hyperspectral imageAnd high resolution multispectral image/>Splice together in the channel dimension to obtain splice characteristics/>，/>Expressed as a concatenation of channel dimensions, respectively representing low resolution hyperspectral images/>And splice features/>And (3) inputting a spectrum coding function and a space coding function, and calculating spectrum modal characteristics and space modal characteristics of the spectrum coding function and the space coding function: /(I)
In the method, in the process of the invention,Is characterized by spectrum mode, and the space size is/>The number of channels is/>；Is characterized by spectrum mode, and the space size is/>The number of channels is/>；/>As a function of the spectral encoding,Learning parameters expressed as spectral encoding functions, likewise,/>As a spatial coding function,/>The learnable parameters, denoted spatial encoding functions, are both implemented with EDSR encoder networks.
Step 2.2: in general, implicit neural representations use relative position coordinatesIn the present invention,/>Normalizing a two-dimensional graph for a high resolution domain, the spatial size of which is/>The number of channels is 2. Which represents the pixel with its center position and will/>Scaled to a magnitude/>Allowing the invention to use a continuous image representation that can share the same coordinates in the High Resolution (HR) and Low Resolution (LR) fields. The location coordinates of the query are/>The relative position coordinates are then expressed as: /(I)
Wherein,For and query coordinates/>Nearest coordinates,/>The four corner pixels closest to q in HR are in this invention the four corners in square grid. In obtaining relative position coordinates/>Then position coding is carried out, thus obtaining the coordinate modal characteristic/>：/>
In the method, in the process of the invention,As a position coding function, L is a super parameter, and in this embodiment, L is set to 10.
The up-sampled low-resolution hyperspectral image I LR_up and high-resolution multispectral image are combinedSplice together in the channel dimension to obtain splice characteristics/>，/>Expressed as a concatenation of channel dimensions, respectively representing low resolution hyperspectral images/>The spectral coding function and the spatial coding function are input with the splicing feature Cat (I LR_up,IHR), and the spectral mode feature and the spatial mode feature of the spectral coding function are calculated and expressed as follows: /(I)；
And step 3, calculating interpolation pixel values and similarity weights, and carrying out fusion in the first stage by utilizing neural implicit image interpolation. The specific process is as shown in fig. 3: step 3.1: characterizing the spectral modality calculated in step 2Spatial modal characteristicsAnd coordinate modality feature/>Input to implicit feature fusion function/>Fusing the characteristics and generating similarity weight/>. The process is expressed as: /(I)
Wherein V is an interpolation value feature vector obtained by the function, and W is an interpolation weight feature vector. In this embodiment, the implicit feature fusion functionMultilayer perceptron (MLP) being a 2-layer fully connected layer,/>Is its learnable parameter that integrates the learning of interpolation weights and values. As shown in fig. 2, for query pixel q, it corresponds to coordinates/>Corresponds to nearest position coordinates/>By implicit feature fusion function/>Obtaining interpolation pixel value/>, corresponding to adjacent coordinatesAnd learned similarity weights/>Finally, the similarity weight is normalized by applying a softmax function:
step 3.2: and (5) performing neural implicit image interpolation to achieve the purpose of feature integration. The specific interpolation formula is:
wherein, Is the coordinates/>Fusion features of positions for/>And/>，/>I.e. after interpolation of the imageAnd/>Is a fusion feature map of the formula (I), the space size of which is/>The number of channels is/>。
Step 4. In the decoding stage, as shown in FIG. 3, the method comprisesPerforming Fourier transform to transform into Fourier domain, capturing spectrum features in frequency domain by using convolution layer, and performing Fourier inverse transform to transform spectrum features into space domain to obtain spectrum features/>; The above process is called a fourier unit and can be expressed as:
wherein, Is a fourier unit.
Obtaining spectral featuresAfter that, it is fused with the feature map/>Common input to dual-branch decoderObtaining final decoder output characteristics:
wherein, Is a dual-branch decoder,/>Is a learnable parameter of a decoder consisting of four layers/>The convolution intermediate interleaving ReLU activation function and the Complex Gabor wavelet activation function is a double-branch interactive fusion decoder. Superior to fusion features/>For input, the lower branch takes the spectral features/>For input, the spectrum feature and the fusion feature enter a convolution layer of 1×1 respectively, then the obtained two feature graphs are subjected to matrix multiplication to be used as new spectrum features, the fusion feature of the feature and the input decoder is used as new fusion feature after channel dimension splicing, after the new spectrum feature and the fusion feature are obtained, the upper branch uses a ReLU activation function to activate nonlinearity, the lower branch uses a complex Gabor wavelet activation function, the new two features enter the convolution layer of the next round, after the operation is performed for 4 times, the output feature/> is obtained through the last convolution layer after the multiplication of the two branch features. The specific structure is shown in fig. 4. Wherein the ReLU activation function is denoted as/>：/>
The Complex Gabor wavelet activation function is：
Wherein,Are all features input to the activation function,/>Controlling the frequency of the wavelet,/>The width of the wavelet is controlled.
Step 5: decoder with double branchesOutput characteristics/>And =/>, after up-sampling by bicubic interpolationAdding elements by elements to obtain the expected hyperspectral fusion image/>：
Wherein,For the output of a dual-branch decoder,/>Is thatThe upsampled image is bicubic interpolated.
In the embodiment, training and testing are realized by using an Adam optimizer under PyTorch.0.1 and Python 3.9.15 environments, in the training process, 2000 epochs are trained altogether, the learning rate of the first 1000 epochs training is set to be 0.0001, the learning rate of the second 1000 epochs training is set to be 0.00001, and the linux operating system is NVIDIA RTX3090 GPU (24 GB). Optimization is performed by minimizing the standard L1 loss absolute difference:
Where N is the total number of sampled pixels, For arbitrarily sampling the coordinates of the pixel,/>For the real pixel value group-Truth,/>For predicting pixel values. In testing, the present embodiment queries the coordinates of all pixels in the target domain to recover the complete upsampled fusion image.
When the two data sets are tested by using the optimal model obtained through training, four evaluation indexes, namely PSNR (peak signal to noise ratio), SAM (spectrum angle), ERGAS (relative integral dimension integrated error) and SSIM (structural similarity) are adopted. These metrics are used to evaluate model performance. We performed four-fold super-resolution fusion tests on the CAVE dataset and the Harvard dataset, resulting in the following results: the CAVE test set had an average PSNR value of 52.40dB, an average SAM value of 1.925, an average ERGAS value of 0.996, and an average SSIM value of 0.999; the Harvard test set had an average PSNR value of 49.07dB, an average SAM value of 2.093, an average ERGAS value of 1.792, and an average SSIM value of 0.996.
Those of ordinary skill in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the practice of the invention and that the scope of the invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific modifications and combinations from the teachings of the present disclosure without departing from the spirit thereof, and such modifications and combinations remain within the scope of the present disclosure.
Claims (6)
1. An implicit fusion method for enhancing image resolution fusion, which is characterized by comprising the following steps:
Step 1: acquiring and processing a data set; acquiring an image dataset, preprocessing images in the dataset to obtain a low-resolution hyperspectral image I LR, a high-resolution multispectral image I HR and a corresponding real pixel value image I GT, and dividing a training dataset and a testing dataset;
Step 2: coding and multi-mode information generation;
Firstly, performing bicubic interpolation on a low-resolution hyperspectral image I LR to obtain an up-sampled low-resolution hyperspectral image I LR_up;
Then splicing the up-sampled low-resolution hyperspectral image I LR_up and the high-resolution multispectral image I HR together in the channel dimension to obtain a spliced characteristic Cat (I LR_up,IHR);
Respectively inputting a low-resolution hyperspectral image I LR and a splicing feature Cat (I LR_up,IHR) into a spectral coding function and a spatial coding function, and calculating a spectral mode feature S pe and a spatial mode feature S pa;
The center position of a pixel is expressed as a coordinate point of the pixel, and the coordinate graph is scaled into a square grid with the size of 2 multiplied by 2, so that a normalized two-dimensional coordinate graph X HR with a high resolution domain is obtained, and further, the relative positions of a query coordinate X q and a nearest coordinate X q,i are obtained, and coordinate mode information PE (delta X) is obtained; the specific process of generating the coordinate mode information PE (delta X) is as follows:
normalized two-dimensional graph of the high resolution domain The space size is H multiplied by W, and the channel number is 2; the method uses the central position of the pixel to represent the pixel, scales the H multiplied by W coordinate graph into square grids with the size of [ -1,1] × [ -1,1], and adopts a continuous image representation mode to share the same coordinate in the high-resolution and low-resolution fields; the position coordinate of the query is x q∈XHR, and the relative position coordinate is expressed as:
Where x q,i is the coordinate nearest to query coordinate x q, The four corner pixels closest to q in HR are the four corners in the square grid; after obtaining the relative position coordinates δx, position encoding is performed, so as to obtain the coordinate modal feature PE (δx):
PE(δX)＝[sin(20δX),cos(20δX),...,sin(2L-1δX),cos(2L-1δX)]
Wherein PE is a position coding function, and L is a super parameter;
Step 3: the implicit neural image interpolation is fused in the first stage; splicing the spectrum modal feature S pe, the spatial modal feature S pa and the coordinate modal feature PE (delta X) obtained in the step 2 in the channel dimension, obtaining an interpolation pixel value v q,i and a similarity weight w q,i of a pixel by using an implicit feature fusion function, and then carrying out similarity feature integration to obtain a fusion feature diagram epsilon; the method comprises the following specific steps:
Step 3.1: the spectral modal feature S pe, the spatial modal feature S pa and the coordinate modal feature PE (δX) are input into an implicit feature fusion function f θ, the features are fused, and meanwhile, a similarity weight w q,i is generated, and the process is expressed as follows:
W，V＝fθ(Spe,Spa，PE(δX))
Wherein the implicit feature fusion function f θ is a multi-layer perceptron of 2 layers of full-connection layers, V is an interpolation value feature vector obtained by the function, W is an interpolation weight feature vector, and θ is a learnable parameter; for the query pixel q, the corresponding coordinate x q∈XHR corresponds to the nearest position coordinate x q,i, the interpolation pixel value V q,i epsilon V corresponding to the nearest position coordinate and the learned similarity weight W q,i epsilon W are obtained by the implicit feature fusion function f θ, and finally the similarity weight is normalized by the softmax function:
step 3.2: the neural implicit image interpolation is carried out, and a specific interpolation formula is as follows:
Where ε q is the fusion feature of the coordinate x q position, for I LR and I HR, The method is a fusion characteristic diagram of I LR and I HR after image interpolation, the space size is H multiplied by W, and the channel number is S;
Step 4: a decoding stage; spectral response information obtained from the high resolution multispectral image I HR through a Fourier unit, Inputting the characteristic diagram epsilon and the coding output fusion characteristic diagram epsilon into a dual-branch Decoder to be fused at the same time, so as to obtain an output I fusion of a decoding stage; the dual-branch Decoder consists of four layers of 1 multiplied by 1 convolution modules, and the convolution middle is inserted with a ReLU activation function and a Complex Gabor wavelet activation function, so that the dual-branch Decoder is a dual-branch interactive fusion Decoder; the decoder is upscaled with fusion features/>For input, the lower branch takes the spectral features/>For input, the spectrum feature and the fusion feature enter a convolution layer of 1×1 respectively, then the obtained two feature graphs are subjected to matrix multiplication to be used as new spectrum features, the fusion feature of the feature and the input decoder is spliced in the channel dimension to be used as new fusion feature, after the new spectrum features and the fusion feature are obtained, the upper branch uses a ReLU activation function to activate nonlinearity, the lower branch uses a complex Gabor wavelet activation function, the new two features enter the convolution layer of the next round, after the operation is performed for 4 times, the output feature I fusion is obtained through the last convolution layer after the multiplication of the double-branch features, and the method is expressed as follows:
wherein, Is a dual-branch decoder,/>Is a learnable parameter of the decoder;
Wherein the ReLU activation function is denoted as σ ReLU:
σReLU(x)＝max(0，x)
the Complex Gabor wavelet activation function is σ Gabor:
Wherein x is the characteristic input to the activation function, omega 0 controls the frequency of the wavelet, and s 0 controls the width of the wavelet;
Step 5: obtaining output; adding the output I fusion obtained in the step 4 and the up-sampled low-resolution hyperspectral image I LR_up element by adopting a long jumper connection mode to obtain a fusion result And obtaining a final fusion result after training and testing.
2. The method of claim 1, wherein the fusion result is obtained by minimizing the standard L1 loss absolute difference during trainingAnd (3) optimizing:
Where N is the total number of sampled pixels, x i is the coordinates of any sampled pixel, I GT(xi) is the true pixel value, For predicting pixel values.
3. An implicit fusion method for image resolution fusion enhancement according to claim 2, characterized in that the step of preprocessing the images in the dataset is as follows:
m overlapping small blocks are cut out from a training image in a data set, and each patch is 64 multiplied by 31 in size and is used as a real pixel value image I GT;
performing fuzzy processing on the initial high-resolution multispectral image by using a Gaussian kernel with standard deviation sigma, and then downsampling a fuzzy block by using a scale factor with value tau to obtain a low-resolution multispectral image I LR;
Creating a high-resolution multispectral image patch by using a common spectral response function of the camera and the high-resolution multispectral image to obtain a high-resolution multispectral image I HR; the low resolution hyperspectral image I LR, the high resolution multispectral image I HR, and the corresponding real pixel value images are then randomly separated into training data and test data.
4. An implicit fusion method of image resolution fusion enhancement according to claim 3, wherein the process of generating spectral modality features S pe, spatial modality features S pa is specifically as follows:
The process of obtaining the up-sampled low-resolution hyperspectral image I LR_up from the low-resolution hyperspectral image I LR through bicubic interpolation is expressed as follows:
ILR_up＝fBI(ILR)
wherein f BI (·) is a bicubic interpolation upsampling function, Is a low-resolution hyperspectral image, the space size is h multiplied by w, and the channel number is S,/>The spatial size of the image is H multiplied by W, the channel number is S, and/>, the image is obtained by bicubic interpolation and up-sampling of the low-resolution hyperspectral imageThe space size of the high-resolution multispectral image is H multiplied by W, and the channel number is s;
splicing the up-sampled low-resolution hyperspectral image I LR_up and the high-resolution multispectral image I HR together in the channel dimension to obtain splicing characteristics Cat (·) is expressed as a concatenation of channel dimensions, the low-resolution hyperspectral image I LR and the concatenation feature Cat (I LR_up,IHR) are respectively input into a spectral coding function and a spatial coding function, and the spectral mode feature and the spatial mode feature of the low-resolution hyperspectral image I LR and the concatenation feature Cat (I LR_up,IHR) are calculated and expressed as follows:
In the method, in the process of the invention, The space size of the spectrum is h multiplied by w, and the channel number is D; /(I)The space size of the spectrum is H multiplied by W, and the channel number is D; e φ is a spectral coding function, phi is a learnable parameter of the spectral coding function, E ψ is a spatial coding function, and ψ is a learnable parameter of the spatial coding function.
5. The method according to claim 4, wherein the process of passing through a fourier unit is specifically: will firstPerforming Fourier transform to transform into Fourier domain, capturing spectrum features in frequency domain by using convolution layer, and performing Fourier inverse transform to transform spectrum features into space domain to obtain spectrum features/>The above process is called a fourier cell, denoted as:
Wherein FU (·) is a fourier unit.
6. The method of claim 5, wherein the spectral encoding function and the spatial encoding function are implemented in EDSR encoder networks.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410181120.XA CN117726916B (en) | 2024-02-18 | 2024-02-18 | Implicit fusion method for enhancing image resolution fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410181120.XA CN117726916B (en) | 2024-02-18 | 2024-02-18 | Implicit fusion method for enhancing image resolution fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117726916A CN117726916A (en) | 2024-03-19 |
| CN117726916B true CN117726916B (en) | 2024-04-19 |
Family
ID=90205693
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410181120.XA Active CN117726916B (en) | 2024-02-18 | 2024-02-18 | Implicit fusion method for enhancing image resolution fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117726916B (en) |
Citations (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018160963A1 (en) * | 2017-03-02 | 2018-09-07 | Spectral Md, Inc. | Machine learning systems and techniques for multispectral amputation site analysis |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| CN112184553A (en) * | 2020-09-25 | 2021-01-05 | 西北工业大学 | Hyperspectral image super-resolution method based on depth prior information |
| CN112634137A (en) * | 2020-12-28 | 2021-04-09 | 西安电子科技大学 | Hyperspectral and full-color image fusion method based on AE extraction of multi-scale spatial spectrum features |
| CN113763299A (en) * | 2021-08-26 | 2021-12-07 | 中国人民解放军军事科学院国防工程研究院工程防护研究所 | Panchromatic and multispectral image fusion method and device and application thereof |
| CN114119444A (en) * | 2021-11-29 | 2022-03-01 | 武汉大学 | Multi-source remote sensing image fusion method based on deep neural network |
| WO2022051421A1 (en) * | 2020-09-04 | 2022-03-10 | Nvidia Corporation | Processor and system for automatic fusion of matrix multiplication and reduction operations |
| CN114742985A (en) * | 2022-03-17 | 2022-07-12 | 苏州大学 | Hyperspectral feature extraction method and device and storage medium |
| CN114782246A (en) * | 2022-03-19 | 2022-07-22 | 南京理工大学 | Hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection |
| CN114862731A (en) * | 2022-03-29 | 2022-08-05 | 武汉大学 | Multi-hyperspectral image fusion method based on low-rank prior and spatial-spectral information guidance |
| CN115147321A (en) * | 2022-07-14 | 2022-10-04 | 武汉大学 | Multi-spectral image fusion method based on interpretable neural network |
| CN115512192A (en) * | 2022-08-16 | 2022-12-23 | 南京审计大学 | Multispectral and hyperspectral image fusion method based on cross-scale octave convolution network |
| CN115564692A (en) * | 2022-09-07 | 2023-01-03 | 宁波大学 | Panchromatic-multispectral-hyperspectral integrated fusion method considering width difference |
| CN115760814A (en) * | 2022-11-25 | 2023-03-07 | 山东师范大学 | Remote sensing image fusion method and system based on double-coupling deep neural network |
| CN115861076A (en) * | 2022-12-28 | 2023-03-28 | 江南大学 | Unsupervised hyperspectral image super-resolution method based on matrix decomposition network |
| CN116051982A (en) * | 2022-12-20 | 2023-05-02 | 北京林业大学 | Deep learning method for identifying pine wood nematode disease damage wood by fusing multi-source heterogeneous data |
| CN116205830A (en) * | 2023-02-09 | 2023-06-02 | 武汉大学 | Remote sensing image fusion method based on combination of supervised learning and unsupervised learning |
| CN116416375A (en) * | 2023-02-17 | 2023-07-11 | 贵州大学 | Three-dimensional reconstruction method and system based on deep learning |
| CN116433548A (en) * | 2023-04-14 | 2023-07-14 | 西安电子科技大学 | Hyperspectral and panchromatic image fusion method based on multistage information extraction |
| CN116468645A (en) * | 2023-06-20 | 2023-07-21 | 吉林大学 | Antagonistic hyperspectral multispectral remote sensing fusion method |
| WO2023140488A1 (en) * | 2022-01-20 | 2023-07-27 | Samsung Electronics Co., Ltd. | Bandwidth extension and speech enhancement of audio |
| CN116740340A (en) * | 2023-07-11 | 2023-09-12 | 西安电子科技大学 | Calculation spectrum imaging error correction method based on deep learning |
| CN116883799A (en) * | 2023-07-11 | 2023-10-13 | 西安电子科技大学 | Hyperspectral image depth space spectrum fusion method guided by component replacement model |
| WO2023219963A1 (en) * | 2022-05-08 | 2023-11-16 | The Medical College Of Wisconsin, Inc. | Deep learning-based enhancement of multispectral magnetic resonance imaging |
| CN117197008A (en) * | 2023-08-15 | 2023-12-08 | 武汉大学 | Remote sensing image fusion method and system based on fusion correction |
| CN117274759A (en) * | 2023-09-21 | 2023-12-22 | 大连大学 | Infrared and visible light image fusion system based on distillation-fusion-semantic joint driving |
| CN117474781A (en) * | 2023-11-06 | 2024-01-30 | 东方通信股份有限公司 | High spectrum and multispectral image fusion method based on attention mechanism |
| CN117557476A (en) * | 2023-10-12 | 2024-02-13 | 杭州电子科技大学 | Image reconstruction method and system based on FCTFT |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11676284B2 (en) * | 2019-03-22 | 2023-06-13 | Nvidia Corporation | Shape fusion for image analysis |
| US20220122250A1 (en) * | 2020-10-19 | 2022-04-21 | Northwestern University | Brain feature prediction using geometric deep learning on graph representations of medical image data |
| US20230062811A1 (en) * | 2021-08-24 | 2023-03-02 | PAIGE.AI, Inc. | Systems and methods for processing electronic images in forensic pathology |
| US20230154007A1 (en) * | 2021-11-15 | 2023-05-18 | Elekta Limited | Few-shot semantic image segmentation using dynamic convolution |
| US20230360180A1 (en) * | 2022-05-04 | 2023-11-09 | Adobe Inc. | Digital image inpainting utilizing a cascaded modulation inpainting neural network |
| US20230368339A1 (en) * | 2022-05-13 | 2023-11-16 | Adobe Inc. | Object class inpainting in digital images utilizing class-specific inpainting neural networks |
-
2024
- 2024-02-18 CN CN202410181120.XA patent/CN117726916B/en active Active
Patent Citations (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018160963A1 (en) * | 2017-03-02 | 2018-09-07 | Spectral Md, Inc. | Machine learning systems and techniques for multispectral amputation site analysis |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| WO2022051421A1 (en) * | 2020-09-04 | 2022-03-10 | Nvidia Corporation | Processor and system for automatic fusion of matrix multiplication and reduction operations |
| CN112184553A (en) * | 2020-09-25 | 2021-01-05 | 西北工业大学 | Hyperspectral image super-resolution method based on depth prior information |
| CN112634137A (en) * | 2020-12-28 | 2021-04-09 | 西安电子科技大学 | Hyperspectral and full-color image fusion method based on AE extraction of multi-scale spatial spectrum features |
| CN113763299A (en) * | 2021-08-26 | 2021-12-07 | 中国人民解放军军事科学院国防工程研究院工程防护研究所 | Panchromatic and multispectral image fusion method and device and application thereof |
| CN114119444A (en) * | 2021-11-29 | 2022-03-01 | 武汉大学 | Multi-source remote sensing image fusion method based on deep neural network |
| WO2023140488A1 (en) * | 2022-01-20 | 2023-07-27 | Samsung Electronics Co., Ltd. | Bandwidth extension and speech enhancement of audio |
| CN114742985A (en) * | 2022-03-17 | 2022-07-12 | 苏州大学 | Hyperspectral feature extraction method and device and storage medium |
| WO2023173884A1 (en) * | 2022-03-17 | 2023-09-21 | 苏州大学 | Hyper-spectral feature extraction method and apparatus, and storage medium |
| CN114782246A (en) * | 2022-03-19 | 2022-07-22 | 南京理工大学 | Hyperspectral super-resolution method using asymmetric attention and wavelet sub-band injection |
| CN114862731A (en) * | 2022-03-29 | 2022-08-05 | 武汉大学 | Multi-hyperspectral image fusion method based on low-rank prior and spatial-spectral information guidance |
| WO2023219963A1 (en) * | 2022-05-08 | 2023-11-16 | The Medical College Of Wisconsin, Inc. | Deep learning-based enhancement of multispectral magnetic resonance imaging |
| CN115147321A (en) * | 2022-07-14 | 2022-10-04 | 武汉大学 | Multi-spectral image fusion method based on interpretable neural network |
| CN115512192A (en) * | 2022-08-16 | 2022-12-23 | 南京审计大学 | Multispectral and hyperspectral image fusion method based on cross-scale octave convolution network |
| CN115564692A (en) * | 2022-09-07 | 2023-01-03 | 宁波大学 | Panchromatic-multispectral-hyperspectral integrated fusion method considering width difference |
| CN115760814A (en) * | 2022-11-25 | 2023-03-07 | 山东师范大学 | Remote sensing image fusion method and system based on double-coupling deep neural network |
| CN116051982A (en) * | 2022-12-20 | 2023-05-02 | 北京林业大学 | Deep learning method for identifying pine wood nematode disease damage wood by fusing multi-source heterogeneous data |
| CN115861076A (en) * | 2022-12-28 | 2023-03-28 | 江南大学 | Unsupervised hyperspectral image super-resolution method based on matrix decomposition network |
| CN116205830A (en) * | 2023-02-09 | 2023-06-02 | 武汉大学 | Remote sensing image fusion method based on combination of supervised learning and unsupervised learning |
| CN116416375A (en) * | 2023-02-17 | 2023-07-11 | 贵州大学 | Three-dimensional reconstruction method and system based on deep learning |
| CN116433548A (en) * | 2023-04-14 | 2023-07-14 | 西安电子科技大学 | Hyperspectral and panchromatic image fusion method based on multistage information extraction |
| CN116468645A (en) * | 2023-06-20 | 2023-07-21 | 吉林大学 | Antagonistic hyperspectral multispectral remote sensing fusion method |
| CN116740340A (en) * | 2023-07-11 | 2023-09-12 | 西安电子科技大学 | Calculation spectrum imaging error correction method based on deep learning |
| CN116883799A (en) * | 2023-07-11 | 2023-10-13 | 西安电子科技大学 | Hyperspectral image depth space spectrum fusion method guided by component replacement model |
| CN117197008A (en) * | 2023-08-15 | 2023-12-08 | 武汉大学 | Remote sensing image fusion method and system based on fusion correction |
| CN117274759A (en) * | 2023-09-21 | 2023-12-22 | 大连大学 | Infrared and visible light image fusion system based on distillation-fusion-semantic joint driving |
| CN117557476A (en) * | 2023-10-12 | 2024-02-13 | 杭州电子科技大学 | Image reconstruction method and system based on FCTFT |
| CN117474781A (en) * | 2023-11-06 | 2024-01-30 | 东方通信股份有限公司 | High spectrum and multispectral image fusion method based on attention mechanism |
Non-Patent Citations (3)
| Title |
|---|
| Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion;Deng, ShangQi等;《arXiv preprint arXiv:2307.07288》;20231029;1-10 * |
| MIMO-SST: Multi-Input Multi-Output Spatial-Spectral Transformer for Hyperspectral and Multispectral Image Fusion;Fang, Jian等;《IEEE Transactions on Geoscience and Remote Sensing》;20240202;第62卷;1-20 * |
| 基于多层小波深度聚合网络的高光谱图像超分辨率方法;方健等;《电子学报》;20240131;第52卷(第1期);201-216 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117726916A (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fan et al. | Balanced two-stage residual networks for image super-resolution | |
| CN110660038B (en) | Multispectral image and full-color image fusion method based on generation countermeasure network | |
| Yue et al. | Deep recursive super resolution network with Laplacian Pyramid for better agricultural pest surveillance and detection | |
| CN109727207B (en) | Hyperspectral image sharpening method based on spectrum prediction residual convolution neural network | |
| CN109102469B (en) | Remote sensing image panchromatic sharpening method based on convolutional neural network | |
| CN109064396A (en) | A kind of single image super resolution ratio reconstruction method based on depth ingredient learning network | |
| Li et al. | Deep learning methods in real-time image super-resolution: a survey | |
| Mei et al. | Hyperspectral image super-resolution via convolutional neural network | |
| Liu et al. | An efficient residual learning neural network for hyperspectral image superresolution | |
| Wang et al. | Dilated projection correction network based on autoencoder for hyperspectral image super-resolution | |
| CN115272093A (en) | Hyperspectral image unmixing method based on spatial structure information constraint | |
| CN116343052A (en) | Attention and multiscale-based dual-temporal remote sensing image change detection network | |
| Mantripragada et al. | The effects of spectral dimensionality reduction on hyperspectral pixel classification: A case study | |
| Qin et al. | Deep ResNet based remote sensing image super-resolution reconstruction in discrete wavelet domain | |
| Wang et al. | Image super-resolution using multi-granularity perception and pyramid attention networks | |
| Deng et al. | Multiple frame splicing and degradation learning for hyperspectral imagery super-resolution | |
| Liu et al. | Cross-resolution feature attention network for image super-resolution | |
| Nasrollahi et al. | Deep artifact-free residual network for single-image super-resolution | |
| Lei et al. | Convolution neural network with edge structure loss for spatiotemporal remote sensing image fusion | |
| CN117726916B (en) | Implicit fusion method for enhancing image resolution fusion | |
| Ahmadian et al. | Single image super-resolution with self-organization neural networks and image laplace gradient operator | |
| Liang et al. | Blind super-resolution of single remotely sensed hyperspectral image | |
| Liu et al. | Frequency separation-based multi-scale cascading residual block network for image super resolution | |
| Zhang et al. | Thermal image super-resolution via multi-path residual attention network | |
| Guo et al. | A novel lightweight multi-dimension feature fusion network for single-image super-resolution reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |