CN112669248A - Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid - Google Patents
Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid Download PDFInfo
- Publication number
- CN112669248A CN112669248A CN202011576102.XA CN202011576102A CN112669248A CN 112669248 A CN112669248 A CN 112669248A CN 202011576102 A CN202011576102 A CN 202011576102A CN 112669248 A CN112669248 A CN 112669248A
- Authority
- CN
- China
- Prior art keywords
- image
- layer
- hyperspectral
- spatial
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 32
- 230000004927 fusion Effects 0.000 claims abstract description 73
- 238000000605 extraction Methods 0.000 claims abstract description 23
- 238000001228 spectrum Methods 0.000 claims abstract description 20
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 10
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 230000006870 function Effects 0.000 claims description 56
- 238000013527 convolutional neural network Methods 0.000 claims description 33
- 230000004913 activation Effects 0.000 claims description 23
- 238000005070 sampling Methods 0.000 claims description 17
- 238000010586 diagram Methods 0.000 claims description 12
- 238000010606 normalization Methods 0.000 claims description 11
- 230000007704 transition Effects 0.000 claims description 7
- 230000003213 activating effect Effects 0.000 claims description 4
- 238000000034 method Methods 0.000 abstract description 51
- 230000003595 spectral effect Effects 0.000 abstract description 30
- 238000012545 processing Methods 0.000 abstract description 5
- 238000002474 experimental method Methods 0.000 description 9
- 238000004458 analytical method Methods 0.000 description 7
- 238000000513 principal component analysis Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 239000000243 solution Substances 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004445 quantitative analysis Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000013398 bayesian method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 230000001976 improved effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000000116 mitigating effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of image processing, discloses a hyperspectral and full-color image fusion method based on CNN and Laplacian pyramid, and solves the problems of insufficient extraction of space detail information and excessive loss of spectrum information in the conventional hyperspectral and full-color image fusion method. The hyperspectral image and full-color image fusion method comprises the following implementation steps: acquiring an image data set, and preprocessing the data; carrying out full-color image decomposition by utilizing a Laplacian pyramid structure; extracting spatial details by using a Dense block network; combining the low-spatial-resolution hyperspectral image with the extracted spatial detail features to reconstruct the image; and carrying out model training by using the loss function to obtain a final hyperspectral fusion image with high spatial resolution. The method can fully extract the spatial detail characteristics while reducing the loss of spectral information, improves the spatial information of the image, and can obtain a high-quality hyperspectral fusion image.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a hyperspectral and full-color image fusion method based on CNN and Laplacian pyramid.
Background
At present: the hyperspectral technology represents the development front of the field of remote sensing, but due to the limitation of some physical conditions, a single hyperspectral sensor cannot obtain an image with high spatial resolution and rich spectral information. Full color images imaged in a single electromagnetic spectrum have high spatial resolution but low spectral diversity. The spectral information of hyperspectral images containing dozens or even hundreds of spectral band information is abundant, but the spatial resolution is low. Therefore, it is known that the single full-color image and the single hyperspectral image cannot meet the requirements of performing the problems of target detection, change detection and the like, and therefore, it is necessary to integrate the spatial information of the full-color image and the spectral information of the hyperspectral image by the image fusion technology.
At present, a hyperspectral image fusion technology becomes a research hotspot in the field of remote sensing image processing. Researchers have developed and designed many classical hyperspectral image fusion algorithms, and representative methods mainly include a component replacement method, a multiresolution analysis method, a Bayesian method, a matrix decomposition method and the like. The fusion algorithm based on Component replacement mainly comprises Schmidt-Schmidt (GS), Principal Component Analysis (PCA) and the like, and the main idea is to map a hyperspectral image to a new space to separate a spectral Component and a spatial Component, and replace the spatial Component separated from the hyperspectral image with the spatial Component of a full-color image. This method performs well in terms of spatial resolution, but results in some loss of spectral information resulting in spectral distortion. The fusion algorithm based on multi-resolution analysis mainly comprises a Laplacian Pyramid (LP) and a smooth filtering method (SFIM) based on brightness adjustment, and the fusion algorithm mainly has the main idea that spatial details lost in a hyperspectral image are extracted from a full-color image and are injected into the hyperspectral image. This method can maintain good spectral information, but has a certain spatial distortion and may even have the problem of injecting too much spatial detail. The fusion algorithm based on the Bayesian method comprises Bayesian Sparse Representation (BSR), Hysure and the like, and the main idea is to consider the statistical relationship between the panchromatic image and the hyperspectral image, calculate the maximum posterior probability by using a Bayesian formula, and obtain the reconstructed image with high spatial resolution and high spectral resolution as the fusion result. Although the method has better performance in the aspects of spatial resolution and spectral information, the ideal effect can be obtained only by relying on enough prior information, and the method has large computation amount and is inconvenient to implement. The fusion algorithm based on the Matrix decomposition method is typically a Coupled non-negative Matrix decomposition (CNMF), and the main idea is to perform non-negative Matrix decomposition on two complex input image data sources to obtain end members and abundance maps of the two data sources, and then alternately fuse the abundance of the multispectral image and the end member information of the hyperspectral image, so as to generate a better fusion result in the aspects of space and spectrum.
In recent years, deep learning has been successfully applied to many fields of image processing, particularly the field of image super-resolution, showing great potential. Dong et al, in the document "Image super-resolution using deep-visual networks," IEEE trans. pattern an. mach. inner, vol.38, No.2, pp.295-307, feb.2015 ", propose a first deep CNN-based Image super-resolution (SRCNN) method that shows better performance than some classical fusion methods. The purpose of super-resolution and image fusion is to improve the image resolution. Inspired by the success of CNNs in image super-resolution, Masi et al successfully applied SRCNNs to image fusion in the document "panschpening by volumetric neural networks," Remote sens, vol.8, No.7, pp.594, 2016 ". On the basis of this, Wei, Y et al also propose some residual learning-based image fusion methods in the literature "Boosting the acquisition of multispectral images panying by left learning a deep residual network," IEEE geosci.remote sens.lett., vol.14, No.10, pp.1795-1799, oct.2017 ", which provide for image fusion by deeper CNNs. Due to the complexity of different spectral responses and ground features, a classic image fusion algorithm is difficult to obtain a fusion result with rich spectral information and good spatial resolution performance, and the high nonlinearity of the CNN provides a new solution to the problem.
The success of CNN-based approaches demonstrates the great potential of CNN in solving the fusion problem. However, almost all CNN-based fusion methods are implemented by changing the network architecture for image super-resolution. These methods essentially take a full color image as one input to the network and take it as an additional band of the hyperspectral image. CNN networks act like a black box, treating each band indiscriminately. Therefore, they cannot sufficiently extract spatial details of a full-color image, which play an important role in image fusion.
Through the above analysis, the problems and defects of the prior art are as follows: all the fusion methods based on the CNN are realized by changing a network architecture of image super-resolution, the CNN network treats each wave band indiscriminately like a black box, the space details of a full-color image cannot be fully extracted, and the space details play an important role in image fusion.
The difficulty in solving the above problems and defects is: due to the complexity of different spectral responses and ground features, the relationship between a high-resolution hyperspectral image, a full-color image and a low-resolution hyperspectral image is difficult to establish by a classical hyperspectral image fusion algorithm. Due to the limitation of a network architecture, the existing CNN-based method is difficult to fully extract the detailed features of the full-color image.
The significance of solving the problems and the defects is as follows: in the hyperspectral image fusion process, the extraction of detail features is very important, the quality of a final fusion result is directly determined, the problems of artifacts and the like can be caused if the space details are injected too much, and the problems of image blurring and the like can be caused if the space details are not extracted sufficiently. Therefore, it is necessary to select a fusion method which can not only maintain the spectral information of the hyperspectral image, but also effectively improve the spatial resolution of the image.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a method for fusing a hyperspectral image and a full-color image.
The invention is realized by a method for fusing a hyperspectral image and a full-color image, which comprises the following steps:
acquiring an image data set, and preprocessing the data;
the method comprises the steps of obtaining a high-resolution hyperspectral image, generating a low-spatial-resolution hyperspectral image through spatial downsampling, generating a full-color image through spectral sampling, dividing the obtained image according to a certain number of proportions, and obtaining enough training images and testing images for subsequent operation.
Carrying out full-color image decomposition by utilizing a Laplacian pyramid structure;
the laplacian pyramid structure decomposes the original full-color image into sub full-color images with different sizes, so that the spatial detail features of the full-color images of all layers can be conveniently extracted by each layer of sub-network, and the more comprehensive and sufficient full-color image spatial detail features are finally obtained for the subsequent image reconstruction process.
Extracting spatial details by using a Dense block network;
DenseNet avoids the gradient vanishing problem by propagating the output of each layer to all subsequent layers, effectively mitigating the propagation of forward and reverse information during network training, which can be used to train very deep networks. In the invention, by using a Dense block network, the detail information of full-color images of all sizes can be fully extracted.
Combining the low-spatial-resolution hyperspectral image with the extracted spatial detail features to reconstruct the image;
the original low-resolution hyperspectral image is sampled step by step and fused with the spatial detail features from the detail feature extraction network, so that the process of gradually reconstructing the hyperspectral image from low resolution to high resolution is realized, rich spectral information is kept, and comprehensive spatial detail information is added.
And carrying out model training by using the loss function to obtain a final hyperspectral fusion image with high spatial resolution.
The difference between the hyperspectral fusion image obtained by the method and the reference high-resolution hyperspectral image can be quantitatively divided by using a loss function so as to estimate the prediction quality of the network on the training data. The reference high-resolution hyperspectral images corresponding to the reconstructed images of different layers can be obtained by down-sampling the reference high-resolution hyperspectral images, so that the loss of the different layers can be calculated, and the most ideal hyperspectral fusion image with the minimum difference with the reference hyperspectral images is obtained.
Further, the hyperspectral image and full-color image fusion method specifically comprises the following steps:
firstly, acquiring an image data set, and preprocessing data;
(1) obtaining a hyperspectral image H from the published dataset as a reference to the original hyperspectral imageREF；
(2) Obtaining a required full-color image P and a low-spatial-resolution hyperspectral image H by using a given reference hyperspectral image according to a Wald protocol;
secondly, forming a spatial feature extraction network by using a Laplacian pyramid, and decomposing an original full-color image into a plurality of layers, wherein each layer of image has different sizes;
(1) sequentially downsampling the input high-spatial-resolution panchromatic image P to generate 3 sub-panchromatic images S (P) with different sizes;
(2) up-sampling the (I + 1) th layer image after down-sampling to the size of the original I-th layer image, and subtracting the pixel points of the two obtained images to obtain the corresponding I-th layer Laplacian pyramid output Ii(P)：
Ii(P)＝Si(P)-upsample(Si+1(P))；
Wherein i is 1,2,3, Si(P) represents the ith Laplacian pyramid, S1(P) ═ P, upsample () represents an upsample operation;
thirdly, constructing a Dense block pyramid structure for extracting high-frequency detail information of the full-color image, and extracting spatial detail features;
(1) building three layers of pyramid-structured Dense block sub-networks based on a DenseNet convolutional neural network architecture;
(2) full-color image I with different Laplacian layersi(P) respectively sending the depth feature map into a Dense block subnetwork corresponding to each layer for spatial detail extraction to obtain a depth feature map
(3) Inputting the depth characteristic diagram obtained by each layer into a 3 multiplied by 3 convolution layer, and obtaining the final required detail characteristic diagram by taking a nonlinear function Relu as an activation function
Step four, the low-spatial resolution hyperspectral image is up-sampled step by step and is fused with the details obtained from the spatial detail extraction branch for image reconstruction;
the fifth step is to
The error loss function and the (1-CC) loss function are used as loss functions between the hyperspectral fusion image and the reference high-resolution hyperspectral image and used for estimating the difference between the output fusion image and the reference real image to obtain an ideal output fusion image; for layers 2 and 3 only consideration is given
Error loss function, layer 1 simultaneous consideration
Error loss function and (1-CC) loss function:
wherein Q represents the number of training samples, F1＝F，HREFiRepresenting a reference picture by 2i-1Image obtained by multiplying down-sampling, HREF1＝HREF。
Further, the Dense block subnetwork structure for constructing the three-layer pyramid structure based on the DenseNet convolutional neural network architecture is as follows: the Dense block subnetworks in the 1,2 and 3-level pyramids respectively comprise 8, 6 and 4 volume blocks, and each volume block comprises a batch normalization layer and 32 volume layers with the size of 3 x 3 and takes a nonlinear function Relu as an activation function; the transition layer is implemented as a combination of batch normalization, 1 x 1 convolution kernel, and Relu activation function, concatenated in each Dense block with the feature maps from the respective convolution layer with the feature maps from all previous layers to implement the DenseNet structure.
Further, a full-color image I of different Laplace layersi(P) respectively sending the space details into a Dense block subnetwork corresponding to each layer for space detail extraction, and performing the following steps:
1) the full-color image output by the Laplacian pyramid of the i-th layer enters a Dense block sub-network, and the mth convolution block in the network generates a feature map
Wherein
Representing the feature map generated by the p-th convolutional layer in the depth block in the i-th laplacian pyramid,
representing a join operation with the previous all-layer profile, fTBatch normalization, activating function and 3 multiplied by 3 convolution continuous operation;
the feature map generated by each convolution block passes through a transition layer to obtain the feature map output by the Dense block subnetwork of the ith Laplacian pyramid
Wherein q isiIndicates the number of the convolution blocks in the Laplacian pyramid of the ith layer, fTBatch normalization, activation function and 1 × 1 convolution continuous operation;
2) the feature map obtained by the 2 nd layer Dense block subnetwork of the pyramid is up-sampled and is added with the feature map obtained by the 1 st layer Dense block subnetwork pixel by pixel, and a corresponding depth feature map is obtained through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function
The feature map obtained by the pyramid depth block subnetwork of the 3 rd layer is up-sampled and is added with the feature map obtained by the depth block subnetwork of the 2 nd layer pixel by pixel, and a corresponding depth feature map is obtained through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function
The feature map obtained from the pyramid layer 3 Dense block subnetwork directly passes through the convolution layer with the size of 3 multiplied by 3 andobtaining corresponding depth characteristic map by nonlinear Relu activation function
Further, the low spatial resolution hyperspectral image is up-sampled step by step and is fused with details obtained from the spatial detail extraction branch, and the method comprises the following steps:
1) detailed feature map obtained from layer 3
Adding the high-resolution high-spectrum image H with the input low-resolution high-spectrum image to obtain a 3 rd-layer high-spectrum fusion image F3：
2) Fusing the 3 rd layer high spectrum into an image F3Upsampling and matching with layer 2 derived detail feature maps
Adding pixel points by pixel points to obtain a hyperspectral fusion image F of the 2 nd layer2：
3) Fusing the 2 nd layer high spectrum into an image F2Upsampling and matching with layer 1 derived detail feature maps
Adding pixel points one by one, and obtaining a final hyperspectral fusion image F by passing the obtained fusion result through a convolution layer with the size of 3 x 3 and a Relu activation layer and then passing through a convolution layer with the size of 3 x 3:
further, CC in the (1-CC) loss function is a cross-correlation coefficient that characterizes geometric distortion by calculating the similarity between corresponding bands of the reference image and the fused image, and the optimal value of CC is 1.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention relates to a hyperspectral and full-color image fusion method based on CNN and Laplace pyramid, which aims to improve the spatial resolution of a hyperspectral image and provide a high-quality image for the fields of target detection, change detection and the like.
The invention has definite physical explanation to the fusion process, follows the general idea of multi-resolution analysis algorithm, and divides the fusion into two continuous processes: and extracting and reconstructing details. Firstly, extracting the space detail characteristics of the full-color image by utilizing the Laplacian pyramid, and then injecting the space detail into the low-resolution high-spectrum image to realize the reconstruction of the image. In the detail extraction module, the full-color image is decomposed into a plurality of layers by utilizing the Laplacian pyramid, so that the details of different layers can be distinguished, a simple detail extraction sub-network is constructed for each layer, and the depth features of different layers can be fully extracted. In the reconstruction module, the spatial detail features obtained for each layer are injected into the corresponding hyperspectral image, while they are upsampled and input into the next subnetwork, which can help to fully exploit the complementary details between the different levels.
In the invention, a DenseNet convolutional neural network architecture is utilized, and DenseNet transmits the output of each layer to the subsequent layer, so that the characteristic information of the initial layer is not lost, meanwhile, the transmission of forward and reverse information during the network training period is effectively reduced, all layers in the network are allowed to talk with each other, the characteristic transmission is enhanced, and the network efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
FIG. 1 is a flow chart of a method for fusing a hyperspectral image and a panchromatic image according to an embodiment of the invention.
FIG. 2 is a flowchart of an implementation of a method for fusing a hyperspectral image and a panchromatic image according to an embodiment of the invention.
Fig. 3 is a structural diagram of a laplacian pyramid panchromatic image decomposition network according to an embodiment of the present invention.
Fig. 4 is a three-layer Dense block diagram for detail feature extraction according to an embodiment of the present invention.
Fig. 5 is a structural diagram of a basic densnet convolutional neural network module constituting a three-layer Dense block diagram in fig. 4 according to an embodiment of the present invention.
FIG. 6 is a structural diagram of image reconstruction performed by injecting full-color image detail features into a hyperspectral image according to an embodiment of the invention.
FIG. 7 is a graph showing the fusion results of the present invention and the existing six fusion methods on the Cave data set on low-resolution hyperspectral images and full-color images of the same scene.
Fig. 8 is a graph of the fusion results of the present invention and the prior six fusion methods on the Pavia Center dataset for low resolution hyperspectral images and full-color images of the same scene.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Aiming at the problems in the prior art, the invention provides a method for fusing a hyperspectral image and a full-color image, and the invention is described in detail below by combining the attached drawings.
As shown in fig. 1, the method for fusing the hyperspectral image and the panchromatic image provided by the invention comprises the following steps:
s101: acquiring an image data set, and preprocessing the data;
s102: carrying out full-color image decomposition by utilizing a Laplacian pyramid structure;
s103: extracting spatial details by using a Dense block network;
s104: combining the low-spatial-resolution hyperspectral image with the extracted spatial detail features to reconstruct the image;
s105: and carrying out model training by using the loss function to obtain a final hyperspectral fusion image with high spatial resolution.
The hyperspectral image and panchromatic image fusion method provided by the invention can be implemented by other steps by persons skilled in the art, and the hyperspectral image and panchromatic image fusion method provided by the invention in fig. 1 is only a specific example.
The technical solution of the present invention is further described below with reference to the accompanying drawings.
As shown in fig. 2, the hyperspectral and panchromatic image fusion method based on CNN and laplacian pyramid provided by the invention includes the following stages: the method comprises the steps of image preprocessing, full-color image decomposition by utilizing a Laplacian pyramid, construction of a DenseNet convolutional neural network architecture-based space detail feature extraction network, image reconstruction and model training by utilizing a loss function.
The method comprises the following steps: an image data set is acquired and the data is preprocessed.
1.1) obtaining an original hyperspectral image from the published data set as a reference hyperspectral image.
1.2) obtaining a required full-color image and a low spatial resolution hyperspectral image according to the Wald protocol by utilizing a given reference hyperspectral image.
Step two: the full-color image is decomposed using the laplacian pyramid.
As shown in fig. 3, the laplacian pyramid is a three-layer network architecture, and performs down-sampling on an input full-color image, decomposes the original full-color image into multiple layers, and connects two adjacent layers of full-color images through up-sampling and pixel-by-pixel subtraction to obtain more accurate detail features of the full-color image. The concrete implementation is as follows:
2.1) downsampling the high spatial resolution panchromatic image P in sequence to generate 3 sub-panchromatic images S (P) of different sizes.
2.2) up-sampling the down-sampled 2 nd layer image to the size of the original full-color image, and subtracting the pixel points of the two obtained images one by one to obtain the corresponding 1 st layer Laplacian pyramid output I1(P)。
The 3 rd layer image after down sampling is up sampled to the size of the 2 nd layer image, and the obtained two images are subtracted pixel by pixel to obtain the corresponding 2 nd layer Laplacian pyramid output I2(P)。
The 4 th layer image after down sampling is up sampled to the size of the 3 rd layer image, and the obtained two images are subtracted pixel by pixel to obtain the corresponding 3 rd layer Laplacian pyramid output I3(P)
Ii(P)＝Si(P)-upsample(Si+1(P))；
Wherein i is 1,2,3, Si(P) represents the ith Laplacian pyramid, S1(P) ═ P, upsample () represents an upsample operation.
Step three: and constructing a spatial detail feature extraction network based on the DenseNet convolutional neural network architecture.
As shown in fig. 4, the spatial detail feature extraction network based on the DenseNet convolutional neural network architecture is a convolutional neural network comprising 3 sense blocks, and each sense block sequentially performs spatial detail extraction on each layer of panchromatic image. The specific implementation of the step is as follows:
3.1) building three-layer pyramid structured Dense block sub-networks based on a DenseNet convolutional neural network architecture.
3.1.1) Dense block subnetworks in the 1 st, 2 nd, 3 rd level pyramid contain 8, 6, 4 volume blocks, respectively. As shown in fig. 5, for each volume block, one batch normalization layer and 32 volume layers of size 3 × 3 are included and the nonlinear function Relu is the activation function. To reduce feature set dimensions, a "transition layer" is connected to the last layer of the volume block. The transition layer is implemented as a combination of batch normalization, a 1 x 1 convolution kernel, and a Relu activation function. The DenseNet structure is implemented in each volume block by stitching the feature maps of the layers with the feature maps from all previous layers.
3.2) full-color image I with different Laplace layersiAnd (P) respectively sending the data into a Dense block subnetwork corresponding to each layer for spatial detail extraction.
3.2.1) the full-color image output by the Laplacian pyramid of the ith layer enters a Dense block sub-network, and the mth convolution block in the network generates a feature map
Wherein
Representing the feature map generated by the p-th convolutional layer in the depth block in the i-th laplacian pyramid,
representing a join operation with the previous all-layer profile, fTBatch normalization, activating function and 3 multiplied by 3 convolution continuous operation;
the feature map generated by each convolution block passes through a transition layer to obtain the feature map output by the Dense block subnetwork of the ith Laplacian pyramid
Wherein q isiIndicating the number of the rolling blocks in the ith Laplacian pyramid. f. ofTIs to make one in batchesContinuous operation of changing and activating functions and 1 multiplied by 1 convolution;
3.2.2) feature maps obtained by the pyramid 2 nd layer Dense block subnetwork are up-sampled and added with the feature maps obtained by the 1 st layer Dense block subnetwork pixel by pixel, and then the feature maps are obtained by a convolution layer with the size of 3 x 3 and a nonlinear Relu activation function to obtain corresponding depth feature maps
The feature map obtained by the pyramid depth block subnetwork of the 3 rd layer is up-sampled and is added with the feature map obtained by the depth block subnetwork of the 2 nd layer pixel by pixel, and a corresponding depth feature map is obtained through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function
The feature map obtained by the pyramid layer 3 Dense block subnetwork directly passes through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function to obtain a corresponding depth feature map
3.3) inputting the depth characteristic map obtained by each layer into a 3 x 3 convolutional layer, and obtaining a final required detail characteristic map by taking a nonlinear function Relu as an activation function
Step four: and (5) image reconstruction.
As shown in fig. 6, the image reconstruction network performs image reconstruction by up-sampling the low-spatial-resolution hyperspectral image stage by stage and fusing the up-sampled low-spatial-resolution hyperspectral image with the details obtained from the spatial detail extraction branch. The specific implementation of the step is as follows:
4.1) detail feature map obtained from the third layer
Adding the high-resolution high-spectrum image H with the input low-resolution high-spectrum image one by one to obtain a third-layer high-spectrum fusion image F3：
4.2) fusing the third layer of high spectrum with an image F3Up-sampling and obtaining detail characteristic diagram with second layer
Adding pixel points by pixel points to obtain a hyperspectral fusion image F of a second layer2：
4.3) fusing the second layer high spectrum into an image F2Detail characteristic diagram G obtained by up-sampling and combining with the first layer1 PAdding pixel points one by one, and obtaining a final hyperspectral fusion image F by passing the obtained fusion result through a convolution layer with the size of 3 x 3 and a Relu activation layer and then passing through a convolution layer with the size of 3 x 3:
step five: and carrying out model training by using the loss function.
5.1) to
Error loss function and (1-CC) loss function as between hyperspectral fused image and reference high-resolution hyperspectral imageAnd the loss function is used for estimating the difference between the output fusion image and the reference real image so as to obtain an ideal output fusion image. For layers 2 and 3 only consideration is given
Error loss function, layer 1 simultaneous consideration
Error loss function and (1-CC) loss function:
wherein Q represents the number of training samples, F1＝F，
Representing a reference picture by 2i-1The image obtained by the down-sampling is multiplied,
CC is a cross-correlation coefficient that characterizes geometric distortion by calculating the similarity between corresponding bands of the reference image and the fused image, with an optimal value of CC of 1.
The effect of the present invention will be further explained with the simulation experiment.
First, experimental conditions
The first data set used in the experiment was the Cave data set, which contained 31 bands of hyperspectral images within the spectral range of 400-700 nm. In this experiment, the size of a full-color image obtained by the Wald protocol and the segmentation processing was 128 × 128, and the size of a low-resolution hyperspectral image was 32 × 32.
The second dataset used in the experiment was the Pavia Center dataset, which contained hyperspectral images in 115 spectral bands within the wavelength range of 430-860 nm. 13 of these water absorption and noise bands were discarded, and the remaining 102 bands were used for the experiment. The full-color image obtained by the Wald protocol and the segmentation process in this experiment was 160 × 160, and the low-resolution hyperspectral image was 40 × 40.
Second, the experimental contents
Experiment 1: the first experiment was performed on a Cave dataset, and the low-resolution hyperspectral image and the same-scene panchromatic image were fused using the present invention and the existing six fusion methods, the result is shown in fig. 7, where:
figure 7(a) is an original high resolution hyperspectral image,
FIG. 7(b) is a graph showing the fusion result of the conventional GS method,
FIG. 7(c) is a graph showing the fusion result of the conventional PCA method,
FIG. 7(d) is a graph showing the fusion result of the conventional GFPCA method,
FIG. 7(e) is a graph showing the fusion result of the conventional BSF method,
FIG. 7(f) is a graph showing the fusion result of the conventional CNMF method,
FIG. 7(g) is a graph showing the fusion result of the conventional MGH method,
FIG. 7(h) is a graph showing the result of fusion using the present invention.
Referring to the results in fig. 7, analysis can find that the GS and MGH methods yield results that exhibit good spectral fidelity, but image edges are subject to artifacts. The results obtained by the PCA method are obviously lack of spectral information, and the results obtained by the CNMF and the BSF have obvious color darkening problems. The GFPCA method results in a blur due to too little spatial detail injection. The results obtained by the present invention can provide the highest fidelity of spectral information and the clearest details.
In order to further evaluate the performance of the comparison method, objective quantitative analysis is carried out on the fusion result of the method, correlation coefficient CC, spectrum angle SAM, mean square error RMSE and global relative error ERGAS indexes are respectively calculated, and the fusion effect is evaluated from two aspects of space and spectrum. Table 1 shows objective quantitative analysis of the Cave data set, and respectively lists performance indexes of the invention and the existing six fusion methods for the Cave data set hyperspectral image fusion result.
Table 1. Performance indexes of fusion results of hyperspectral images of Cave data sets by the invention and six existing fusion methods
As can be seen from the table, the cross-correlation coefficient CC value of the invention is the largest, which shows that the hyperspectral image space information fused by the invention is the richest; the smallest spectral angle SAM of the invention indicates that the invention has the best performance in the aspect of spectral information retention; the root mean square error RMSE and the overall synthetic dimension error ERGAS of the method are minimum, which shows that the method is superior to all other methods in the aspect of global index evaluation.
Experiment 2: the second experiment was performed on a Pavia Center dataset, and the low resolution hyperspectral image was fused with the co-scene panchromatic image using the present invention and the existing six fusion methods, the results are shown in fig. 8, where:
figure 8(a) is an original high resolution hyperspectral image,
FIG. 8(b) is a graph showing the fusion result of the conventional GS method,
FIG. 8(c) is a graph showing the fusion result of the conventional PCA method,
FIG. 8(d) is a graph showing the fusion result of the conventional GFPCA method,
FIG. 8(e) is a graph showing the fusion result of the conventional BSF method,
FIG. 8(f) is a graph showing the fusion result of the conventional CNMF method,
FIG. 8(g) is a graph showing the fusion result of the conventional MGH method,
FIG. 8(h) is a graph showing the result of fusion using the present invention.
Referring to the results in fig. 8, the analysis can find that there are different degrees of color distortion in the results obtained by GS, PCA, CNMF methods. The results obtained by the GFPCA, MGH and BSF methods have obvious fuzzy artifacts. The result obtained by the invention can effectively retain the spectral information of the original image while improving the spatial resolution.
Table 2 shows objective quantitative analysis of the Pavia Center dataset, and lists performance indexes of the fusion result of the invention and the hyperspectral image of the Pavia Center dataset by using the existing six fusion methods.
Table 2. Performance indexes of results of fusion of Pavia Center data set hyperspectral images by the invention and six existing fusion methods
As can be seen from the table, the cross-correlation coefficient CC value of the invention is the largest, which shows that the fused hyperspectral image space detail performance is the best; the spectral angle SAM of the invention is minimum, which shows that the loss of spectral information of the invention is minimum; the root mean square error RMSE and the synthetic dimension total error ERGAS of the invention are minimum, which shows that the error of the fusion result and the original hyperspectral image of the invention is very minimum.
The above simulation experiments show that: the invention provides a hyperspectral image and full-color image fusion method based on a CNN and Laplacian pyramid network, and solves the problems that space detail information is not fully extracted and spectrum information is excessively lost in the conventional hyperspectral and full-color image fusion method. According to the method, the full-color image is decomposed into a plurality of layers by utilizing the Laplacian pyramid, so that the details of different layers can be distinguished, a simple detail extraction sub-network is constructed for each layer, and the spatial detail features of different layers can be fully extracted. According to simulation result analysis, the fusion index and the visual effect of the hyperspectral imager are good in space and spectral performance, the space resolution is effectively improved while good spectral information is kept, and a high-quality hyperspectral fusion image can be obtained.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided on a carrier medium such as a disk, CD-or DVD-ROM, programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.
Claims (6)
1. A CNN and Laplace pyramid-based hyperspectral and panchromatic image fusion method is characterized by comprising the following steps:
acquiring an image data set, and preprocessing the data;
carrying out full-color image decomposition by utilizing a Laplacian pyramid structure;
extracting spatial details by using a Dense block network;
combining the low-spatial-resolution hyperspectral image with the extracted spatial detail features to reconstruct the image;
and carrying out model training by using the loss function to obtain a final hyperspectral fusion image with high spatial resolution.
2. The CNN and laplacian pyramid based hyperspectral and panchromatic image fusion method of claim 1, comprising in particular:
firstly, acquiring an image data set, and preprocessing data;
(1) obtaining a hyperspectral image H from the published dataset as a reference to the original hyperspectral imageREF；
(2) Obtaining a required full-color image P and a low-spatial-resolution hyperspectral image H by using a given reference hyperspectral image according to a Wald protocol;
secondly, forming a spatial feature extraction network by using a Laplacian pyramid, and decomposing an original full-color image into a plurality of layers, wherein each layer of image has different sizes;
(1) sequentially downsampling the input high-spatial-resolution panchromatic image P to generate 3 sub-panchromatic images S (P) with different sizes;
(2) up-sampling the (I + 1) th layer image after down-sampling to the size of the original I-th layer image, and subtracting the pixel points of the two obtained images to obtain the corresponding I-th layer Laplacian pyramid output Ii(P)：
Ii(P)＝Si(P)-upsample(Si+1(P))；
Wherein i is 1,2,3, Si(P) represents the ith Laplacian pyramid, S1(P) ═ P, upsample () represents an upsample operation;
thirdly, constructing a Dense block pyramid structure for extracting the full-color image space detail information, and extracting the space detail characteristics;
(1) building three layers of pyramid-structured Dense block sub-networks based on a DenseNet convolutional neural network architecture;
(2) full-color image I with different Laplacian layersi(P) respectively sending the depth feature map into a Dense block subnetwork corresponding to each layer for spatial detail extraction to obtain a depth feature map
(3) Inputting the depth characteristic diagram obtained by each layer into a 3 multiplied by 3 convolution layer, and obtaining the final required detail characteristic diagram by taking a nonlinear function Relu as an activation function
Step four, the low-spatial resolution hyperspectral image is up-sampled step by step and is fused with the details obtained from the spatial detail extraction branch for image reconstruction;
the fifth step is to1The error loss function and the (1-CC) loss function are used as loss functions between the hyperspectral fusion image and the reference high-resolution hyperspectral image and used for estimating the difference between the output fusion image and the reference real image to obtain an ideal output fusion image; consider only l for layers 2 and 31Error loss function, layer 1 taking into account l simultaneously1Error loss function and (1-CC) loss function:
wherein Q represents the number of training samples, F1＝F，HREFiRepresenting a reference picture by 2i-1The image obtained by the down-sampling is multiplied,
3. the CNN and laplacian pyramid based hyperspectral and panchromatic image fusion method of claim 2, wherein the Dense block subnetwork structure for constructing the three-layer pyramid structure based on the DenseNet convolutional neural network architecture is as follows: the Dense block subnetworks in the 1,2 and 3-level pyramids respectively comprise 8, 6 and 4 volume blocks, and each volume block comprises a batch normalization layer and 32 convolution kernels with the size of 3 x 3 and takes a nonlinear function Relu as an activation function; joining one transition layer is implemented as a combination of batch normalization, 1 x 1 convolution kernel, and Relu activation function. In each density block, the density structure is implemented by concatenating the characteristic maps of the individual convolutional layers with the characteristic maps from all preceding layers.
4. The CNN and laplacian pyramid based hyperspectral and panchromatic image fusion method of claim 2, wherein different laplacian layer panchromatic images I are fusedi(P) respectively sending the space details into a Dense block subnetwork corresponding to each layer for space detail extraction, and performing the following steps:
1) the full-color image output by the Laplacian pyramid of the i-th layer enters a Dense block sub-network, and the mth convolution block in the network generates a feature map
Wherein
Representing the feature map generated by the p-th convolutional layer in the depth block in the i-th laplacian pyramid,
representing a join operation with the previous all-layer profile, fTBatch normalization, activating function and 3 multiplied by 3 convolution continuous operation;
the feature map generated by each convolution block passes through a transition layer to obtain the feature map output by the Dense block subnetwork of the ith Laplacian pyramid
Wherein q isiIndicating the ith layer of drawingNumber of convolution blocks in the Prasiian pyramid, fTBatch normalization, activation function and 1 × 1 convolution continuous operation;
2) the feature map obtained by the 2 nd layer Dense block subnetwork of the pyramid is up-sampled and is added with the feature map obtained by the 1 st layer Dense block subnetwork pixel by pixel, and a corresponding depth feature map is obtained through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function
The feature map obtained by the pyramid depth block subnetwork of the 3 rd layer is up-sampled and is added with the feature map obtained by the depth block subnetwork of the 2 nd layer pixel by pixel, and a corresponding depth feature map is obtained through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function
The feature map obtained by the pyramid layer 3 Dense block subnetwork directly passes through a convolution layer with the size of 3 multiplied by 3 and a nonlinear Relu activation function to obtain a corresponding depth feature map
5. The CNN and laplacian pyramid based hyperspectral and panchromatic image fusion method of claim 2, wherein the low spatial resolution hyperspectral image is up-sampled stage by stage and fused with details from the spatial detail extraction branch by the following steps:
1) detailed feature map obtained from layer 3
With the input low-resolution hyperspectral image H by HAdding pixel points to obtain a 3 rd-layer high-spectrum fusion image F3：
2) Fusing the 3 rd layer high spectrum into an image F3Upsampling and matching with layer 2 derived detail feature maps
Adding pixel points by pixel points to obtain a hyperspectral fusion image F of the 2 nd layer2：
3) Fusing the 2 nd layer high spectrum into an image F2Upsampling and obtaining detail characteristic diagram G with layer 11 PAdding pixel points one by one, and obtaining a final hyperspectral fusion image F by passing the obtained fusion result through a convolution layer with the size of 3 x 3 and a Relu activation layer and then passing through a convolution layer with the size of 3 x 3:
F＝f′cnn{fcnn[upsampling(F2)+G1 P]}。
6. the CNN and laplacian pyramid based hyperspectral and panchromatic image fusion method of claim 2 wherein CC in the (1-CC) loss function is a cross-correlation coefficient that characterizes geometric distortion by calculating similarity between corresponding bands of the reference image and the fused image, with an optimal value of CC being 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011576102.XA CN112669248B (en) | 2020-12-28 | 2020-12-28 | Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011576102.XA CN112669248B (en) | 2020-12-28 | 2020-12-28 | Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112669248A true CN112669248A (en) | 2021-04-16 |
| CN112669248B CN112669248B (en) | 2024-03-05 |
Family
ID=75410225
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011576102.XA Active CN112669248B (en) | 2020-12-28 | 2020-12-28 | Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112669248B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114240814A (en) * | 2021-12-19 | 2022-03-25 | 西北工业大学 | Multispectral and panchromatic remote sensing image fusion method based on pyramid modulation injection |
| CN114581347A (en) * | 2022-01-24 | 2022-06-03 | 中国科学院空天信息创新研究院 | Optical remote sensing space spectrum fusion method, device, equipment and medium without reference image |
| CN115565037A (en) * | 2022-09-14 | 2023-01-03 | 中国科学院空天信息创新研究院 | Image fusion method and device |
| CN116579959A (en) * | 2023-04-13 | 2023-08-11 | 北京邮电大学 | Fusion imaging method and device for hyperspectral image |
| CN117690096A (en) * | 2024-02-04 | 2024-03-12 | 成都中轨轨道设备有限公司 | Contact net safety inspection system adapting to different scenes |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017215284A1 (en) * | 2016-06-14 | 2017-12-21 | 山东大学 | Gastrointestinal tumor microscopic hyper-spectral image processing method based on convolutional neural network |
| CN109859110A (en) * | 2018-11-19 | 2019-06-07 | 华南理工大学 | The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum |
| CN110428387A (en) * | 2018-11-16 | 2019-11-08 | 西安电子科技大学 | EO-1 hyperion and panchromatic image fusion method based on deep learning and matrix decomposition |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| CN110544212A (en) * | 2019-07-31 | 2019-12-06 | 华南理工大学 | convolutional neural network hyperspectral image sharpening method based on hierarchical feature fusion |
-
2020
- 2020-12-28 CN CN202011576102.XA patent/CN112669248B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017215284A1 (en) * | 2016-06-14 | 2017-12-21 | 山东大学 | Gastrointestinal tumor microscopic hyper-spectral image processing method based on convolutional neural network |
| CN110428387A (en) * | 2018-11-16 | 2019-11-08 | 西安电子科技大学 | EO-1 hyperion and panchromatic image fusion method based on deep learning and matrix decomposition |
| CN109859110A (en) * | 2018-11-19 | 2019-06-07 | 华南理工大学 | The panchromatic sharpening method of high spectrum image of control convolutional neural networks is tieed up based on spectrum |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| CN110544212A (en) * | 2019-07-31 | 2019-12-06 | 华南理工大学 | convolutional neural network hyperspectral image sharpening method based on hierarchical feature fusion |
Non-Patent Citations (2)
| Title |
|---|
| 罗运;夏晓峰;: "基于多尺度金字塔的多光谱图像Pan-Sharpening方法", 西南师范大学学报(自然科学版), no. 06 * |
| 薛洋;曾庆科;夏海英;王文涛;: "基于卷积神经网络超分辨率重建的遥感图像融合", 广西师范大学学报(自然科学版), no. 02 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114240814A (en) * | 2021-12-19 | 2022-03-25 | 西北工业大学 | Multispectral and panchromatic remote sensing image fusion method based on pyramid modulation injection |
| CN114240814B (en) * | 2021-12-19 | 2024-02-23 | 西北工业大学 | Multispectral and panchromatic remote sensing image fusion method based on pyramid modulation injection |
| CN114581347A (en) * | 2022-01-24 | 2022-06-03 | 中国科学院空天信息创新研究院 | Optical remote sensing space spectrum fusion method, device, equipment and medium without reference image |
| CN114581347B (en) * | 2022-01-24 | 2024-03-12 | 中国科学院空天信息创新研究院 | Optical remote sensing spatial spectrum fusion method, device, equipment and medium without reference image |
| CN115565037A (en) * | 2022-09-14 | 2023-01-03 | 中国科学院空天信息创新研究院 | Image fusion method and device |
| CN115565037B (en) * | 2022-09-14 | 2023-08-22 | 中国科学院空天信息创新研究院 | Image fusion method and device |
| CN116579959A (en) * | 2023-04-13 | 2023-08-11 | 北京邮电大学 | Fusion imaging method and device for hyperspectral image |
| CN116579959B (en) * | 2023-04-13 | 2024-04-02 | 北京邮电大学 | Fusion imaging method and device for hyperspectral image |
| CN117690096A (en) * | 2024-02-04 | 2024-03-12 | 成都中轨轨道设备有限公司 | Contact net safety inspection system adapting to different scenes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112669248B (en) | 2024-03-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Cai et al. | Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction | |
| Li et al. | Survey of single image super‐resolution reconstruction | |
| CN110969577B (en) | Video super-resolution reconstruction method based on deep double attention network | |
| CN112669248A (en) | Hyperspectral and panchromatic image fusion method based on CNN and Laplacian pyramid | |
| CN111127374B (en) | Pan-sharing method based on multi-scale dense network | |
| Hui et al. | Progressive perception-oriented network for single image super-resolution | |
| CN110415199B (en) | Multispectral remote sensing image fusion method and device based on residual learning | |
| CN111161146B (en) | Coarse-to-fine single-image super-resolution reconstruction method | |
| Luo et al. | Lattice network for lightweight image restoration | |
| CN111640060A (en) | Single image super-resolution reconstruction method based on deep learning and multi-scale residual dense module | |
| Li et al. | HyperNet: A deep network for hyperspectral, multispectral, and panchromatic image fusion | |
| CN111402138A (en) | Image super-resolution reconstruction method of supervised convolutional neural network based on multi-scale feature extraction fusion | |
| Long et al. | Bishift networks for thick cloud removal with multitemporal remote sensing images | |
| CN114723631A (en) | Image denoising method, system and device based on depth context prior and multi-scale reconstruction sub-network | |
| CN115375711A (en) | Image segmentation method of global context attention network based on multi-scale fusion | |
| Liu et al. | An efficient unfolding network with disentangled spatial-spectral representation for hyperspectral image super-resolution | |
| CN117474781A (en) | High spectrum and multispectral image fusion method based on attention mechanism | |
| CN116486074A (en) | Medical image segmentation method based on local and global context information coding | |
| Chong et al. | High-order Markov random field as attention network for high-resolution remote-sensing image compression | |
| CN117593199A (en) | Double-flow remote sensing image fusion method based on Gaussian prior distribution self-attention | |
| CN116433548A (en) | Hyperspectral and panchromatic image fusion method based on multistage information extraction | |
| Luo et al. | A fast denoising fusion network using internal and external priors | |
| Gupta et al. | A robust and efficient image de-fencing approach using conditional generative adversarial networks | |
| CN115760875A (en) | Full-field medical picture region segmentation method based on self-supervision learning | |
| CN110807746B (en) | Hyperspectral image sharpening method based on detail embedded injection convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |