CN113744136A - Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion - Google Patents
Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion Download PDFInfo
- Publication number
- CN113744136A CN113744136A CN202111165041.2A CN202111165041A CN113744136A CN 113744136 A CN113744136 A CN 113744136A CN 202111165041 A CN202111165041 A CN 202111165041A CN 113744136 A CN113744136 A CN 113744136A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- spatial
- feature
- spectral
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000003595 spectral effect Effects 0.000 claims abstract description 88
- 238000012549 training Methods 0.000 claims abstract description 35
- 238000001228 spectrum Methods 0.000 claims abstract description 32
- 239000013598 vector Substances 0.000 claims abstract description 23
- 238000000605 extraction Methods 0.000 claims abstract description 11
- 230000008859 change Effects 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 43
- 230000008569 process Effects 0.000 claims description 23
- 230000002776 aggregation Effects 0.000 claims description 17
- 238000004220 aggregation Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 5
- 208000036357 GUCY2D-related recessive retinopathy Diseases 0.000 claims description 4
- 230000007423 decrease Effects 0.000 claims 1
- 230000008901 benefit Effects 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 10
- 238000013461 design Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 101100043438 Mus musculus Srsf10 gene Proteins 0.000 description 1
- 230000003416 augmentation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image super-resolution reconstruction method and system based on channel constraint multi-feature fusion, and belongs to the field of super-resolution image reconstruction. The method comprises the following steps: acquiring a high-spatial-resolution high-spectrum image, a low-spatial-resolution high-spectrum image and a high-spatial-resolution multi-spectrum image pair under the same scene to construct a training set; constructing a double-channel super-resolution network, comprising: the characteristic extraction module is used for simultaneously extracting spatial spectrum characteristics from the low-spatial-resolution high-spectral image and the high-spatial-resolution multi-spectral image in the same scene; the characteristic fusion module is used for fusing the spatial information of the multispectral image and the spectral information of the hyperspectral image in the same scene; the image reconstruction module is used for reconstructing to obtain a reconstructed image; training the network until the change rule of each element in the spectral vector corresponding to the reconstructed image is consistent with that of each element in the spectral vector corresponding to the original image; and acquiring low-spatial-resolution hyperspectral and high-spatial-resolution multispectral images under a scene to be reconstructed, and inputting the images into a trained network to obtain a reconstructed super-resolution hyperspectral image.
Description
Technical Field
The invention belongs to the field of super-resolution image reconstruction, and particularly relates to an image super-resolution reconstruction method and system based on channel constraint multi-feature fusion.
Background
In the field of remote sensing images, certain balance is required between spectrum and space detail information, and partial space detail information of the spectrum and the space detail information has to be sacrificed to ensure the integrity of the spectrum band information, however, the spectrum and the space detail information bring great difficulty to subsequent high-level tasks and limit the application of hyperspectral images. Therefore, the spatial quality of the image is improved while the spectral resolution is ensured, so that the scene information can be restored to the maximum extent, and the method is a key point and a difficulty point of the current hyperspectral field research.
There are two general approaches to improve the spatial resolution of hyperspectral images: one approach is to increase the hardware capabilities of the imaging device, but this relies on a breakthrough in imaging technology and semiconductor processing, which not only has a long development time period, but also increases the hardware cost. The other approach is to use an algorithm to improve the image quality by solving the problem at a software level. In recent years, Super-resolution (SR) technology has attracted attention from a large number of researchers. The technology aims to reconstruct a Low-spatial Resolution (LR) image with less spatial detail information into a High-spatial Resolution (HR) image with richer detail features, and eliminate the influence caused by image noise and blurring to a certain extent. With the rapid update of computer equipment, the computing power of the computer is remarkably improved, and under the support of sufficient computing power and a large amount of data, the deep learning obtains a very dazzling achievement at the super-resolution of the image. Compared with the traditional machine learning method, the super-resolution method based on deep learning can avoid manual setting of feature rules, automatically extract image and spectrum level features, learn a complex nonlinear mapping relation between high and low spatial resolution images from a large amount of data, and have a better reconstruction effect. Therefore, the task of super-resolution reconstruction of the hyperspectral image by utilizing the deep learning technology has quite important significance and value.
Although researchers have achieved some performance in the field of image super-resolution, due to the particularity of the data structure of the hyperspectral image, it is a challenging task to effectively utilize spectral information and reduce the distortion of the spectral information in the reconstruction process. Aiming at the situation, the hyperspectral and multispectral image combination method based on multi-feature fusion is applied to the image super-resolution field, improvement is carried out according to the problems existing in the practical application of a network model, and the detail reconstruction capability of an algorithm image and the model spectrum information reduction capability are enhanced.
Disclosure of Invention
Aiming at the defects and improvement requirements of the prior art, the invention provides an image super-resolution reconstruction method and system based on channel constraint multi-feature fusion, and aims to solve the problems that an image after the super-resolution reconstruction of a hyperspectral image is distorted on spectral information frequently, the image reconstruction difficulty is high when the super-resolution multiple is high, and the like.
To achieve the above object, according to a first aspect of the present invention, there is provided an image super-resolution reconstruction method based on channel-constrained multi-feature fusion, the method comprising:
a training stage:
acquiring a high-spatial-resolution hyperspectral image, a low-spatial-resolution hyperspectral image and a high-spatial-resolution multispectral image in the same scene, taking the low-spatial-resolution hyperspectral image as a training sample, and taking a corresponding high-spatial-resolution hyperspectral image as a label to construct a training set;
constructing a double-channel super-resolution network, wherein the double-channel super-resolution network comprises: the characteristic extraction module is used for simultaneously extracting spatial characteristics and spectral characteristics from the low-spatial-resolution high-spectral image and the high-spatial-resolution multi-spectral image in the same scene; the characteristic fusion module is used for fusing the multispectral image and the spatial spectral characteristics of the hyperspectral image in the same scene; the image reconstruction module is used for reconstructing the fused features to obtain a reconstructed high-spatial-resolution hyperspectral image;
training the two-channel super-resolution network by adopting a training set until the change rule of each element in the corresponding spectral vector of the reconstructed high-spatial-resolution hyperspectral image and the original high-spatial-resolution hyperspectral image is consistent, and obtaining the trained two-channel super-resolution network;
an application stage:
and acquiring a low-spatial-resolution hyperspectral image and a high-spatial-resolution multispectral image under a scene to be reconstructed, inputting the images into a trained dual-channel super-resolution network for super-resolution reconstruction, and acquiring a reconstructed high-spatial-resolution hyperspectral image.
Preferably, the low spatial resolution hyperspectral image corresponding to the high spatial resolution hyperspectral image is obtained by the following method: down-sampling an original high-spatial-resolution hyperspectral image in a bicubic interpolation mode, and adding a Gaussian blur with a standard deviation of 0.5;
the high spatial resolution multispectral image corresponding to the high spatial resolution hyperspectral image is obtained by the following method: and performing spectrum downsampling on the original high-spatial-resolution high-spectral image by combining the spectral response curve.
Preferably, the feature extraction module comprises a first branch and a second branch connected in parallel;
the first branch is used for extracting high-spatial-resolution multispectral image information and comprises a first three-dimensional convolution layer, a first three-dimensional residual error feature aggregation module and a second three-dimensional convolution layer which are connected in series, wherein the first three-dimensional convolution layer is used for extracting shallow-layer spatial spectral features of the multispectral image, the first three-dimensional residual error feature aggregation module is used for further extracting spatial dimensional features and spectral dimensional features of the multispectral image, corresponding weight is distributed to each feature channel in the obtained feature graph, and the second three-dimensional convolution layer is used for extracting high-level features and outputting the high-level features to the feature fusion module;
the second branch is used for extracting low-spatial-resolution hyperspectral image information and comprises a first three-dimensional deconvolution layer, a third three-dimensional convolution layer, a second three-dimensional residual error feature aggregation module and a fourth three-dimensional convolution layer which are connected in series, wherein the first three-dimensional deconvolution layer is used for adjusting the size of a hyperspectral image to be the same as that of a multispectral image, the third three-dimensional convolution layer is used for extracting a hyperspectral image shallow layer, the second three-dimensional residual error feature aggregation module is used for extracting space dimension features and spectrum dimension features of the second three-dimensional residual error feature aggregation module, corresponding weight is distributed to each feature channel in an obtained feature map, and the fourth three-dimensional convolution layer is used for extracting high-level features and outputting the high-level features to the feature fusion module.
According to the invention, the spatial characteristics and the spectral characteristics are extracted simultaneously by optimizing the three-dimensional convolution, so that the utilization rate of the model on the spectral information is enhanced, and the model super-resolution reconstruction precision is improved.
Preferably, the feature fusion module comprises a three-dimensional residual error module and a three-dimensional convolution layer which are connected in series, wherein the three-dimensional residual error module is used for fusing features output by the feature extraction network, and the fusion adopts a jump connection mode; the three-dimensional convolution layer is used for fusing the feature map output by each residual error module and the feature maps of all residual error features, and the N feature maps are connected together in a cascading mode to form a feature map group M and are jointly transmitted to the image reconstruction module.
Has the advantages that: the preferred fusion module of the invention adds the residual error characteristics in each residual error module in the process of feature fusion so as to strengthen the reconstruction of the detail information of the image. The fusion module adopts a jump connection mode, the feature graph output by each residual error module and the feature graph after convolution of all residual error features are connected to the output end of the module together, and each feature graph is connected together in a cascading mode and transmitted to the image reconstruction module together, so that the features of each layer are fully utilized to the maximum extent, and the information loss caused by convolution is reduced.
Preferably, the image reconstruction module comprises a feature layer weighting constraint module, a first three-dimensional convolution layer, a residual feature aggregation module, a second three-dimensional convolution layer and a third three-dimensional convolution layer which are connected in series;
the feature layer weighting constraint module is used for carrying out weighting constraint on feature maps of different images and different levels in the feature map group output by the feature fusion module; the first three-dimensional convolution layer module is used for further extracting features from the feature map after weighting constraint; the residual error feature aggregation module is used for extracting the fused empty spectrum features and distributing corresponding weights; and the second three-dimensional convolution layer and the third three-dimensional convolution layer are used for nonlinear mapping to generate a final hyperspectral image.
Has the advantages that: according to the method, a feature Layer weighting constraint Module (LAM) is applied to a hyperspectral image reconstruction task, different weight values are automatically allocated to feature maps from different depths through network learning, and the expression capability of feature extraction is improved.
Preferably, the residual error feature aggregation module consists of four three-dimensional residual error modules, the first three positions are stacked together in a conventional manner, the last one is removed from an identity mapping part, and only a 3D-CRB part is reserved; and performing weighted fusion on the feature maps output by the four 3D-CRBs in a cascading manner.
Preferably, the constrained feature map set is weighted
The calculation process is expressed as:
where ρ denotes a scaling factor, MjRepresenting input feature sets, MiRepresenting sets of feature maps after dimension conversion, wi,jRepresenting the correlation coefficient between the ith and jth set of features, δ (-) and
respectively representing Softmax and dimension conversion operations, and N representing a constituent featureNumber of feature maps of a set of maps.
Has the advantages that: the method carries out further calculation on the fused features, adaptively distributes weights to the features of different images and different levels, and reconstructs the feature map into the high-spatial-resolution hyperspectral image, so that the advantages of the two images are fully complemented, and the reconstruction task under the requirements of higher super-resolution multiple and higher image precision is completed.
Preferably, during the training process, the spatial-spectral joint constraint loss function is calculated, and the calculation is stopped when the loss value reaches the expected range or is not reduced any more.
Preferably, the spatial-spectral joint constraint loss function is:
L＝LMSE+αLspecral+βLdiff
wherein L represents the total loss function, LMSERepresenting the mean square error loss function, LspecralRepresenting a loss function of spectral difference, LdiffRepresenting a spectral difference characteristic error loss function, alpha, beta are coefficients for balancing the loss function, H, W represent the length and width in the image space direction, respectively, DeltaxijOne-dimensional spectral difference vector, Δ y, representing the original image in the ith row, j, and the th column of spaceijA one-dimensional spectral difference vector representing the reconstructed image at the ith row and the jth column in space.
Has the advantages that: aiming at the problem of spectral distortion easily occurring in the reconstruction process, the invention provides a space-spectrum combined constraint loss function to reduce the distortion of spectral information. In order to ensure the authenticity of spectral information while improving the spatial resolution of an image, the spectral difference is added into a loss function as a constraint; in order to prevent the adjacent elements of the spectral vector of the reconstructed image from generating abnormal jump inconsistent with the original image, a spectral difference characteristic error is added in the design of the loss function.
To achieve the above object, according to a second aspect of the present invention, there is provided an image super-resolution reconstruction system based on channel-constrained multi-feature fusion, the system comprising: a computer-readable storage medium and a processor;
the computer-readable storage medium is used for storing executable instructions;
the processor is configured to read executable instructions stored in the computer-readable storage medium, and execute the method for reconstructing image super-resolution based on channel-constrained multi-feature fusion according to the first aspect.
Generally, by the above technical solution conceived by the present invention, the following beneficial effects can be obtained:
aiming at the low utilization rate of spectral information in the super-resolution reconstruction process, the invention provides a method for improving the multi-level feature expression capability of a model by fusing the spatial spectral features of a multispectral image (high spatial resolution) and a hyperspectral image (low spatial resolution) in the same scene, combining the spatial information advantage of the multispectral image with the spectral information advantage of the hyperspectral image, and reconstructing the high-spatial-resolution hyperspectral image by combining the spatial spectral information of the multispectral image and the spectral information of the hyperspectral image, so that the spatial spectral features of the two are fully fused to meet the reconstruction accuracy requirement under a higher super-resolution multiple.
Drawings
FIG. 1 is a flow chart of an image super-resolution reconstruction method based on channel constraint multi-feature fusion provided by the invention;
FIG. 2 is a schematic diagram of a dual-channel super-resolution network structure based on multi-feature fusion provided by the present invention;
FIG. 3 is a schematic structural diagram of a feature fusion module provided in the present invention;
FIG. 4 is a schematic structural diagram of an image reconstruction module according to the present invention;
FIG. 5 is a schematic structural diagram of a feature layer weighting constraint module provided in the present invention;
FIG. 6 is an error plot based on root mean square error in a Chikusei data set in accordance with an embodiment of the present invention;
FIG. 7 is a comparison of results of different super-resolution reconstruction algorithms in a surface feature classification task according to an embodiment of the present invention;
FIG. 8 is a scatter diagram of the classification accuracy statistics of various ground objects by different super-resolution reconstruction algorithms in the embodiment of the present invention;
fig. 9 is a statistical histogram of the accuracy of the overall classification ground object of different super-resolution reconstruction algorithms in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Aiming at a task of reconstructing the combined Super-Resolution of a Hyperspectral Image and a Multispectral Image, the invention provides a Dual-channel Super-Resolution network (DSR-MFF) based on Multi-Feature Fusion, which extracts spatial features and spectral features simultaneously through three-dimensional convolution, enhances the utilization rate of a model to spectral information, improves the reconstruction precision of the model Super-Resolution, combines the spatial information advantage of the High-spatial Resolution Multispectral Image (HR-MSI) and the Low-spatial Resolution Hyperspectral Image (LR-HSI) in the same scene through fusing Multi-level Feature information of the High-spatial Resolution Multispectral Image (HR-MSI) and the Low-spatial Resolution Hyperspectral Image (LR-HSI), reconstructs the High-spatial Resolution Hyperspectral Image (High-Resolution Image) by combining the spatial information and the spectral information of the two spatial information, HR-HSI) and improving the expression capacity of the multi-level characteristics of the model. Aiming at the problem of spectral distortion easily occurring in the reconstruction process, the invention provides a space-spectrum combined constraint loss function to reduce the distortion of spectral information.
The invention provides an image super-resolution reconstruction method based on channel constraint multi-feature fusion, which comprises the following steps:
and (1) making a data set, selecting a part of hyperspectral image, and inputting the part of hyperspectral image as an original high-spatial-resolution image.
And (2) obtaining the low-spatial-resolution hyperspectral image by operations of down-sampling, addition of fuzzy and noise and the like.
And (3) carrying out spectrum downsampling operation on the original image by combining the spectrum response curve to obtain the multispectral image with high spatial resolution.
And (4) constructing a dual-channel super-resolution network based on multi-feature fusion, inputting a low-spatial-resolution hyperspectral image and a high-spatial-resolution multispectral image, adjusting the two images to the same spatial size, and extracting and fusing multi-level features of the input multispectral image and the input hyperspectral image through a feature fusion module.
The multi-feature fusion dual-channel super-resolution network in the step (4) is specifically realized as follows:
the image super-resolution reconstruction method based on the dual-channel super-resolution network with multi-feature fusion is provided, and the high-spatial-resolution multi-spectral image HR-MSI with abundant spatial information and the low-spatial-resolution multi-spectral image LR-HSI with abundant spectral information in the same scene are combined to perform super-resolution reconstruction, so that the spatial spectral features of the high-spatial-resolution multi-spectral image HR-MSI and the low-spatial-resolution multi-spectral image LR-HSI are fully fused to meet the reconstruction accuracy requirement under a higher super-resolution multiple.
The feature fusion module in the step (4) is specifically realized as follows:
the fusion module adds the residual error characteristics in each residual error module in the process of feature fusion so as to strengthen the reconstruction of image detail information. The fusion module adopts a jump connection mode, the feature graph output by each residual error module and the feature graph after convolution of all residual error features are connected to the output end of the module together, and each feature graph is connected together in a cascading mode and transmitted to the image reconstruction module together, so that the features of each layer are fully utilized to the maximum extent, and the information loss caused by convolution is reduced. The overall structure of the fusion module is shown in fig. 3.
The cascade of the feature maps adopted by the feature fusion module in the step (4) is specifically realized as follows:
the feature map cascade needs to ensure that the space sizes of the feature maps are consistent, and the space sizes of the image feature maps are not changed in the convolution process. Therefore, the convolution step of each channel weighting constraint residual module in the feature fusion module is set to be 1, and the feature maps are filled in the SAME mode to ensure that the feature maps output by each layer have the SAME space size.
And (5) outputting the fused features to an image reconstruction module, further calculating the fused features, adaptively distributing weights to the features of different images and different levels, and reconstructing the feature map into a high-spatial-resolution hyperspectral image.
The image reconstruction module in the step (5) is specifically implemented as follows:
a feature Layer weighting constraint Module (LAM) is applied to a hyperspectral image reconstruction task, different weight values are automatically allocated to feature maps from different depths through network learning, and the expression capability of feature extraction is improved. For a feature map group M formed by cascading N feature maps input in the image fusion module, firstly, the feature maps are converted from dimension H multiplied by W multiplied by NC to dimension N multiplied by HWC, and a new two-dimensional matrix is obtained. The matrix calculates the correlation between different layers by performing a matrix multiplication operation with its corresponding transpose matrix, and the calculation process can be expressed as:
wherein, wi,jRepresenting the correlation coefficient between the ith and jth set of features, δ (-) and
respectively representing Softmax and dimension conversion operations. Finally, multiplying the feature map group after dimension conversion by the correlation matrix coefficient and the scaling factor alpha, and adding the obtained result with the input feature map group to obtain the feature map group after weighting constraint, wherein the calculation process is represented as:
and (6) training the constructed model, calculating a space-spectrum combined constraint loss function, and stopping model training when the loss value reaches an expected range or is not reduced any more.
The spatial spectrum joint constraint loss function in the step (6) is as follows:
for the hyperspectral image, adjacent wave bands have strong correlation, and if the pixel loss is considered only and the expression of the spectral characteristics of the hyperspectral image is not facilitated, the distortion exists on the spectral information of the reconstructed image. Therefore, in a hyperspectral image reconstruction task, in order to ensure the authenticity of spectral information while improving the spatial resolution of an image, the spectral difference degree is added into a loss function as a constraint:
wherein H, W denotes the length and width in the image space direction, LspectrumRepresenting a loss function of spectral difference, xijAnd yijRespectively representing the one-dimensional spectral vectors of the original image and the reconstructed image in the ith row and the jth column in space.
Meanwhile, besides the restraint of the whole spectral information, in order to ensure that each element between the original image and the spectral vector of the reconstructed image keeps the consistent change rule and prevent the adjacent elements of the spectral vector of the reconstructed image from generating abnormal jump inconsistent with the original image, the spectral difference characteristic error is added in the design of the loss function. For a one-dimensional spectral vector x with C elementsijAnd yijThe spectral difference at the band k (2. ltoreq. k. ltoreq.C) is expressed as:
Δxij(k)＝xij(k)-xij(k-1)
Δyij(k)＝yij(k)-yij(k-1)
the spectral difference vector can be expressed as:
Δxij＝(Δxij(2),Δxij(3),…,Δxij(C))T
Δyij＝(Δyij(2),Δyij(3),…,Δyij(C))T
the spectral difference characteristic error loss function LdiffCan be expressed as:
total loss function:
L＝LMSE+αLspecral+βLdiff
where α and β are coefficients for balancing the loss function, and both α and β are set to 0.5 in this embodiment.
And (7) performing super-resolution calculation on the input high-spatial-resolution multispectral image and the low-spatial-resolution hyperspectral image jointly by using the trained model, and reconstructing to generate the high-spatial-resolution hyperspectral image.
The invention carries out multi-angle verification of visual effect, parameter index and influence on ground feature classification performance on reconstruction quality of two model images in a public data set.
The embodiment comprises the following steps:
1. data set production and analysis
The present embodiment selects CAVE and Chikusei data sets. The CAVE data set comprises 32 indoor scene hyperspectral images with the space size of 512 x 512, 31 spectral wave bands from 400nm to 700nm, and the interval between the wave bands is 10 nm. The Chikusei data set is a remote sensing hyperspectral image taken in Japan, which contains 128 bands and has the image size of 2517 pixels × 2335 pixels.
In the training set production, an original image is used as a high spatial resolution label, a corresponding low spatial resolution image is generated by down-sampling in a double cubic interpolation mode, and Gaussian blur with a standard deviation of 0.5 is added. In the CAVE data set, 20 images are randomly extracted to produce a training set of the model, and for the Chikusei data set with only a single image, five 320 × 320 areas in the image are selected as a test set, and the rest areas are divided into small image blocks which are partially overlapped through cropping to produce the training set. For the Pavia Centre and University data sets, in order to ensure that the image classification task is not interfered, the whole Pavia Centre image is selected as a training set, and a plurality of training images are generated in an image cropping mode.
In a verification experiment for improving the effect of super-resolution reconstruction on a ground object classification task, a general Pavia Centre and University data set in a classical super-resolution task and a ground object classification task is selected. For the Pavia Centre and University data sets, in order to ensure that the image classification task is not interfered, the whole Pavia Centre image is selected as a training set, and a plurality of training images are generated in an image cropping mode.
The DSR-MFF model needs a low spatial resolution hyperspectral image (LR-HSI) and a corresponding high spatial resolution hyperspectral image (HR-HIS) image pair as training data, and additionally needs a corresponding high spatial resolution multispectral image (HR-MSI), the HR-MSI needed by the DSR-MFF model is synthesized by the HR-HSI and curves of three wave bands of RGB in corresponding spectral response curves, a spectral response curve of a Nikon D700 camera is selected in a CAVE data set, and a spectral response curve of a Landsat-8 satellite-borne sensor is selected in a Chikusei and a Pavia Centre and University data set.
The image pixel values need to be normalized before the experiment, adjusting each pixel value to between 0 and 1. In the experiment of this embodiment, each image is normalized by dividing the image by the maximum pixel value of the image:
I′＝I/max(I)
where I' represents the normalized image, I represents the original image, and max (·) represents the maximum value of all pixels in the image.
Sample augmentation of images in training set: and (3) carrying out scaling processing of 0.5, 0.75 and 1 on each group of samples of the training set, respectively rotating the samples to 0 degree, 90 degrees and 270 degrees in an image rotation and overturning mode, and horizontally overturning to obtain more training data.
2. Network training process
The invention provides a multispectral and hyperspectral image combined super-resolution reconstruction method based on multi-feature fusion, the overall flow of the method is shown in figure 1, and the specific process of the method mainly comprises three parts, namely network building, loss function design and network training.
Network construction
The double-channel super-resolution network process provided by the invention can be divided into the following steps: and (3) performing feature fusion and image reconstruction. Firstly, feature extraction and feature fusion are carried out on two input images by a feature fusion module, the fused feature images are output to an image reconstruction module, then the fused features are reconstructed by the image reconstruction module to obtain the final HR-HIS, and the overall network structure is shown in figure 2.
Dual channel feature fusion module
The traditional image fusion method can be divided into three levels of pixel level fusion, feature level fusion and decision level fusion according to different information levels. Generally speaking, a deeper network model can bring better nonlinear expression capability, a higher-level network has stronger global mapping capability, but the resolution of a feature map is lower, and the expression capability of geometric feature details of an image is lacked, while a feature map extracted by a lower-level network usually contains stronger geometric feature information but lacks semantic information. In order to integrate the advantages of the two, the feature fusion module inputs the lower-level features and the higher-level features into a subsequent reconstruction module together so as to improve the performance of the model.
First, before fusion, to adjust the two images to the same spatial size, an deconvolution layer was placed at the LR-HSI input to enlarge the image size.
The fusion module adds the residual error characteristics in each residual error module in the process of feature fusion so as to strengthen the reconstruction of image detail information. And the feature graph output by each residual error module and the feature graph after convolution of all residual error features are connected to the output end of the module together in a jump connection mode, and the feature graphs are connected together in a cascade mode and transmitted to the image reconstruction module together, so that the features of each layer are fully utilized to the maximum extent, and the information loss caused by convolution is reduced. The overall structure of the fusion module is shown in fig. 3.
The feature map cascade needs to ensure that the space sizes of the feature maps are consistent, and the space sizes of the image feature maps are not changed in the convolution process. For this purpose, the convolution step in each CRM in the feature fusion module is set to 1, and the feature maps are filled in by the SAME way, so as to ensure that the feature maps output by each layer have the SAME spatial size.
Image reconstruction module
The image reconstruction module takes the multi-level spatial spectrum feature map group output by the image fusion module as input, and the feature maps from different images and different levels are not consistent in importance of image reconstruction. In addition, there is a correlation between feature maps of different layers, and simply splicing feature maps of different layers does not effectively represent the correlation and difference between layers.
The invention provides a method for applying a characteristic layer weighting constraint module to a hyperspectral image reconstruction task, automatically distributing different weight values to characteristic graphs from different depths through network learning, and improving the expression capability of characteristic extraction.
After the feature map group output by the image fusion module is input into the image reconstruction module, firstly, weighting constraint is carried out on feature maps of different images and different layers through LAM, and then nonlinear mapping and image reconstruction are carried out on the feature maps after weighting constraint through the convolution layer and a residual error feature aggregation module consisting of four CRM to generate a final hyperspectral image. The structure of the image reconstruction module is shown in fig. 4. The implementation process of the feature-layer weighting constraint module will be described below, and the specific configuration is shown in fig. 5.
For a feature map group M formed by cascading N feature maps input in the image fusion module, firstly, the feature maps are converted from dimension H multiplied by W multiplied by NC to dimension N multiplied by HWC, and a new two-dimensional matrix is obtained. The matrix calculates the correlation between different layers by performing a matrix multiplication operation with its corresponding transpose matrix, and the calculation process can be expressed as:
wherein, wi,jRepresenting the correlation coefficient between the ith and jth set of features, δ (-) and
respectively representing Softmax and dimension conversion operations. Finally, multiplying the feature map group after dimension conversion by the correlation matrix coefficient and the scaling factor alpha, and adding the obtained result to the input feature map group to obtain the feature map group after weighting constraint, wherein the calculation process can be expressed as:
loss function design
Aiming at the problem that spectral dimension information is easy to distort when a multispectral image and a hyperspectral image are jointly reconstructed, the invention provides a space-spectrum joint constraint loss function combining spectral difference, spectral difference characteristic error and mean square error, and spectral distortion can be effectively reduced in the reconstruction process.
Mean square error loss function of hyperspectral image:
spectral dissimilarity loss function:
in order to ensure the authenticity of spectral information while improving the spatial resolution of an image, the spectral difference degree is added into a loss function as a constraint:
wherein H, W denotes the length and width in the image space direction, LspectrumRepresenting a loss function of spectral difference, xijAnd yijRespectively represent the originalAnd the one-dimensional spectral vectors of the initial image and the reconstructed image on the ith row and the jth column in the space. By utilizing the spectrum difference degree, the spectrum vector between the original image and the reconstructed image can be constrained, and the integral spectrum difference of the two images is reduced.
Meanwhile, besides the restraint of the whole spectral information, in order to ensure that each element between the original image and the spectral vector of the reconstructed image keeps the consistent change rule and prevent the adjacent elements of the spectral vector of the reconstructed image from generating abnormal jump inconsistent with the original image, the spectral difference characteristic error is added in the design of the loss function. For a one-dimensional spectral vector x with C elementsijAnd yijThe spectral difference at the band k (2. ltoreq. k. ltoreq.C) is expressed as:
Δxij(k)＝xij(k)-xij(k-1)
Δyij(k)＝yij(k)-yij(k-1)
the spectral difference vector can be expressed as:
Δxij＝(Δxij(2),Δxij(3),…,Δxij(C))T
Δyij＝(Δyij(2),Δyij(3),…,Δyij(C))T
spectral difference characteristic error loss function
By using the spectral difference characteristic error, the value of the phase difference between adjacent elements in the spectral vector between the reconstructed images can be restrained, so that the change rule of each element in the corresponding spectral vector of the reconstructed image and the original image is kept consistent, and the abnormal jump of the adjacent elements of the spectral vector of the reconstructed image is prevented.
Spectral difference characteristic error loss function LdiffExpressed as:
total loss function
In order to reduce the pixel error of the reconstructed image and enhance the spectrum similarity, a spectrum difference degree loss function, a spectrum difference characteristic error loss function and a mean square error loss function are combined to obtain a total loss function:
L＝LMSE+αLspecral+βLdiff
where α and β are coefficients for balancing the loss function, and both α and β are 0.5 in this embodiment.
Network training
The network training process specifically includes inputting the prepared training samples into a built network for training, calculating a loss function, performing a back propagation gradient, and updating the network. The training steps are as follows:
initializing a dual-channel feature fusion network and an image reconstruction network;
respectively sending the low-spatial-resolution high-spectral image y and the high-spatial-resolution multi-spectral image z into a dual-channel feature fusion network, and extracting content features and style features of the multi-spectral image and the high-spatial-resolution multi-spectral image;
applying geometric transformation, such as rotation, mirror image and the like, to the content features, and then sending the content features after the geometric transformation and the original respective style features to an image reconstruction network for image generation;
comparing the original image with the generated image, and calculating the spatial-spectral joint constraint loss;
and (5) turning to the next iteration process to generate a high spatial resolution hyperspectral image.
Fusion model performance analysis
In order to comprehensively evaluate the overall performance of the DSR-MFF, the DSR-MFF is compared with four latest hyperspectral and multispectral image combined super-resolution reconstruction algorithms on a Chikusei data set, and the super-resolution multiples are set to be 16 and 32. The four methods comprise NSSR, HySure, DHSIS and uSDN, and because the numerical distribution of each pixel point in the Chikusei data set is wide, an error graph based on root mean square error is adopted. The reconstruction results of the different methods in the Chikusei dataset are shown in fig. 6, with DSR-MFF on the left and the true image on the right. The objective evaluation index comparison results are shown in table 1.
TABLE 1
The influence of the reconstruction task on the actual scene application value of the hyperspectral image is further explored. In a hyperspectral image application scene, a hyperspectral image classic task-ground object classification is selected as a test task. And reconstructing LR-HSI by using a DSR-MFF network model, wherein the super-resolution multiplying power is set to be 4.
After the model training is completed, the LR image, the image after the reconstruction of the two models and the truth image before the down sampling are respectively subjected to classification test, the classification result is shown in FIG. 7, the DER-MFF reconstruction classification result is on the left, the original image classification result is in the middle, and the labeled ground object pair is on the right. The classification accuracy statistical chart is shown in fig. 8 and 9.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A super-resolution image reconstruction method based on channel constraint multi-feature fusion is characterized by comprising the following steps:
a training stage:
acquiring a high-spatial-resolution hyperspectral image, a low-spatial-resolution hyperspectral image and a high-spatial-resolution multispectral image in the same scene, taking the image pairs as training samples, taking the corresponding high-spatial-resolution hyperspectral images as labels, and constructing a training set;
constructing a double-channel super-resolution network, wherein the double-channel super-resolution network comprises: the characteristic extraction module is used for simultaneously extracting spatial characteristics and spectral characteristics from the low-spatial-resolution high-spectral image and the high-spatial-resolution multi-spectral image in the same scene; the characteristic fusion module is used for fusing the multispectral image and the spatial spectral characteristics of the hyperspectral image in the same scene; the image reconstruction module is used for reconstructing the fused features to obtain a reconstructed high-spatial-resolution hyperspectral image;
training the two-channel super-resolution network by adopting a training set until the change rule of each element in the corresponding spectral vector of the reconstructed high-spatial-resolution hyperspectral image and the original high-spatial-resolution hyperspectral image is consistent, and obtaining the trained two-channel super-resolution network;
an application stage:
and acquiring a low-spatial-resolution hyperspectral image and a high-spatial-resolution multispectral image under a scene to be reconstructed, inputting the images into a trained dual-channel super-resolution network for super-resolution reconstruction, and acquiring a reconstructed high-spatial-resolution hyperspectral image.
2. The method of claim 1, wherein the low spatial resolution hyperspectral image corresponding to the high spatial resolution hyperspectral image is obtained by: down-sampling an original high-spatial-resolution hyperspectral image in a bicubic interpolation mode, and adding a Gaussian blur with a standard deviation of 0.5;
the high spatial resolution multispectral image corresponding to the high spatial resolution hyperspectral image is obtained by the following method: and performing spectrum downsampling on the original high-spatial-resolution high-spectral image by combining the spectral response curve.
3. The method of claim 1, wherein the feature extraction module comprises a first branch and a second branch in parallel;
the first branch is used for extracting high-spatial-resolution multispectral image information and comprises a first three-dimensional convolution layer, a first three-dimensional residual error feature aggregation module and a second three-dimensional convolution layer which are connected in series, wherein the first three-dimensional convolution layer is used for extracting shallow-layer spatial spectral features of the multispectral image, the first three-dimensional residual error feature aggregation module is used for further extracting spatial dimensional features and spectral dimensional features of the multispectral image, corresponding weight is distributed to each feature channel in the obtained feature graph, and the second three-dimensional convolution layer is used for extracting high-level features and outputting the high-level features to the feature fusion module;
the second branch is used for extracting low-spatial-resolution hyperspectral image information and comprises a first three-dimensional deconvolution layer, a third three-dimensional convolution layer, a second three-dimensional residual error feature aggregation module and a fourth three-dimensional convolution layer which are connected in series, wherein the first three-dimensional deconvolution layer is used for adjusting the size of a hyperspectral image to be the same as that of a multispectral image, the third three-dimensional convolution layer is used for extracting a hyperspectral image shallow layer, the second three-dimensional residual error feature aggregation module is used for extracting space dimension features and spectrum dimension features of the second three-dimensional residual error feature aggregation module, corresponding weight is distributed to each feature channel in an obtained feature map, and the fourth three-dimensional convolution layer is used for extracting high-level features and outputting the high-level features to the feature fusion module.
4. The method of claim 1, wherein the feature fusion module comprises a three-dimensional residual module and a three-dimensional convolution layer connected in series, wherein the three-dimensional residual module is used for fusing features of the feature extraction network output, and the fusion adopts a jump connection mode; the three-dimensional convolution layer is used for fusing the feature map output by each residual error module and the feature maps of all residual error features, and the N feature maps are connected together in a cascading mode to form a feature map group M and are jointly transmitted to the image reconstruction module.
5. The method of claim 1, wherein the image reconstruction module comprises a feature layer weight constraint module, a first three-dimensional convolutional layer, a residual feature aggregation module, a second three-dimensional convolutional layer, and a third three-dimensional convolutional layer in series;
the feature layer weighting constraint module is used for carrying out weighting constraint on feature maps of different images and different levels in the feature map group output by the feature fusion module; the first three-dimensional convolution layer module is used for further extracting features from the feature map after weighting constraint; the residual error feature aggregation module is used for extracting the fused empty spectrum features and distributing corresponding weights; and the second three-dimensional convolution layer and the third three-dimensional convolution layer are used for nonlinear mapping to generate a final hyperspectral image.
6. The method of claim 5, wherein the residual feature aggregation module consists of four three-dimensional residual modules, the first three positions are stacked together in a conventional manner, the last one has an identity mapping part removed, and only a 3D-CRB part remains; and performing weighted fusion on the feature maps output by the four 3D-CRBs in a cascading manner.
7. The method of claim 5, wherein the constrained set of feature maps is weighted
The calculation process is expressed as:
where ρ denotes a scaling factor, MjRepresenting input feature sets, MiRepresenting sets of feature maps after dimension conversion, wi，jRepresenting the correlation coefficient between the ith and jth set of features, δ (-) and
softmax and dimension conversion operation are respectively expressed, and N represents the number of feature maps constituting a feature map group.
8. The method of claim 1, wherein during training, the spatial-spectral joint constraint loss function is computed and stopped when the loss value is within a desired range or no longer decreases.
9. The method of claim 8, wherein the spatial-spectral joint constraint loss function is:
L＝LMSE+αLspecral+βLdiff
wherein L represents the total loss function, LMSERepresenting the mean square error loss function, LspecralRepresenting a loss function of spectral difference, LdiffRepresenting a spectral difference characteristic error loss function, alpha, beta are coefficients for balancing the loss function, H, W represent the length and width in the image space direction, respectively, DeltaxijOne-dimensional spectral difference vector, Δ y, representing the original image in the ith row, j, and the th column of spaceijA one-dimensional spectral difference vector representing the reconstructed image at the ith row and the jth column in space.
10. An image super-resolution reconstruction system based on channel constraint multi-feature fusion is characterized by comprising: a computer-readable storage medium and a processor;
the computer-readable storage medium is used for storing executable instructions;
the processor is used for reading executable instructions stored in the computer-readable storage medium and executing the image super-resolution reconstruction method based on channel constraint residual feature fusion of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111165041.2A CN113744136A (en) | 2021-09-30 | 2021-09-30 | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111165041.2A CN113744136A (en) | 2021-09-30 | 2021-09-30 | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113744136A true CN113744136A (en) | 2021-12-03 |
Family
ID=78725957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111165041.2A Pending CN113744136A (en) | 2021-09-30 | 2021-09-30 | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113744136A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114092834A (en) * | 2022-01-24 | 2022-02-25 | 南京理工大学 | Unsupervised hyperspectral image blind fusion method and system based on space-spectrum combined residual correction network |
| CN115222600A (en) * | 2022-07-29 | 2022-10-21 | 大连理工大学 | Multispectral remote sensing image super-resolution reconstruction method for contrast learning |
| CN115564692A (en) * | 2022-09-07 | 2023-01-03 | 宁波大学 | Panchromatic-multispectral-hyperspectral integrated fusion method considering width difference |
| CN117132473A (en) * | 2023-10-20 | 2023-11-28 | 中国海洋大学 | Underwater rare earth spectrum detection method and spectrum super-resolution reconstruction model building method thereof |
-
2021
- 2021-09-30 CN CN202111165041.2A patent/CN113744136A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114092834A (en) * | 2022-01-24 | 2022-02-25 | 南京理工大学 | Unsupervised hyperspectral image blind fusion method and system based on space-spectrum combined residual correction network |
| CN115222600A (en) * | 2022-07-29 | 2022-10-21 | 大连理工大学 | Multispectral remote sensing image super-resolution reconstruction method for contrast learning |
| CN115564692A (en) * | 2022-09-07 | 2023-01-03 | 宁波大学 | Panchromatic-multispectral-hyperspectral integrated fusion method considering width difference |
| CN115564692B (en) * | 2022-09-07 | 2023-12-05 | 宁波大学 | Full color-multispectral-hyperspectral integrated fusion method considering breadth difference |
| CN117132473A (en) * | 2023-10-20 | 2023-11-28 | 中国海洋大学 | Underwater rare earth spectrum detection method and spectrum super-resolution reconstruction model building method thereof |
| CN117132473B (en) * | 2023-10-20 | 2024-01-23 | 中国海洋大学 | Underwater rare earth spectrum detection method and spectrum super-resolution reconstruction model building method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020168844A1 (en) | Image processing method, apparatus, equipment, and storage medium | |
| Liu et al. | A spectral grouping and attention-driven residual dense network for hyperspectral image super-resolution | |
| CN111369440B (en) | Model training and image super-resolution processing method, device, terminal and storage medium | |
| CN113744136A (en) | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion | |
| CN111767979A (en) | Neural network training method, image processing method, and image processing apparatus | |
| CN110717851A (en) | Image processing method and device, neural network training method and storage medium | |
| CN112819910B (en) | Hyperspectral image reconstruction method based on double-ghost attention machine mechanism network | |
| CN109087375B (en) | Deep learning-based image cavity filling method | |
| CN112884668A (en) | Lightweight low-light image enhancement method based on multiple scales | |
| CN113888491B (en) | Multistage hyperspectral image progressive superdivision method and system based on non-local features | |
| CN109447897B (en) | Real scene image synthesis method and system | |
| CN116309070A (en) | Super-resolution reconstruction method and device for hyperspectral remote sensing image and computer equipment | |
| CN104899835A (en) | Super-resolution processing method for image based on blind fuzzy estimation and anchoring space mapping | |
| CN109948575A (en) | Eyeball dividing method in ultrasound image | |
| Xu et al. | AutoSegNet: An automated neural network for image segmentation | |
| Shi et al. | Structure-aware deep networks and pixel-level generative adversarial training for single image super-resolution | |
| Li et al. | D2c-sr: A divergence to convergence approach for real-world image super-resolution | |
| CN117788295A (en) | Super-resolution reconstruction method, system and medium for remote sensing image | |
| CN117575915A (en) | Image super-resolution reconstruction method, terminal equipment and storage medium | |
| CN113313625A (en) | Ink and wash painting artistic style conversion method, system, computer equipment and storage medium | |
| Xu et al. | Depth map super-resolution via joint local gradient and nonlocal structural regularizations | |
| Luo et al. | A fast denoising fusion network using internal and external priors | |
| Liu et al. | A fast and accurate super-resolution network using progressive residual learning | |
| Yang | Super resolution using dual path connections | |
| Zhao et al. | Joint Discontinuity-Aware Depth Map Super-Resolution via Dual-Tasks Driven Unfolding Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240315 Address after: Room A348, 4th Floor, Building 1, Phase III, International Enterprise Center, No. 1 Guanggu Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province, 430073 (Wuhan Free Trade Zone) Applicant after: AVIC Tianhai (Wuhan) Technology Co.,Ltd. Country or region after: China Address before: 430074 Hubei Province, Wuhan city Hongshan District Luoyu Road No. 1037 Applicant before: HUAZHONG University OF SCIENCE AND TECHNOLOGY Country or region before: China |