CN108921783A - A kind of satellite image super resolution ratio reconstruction method based on losses by mixture function constraint - Google Patents

A kind of satellite image super resolution ratio reconstruction method based on losses by mixture function constraint Download PDF

Info

Publication number
CN108921783A
CN108921783A CN201810556589.1A CN201810556589A CN108921783A CN 108921783 A CN108921783 A CN 108921783A CN 201810556589 A CN201810556589 A CN 201810556589A CN 108921783 A CN108921783 A CN 108921783A
Authority
CN
China
Prior art keywords
image
network
losses
satellite image
resolution ratio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810556589.1A
Other languages
Chinese (zh)
Other versions
CN108921783B (en
Inventor
王中元
江奎
易鹏
韩镇
邵振峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN201810556589.1A priority Critical patent/CN108921783B/en
Publication of CN108921783A publication Critical patent/CN108921783A/en
Application granted granted Critical
Publication of CN108921783B publication Critical patent/CN108921783B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a kind of satellite image super resolution ratio reconstruction methods based on losses by mixture function constraint, including non-local network's building, L1 and L2 to mix two processes of normal form constrained learning.In non-local network's building, for nontarget area, the correlation with the feature vector in current goal region is calculated, the related description information of target is extracted by convolution, information enhancement is carried out to current input, to promote the ability to express of network.During L1 and L2 mixing normal form constrained learning, it is contemplated that feature of the L1 and L2 norm in the expression of different levels characteristic information sparsity uses the two norms as target loss function, to promote the convergence rate and performance of network stage by stage.The present invention significantly enhances the feature representation ability of deep learning network, can rebuild the high-resolution satellite image of texture exquisiteness.

Description

A kind of satellite image super resolution ratio reconstruction method based on losses by mixture function constraint
Technical field
The invention belongs to digital image processing techniques fields, are related to a kind of satellite image super resolution ratio reconstruction method, specifically It is related to a kind of satellite image super resolution ratio reconstruction method of L1 and L2 mixing normal form constraint non-locality's network.
Background technique
The spatial resolution of satellite image is to measure an important indicator of satellite shooting ability, how in existing hardware water Equalling the upper higher resolution ratio of acquisition, oneself becomes the developing goal of various countries' satellite.Super-resolution technique is answered remote sensing satellite field With, it will the spatial resolution of satellite is improved, the coke of optical instrument can also be reduced under conditions of keeping satellite resolution ratio Away from minimizing Satellite Camera, reduce its volume and weight.
Video satellite is a kind of novel remote sensing satellite, can capture continuous dynamic video rather than static image, because This is very suitable to the observation of the large-scale dynamic object such as ship and aircraft.Compared with traditional remote sensing satellite, video satellite is to sacrifice sky Between resolution ratio be that cost obtains high time resolution, it is therefore necessary to restore its ground by the means of image super-resolution Resolution ratio and fine object construction.
Image super-resolution technology is from the interpolation algorithm of early stage to the method for being currently based on deep learning, the property of reconstruction image Can have and be obviously improved.Method based on interpolation, although arithmetic speed is fast, result images lack details, especially for This application for needing high-precision detail textures of video satellite image, interpolation method are not suitable for.SA (the Shift of the propositions such as Elad And Add) the method principle fuzzy based on translation and space invariance, pass through information non-iterative fusion process and obtains high resolution graphics Picture, but algorithm requires to must satisfy certain translation relation between sequence of low resolution pictures, is otherwise easy to appear " black lattice Net " phenomenon.Recently, Patrick Hagerty et al. proposes a kind of method based on deep learning, in deep neural network It is embedded in the details of high-definition picture, and extracts these details when enhancing similar image.But this method does not account for and benefit The spatial simlanty feature of geomorphological features under the large scale range existing for video satellite image, to image self information using not Sufficiently, oversubscription result is caused to be overly dependent upon network model parameter, scalability is poor.
Summary of the invention
It is super that in order to solve the above-mentioned technical problems, the present invention provides a kind of satellite images based on losses by mixture function constraint Resolution reconstruction method is inserted into non local enhancing block in residual error network, makes full use of the self similarity of satellite image spatially Property, obtaining the useful information except receptive field target area enhances current region, while using L1 and L2 model stage by stage Number is constrained, and fine reconstruction effect is obtained.
The technical scheme adopted by the invention is that:A kind of satellite image Super-resolution reconstruction based on losses by mixture function constraint Construction method, which is characterized in that include the following steps:
Step 1:Several satellite image data are chosen as training sample, it is N × N pixel that size is intercepted from every frame image Image block obtain the low-resolution image block that corresponding size is M × M as data sample, while by n times of its down-sampling, as The input of residual error network, wherein N=n × M;
Step 2:Characteristics of image is extracted by convolution kernel;
Step 3:The output information of each residual block is passed to non local piece, feature enhancing processing is carried out to input;
Step 4:The characteristic information of acquisition is input to reconstruction of layer, while using L1 and L2 norm as the damage of network stage by stage Function is lost, the parameter of network is optimized, output obtains residual plot;
Step 5:Simultaneously by original input interpolation amplification, the interpolation image with high-definition image comparable size is obtained;
Step 6:By interpolation image and residual error e-learning to residual image be added, obtain final super-resolution output Image.
The present invention pointedly carries out characterology using L1 and L2 norm because it is contemplated that non local input information It practises, greatly enhances the ability to express of network, clearly oversubscription image can be obtained.
Detailed description of the invention
Fig. 1 is the schematic diagram of non-local network's block of the present invention.
Specific embodiment
Understand for the ease of those of ordinary skill in the art and implement the present invention, with reference to the accompanying drawings and embodiments to this hair It is bright to be described in further detail, it should be understood that implementation example described herein is merely to illustrate and explain the present invention, not For limiting the present invention.
Referring to Fig.1, a kind of satellite image super-resolution rebuilding side based on losses by mixture function constraint provided by the invention Method, which is characterized in that include the following steps:
Step 1:Several satellite image data are chosen as training sample, it is N × N (this that size is intercepted from every frame image Example N takes the image block of 128) pixel as data sample, while it is M × M (this example N that its down-sampling, which is obtained corresponding size, Take low-resolution image block 32), the input as residual error network;Wherein N=n × M;
Step 2:Characteristics of image is extracted by convolution kernel;
Step 3:The output information of each residual block is passed to non local piece, by increasing additional non-local information pair Input carries out feature enhancing processing;
In order to obtain preferable effect, non local piece of insertion position is after shallow-layer residual block, for front convolution area One enhancing of domain receptive field.
The present embodiment carries out enhancing processing to feature using non-local network, and formula is utilized during network training (1), it by softmax function, determines the interconnection vector between non local region and target area, is calculated by way of weight Non local region y outiTo the contribution W of target areaz*yi
zi=Wzyi+xi (1)
Wherein, xiIt indicates a residual error connection, while being also non local piece of input, ziIt indicates through non local piece of enhancing Information output, yiCorrespond to the image block areas being different from except target area, WzIndicate different image block areas to current The percentage contribution of target area obtains weight in such a way that network self-adapting learns.
Step 4:The characteristic information of acquisition is input to reconstruction of layer, while using L1 and L2 norm as the damage of network stage by stage Function is lost, the parameter of network is optimized, output obtains residual plot;
Due to the sparse characteristic of L1 norm, most feature vector coefficients is 0, only the feature vector of fraction It is utilized.For this purpose, feature of the L1 as the loss function of network for low frequency level in learning is used early period, to be compared Smooth result.L2 norm also has sparsity, but its sparse coefficient is close to 0 but not equal to 0, it is hereby achieved that more non- Zero feature vector can learn the information to high frequency level.Therefore, in the network optimization later period, combined using L2 and L1 norm The form of constraint, to rebuild better result.
The present embodiment is constrained stage by stage using L1 and L2 mixing normal form, finally unified about with two loss functions again Beam controls reconstructed image signal representation error using formula (2);
Wherein, first part | | y-Wx | |1Indicate L1 loss function, second partIndicate L2 loss, λ is indicated Weight, x correspond to the initial input of network, i.e. low-resolution image, and y is high-resolution supervision image, and W indicates super-resolution net The parameter of filter in network, while being also the reconstruction coefficients of high-definition picture, LOSS represents the whole loss function of network, wraps It includes L1 loss function and L2 loses two parts;
It is reconstructed using sub-pix convolution operation, obtains output layer (r relevant to amplification coefficient2* C, wherein r is amplification system Number, C are output image color channel numbers), reconstruct obtains residual plot;It is reconstructed, is different from traditional using sub-pix convolution operation Deconvolution operation, it is thus possible to retain the structural information of high-level feature to greatest extent, obtain finer and smoother detail textures.
Step 5:The method of original input bi-cubic interpolation is amplified simultaneously, is obtained and high-definition image comparable size Interpolation image;
Step 6:By interpolation image and residual error e-learning to residual error be added, obtain final oversubscription output image.
The present invention can extract characteristic information related with current goal region from non local region, to input information into Row enhancing, is greatly improved the ability to express of shallow-layer network;Performance is obtained more preferably under the premise of guaranteeing computational efficiency simultaneously Oversubscription result.
It should be understood that the part that this specification does not elaborate belongs to the prior art.
It should be understood that the above-mentioned description for preferred embodiment is more detailed, can not therefore be considered to this The limitation of invention patent protection range, those skilled in the art under the inspiration of the present invention, are not departing from power of the present invention Benefit requires to make replacement or deformation under protected ambit, fall within the scope of protection of the present invention, this hair It is bright range is claimed to be determined by the appended claims.

Claims (5)

1. a kind of satellite image super resolution ratio reconstruction method based on losses by mixture function constraint, which is characterized in that including following Step:
Step 1:Several satellite image data are chosen as training sample, the figure that size is N × N pixel is intercepted from every frame image Picture block obtains the low-resolution image block that corresponding size is M × M as data sample, while by n times of its down-sampling, as residual error The input of network, wherein N=n × M;
Step 2:Characteristics of image is extracted by convolution kernel;
Step 3:The output information of each residual block is passed to non local piece, feature enhancing processing is carried out to input;
Step 4:The characteristic information of acquisition is input to reconstruction of layer, while using L1 and L2 norm as the loss letter of network stage by stage Number, optimizes the parameter of network, and output obtains residual plot;
Step 5:Simultaneously by original input interpolation amplification, the interpolation image with high-definition image comparable size is obtained;
Step 6:By interpolation image and residual error e-learning to residual image be added, obtain final super-resolution output figure Picture.
2. the satellite image super resolution ratio reconstruction method according to claim 1 based on losses by mixture function constraint, special Sign is:In step 3, enhancing processing is carried out to feature using non-local network, formula is utilized during network training (1), it by softmax function, determines the interconnection vector between non local region and target area, is calculated by way of weight Non local region y outiContribution to target area;
zi=Wzyi+xi (1)
Wherein, xiIt indicates a residual error connection, while being also non local piece of input, ziIndicate the letter by non local piece of enhancing Breath output, yiCorrespond to the image block areas being different from except target area, WzIndicate different image block areas to current goal The percentage contribution in region obtains weight in such a way that network self-adapting learns.
3. the satellite image super resolution ratio reconstruction method according to claim 1 based on losses by mixture function constraint, special Sign is:It in step 4, is constrained using L1 and L2 mixing normal form, is finally constrained again with two loss functions are unified stage by stage, Using formula (2), reconstructed image signal representation error is controlled;
Wherein, first part | | y-Wx | |1Indicate L1 loss function, second partIndicate L2 loss, λ indicates weight; X corresponds to the initial input of network, i.e. low-resolution image;Y is high-resolution supervision image, and W is indicated in super-resolution network The parameter of filter, while being also the reconstruction coefficients of high-definition picture, LOSS represents the whole loss function of network, including L1 Loss function and L2 lose two parts.
4. the satellite image super resolution ratio reconstruction method according to claim 1 based on losses by mixture function constraint, special Sign is:It in step 4, is reconstructed using sub-pix convolution operation, obtains output layer relevant to amplification coefficient, reconstruct obtains residual error Figure.
5. the satellite image super resolution ratio reconstruction method according to claim 1 based on losses by mixture function constraint, special Sign is:In step 5, the method for original input bi-cubic interpolation is amplified.
CN201810556589.1A 2018-06-01 2018-06-01 Satellite image super-resolution reconstruction method based on mixed loss function constraint Active CN108921783B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810556589.1A CN108921783B (en) 2018-06-01 2018-06-01 Satellite image super-resolution reconstruction method based on mixed loss function constraint

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810556589.1A CN108921783B (en) 2018-06-01 2018-06-01 Satellite image super-resolution reconstruction method based on mixed loss function constraint

Publications (2)

Publication Number Publication Date
CN108921783A true CN108921783A (en) 2018-11-30
CN108921783B CN108921783B (en) 2022-04-15

Family

ID=64418700

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810556589.1A Active CN108921783B (en) 2018-06-01 2018-06-01 Satellite image super-resolution reconstruction method based on mixed loss function constraint

Country Status (1)

Country Link
CN (1) CN108921783B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109671022A (en) * 2019-01-22 2019-04-23 北京理工大学 A kind of picture texture enhancing super-resolution method based on depth characteristic translation network
CN110009568A (en) * 2019-04-10 2019-07-12 大连民族大学 The generator construction method of language of the Manchus image super-resolution rebuilding
CN110245310A (en) * 2019-03-06 2019-09-17 腾讯科技(深圳)有限公司 A kind of behavior analysis method of object, device and storage medium
CN111010566A (en) * 2019-12-04 2020-04-14 杭州皮克皮克科技有限公司 Non-local network-based video compression distortion restoration method and system
CN111353944A (en) * 2018-12-20 2020-06-30 深圳市中兴微电子技术有限公司 Image reconstruction method and device and computer readable storage medium
TWI733341B (en) * 2020-02-20 2021-07-11 瑞昱半導體股份有限公司 Method and system for image scaling and enhancement
CN113313636A (en) * 2020-02-27 2021-08-27 瑞昱半导体股份有限公司 Method and apparatus for image magnification and enhancement
CN118037549A (en) * 2024-04-11 2024-05-14 华南理工大学 Video enhancement method and system based on video content understanding

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102142137A (en) * 2011-03-10 2011-08-03 西安电子科技大学 High-resolution dictionary based sparse representation image super-resolution reconstruction method
CN105825477A (en) * 2015-01-06 2016-08-03 南京理工大学 Remote sensing image super-resolution reconstruction method based on multi-dictionary learning and non-local information fusion
WO2017004890A1 (en) * 2015-07-08 2017-01-12 北京大学深圳研究生院 Dictionary-type image super-resolution system and method based on iteration projection reconstruction
CN106683067A (en) * 2017-01-20 2017-05-17 福建帝视信息科技有限公司 Deep learning super-resolution reconstruction method based on residual sub-images
CN106952228A (en) * 2017-03-10 2017-07-14 北京工业大学 The super resolution ratio reconstruction method of single image based on the non local self-similarity of image
CN106960416A (en) * 2017-03-20 2017-07-18 武汉大学 A kind of video satellite compression image super-resolution method of content complexity self adaptation
CN107833183A (en) * 2017-11-29 2018-03-23 安徽工业大学 A kind of satellite image based on multitask deep neural network while super-resolution and the method for coloring

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102142137A (en) * 2011-03-10 2011-08-03 西安电子科技大学 High-resolution dictionary based sparse representation image super-resolution reconstruction method
CN105825477A (en) * 2015-01-06 2016-08-03 南京理工大学 Remote sensing image super-resolution reconstruction method based on multi-dictionary learning and non-local information fusion
WO2017004890A1 (en) * 2015-07-08 2017-01-12 北京大学深圳研究生院 Dictionary-type image super-resolution system and method based on iteration projection reconstruction
CN106683067A (en) * 2017-01-20 2017-05-17 福建帝视信息科技有限公司 Deep learning super-resolution reconstruction method based on residual sub-images
CN106952228A (en) * 2017-03-10 2017-07-14 北京工业大学 The super resolution ratio reconstruction method of single image based on the non local self-similarity of image
CN106960416A (en) * 2017-03-20 2017-07-18 武汉大学 A kind of video satellite compression image super-resolution method of content complexity self adaptation
CN107833183A (en) * 2017-11-29 2018-03-23 安徽工业大学 A kind of satellite image based on multitask deep neural network while super-resolution and the method for coloring

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
HONGGANG CHEN等: "Single Image Super-Resolution via Adaptive Transform-Based Nonlocal Self-Similarity Modeling and Learning-Based Gradient Regularization", 《 IEEE TRANSACTIONS ON MULTIMEDIA》 *
向文等: "结合结构自相似性和卷积网络的单幅图像超分辨率", 《计算机应用》 *
李银辉等: "基于L1和L2混合范式的序列图像超分辨率重建", 《计算机应用》 *
沈松等: "基于稀疏表示的超分辨率图像重建", 《电子测量技术》 *
邱康等: "协作稀疏字典学习实现单幅图像超分辨率重建", 《光学学报》 *
陈伟业等: "结合压缩感知与非局部信息的图像超分辨率重建", 《计算机应用》 *

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111353944A (en) * 2018-12-20 2020-06-30 深圳市中兴微电子技术有限公司 Image reconstruction method and device and computer readable storage medium
CN111353944B (en) * 2018-12-20 2024-05-28 深圳市中兴微电子技术有限公司 Image reconstruction method, device and computer readable storage medium
US11810265B2 (en) 2018-12-20 2023-11-07 Sanechips Technology Co., Ltd. Image reconstruction method and device, apparatus, and non-transitory computer-readable storage medium
CN109671022B (en) * 2019-01-22 2022-11-18 北京理工大学 Image texture super-resolution enhancement method based on depth feature translation network
CN109671022A (en) * 2019-01-22 2019-04-23 北京理工大学 A kind of picture texture enhancing super-resolution method based on depth characteristic translation network
CN110245310A (en) * 2019-03-06 2019-09-17 腾讯科技(深圳)有限公司 A kind of behavior analysis method of object, device and storage medium
CN110245310B (en) * 2019-03-06 2023-10-13 腾讯科技(深圳)有限公司 Object behavior analysis method, device and storage medium
CN110009568A (en) * 2019-04-10 2019-07-12 大连民族大学 The generator construction method of language of the Manchus image super-resolution rebuilding
CN111010566A (en) * 2019-12-04 2020-04-14 杭州皮克皮克科技有限公司 Non-local network-based video compression distortion restoration method and system
US11416966B2 (en) 2020-02-20 2022-08-16 Realtek Semiconductor Corp. Method and system for image scaling and enhancement
TWI733341B (en) * 2020-02-20 2021-07-11 瑞昱半導體股份有限公司 Method and system for image scaling and enhancement
CN113313636A (en) * 2020-02-27 2021-08-27 瑞昱半导体股份有限公司 Method and apparatus for image magnification and enhancement
CN118037549A (en) * 2024-04-11 2024-05-14 华南理工大学 Video enhancement method and system based on video content understanding
CN118037549B (en) * 2024-04-11 2024-06-28 华南理工大学 Video enhancement method and system based on video content understanding

Also Published As

Publication number Publication date
CN108921783B (en) 2022-04-15

Similar Documents

Publication Publication Date Title
CN108921783A (en) A kind of satellite image super resolution ratio reconstruction method based on losses by mixture function constraint
CN107123089B (en) Remote sensing image super-resolution reconstruction method and system based on depth convolution network
CN106683067B (en) Deep learning super-resolution reconstruction method based on residual sub-images
Zhang et al. Group-based sparse representation for image restoration
CN107025632B (en) Image super-resolution reconstruction method and system
WO2021022929A1 (en) Single-frame image super-resolution reconstruction method
CN109741256A (en) Image super-resolution rebuilding method based on rarefaction representation and deep learning
CN109671023A (en) A kind of secondary method for reconstructing of face image super-resolution
CN114119444B (en) Multi-source remote sensing image fusion method based on deep neural network
CN103455988B (en) The super-resolution image reconstruction method of structure based self-similarity and rarefaction representation
CN110136060B (en) Image super-resolution reconstruction method based on shallow dense connection network
CN109146787B (en) Real-time reconstruction method of dual-camera spectral imaging system based on interpolation
CN106920220A (en) Based on the turbulent flow method for blindly restoring image that dark primary and alternating direction multiplier method optimize
CN112837224A (en) Super-resolution image reconstruction method based on convolutional neural network
CN103136728B (en) Based on the image super-resolution method of dictionary learning and non local total variance
Chen et al. Single image super-resolution using deep CNN with dense skip connections and inception-resnet
CN106920214A (en) Spatial target images super resolution ratio reconstruction method
CN110288526A (en) A kind of image reconstruction algorithm based on deep learning promotes the optimization method of single pixel camera imaging quality
CN107833182A (en) The infrared image super resolution ratio reconstruction method of feature based extraction
CN110349085A (en) A kind of single image super-resolution feature Enhancement Method based on generation confrontation network
CN114998141B (en) Space environment high dynamic range imaging method based on multi-branch network
CN112163998A (en) Single-image super-resolution analysis method matched with natural degradation conditions
CN105550989A (en) Image super-resolution method based on nonlocal Gaussian process regression
WO2020231016A1 (en) Image optimization method, apparatus, device and storage medium
CN115526779A (en) Infrared image super-resolution reconstruction method based on dynamic attention mechanism

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant