CN106254720B - A kind of video super-resolution method for reconstructing based on joint regularization - Google Patents
A kind of video super-resolution method for reconstructing based on joint regularization Download PDFInfo
- Publication number
- CN106254720B CN106254720B CN201610567213.1A CN201610567213A CN106254720B CN 106254720 B CN106254720 B CN 106254720B CN 201610567213 A CN201610567213 A CN 201610567213A CN 106254720 B CN106254720 B CN 106254720B
- Authority
- CN
- China
- Prior art keywords
- video
- resolution
- regularization
- super
- joint
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/142—Edging; Contouring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/20—Circuitry for controlling amplitude response
- H04N5/205—Circuitry for controlling amplitude response for correcting amplitude versus frequency characteristic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
- H04N5/213—Circuitry for suppressing or minimising impulsive noise
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of method for reconstructing of video super-resolution based on joint regularization.The following steps are included: establishing low resolution observing and nursing；By the least-square methods based on regularization, the cost function that a general video is rebuild is established；It constructs TV (CTV) regularization term based on compensation and assigns the adaptive weighted coefficient of regional space to reduce registration error bring adverse effect；Construct several non-local low rank (MNLR) regularization terms；The present invention is based on the cost functions that the video super-resolution of joint regularization is rebuild for foundation；Cost function is solved with Split-Bregman alternative manner, reconstructs high-resolution video.Video super-resolution method for reconstructing of the present invention, which reconstructs the video frame come, has marginal information abundant, and almost without sawtooth effect, from video reconstruction frames it can be seen that this method inhibits the ability of noise also fine, there is very high reference value on objectively evaluating parameter.So the present invention is a kind of effective video super resolution ratio reconstruction method.
Description
Technical field
The present invention had designed a kind of video super-resolution method for reconstructing based on joint regularization, especially this regularization term both
The correlation in video frame is considered, it is also contemplated that the correlation of video interframe, can keep video frame edge details letter
While breath, inhibit noise well, belongs to digital image processing field.
Background technique
During obtaining video, due to atmospheric perturbation, lack sampling, system noise and optics and motion blur etc. because
The influence of element, the video quality degradation caused, resolution ratio reduce.Requirement with people to the clarity of video is more next
Higher, the resolution ratio for how improving video becomes urgent problem to be solved.Super-resolution rebuilding technology is high hard because not needing
Part cost can have good reconstruction effect and the extensive concern by domestic and international researcher, be in recent years at image
The research hotspot in the fields such as reason and computer vision.Using similar and different low point continuous when video super-resolution is rebuild
Resolution video frame obtains high-resolution video frame by redundancy, inhibition noise, the reduction blooming between video frame.
Video frame is made Super-resolution reconstruction be built into ill-conditioning problem by the deficiency of various noises or redundancy, therefore passes through regularization term
The method of constraint high-definition picture solution space can solve this problem.The realization of Tikhonov regularization term is simple, but this
Regularization has obscured image border while constraining noise, and this method can make the image rebuild excessively smooth.Become based on complete
The regularization constraint for dividing (TV) and bilateral full variation (BTV) is more generally used prior image model, since they can
Removal noise while reconstruct the image of sharpening, save image edge detailss information, but when video is by larger noise
It is bad in flat site treatment effect when influence, it is easy to produce pseudo-edge, edge region is also easy to produce sawtooth effect.
Summary of the invention
The present invention is directed to the deficiency of the above regularization, a kind of joint regularization method is proposed, by the TV based on compensation
(CTV) combine with several non-local low rank regularization terms, make full use of the different advantages of two kinds of regularizations, considering
While video in-frame correlation, the correlation of video interframe is explored, it is superfluous between adjacent video frames being introduced in regularization term
Remaining information.Simultaneously in view of being registrated the adverse effect of residual error bring in CTV, regional space information filter is devised to CTV and is added
Weight coefficient reduces negative influence of the registration residual error to the reconstruction of video.So that algorithm for reconstructing is kept while eliminating noise
The edge detail information of video frame.
The present invention proposes the video super-resolution method for reconstructing based on joint regularization, mainly includes following operating procedure:
(1) low resolution observing and nursing is established；
(2) by the least-square methods based on regularization, the cost function that a general video is rebuild is established；
(3) it constructs TV (CTV) regularization term based on motion compensation and assigns the adaptive weighted coefficient of regional space to subtract
The adverse effect of small registration error；
(4) several non-local low rank (MNLR) regularization terms are constructed；
(5) according to the regularization term in step (3) and step (4), the present invention is based on the videos of joint regularization to surpass for foundation
The cost function of resolution reconstruction；
(6) cost function is solved with Split-Bregman alternative manner, reconstructs high-resolution video.
Detailed description of the invention
Fig. 1 is the block diagram of the video super-resolution method for reconstructing the present invention is based on joint regularization
Fig. 2 is reconstructed results comparison diagram of the present invention with four kinds of methods to the 15th frame of video " Mobile "
Fig. 3 is reconstructed results comparison diagram of the present invention with four kinds of methods to the 15th frame of video " Grandma "
Fig. 4 is reconstructed results comparison diagram of the present invention with four kinds of methods to the 15th frame of video " Akiyio "
Specific embodiment
Video method for reconstructing based on joint regularization mainly comprises the steps of:
(1) low resolution observing and nursing is established；
(2) by the least-square methods based on regularization, the cost function that a general video is rebuild is established；
(3) it constructs TV (CTV) regularization term based on motion compensation and assigns the adaptive weighted coefficient of regional space to subtract
The adverse effect of small registration error；
(4) several non-local low rank (MNLR) regularization terms are constructed；
(5) according to the regularization term in step (3) and step (4), the present invention is based on the videos of joint regularization to surpass for foundation
The cost function of resolution reconstruction；
(6) cost function is solved with Split-Bregman alternative manner.
Specifically, in the step (1) video frame low resolution observing and nursing are as follows:
yk=DBfk+N 1≤k≤K, (1)
Wherein yk, fkRespectively kth frame low resolution video frame and kth frame high-resolution video frame, D are the down-sampling factor,
B is fuzzy factor, and N is random noise, and K is video frame number.
In the step (2), prior information appropriate is introduced to construct high-resolution view when video super-resolution is rebuild
The constraint of frequency can be public by being realized based on the least-square methods of regularization to guide the super-resolution rebuilding of video
Formula is as follows:
Wherein, first item is data fidelity term, and Section 2 is regularization term, and λ is the weighting coefficient of regularization.
TV (CTV) regularization term based on compensation is constructed in the step (3).The method includes the steps of:
The first step, it is assumed that each frame fk, haveWherein Qk-1, Bk+1It respectively indicates with fkFor
Reference frame is to fk-1,fk+1Operation of motion compensation, the present invention using optical flow method carry out estimation；
Residual error between consecutive frame after reference frame and motion compensation is arranged in gradient field second step, and formula is as follows:
Wherein G indicates gradient operation, G=[Gh,Gv],GhWith GvRespectively indicate difference operation both horizontally and vertically.
Residual error is arranged in gradient field in the present invention, rather than equally original image is arranged in ladder with traditional full variation (TV)
Domain is spent, is because if estimation is accurate enough, residual error will be more more sparse than original image.
Due to CTV dependence and the difference between the content and consecutive frame of video frame, so it is insensitive to edge.However,
There is complicated local motion between consecutive frame, accurate motion registration is difficult to obtain, therefore registration residual error will will affect view
The reconstructed results of frequency.In view of s in step (3)kAccuracy, to every s in gradient fieldkIt is adaptive to add a regional space
Weight is answered, so that CTV is converted into CATV, formula is as follows:
Wherein step WkSolution procedure it is as follows；
The first step needs to extract spatial structural form, since for radio-frequency component, second dervative can distinguish image border
With noise region, defined so second derivative-based curvature difference arithmetic operators are applied to regularization weight coefficient
In.Curvature difference at pixel j point are as follows:
Cj=| | eηη|-|eεε||
Wherein, ex,eyFirst derivative respectively at pixel horizontally and vertically, exx,exy,eyyAt pixel
Second dervative.
Second step calculates the partial weight at pixel, and formula is as follows:
Wherein, τ is constant, control parameter intensity.
Third step, using clustering algorithm, by wjM class is divided, then image is divided into discontinuous region, has similarity weight
The pixel of weight is then divided into identical region, and the regularization intensity in each region is by of a sort cluster centre rm(m=
1 ..., M) control.
4th step calculates final weight equation:
Wherein Γ1,Γ2Respectively represent the smooth region and details area of image.
It should be noted weight Wk(k=1 ..., K) is designed for residual error in gradient field, rather than video frame.
It can be seen that edge region weight is smaller, therefore has lesser regularization constraint, registration error pair is reduced
The influence of reconstruction maintains the detailed information of reconstruction.In flat site, weight is larger, effectively inhibits noise.
In the step (4), the step of constructing several non-local low rank (MNLR) regularization terms are as follows:
Video frame is divided into the block of overlapping by the first step, it is assumed that will in kth frame, centered on pixel j, size be n ×
The image block of n indicates to become pk,j, centered on j, determine search window；
Second step reuses the optical flow field in step (3), to guide the search window in (k-1) frame and (k+1) frame
Center position.Accordingly, it is determined that the center position of the search window in kth frame, so that it may with this in optical flow field
The motion vector of central point finds out the center position that search window is corresponded in consecutive frame；
Third step, calculate (k-1) frame, in k frame and (k+1) frame in search window in all candidate blocks and reference block
pk,jBetween l2Difference finds out pk,jThe N of blockpA similar block；
4th step, similar block is grouped, each building matrix Xj,k, each column of this matrix represent a similar block.Root
According to similar characteristic, this matrix has low-rank characteristic, but due to the interference of noise, this characteristic is destroyed；
5th step constructs MNLR regularization term, and formula is as follows:
Wherein, Sk,jF expression seeks the reference block in k frame centered on pixel j in (k-1) frame, k frame and (k+1) frame
Similar block, and Xj,k=Sk,jF, Zk,jIt is the approximate S an of low-rankk,jThe matrix of F, λ are a tradeoff parameters.
Cost function in step (5) are as follows:
Wherein α, β, λ regularization parameter replace Rank (Z using with minor functionk,j):
δ is the constant of a very little, and I indicates unit matrix, then cost equation be converted in order to:
The step of in step (7) using the optimization algorithm of the super-resolution rebuilding of joint regularization term, is as follows:
The first step increases a variable d, so thatCost equation is converted into for following formula:
Second step solves problem above using Split-Bregman iterative algorithm, then cost function can be decomposed into following
Four subproblems:
Third step solves above four subproblems respectively:
1 solves the subproblem of Z:
Zn+1=U (Σ-λ diag (θn))+VT (16)
Wherein, It isFirst of characteristic value, Sk,jFnCan carry out Eigenvalues Decomposition is U Σ VT,
And (x)+=max (x, 0).
2 solve the subproblem of F:
It is solved herein with conjugate gradient method (CG).
3 solve the subproblem of d:
Wherein,
In order to illustrate effectiveness of the invention, the present invention will choose 3 test videos and test, i.e. " Mobile "
" Grandma " and " Akiyo ".First by video size 5 × 5, the fuzzy core that variance is 1 is obscured, is then carried out down
Sampling is eventually adding the zero-mean Gaussian noise that variance is 3, generates low-resolution video.Bicubic interpolation is chosen in control methods
Bicubic and 3 kinds of video super-resolution method for reconstructing.Experimental result is as shown in Fig. 2, Fig. 3 and Fig. 4, (b) respectively to (f)
Represent bicubic interpolation image, the video frame that method 1, method 2, method 3 and the present invention reconstruct.
Three kinds of video super-resolution algorithm for reconstructing are respectively as follows:
Method one: software algorithm, Infognition Video Enhancer [EB/OL], vol.5, no.1,2014.
http://www.infognition.com/videoenhancer/.
The method that method two: Yuan et al. proposes.Bibliography " Q.Yuan, L.Zhang, H.Shen. " Regional
spatially adaptive total variation super-resolution with spatial information
filtering and clustering."IEEE Transactions on Image Processing,vol.22,no.6,
pp.2327-2342,2013.”。
The method that method three: Turgay et al. proposes.Bibliography " E.Turgay, G.B.Akar. " Texture and
edge preserving multiframe super-resolution."Image Processing,IET,vol.8,
no.9,pp.499-508,2014.”。
The content of comparative experiments is as follows:
Experiment 1 be with Bicubic method to the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " into
2 times of super-resolution rebuildings of row, shown in experimental result such as Fig. 2 (b), Fig. 3 (b) and Fig. 4 (b).
Experiment 2 is to carry out 2 times with 10 to 20th frame of the method 1 to video " Mobile ", " Grandma " and " Akiyo "
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (c), Fig. 3 (c) and Fig. 4 (c).
Experiment 3 is to carry out 2 times with 10 to 20th frame of the method 2 to video " Mobile ", " Grandma " and " Akiyo "
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (d), Fig. 3 (d) and Fig. 4 (d).
Experiment 4 is that usage 3 is super to 2 times of the 10 to 20th frame progress of video " Mobile ", " Grandma " and " Akiyo "
Resolution reconstruction, shown in experimental result such as Fig. 2 (e), Fig. 3 (e) and Fig. 4 (e).
Experiment 5 is to carry out 2 times to the 10 to 20th frame of video " Mobile ", " Grandma " and " Akiyo " with the present invention
Super-resolution rebuilding, shown in experimental result such as Fig. 2 (f), Fig. 3 (f) and Fig. 4 (f).
Several groups of comparison comparative experimentss show that the video frame arrived that Bicubic is rebuild is very fuzzy, and there is sawtooth effect at edge,
Noise is still serious in video frame；The method that method 1 is rebuild the video frame of reconstruction still cannot be overcome still fuzzy on the whole and
The serious disadvantage of noise；Rebuilding obtained video frame from method 2 can be seen that the denoising of method 2 effect is relatively good, but at edge
Still there is sawtooth effect in part；Method 3 rebuilds obtained video frame in edge sharpening, but noise phenomenon is still serious；With
No matter the video frame that the present invention is rebuild in terms of being also to maintain edge in terms of removing noise has all obtained good result.
For objective appraisal effectiveness of the invention, with Y-PSNR (PSNR:Peak Signal to Noise
Ratio it) is used as with structural similarity (SSIM:Structure Similarity Index) and objectively evaluates evaluation criterion, table one
Illustrate the average PSNR and average SSIM of various algorithms.
Table one illustrates that two kinds of objective standards, the present invention has highest value, for video " Mobile ",
Average PSNR value method 2 " Grandma " and " Akiyo " of the invention be higher by respectively 1.09dB, 1.63dB and
2.01dB, average SSIM value ratio method 2 are higher by 0.0303,0.0121 and 0.0192 respectively.
In conclusion no matter the present invention in subjective vision or in the standard of objectively evaluating has apparent advantage.Therefore originally
Invention is a kind of effective video super resolution ratio reconstruction method.
Table one
Claims (3)
1. a kind of video super-resolution method for reconstructing based on joint regularization mainly includes following operating procedure:
Step 1: low resolution observing and nursing is established；
Step 2: by the least-square methods based on regularization, the cost function that a general video is rebuild is established；
Step 3: building the full variation CTV regularization term based on motion compensation and assign the adaptive weighted coefficient of regional space with
Reduce the adverse effect of registration error；
Step 4: several non-local low ranks MNLR regularization term is constructed；
Step 5: according to the regularization term in step 3 and step 4, the present invention is based on the video oversubscription of joint regularization for foundation
The cost function that resolution is rebuild；
Step 6: cost function is solved with Split-Bregman alternative manner, reconstructs high-resolution video.
2. a kind of video super-resolution method for reconstructing based on joint regularization according to claim 1, it is characterised in that
Step 3: the full variational regularization item based on motion compensation is mutually tied with several non-local low ranks MNLR regularization term in step 4
Cooperation is the prior-constrained carry out super-resolution rebuilding that video super-resolution is rebuild.
3. a kind of video super-resolution method for reconstructing based on joint regularization according to claim 1, it is characterised in that
Since accurate estimation is difficult to obtain in step 3, registration error not can avoid in CTV, therefore to be based on spatial information
It filters with the weighting coefficient of classification and reduces registration error bring negative influence in CTV, configured in the high region of registration error
Small weight reduces the regularization constraint of CTV, configures big weight in the low region of registration error, reconstructs high-resolution with this
The video of rate.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610567213.1A CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610567213.1A CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106254720A CN106254720A (en) | 2016-12-21 |
| CN106254720B true CN106254720B (en) | 2019-07-09 |
Family
ID=57613338
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610567213.1A Active CN106254720B (en) | 2016-07-19 | 2016-07-19 | A kind of video super-resolution method for reconstructing based on joint regularization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106254720B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107194889B (en) * | 2017-05-11 | 2021-02-02 | 西安电子科技大学 | Block bilateral total variation regularization image noise elimination method |
| CN108416371A (en) * | 2018-02-11 | 2018-08-17 | 艾视医疗科技成都有限公司 | A kind of diabetic retinopathy automatic testing method |
| CN108765322B (en) * | 2018-05-16 | 2021-04-27 | 上饶师范学院 | Image denoising method and device |
| CN110208211B (en) * | 2019-07-03 | 2021-10-22 | 南京林业大学 | Near infrared spectrum noise reduction method for pesticide residue detection |
| CN113034640A (en) * | 2021-03-26 | 2021-06-25 | 浙江工业大学 | Compressed sensing image reconstruction method combining bilateral total variation and non-local low-rank regularization |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103971354A (en) * | 2014-05-19 | 2014-08-06 | 四川大学 | Method for reconstructing low-resolution infrared image into high-resolution infrared image |
| CN104463793A (en) * | 2014-11-12 | 2015-03-25 | 合肥工业大学 | Video super-resolution reestablishing method and system based on sparse representation and vector continued fraction interpolation under polar coordinates |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9558712B2 (en) * | 2014-01-21 | 2017-01-31 | Nvidia Corporation | Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image |
-
2016
- 2016-07-19 CN CN201610567213.1A patent/CN106254720B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103971354A (en) * | 2014-05-19 | 2014-08-06 | 四川大学 | Method for reconstructing low-resolution infrared image into high-resolution infrared image |
| CN104463793A (en) * | 2014-11-12 | 2015-03-25 | 合肥工业大学 | Video super-resolution reestablishing method and system based on sparse representation and vector continued fraction interpolation under polar coordinates |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106254720A (en) | 2016-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106254720B (en) | A kind of video super-resolution method for reconstructing based on joint regularization | |
| CN110443768B (en) | Single-frame image super-resolution reconstruction method based on multiple consistency constraints | |
| Zhang et al. | Image super-resolution based on structure-modulated sparse representation | |
| Song et al. | Unsupervised denoising for satellite imagery using wavelet directional CycleGAN | |
| Cheon et al. | Generative adversarial network-based image super-resolution using perceptual content losses | |
| CN106952228A (en) | The super resolution ratio reconstruction method of single image based on the non local self-similarity of image | |
| CN103093433B (en) | Natural image denoising method based on regionalism and dictionary learning | |
| CN101980289A (en) | Frequency domain registration and convex set projection-based multi-frame image super-resolution reconstruction method | |
| CN102902961A (en) | Face super-resolution processing method based on K neighbor sparse coding average value constraint | |
| CN112396672B (en) | Sparse angle cone-beam CT image reconstruction method based on deep learning | |
| CN110211084B (en) | Image multi-resolution reconstruction method based on weight wavelet transform | |
| CN104504672B (en) | Low-rank sparse neighborhood insertion ultra-resolution method based on NormLV features | |
| Li et al. | Perceptual quality evaluation for image defocus deblurring | |
| CN109447903B (en) | Method for establishing semi-reference super-resolution reconstruction image quality evaluation model | |
| CN107169928A (en) | A kind of human face super-resolution algorithm for reconstructing learnt based on deep layer Linear Mapping | |
| CN108171654A (en) | Chinese character image super resolution ratio reconstruction method with AF panel | |
| Yan et al. | SMIR: A Transformer-Based Model for MRI super-resolution reconstruction | |
| CN117769722A (en) | Generating a lossless image compression model for tomographic imaging based on an antagonism network | |
| CN114187181A (en) | Double-path lung CT image super-resolution method based on residual information refining | |
| CN106846286B (en) | Video super-resolution algorithm for reconstructing based on a variety of complementary priori | |
| Liu et al. | Undersampled MR Image Reconstruction with Data‐Driven Tight Frame | |
| Zhang et al. | Deep residual network based medical image reconstruction | |
| Al-Hassan et al. | Enhancing face recognition at a distance using super resolution | |
| Thilagavathi et al. | An efficient dictionary learning algorithm for 3d Medical Image Denoising based on Sadct | |
| CN108665412B (en) | Method for performing multi-frame image super-resolution reconstruction by using natural image priori knowledge |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |