CN112200732A

CN112200732A - Video deblurring method with clear feature fusion

Info

Publication number: CN112200732A
Application number: CN202010368483.6A
Authority: CN
Inventors: 魏颢; 项欣光; 潘金山
Original assignee: Nanjing University of Science and Technology
Current assignee: Nanjing University of Science and Technology
Priority date: 2020-04-30
Filing date: 2020-04-30
Publication date: 2021-01-08
Anticipated expiration: 2040-04-30
Also published as: CN112200732B

Abstract

The invention discloses a video deblurring method with fused clear features. Firstly, selecting a plurality of continuous fuzzy video frames, estimating optical flow between the continuous frames by using an optical flow estimation network, twisting images of adjacent frames by using the estimated optical flow, then taking a twisted result and an original fuzzy frame sequence as input of a deblurring network, then selecting a plurality of clear frames, passing through a clear feature extraction module, obtaining clear features, fusing the clear features into the deblurring network, and finally outputting relatively clear video frames by the deblurring network. The method is robust to scenes with clear frames, clear frames of any scene can be used for feature fusion, and reconstruction of video frames is facilitated, so that the method is convenient and effective.

Description

Video deblurring method with clear feature fusion

Technical Field

The invention relates to an end-to-end video deblurring network, in particular to a video deblurring algorithm based on clear feature fusion.

Background

In recent years, with the development of portable imaging devices, such as mobile phones, cameras, image/video deblurring techniques have received much attention. The reasons for the formation of blur are many, including the motion of objects during imaging, camera shake and depth of field, which provide much resistance to computer vision studies (object detection, object recognition). Therefore, it is very necessary to conduct the research of the deblurring algorithm.

At present, the deblurring algorithm is mainly divided into two types, one is a method based on a physical model, and the other is a method based on learning. Early, according to a degradation model (B ═ K × S + N), where B denotes a blurred image, K denotes a blur kernel, S denotes a sharp image, and N denotes additive noise. In the case of the known blurred image B, it is very difficult to solve the blurred kernel K and the sharp image S, and a multi-solution situation may occur, so that it is a pathological problem. In order to constrain the solution space, a plurality of natural image priors are designed, including an L0 gradient prior, a dark channel prior and the like to constrain the natural image, and the solution is carried out under a maximum posterior framework. However, the method is difficult to optimize based on a physical model, time-consuming, without universality, and requires artificially designed prior information for constraint.

Another learning-based approach is the one that is currently more popular. People learn the inherent distribution of natural images from data sets through designing a reasonable neural network, and finally achieve the aim of deblurring.

The invention provides an end-to-end video deblurring network, which is different from image deblurring, and the video deblurring needs to consider the relation between adjacent frames.

Disclosure of Invention

The invention aims to provide an end-to-end video deblurring algorithm based on clear feature fusion, which is used for inputting a plurality of continuous blurred video frames and recovering clear intermediate frames.

The technical solution for realizing the purpose of the invention is as follows: a video deblurring algorithm based on sharp feature fusion comprises the following steps:

step A: designing an optical flow estimation module, inputting three continuously blurred frames into the optical flow estimation module, and outputting a result obtained by performing image torsion (image warp) on two adjacent frames;

and B: designing a clear feature fusion module, inputting any clear three frames into the module, and outputting clear feature graphs with different scales;

and C: and designing a deblurring module, stacking the result of the step A and the original input, sending the result into the module, and fusing the result of the step B into the whole deblurring process to finally obtain a clear intermediate frame.

Compared with the prior art, the invention has the following remarkable advantages: the invention considers the motion compensation and the image deblurring at the same time. The method is robust to scenes with clear frames, clear frames of any scene can be used for feature fusion, and reconstruction of video frames is facilitated, so that the method is convenient and effective.

Drawings

FIG. 1 is an overall flow chart of the present invention.

Fig. 2 is a diagram of a network architecture designed by the present invention.

Fig. 3 is a comparison graph of the deblurring effect of the present invention. (a) Is a blurred video frame and (b) is the reconstruction result.

Detailed Description

The invention is further described below with reference to the accompanying drawings.

As shown in fig. 2, the whole network includes 3 modules, which are an optical flow estimation module, a sharp feature fusion module and a deblurring module.

According to the first panel of fig. 1, the data preparation steps are as follows:

step 1, downloading a GOPRO _ Su data set as a training sample, wherein the GOPRO _ Su data set comprises 71 video sets, and each video is provided with a plurality of paired fuzzy-clear video frames;

and 2, dividing 71 videos into two parts, wherein 61 videos serve as training samples, and 10 videos serve as testing samples.

According to the second block of fig. 1, the optical flow estimation module has the following steps:

step 1, presetting training parameters, including a learning rate 1e-6 of an optical flow estimation module, a learning rate 1e-4 of a deblurring module, and a maximum iteration epoch of 500;

step 2, three continuous blurred images { I ] are selected₁,I₂,I₃Estimating optical flow { f) between two adjacent frames by FlowNet S_1→2,f_3→2Obtaining two 2-channel optical flow graphs respectively,performing image torsion (image warp) operation on the optical flow and the adjacent frame, and twisting the adjacent frame to the intermediate frame to finally obtain two 3-channel RGB images { warp_1→2,warp_3→2}. The process can be described as follows:

f_1→2＝F([I₁,I₂]),f_3→2＝F([I₃,I₂]),

warp_1→2＝W([f_1→2,I₁]),warp_3→2＝W([f_3→2,I₃])；

where F () denotes the optical flow estimation network, [ ] denotes the operation of the image concat, and W () denotes the warp operation.

According to the third panel of fig. 1, the steps of the sharp feature fusion module are as follows:

selecting three clear images { S₁,S₂,S₃And the three clear images are passed through a feature extraction module, wherein the feature extraction module is formed by two layers of convolution respectively, and 32 256 × 256 feature maps { feature _ coarse } and 64 × 64 feature maps { feature _ fine } are finally obtained respectively. The process can be described as follows:

Feature_coarse,feature_fine＝E([S₁,S₂,S₃])

where E () represents the feature extraction network and [ ] represents the operation of the image concat.

According to the fourth panel of fig. 1, the step of deblurring the module is as follows:

step 1, obtaining a warp result { warp ] from an optical flow estimation module_1→2,warp_3→2And original blurred video frame { I }₁,I₂,I₃Sending the data into a deblurring module together;

and 2, the deblurring module is a coding and decoding structure, wherein each coding block is formed by adding three residual blocks to 1-layer convolution, and each decoding block is formed by adding one deconvolution to the three residual blocks. In the process of deblurring, the clear feature map { feature _ coarse, feature _ fine } obtained by the clear feature fusion module is embedded into an encoder and a decoder of the deblurring modulePart of, finally obtaining a sheet and I₂Corresponding sharp image R₂. Clear image R generated by calculation₂With the original I₂Reference sharp image G₂To update the entire network. The process can be described as follows:

input＝[warp_1→2,I₁,I₂,I₃,warp_3→2],

En_out＝En(input,feature_coarse,feature_fine),

De_out＝De(En_out,feature_coarse,feature_fine)；

where En () denotes an encoder part and De () denotes a decoder part.

And 3, training according to the steps until the maximum iteration number is reached, and finishing training when the generated model is reached.

According to the fifth panel of fig. 1, the test procedure is as follows:

step 1, inputting 10 video test samples obtained by the data preparation part into a trained model;

and 2, selecting the clear frame, namely selecting any clear frame different from the fuzzy scene, and inputting the clear frame into the trained model to finally obtain a clear image. Fig. 3 is a diagram of the visual effect of the present invention.

Claims

1. A video deblurring method with fusion of clear features specifically comprises the following steps

Step A: designing an optical flow estimation module, inputting three continuous fuzzy frames into the optical flow estimation module, and outputting a result obtained by performing image torsion on two adjacent frames;

2. The method according to claim 1, wherein step a comprises in particular the steps of:

step A01, presetting training parameters, including a learning rate 1e-6 of an optical flow estimation module, a learning rate 1e-4 of a deblurring module, and a maximum iteration epoch of 500;

step A02. selecting three continuous blurred images { I₁,I₂,I₃Estimating the optical flow between two adjacent frames by FlowNet S_1→2,flow_3→2Obtaining two 2-channel optical flow graphs respectively, performing image torsion (image warp) operation on the optical flow and an adjacent frame, and twisting the adjacent frame to an intermediate frame to finally obtain two 3-channel RGB images { warp_1→2,warp_3→2}。

3. The method according to claim 1, wherein step B comprises in particular the steps of:

step B01, selecting three clear images { S₁,S₂,S₃And the three clear images are different from the content of the blurred image in the step A02, and the three clear images are passed through a feature extraction module to obtain 32 256 × 256 feature maps { feature _ coarse } and 64 × 64 feature maps { feature _ fine } respectively.

4. The method according to claim 1, wherein step C comprises in particular the steps of:

step C01. get the result after warp from step A02 { warp_1→2,warp_3→2And original blurred video frame { I }₁,I₂,I₃Sending the data into a deblurring module together; in the process of deblurring, embedding the clear feature map { feature _ coarse, feature _ fine } obtained in the step B01 into a deblurring module to finally obtain a feature map I₂Corresponding sharp image R₂(ii) a Clear image R generated by calculation₂With the original I₂Reference sharp image G₂To update the entire network;

c02, training according to the steps until the maximum iteration number is reached, and finishing training when the generated model is reached;

and C03, after the model training is finished, inputting three continuous fuzzy video frames, and selecting a clear frame of any scene to extract clear features to finally obtain a clear image.