CN103888767A - Frame rate improving method with UMH block matching motion estimation and optical flow field motion estimation combined - Google Patents

Abstract

The invention provides a frame rate improving method which mainly comprises four steps that image segmentation is carried out, and the foreground, the background and the object edge are obtained; the foreground and the background are subjected to variable size block matching motion estimation, and the object edge is subjected to motion estimation based on an optical flow field; a motion vector is subjected to post processing, and a proper motion vector is obtained; and a method of overlapped block motion compensation and bilinear interpolation is used for motion compensation, and interpolation frames are synthesized. According to the frame rate improving method, the problems that the halo effect, the edge block sawtooth effect and the like happen in a traditional frame rate improving method can be solved, and the method is widely used in the field of frame rate improving.

Description

UMH block matching motion is estimated a kind of frame per second method for improving combining with optical flow field estimation

Technical field

The present invention relates to a kind of method that frame-rate video promotes, belong to video data process field.

Background technology

The frame per second method for improving of video is the visual quality that the frame by insert prediction in original frame per second video sequence promotes low frame-rate video, obtains the video of high frame per second.Because video frame rate promotes diversified application, frame per second lift technique is more and more important in consumer electronics field.HDTV and multimedia PC system can be play the video higher than broadcast video stream frame per second, and video frame rate lift technique just can be applied to lifting original video frame per second and improve terminal use's viewing effect.

It is the motion compensation process based on estimation that video frame rate lifting at present adopts maximum methods, and most frame per second method for improving is all the method for estimating based on piece coupling.But at the intersection of prospect and background, due to the inaccuracy of the object edge piece coupling of moving, cause estimating inaccurate motion vector, thereby there will be the problem such as halo effect, jagged edges in inaccessible region, cause the video quality after frame per second promotes to decline, affect terminal use's visual effect.

Summary of the invention

The halo effect existing in promoting for frame per second, the problem of jagged edges, prospect, background separation that the application provides a kind of moving object to detect, the video frame rate method for improving that the estimation that employing is mated based on piece and optical flow field estimation combine, in order to improve the video quality after frame per second promotes.

Technical solution of the present invention is as follows:

Based on the video frame rate method for improving that UMHexagonS block matching motion is estimated and optical flow field estimation combines, it is characterized in that the method comprises the following steps:

Step 1: original video is processed, adopted frame-to-frame differences method to separate prospect background, and marker edge pixel;

Step 2: adopt variable-sized UMHexagonS block matching motion to estimate to obtain the motion vector of prospect, background;

Step 3: adopt optical flow field estimation to obtain moving object edge pixel point motion vector;

Step 4: the motion vector obtaining is carried out to reprocessing;

Step 5: prospect and background are carried out to overlapped block motion compensation, object edge is carried out to bilinear interpolation motion compensation, obtain inserting frame;

Step 6: will insert the video of the synthetic high frame per second of frame and primitive frame.

Preferably, in step 2 and step 3, prospect, background, object edge are adopted respectively to adaptive movement estimation method, to improve the accuracy of object edge pixel motion vector.

Preferably, in step 4, the motion vector obtaining is carried out to reliability judgement, insecure motion vector is carried out to medium filtering, to improve the accuracy of motion vector.

Preferably, in step 5, carry out overlapped block motion compensation, to reduce blocking effect, to improve video quality.

Brief description of the drawings

Fig. 1: be disposed of in its entirety block diagram of the present invention.

Fig. 2: UMHexagonS block matching motion estimation method schematic diagram.

Fig. 3: simulation result figure.

Embodiment

The present invention is directed to the motion conditions of objects in images, be divided into prospect, background, fringe region, adopt respectively size variable block matching motion method of estimation, method based on optical flow field estimation, the method construct that adopts overlapped block motion compensation after motion vector reprocessing is inserted to frame, play the effect that reduces halo effect, edge sawtooth, reach the target of the high frame-rate video of reconstruct high-quality.

Below in conjunction with specific embodiment (but being not limited to this example) and accompanying drawing, the present invention is further detailed.

(1) processing to original digital image:

(1) read in video;

(2) counter t=1 is set, preserves successively i frame as present frame, t+2 frame is as next frame, and reserved t+1 frame is as incoming frame to be inserted;

(3) utilize dynamic self-adapting threshold value frame-to-frame differences method to carry out moving object detection and come separation prospect and background.Motion detection based on inter-frame difference is frame difference method, and it detects moving target according to the size of consecutive frame or brightness variation every between two field picture.Specifically comprise following step:

A. the difference of calculating t two field picture and t+2 two field picture by formula (1), is designated as D (x, y):

D (x, y)=| F _t+2(x, y)-F _t(x, y) | (formula 1)

Wherein: F _t(x, y), F _t+2(x, y) represents respectively the image in t and t+2 moment;

B. choosing of dynamic self-adapting threshold value TH, the value of threshold value TH be the highest gray value of difference diagram in frame difference method and lowest gray value poor 1/2.The maximum that records D (x, y) by step a is D _max, minimum value is D _min, TH=(D _max-D _min)/2;

C. difference image D (x, y) is carried out to binary conversion treatment, carries out image according to formula 2 and cut apart:

$R(x,y)=\left\{\begin{array}{cc}0& D(x,y)>\mathrm{TH}\\ 1& D(x,y)\≤\mathrm{TH}\end{array}\right.$ (formula 2)

In formula 2, R (x, y) is binaryzation difference image afterwards, and TH is the threshold value that image is cut apart.

(2) estimation stages of motion vector:

(1), with reference to the accompanying drawings shown in 2, for moving object, adopt piece to be of a size of the UMHexagonS bi-directional motion estimation method of 4*4.Concrete steps are as follows:

A. virtual incoming frame t+1 frame to be inserted, according to step (one's) segmentation result, is divided into the position at foreground object place the rectangular block of some 4*4, successively each fritter is carried out to bi-directional motion estimation;

B. the criterion of piece coupling is as follows: calculate least absolute error and (Sum of Absolute Difference, the SAD) of t frame and t+2 frame corresponding blocks according to formula 3, SAD is defined as follows:

$\mathrm{SAD}[B_{i,j},v]=\underset{s\∈B_{i,j}}{\mathrm{\Σ}}\left|f_t\right[s-v]-f_{t+1}[s+v]$ (formula 3)

Wherein, B _i,jbe the piece that will estimate in t+1 frame, the motion vector that v is candidate, s is the pixel that will insert in t+1, f _t[s-v] is that t+1 is mapped to forward pixel corresponding in t frame, f _t+1[s+v] is that t+1 frame is mapped to pixel corresponding in t+2 frame backward;

C. according to matching criterior, asymmetric cross multi-level hexagon lattice point (Unsymmetrical Multi-resolution Hexagon, UMHexagonS) searching method has adopted hybrid multilayer time way of search, and concrete steps are as follows:

Step1: the prediction of central point, first initial center point is carried out to median prediction, be then upper strata prediction, finally carry out the motion-vector prediction of front frame corresponding blocks;

Step2: the motion search that hybrid multilayer is inferior;

The asymmetric Cross Search of Step2.1, then centered by the optimum point searching, point carries out grid search, then point carries out large hexagon search centered by current optimum point;

Step2.2 expands hexagon search, until point or stop while reaching maximum search number of times centered by optimum point;

Step2.3 dwindles hunting zone, carries out diamond search, until point or stop while reaching maximum search number of times centered by optimum point.

(2), for relatively static background, adopt piece to be of a size of the UMHexagonS search bi-directional motion estimation method (concrete step is as step (1)) of 8*8;

(3) for the edge contour of prospect and background, adopt the estimation based on optical flow field method.The basic model of optical flow computation is: suppose that the brightness value that faces pixel on territory in a less space keeps constant, can calculate faster the local light flow field of moving image by least square method optimal method, concrete steps are as follows:

A. make f (x, y, t) represent continuous space-time Luminance Distribution, if along its brightness preservation of movement locus constant we can obtain:

$\frac{\mathrm{df}(x,y,t)}{\mathrm{dt}}=0$ (formula 4)

In formula 4, x, y is respectively along movement locus t variation in time.Use differential chain rule to obtain to formula 4:

$\frac{\∂f(x,y,t)}{\∂x}{v}_x(x,y,t)+\frac{\∂f(x,y,t)}{\∂y}{v}_y(x,y,t)+\frac{\∂f(x,y,t)}{\∂t}=0$ (formula 5)

In formula 5,

represent respectively along the component of the motion vector of space coordinates.Formula 5 is called optical flow equation or optical flow constraint condition, and formula 5 can also be write as the form of < inner product of vector >:

$\&lang;\&dtri;f(x,y,t),v(x,y,t)\&rang;+\frac{\mathrm{df}(x,y,t)}{\mathrm{dt}}=0$ (formula 6)

B. the minimum value that the flow vector of sports ground changes by pixel should meet optical flow equation, order:

${\mathrm{\&epsiv;}}_{\mathrm{of}}\left(v(x,y,t)\right)=\&lang;\&dtri;f(x,y,t),v(x,y,t)\&rang;+\frac{\mathrm{df}(x,y,t)}{\mathrm{dt}}$ (formula 7)

Formula 7 represents the error in optical flow equation, works as ε _of(v (x, y, t)) equal 0 time, meet optical flow equation.In the situation that blocking with noise, obtain ε _ofthe minimum value of (v (x, y, t)) quadratic power.We can ask light stream with regularization method, make following formula minimum:

$\&Integral;{(\&dtri;\mathrm{fgv}+\frac{\&PartialD;f}{\&PartialD;t})}^2+\mathrm{\&lambda;}\left(\right||{\&dtri;}_{v_x}{\left|\right|}^2+|\left|{\&dtri;}_{v_y}{\left|\right|}^2\right)\mathrm{dx}$ (formula 8)

In formula 8, ${\&dtri;}_{v_x}={(\frac{{\&PartialD;}_{v_x}}{{\&PartialD;}_x},\frac{{\&PartialD;}_{v_x}}{{\&PartialD;}_y})}^T,{\&dtri;}_{v_y}={(\frac{{\&PartialD;}_{v_y}}{{\&PartialD;}_x},\frac{{\&PartialD;}_{v_y}}{{\&PartialD;}_y})}^T,$ λ Lagrange's multiplier, if derivative ▽ f and

more accurate the obtaining of energy, the desirable the greater of parameter, otherwise desirable smaller.

(3) motion vector post-processing stages:

A. the judgement of motion vector reliability:

Step1: calculate and want decision block (being designated as B piece) and the mean value of the motion vector of eight pieces around thereof:

$v_m=\frac{1}{9}\underset{i=1}{\overset{9}{\mathrm{\&Sigma;}}}{v}_i$ (formula 9)

V in formula _mfor mean value, v _ithe representative motion vector of eight pieces around respectively.

Step2: calculated difference:

$\mathrm{Dn}=\frac{1}{8}\underset{i=2}{\overset{9}{\mathrm{\&Sigma;}}}{|v}_m-v_i|$ (formula 10)

V in formula _mfor mean value, v _irepresent the motion vector of B piece.

Step3: calculate mean difference:

Dc=|v _m-v ₁| (formula 11)

Step4: judgement, if Dc > is Dn, v ₁for unreliable motion vector, need medium filtering.

B. unreliable motion vector is carried out to medium filtering:

V _1smooth=median[v ₁, v ₂, v ₃..., v ₉] (formula 12)

(4) motion compensation stage: prospect and background object are adopted to overlapped block motion compensation method (OBMC), to the motion compensation process of prospect and background edge employing bilinear interpolation.When estimation of motion vectors is inaccurate or object of which movement is not, while having multiple different objects motion in simple translational motion and a piece, to adopt overlapped block motion compensation method can solve blocking effect problem.Adopt OBMC method, the prediction of a pixel is the estimation of the motion vector based on piece under it not only, also the estimation of motion vectors based on adjacent block.

In traditional video frame rate method for improving, when moving object edge is done to block-based estimation, can use the pixel of background, this has just caused the inaccuracy of margin estimation.Adopt the estimation based on pixel just can not have the problem of estimating moving object Pixel Information by background pixel information at moving object edge, thereby can obtain correct motion vector, effectively solved edge-light toroidal effect and sawtooth piece problem.

As shown in Figure 3, in figure, from left to right, be once respectively from top to bottom UMHexagonSexagonS block matching motion and estimate first motion compensation, overlapped block motion compensation, the motion compensation of optical flow field estimation bilinear interpolation, this patent motion compensation simulation result.

The present invention adopts standard yuv video cycle tests foreman sequence to obtain simulation result, with based on UMHexagonSexagonS motion estimation and compensation method, compare based on optical flow field estimation interpolation method, overlapped block motion compensation method, can find out that method of the present invention efficiently solves the problem of edge-light toroidal effect, edge sawtooth piece.

Claims (4)

1. the video frame rate method for improving based on UMHexagonS block matching motion is estimated and optical flow field estimation combines, is characterized in that the method comprises the following steps:

Step 1: original video is processed, adopted frame-to-frame differences method to separate prospect background, and marker edge pixel;

Step 2: adopt variable-sized UMHexagonS block matching motion to estimate to obtain the motion vector of prospect, background;

Step 3: adopt optical flow field estimation to obtain moving object edge pixel point motion vector;

Step 4: the motion vector obtaining is carried out to reprocessing;

Step 5: prospect and background are carried out to overlapped block motion compensation, object edge is carried out to bilinear interpolation motion compensation, obtain inserting frame;

Step 6: will insert the video of the synthetic high frame per second of frame and primitive frame.

2. the video frame rate method for improving based on UMHexagonS block matching motion is estimated and optical flow field estimation combines according to claim 1, it is characterized in that, in step 2 and step 3, prospect, background, object edge are adopted respectively to adaptive movement estimation method, to improve the accuracy of object edge pixel motion vector.

3. the video frame rate method for improving based on UMHexagonS block matching motion is estimated and optical flow field estimation combines according to claim 1, it is characterized in that, in step 4, the motion vector obtaining is carried out to reliability judgement, insecure motion vector is carried out to medium filtering, to improve the accuracy of motion vector.

4. the video frame rate method for improving based on UMHexagonS block matching motion is estimated and optical flow field estimation combines according to claim 1, is characterized in that carrying out overlapped block motion compensation in step 5, to reduce blocking effect, to improve video quality.

Images