CN101621693B - Frame frequency lifting method for combining target partition and irregular block compensation - Google Patents

Abstract

The invention discloses a frame frequency lifting method for combining target partition and irregular block compensation, which is characterized by comprising the following steps that: firstly, establishing a blank frame to be inserted; secondly, sampling a reference frame and the blank frame to be inserted; thirdly, calculating the frame difference between the reference frame and the blank frameto obtain a moving target outline and a moving target area mask by applying the morphological method; fourthly, partitioning the two reference frames into a target area and a background area; fifthly, moving and estimating subarea processing to obtain an initial motion vector; sixthly, smoothening the obtained initial motion vector; seventhly, trimming the initial motion vector after being smoothened to obtain a more accurate motion vector; eighthly, calculating the compensation weight coefficient Mu; ninthly, partitioning the block into a high relevant area and a low relevant area; tenthly, compensating subarea processing; and eleventhly, generating an interpolated frame. The method can be widely applied to the technical fields of video coding and encoding and video processing.

Description

A kind of combining target is cut apart the frame frequency lifting method with irregular block compensation

Technical field

The present invention relates to technical field of video processing, relate in particular to a kind of combining target and cut apart frame frequency lifting method with irregular block compensation.

Background technology

In low bit rate video coding, some encoders adopt frequency hopping to satisfy the desired compression rate.In this case, the picture frame in the list entries is periodically abandoned from cataloged procedure, thereby time pseudomorphism (as shake) is introduced in receiving terminal, will cause the serious decline of video quality.At this moment, need on decoder, use frame rate up conversion (FRUC) to regenerate the picture frame of being skipped, so that reduce the time pseudomorphism.

In addition, the liquid crystal LCD display obtains development and application widely in recent years, and its defective is to produce motion blur phenomenons such as hangover and ghost when video display, and main cause is that the liquid crystal display retention time is long and the response time is slower.Because it is limited to reduce liquid crystal response time degree, has reached bottleneck at present, therefore reducing the liquid crystal retention time becomes the main solution of this problem.Can effectively reduce the retention time by the method that improves the LCD refreshing frequency, promptly on former frame frequency basis with one times or many times of frame frequency lifting.

Promote the method for frame frequency, have and adopt the method that repeats interleave that each frame is repeated once to input to the 120Hz panel, but display effect is suitable with 60Hz like this, the motion flicker also can occur.Also have and insert black frame raising frame frequency between two frames, such result is that whole video display effect can deepening.

Frame frequency lifting algorithm based on motion compensation is more effective interleave method, two frames before and after can connecting more realistically, more changing naturally near video sequence.Obtaining the key of better interpolation frame based on the frame frequency lifting method of motion compensation, is the optimization process of the accuracy and the motion compensated result of estimation.Therefore, thus how to estimate motion vector accurately and carry out more excellent motion compensation that to produce high-quality interpolation frame be the direction that the industry researcher makes joint efforts.

Block-based estimation is a method for estimating at present commonly used, and minimum sad value criterion is based on piece coupling matching criterior commonly used in the estimation of piece.The bilinear interpolation algorithm is usually used in accessing the high-quality zoomed image in the image zoom processing, dwindles the reference frame size with this algorithm, can reduce the amount of calculation in the motion estimation process.

In traditional method for estimating, with a certain reference frame or treat interleave integral body, then in another reference frame search or the search of front and back reference frame symmetry by certain regular piecemeal.Consider in the frame of video disunities such as the target object amount of exercise is big, motion size/direction, and background area motion is mainly moved by camera and is caused, amount of exercise is less, and motion is consistent, for estimation and compensation accurately, be necessary with target, background area separate processes.On method for estimating, mainly contain two kinds of unidirectional estimation and bilateral estimations, bilateral estimation because of its to treat that interleave is the mirror image center, interpolation frame can not overlap with leak and be widely used when having guaranteed compensation.In addition, in order to obtain motion vector accurately, Many researchers has been done a lot of improvement to method for estimating, as document [1] (Suk-Ju Kang, Kyoung-Rok Cho, Young Hwan Kim.Motion Compensated Frame Rate Up-ConversionUsing Extended Bilateral Motion Estimation.IEEE Transactions onConsumer Electronics, Nov.2007,53 (4): the bilateral estimation of expansion (EBME) that adopts 1759-1761.), as shown in Figure 1, suppose that the compensation block sizes size is n * n, the piece that at first interpolation frame is divided into 2n * 2n carries out bilateral estimation, obtains the mass movement vector by minimum SAD criterion.The capable pixel of matching window n that glides is then estimated the new window motion vector, and this moment, new window had the repetitive of 2 n * n with former matching window, relatively before and after the sad value of twice coupling, the motion vector of the SAD correspondence of choosing minimum is as the motion vector of repetitive.The method can obtain accurately motion vector, but does not consider the matching degree of fritter when mating, and might be that the non-optimum Match of optimum Match fritter is made mistakes because of its bulk.

Block-based compensation method commonly used at present is that pixel is not a pixel in the same objective body in the if block with the motion of whole interior pixel of a motion vector representative, and then its motion is inconsistent possibly.Error will appear in therefore, traditional block compensation.Document [2] (is coerced a Thailand greatly, her rattan is firm, three islands are straight. produce the method and apparatus of interpolation frame. and Toshiba Corporation, 2006,10) after finding match block, according to the degree of correlation between matched pixel match block is divided into irregular high and low relevant portion, adopts diverse ways to compensate in when compensation according to the two characteristics.But the division of its height relevant portion is to judge according to the pixel value difference setting threshold to 2 match block, certainly will set different produce erroneous judgements, compensation for loss and damage effect because of thresholding.In order accurately to compensate interpolation, propose rationally to judge the method for height relevant range, significant.

Block-based motion compensation produces the inevitable problem of interpolation frame and occurs blocking effect exactly, it can have a strong impact on the visual quality and the objective evaluation of interpolation frame, discover overlapped block motion compensation (Overlapped Block Motion Compensation through numerous researchers, OBMC) can effectively eliminate blocking effect, improve compensation effect.Document [3] (M.T.Orchardand C.J.Sullivan, Overlapped block motion compensation:an estimation-theoretic approach, IEEE Trans.Image Processing, Sept.1994,3 (9): 693-699.) the overlapped block motion compensation effect has been done valuable assessment, and the optimum superposing window shape has been discussed.The method has very high practical value for the block-based movement compensating algorithm that generally adopts at present.

Summary of the invention

Technical problem to be solved by this invention provides a kind of a kind of combining target that is used for coding and decoding video and improves the liquid crystal display quality and cuts apart frame frequency lifting method with irregular block compensation.

According to a kind of technical scheme of the present invention, a kind of combining target is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that: may further comprise the steps:

The first, from video sequence, the former frame that extracts present frame and present frame is set up blank and is treated interleave as the reference frame;

The second,, utilize bilinear interpolation that reference frame and blank are treated that interleave descends sampling for reducing the motion estimation process amount of calculation;

Three, two reference frame frames are poor before and after the calculating, obtain the moving target profile, and the utilization morphological method obtains the motion target area mask;

Four,, respectively two reference frames are divided into target area and background area according to the motion target area mask;

Five, estimation subarea processing, treat the interleave piecemeal after will descending sampling, adopt neighborhood to select bilateral method for estimating, below treat that the interleave piece is the mirror image center after the sampling, seek symmetrical match block respectively in target area in the reference frame of front and back and the background area, obtain initial motion vectors;

Six, the initial motion vectors that obtains is carried out smoothly the blank that first step obtains being treated the interleave piecemeal;

Seven, the initial motion vectors block-by-block after will be level and smooth corresponds to the blank that first step obtains and treats interleave, treat that with blank the interleave piece is that the piece match search is carried out in the peripheral region in the mirror image center in the front and back reference frame that level and smooth back initial motion vectors is pointed to, obtain final motion vector, promptly the initial motion vectors after level and smooth is finely tuned to obtain motion vector more accurately;

Eight, according to the final motion vector that obtains in the 7th step, generate initial interpolation frame match block is got averaging method, its pixel constantly is updated to correct interpolating pixel in the interpolation compensation process simultaneously; The backoff weight coefficient μ of match block in the front and back reference frame of each piece correspondence in the interleave is treated in calculating;

Nine, according to the final motion vector that obtains in the 7th step, treat each piece in the interleave, match block in the reference frame according to its respective pixel similarity degree, is divided into high relevant range and low relevant range with piece before and after finding;

Ten, compensation subarea processing, treat each piece in the interleave, if match block belongs to the target area in the front and back reference frame of its correspondence, adopt target area interpolation compensation method, the two-way OBMC method compensation of match block respective pixel average in the reference frame before and after then its high relevant range interpolate value is got, low relevant range based on weight; If match block belongs to the background area in the front and back reference frame of its correspondence, adopt background area interpolation compensation method, then its same target area of high relevant range compensation method is hanged down the relevant range and is adopted the unidirectional OBMC method based on weight to compensate;

The 11, repeat the 8th, the 9th, the tenth step, until generating interpolation frame.

According to a preferred version of the present invention, the estimation subarea processing in the 5th step is: estimation is according to the motion target area mask of being divided, and the estimation of object block is carried out in the target area, and the estimation of background piece is carried out in the background area.

According to a preferred version of the present invention, neighborhood in the 5th step selects bilateral method for estimating to be: according to motion vector zone consistency characteristics, for treating in the interleave a certain, the match block size is selected 4 times of sizes of its size during search, in the reference frame of front and back, carry out big block search, based on minimum sad value criterion, the motion vector that obtains bulk is as the alternative motion vector of current block; And then get its left side, go up, the motion vector of upper left these 3 pieces is as alternative motion vector, calculates the sad value of these 4 the corresponding match block of alternative motion vector respectively, selects the best initial motion vectors of the motion vector of minimum sad value correspondence as current block.

According to a preferred version of the present invention, the method of calculating the backoff weight coefficient μ of match block in the front and back reference frame of each piece correspondence in the frame to be inserted in the 8th step is: according to the final motion vector that obtains in the 7th step, generate initial interpolation frame match block is got averaging method

Its pixel constantly is updated to correct interpolating pixel in compensation process simultaneously, at certain to be compensated, calculates itself and the absolute value error that extends out 4 pixel regions around the corresponding match block of front and back reference frame, is designated as MAD with last reference frame corresponding region absolute value error _f, be designated as MAD with back one reference frame corresponding region absolute value error _b,

Backoff weight coefficient μ is obtained by following formula:

$\mathrm{\μ}=\frac{{\mathrm{MAD}}_b}{{\mathrm{MAD}}_f+{\mathrm{MAD}}_b}$

Reference frame match block respective pixel was to the contribution proportion of interpolation frame interpolating pixel before and after backoff weight coefficient μ was used for determining.

According to a preferred version of the present invention, the division methods of high relevant range and low relevant range is in the 9th step: when high relevant range and low relevant range are divided, suppose that the former frame of to be divided correspondence and back one frame match block are respectively B _N-1(s), B _N+1(s), wherein s represents location of pixels in the piece, then makes the respective pixel of two pieces be divided by, and obtains matched pixel merchant dif_blk (s)=B _N-1(s)/B _N+1(s)

Set reference value Th=1, ε is a minimum as thresholding, each location of pixels s in the corresponding blocks, if | dif_blk (s)-Th|＜ε, then think in the match block respective pixel position for high relevant, otherwise for low relevant.

According to a preferred version of the present invention, the compensation method of target area interpolation described in the tenth step:

To be compensated of target area, according to the match block similarity degree difference of to be compensated correspondence it is divided into high and low relevant range, matched pixel average compensation before and after adopting for the relevant zone of height, wherein, v is a block motion vector to be compensated,

Be pixel value to be compensated, f _N-1(s-v) be corresponding blocks pixel value in the last reference frame with to be compensated coupling, f _N+1(s+v) be corresponding blocks pixel value in the back reference frame with to be compensated coupling,

${\hat{f}}_n\left(s\right)=0.5*f_{n-1}(s-v)+0.5*f_{n+1}(s+v)$

The two-way OBMC compensation based on weight, wherein w are adopted in low relevant range _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling, penalty coefficient ρ gets backoff weight coefficient μ

${\hat{f}}_n\left(s\right)=\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)*(\mathrm{\ρ}*f_{n-1}(s-v_i)+(1-\mathrm{\ρ})*f_{n+1}(s+v_i)),\mathrm{\ρ}=\mathrm{\μ}.$

According to a preferred version of the present invention, the compensation method of background area interpolation described in the tenth step:

To the same method of dividing high and low relevant range that adopts of the interpolation compensation of background area, can directly get average as interpolation frame correspondence position pixel to high relevant range to matched pixel in the reference frame of front and back, wherein, v is a block motion vector to be compensated, Be pixel value to be compensated, f _N-1(s-v) be corresponding blocks pixel value in the last reference frame with to be compensated coupling, f _N+1(s+v) be corresponding blocks pixel value in the back reference frame with to be compensated coupling,

${\hat{f}}_n\left(s\right)=0.5*f_{n-1}(s-v)+0.5*f_{n+1}(s+v)$

For low relevant range, adopt unidirectional compensation based on OBMC:

${\hat{f}}_n\left(s\right)=\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)*(\mathrm{\ρ}*f_{n-1}(s-v_i)+(1-\mathrm{\ρ})*f_{n+1}(s+v_i))$

Wherein, the value of penalty coefficient ρ is by backoff weight coefficient μ decision, if backoff weight coefficient μ＞0.5, and penalty coefficient ρ=1 then, otherwise ρ=0;

w _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling.

The beneficial effect that a kind of combining target of the present invention is cut apart with the frame frequency lifting method of irregular block compensation is: the present invention is based on target and cut apart with neighborhood and select bilateral estimation method can estimate motion vector exactly, the self-adapting compensation method that combining target is cut apart with irregular block compensation can accurately compensate and reduce blocking effect, during compensation by the backoff weight coefficient of reference frame is set, the realization self adaptation is single, bidirectional compensating, quotient with front and back reference frame match block is judged height, low relevant irregular block, mark off the height relevant range more accurately, OBMC is applied to self-adapting compensation method, effectively remove the blocking effect that compensation produces, compared with prior art, this method can produce the interpolation frame of better quality, can be applied in coding and decoding video widely, technical field of video processing, can improve the liquid crystal display quality, so the present invention has important use value.

Description of drawings

Below with reference to accompanying drawing embodiments of the invention are further described.

Fig. 1 a is an estimated motion vector territory exemplary plot for the first time in the EBME method for estimating that adopts in the document [1];

Fig. 1 b is a middle estimated motion vector territory exemplary plot in the EBME method for estimating that adopts in the document [1];

Fig. 1 c is a final motion vector territory exemplary plot in the EBME method for estimating that adopts in the document [1];

Fig. 2 is that a kind of combining target of the present invention is cut apart the frame frequency lifting method embodiment flow chart with irregular block compensation;

Fig. 3 is a generation interpolation frame motion compensation flow chart of the present invention;

Fig. 4 a is that the motion search match block is divided schematic diagram in the method for estimating of the present invention;

Fig. 4 b is the alternative motion vector schematic diagram of current block in the method for estimating of the present invention;

Fig. 5 is that the relevant irregular area of height of the present invention is divided example;

Fig. 6 is a bilinearity window exemplary plot among the used OBMC among the present invention;

Fig. 7 is 9 neighborhood window schematic diagrames among the OBMC of the present invention;

Fig. 8 is a backoff weight coefficient calculations neighborhood schematic diagram of the present invention.

Embodiment

Fig. 2 is that a kind of combining target of the present invention is cut apart the frame frequency lifting method specific embodiment flow chart with irregular block compensation.Referring to Fig. 2, a kind of combining target is cut apart the frame frequency lifting method with irregular block compensation, and following steps are arranged:

The first, from video sequence, the former frame that extracts present frame and present frame is set up blank and is treated interleave as the reference frame;

The second,, utilize bilinear interpolation that reference frame and blank are treated that interleave descends sampling for reducing the motion estimation process amount of calculation;

Three, two reference frame frames are poor before and after the calculating, obtain the moving target profile, and the utilization morphological method obtains the motion target area mask;

Four,, respectively two reference frames are divided into target area and background area according to the motion target area mask;

Five, estimation subarea processing, treat the interleave piecemeal after will descending sampling, adopt neighborhood to select bilateral method for estimating, below treat that the interleave piece is the mirror image center after the sampling, seek symmetrical match block respectively in target area in the reference frame of front and back and the background area, obtain initial motion vectors;

Six, the initial motion vectors that obtains is carried out smoothly the blank that first step obtains being treated the interleave piecemeal;

Seven, the initial motion vectors block-by-block after will be level and smooth corresponds to the blank that first step obtains and treats interleave, treat that with blank the interleave piece is that the piece match search is carried out in the peripheral region in the mirror image center in the front and back reference frame that level and smooth back initial motion vectors is pointed to, obtain final motion vector, promptly the initial motion vectors after level and smooth is finely tuned to obtain motion vector more accurately;

Eight, according to the final motion vector that obtains in the 7th step, generate initial interpolation frame match block is got averaging method, its pixel constantly is updated to correct interpolating pixel in the interpolation compensation process simultaneously; The backoff weight coefficient μ of match block in the front and back reference frame of each piece correspondence in the interleave is treated in calculating;

Nine, according to the final motion vector that obtains in the 7th step, treat each piece in the interleave, match block in the reference frame according to its respective pixel similarity degree, is divided into high relevant range and low relevant range with piece before and after finding;

Ten, compensation subarea processing, treat each piece in the interleave, if match block belongs to the target area in the front and back reference frame of its correspondence, adopt target area interpolation compensation method, the two-way OBMC method compensation of match block respective pixel average in the reference frame before and after then its high relevant range interpolate value is got, low relevant range based on weight; If match block belongs to the background area in the front and back reference frame of its correspondence, adopt background area interpolation compensation method, then its same target area of high relevant range compensation method is hanged down the relevant range and is adopted the unidirectional OBMC method based on weight to compensate;

The 11, repeat the 8th, the 9th, the tenth step, until generating interpolation frame.

Below the embodiment of the invention is described in further detail:

As shown in Figure 2, S1 reads in 2 frame successive video frames from video sequence, respectively as last reference frame f _N-1(s) and the back one reference frame f _N+1(s), the measure-alike blank of foundation and reference frame is treated interleave

S2 adopts bilinear interpolation that front and back two reference frames and blank are treated sampling under the interleave, obtains reference frame f ' _N-1(s), f ' _N+1(s) and treat interleave

Following sampling can not change the structural information of reference frame, but can reduce amount of calculation in the search matching process;

Two reference frame frames were poor before and after S3 calculated, and abs takes absolute value to it by function, obtains the moving target profile to be:

f _obj＝abs(f _n-1(s)-f _n+1(s))；

S4 at first by setting threshold with moving target profile f _ObjBecome bianry image, use morphological method to handle again: to be that 1 rhombus is a structural element with radius earlier, bianry image is carried out ON operation, to remove thinner and more delicate edge and noise in the bianry image, these edge major parts are that non-motion change causes, are that 16 rhombus is a structural element with radius again, image after handling is carried out closed operation, the moving region is connected as a single entity, obtains the moving target bianry image, draw the motion target area mask thus; In the time of necessary, obvious as background area frame difference, need the estimating target position, limited target intercepting zone is so that complete separate targets zone, background area; Target area, background area separation method not only are confined to method described herein, and other can be used for the method that target cuts apart and all can adopt;

S5 is according to the motion target area mask, with reference frame f ' _N-1(s), f ' _N+1(s) be divided into target area and background area, use for next step motion search;

S6 is to treat that interleave is the mirror image center, and the bilateral method for estimating that adopts neighborhood to select carries out the estimation subarea processing.The bilateral method for estimating that neighborhood is selected is used for reference document [1] algorithm thinking, consider regional movement vector consistency characteristics, and take into full account the coupling of fritter, on the basis of big block search coupling, carry out the fritter coupling, from neighborhood and current motion vector, select optimum movement vector as initial motion vectors;

Concrete grammar is described as: at first will

Be divided into the piece that is of a size of n * n, with wherein one

Estimation be example, to comprise

The bulk of 2n * 2n that is of a size of its 4 times of sizes is as coupling mirror image center, shown in Fig. 4 (a), at reference frame f ' _N-1(s), f ' _N+1(s) search for symmetrical match block in, obtain the motion vector V_cur of bulk, assign it to current block, as piece

Alternative motion vector; If piece

A left side, go up, upper left 3 adjacent block motion vectors are respectively V_left, V_up, V_dia, shown in Fig. 4 (b), relatively under 4 motion vector situations,

The sad value of corresponding match block is selected the motion vector conduct of minimum value correspondence

Initial motion vectors.

In, according to the target that obtains among the S5, background area, if Belong to the target area, then in the target area of reference frame, carry out above-mentioned estimation, otherwise in the background area, carry out.Block-by-block carries out, until finishing whole frame estimation.It is benchmark that the method is mated with current block most, considers the consistency of regional movement again, can obtain motion vector more accurately;

The initial motion vectors that S7 obtained the last step adopts medium filtering or other filtering methods to carry out smoothly;

S8 treats interleave with blank

Be divided into the piece of 2n * 2n, will

In every motion vector correspond to

In corresponding fritter (size is the motion vector of n * n); With

By piecemeal is the mirror image center, at the f of its motion vector points _N-1(s) and f _N+1(s) near zone of position carries out the piece match search in, obtains final motion vector, promptly the initial motion vectors that obtains among the S7 is finely tuned to obtain motion vector more accurately;

S9 generates initial interpolation frame according to the final motion vector that obtains match block is got averaging method, and its pixel constantly is updated to correct interpolating pixel in the interpolation compensation process simultaneously, calculates the backoff weight coefficient μ that treats the relative front and back of each piece reference frame in the interleave;

Right

In each piece, match block in the reference frame before and after finding, the pixel similarity degree according to match block in its corresponding reference frame is divided into high relevant range and low relevant range with piece, example is divided as shown in Figure 5 in the height relevant range; When carrying out motion compensation, if match block belongs to the target area in the front and back reference frame of its correspondence, match block respective pixel average in the reference frame before and after then its high relevant range interpolate value is got, of the two-way OBMC method compensation of low relevant range based on weight; If match block belongs to the background area in the front and back reference frame of its correspondence, then its same target area of high relevant range compensation method is hanged down the relevant range and is adopted the unidirectional OBMC method based on weight to compensate;

Repeat the S9 step, until the traversal interpolation frame.

Fig. 3 is the irregular block compensation detail flowchart that S9 unit combining target is cut apart among Fig. 2.

S91 at first scans and treats interleave, and order is extracted and treated interleave

In to be compensated;

S92 extracts reference frame f according to the motion vector of to be compensated correspondence _N-1(s), f _N+1(s) the corresponding match block B in _N-1(s), B _N+1(s);

S93 divides the relevant irregular area of height according to pixel value similarity degree in the match block.Wherein s remarked pixel coordinate makes two piece respective pixel be divided by, and obtains the matched pixel merchant:

dif_blk(s)＝B _n-1(s)/B _n+1(s)，

Set reference value Th=1, ε is a threshold value, it is a minimum, each position s in the corresponding blocks, if | dif_blk (s)-Th|＜ε, think that then the relevant position is for high relevant in the match block, the value M (s)=1 of correspondence position in the mask is set, otherwise for low relevant, M (s)=0 is set, thereby obtains the relevant irregular mask of height; Mask generates example as shown in Figure 5.By thresholding ε is set, the high accuracy that can obtain needing neatly is correlated with or low precision is correlated with, and this method makes the division of height relevant range more accurate and convenient.

According to high and low relevant region of disorder domain classification method as can be known, high relevant range is to mate part preferably in the reference frame of front and back, so the two average promptly can be used for compensation.Low relevant range then is the relatively poor part of coupling in the reference frame, need belong to the target area or the background area is treated with a certain discrimination according to match block during compensation.

The S94 combining target is cut apart to divide with irregular area and is treated compensation block and compensate respectively.

The problem that block-based motion compensation is difficult to avoid is exactly a blocking effect, in order to eliminate blocking effect, obtain desirable interpolation frame, the present invention has used the OBMC technology, it can effectively remove blocking effect, and its core concept is to take into full account to be compensated motion vector correlation with its neighborhood piece, and the weighted sum of the offset by asking neighborhood block motion vector correspondence obtains interpolation, and the weighting weight is determined by window function, obtains more level and smooth sports ground by weighted interpolation.Window function is generally raised cosine window or bilinearity window, and it is as follows that the bilinearity window that the present invention uses generates formula, and its shape as shown in Figure 6.

$w\left(u\right)=\left\{\begin{array}{cc}\frac{1}{16}(u+1/2)& u=\mathrm{0,1}...15\\ w(31-u)& u=\mathrm{16,17}...31\end{array}\right.,w\left(v\right)=w\left(u\right),w(u,v)=w\left(u\right)*w\left(v\right)$

The setting compensation block size is 16 * 16, the rectangular window for 32 * 32, and its coverage comprises the part of current to be compensated adjacent 8 pieces with it.As shown in Figure 7, piece 5 is in the interpolation frame to be compensated, and its offset is relevant with 8 adjacent pieces around it and itself motion vector, establishes

Be pixel value to be inserted in the piece 5, s is a pixel coordinate in the piece 5, and compensation method is:

${\hat{f}}_n\left(s\right)=\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)*(\mathrm{\ρ}*f_{n-1}(s-v_i)+(1-\mathrm{\ρ})*f_{n+1}(s+v_i)),$ And $\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)=1$

Wherein, w _i(s) be coefficient of correspondence in the bilinearity window at center for center with piece i that current block coordinate s is corresponded to, ρ is a penalty coefficient, can get 0,1 or the backoff weight coefficient μ that obtains of previous calculations according to the classification in the compensation deals, concrete value is the compensation method summation of face as follows; v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling.This formula shows that each offset all is to comprise 9 smoothed motion vector of 3 neighborhoods of himself and the result of spectral window weighting in the piece 5, thereby effectively eliminates blocking effect.

In addition, consider for

In to be compensated, when match block compensate it in the corresponding front and back reference frame because of with its on every side piece suit that degree is different to have different contributions, the present invention introduces backoff weight coefficient μ.

According to the motion vector that S8 obtains, employing is got averaging method to match block and is generated initial interpolation frame

Its pixel constantly is updated to correct interpolating pixel in the interleave compensation process simultaneously.At

In certain to be compensated, as shown in Figure 8, calculate the absolute value error that extends out 4 pixel regions around itself and the corresponding match block of front and back reference frame, be designated as MAD with last reference frame corresponding region absolute value error _f, be designated as MAD with back one reference frame corresponding region absolute value error _b, N wherein _{I, j}For extending out 4 pixel regions, N ₁N ₂Be sum of all pixels in the flared region, v is the motion vector of interpolation frame piece s position correspondence,

${\mathrm{MAD}}_f=\frac{1}{N_1{N}_2}\underset{s\∈N_{i,j}}{\mathrm{\Σ}}|f_{n-1}(s-v)-{\hat{f}}_n^{\′\′}\left(s\right)|$

${\mathrm{MAD}}_b=\frac{1}{N_1{N}_2}\underset{s\∈N_{i,j}}{\mathrm{\Σ}}|f_{n+1}(s+v)-{\hat{f}}_n^{\′\′}\left(s\right)|$

Backoff weight coefficient μ is obtained by following formula:

$\mathrm{\μ}=\frac{{\mathrm{MAD}}_b}{{\mathrm{MAD}}_f+{\mathrm{MAD}}_b}$

The compensating proportion of reference frame respective pixel before and after backoff weight coefficient μ uses in the interleave compensation and decides.

Two kinds of situations in compensation partial objectives for zone and background area of low relevant range are handled, the target area motion is complicated, be easy to occur blocking effect, so low relevant portion adopts the two-way OBMC compensation based on weight in the target area, match block information before and after making full use of, and utilize the OBMC method to remove blocking effect; The background area general motion is more single, coherent, and motion estimation error is less, and the background area is generally more smooth, similarity degree is bigger, when 2 frame reference frame information compensated before and after only utilizing, in order to reduce the interpolated pixel noise, the unidirectional compensation based on OBMC was adopted in low relevant range.Certainly, if make full use of the continuity of background information, in the low relevant range that the former frame or back one frame of front and back reference frame are searched for the background area again, then for the translational motion sequence, as the car with fast moving is the video sequence victra of target object, and effect is preferably arranged.

Below combining target of the present invention is cut apart the compensation method of dividing with irregular block and does a summation:

During compensation, interleave is treated in block-by-block scanning

If match block belongs to the target area in the front and back reference frame of its correspondence, the average of match block pixel in the reference frame before and after then its high relevant range interpolate value is got, wherein, v is a block motion vector to be compensated,

Be pixel value to be compensated, f _N-1(s-v) be corresponding blocks pixel value in the last reference frame with to be compensated coupling, f _N+1(s+v) be corresponding blocks pixel value in the back reference frame with to be compensated coupling,

${\hat{f}}_n\left(s\right)=0.5*f_{n-1}(s-v)+0.5*f_{n+1}(s+v),$

The two-way OBMC filtering compensation based on weight is adopted in low relevant range; W wherein _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling, penalty coefficient ρ gets backoff weight coefficient μ

${\hat{f}}_n\left(s\right)=\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)*(\mathrm{\ρ}*f_{n-1}(s-v_i)+(1-\mathrm{\ρ})*f_{n+1}(s+v_i)),\mathrm{\ρ}=\mathrm{\μ}$

If match block belongs to the background area in the front and back reference frame of its correspondence, average is got as the interpolation frame respective pixel to match block pixel in the reference frame of front and back equally in high relevant range, and the unidirectional compensation based on OBMC is adopted in low relevant range,

${\hat{f}}_n\left(s\right)=\underset{i=1}{\overset{9}{\mathrm{\Σ}}}{w}_i\left(s\right)*(\mathrm{\ρ}*f_{n-1}(s-v_i)+(1-\mathrm{\ρ})*f_{n+1}(s+v_i))$

Wherein, w _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling, the value of penalty coefficient ρ is by backoff weight coefficient μ decision, if backoff weight coefficient μ＞0.5, penalty coefficient ρ=1 then, otherwise finish to scanning ρ=0, generates interpolation frame.

Above the specific embodiment of the present invention is described, still, the scope that is not limited only to embodiment of the present invention's protection.

Claims (7)

1. a combining target is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that: may further comprise the steps:

The first, from video sequence, the former frame that extracts present frame and present frame is set up blank and is treated interleave as the reference frame;

The second,, utilize bilinear interpolation that reference frame and blank are treated that interleave descends sampling for reducing the motion estimation process amount of calculation;

Three, two reference frame frames are poor before and after the calculating, obtain the moving target profile, and the utilization morphological method obtains the motion target area mask;

Four,, respectively two reference frames are divided into target area and background area according to the motion target area mask;

Five, estimation subarea processing, treat the interleave piecemeal after will descending sampling, adopt neighborhood to select bilateral method for estimating, below treat that the interleave piece is the mirror image center after the sampling, seek symmetrical match block respectively in target area in the reference frame of front and back and the background area, obtain initial motion vectors;

Six, the initial motion vectors that obtains is carried out smoothly the blank that first step obtains being treated the interleave piecemeal;

Seven, the initial motion vectors block-by-block after will be level and smooth corresponds to the blank that first step obtains and treats interleave, treat that with blank the interleave piece is that the piece match search is carried out in the peripheral region in the mirror image center in the front and back reference frame that level and smooth back initial motion vectors is pointed to, obtain final motion vector, promptly the initial motion vectors after level and smooth is finely tuned to obtain motion vector more accurately;

Eight, according to the final motion vector that obtains in the 7th step, generate initial interpolation frame match block is got averaging method, its pixel constantly is updated to correct interpolating pixel in the interpolation compensation process simultaneously; The backoff weight coefficient μ of match block in the front and back reference frame of each piece correspondence in the interleave is treated in calculating;

Nine, according to the final motion vector that obtains in the 7th step, treat each piece in the interleave, match block in the reference frame according to its respective pixel similarity degree, is divided into high relevant range and low relevant range with piece before and after finding;

Ten, compensation subarea processing, treat each piece in the interleave, if match block belongs to the target area in the front and back reference frame of its correspondence, adopt target area interpolation compensation method, the two-way OBMC method compensation of match block respective pixel average in the reference frame before and after then its high relevant range interpolate value is got, low relevant range based on weight; If match block belongs to the background area in the front and back reference frame of its correspondence, adopt background area interpolation compensation method, then its same target area of high relevant range compensation method is hanged down the relevant range and is adopted the unidirectional OBMC method based on weight to compensate;

The 11, repeat the 8th, the 9th, the tenth step, until generating interpolation frame.

2. a kind of combining target according to claim 1 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that, estimation subarea processing in the 5th step is: estimation is according to the motion target area mask of being divided, the estimation of object block is carried out in the target area, and the estimation of background piece is carried out in the background area.

3. a kind of combining target according to claim 2 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that, neighborhood in the 5th step selects bilateral method for estimating to be: according to motion vector zone consistency characteristics, for treating in the interleave a certain, the match block size is selected 4 times of sizes of its size during search, carry out big block search in the reference frame of front and back, based on minimum sad value criterion, the motion vector that obtains bulk is as the alternative motion vector of current block; And then get its left side, go up, the motion vector of upper left these 3 pieces is as alternative motion vector, calculates the sad value of these 4 the corresponding match block of alternative motion vector respectively, selects the best initial motion vectors of the motion vector of minimum sad value correspondence as current block.

4. a kind of combining target according to claim 3 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that, the method of calculating the backoff weight coefficient μ of match block in the front and back reference frame of each piece correspondence in the frame to be inserted in the 8th step is: according to the final motion vector that obtains in the 7th step, generate initial interpolation frame match block is got averaging method

Its pixel constantly is updated to correct interpolating pixel in compensation process simultaneously, at certain to be compensated, calculates itself and the absolute value error that extends out 4 pixel regions around the corresponding match block of front and back reference frame, is designated as MAD with last reference frame corresponding region absolute value error _f, be designated as MAD with back one reference frame corresponding region absolute value error _b,

Backoff weight coefficient μ is:

5. a kind of combining target according to claim 4 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that the division methods of high relevant range and low relevant range is in the 9th step: suppose that the former frame of to be divided correspondence and back one frame match block are respectively B _N-1(s), B _N+1(s), wherein s represents location of pixels in the piece, then makes the respective pixel of two pieces be divided by, and obtains the matched pixel merchant:

dif_blk(s)＝B _n-1(s)/B _n+1(s)

Set reference value Th=1, ε is a minimum as thresholding, each location of pixels s in the corresponding blocks, if | dif_blk (s)-Th|＜ε, then think in the match block respective pixel position for high relevant, otherwise for low relevant.

6. a kind of combining target according to claim 5 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that the compensation method of target area interpolation described in the tenth step:

To be compensated of target area, according to the match block similarity degree difference of to be compensated correspondence it is divided into high and low relevant range, matched pixel average compensation before and after adopting for high relevant range, wherein, v is a block motion vector to be compensated,

Be pixel value to be compensated, f _N-1(s-v) be corresponding blocks pixel value in the last reference frame with to be compensated coupling, f _N+1(s+v) be corresponding blocks pixel value in the back reference frame with to be compensated coupling,

The two-way OBMC compensation based on weight, wherein w are adopted in low relevant range _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling, penalty coefficient ρ gets backoff weight coefficient μ

7. a kind of combining target according to claim 6 is cut apart the frame frequency lifting method with irregular block compensation, it is characterized in that the compensation method of background area interpolation described in the tenth step:

To the same method of dividing high and low relevant range that adopts of the interpolation compensation of background area, can directly get average as interpolation frame correspondence position pixel to high relevant range to matched pixel in the reference frame of front and back, wherein, v is a block motion vector to be compensated,

Be pixel value to be compensated, f _N-1(s-v) be corresponding blocks pixel value in the last reference frame with to be compensated coupling, f _N+1(s+v) be corresponding blocks pixel value in the back reference frame with to be compensated coupling,

For low relevant range, adopt unidirectional compensation based on OBMC:

Wherein, the value of penalty coefficient ρ is by backoff weight coefficient μ decision, if backoff weight coefficient μ＞0.5, and penalty coefficient ρ=1 then, otherwise ρ=0;

w _i(s) be bilinearity window coefficient, v _iBe the motion vector of to be compensated neighborhood i piece, f _N-1(s-v _i) be corresponding blocks pixel value in the last reference frame with to be compensated neighborhood i piece coupling, f _N+1(s+v _i) be corresponding blocks pixel value in the back reference frame with to be compensated neighborhood i piece coupling.

Images