CN104202603B - Motion vector field generation method applied to video frame rate up-conversion - Google Patents
Motion vector field generation method applied to video frame rate up-conversion Download PDFInfo
- Publication number
- CN104202603B CN104202603B CN201410489709.2A CN201410489709A CN104202603B CN 104202603 B CN104202603 B CN 104202603B CN 201410489709 A CN201410489709 A CN 201410489709A CN 104202603 B CN104202603 B CN 104202603B
- Authority
- CN
- China
- Prior art keywords
- block
- motion vector
- particle
- frame
- motion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Television Systems (AREA)
Abstract
The invention relates to the field of video image processing, in particular to a motion vector field generation method applied to video frame rate up-conversion. The motion vector field generation method includes: firstly, determining the size of blocks in a self-adaptive mode under the framework of particle filtering, adopting the larger blocks in the area presenting single motion so as to beneficially reduce influence of image noise on the matching searching process, adopting the smaller blocks in the area presenting complex motion so as to capture those complex motion, and therefore, beneficially, correctly and reliably structuring motion vector fields used for motion compensation interpolation; secondly, for areas with uniform gray level and lack of texture information, of the video image, adopting a two-pass scanning mode to search best match in a smaller range by taking motion vectors of the neighborhood blocks as candidate vectors after other parts of an image are estimated to acquire the motion vectors; thirdly, for the video image with exposing areas, adopting two consecutive frames of I1 and I2 which are located after interpolation frames, determining the motion vectors of the corresponding positions of the interpolation frames by the motion vectors with the I1 pointing to the I2 so as to effectively prevent error vectors from occurring in the exposing areas.
Description
Technical field
The present invention relates to field of video image processing, more particularly to a kind of motion for being applied to be changed on video frame rate to
Amount field generation method.
Background technology
The frame rate that new frame is periodically inserted in the video of relatively low frame per second to improve video is changed in frame rate.
In the applications such as the playback of low frame-rate video coding normal frame rate, video format conversion, the viewing experience of beholder can be effectively lifted
Degree.
In transfer algorithm in existing frame rate, the algorithm based on motion compensation is repeated or front and rear frame than those simple-frames
Average algorithm has better performance, and reason is that video image often has the dynamic area introduced by moving object, base
Each pixel of interpolation on movement locus is attempted in the algorithm of motion compensation, by estimation and the tight knot of Neighborhood Filtering
Close, it is possible to avoid due to the frame-to-frame jump being simply repeatedly introduced, and the motion blur averagely introduced due to frame.
For the algorithm based on motion compensation, following two key elements will determine the quality of interpolated frame：(1) how each is estimated
The real motion of pixel is reducing their movement locus；(2) how more or less there are some insecure motion vectors
In the case of, build the interpolated frame that visually object is presented continuous and smooth motion.For first key element, difficulty comes from
Estimation is in itself an ill-conditioning problem, and existing motion estimation techniques, most of to be directed to video encoding design.
They are target to minimize compensation redundancy, reduce encoder bit rate, and the motion vector of estimation does not reflect the true of object sometimes
Motion.
It is pointed out that for block matching algorithm, the size of block will influence the reliability and precision of motion vector, when
One region is presented consistent motion, is conducive to reducing shadow of the picture noise for matching search procedure using the block of large-size
Ring, on the contrary, the region such as edge in moving object, using the less piece of motion for being conducive to catching those complexity；Positioned at ash
The block of homogeneous area is spent, Block- matching search procedure is easily influenceed by picture noise, for such region, in other regions
Determine after motion vector, the motion vector using contiguous block is more beneficial for obtaining reflecting that the region is true as reference vector
The motion vector of motion, and be also beneficial to reduce calculation cost；In addition, for exposed area, using close to before interpolated frame
It is insecure, it is necessary to company after being in interpolated frame that the motion vector that one frame and a later frame are obtained as block-matching search is doomed
Continuous two frames carry out the anti-motion vector for pushing away interpolated frame correspondence position.
The content of the invention
The technical problems to be solved by the invention are to provide a kind of motion vector for being applied to and being changed on video frame rate
Field generation method, the method is adaptively used according to video image time-domain motion feature in the region that single movement is presented
Larger block, present compound movement region carry out block-based motion estimation using less piece, so as to be effectively prevented from by
In the wrong motion vector introduced comprising the pixel from different motion object an inside for block；According to spatial domain line
Reason information, with inhomogeneous intensity, lack the region of grain details using larger block size, and with adjacent thereto and have
The motion vector of the neighborhood block of obvious texture information estimates the motion vector of these blocks as candidate vector, insecure to avoid
Motion vector.
To achieve the above object, the present invention provides a kind of motion vector field generation side for being applied to and being changed on video frame rate
Method, comprises the following steps：
Step one, read access time domain is close to interpolation frame, and a frame before it, is denoted as I0, close to interpolation frame and
Two frames after which, are denoted as I1And I2, with I0It is present frame, I1It is reference frame, is estimated in the way of multiresolution, change block size
Meter motion vector field MF_0, each vector correspondence I therein0In a block, be given in vector form the block with reference to figure
The relative skew between best matching blocks as in；
Step 2, judges whether each motion vector in motion vector field MF_0 is reliable；
Step 3, motion vector projection treatment, according to interpolated frame IΔtAnd I0Between time interval, by vector field MF_0
Each reliable motion vector projection assign the pixel of interpolated frame, be as a result one and IΔtThe vector field MF_X of formed objects；
Step 4, for the region that motion vector is not endowed in vector field MF_X, seeks one by interpolated frame IΔtGo out
Hair, through I1And I2, and I can be obtained1And I2Between smallest match cost vector as motion vector.
Further, the step one, the estimating motion vector in the way of multiresolution, variable-size-block are matched, will be adaptive
Ground is answered to divide image into the process model building of various sizes of sub-block into a dynamical system, with state vector s=[s1, s2...,
sN]TDescription, wherein N is total block number, si=(xi, yi, wi) i-th particle is represented, (x corresponding with an image blocki, yi)
It is the upper left angular coordinate of block, wiIt is width and short transverse block size in pixels, under the framework of particle filter, by weight
The property wanted density function q drives, and realizes that motion vector field is estimated in the form of iteration, comprises the following steps：
Step 1, builds L tomographic image pyramids, wherein the 0th layer of correspondence original image, L-1 layers of correspondence minimum resolution figure
Picture, if the size of kth tomographic image is H × W, the size of the tomographic image of kth+1 isAnd each pixel therein is equal to the
4 averages of pixel of k tomographic images；
Step 2, is input with the 0th tomographic image, calculates image gradient intensity, and gradient intensity is detected in the way of window is scanned
Figure, if the number of pixels with larger Grad is less than a preassigned threshold value in window, judges that window is located at gray scale
Homogeneous area, such region does not deal with follow-up step 3 to 7；
Step 3, judges whether corresponding piece of particle is divided by the block of last layer in the form of quaternary tree, or currently
Treatment is L-1 tomographic images, is matching criterior to minimize absolute frame difference sum if so, then using fast block matching algorithm,
A best match is searched in a reference image, with the relative skew (dx of the twoi, dyi) as motion vector；Otherwise, according to upper
Search for the motion vector for obtaining for one layer and determine initial searching position, best match is searched in a less scope；
Step 4, is measured, specifically, first by the distribution of motion vectors uniformity calculating observation of matching error and neighborhood block
Each particle is calculated as follows：
Wherein bkThe block size of current layer, SAD be obtain in step 3 matching process, correspondence best match it is absolute
Frame difference sum, eV∈ [0,1] is the parameter that a reflection current block moves consistent degree with its neighborhood block, if current block is adjacent with it
Domain block has consistent motion, then the parameter takes smaller value, otherwise takes higher value；
Secondly, observation measurement o is calculated as follows to each particleK, i
Step 5, is measured by observation, is calculated as follows the observation likelihood density function p of kth time iterationk(z|s)
Wherein, N () represents gauss of distribution function, uK, iBlock center when representing kth time iteration corresponding to i-th particle
Position, ∑K, iIt is covariance matrix, the size with block is relevant, z has corresponded to the observation measurement relevant with division；
Step 6, updates the importance density function as the following formula
Wherein ak∈ (0,1] be kth time iteration rewriting coefficient；
Step 7, according to the importance density function q, sampling produces the particle collection for carrying out next iteration calculating.Kth time is repeatedly
In generation, is directed to L-k tomographic images, one of bk×bkBlock has corresponded to L-k-1 layers of a 2bk×2bkBlock, if certain grain
Son has larger q values, then L-k-1 layers in next iteration is divided into four sons by its corresponding piece in the form of quaternary tree
Block, the particle that each sub-block correspondence+1 circulating particle of kth is concentrated, the otherwise corresponding sub-block of the particle is not divided, under
Size is taken in an iteration for 2bk×2bk, using step 3 methods described, with the motion vector obtained by last layer estimation as reference
Vector, best match is found in a less scope；
Step 8, if having been carried out L circulation, order down performs step 9, otherwise goes to step 3 and continue cycling through；
Step 9, is positioned at I0The block estimating motion vector in middle uniform gray level region, if in step 2, corresponding to particle
Block is detected as being located at uniform gray level region, then follow-up step 3 in 7 it will be ignored, after above-mentioned cyclic process terminates,
To these block estimating motion vectors by the way of two-pass scan, first pass is processed from top to bottom, left to right, right
In the block of need treatment, it is adjacent thereto and be positioned above, upper left side, the block on upper right side and the left side estimated are transported
Moving vector, using the motion vector of these blocks as candidate vector, selects that to produce minimum absolute in set of candidate vectors
The vectorial result as first pass of frame difference sum；Second time scanning is right by sequential processes from top to bottom, from right to left
In a pending block, it is adjacent thereto and be disposed below, lower left, lower right, the block on the right estimated are moved
Vector, using these vectors plus the motion vector itself estimated in first pass as candidate vector, in candidate vector
That can produce the vector of minimum absolute frame difference sum as final result to concentrate selection.
Further, the step 4, for the region that motion vector is not endowed in vector field MF_X, seek one by
Interpolated frame IΔtSet out, through I1And I2, and I can be obtained1And I2Between smallest match cost vector as motion vector, including：
Block- matching search is limited in certain scope, vector (u possible to each in hunting zonex, uy) calculate
Matching error during different i and j：
Wherein i and j is less integer, to all of i and j, is calculated as follows
Finally, the motion vector for determining the block is
The method have the benefit that：Under the framework of particle filter, the size of block is determined in an adaptive way,
Larger block is used in the region that single movement is presented, less piece is used in the region that compound movement is presented, be conducive to correct
And reliably build the motion vector field for motion compensated interpolation；For there is exposed area in video image, this part area
Can not possibly be present the motion vector of a later frame that interpolated frame is pointed to by the frame before interpolated frame in domain, tightened using time shaft
Adjacent interpolated frame, and positioned at two continuous frames I thereafter1And I2, by I1Point to I2Motion vector determine interpolated frame relevant position
Motion vector, with the wrong interpolation for being effectively prevented from occurring in exposed area.
Brief description of the drawings
Fig. 1 is the specific embodiment block diagram that present invention generation is applied to the motion vector field of conversion on video frame rate；
Fig. 2 multiresolutions become the flow chart of block size estimating motion vector；
Fig. 3 two-step-rooting method algorithm search position views；
The 4- neighborhood schematic diagrames of Fig. 4 image blocks；
Fig. 5 processes the schematic diagram of the block positioned at uniform gray level region in the way of two-pass scan；
Fig. 6 different zones have been assigned the schematic diagram of different number of motion vector；
Fig. 7 is determined the motion vector schematic diagram of hole region in interpolated frame by the Block- matching of follow-up two frame.
Specific embodiment
The specific embodiment of method provided by the present invention is illustrated below in conjunction with accompanying drawing.If continuous the three of input
Frame is I0、I1And I2, that to be estimated is interpolated frame IΔtThe motion vector of each location of pixels in (0 ＜ Δ t ＜ 1), Fig. 1 shows
The block diagram of the specific embodiment of the invention, including following steps：
Step 101, by the way of multiresolution, change block size, with I0It is present frame, I1For reference frame estimate motion to
Amount field MF_0, each vector correspondence I therein0In a block, be given in vector form in the block and reference picture most
Relative skew between good match block；
Step 102, judges whether each motion vector in MF_0 is reliable；
Step 103, motion vector projection treatment, according to interpolated frame IΔtAnd I0Between time interval, by vector field MF_0
In each reliable motion vector the pixel of interpolated frame is assigned in the way of projecting, be as a result one and IΔtFormed objects to
Amount field MF_X；
Step 104, for the region that motion vector is not endowed in vector field MF_X, seeks one by interpolated frame IΔt
Set out, through I1And I2, and I can be obtained1And I2Between smallest match cost vector as motion vector.
The step 101, using multiresolution, variable-size-block matching algorithm estimating motion vector MF_0.Based on motion
Changed in the frame rate of compensation, whether its algorithm performance is largely dependent upon the motion vector obtained by estimating can be anti-well
Reflect the real motion of each pixel.For block matching algorithm, the size of block will influence the reliability and precision of motion vector, when
One region is presented consistent motion, is conducive to reducing shadow of the picture noise for matching search procedure using the block of large-size
Ring, on the contrary, the region such as edge in moving object, using the less piece of motion for being conducive to catching those complexity.Therefore, root
The motion feature showed according to image, in an adaptive way determine block size, present single movement region use compared with
Big block, less piece is used in the region that compound movement is presented, and is conducive to correctly and reliably building and is inserted for motion compensation
The motion vector field of value.
The present invention will adaptively divide image into various sizes of piece of process model building into a dynamical system, and with shape
State vector s=[s1, s2..., sN]TDescription, wherein N is total block number, si=(xi, yi, wi) i-th particle is represented, with one
Image block is corresponding, contains the information for describing the tile location and size, (xi, yi) be sub-block upper left angular coordinate, wi
It is width and short transverse block size in pixels.Under the framework of particle filter, in importance density in the form of iteration
Under the driving of function q, larger block is used in background area, Moving Objects edge will be located at or the region of compound movement is presented
Less sub-block is divided into, so as to the motion vector field estimation realized multiresolution, become block size, Fig. 2 shows specific steps.
Step 200, builds L tomographic image pyramids, wherein the 0th layer of correspondence original image, L-1 layers of correspondence minimum resolution
Image, if the size of kth tomographic image is H × W, the size of the tomographic image of kth+1 isAnd each pixel therein is equal to
4 averages of pixel of kth tomographic image.Image of a preferred embodiment of the present invention to resolution ratio less than 1920 × 1280 takes L
=3, the image more than or equal to the resolution ratio takes L=4.
The L-1 tomographic images of minimum resolution are evenly divided into b0×b0Sub-block, each block correspondence one particle,
One embodiment of the present of invention takes b0=8.By initial the importance density function q0It is defined as being uniformly distributed, by all particle structures
Into particle assemblyWhereinN0The block number total equal to L-1 tomographic images.
Step 201, is input detection uniform gray level region with the 0th tomographic image, and one embodiment of the present of invention uses gradient
Operator [- 1,0 ,+1] and [- 1,0 ,+1]TMake convolution algorithm with image respectively, calculate the gradient image I of X and Y both directionsxWith
Iy, it is calculated as follows gradient intensity
Step 203, with fixed size, the window not overlapped each other scanning gradient intensity figure, if window inside gradient
Intensity level is more than threshold value TeNumber of pixels be less than a given number Tn, then corresponding piece is judged in inhomogeneous intensity region,
Do not dealt with follow-up step 204 to 207.One embodiment of the present of invention takes window size for 32 × 32, Te=6, Tn=
4。
The thought of above-mentioned steps behind is：If block is located at uniform gray level region, Block- matching search procedure is easily schemed
As the influence of noise, often there are some positions in search procedure, its matching error and minimum match error closely, and
The position of that generation minimum match error may not reflect real motion.For such region, determine in other regions
After motion vector, the motion vector using contiguous block is more beneficial for obtaining reflecting the region real motion as candidate vector
Motion vector, and be also beneficial to reduce calculation cost.
Step 204, judges whether corresponding piece of particle is divided by the block of last layer in the form of quaternary tree, or works as
Pre-treatment is L-1 tomographic images.
Step 205, for the image block being judged to by step 204 corresponding to qualified particle, using Rapid matching
Algorithm, is matching criterior to minimize absolute frame difference sum, a best match is searched in a reference image, with the relative of the two
Skew (dxi, dyi) as motion vector.Specifically, one embodiment of the present of invention uses diamond search algorithm, is calculated as follows
Absolute frame difference sum SAD as matching error
Wherein (x0, y0) it is particle siThe top left co-ordinate of correspondence sub-block, bkIt is L-k layers of block size.
Step 206, if the condition judgment of step 204 is invalid, determines according to the motion vector that last layer search is obtained
Initial searching position, specifically, if the motion vector of last layer is (dx, dy), (x0, y0) be block top left co-ordinate, then with
(x0+ 2dx, y0+ 2dy) as initial searching position, use two-step-rooting method algorithm search best match.If (1,1) is last layer
Motion vector, Fig. 3 shows the searching position of the first step and second step, and wherein the first step searches for 9 positions of filled circle marker
Put, if the point of right positions is the corresponding point of minimum match error, second step searches for 8 points of open circle markers.
Step 207, observation measurement is determined by the distribution of motion vectors uniformity of matching error and neighborhood block, specifically, first
First each particle is calculated as follows：
Wherein, eV∈ [0,1] is the parameter that reflection current block and its neighborhood block move consistent degree, if current block with
Its adjacent block has consistent motion, then the parameter takes smaller value, otherwise takes higher value.Reference picture 4, if current block is BM, n, with
Its four adjacent neighborhood block is respectively BM-1, n、BM+1, n、BM, n-1And BM, n+1, one embodiment of the present of invention determines parameter as the following formula
eV
Wherein udxIt is four neighborhood block BM-1, n、BM+1, n、BM, n-1And BM, n+1Motion vector X-component average, udyIt is Y points
The average of amount, min (a, b) and max (a, b) represent the smaller value and higher value for taking a and b two numbers respectively.
Secondly, observation measurement o is calculated as follows to each particleK, i
Step 208, judges whether all particles that processed current particle is concentrated, if so, step 209 is performed, otherwise
Go to step the 203 next particles for continuing with particle concentration.
Step 209, is measured by observation, is calculated as follows the observation likelihood density function p of kth time iterationk(z|s)
Wherein, N () represents gauss of distribution function, uK, iIn block when representing kth time iteration corresponding to i-th particle
Heart position, ∑K, iIt is covariance matrix, the size with block is relevant, and one embodiment of the present of invention takes
Step 210, updates the importance density function as the following formula
Wherein ak∈ (0,1] be kth time iteration rewriting coefficient, one embodiment of the present of invention is taken as constant：ak=
0.75。
Step 211, according to the importance density function q, sampling produces the particle collection for carrying out next iteration calculating.Kth time
Iteration is directed to L-k tomographic images, one of bk×bkBlock has corresponded to L-k-1 layers of a 2bk×2bkBlock, if certain
Particle has larger q values, then L-k-1 layers in next iteration is divided into four by its corresponding piece in the form of quaternary tree
Sub-block, the particle that each sub-block correspondence+1 particle of circulation of kth is concentrated, the otherwise corresponding sub-block of the particle is not drawn
Point, next iteration takes size for 2bk×2bk, its motion vector is estimated using the two-step-rooting method method described in step 206.
Step 212, judges whether processed 0th tomographic image, if so, going to step 213, otherwise goes to step under 202 continuation
An iteration.
Step 213, is positioned at I0The block estimating motion vector in middle uniform gray level region.If in step 201, particle institute is right
The block answered is detected as being located at uniform gray level region, then it will be ignored in follow-up step, reference picture 5, and such block is with white
Color square frame is marked.The present invention to these block estimating motion vectors using two-pass scan by the way of, press from top to bottom, from a left side by first pass
To right sequential processes, if the block that need to be processed is A, then it is adjacent thereto be positioned above, upper left side, upper right side and the left side block
It is estimated to have obtained motion vector, using the motion vector of these blocks as candidate vector, that is selected in set of candidate vectors
The vector of minimum SAD can be produced as the result of first pass.Second time scanning is pressed at order from top to bottom, from right to left
Reason, if pending block is B, then it is adjacent thereto be disposed below, lower left, lower right, the estimated of the block on the right obtain
Motion vector, itself also estimates to obtain a motion vector in first pass, using these vectors as candidate vector,
That is selected in set of candidate vectors can produce the vector of minimum SAD as second time result of scanning.
After above-mentioned steps 200 to 213 terminate, output is motion vector field MF_0, each vector correspondence I therein0In
A block.
The step 102, judges whether the motion vector in MF_0 is reliable, due to using foregoing multiresolution, becoming size
Block- matching searching algorithm estimating motion vector, and with two-pass scan and with the motion vector of its neighborhood block be candidate vector to place
Block in uniform gray level region makees estimation, thus in MF_0 insecure motion vector more come from occlusion area and
Exposed area.A motion vector in MF_0 is noticed, it is one 64 × 64 pieces of maximum possible the 0th tomographic image of correspondence, minimum
The block of one 8 × 8 may be corresponded to, for treatment is convenient, corresponding one 8 × 8 block of each vector is uniformly processed into MF_0, specifically
Ground a, if vector is corresponding one 2r×2rBlock, wherein r is integer, and 4≤r≤6, then by it both horizontally and vertically each
It is divided into 2r-3Individual sub-block, and assign all of sub-block by the vector.One embodiment of the present of invention uses two following steps
Judge the reliability of motion vector：
Step one, set BM, nThe motion vector of block is (vx, vy), its four neighborhoods block is respectively BM-1, n、BM, n-1、BM, n+1With
BM+1, n, it is calculated as follows BM, nWith the maximum of its four neighborhood motion vectors absolute difference
Δvmax=max | vx(m, n)-vx(m+i, n+j) |+| vy(m, n)-vy(m+i, n+j) | } (9)
Wherein i, j=0, ± 1, and | i | ≠ | j |.If Δ vmax≤ε1, then B is judgedM, nBlock is located at motion uniform domain, nothing
The judgement of subsequent step two need to be carried out, is not otherwise to go to step two.Wherein, ε1It is a less threshold value, one of the invention
Embodiment takes ε1=2.
Step 2, if present frame I0BM, nBlock is with it in reference frame I1In best matching blocks between relative skew be
(vx, vy).In reference frame with (m+vx, n+vy) it is top left co-ordinate, take size and BM, nIdentical image block, finds the block in I0
Best matching blocks in frame, note motion vector is (ux, uy).If following formula is set up, (vx, vy) it is a reliable motion vector,
Otherwise the motion vector is unreliable：
|vx+ux|+|vy+uy|≤ε2 (10)
Wherein ε2It is a threshold value determined by experiment, a preferred embodiment of the invention takes ε2=2.
The step 103, motion vector projection treatment, according to interpolated frame IΔtAnd I0Between time interval, by vector field
Each reliable motion vector in MF_0 assigns the pixel of interpolated frame, is as a result one and IΔtThe vector field MF_ of formed objects
X.Specifically, if (x in vector field MF_00, y0) position vector be M (x0, y0)=(vx, vy), and and I0Frame top left co-ordinate
It is (x0N, y0N N × N block correspondence), then top left co-ordinate is (x in MF_X0N+round(vxΔ t), y0N+round(vy
Δ t)), size is endowed motion vector (v for all positions of N × N blocksxΔ t, vyΔ t), wherein round () are represented and are rounded fortune
Calculate.
Fig. 6 shows that, by above-mentioned motion vector projection, MF_X diverse locations have showing for different number of motion vector
It is intended to：One position of region A is endowed unique motion vector, and region B forms the hole region of interpolated frame, unit therein
Element is not endowed motion vector, and region C then forms the overlapping region of interpolated frame, each position correspondence two therein even two
Motion vector more than individual.
If the pixel (x, y) in interpolated frame has been assigned unique motion vector (dx, dy), then interpolated frame is being subsequently generated
During directly from I0Frame copies pixel value, specifically,
IΔt(x, y)=I0(x-dx, y-dy) (11)
If the pixel (x, y) of interpolated frame has been assigned n motion vectorThen be subsequently generated it is slotting
The pixel value of the form calculus interpolated frame of following weighted sum is used during value frame
Wherein wkIt is and motion vector MkCorresponding weights, specifically, are calculated as follows matching error first
Secondly, weights are calculated as follows
Wherein e is a less number, and one embodiment of the present of invention takes e=0.5, and E is then calculated as follows
Due to there is insecure motion vector in MF_0, and there is the difference of direction and modulus value in adjacent motion vector
It is different, can there are some regions for not being endowed motion vector in interpolated frame.In order to detect these regions, a reality of the invention
Apply example and one is used in motion vector projection with interpolated frame IΔtThe Matrix C of formed objects records the fortune that each pixel is endowed
The number of moving vector, i.e.,
C (x, y)=a (16)
Wherein a=0,1,2... is the number through the motion vector of the pixel.After projection process terminates, one 3 is used
× 3 window scanning C, if window center position is located at (x0, y0) when following formula set up, then go to step 104, otherwise continue to scan on down
One position.
The step 104, reference picture 7, with the center (x of scanning window0, y0) centered on, an implementation of the invention
Example takes block size for N=8, and hunting zone is -17~+17, vector (u possible to each in hunting zonex, uy) meter
Calculate matching error during different i and j：
Wherein -2≤i, j≤2, and be integer.To all of i and j, it is calculated as follows
Finally, the motion vector of the block where determining scanning window is：
And according to the vector by I1The pixel value of frame determines the value of interpolated frame correspondence position.
Presently preferred embodiments of the present invention is the foregoing is only, but protection scope of the present invention is not limited thereto, it is all at this
Within the spirit and principle of invention, any modification or replacement for being made etc. should all be covered within the scope of the present invention.
Claims (5)
1. it is a kind of be applied on video frame rate change motion vector field generation method, it is characterised in that：With multiresolution, become
The mode estimating motion vector of size Block- matching, will adaptively divide image into various sizes of piece of process model building into one
Individual dynamical system, with state vector s=[s1, s2..., sN]TThe system is described, wherein N is total block data, si=(xi, yi, wi) generation
Table i-th particle corresponding with an image block, (xi, yi) be block top left co-ordinate, wiIt is width and short transverse with picture
The block size of element meter, is driven under the framework of particle filter by the importance density function, and motion vector is realized in the form of iteration
Estimate, comprise the following steps in field：
(1) L tomographic image pyramids are built, wherein the 0th layer of correspondence original image, the minimum resolution in L-1 layers of correspondence pyramid
Rate image, if the size of kth tomographic image is H × W, the size of the tomographic image of kth+1 is
(2) it is input with the 0th tomographic image, calculates image gradient intensity, gradient intensity figure is detected in the way of window is scanned, if window
It is intraoral more than preassigned threshold value TeNumber of pixels be less than a preassigned threshold value Tn, then judge that window is located at gray scale
Homogeneous area, such region does not deal with follow-up step (3) to (7)；
(3) to each particle, judge whether corresponding piece of particle is divided by the block of last layer in the form of quaternary tree, or
Currently processed is L-1 tomographic images, is matching to minimize absolute frame difference sum if so, then using fast block matching algorithm
Criterion, searches for a best match, with the relative skew (dx, dy) of the two as motion vector in a reference image；Otherwise, root
The motion vector obtained according to last layer search determines initial searching position, is respectively in the range of -3~+3 in the position both direction
Search best match；
(4) observation measurement is determined by the motion vector uniformity of matching error and neighborhood block, first, each particle is counted as the following formula
Calculate：
Wherein bkThe block size of current layer, SAD be obtain in step (3) matching process, the absolute frame of correspondence best match it is poor
Sum, eV∈ [0,1] is a reflection current block BM, nParameter with its neighborhood block moves consistent degree, is calculated as follows：
Wherein udxIt is four neighborhood block BM-1, n、BM+1, n、BM, n-1And BM, n+1Motion vector X-component average, udyIt is Y-component
Average, min (a, b) and max (a, b) represent the smaller value and higher value for taking a and b two numbers respectively；Secondly, each particle is pressed
Formula calculating observation measures oK, i
(5) measured by observation, be calculated as follows the observation likelihood density function p of kth time iterationk(z|s)
Wherein, N () represents gauss of distribution function, uK, iThe centre bit of block when representing kth time iteration corresponding to i-th particle
Put, ∑K, iIt is covariance matrix, the size with block is relevant, is calculated as follows
(6) the importance density function is updated as the following formula
Wherein ak∈ (0,1] be kth time iteration rewriting coefficient；
(7) according to the importance density function q, sampling produces the particle collection for carrying out next iteration calculating, re-executes step
(3)-(7), until having performed the L times circulation.
2. the method for claim 1, it is characterised in that described motion vector field is with two frames of continuous input
First frame I0It is present frame, the second frame I1It is reference frame calculating, I0Before interpolated frame, I1After interpolated frame.
3. the method for claim 1, it is characterised in that further include：
(8) according to interpolated frame IΔtAnd I0Between time interval Δ t, make motion vector projection treatment, if to the 0th tomographic image estimate
The motion vector field of gained is MF_0, and its size isWherein Width, Height and b represent the 0th layer respectively
Wide, the high and block size of image, each vector of MF_0 has corresponded to I0One image block of frame, wherein position are (x, y)
The top left co-ordinate of vectorial corresponding image block be (xb, yb), if MF_0 (x, y)=(dx (x, y), dy (x, y)), through move to
After amount projection, interpolated frame IΔtTop left co-ordinate is (xb+round (Δ tdx (x, y)), yb+round (Δ tdy (x, y))), greatly
Small all positions for b × b blocks are endowed motion vector (Δ tdx (x, y), Δ tdy (x, y)), and wherein round () is represented and taken
Whole computing；
(9) scan process is made to the motion vector field obtained by projection, if contained in scanning window not being endowed motion vector
Pixel, then seek one by interpolated frame IΔtSet out, through I1Thereafter a frame I2, and I can be obtained1And I2Between minimum
Vector with cost is used as motion vector.
4. the method for claim 1, it is characterised in that the step (7), kth time iteration is directed to L-k layer and schemes
Picture, one of bk×bkBlock has corresponded to L-k-1 layers of a 2bk×2bkBlock, if certain particle has larger q values,
L-k-1 layers in next iteration is divided into four sub-blocks, the size of each sub-block by its corresponding piece in the form of quaternary tree
It is bk×bk, and the particle that+1 circulating particle of kth is concentrated is corresponded to respectively, the otherwise corresponding sub-block of the particle is not divided,
Corresponding piece of size is 2bk×2bk, a particle is only corresponded in+1 particle collection of circulation of kth.
5. the method for claim 1, it is characterised in that the method for estimation of the step (9) is as follows：Block- matching is searched for
It is limited in certain scope, vector (u possible to each in hunting zonex, uy) calculate different i and j when matching
Error
Wherein (x0, y0) it is the center of step (9) described scanning window, i and j is integer, and -2≤i, j≤2, to all
I and j, be calculated as follows
Finally, the motion vector for determining the block is
。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410489709.2A CN104202603B (en) | 2014-09-23 | 2014-09-23 | Motion vector field generation method applied to video frame rate up-conversion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410489709.2A CN104202603B (en) | 2014-09-23 | 2014-09-23 | Motion vector field generation method applied to video frame rate up-conversion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104202603A CN104202603A (en) | 2014-12-10 |
| CN104202603B true CN104202603B (en) | 2017-05-24 |
Family
ID=52087821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410489709.2A Active CN104202603B (en) | 2014-09-23 | 2014-09-23 | Motion vector field generation method applied to video frame rate up-conversion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104202603B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104796721B (en) * | 2015-04-24 | 2018-03-16 | 宏祐图像科技(上海)有限公司 | The method for carrying out estimation to image light scene change using MEMC technologies |
| CN105376584B (en) * | 2015-11-20 | 2018-02-16 | 信阳师范学院 | Turn evidence collecting method in video motion compensation frame per second based on noise level estimation |
| CN106131567B (en) * | 2016-07-04 | 2019-01-08 | 西安电子科技大学 | Ultraviolet aurora up-conversion method of video frame rate based on Lattice Boltzmann |
| CN108810317B (en) * | 2017-05-05 | 2021-03-09 | 展讯通信（上海）有限公司 | True motion estimation method and device, computer readable storage medium and terminal |
| CN110662037B (en) * | 2018-06-29 | 2022-06-28 | 北京字节跳动网络技术有限公司 | Limitation of motion information sharing |
| CN111174782B (en) * | 2019-12-31 | 2021-09-17 | 智车优行科技（上海）有限公司 | Pose estimation method and device, electronic equipment and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102523419A (en) * | 2011-12-31 | 2012-06-27 | 上海大学 | Digital video signal conversion method based on motion compensation |
| CN102685438A (en) * | 2012-05-08 | 2012-09-19 | 清华大学 | Up-conversion method of video frame rate based on time-domain evolution |
| CN103220488A (en) * | 2013-04-18 | 2013-07-24 | 北京大学 | Up-conversion device and method of video frame rate |
| CN103260024A (en) * | 2011-12-22 | 2013-08-21 | 英特尔公司 | Complexity scalable frame rate up-conversion |
| CN103702128A (en) * | 2013-12-24 | 2014-04-02 | 浙江工商大学 | Interpolation frame generating method applied to up-conversion of video frame rate |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4645736B2 (en) * | 2008-12-22 | 2011-03-09 | ソニー株式会社 | Image processing apparatus, image processing method, and program |
-
2014
- 2014-09-23 CN CN201410489709.2A patent/CN104202603B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103260024A (en) * | 2011-12-22 | 2013-08-21 | 英特尔公司 | Complexity scalable frame rate up-conversion |
| CN102523419A (en) * | 2011-12-31 | 2012-06-27 | 上海大学 | Digital video signal conversion method based on motion compensation |
| CN102685438A (en) * | 2012-05-08 | 2012-09-19 | 清华大学 | Up-conversion method of video frame rate based on time-domain evolution |
| CN103220488A (en) * | 2013-04-18 | 2013-07-24 | 北京大学 | Up-conversion device and method of video frame rate |
| CN103702128A (en) * | 2013-12-24 | 2014-04-02 | 浙江工商大学 | Interpolation frame generating method applied to up-conversion of video frame rate |
Non-Patent Citations (4)
| Title |
|---|
| Bo-Won Joen.Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure.《IEEE Transactions on Consumer Electronics》.2003,第49卷(第3期),第499-508页. * |
| Byeong-Doo Choi et al..Motion-Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Motion Compensation.《IEEE Transactions on Circuits & Systems for Video Technology》.2007,第17卷(第4期),第407-416页. * |
| R.Castagno et al..A method for motion adaptive frame rate up-conversion.《IEEE Transactions on Circuits and Systems for Video Technology》.2008,第6卷(第5期),第436-446页. * |
| Yen-Kuang Chen et al..Frame-rate up-conversion using transmitted true motion vectors.《Multimedia Signal Processing，1998 IEEE Second Workshop on》.2002,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104202603A (en) | 2014-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104202603B (en) | Motion vector field generation method applied to video frame rate up-conversion | |
| Elgharib et al. | Video magnification in presence of large motions | |
| KR101130060B1 (en) | Temporal motion vector filtering | |
| CN105100807B (en) | A kind of frame per second method for improving based on motion vector post-processing | |
| JP3679426B2 (en) | A system that encodes image data into multiple layers, each representing a coherent region of motion, and motion parameters associated with the layers. | |
| TWI359387B (en) | Robust camera pan vector estimation using iterativ | |
| US8711938B2 (en) | Methods and systems for motion estimation with nonlinear motion-field smoothing | |
| WO2012043841A1 (en) | Systems for producing a motion vector field | |
| KR20000069107A (en) | Motion estimation and motion-compensated interpolation | |
| KR20050065298A (en) | Motion compensated frame rate conversion | |
| KR19990077203A (en) | Image segmentation | |
| CN1488123A (en) | Segmentation unit for and method of determining a second segment and image processing apparatus | |
| CN107292819A (en) | A kind of infrared image super resolution ratio reconstruction method protected based on edge details | |
| CN108805832A (en) | Improvement Gray Projection digital image stabilization method suitable for tunnel environment characteristic | |
| Choi et al. | Affine transformation-based deep frame prediction | |
| CN108270945B (en) | Motion compensation denoising method and device | |
| CN114782628A (en) | Indoor real-time three-dimensional reconstruction method based on depth camera | |
| CN111598775B (en) | Light field video time domain super-resolution reconstruction method based on LSTM network | |
| CN113724155A (en) | Self-boosting learning method, device and equipment for self-supervision monocular depth estimation | |
| CN107295217B (en) | Video noise estimation method based on principal component analysis | |
| Wu et al. | Divide-and-conquer completion network for video inpainting | |
| Lee et al. | Fast 3D video stabilization using ROI-based warping | |
| Kanchana et al. | Revealing disocclusions in temporal view synthesis through infilling vector prediction | |
| CN107135393B (en) | Compression method of light field image | |
| CN1328699C (en) | Feature Point selection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210114 Address after: No.477 Yongle Road, Wangdian Town, Xiuzhou District, Jiaxing City, Zhejiang Province Patentee after: ZHEJIANG OUTUO ELECTRICAL Co.,Ltd. Address before: No.18 Xuezheng street, Xiasha Economic Development Zone, Hangzhou City, Zhejiang Province, 310018 Patentee before: ZHEJIANG GONGSHANG University |
|
| TR01 | Transfer of patent right |