GB2500909A - Selecting motion vectors on the basis of acceleration - Google Patents
Selecting motion vectors on the basis of acceleration Download PDFInfo
- Publication number
- GB2500909A GB2500909A GB1206067.9A GB201206067A GB2500909A GB 2500909 A GB2500909 A GB 2500909A GB 201206067 A GB201206067 A GB 201206067A GB 2500909 A GB2500909 A GB 2500909A
- Authority
- GB
- United Kingdom
- Prior art keywords
- vectors
- candidate
- backward
- acceleration
- motion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/521—Processing of motion vectors for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
In video processing, two candidate forward motion vectors and two candidate backward motion vectors are derived for each frame. Four candidate pairs of motion vectors are designated with each candidate pair comprising one of the candidate forward vectors and one of the candidate backward vectors. An acceleration is calculated for each candidate pair and one of the candidate pairs is selected based on that acceleration.
Description
SELECTING MOTION VECTORS
FIELD OF INVENTION
This invention concerns selecting motion vectors in motion estimation of video signals.
BACKGROUND OF THE INVENTION
This invention relates to the estimation of motion vectors between video frames.
In motion-compensated image processing motion vectors are assigned to pixels, or blocks of pixels, in each frame and describe the estimated displacement of each pixel or block in a next frame or a previous frame in a sequence of frames.
In the following description, the motion estimation is considered to be "dense" meaning that a motion vector is calculated for every pixel. The definition of "dense" may be widened to cover the calculation of a motion vector for each small block in the picture, for each pixel in a subsampled version of the picture or for each small region of arbitrary shape within which the motion is expected to be uniform. The invention can be applied with trivial modification to these wider cases.
Motion estimation has application in many image and video processing tasks, including video compression, motion-compensated temporal interpolation for standards conversion or slow-motion synthesis, motion-compensated noise reduction, object tracking, image segmentation and, in the form of displacement estimation, stereoscopic 3D analysis and view synthesis from multiple cameras.
Most applications of motion estimation involve the "projection" (also described as "shifting") of picture information forward or backward in time according to the motion vector that has been estimated. This is known as "motion-compensated" projection. The projection may be to the time instant of an existing frame or field, for example in compression, where a motion-compensated projection of a past or future frame to the current frame instant serves as a prediction of the current frame. Alternatively, the projection may be to a time instant not in the input sequence, for example in motion-compensated standards conversion, where information from a current frame is projected to an output time instant, where it will be used to build a motion-compensated interpolated output frame.
In most cases, it is desirable to estimate both "forward" vectors, which describe motion between the current frame and a future frame, and "backward" vectors, which describe motion between a current frame and a past frame. It is important to note that forward vectors estimated for the current frame are not necessarily the same as backward vectors estimated for the future frame. This is because a motion-compensated projection from current-frame pixels to future-frame pixels is not reversible in general. Not all future-frame pixels will be destinations for motion-compensated projection from the current frame. This mismatch typically occurs in areas known as "occlusions" whore relative motion of objects in the scene leads to parts of a picture being revealed or obscured from one frame to the next. Explicit measurement of forward and backward vectors is helpful for handling occlusions in motion-compensated processing.
An example of a known method of estimating forward and backward vectors for pixels of an image forming part of an image sequence will now be described.
Some terminology will first be defined with reference to Figure 1.
Figure 1 shows one-dimensional sections through three successive frames in a sequence. The horizontal axis of Figure 1 represents time, and the vertical axis represents position. Of course, the skilled person will recognise that Figure 1 is a simplification and that motion vectors used in image processing are generally two dimensional. The illustrated frames are: a previous frame (102); a current frame (101); and, a next frame (103). A forward motion vector (104) is shown assigned to a pixel (105) in the current frame. The motion vector (104) indicates a point (106) in the next frame which is the estimated destination of the current frame pixel (105) in the next frame. Similarly, a backward motion vector (107) assigned to the same current frame pixel (105) indicates a point (108) in the previous frame which is the estimated source of the current frame pixel in the previous frame.
The following descriptions assume that these three frames are consecutive in the sequence, but the described processes are equally applicable in cases where there are intervening frames, for example in some compression algorithms. It should also be noted that a convention has been chosen here whereby forward vectors point to frames in the future. In some compression literature, for example in the MPEG-2 video coding specification (ISO/IEC 13818-2), forward vectors point to frames in the past. However, in the descriptions that follow, there is complete symmetry between forward and backward vectors, so no ambiguity arises from the difference in naming conventions.
In the remainder of this specification temporal samples of an image will be referred to as fields, as would be the case when processing interlaced images.
However, as the skilled person will appreciate, in non-interlaced image formats a temporal sample is represented by a frame; and, fields may be de-interlaced' to form frames within an image process. The spatial sampling of the image is not relevant to the discussion which follows.
We begin the description of a known method of estimating forward and backward vectors by describing an algorithm that estimates backward vectors only. An example of such an algorithm is given in GB2188510 (Thomas). This algorithm is summarised in Figure 2 and assigns a single backward vector to every pixel of a current field in a sequence of fields. The process of Figure 2 is assumed to operate sequentially on the pixels of the current field; the pixel whose vector assignment is currently being determined will be referred to as the current pixel.
The current field (201) and the previous field (202) are applied to a phase correlation unit (203) which calculates a "menu" for every pixel of the current field consisting of a number (three in this example) of candidate motion vectors (204).
Each candidate vector controls a respective member of a set of shift units (205) which displaces the previous field (202) by the respective candidate vector to produce a shifted pixel corresponding to the current pixel of the current field in the respective member of a set of displaced fields (206).
A set of error calculation units (207) produces a set of error values (208), one error value for every menu vector for every pixel of the current field. Each of the error calculation units (207) subtracts the respective one of the displaced fields (206) from the current field (201) and rectifies the result to produce a field of difference magnitudes, which is spatially filtered in a filter centered on the current pixel to give an error value for that pixel and menu vector. The set of three error values (208) for each pixel are compared in a comparison unit (209), which finds the minimum value error. The comparison unit (209) outputs a candidate index (210). which identifies the vector that gave rise to the minimum error. The candidate index (210) for every pixel is then applied to a vector selection unit (211)to select the identified candidate from the menu of vectors (204) as the respective output assigned backward vector (212).
Such an algorithm may trivially be adapted to calculate and assign forward vectors, using the next field in place of the previous field. Thus, the double application of the algorithm of Thomas will estimate a forward and a backward vector for every pixel, as required.
It is the object of this invention to provide an improved method of assigning forward and backward vectors to pixels.
SUMMARY OF THE INVENTION
The invention consists of a method and apparatus for assigning forward and backward motion vectors to image pixels, characterised in that forward and backward vectors are assigned as a pair and the acceleration derived from the pair of vectors is taken into account in the assignment process.
The inventors have noted that methods of calculating forward and backward vectors such as that just described have a significant disadvantage, which arises because the calculation of forward vectors is independent of the calculation of backward vectors. A difference between the forward and backward vectors shows that the motion of a portrayed object between the time instants of the previous and current fields is different from the motion of the object between the time instants of the current and next frames. This change in motion corresponds to an acceleration of the object.
The selection of a vector from a list of candidates does not always result from a clear-cut decision. Two or more candidates may produce errors that are very close in value, so that the selection of the vector corresponding to the minimum error is to some extent arbitrary. If this ambiguity occurs in either or both of the forward and backward directions, then the resulting pair of forward and backward vectors may represent an acceleration in the motion of a portrayed object that is very unlikely, or even impossible, at that point in the video sequence. In that eventuality, then at least one of the pair of the assigned vectors will be wrong and, when used for motion-compensated interpolation for example, will produce undesirable artefacts in the output picture.
The inventors have recognised that the problem of incorrectly assigned vectors can be ameliorated by making use the acceleration of any particular feature of the image is likely to be small. In terms of Figure 1, this is equivalent to asserting that the angle between forward and backward vectors (104 and 107) is close to 180 degrees, or that the sum of the values of the two vectors is small.
BRIEF DESCRIPTION OF THE DRAWINGS
An example of the invention will now be described with reference to the drawings in which: Figure 1 is a diagram showing previous, current and next frames in an image sequence; and, forward and backward motion vectors extending from a pixel in the current frame; Figure 2 is a block diagram of apparatus for assigning backward motion
vectors to pixels according to the prior art;
Figure 3 is a block diagram of apparatus for assigning forward and backward motion vectors to pixels according to invention
DETAILED DESCRIPTION OF THE INVENTION
Forward and backward motion vector assignment apparatus according to an exemplary embodiment of the invention will now be described.
Referring to Figure 3, the current field (301)and the previous field (302) are applied to a phase correlation unit (303) which calculates a "backward menu" consisting of a number (three in this example) of candidate backward motion vectors (304) for each pixel of the current field. Each candidate backward vector controls a respective member of a set of shift units (305) which, for every pixel in the current field, displaces the previous field (302) by the respective candidate backward vector to produce a respective member of a set of displaced previous fields (306). Each of the displaced previous fields (306) is subtracted from the current field and then filtered in a respective member of a set of error calculation units (307) to produce a set of backward-vector errors (308). This process is analogous to the previously described derivation of the set of error values (208) in the system of Figure 2.
In parallel with the above, the current field (301) and the next field (312) are also applied to a phase correlation unit (313) which calculates a "forward menu" (314) consisting of a number of candidate forward motion vectors. Each candidate forward vector controls a respective member of a set of shift units (315) which displaces the next field (312) by the respective candidate vector to produce a set of displaced next fields (316). Each displaced next field is subtracted from the current field and then filtered in a respective member of a set of error calculation units (317) to produce a set of forward-vector errors (318).
For each pixel in the current field, the vectors in the forward and backward menus (304) and (314) are combined in a combining unit (320) to produce a list (321) of all possible pairs of forward and backward candidate vectors. In this example, with three backward and three forward vectors, there will be nine possible candidate vector pairs. Likewise, the forward and backward error values (308) and (318) are added together in pairs in an adding unit (322) to produce a summed error (323) associated with each candidate vector pair (321).
The two vectors in each of the pairs in the list (321) are respectively added together in an adding unit (324) to produce a set of vector sums (325), one for each pair. Each vector sum is a vector having a horizontal component equal to the sum of the horizontal components of the respective forward and backward vectors, and a vertical component equal to the sum of the respective vertical components. The convention used here for representing forward and backward vectors is that each is regarded as a spatial displacement from the current field to the next or previous field. With this convention, the horizontal and vertical components of vectors representing constant motion across the two field periods will be equal in magnitude and opposite in sign, and will therefore sum to zero. In the general case, each sum of forward and backward vectors represented according to this convention will be a vector equal to the acceleration measured
between the two field intervals.
The set of summed errors (323) for each pair of vectors is then respectively combined with the set of vector sums (325) to produce a set of acceleration-aware errors (327), one for each pair. A suitable combining algorithm would be to add the summed error to a constant multiple of the magnitude of the acceleration (or an approximation thereto). Suitable methods of calculating or approximating the magnitude of the acceleration include: (a) the true magnitude, which is the square root of the sum of the squares of the horizontal and vertical acceleration components (b) the sum or average of the absolute values of the two acceleration components (c) the sum of the higher of the two absolute values and half the lower of the two absolute values of the acceleration components.
Additionally, the magnitude of the acceleration may be "cored" so that values below a small threshold are set to zero. It may also be "clipped" so that values above a large threshold are limited to the threshold. In general, any suitable non-decreasing nonlinearfunction of the magnitude may be employed.
Finally, a comparison unit (328) finds the minimum value member of the set of acceleration-aware errors (327) and outputs an index (329) that identifies the corresponding pair of vectors in the list (321). A selection unit (330) outputs that pair of vectors as an assigned pair (331) of forward and backward vectors for the
current pixel in the current field.
The method has been described here with reference to forward and backward vectors and takes into account the acceleration between two consecutive field intervals. It will be apparent to the skilled person that the method may equally be applied to triplets or higher numbers of vectors calculated across three or more field intervals. It will also be apparent to the skilled person that the method could be applied to the calculation of displacement vectors between multiple cameras, where the acceleration in time is replaced by, or considered together with, suitably normalised displacement variations between overlapping views from pairs of cameras. Such overlapping views may or may not correspond to the same point in time. The term acceleration is thus intended to refer to the difference in the displacement of an image feature between two consecutive inter-image intervals; regardless of whether the sequence of images is a time sequence, a spatial sequence, or sequence of different viewpoints at different times.
The invention may be implemented in many ways including, for example, a sequential process applied in real time to the pixels of the current field or frame in an order corresponding to a scanning raster, or in another example, a processor may retrieve pixel values from field or frame stores, or a file server, at a rate unrelated to the intended rate of display of the images. And, as explained in the introduction, appropriate vectors may be selected according to the invention for subsamples or regions of an image.
Claims (13)
- CLAIMS1. A method of selecting a forward and backward motion vector for each region of a picture in which the selection criterion includes acceleration.
- 2. A method according to Claim I in which the regions are pixels or uniform rectangular blocks of pixels.
- 3. A method according to Claim 1 in which the selection criterion includes the sum of a forward-vector error, a backward-vector error and a function of acceleration.
- 4. A method according to Claim 1 in which the selection is made from lists of candidate forward and backward vectors that are shorter than the lists of all possible forward and backward vectors.
- 5. A method according to Claim 4 in which the lists of candidate vectors are derived from phase correlation.
- 6. Apparatus for selecting a forward and backward motion vector for each region of a picture in which the selection criterion includes acceleration.
- 7. Apparatus according to Claim 6 in which the regions are pixels or uniform rectangular blocks of pixels.
- 8. Apparatus according to Claim 6 in which the selection criterion includes the sum of a forward-vector error, a backward-vector error and a function of acceleration.
- 9. Apparatus according to Claim 6 in which the selection is made from lists of candidate forward and backward vectors that are shorter than the lists of all possible forward and backward vectors.
- 10. Apparatus according to Claim 9 in which the lists of candidate vectors are derived from phase correlation.
- II. A method of video sequence processing, comprising the steps of deriving for a current frame at least two candidate forward motion vectors and at least two candidate backward motion vectors, each motion vector -10-representing the displacement of an object between frames of the video sequence; designating at least four candidate pairs of motion vectors, each candidate pair comprising one of the candidate forward motion vectors and one of the candidate backward motion vectors; calculating an acceleration for each candidate pair and selecting one of the candidate pairs on selection criteria which include said acceleration
- 12. Programmable apparatus programmed to implement a method according to any one of Claims 1 to 5 or 11.
- 13. A computer program product adapted to cause programmable apparatus to implement a method according to any one of Claims ito 5 or 11.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB1206067.9A GB2500909A (en) | 2012-04-04 | 2012-04-04 | Selecting motion vectors on the basis of acceleration |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB1206067.9A GB2500909A (en) | 2012-04-04 | 2012-04-04 | Selecting motion vectors on the basis of acceleration |
Publications (2)
Publication Number | Publication Date |
---|---|
GB201206067D0 GB201206067D0 (en) | 2012-05-16 |
GB2500909A true GB2500909A (en) | 2013-10-09 |
Family
ID=46160351
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
GB1206067.9A Withdrawn GB2500909A (en) | 2012-04-04 | 2012-04-04 | Selecting motion vectors on the basis of acceleration |
Country Status (1)
Country | Link |
---|---|
GB (1) | GB2500909A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8860880B2 (en) | 2012-07-24 | 2014-10-14 | Snell Limited | Offset interpolation of a sequence of images to obtain a new image |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05110934A (en) * | 1991-10-11 | 1993-04-30 | Sony Corp | Motion vector detector for image |
JPH05110933A (en) * | 1991-10-11 | 1993-04-30 | Sony Corp | Motion vector detector for image |
JPH07298270A (en) * | 1994-04-26 | 1995-11-10 | Matsushita Electric Ind Co Ltd | Inter-motion compensation frame prediction coder |
US20060159177A1 (en) * | 2004-12-14 | 2006-07-20 | Stmicroelectronics Sa | Motion estimation method, device, and system for image processing |
GB2469679A (en) * | 2009-04-23 | 2010-10-27 | Imagination Tech Ltd | Object tracking using momentum and acceleration vectors in a motion estimation system |
WO2011070730A1 (en) * | 2009-12-07 | 2011-06-16 | 日本電気株式会社 | Video coding device and video decoding device |
-
2012
- 2012-04-04 GB GB1206067.9A patent/GB2500909A/en not_active Withdrawn
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH05110934A (en) * | 1991-10-11 | 1993-04-30 | Sony Corp | Motion vector detector for image |
JPH05110933A (en) * | 1991-10-11 | 1993-04-30 | Sony Corp | Motion vector detector for image |
JPH07298270A (en) * | 1994-04-26 | 1995-11-10 | Matsushita Electric Ind Co Ltd | Inter-motion compensation frame prediction coder |
US20060159177A1 (en) * | 2004-12-14 | 2006-07-20 | Stmicroelectronics Sa | Motion estimation method, device, and system for image processing |
GB2469679A (en) * | 2009-04-23 | 2010-10-27 | Imagination Tech Ltd | Object tracking using momentum and acceleration vectors in a motion estimation system |
WO2011070730A1 (en) * | 2009-12-07 | 2011-06-16 | 日本電気株式会社 | Video coding device and video decoding device |
Non-Patent Citations (2)
Title |
---|
"Adaptive de-interlacing algorithm based on motion compensation", Proc of Int Conf on Signal Processing, 2004.(ICSP '04), 31 Aug.-4 Sept. 2004, Gu Junxia; Gao Xinbo; Ishimura, H., Vol 1, pages 800-803 * |
High Visual Quality Particle Based Frame Rate Up Conversion with "Acceleration Assisted Motion Trajectory Calibration", IEEE Journal of Display Technology, June 2012,USA,Tsung-Han Tsai; Hsueh-Yi Lin, Vol 8,Nr 6, pages 341 - 351 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8860880B2 (en) | 2012-07-24 | 2014-10-14 | Snell Limited | Offset interpolation of a sequence of images to obtain a new image |
Also Published As
Publication number | Publication date |
---|---|
GB201206067D0 (en) | 2012-05-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8144778B2 (en) | Motion compensated frame rate conversion system and method | |
JP5877469B2 (en) | Object tracking using moment and acceleration vectors in motion estimation systems | |
KR100582856B1 (en) | Motion estimation and motion-compensated interpolation | |
JP3226539B2 (en) | Video image processing | |
JP2911055B2 (en) | Motion estimation method and apparatus | |
US8358878B2 (en) | Method and apparatus for interpolating an image | |
CN104219533B (en) | A kind of bi-directional motion estimation method and up-conversion method of video frame rate and system | |
EP1841234A2 (en) | Apparatus for creating interpolation frame | |
KR100720722B1 (en) | Intermediate vector interpolation method and 3D display apparatus | |
EP2555522A2 (en) | External vectors in a motion estimation system | |
US20090208123A1 (en) | Enhanced video processing using motion vector data | |
JP2009003507A (en) | Image processing method, image processor, and image processing program | |
JP2006504175A (en) | Image processing apparatus using fallback | |
US20050195324A1 (en) | Method of converting frame rate of video signal based on motion compensation | |
KR100565066B1 (en) | Method for interpolating frame with motion compensation by overlapped block motion estimation and frame-rate converter using thereof | |
US8306123B2 (en) | Method and apparatus to improve the convergence speed of a recursive motion estimator | |
US20170085912A1 (en) | Video sequence processing | |
KR100942887B1 (en) | Motion estimation | |
KR20050097936A (en) | Efficient predictive image parameter estimation | |
JP2009533887A (en) | Motion vector field correction apparatus and method | |
GB2514334A (en) | Video processing | |
KR100955415B1 (en) | Method, computer program product and integrated circuit for providing a motion parameter set or a picture repetition pattern, and computer for providing a motion parameter | |
US20080144716A1 (en) | Method For Motion Vector Determination | |
GB2500909A (en) | Selecting motion vectors on the basis of acceleration | |
EP1606952A1 (en) | Method for motion vector determination |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WAP | Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1) |