GB2500909A

GB2500909A - Selecting motion vectors on the basis of acceleration

Info

Publication number: GB2500909A
Application number: GB1206067.9A
Authority: GB
Inventors: Michael James Knee; Martin Weston
Original assignee: Snell Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2012-04-04
Filing date: 2012-04-04
Publication date: 2013-10-09
Also published as: GB201206067D0

Abstract

In video processing, two candidate forward motion vectors and two candidate backward motion vectors are derived for each frame. Four candidate pairs of motion vectors are designated with each candidate pair comprising one of the candidate forward vectors and one of the candidate backward vectors. An acceleration is calculated for each candidate pair and one of the candidate pairs is selected based on that acceleration.

Description

SELECTING MOTION VECTORS

FIELD OF INVENTION

This invention concerns selecting motion vectors in motion estimation of video signals.

BACKGROUND OF THE INVENTION

This invention relates to the estimation of motion vectors between video frames.

In motion-compensated image processing motion vectors are assigned to pixels, or blocks of pixels, in each frame and describe the estimated displacement of each pixel or block in a next frame or a previous frame in a sequence of frames.

In the following description, the motion estimation is considered to be "dense" meaning that a motion vector is calculated for every pixel. The definition of "dense" may be widened to cover the calculation of a motion vector for each small block in the picture, for each pixel in a subsampled version of the picture or for each small region of arbitrary shape within which the motion is expected to be uniform. The invention can be applied with trivial modification to these wider cases.

Motion estimation has application in many image and video processing tasks, including video compression, motion-compensated temporal interpolation for standards conversion or slow-motion synthesis, motion-compensated noise reduction, object tracking, image segmentation and, in the form of displacement estimation, stereoscopic 3D analysis and view synthesis from multiple cameras.

Most applications of motion estimation involve the "projection" (also described as "shifting") of picture information forward or backward in time according to the motion vector that has been estimated. This is known as "motion-compensated" projection. The projection may be to the time instant of an existing frame or field, for example in compression, where a motion-compensated projection of a past or future frame to the current frame instant serves as a prediction of the current frame. Alternatively, the projection may be to a time instant not in the input sequence, for example in motion-compensated standards conversion, where information from a current frame is projected to an output time instant, where it will be used to build a motion-compensated interpolated output frame.

In most cases, it is desirable to estimate both "forward" vectors, which describe motion between the current frame and a future frame, and "backward" vectors, which describe motion between a current frame and a past frame. It is important to note that forward vectors estimated for the current frame are not necessarily the same as backward vectors estimated for the future frame. This is because a motion-compensated projection from current-frame pixels to future-frame pixels is not reversible in general. Not all future-frame pixels will be destinations for motion-compensated projection from the current frame. This mismatch typically occurs in areas known as "occlusions" whore relative motion of objects in the scene leads to parts of a picture being revealed or obscured from one frame to the next. Explicit measurement of forward and backward vectors is helpful for handling occlusions in motion-compensated processing.

An example of a known method of estimating forward and backward vectors for pixels of an image forming part of an image sequence will now be described.

Some terminology will first be defined with reference to Figure 1.

Figure 1 shows one-dimensional sections through three successive frames in a sequence. The horizontal axis of Figure 1 represents time, and the vertical axis represents position. Of course, the skilled person will recognise that Figure 1 is a simplification and that motion vectors used in image processing are generally two dimensional. The illustrated frames are: a previous frame (102); a current frame (101); and, a next frame (103). A forward motion vector (104) is shown assigned to a pixel (105) in the current frame. The motion vector (104) indicates a point (106) in the next frame which is the estimated destination of the current frame pixel (105) in the next frame. Similarly, a backward motion vector (107) assigned to the same current frame pixel (105) indicates a point (108) in the previous frame which is the estimated source of the current frame pixel in the previous frame.

The following descriptions assume that these three frames are consecutive in the sequence, but the described processes are equally applicable in cases where there are intervening frames, for example in some compression algorithms. It should also be noted that a convention has been chosen here whereby forward vectors point to frames in the future. In some compression literature, for example in the MPEG-2 video coding specification (ISO/IEC 13818-2), forward vectors point to frames in the past. However, in the descriptions that follow, there is complete symmetry between forward and backward vectors, so no ambiguity arises from the difference in naming conventions.

In the remainder of this specification temporal samples of an image will be referred to as fields, as would be the case when processing interlaced images.

However, as the skilled person will appreciate, in non-interlaced image formats a temporal sample is represented by a frame; and, fields may be de-interlaced' to form frames within an image process. The spatial sampling of the image is not relevant to the discussion which follows.

We begin the description of a known method of estimating forward and backward vectors by describing an algorithm that estimates backward vectors only. An example of such an algorithm is given in GB2188510 (Thomas). This algorithm is summarised in Figure 2 and assigns a single backward vector to every pixel of a current field in a sequence of fields. The process of Figure 2 is assumed to operate sequentially on the pixels of the current field; the pixel whose vector assignment is currently being determined will be referred to as the current pixel.

The current field (201) and the previous field (202) are applied to a phase correlation unit (203) which calculates a "menu" for every pixel of the current field consisting of a number (three in this example) of candidate motion vectors (204).

Each candidate vector controls a respective member of a set of shift units (205) which displaces the previous field (202) by the respective candidate vector to produce a shifted pixel corresponding to the current pixel of the current field in the respective member of a set of displaced fields (206).

A set of error calculation units (207) produces a set of error values (208), one error value for every menu vector for every pixel of the current field. Each of the error calculation units (207) subtracts the respective one of the displaced fields (206) from the current field (201) and rectifies the result to produce a field of difference magnitudes, which is spatially filtered in a filter centered on the current pixel to give an error value for that pixel and menu vector. The set of three error values (208) for each pixel are compared in a comparison unit (209), which finds the minimum value error. The comparison unit (209) outputs a candidate index (210). which identifies the vector that gave rise to the minimum error. The candidate index (210) for every pixel is then applied to a vector selection unit (211)to select the identified candidate from the menu of vectors (204) as the respective output assigned backward vector (212).

Such an algorithm may trivially be adapted to calculate and assign forward vectors, using the next field in place of the previous field. Thus, the double application of the algorithm of Thomas will estimate a forward and a backward vector for every pixel, as required.

It is the object of this invention to provide an improved method of assigning forward and backward vectors to pixels.

SUMMARY OF THE INVENTION

The invention consists of a method and apparatus for assigning forward and backward motion vectors to image pixels, characterised in that forward and backward vectors are assigned as a pair and the acceleration derived from the pair of vectors is taken into account in the assignment process.

The inventors have noted that methods of calculating forward and backward vectors such as that just described have a significant disadvantage, which arises because the calculation of forward vectors is independent of the calculation of backward vectors. A difference between the forward and backward vectors shows that the motion of a portrayed object between the time instants of the previous and current fields is different from the motion of the object between the time instants of the current and next frames. This change in motion corresponds to an acceleration of the object.

The selection of a vector from a list of candidates does not always result from a clear-cut decision. Two or more candidates may produce errors that are very close in value, so that the selection of the vector corresponding to the minimum error is to some extent arbitrary. If this ambiguity occurs in either or both of the forward and backward directions, then the resulting pair of forward and backward vectors may represent an acceleration in the motion of a portrayed object that is very unlikely, or even impossible, at that point in the video sequence. In that eventuality, then at least one of the pair of the assigned vectors will be wrong and, when used for motion-compensated interpolation for example, will produce undesirable artefacts in the output picture.

The inventors have recognised that the problem of incorrectly assigned vectors can be ameliorated by making use the acceleration of any particular feature of the image is likely to be small. In terms of Figure 1, this is equivalent to asserting that the angle between forward and backward vectors (104 and 107) is close to 180 degrees, or that the sum of the values of the two vectors is small.

BRIEF DESCRIPTION OF THE DRAWINGS

An example of the invention will now be described with reference to the drawings in which: Figure 1 is a diagram showing previous, current and next frames in an image sequence; and, forward and backward motion vectors extending from a pixel in the current frame; Figure 2 is a block diagram of apparatus for assigning backward motion

vectors to pixels according to the prior art;

Figure 3 is a block diagram of apparatus for assigning forward and backward motion vectors to pixels according to invention

DETAILED DESCRIPTION OF THE INVENTION

Forward and backward motion vector assignment apparatus according to an exemplary embodiment of the invention will now be described.

Referring to Figure 3, the current field (301)and the previous field (302) are applied to a phase correlation unit (303) which calculates a "backward menu" consisting of a number (three in this example) of candidate backward motion vectors (304) for each pixel of the current field. Each candidate backward vector controls a respective member of a set of shift units (305) which, for every pixel in the current field, displaces the previous field (302) by the respective candidate backward vector to produce a respective member of a set of displaced previous fields (306). Each of the displaced previous fields (306) is subtracted from the current field and then filtered in a respective member of a set of error calculation units (307) to produce a set of backward-vector errors (308). This process is analogous to the previously described derivation of the set of error values (208) in the system of Figure 2.

In parallel with the above, the current field (301) and the next field (312) are also applied to a phase correlation unit (313) which calculates a "forward menu" (314) consisting of a number of candidate forward motion vectors. Each candidate forward vector controls a respective member of a set of shift units (315) which displaces the next field (312) by the respective candidate vector to produce a set of displaced next fields (316). Each displaced next field is subtracted from the current field and then filtered in a respective member of a set of error calculation units (317) to produce a set of forward-vector errors (318).

For each pixel in the current field, the vectors in the forward and backward menus (304) and (314) are combined in a combining unit (320) to produce a list (321) of all possible pairs of forward and backward candidate vectors. In this example, with three backward and three forward vectors, there will be nine possible candidate vector pairs. Likewise, the forward and backward error values (308) and (318) are added together in pairs in an adding unit (322) to produce a summed error (323) associated with each candidate vector pair (321).

The two vectors in each of the pairs in the list (321) are respectively added together in an adding unit (324) to produce a set of vector sums (325), one for each pair. Each vector sum is a vector having a horizontal component equal to the sum of the horizontal components of the respective forward and backward vectors, and a vertical component equal to the sum of the respective vertical components. The convention used here for representing forward and backward vectors is that each is regarded as a spatial displacement from the current field to the next or previous field. With this convention, the horizontal and vertical components of vectors representing constant motion across the two field periods will be equal in magnitude and opposite in sign, and will therefore sum to zero. In the general case, each sum of forward and backward vectors represented according to this convention will be a vector equal to the acceleration measured

between the two field intervals.

The set of summed errors (323) for each pair of vectors is then respectively combined with the set of vector sums (325) to produce a set of acceleration-aware errors (327), one for each pair. A suitable combining algorithm would be to add the summed error to a constant multiple of the magnitude of the acceleration (or an approximation thereto). Suitable methods of calculating or approximating the magnitude of the acceleration include: (a) the true magnitude, which is the square root of the sum of the squares of the horizontal and vertical acceleration components (b) the sum or average of the absolute values of the two acceleration components (c) the sum of the higher of the two absolute values and half the lower of the two absolute values of the acceleration components.

Additionally, the magnitude of the acceleration may be "cored" so that values below a small threshold are set to zero. It may also be "clipped" so that values above a large threshold are limited to the threshold. In general, any suitable non-decreasing nonlinearfunction of the magnitude may be employed.

Finally, a comparison unit (328) finds the minimum value member of the set of acceleration-aware errors (327) and outputs an index (329) that identifies the corresponding pair of vectors in the list (321). A selection unit (330) outputs that pair of vectors as an assigned pair (331) of forward and backward vectors for the

current pixel in the current field.

The method has been described here with reference to forward and backward vectors and takes into account the acceleration between two consecutive field intervals. It will be apparent to the skilled person that the method may equally be applied to triplets or higher numbers of vectors calculated across three or more field intervals. It will also be apparent to the skilled person that the method could be applied to the calculation of displacement vectors between multiple cameras, where the acceleration in time is replaced by, or considered together with, suitably normalised displacement variations between overlapping views from pairs of cameras. Such overlapping views may or may not correspond to the same point in time. The term acceleration is thus intended to refer to the difference in the displacement of an image feature between two consecutive inter-image intervals; regardless of whether the sequence of images is a time sequence, a spatial sequence, or sequence of different viewpoints at different times.

The invention may be implemented in many ways including, for example, a sequential process applied in real time to the pixels of the current field or frame in an order corresponding to a scanning raster, or in another example, a processor may retrieve pixel values from field or frame stores, or a file server, at a rate unrelated to the intended rate of display of the images. And, as explained in the introduction, appropriate vectors may be selected according to the invention for subsamples or regions of an image.

Claims

CLAIMS1. A method of selecting a forward and backward motion vector for each region of a picture in which the selection criterion includes acceleration.
2. A method according to Claim I in which the regions are pixels or uniform rectangular blocks of pixels.
3. A method according to Claim 1 in which the selection criterion includes the sum of a forward-vector error, a backward-vector error and a function of acceleration.
4. A method according to Claim 1 in which the selection is made from lists of candidate forward and backward vectors that are shorter than the lists of all possible forward and backward vectors.
5. A method according to Claim 4 in which the lists of candidate vectors are derived from phase correlation.
6. Apparatus for selecting a forward and backward motion vector for each region of a picture in which the selection criterion includes acceleration.
7. Apparatus according to Claim 6 in which the regions are pixels or uniform rectangular blocks of pixels.
8. Apparatus according to Claim 6 in which the selection criterion includes the sum of a forward-vector error, a backward-vector error and a function of acceleration.
9. Apparatus according to Claim 6 in which the selection is made from lists of candidate forward and backward vectors that are shorter than the lists of all possible forward and backward vectors.
10. Apparatus according to Claim 9 in which the lists of candidate vectors are derived from phase correlation.
II. A method of video sequence processing, comprising the steps of deriving for a current frame at least two candidate forward motion vectors and at least two candidate backward motion vectors, each motion vector -10-representing the displacement of an object between frames of the video sequence; designating at least four candidate pairs of motion vectors, each candidate pair comprising one of the candidate forward motion vectors and one of the candidate backward motion vectors; calculating an acceleration for each candidate pair and selecting one of the candidate pairs on selection criteria which include said acceleration
12. Programmable apparatus programmed to implement a method according to any one of Claims 1 to 5 or 11.
13. A computer program product adapted to cause programmable apparatus to implement a method according to any one of Claims ito 5 or 11.