GB2281167A - Motion compensated video signal processing - Google Patents

Abstract

A motion compensated video signal processing apparatus, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, comprises a means for comparing blocks of the pair of input images to generate a plurality of correlation surfaces. Each correlation surface is examined to detect points of maximum correlation within the correlation surface and a motion vector is generated from the detected points of maximum correlation. The apparatus also provides detection of a plurality of points of maximum correlation within a correlation surface for a block, and assigns coresponding motion vectors accordingly. A plurality of motion vectors can be assigned to a block by growing the correlation surfaces (see figs. 4a - 4f) and detecting the points of maximum correlation in the original and combined correlation surfaces. <IMAGE>

Description

MOTION COMPENSATED VIDEO SIGNAL PROCESSING This invention relates to motion compensated video signal processing.

Motion compensated video signal processing is used in applications such as television standards conversion, film standards conversion and conversion between video and film standards.

In a motion compensated television standards converter, such as the converter described in UK patent application number GB-A-2 231 749, pairs of successive input images are processed to generate sets of motion vectors representing image motion between the pair of input images. The processing is carried out on discrete blocks of the images, so that each motion vector represents the inter-image motion of the content of a respective block.

In the motion vector estimation process a correlation surface representing spatial correlation between blocks of the two input images is examined to detect points of maximum correlation. (The correlation surface actually represents the difference between the two input images so that the points of maximum correlation are in fact minima on the correlation surface). When a minimum has been detected, a motion vector is generated from the spatial position of the minimum in the correlation surface. Tests are performed to determine whether that minimum represents a significant peak in correlation with respect to the remainder of the correlation surface. If the minimum passes this test it is treated as being "valid", and a confidence flag associated with the motion vector is set.

After the individual motion vectors, termed "local" motion vectors, have been derived for the blocks of the input image, the local motion vectors are supplied to a motion vector reducer. The motion vector reducer assigns to each block a set of motion vectors selected, in order of preference, from: the zero motion vector; the local motion vector for that block; the local motion vectors for blocks neighbouring that block in the input image; and a set of vectors known as "global" motion vectors. The global motion vectors are derived for each image by ranking all the (valid) local motion vectors for this image in order of decreasing frequency, and then selecting from the most common of the local motion vectors a number of unique motion vectors to be used as global motion vectors.A constraint may be applied here to ensure that the global motion vectors differ from each other by at least a minimum amount.

The sets of motion vectors for all the blocks of the input image are then supplied to a motion vector selector. The purpose of the motion vector selector is to allocate to each picture element (pixel) of a block of an output image a single motion vector selected from the set of motion vectors supplied for a corresponding block of the input image. As this motion vector will be used in interpolation of the output pixel, the motion vectors supplied to the motion vector selector are tested so that the "best" motion vector is selected for each pixel from the set of motion vectors for the corresponding block.The selected motion vector for each pixel is then supplied to a motion compensated interpolator which uses these motion vectors to interpolate the pixels of an output image from the pair of input images in dependence upon the temporal offset of the output image from the pair of input images.

If a search block used in the generation of a correlation surface is dominated by a large object, the motion of that large object will cause a large minimum in the correlation surface. This in turn will mean that a motion vector is generated representing the motion of the large object. Because only one motion vector is derived from each correlation surface, the motion of a smaller object in the same block will be masked by that of the large object. If the smaller object does not extend into other blocks, it is likely that a suitable motion vector representing the motion of the smaller object will not be generated.

An example of this problem is shown in Figure 1 of the accompanying drawings, in which a large stationary object 10 (e.g. a wall) dominates a search block 20, which also contains a much smaller moving object 30 (a lorry). A cross-section of a correlation surface generated using this search block is shown in Figure 2 of the accompanying drawings; a large minimum 40 (from which a motion vector is generated) represents the motion of the stationary wall 10, while a smaller, localised, minimum 50 (which is ignored in favour of the larger minimum 40) represents the lorry's motion.

This invention provides a motion compensated video signal processing apparatus in which motion vectors are generated for use in interpolation of an output image of an output video signal, the motion vectors representing image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; means for examining the correlation surfaces to detect points of maximum correlation within the correlation surfaces; and means for generating motion vectors from the detected points of maximum correlation in the correlation surfaces; in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface.

In an apparatus according to the invention, more than one motion vector may be generated from a single correlation surface. This means that motion vectors can be generated to reflect the motion of a number of objects within a search block, rather than simply the largest object or part of the background.

For convenient later processing of the motion vectors, it is preferable to limit the maximum number of motion vectors which can be generated from a single correlation surface. To this end, it is preferred that the means for generating. motion vectors is operable to generate up to a predetermined number of motion vectors from each correlation surface.

Although the predetermined number of motion vectors could be selected in an arbitrary fashion in the case of a correlation surface having greater than the predetermined number of local correlation maxima, it is preferred that the means for generating motion vectors is operable to generate a respective motion vector from the predetermined number of most significant local correlation maxima.

In order to reduce the risk of generating a motion vector, which may ultimately be used in the interpolation of a pixel of the output image, from noise or alias components in the correlation surfaces, it is preferred that the apparatus comprises means for performing a confidence test on each motion vector generated from a secondary correlation maximum, a secondary correlation maximum being a point of local maximum correlation in a correlation surface other than the point of absolute maximum correlation in that correlation surface; and means for preventing the use, in interpolation of the output image, of motion vectors derived from secondary correlation maxima and which failed the confidence test.

In one preferred embodiment the means for performing a confidence test comprises: means for detecting whether each motion vector generated from a secondary correlation maximum lies within a predetermined threshold amount of one or more motion vectors derived from correlation surfaces generated from adjacent blocks of the input image. A motion vector generated from a secondary correlation maximum passes the confidence test if it does indeed lie within the predetermined threshold amount of a neighbouring motion vector.

Preferably the apparatus comprises means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the motion vectors.

In another preferred embodiment the means for performing a confidence test comprises means for detecting whether each motion vector generated from a secondary correlation maximum lies within a predetermined threshold amount of a global motion vector.

In a further preferred embodiment the means for performing a confidence test comprises means for detecting a change in sign of the rate of change of a correlation surface in a predetermined number of non-parallel linear directions passing through a secondary correlation maximum.

Preferably the predetermined number of non parallel linear directions comprises a direction indicative of horizontal image motion and a direction indicative of vertical image motion.

A further criterion may be added to the "rate of change" test described above; it is preferred that the means for performing a confidence test comprises means for detecting whether the rate of change of the correlation surface adjacent to a secondary correlation maximum exceeds a predetermined amount.

Vector reduction (the assignment of a group of motion vectors to each block, from which vectors are later selected for use in interpolation of pixels of the output image) is preferably modified to reflect the availability of multiple vectors from a single correlation surface. To this end, it is preferred that the apparatus comprises means for assigning to each of the blocks a group of motion vectors selected from, in the following order of preference: (i) a motion vector indicative of zero image motion; (ii) the motion vector generated from a point of absolute maximum correlation in the correlation surface derived from that block; (iii) motion vectors generated from secondary correlation maxima in the correlation surface derived from that block; and (iv) the global motion vectors.

Viewed from a second aspect this invention provides a method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; examining the correlation surfaces to detect points of maximum correlation within the correlation surfaces; and generating motion vectors from the detected points of maximum correlation in the correlation surfaces; in which, for a correlation surface having a plurality of points of local maximum correlation, the step of generating motion vectors comprises generating a corresponding plurality of motion vectors from that correlation surface.

A previously proposed technique for use in a motion compensated standards conversion apparatus was to "grow" correlation surfaces during vector estimation. In this technique, each correlation surface was examined for minima in its own right, and was also added to adjacent correlation surfaces to form "grown" surfaces for examination.

The process of correlation surface "growing" is illustrated in Figures 3 and 4a to 4f of the accompanying drawings. Figure 3 shows an array of search blocks 100 in an input image forming one of a pair of input images from which motion vectors are to be derived. A correlation surface is generated from each of the search blocks 100 by comparing that search block with a larger search area in the other of the pair of input images. Each of a group of 15 search blocks (in fact a rectangular array of five search blocks horizontally by three search blocks vertically) is referred to by a lower case letter from "a" to tot .

In the previously proposed technique of correlation surface growing, each correlation surface is added to predetermined groups of correlation surfaces generated from surrounding search blocks. A normalisation process then takes place. An example is shown in Figures 4a to 4f, in which six "grown" correlation surfaces generated from a correlation surface "h" (Figure 3) are shown. In each case, Figures 4a to 4f show the spatial relationship of the correlation surfaces from which each "grown" correlation surface is generated.In particular, Figure 4a illustrates the correlation surface "h" itself; Figure 4b represents a sum of the correlation surfaces "g", "h" and "i"; Figure 4b represents a sum of the correlation surfaces "g", "h" and "i"; Figure 4c represents a sum of the correlation surfaces "f", "gel, "h", "i" and "j"; Figure 4d represents a sum of the correlation surfaces "c", "h" and "m"; Figure 4e represents a sum of the correlation surfaces "b", "c", "d", "g", "h", "i", "1", "m" and "n"; and Figure 4f represents a sum of all of the fifteen correlation surfaces "a" to "o".

Figure 5 of the accompanying drawings is a schematic diagram of a previously proposed circuit for performing correlation surface growing. Each correlation surface is passed sequentially to "growing circuits" 110, which add groups of adjacent surfaces to generate the six output surfaces shown in Figures 4a to 4f. Minima detection and vector testing (e.g. the confidence test mentioned above) are then performed 120 in parallel on each of the six surfaces. The results of the minima testing are then passed to selection logic 130 which selects a motion vector derived from a single one of the six surfaces, to be passed to vector reduction.

If a minimum derived from a "grown" surface was used, this meant that vector estimation effectively took place over a wider image area.

However, the ability to track the movement of small objects was reduced. The selection of a motion vector derived from one of the "grown" correlation surfaces, to be passed to vector reduction, involved a compromise between the detection of detailed motion (by using a vector derived from a small correlation surface) and the greater accuracy offered by the use of a larger ("grown") correlation surface.

Viewed from a third aspect this invention provides a motion compensated video signal processing apparatus in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of original correlation surfaces, each representing correlation between a respective search block of one input image of the pair and a search area, comprising a plurality of blocks, in the other image of the pair; means for combining each original correlation surface with groups of original correlation surfaces generated from surrounding search blocks, to generate a respective plurality of combined correlation surfaces; and means for generating a motion vector from each of the original correlation surfaces and the combined correlation surfaces.

According to this aspect of the invention, motion vectors are generated from the various "grown" correlation surfaces and are passed to vector reduction. This avoids the need to make an early compromise between the detection of detailed motion (by using a vector derived from a small correlation surface) and the greater accuracy offered by the use of a larger ("grown") correlation surface.

Although various configurations could be employed, it is preferred that the means for combining is operable to combine each original correlation surface with five groups of original correlation surfaces generated from search blocks in a surrounding array of search blocks.

Preferably the array is a 5 x 3 array of search blocks.

In an advantageously simple embodiment the means for combining comprises: means for receiving serial data representing successive original correlation surfaces; a plurality of delay circuits operable to generate respective delayed data from the serial data; and means for adding the delayed data generated by a predetermined group of the delay circuits, to generate output data representing a combined correlation surface.

Preferably the means for adding comprises means for performing normalised addition of the delayed data.

In a preferred embodiment the apparatus comprises means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the motion vectors.

Preferably the vector reduction process is modified to reflect the availability of multiple motion vectors generated from "grown" correlation surfaces. It is preferred that the apparatus comprises means for assigning to each of the blocks a group of motion vectors selected from, in the following order of preference: (i) a motion vector indicative of zero image motion; (ii) the motion vector generated from a selected combined correlation surface; (iii) motion vectors having at least a predetermined number of occurrences amongst those motion vectors generated from the original correlation surfaces derived from the current block and adjacent blocks; and (iv) the global motion vectors.

In another embodiment, the motion vectors derived from all of the combined correlation surfaces could be selected in preference to the global motion vectors.

In a preferred embodiment the selected combined correlation surface is a combined correlation surface generated by combining original correlation surfaces generated from each of the array of search blocks.

Preferably the apparatus comprises means for examining the correlation surface to detect points of maximum correlation within the original and combined correlation surfaces; and means for generating motion vectors from the detected points of maximum correlation; in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface. It is also preferred that the apparatus comprises means for performing a predetermined confidence test on each motion vector; and means for preventing the assignment of motion vectors which failed the confidence test.

Viewed from a fourth aspect this invention provides a method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of original correlation surfaces, each representing correlation between a respective search block of one input image of the pair and a search area, comprising a plurality of blocks, in the other image of the pair; combining each original correlation surface with groups of original correlation surfaces generated from surrounding search blocks, to generate a respective plurality of combined correlation surfaces; and generating a motion vector from each of the original correlation surfaces and the combined correlation surfaces.

As mentioned above, the purpose of the motion vector reducer is to provide the motion vector selector with a reasonable selection of motion vectors for each block from which to select the most appropriate motion vector for each output pixel corresponding to that block. The global motion vectors each represent a motion which was frequently detected in the image as a whole, so that it is often reasonable to include these vectors in the sets of vectors from which the most appropriate vectors are selected by the motion vector selector.

However, the motion vector selection process is not completely reliable, and may under some circumstances, erroneously select a global motion vector (derived from the motion of a different part of the image) rather than the correct local motion vector. This can result in disturbing artifacts in the output image, such as those illustrated in Figure 6 of the accompanying drawings. Here, a global motion vector corresponding to the motion of a vehicle 140 provides a good match between test blocks 150, 160 in a stationary window in the background of the image. The vehicle motion vector has therefore been inappropriately selected for use in interpolation of the region of the image at the centre of the window. The result is a gap 170 in the central vertical frame member of the window.

Viewed from a fifth aspect this invention provides a motion compensated video signal processing apparatus in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; means for generating a plurality of motion vectors from the correlation surfaces, each motion vector depending on a point of maximum correlation in that correlation surface; means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the plurality of motion vectors; means for examining the correlation surfaces to detect those global motion vectors which are locally valid for each block, by detecting whether the correlation surface generated from that block has a local correlation maximum indicative of image motion substantially identical to that represented by each global motion vector; and means for assigning to each of the blocks a group of motion vectors selected from: the zero motion vector; the motion vector representing image motion of that block; and those global motion vectors which are locally valid for that block.

In accordance with this aspect of the present invention, a test is performed to detect whether each global motion vector is "locally valid" for each block of the output image, by examining the corresponding correlation surface to detect evidence of the motion represented by the global motion vector. If no such evidence is found, that global vector is not allowed to be selected for possible use in interpolation of the output block. This can avoid or alleviate the problems described above.

Preferably the detecting means comprises: means for detecting a change in sign of the rate of change of a correlation surface in a predetermined number of non-parallel linear directions passing through a position indicative of image motion represented by a global motion vector.

Although various linear directions could be used, in an advantageously simple embodiment the predetermined number of non parallel linear directions comprises a direction indicative of horizontal image motion and a direction indicative of vertical image motion.

Preferably the apparatus comprises means for examining the correlation surface to detect points of maximum correlation within the correlation surfaces; and in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface.

In a preferred embodiment the apparatus comprises means for combining each correlation surface with groups of correlation surfaces generated from surrounding blocks, to generate a respective plurality of combined correlation surfaces; and means for generating a respective motion vector from each of the combined correlation surfaces.

Viewed from a sixth aspect this invention provides a method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; generating a plurality of motion vectors from the correlation surfaces, each motion vector depending on a point of maximum correlation in that correlation surface; deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the plurality of motion vectors; detecting those global motion vectors which are locally valid for each block, by detecting whether the correlation surface generated from that block has a local correlation maximum indicative of image motion substantially identical to that represented by each global motion vector; and assigning to each of the blocks a group of motion vectors selected from: the zero motion vector; the motion vector representing image motion of that block; and those global motion vectors which are locally valid for that block.

Apparatus according to the invention may be particularly advantageously employed in a television standards conversion apparatus.

Features of the respective aspects of the invention defined above may be combined according to all of the possible various combinations.

An embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings, throughout which like parts are referred to by like references, and in which: Figure 1 is a schematic diagram of a portion of an image representing a large stationary object and a small moving object; Figure 2 is a schematic diagram illustrating a cross-section through a correlation surface generated from the image of Figure 1; Figure 3 illustrates an array of search blocks; Figures 4a to 4f illustrate the "growing" of correlation surfaces; Figure 5 is a schematic diagram of a circuit for performing correlation surface growing; Figure 6 illustrates artifacts in an output image due to the erroneous selection of a global motion vector; Figure 7 is a schematic block diagram of a motion compensated television standards converter;; Figure 8 illustrates vertical subsampling of an interlaced video field; Figure 9 is a schematic diagram of a correlation surface; Figure 10 illustrates a part of a modified motion processor; Figure 11 is a schematic diagram of a correlation surface having two minima; Figure 12 schematically illustrates the examination of a correlation surface along two perpendicular directions; Figures 13a and 13b illustrate cross-sections through the correlation surface of Figure 12 along the two perpendicular directions; Figure 14 illustrates a part of a modified motion processor; Figure 15 is a schematic block diagram of a part of a correlation surface processor; Figure 16 illustrates a normalising adder; Figure 17 illustrates a part of a modified motion processor;; Figure 18 is a representation of the image of Figure 6, showing blocks of the image for which evidence of a global vector was found; Figures 19 illustrates a correlation surface; and Figure 20 illustrates a cross-section through the correlation surface of Figure 19.

Figure 7 is a schematic block diagram of a motion compensated television standards conversion apparatus. The apparatus receives an input interlaced digital video signal 250 (e.g. an 1125/60 2:1 high definition video signal (HDVS)) and generates an output interlaced digital video signal 260 (e.g a 1250/50 2:1 signal).

The input video signal 250 is first supplied to an input buffer/packer 310. In the case of a conventional definition input signal, the input buffer/packer 310 formats the image data into a high definition (16:9 aspect ratio) format, padding with black pixels where necessary. For a HDVS input the input buffer/packer 310 merely provides buffering of the data.

The data are passed from the input buffer/packer 310 to a matrix circuit 320 in which (if necessary) the input video signal's colorimetry is converted to the colorimetry of the desired output signal, such as the standard "CCIR recommendation 601" (Y,Cr,Cb) colorimetry.

From the matrix circuit 320 the input video signal is passed to a time base changer and delay 330, and via a sub-sampler 370 to a subsampled time base changer and delay 380. The time base changer and delay 330 determines the temporal position of each field of the output video signal, and selects the two fields of the input video signal which are temporally closest to that output field for use in interpolating that output field. For each field of the output video signal, the two input fields selected by the time base changer are appropriately delayed before being supplied to an interpolator 340 in which that output field is interpolated. A control signal t, indicating the temporal position of each output field with respect to the two selected input fields, is supplied from the time base changer and delay 330 to the interpolator 340.

The subsampled time base changer and delay 380 operates in a similar manner, but using spatially subsampled video supplied by the subsampler 370. Pairs of fields, corresponding to the pairs selected by the time base changer 330, are selected by the subsampled time base changer and delay 380 from the subsampled video, to be used in the generation of motion vectors.

The time base changers 330 and 380 can operate according to synchronisation signals associated with the input video signal, the output video signal, or both. In the case in which only one synchronisation signal is supplied, the timing of fields of the other of the two video signals is generated deterministically within the time base changers 330. 380.

The pairs of fields of the subsampled input video signal selected by the subsampled time base changer and delay 380 are supplied to a motion processor 385 comprising a direct block matcher 390, a correlation surface processor 400, a motion vector estimator 410, a motion vector reducer 420, a motion vector selector 430 and a motion vector post-processor 440. The pairs of input fields are supplied first to the direct block matcher 390 which calculates correlation surfaces representing the spatial correlation between search blocks in the temporally earlier of the two selected input fields and (larger) search areas in the temporally later of the two input fields.

From the correlation surfaces output by the block matcher 390, the correlation surface processor 400 generates a larger number of interpolated correlation surfaces, which are then passed to the motion vector estimator 410. The motion vector estimator 410 detects points of greatest correlation in the interpolated correlation surfaces. (The original correlation surfaces actually represent the difference between blocks of the two input fields; this means that the points of maximum correlation are in fact minima on the correlation surfaces, and are referred to as "minima"). In order to detect a minimum, additional points on the correlation surfaces are interpolated, providing a degree of compensation for the loss of resolution caused by the use of subsampled video to generate the surfaces.From the detected minimum on each correlation surface, the motion vector estimator 410 generates a motion vector which is supplied to the motion vector reducer 420.

The motion vector estimator 410 also performs a confidence test on each generated motion vector to establish whether that motion vector is significant above the average data level, and associates a confidence flag with each motion vector indicative of the result of the confidence test. The confidence test, known as the "threshold" test, is described (along with certain other features of the apparatus of Figure 7) in GB-A-2 231 749.

A test is also performed by the motion vector estimator 410 to detect whether each vector is aliased. In this test, the correlation surface (apart from an exclusion zone around the detected minimum) is examined to detect the next lowest minimum. If this second minimum does not lie at the edge of the exclusion zone, the motion vector derived from the original minimum is flagged as being potentially aliased.

The motion vector reducer 420 operates to reduce the choice of possible motion vectors for each pixel of the output field, before the motion vectors are supplied to the motion vector selector 430. The output field is notionally divided into blocks of pixels, each block having a corresponding position in the output field to that of a search block in the earlier of the selected input fields. The motion vector reducer compiles a group of four motion vectors to be associated with each block of the output field, with each pixel in that block eventually being interpolated using a selected one of that group of four motion vectors.

Vectors which have been flagged as "aliased" are re-qualified during vector reduction if they are identical to non-flagged vectors in nearby blocks.

As part of its function, the motion vector reducer 420 counts the frequencies of occurrence of "good" motion vectors (i.e. motion vectors which pass the confidence test and the alias test, or which were requalified as non-aliased), with no account taken of the position of the blocks of the input fields used to obtain those motion vectors. The good motion vectors are then ranked in order of decreasing frequency.

The most common of the good motion vectors which are significantly different to one another are then classed as "global" motion vectors.

Three motion vectors which pass the confidence test are then selected for each block of output pixels and are supplied, with the zero motion vector, to the motion vector selector 430 for further processing.

These three selected motion vectors are selected in a predetermined order of preference from: (i) the motion vector generated from the corresponding search block (the "local" motion vector"); (ii) those generated from surrounding search blocks ("neighbouring" motion vectors); and (iii) the global motion vectors.

The motion vector selector 430 also receives as inputs the two input fields which were selected by the subsampled time base changer and delay 380 and which were used to calculate the motion vectors.

These fields are suitably delayed so that they are supplied to the motion vector selector 430 at the same time as the vectors derived from them. The motion vector selector 430 supplies an output comprising one motion vector per pixel of the output field. This motion vector is selected from the four motion vectors for that block supplied by the motion vector reducer 420.

The vector selection process involves detecting the degree of correlation between test blocks of the two input fields pointed to by a motion vector under test. The motion vector having the greatest degree of correlation between the test blocks is selected for use in interpolation of the output pixel. A "motion flag" is also generated by the vector selector. This flag is set to "static" (no motion) if the degree of correlation between blocks pointed to by the zero motion vector is greater than a preset threshold.

The vector post-processor reformats the motion vectors selected by the motion vector selector 430 to reflect any vertical or horizontal scaling of the picture, and supplies the reformatted vectors to the interpolator 340. Using the motion vectors, the interpolator 340 interpolates an output field from the corresponding two (nonsubsampled) interlaced input fields selected by the time base changer and delay 330, taking into account any image motion indicated by the motion vectors currently supplied to the interpolator 340.

If the motion flag indicates that the current output pixel lies in a moving or temporally changing part of the image, pixels from the two selected fields supplied to the interpolator are combined in relative proportions depending on the temporal position of the output field with respect to the two input fields (as indicated by the control signal t), so that a larger proportion of the nearer input field is used. If the motion flag is set to "static" then the temporal weighting is fixed at 50% of each input field. The output of the interpolator 340 is passed to an output buffer 350 for output as a high definition output signal, and to a down-converter 360 which generates a conventional definition output signal 365, using the motion flag to control filter selection.

The down-converter 360 allows a representation of the output of the apparatus (which may be, for example, a high definition video signal) to be monitored, transmitted and/or recorded using conventional definition apparatus. This has benefits because conventional definition recording equipment is significantly cheaper and very much more widespread than high definition equipment. For example, a simultaneous output of conventional and high definition video may be required for respective transmission by terrestrial and satellite channels.

The subsampler 370 performs horizontal and vertical spatial subsampling of the input video fields received from the matrix 320, before those input fields are supplied to the time base changer 380.

Horizontal subsampling is a straightforward operation in that the input fields are first prefiltered by a half-bandwidth low pass filter (in the present case of 2:1 horizontal decimation) and alternate video samples along each video line are then discarded, thereby reducing by one half the number of samples along each video line.

Vertical subsampling of the input fields is complicated by the fact that, in this embodiment, the input video signal 250 is interlaced. This means that successive lines of video samples in each interlaced field are effectively two video lines apart, and that the lines in each field are vertically displaced from those in the preceding or following field by one video line of the complete frame.

One approach to vertical subsampling would be to perform progressive scan conversion (to generate successive progressively scanned video frames each having 1125 lines) and then to subsample the progressively scanned frames by a factor of 2 to perform the vertical subsampling. However, efficient progressive scan conversion would require a degree of motion compensated processing, which processing could adversely affect the operation of the motion processor 385.

Furthermore, real-time progressive scan conversion of a high definition video signal would require particularly powerful and complex processing apparatus.

A simpler approach to vertical spatial subsampling is shown in Figure 8, in which the input fields are first low pass filtered in the vertical direction (to reduce potential aliasing) and a filtering operation is then performed which effectively displaces each pixel vertically by half a video line downwards (for even fields) or upwards (for odd fields). The resulting displaced fields are broadly equivalent to progressively scanned frames which have been subsampled vertically by a factor of two.

In summary, therefore, the result of the subsampling operations described above is that the motion processor 385 operates on pairs of input fields which are spatially subsampled by a factor of two in the horizontal and the vertical directions. This reduces the processing required for motion vector estimation by a factor of four.

Figure 9 is a schematic diagram of a correlation surface 500.

The correlation surface represents the difference between a search block of the earlier of the two input fields from which the surface is generated, and a (larger) search area in the later of the two input fields. A peak in correlation is therefore represented by a minimum point 510 on the correlation surface 500. The position of the minimum point 510 on the correlation surface 500 determines the magnitude and direction of the motion vector derived from that correlation surface.

As mentioned above, if a search block used for the generation of a correlation surface is dominated by a large object (which may be moving or stationary), then the detected motion of that large object will cause a correspondingly large minimum in the correlation surface.

If one motion vector is generated from each correlation surface, the motion vector will reflect the motion of the large object, thus masking the motion of any smaller objects in the same block. This means that if a small moving object falls within a single search block, then in an apparatus in which only one motion vector is generated from each correlation surface, there is no way of accurately representing the motion of the small object.

In accordance with one embodiment of the present invention, Figure 10 shows a modification of part of the motion processor 385' in which a plurality of motion vectors may be generated from each interpolated correlation surface output by the correlation surface processor 400. In particular, a modified vector estimator 550 and a modified vector reducer 560 are employed.

Before the operation of the vector estimator 550 will be described, the nature of a correlation surface having multiple minima representing the motion of objects of differing sizes within the search block will first be described. An example of such a correlation surface is illustrated schematically in Figure 11.

In Figure 11 a correlation surface 570 has a large minimum 580 and a secondary minimum 590. The minimum 580 represents the point of minimum difference (maximum correlation) on the entire correlation surface. However, in contrast, the minimum 590 is only a secondary (localised) minimum representing a point of greater correlation than the immediately surrounding parts of the correlation surface.

Accordingly, the vector estimator 550 must detect localised as well as absolute minima in the correlation surface.

In operation, the vector estimator 550 generates one motion vector for each minimum detected in each interpolated correlation surface, up to a predetermined maximum number of minima. In the example shown in Figure 10, up to six motion vectors are generated from each interpolated correlation surface and are passed to the vector reducer 560. If the correlation surface has more than six minima, motion vectors are generated from the six most significant minima.

It is desirable to take precautions against certain problems which can arise when multiple minima are detected from a single correlation surface. In the example shown in Figure 11, the two minima reflect the motions of a large and a small object within the same search block. However, in an image having an area of regularly spaced patterning, multiple minima can be detected which are regularly spaced across the correlation surface. One of these minima will represent the correct motion of the image patterning; the others are due to alias interactions between the image patterning and the size of the search blocks used to generate the correlation surfaces. In order to help prevent the use of an incorrect motion vector in the interpolation of the output image, the multiple motion vectors generated from a single correlation surface are subjected to thorough testing before use.Four possible techniques for testing a plurality of motion vectors derived from a single correlation surface are: a) Each of the motion vectors derived from secondary minima may be compared with vectors derived from neighbouring correlation surfaces.

This comparison, which may be performed in the vector reducer 560, allows a motion vector derived from a secondary minimum to be used for interpolation of the output image if that motion vector lies within a predetermined threshold of a motion vector derived from a neighbouring correlation surface.

b) A motion vector derived from a secondary minimum may only be used if that vector is substantially identical to a known global motion vector. The comparison of secondary minima and global motion vectors will be described in greater detail below.

c) A differential technique may be employed, in which the differential (rate of change) of the correlation surface is examined along various linear directions. A change in sign of the differential indicates a localised minimum in the direction of examination. If there is a change in sign of the differential along a predetermined number of directions, that minimum can be used to generate a motion vector.

An example of technique (c) is illustrated in Figures 12, 13a and 13b. In Figure 12, a correlation surface 600 is examined along two perpendicular directions A-A and B-B. Figure 13a illustrates a crosssection through the correlation surface along the direction A-A and Figure 13b illustrates a cross-section through the correlation surface along the direction B-B.

The differential (rate of change) of the correlation surface is evaluated along the two directions A-A and B-B. As shown in Figures 13a and 13b, where the differential changes sign (indicating a localised minimum) along both of the two perpendicular directions A-A and B-B, an allowable localised minimum is indicated and a motion vector is generated from that localised minimum.

d) As a modification of technique (c) motion vectors are generated from localised minima only if the differential either side of the localised minimum is greater than a threshold value.

Each of the motion vectors derived from a single correlation surface are passed by the vector estimator 550 to the vector reducer 560. The vector reducer 560 then selects four motion vectors to be assigned to each block of the output image, in a similar manner to the vector reducer 420 described above. The four motion vectors for each block are selected from (in the following order of preference): (i) the zero motion vector; (ii) the local motion vector derived from the most significant minimum in the correction surface for that block; (iii) local motion vectors derived from secondary minima in that correlation surface; and (iv) global motion vectors.

Figure 14 is a schematic block diagram of a modified motion processor 385" according to a second embodiment of the invention. A modified correlation surface processor 610 effects 'growing' of the interpolated correlation surfaces by combining each interpolated correlation surface with groups of other interpolated correlation surfaces in a predetermined spatial relationship to that correlation surface. Various ones of the groups of correlation surfaces are illustrated in Figures 4a to 4f as described above.

Each of the six 'grown' correlation surfaces output by the correlation surface processor is passed to the vector estimator 620.

In contrast to the previously proposed apparatus illustrated in Figure 5, the vector estimator 620 generates a motion vector from each of the grown correlation surfaces and passes those motion vectors to a vector reducer 630.

The vector reducer receives the motion vectors derived from the various grown correlation surfaces and selects four motion vectors for each block to be passed to vector selection. These four motion vectors are selected from (in the following order of preference): (i) the zero motion vector; (ii) the motion vector generated from a selected combined correlation surface (e.g. the grown correlation surface of Figure 4f); (iii) motion vectors having at least a predetermined number of occurrences amongst those motion vectors generated from the original correlation surfaces derived from the current block and adjacent blocks; and (iv) the global motion vectors.

In another embodiment, the motion vectors derived from all of the combined correlation surfaces could be selected in preference to the global motion vectors.

The motion vectors are tested using the confidence test described above. Only those motion vectors which passed the confidence test are selected to be supplied to vector selection.

Figure 15 is a schematic block diagram of the part of the correlation surface processor 610 in which the 'growing' of correlation surfaces is carried out.

In the apparatus of Figure 15, input data 640 representing the interpolated correlation surfaces are supplied in serial form to a number of correlation surface delays 650. Each correlation surface delay 650 delays the input data 640 by a period equivalent to the transmission time of the data representing one correlation surface.

This means that the data at the input and the output of each of the correlation surface delays 650 represent identical points within two adjacent correlation surfaces.

The input data 640 are also supplied to two row delays 660, 670, which delay the data by a period equivalent to the transmission time of a complete row of correlations surfaces. This means that the data at the input and the output of a row delay (660 or 670) represent identical points within two correlation surfaces at corresponding positions in two adjacent rows. The output of each of the row delays 660, 670 is supplied to a series of four correlation surface delays 650.

The input and output of each correlation surface delay 650 is annotated by a lower case letter from a to o. The data at these points represent respective parts of a 5 x 3 array of correlation surfaces in the form shown in Figure 3. Accordingly, in order to generate one of the grown correlation surfaces illustrated in Figures 4a to 4f, the data and the appropriate outputs from the apparatus of Figure 15 is added and normalised.

An example of the use of a normalising adder 680 is shown schematically in Figure 16, in which the grown correlation surface illustrated in Figure 4d is generated. In particular, the data at the outputs c, h and m of the apparatus of Figure 15 are supplied to the normalising adder 680. The normalising adder 680 generates output data according to the following function: CSout = (c+h+m)/3 The correlation surface in Figure 4a (which is simply the current correlation surface) does not require a normalising adder of the type shown in Figure 16. However, normalising adders similar to that shown in Figure 16 are used to generate the remaining grown correlation surfaces of Figures 4b to 4f.

Figure 17 is a schematic block diagram of part of the motion processor 385''' according to a third embodiment of the invention. In this embodiment, interpolated correlation surfaces are supplied from the correlation surface processor 400 in parallel to the vector estimator 410 and to a modified vector reducer 700. Vector estimation is carried out in the manner described above with reference to Figure 7.

In the motion vector reducer 700, each correlation surface is inspected at the position of the global vectors. A differential test (of the type described above with reference to Figures 12, 13a and 13b) is performed to detect whether a localised minimum exists at that position. If such a localised minimum exists, then that global vector is allowed to be used for the current block of the output image.

The effect of this modified vector reduction process is that a global vector is only passed to vector selection for a particular block if there is some evidence that the global vector represents the motion of at least a small part of that block.

Figure 18 is a representation of the image shown in Figure 6, showing those blocks 710 of the image for which evidence of a global vector representing the motion of the car was found. Use of that global vector is restricted to the blocks 710, which helps to avoid the erroneous use of the global vector described with reference to Figure 6.

Figures 19 and 20 illustrate the test performed by the vector reducer 700 to examine the correlation surface for evidence of each global vector. Figure 19 illustrates a correlation surface 720 on which the predominant minimum (corresponding to the local vector) has coordinates (Vl(x) Vl(y)). At the position of a global motion vector, namely (vg(x) Vg(y)) the correlation surface 720 is examined along two perpendicular directions to detect evidence of a localised minimum.

Figure 20 is a cross-section through the correlation surface 720 passing through the predominant minimum at (Vl(X), Vl(y)) and the position of the global vector, namely (vg(x) Vg(y)) In this case, a localised minimum is detected at the position of the global vector, and the global vector is passed for possible use in interpolation of pixels of the current block of the output image.

If desired, the apparatus of Figures 10, 14 and 17 may be combined in various combinations so that, for example, the vector estimator derives one or more motion vectors from each grown correlation surface, with correlation surface re-inspection being also employed.

Claims (39)

1. Motion compensated video signal processing apparatus in which motion vectors are generated for use in interpolation of an output image of an output video signal, the motion vectors representing image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; means for examining the correlation surfaces to detect points of maximum correlation within the correlation surfaces; and means for generating motion vectors from the detected points of maximum correlation in the correlation surfaces; in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface.

2. Apparatus according to claim 1, in which the means for generating motion vectors is operable to generate up to a predetermined number of motion vectors from each correlation surface.

3. Apparatus according to claim 2, in which, in the case of a correlation surface having greater than the predetermined number of local correlation maxima, the means for generating motion vectors is operable to generate a respective motion vector from the predetermined number of most significant local correlation maxima.

4. Apparatus according to any of claims 1 to 3, comprising: means for performing a confidence test on each motion vector generated from a secondary correlation maximum, a secondary correlation maximum being a point of local maximum correlation in a correlation surface other than the point of absolute maximum correlation in that correlation surface; and means for preventing the use, in interpolation of the output image, of motion vectors derived from secondary correlation maxima and which failed the confidence test.

5. Apparatus according to claim 4, in which the means for performing a confidence test comprises: means for detecting whether each motion vector generated from a secondary correlation maximum lies within a predetermined threshold amount of one or more motion vectors derived from correlation surfaces generated from adjacent blocks of the input image.

6. Apparatus according to claim 4, comprising means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the motion vectors.

7. Apparatus according to claim 6, in which the means for performing a confidence test comprises means for detecting whether each motion vector generated from a secondary correlation maximum lies within a predetermined threshold amount of a global motion vector.

8. Apparatus according to claim 4, in which the means for performing a confidence test comprises: means for detecting a change in sign of the rate of change of a correlation surface in a predetermined number of non-parallel linear directions passing through a secondary correlation maximum.

9. Apparatus according to claim 8, in which the predetermined number of non parallel linear directions comprises a direction indicative of horizontal image motion and a direction indicative of vertical image motion.

10. Apparatus according to claim 8 or claim 9, in which the means for performing a confidence test comprises means for detecting whether the rate of change of the correlation surface adjacent to a secondary correlation maximum exceeds a predetermined amount.

11. Apparatus according to claim 6, comprising means for assigning to each of the blocks a group of motion vectors selected from, in the following order of preference: (i) a motion vector indicative of zero image motion; (ii) the motion vector generated from a point of absolute maximum correlation in the correlation surface derived from that block; (iii) motion vectors generated from secondary correlation maxima in the correlation surface derived from that block; and (iv) the global motion vectors.

12. A method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; examining the correlation surfaces to detect points of maximum correlation within the correlation surfaces; and generating motion vectors from the detected points of maximum correlation in the correlation surfaces; in which, for a correlation surface having a plurality of points of local maximum correlation, the step of generating motion vectors comprises generating a corresponding plurality of motion vectors from that correlation surface.

13. Motion compensated video signal processing apparatus in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of original correlation surfaces, each representing correlation between a respective search block of one input image of the pair and a search area, comprising a plurality of blocks, in the other image of the pair; means for combining each original correlation surface with groups of original correlation surfaces generated from surrounding search blocks, to generate a respective plurality of combined correlation surfaces; and means for generating a motion vector from each of the original correlation surfaces and the combined correlation surfaces.

14. Apparatus according to claim 13, in which the means for combining is operable to combine each original correlation surface with five groups of original correlation surfaces generated from search blocks in a surrounding array of search blocks.

15. Apparatus according to claim 14, in which the array is a 5 x 3 array of search blocks.

16. Apparatus according to any one of claims 13 to 15, in which the means for combining comprises: means for receiving serial data representing successive original correlation surfaces; a plurality of delay circuits operable to generate respective delayed data from the serial data; and means for adding the delayed data generated by a predetermined group of the delay circuits, to generate output data representing a combined correlation surface.

17. Apparatus according to claim 16, in which the means for adding comprises means for performing normalised addition of the delayed data.

18. Apparatus according to any one of claims 13 to 17, comprising means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the motion vectors.

19. Apparatus according to claim 18, comprising means for assigning to each of the blocks a group of motion vectors selected from, in the following order of preference: (i) a motion vector indicative of zero image motion; (ii) the motion vector generated from a selected combined correlation surface; (iii) motion vectors having at least a predetermined number of occurrences amongst those motion vectors generated from the original correlation surfaces derived from the current block and adjacent blocks; and (iv) the global motion vectors.

20. Apparatus according to claim 14 and claim 19, in which the selected combined correlation surface is a combined correlation surface generated by combining original correlation surfaces generated from each of the array of search blocks.

21. Apparatus according to any one of claims 13 to 20, comprising: means for examining the correlation surface to detect points of maximum correlation within the original and combined correlation surfaces; and means for generating motion vectors from the detected points of maximum correlation; in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface.

22. Apparatus according to claim 21, comprising: means for performing a confidence test on each motion vector; and means for preventing the assignment of motion vectors which failed the confidence test.

23. A method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of original correlation surfaces, each representing correlation between a respective search block of one input image of the pair and a search area, comprising a plurality of blocks, in the other image of the pair; combining each original correlation surface with groups of original correlation surfaces generated from surrounding search blocks, to generate a respective plurality of combined correlation surfaces; and generating a motion vector from each of the original correlation surfaces and the combined correlation surfaces.

24. Motion compensated video signal processing apparatus in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the apparatus comprising: means for generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; means for generating a plurality of motion vectors from the correlation surfaces, each motion vector depending on a point of maximum correlation in that correlation surface; means for deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the plurality of motion vectors;; means for examining the correlation surfaces to detect those global motion vectors which are locally valid for each block, by detecting whether the correlation surface generated from that block has a local correlation maximum indicative of image motion substantially identical to that represented by each global motion vector; and means for assigning to each of the blocks a group of motion vectors selected from: the zero motion vector; the motion vector representing image motion of that block; and those global motion vectors which are locally valid for that block.

25. Apparatus according to claim 24, in which the detecting means comprises: means for detecting a change in sign of the rate of change of a correlation surface in a predetermined number of non-parallel linear directions passing through a position indicative of image motion represented by a global motion vector.

26. Apparatus according to claim 25, in which the predetermined number of non parallel linear directions comprises a direction indicative of horizontal image motion and a direction indicative of vertical image motion.

27. Apparatus according to any one of claims 24 to 26, comprising means for examining the correlation surface to detect points of maximum correlation within the correlation surfaces; and in which, for a correlation surface having a plurality of points of local maximum correlation, the means for generating motion vectors is operable to generate a corresponding plurality of motion vectors from that correlation surface.

28. Apparatus according to any one of claims 24 to 27, comprising: means for combining each correlation surface with groups of correlation surfaces generated from surrounding blocks, to generate a respective plurality of combined correlation surfaces; and means for generating a respective motion vector from each of the combined correlation surfaces.

29. A method of motion compensated video signal processing, in which motion vectors are generated to represent image motion between a pair of input images of an input video signal, the method comprising the steps of: generating a plurality of correlation surfaces, each representing correlation between a respective block of one input image of the pair and an area, comprising a plurality of blocks, in the other image of the pair; generating a plurality of motion vectors from the correlation surfaces, each motion vector depending on a point of maximum correlation in that correlation surface; deriving a set of global motion vectors comprising a plurality of distinct motion vectors selected from the most common of the plurality of motion vectors;; detecting those global motion vectors which are locally valid for each block, by detecting whether the correlation surface generated from that block has a local correlation maximum indicative of image motion substantially identical to that represented by each global motion vector; and assigning to each of the blocks a group of motion vectors selected from: the zero motion vector; the motion vector representing image motion of that block; and those global motion vectors which are locally valid for that block.

30. Television standards conversion apparatus comprising apparatus according to any one of claims 1 to 11, 13 to 22 or 24 to 28.

31. Motion compensated video signal processing apparatus substantially as hereinbefore described with reference to the accompanying Figures 10, 11, 12, 13a and 13b.

32. A method of motion compensated video signal processing, the method being substantially as hereinbefore described with reference to the accompanying Figures 10, 11, 12, 13a and 13b.

33. Television standards conversion apparatus substantially as hereinbefore described with reference to the accompanying Figures 10, 11, 12, 13a and 13b.

34. Motion compensated video signal processing apparatus substantially as hereinbefore described with reference to the accompanying Figures 14, 15 and 16.

35. A method of motion compensated video signal processing, the method being substantially as hereinbefore described with reference to the accompanying Figures 14, 15 and 16.

36. Television standards conversion apparatus substantially as hereinbefore described with reference to the accompanying Figures 14, 15 and 16.

37. Motion compensated video signal processing apparatus substantially as hereinbefore described with reference to the accompanying Figures 17, 18, 19 and 20.

38. A method of motion compensated video signal processing, the method being substantially as hereinbefore described with reference to the accompanying Figures 17, 18, 19 and 20.

39. Television standards conversion apparatus substantially as hereinbefore described with reference to the accompanying Figures 17, 18, 19 and 20.