GB2263602A - Motion compensated video signal processing - Google Patents
Motion compensated video signal processing Download PDFInfo
- Publication number
- GB2263602A GB2263602A GB9201613A GB9201613A GB2263602A GB 2263602 A GB2263602 A GB 2263602A GB 9201613 A GB9201613 A GB 9201613A GB 9201613 A GB9201613 A GB 9201613A GB 2263602 A GB2263602 A GB 2263602A
- Authority
- GB
- United Kingdom
- Prior art keywords
- video signal
- motion vectors
- input images
- input
- scene change
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
- H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/142—Detection of scene cut or scene change
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/179—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scene or a shot
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/87—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving scene cut or scene change detection in combination with video compression
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/144—Movement detection
- H04N5/145—Movement estimation
Abstract
Video signal processing apparatus for interpolating output images of an output video signal from corresponding pairs of temporally adjacent input images of an input video signal is described. The apparatus comprises a motion vector generator for generating sets of motion vectors representing motion between respective pairs of temporally adjacent input images; an analyzer 70 for performing a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between the pair of temporally adjacent input images from which that set of motion vectors was generated and for detecting a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal; and a motion compensated interpolator 99 for interpolating an output image from a corresponding pair of tempo rally adjacent input images using the sets of motion vectors generated from that pair of input images when a possible scene change is not indicated, and for generating an output image by intra-image processing of one of the corresponding pair of input images when a possible scene change is indicated. <IMAGE>
Description
MOTION COMPENSATED VIDEO SIGNAL PROCESSING
This invention relates to the field of motion compensated video signal processing.
Motion compensated video signal processing is used in video processing applications such as television standards conversion or video to film conversion. An example of a previously proposed motion compensated video processing apparatus is described in the British
Published Patent Application number GB-A-2 231 749, in which pairs of temporally adjacent images (fields or frames) of an input digital video signal are processed to generate corresponding sets of motion vectors.
The processing is carried out on discrete blocks of the images. so that each motion vector represents the inter-image motion of the contents of a respective block. After further processing, the set of motion vectors is supplied to a motion compensated interpolator which interpolates an output image from the pair of input images, taking into account the motion between the input images.
A problem with this previously proposed apparatus, and which occurs generally in motion compensated video signal processing, is that in order to drive the signal processing accurately, the information generated about the motion of objects in the image (ie the set of motion vectors) must itself be accurate. A particular difficulty is that scene changes, and indeed cuts, wipes, fades and some other effects, can appear to the motion vector estimator as rapid random changes in motion. Also, an output image interpolated across a scene change can contain parts of two unrelated images. This can cause unpleasant artifacts in the output images, which may be even more noticeable if any post-processing is applied to the video signal.
This invention provides a video signal processing apparatus for interpolating output images of an output video signal from corresponding pairs of temporally adjacent input images of an input video signal, the video signal processing apparatus comprising:
a motion vector generator for generating sets of motion vectors representing motion between respective pairs of temporally adjacent input images;
an analyzer for performing a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between the pair of temporally adjacent input images from which that set of motion vectors was generated and for detecting a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal; and
a motion compensated interpolator for interpolating an output image from a corresponding pair of temporally adjacent input images using the sets of motion vectors generated from that pair of input images when a possible scene change is not indicated, and for generating an output image by intra-image processing of one of the corresponding pair of input images when a possible scene change is indicated.
The invention provides for the detection of possible scene changes from the sets of motion vectors themselves, by performing a predetermined test on each set of motion vectors and checking for a sudden change in the test result. When a possible scene change is detected the output image is made to be dependent on only one input image, so avoiding the possibility of the output image including parts of input images from two different scenes.
Using the invention, video signal processing apparatus can operate in real time on 'live' video signals for which there is no advance information (such as an edit list) to indicate the temporal position of any scene changes.
The detection of possible scene changes is based on the detection of a change in a test result indicative of the degree of correlation between a pair of input images. If a scene change occurs which is undetectable by this technique then this means that by chance the images before and after the scene change are roughly correlated with one another, so this scene change would not be expected to have a detrimental effect on the operation of the interpolator. In other words, the fact that this scene change was not detected does not matter. Similarly, if the input video signal did not contain a scene change but was such that there was a sudden change in the test results obtained for successive sets of motion vectors. this event would be indicated as a possible scene change by the analyzer.However, since such an event would be likely to have a detrimental effect on the operation of the interpolator, this 'false positive' detection is advantageous as appropriate action (changing to intra-frame generation of that output image) can be taken.
If a scene change occurs between two temporally adjacent input images from which neither a set of motion vectors nor an output image is calculated then that scene change would not be detected by the analyzer. However, that scene change would have no effect whatsoever on the operation of the interpolator, so the fact that it goes undetected is immaterial.
Preferably the motion compensated interpolator comprises: means for receiving a control signal indicative of the temporal position of an output image to be interpolated with respect to the temporal positions of the corresponding pair of temporally adjacent input images; means responsive to the control signal for setting the relative proportions of the corresponding pair of input images to be used in interpolation of the output image in dependence on the temporal position of the output image when a possible scene change is not indicated; means for setting the relative proportions of the corresponding pair of input images to be used in interpolation of the output image so that the output image depends on the content of a selected one of the pair of input images when a possible scene change is indicated; and interpolation means for generating the output image by combining the corresponding pair of input images according to the relative proportions.
The selected one of the input images could be predetermined, but preferably the selected one of the pair of input images is the input image which is temporally closest to the output image.
Since the set of motion vectors generated at a scene change can be meaningless it is preferred that the motion compensated interpolator comprises means for setting the set of motion vectors to be used in interpolation to a set of zero motion vectors when a possible scene change is indicated.
An alternative to using a modified interpolator is to employ an image store for storing at least one of the corresponding pair of input images from which an output image is to be interpolated; and means responsive to the analyzer indicating a possible scene change for setting the output image to be equal to at least a portion of a stored input image.
In one preferred embodiment the motion vector generator comprises means for performing a predetermined confidence test on each motion vector in a set of motion vectors; and the analyzer comprises means generating a test result dependent upon the proportion of the motion vectors, in a set of motion vectors, which passed the confidence test.
In another preferred embodiment the motion vector generator comprises means for detecting a predetermined number of most commonly occurring motion vectors in a set of motion vectors; and the analyzer comprises means for generating a test result dependent upon the sum of the frequencies of occurrence of the predetermined number of most commonly occurring motion vectors.
The invention is particularly suited for use in a system in which the input images are successive video frames of the input video signal.
However, since the actual video signal supplied to the apparatus may be in interlaced form, it is preferred to employ means for receiving an interlaced video signal and a progressive scan converter for converting pairs of temporally adjacent video fields in the interlaced video signal into respective video frames of the input video signal.
The invention is particularly useful when embodied in, for example, a television standards converter comprising image processing apparatus according to any one of the preceding claims.
Viewed from a second aspect this invention provides a method of interpolating output images of an output video signal from corresponding pairs of temporally adjacent input images of an input video signal 9 the method comprising the steps of:
generating sets of motion vectors representing motion between respective pairs of temporally adjacent input images;
performing a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between that pair of temporally adjacent input images;
detecting a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal; and
interpolating an output image from a corresponding pair of temporally adjacent input images using the sets of motion vectors generated from that pair of input images when a possible scene change is not indicated, and generating an output image by intra-image processing of one of the corresponding pair of input images when a possible scene change is indicated.
Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings throughout which like parts are referred to by like references, and in which:
Figure 1 is a block diagram of a previously proposed motion compensated television standards conversion apparatus;
Figure 2 is a schematic diagram showing the operation of the apparatus of Figure 1;
Figure 3 is a schematic diagram showing the operation of the interpolator in Figure 1;
Figure 4 shows one embodiment of an analyzer for detecting possible scene changes in an input video signal;
Figure 5 shows another embodiment of an analyzer for detecting possible scene changes in an input video signal;
Figure 6 shows a modified interpolator for performing intra-frame generation of an output image when a possible scene change is detected;;
Figures 7 and 8 are schematic diagrams illustrating the operation of the interpolator of Figure 6; and
Figures 9 and 10 show alternative modifications to the apparatus of Figure 1 for performing intra-frame generation of an output image when a possible scene change is detected.
Referring now to Figure 1, a block diagram of a previously proposed motion compensated television standards conversion apparatus for converting an input interlaced high definition video signal (HDVS) 8 having 1125 lines per frame and 60 fields per second into an output interlaced video signal 40 having 625 lines per frame and 50 fields per second is shown. The input video signal 8 is first supplied to a progressive scan converter 10 in which the input video fields are converted into video frames at the same rate (60 Hz) as the input video fields. These frames are then supplied to a down converter 12 which reduces the number of lines in each of the frames to 625, the number of lines in a frame of the output video signal 40.The down-converted input frames are then passed to a time base changer 14 which also receives as inputs a 60Hz clocking signal 32 locked to the field frequency of the input video signal 8 and a 50Hz clocking signal 34 locked to the required field frequency of the output video signal 40.
The time base changer 14 determines the temporal position of each field of the output video signal and selects two of the down-converted frames of the input video signal 8 to be used in interpolating that field of the output video signal. The two down-converted input frames selected by the time base changer are supplied on respective outputs 16, with a third output 18 of the time base changer carrying control information.
The video signals corresponding to the two down-converted input frames selected by the time base changer 14 are supplied as inputs to a motion processor 20. The motion processor comprises two substantially identical processing sections: one for even output fields and one for odd output fields. The two down-converted input frames selected by the time base changer 14 are routed to the even field or the odd field processing section as appropriate by means of a switch 21 under the control of a signal (not shown) derived from the 50 Hz clocking signal 34.
In the appropriate section (odd or even) of the motion processor 20, the down-converted input frames are supplied first to a direct block matcher 22 which calculates correlation surfaces representing the spatial correlation between blocks of the two frames. These correlation surfaces are passed to a motion vector estimator 24 which processes the correlation surfaces to generate a set of motion vectors which are supplied to a motion vector reducer 26. The motion vector estimator also performs a confidence test on each generated motion vector to establish whether that motion vector is significant above the general noise level, and associates a confidence flag with each motion vector indicative of the result of the confidence test.The confidence test itself comprises a 'threshold test' and a 'rings test' and is described, along with other features of the apparatus shown in Figure 1, in more detail in the British Published Patent Application number
GB-A-2 231 749.
The motion vector reducer 26 operates to reduce the choice of possible motion vectors for each pixel in each block, before the motion vectors are supplied to a motion vector selector 28. As part of its function the motion vector reducer 26 counts the frequencies of occurrence of the 'good' motion vectors (ie motion vectors which passed the confidence test), with no account taken of the position of the blocks of the input images used to obtain those motion vectors. The good motion vectors are then ranked in order of decreasing frequency.
The three most common of the good motion vectors which are significantly different to one another are then classed as 'global' motion vectors. Three motion vectors which passed the confidence test are then selected for each block and are supplied, with the zero motion vector, to the motion vector selector 28 for further processing. These three selected motion vectors are selected in a predetermined order of preference from the motion vector generated from that block, those generated from the surrounding blocks, and finally the global motion vectors.
The motion vector selector 28 also receives as inputs the two down-converted input frames selected by the time base changer 14 and which were used to calculate the motion vectors (suitably delayed by a system delay compensator 30) and supplies an output comprising one motion vector per pixel of the output field. This motion vector is selected from the four motion vectors for that block supplied by the motion vector reducer 26. Any irregularity in the selection of the motion vectors by the motion vector selector 28 is removed by a motion vector post processor 36, from which the processed motion vectors are supplied to and control an interpolator 38 which also receives the appropriate odd or even pair of down-converted input frames selected by the time base changer, again suitably delayed by the system delay compensator 30.Using the motion vectors, the interpolator 38 interpolates an output field from the two down-converted input frames selected by the time base changer, taking into account any image motion between the two frames. The two down-converted input frames are combined in relative proportions depending on the temporal position of the output field with respect to the two frames, so that a larger proportion of the nearer input image is used. The output 40 of the interpolator 38 is an interlaced video signal having 625 lines per frame and 50 fields per second.
Figure 2 illustrates the way in which pairs of temporally adjacent down-converted frames of the input video signal 8 are selected by the time base changer 14 for use in interpolating a corresponding odd or even field of the output video signal 40. The upper row in
Figure 2 represents the down-converted frames of the input video signal 8. As mentioned above, the input video signal has a field rate of 60Hz and the output of the progressive scan converter 10 is a series of video frames at the same frequency as the field frequency of the input video signal. In order to clarify Figure 2 and so that the downconverted input frames used to interpolate each output field can be identified, each of the down-converted input frames is denoted by a number relating to the corresponding input field.For example, a downconverted input frame 42 is denoted by the number 5+, showing that it originated from an odd field in a fifth frame of the input video signal 8 (before progressive scan conversion). Similarly another downconverted input frame 43 is denoted by the number 8, indicating that it originated from an even field in the 8th frame of the input video signal.
The output video signal has a field frequency of 50Hz, so the temporal separation of adjacent fields in the output video signal is greater than the temporal separation of adjacent down-converted input frames in the input video signal. The time base changer 14 is operable to select the two down-converted input frames closest to the temporal position of an output field to be used in interpolating that output field. For example an (even) output field 44 is interpolated from the two down-converted input frames temporally closest to that output field (those denoted by the numbers 5 and 5+) using motion vectors representing image motion between the same two frames (ie those denoted by the numbers 5 and 5+).
Figure 3 is a schematic illustration showing the operation of the interpolator 38, in which an output field 50 is interpolated from a corresponding pair of input frames (a preceding input frame 52 and a following input frame 54). As mentioned above, each pixel 56 in the output field 50 has associated with it a motion vector 58, selected by the motion vector selector 28. The motion vector 58 is used by the interpolator 38 to select a portion 60 of the preceding input frame 52 and a portion 62 of the following input frame 54 to be used in deriving the pixel 56 in the output field 50. The position of the portion 60 in the preceding input frame 52 is displaced from the position of the pixel 56 in the output field 50 by an amount dependent upon the motion vector 58 and the temporal displacement of the output field 50 from the preceding input frame 52.Similarly the position of the portion 62 in the following input frame 54 depends upon the motion vector 58 and the temporal displacement of the output field 50 from the following input frame 54. A filtering process is used so that the portions 60 and 62 on which the pixel 56 depends comprises a group of pixels in the respective input frame 52, 54.
The interpolation process described above relies on the assumption that the portion 60, the pixel 56, and the portion 62 all relate to the same part of an image, even though that part has moved between the preceding input frame 52 and the following input frame 54.
This assumption can fail for two reasons when a scene change or another dynamic video effect occurs between the preceding input frame 52 and the following input frame 54.
One reason that the motion compensated interpolation process shown in Figure 3 may fall down at a scene change is that the scene change itself may appear to the motion vector estimator as a large amount of rapid image motion, leading to the generation of spurious and potentially large motion vectors. This will result in errors in the calculation of the respective positions of the portion 60 and the portion 62. Furthermore, whatever the positions of the portions 60 and 62 are calculated to be, the actual image content of the portion 60 may be completely unrelated to that of the portion 62, so an interpolation between the portion 60 and the portion 62 will be meaningless.
Two different analyzers for use with the apparatus of Figure 1 and which are capable of detecting a possible scene change in the input video signal will now be described with reference to Figures 4 and 5.
Suitable modifications to the apparatus shown in Figure 1, so that the output image is derived by intra-frame processing (rather than interframe processing) when a possible scene change is detected will then be described with reference to Figures 6 to 10.
Figure 4 is a schematic diagram of an analyzer 70 for detecting a scene change between adjacent input frames of the input video signal.
The analyzer 70 performs a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between the pair of input images used to derive that set of motion vectors and detects a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal.
The predetermined test performed by the analyzer of Figure 4 is to analyze the confidence flags generated by the motion vector estimator 24 and associated with each of the motion vectors in a set of motion vectors. These motion vector confidence flags are received on an input 72 and are passed to a vector pass rate calculator 74. The vector pass rate calculator 74 calculates the number of motion vectors in the set of motion vectors which passed the confidence test, as a proportion of the total number of motion vectors in that set. A test result calculated by the vector pass rate calculator 74 is supplied to a difference calculator 78 and to a latch/delay unit 76 where the test result is temporarily stored.
The difference calculator 78 receives the test result calculated by the vector pass rate calculator 74 corresponding to the set of motion vector confidence flags currently at the input 72 of the analyzer 70, along with the test result stored in the latch/delay unit 76 corresponding to the previous set of motion vector confidence flags supplied to the analyzer 70. The difference calculator 78 calculates the numerical difference between the test result received from the vector pass rate calculator 74 and the previous test result received from the latch/delay unit 76, passing a difference value to a comparator 80.
The comparator 80 compares the difference value supplied to it by the difference calculator 78 with a threshold value stored in a threshold register 82. If the comparator 80 detects a drop in the test results (the proportion of motion vectors in a set which passed the confidence test) corresponding to two successive sets of motion vectors, it supplies an output 84 indicating that there has been a possible scene change in the input video signal.
The output 84 indicates a possible, rather than a definite, scene change for the following reasons. If a scene change occurred in which there was a high degree of correlation between the input images before and after the scene change, enabling successive sets of motion vectors to be generated for which a large proportion passed the confidence test, then it is likely that the interpolator 38 would continue to operate satisfactorily in spite of the scene change. In this case the output 84 would not indicate a possible scene change, but this failure to detect the scene change would not have a detrimental effect on the operation of the interpolator 38.Conversely, if a scene change did not actually occur but the nature of the input video signal was such that the test result calculated by the vector pass rate calculator 74 (reflecting the degree of correlation between the pair of input image used to generate a set of motion vectors) dropped by more than the threshold value between successive sets of motion vectors, then it is likely that the operation of the interpolator 38 would be detrimentally affected even though, strictly speaking, there had not been a scene change in the input video signal. Such an occurrence would be indicated as a possible scene change on the output 84 and appropriate action could be taken.
It may be necessary to apply a delay to the output 84 from the analyzer, in order that the output indicating whether or not a possible scene change has been detected between two input frames is relevant to the pair of input frames currently being supplied to the interpolator from the system delay compensator.
The threshold value stored in the threshold register 82 may be changed, automatically or by an operator, using an input 86 to the analyzer 70. Such changes may be made necessary by the nature of a particular input video signal, in order to tune the detection of possible scene changes performed by the analyzer to that input video signal.
Figure 5 is a schematic illustration of a second embodiment of an analyzer 90 which operates in a similar manner to the analyzer 70 described above to produce an output 84 indicating a possible scene change in an input video signal. The analyzer 90 receives a set of inputs 92 representing the frequencies of occurrence of the three global motion vectors selected by the motion vector reducer 26. For each set of motion vectors a summation unit 94 sums the global vector occurrence frequencies received on the inputs 92, passing this sum as a test result to a difference calculator 78 and to a latch/delay unit 76, both of which are similar to those described above with reference to Figure 4.
The difference calculator calculates the numerical difference between the test result received from the summation unit 94 representing the sum of the global vector occurrence frequencies for a current set of motion vectors and a previous test result received from the latch/delay unit 76 representing that sum for the immediately preceding set of motion vectors. The difference calculator supplies a difference value to a comparator 80 in which the difference value is compared with a threshold value stored in a threshold register 82. If a drop of more than the threshold is detected in the test results corresponding to two successive sets of motion vectors, the output 84 of the analyzer 90 indicates a possible scene change in the input video signal. As above, the threshold value may be varied by means of an input 86 to the threshold register 82.
A combination strategy for detecting possible scene changes could be used in which the outputs 84 from an analyzer 70 and an analyzer 90 are combined using a logical OR gate.
Figure 6 shows one way in which the operation of the interpolator can be modified when a possible scene change is detected by the analyzer 70 or the analyzer 90.
A modified interpolator 99 is shown, which receives a pair of input frames from the system delay compensator at the same time as it receives a set of motion vectors from the motion vector post-processor 36. The set of motion vectors received is that set generated from the two input frames currently being supplied to the interpolator 99, and is stored in a motion vector store 100. The interpolator 99 also receives control information from the time base changer (suitably delayed by the system delay compensator 30) indicative of the temporal position of the required output field with respect to the pair of input frames.
For normal operation (ie when a possible scene change has not been indicated by the analyzer) two switches 101 and 104 in the interpolator are set to "normal" positions 106 and 108 respectively.
An interpolation unit 109 interpolates the output field as described above with reference to Figure 3, in that each pixel 56 in the output field 50 is derived from a portion 60 of the preceding input frame 52 and a portion 62 of the following input frame 54, the position of the portions 60, 62 being dependent upon the motion vector 58 corresponding to the pixel 56 and upon the temporal position of the output field 50 with respect to the preceding input frame 52 and the following input frame 54.
The interpolator also receives an input 110 from the analyzer on which a possible scene change can be indicated. When a possible scene change is indicated the switches 102 and 104 are set to "scene change" positions 112 and 114 respectively. This has two effects on the operation of the interpolator 99: one effect is that the interpolation unit 109 uses a set of zero motion vectors stored in a zero motion vector store 116 instead of the set stored in the motion vector store 100, and another effect is that the output field position is substituted by a value calculated by a single nearest frame calculator 118.
As far as the interpolation unit 109 is concerned, when a possible scene change is indicated on the input 110 the interpolation operation proceeds as normal to produce a field of the output video signal. However, this operation is carried out using a set of motion vectors from the zero motion vector store 116, all of which are the zero motion vector, and also using a substituted output field position calculated by the single nearest frame calculator 118 which specifies to the interpolation unit 109 that the output field is temporally coincident with the nearer of the preceding input frame and the following input frame.
Figures 7 and 8 show the operation of the modified interpolator shown in Figure 6 when a possible scene change is detected by the analyzer (70 or 90) and indicated to the interpolator on the input 110.
Figure 7 shows the situation when the temporal position of the output field 50 is closer to that of the preceding input frame 52 than it is to that of the following input frame 54. Conversely, Figure 8 shows the situation when the temporal position of the output field 50 is closer to that of the following input frame 54 than to the position of the preceding input frame 52.
when a possible scene change is indicated on the input 110 to the interpolator 99 in Figure 6, the switch 102 selects the "scene change" position 112, causing a set of zero motion vectors to be used. This is illustrated in Figures 7 and 8 in which a zero motion vector 120 is used for derivation of the output field 50. Also, when a possible scene change is indicated on the input 110 to the interpolator 99 in
Figure 6, the switch 104 selects the "scene change" position 114. The effect of this, as mentioned above, is that instead of receiving a signal indicative of the temporal position of the output field with respect to the preceding input frame 52 and the following input frame 54, the interpolation unit 109 receives a signal indicating that the output field 50 is temporally coincident with the nearer of the preceding input frame 52 and the following input frame 54.This has the effect that an apparently normal interpolation process can be performed, but the output field produced by the interpolation unit is dependent entirely on the nearer input frame specified by the single nearest frame calculator 118 and not dependent at all on the other of the two input frames.
In Figure 7 the output field 50 is closer to the preceding input frame 52 than to the following input frame 54, so each pixel 56 in the output field 50 is dependent only upon a portion 60 of the preceding input frame. The position of the portion 60 in the preceding input frame 52 corresponds to the position of the pixel 56 in the output field 50 because a zero motion vector 120 is used. Similarly, in
Figure 8 the output field 50 is temporally closer to the following input frame 54 than to the preceding input frame 52, so each pixel 56 in the output field is dependent only upon a portion 62 in the following input frame 54. Again, the portion 62 is at a position in the following input frame 54 corresponding to the position of the pixel 56 in the output field 50 because a zero motion vector 120 is used.
Figures 9 and 10 are schematic diagrams illustrating a second way in which the apparatus shown in Figure 1 can be modified so as to change the way in which output fields are derived when a possible scene change is detected by the analyzer (70 or 90).
In Figure 9 the pair of input frames selected by the time base changer 14 for use in the derivation of an output field are suitably delayed by the system delay compensator 30 and are routed to the interpolator 38 and to two respective frame stores, namely a preceding frame store 122 and a following frame store 124. A switch 126 selects between the outputs of the interpolator 38, the preceding frame store 122 and the following frame store 124, under the control of a control signal 128. The control signal 128 is calculated by a single nearest frame calculator 130 which is responsive to an input from the analyzer indicating whether a possible scene change in the input video signal has been detected, and to an input indicative of the temporal position of the current output field with respect to the pair of input frames.
When a possible scene change is not indicated the control signal 128 controls the switch 126 to select the output of the interpolator 38.
When a possible scene change is indicated, the single nearest frame calculator determines whether the preceding input frame or the
following input frame is closer to the temporal position of the required output field and causes the switch 126 to select the output of
the preceding frame store 122 or the following frame store 124 as appropriate. Depending on whether the current output field is an odd or an even output field, the contents of the selected frame store are subsampled (with or without a filtering process) so that only the odd or the even lines in the selected frame store are used to form the 'output field.
A simpler embodiment is shown in Figure 10, in which a single
frame store 130 stores a predetermined one of the input frames. A
switch 132 selects between the output of the frame store and the output of the interpolator 38, in response to a control signal indicating whether a possible scene change has been detected by the analyzer 70.
In operation, the switch 132 simply selects the output of the
interpolator 38 when a possible scene change has not been detected and selects the output of the frame store 130 when a possible scene change has been detected. As before, the output field is a subsampled version of the contents of the frame store 130.
The apparatus shown in Figures 9 and 10 may be used in situations
such as video to film conversion when the required interpolated output
images are video frames rather than interlaced video fields, by supplying all of the lines of image data in the appropriate frame store
122, 124 or 130 as an output image.
Although the embodiments of the invention described above have
related to television standards conversion, the techniques used are equally applicable to other forms of motion compensated video signal processing. For example, in a film-to-film standards conversion apparatus, an input telecine apparatus would be connected as an input
to, effectively, a television standards conversion apparatus which would then be connected to an output electron beam recorder for re
recording the video information on to film. All of the interpolation processes described above are suitable to be used to produce field- or
frame-based output video signals.
Claims (14)
1. Video signal processing apparatus for interpolating output images of an output video signal from corresponding pairs of temporally adjacent input images of an input video signal, the video signal processing apparatus comprising:
a motion vector generator for generating sets of motion vectors representing motion between respective pairs of temporally adjacent input images;
an analyzer for performing a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between the pair of temporally adjacent input images from which that set of motion vectors was generated and for detecting a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal; and
a motion compensated interpolator for interpolating an output image from a corresponding pair of temporally adjacent input images using the sets of motion vectors generated from that pair of input images when a possible scene change is not indicated, and for generating an output image by intra-image processing of one of the corresponding pair of input images when a possible scene change is indicated.
2. Video signal processing apparatus according to claim 1, in which the motion compensated interpolator comprises:
means for receiving a control signal indicative of the temporal position of an output image to be interpolated with respect to the temporal positions of the corresponding pair of temporally adjacent input images;
means responsive to the control signal for setting the relative proportions of the corresponding pair of input images to be used in interpolation of the output image in dependence on the temporal position of the output image when a possible scene change is not indicated;
means for setting the relative proportions of the corresponding pair of input images to be used in interpolation of the output image so that the output image depends on the content of a selected one of the pair of input images when a possible scene change is indicated; and
interpolation means for generating the output image by combining the corresponding pair of input images according to the relative proportions.
3. Video signal processing apparatus according to claim 2, in which the selected one of the pair of input images is that input image which is temporally closest to the output image.
4. Video signal processing apparatus according to claim 2 or claim 3, in which the motion compensated interpolator comprises means for setting the set of motion vectors to be used in interpolation to a set of zero motion vectors when a possible scene change is indicated.
5. Video signal processing apparatus according to claim 1, comprising:
an image store for storing at least one of the corresponding pair of input images from which an output image is to be interpolated; and
means responsive to the analyzer indicating a possible scene change for setting the output image to be equal to at least a portion of a stored input image.
6. Video signal processing apparatus according to any one of the preceding claims, in which:
the motion vector generator comprises means for performing a predetermined confidence test on each motion vector in a set of motion vectors; and
the analyzer comprises means for generating a test result dependent upon the proportion of the motion vectors, in a set of motion vectors, which passed the confidence test.
7. Video signal processing apparatus according to any one of the claims 1 to 5, in which:
the motion vector generator comprises means for detecting a predetermined number of most commonly occurring motion vectors in a set of motion vectors; and
the analyzer comprises means for generating a test result dependent upon the sum of the frequencies of occurrence of the predetermined number of most commonly occurring motion vectors.
8. Video signal processing apparatus according to any one of the preceding claims, in which the input images are successive video frames of the input video signal.
9. Video signal processing apparatus according to claim 8, comprising:
means for receiving an interlaced video signal; and
a progressive scan converter for converting pairs of temporally adjacent video fields in the interlaced video signal into respective video frames of the input video signal.
10. A television standards converter comprising image processing apparatus according to any one of the preceding claims.
11. A method of interpolating output images of an output video signal from corresponding pairs of temporally adjacent input images of an input video signal, the method comprising the steps of:
generating sets of motion vectors representing motion between respective pairs of temporally adjacent input images;
performing a predetermined test for each set of motion vectors to obtain a respective test result indicative of the degree of correlation between the pair of temporally adjacent input images from which that set of motion vectors was generated;
detecting a change of at least a predetermined size in the test results obtained for successive sets of motion vectors, thereby indicating a possible scene change in the input video signal; and
interpolating an output image from a corresponding pair of temporally adjacent input images using the sets of motion vectors generated from that pair of input images when a possible scene change is not indicated, and generating an output image by intra-image processing of one of the corresponding pair of input images when a possible scene change is indicated.
12. Video signal processing apparatus substantially as hereinbefore described with reference to Figures 3 to 10.
13. A method of image processing substantially as hereinbefore described with reference to Figures 3 to 10.
14. A television standards converter substantially as hereinbefore described with reference to Figures 3 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB9201613A GB2263602B (en) | 1992-01-24 | 1992-01-24 | Motion compensated video signal processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB9201613A GB2263602B (en) | 1992-01-24 | 1992-01-24 | Motion compensated video signal processing |
Publications (3)
| Publication Number | Publication Date |
|---|---|
| GB9201613D0 GB9201613D0 (en) | 1992-03-11 |
| GB2263602A true GB2263602A (en) | 1993-07-28 |
| GB2263602B GB2263602B (en) | 1995-06-21 |
Family
ID=10709235
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| GB9201613A Expired - Fee Related GB2263602B (en) | 1992-01-24 | 1992-01-24 | Motion compensated video signal processing |
Country Status (1)
| Country | Link |
|---|---|
| GB (1) | GB2263602B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0649256A2 (en) * | 1993-10-19 | 1995-04-19 | Canon Kabushiki Kaisha | Motion compensation of a reproduced image signal |
| GB2305569A (en) * | 1995-09-21 | 1997-04-09 | Innovision Res Ltd | Motion compensated interpolation |
| EP0883298A2 (en) * | 1997-06-04 | 1998-12-09 | Hitachi, Ltd. | Conversion apparatus for image signals and TV receiver |
| EP0710031A3 (en) * | 1994-10-26 | 1999-01-20 | Thomson Consumer Electronics, Inc. | System for coding a video signal in the presence of an image intensity gradient |
| EP0909092A2 (en) * | 1997-10-07 | 1999-04-14 | Hitachi, Ltd. | Method and apparatus for video signal conversion |
| EP1006723A2 (en) * | 1998-11-30 | 2000-06-07 | Fujitsu Limited | Apparatus and method for video frame rate conversion |
| GB2351406A (en) * | 1999-06-21 | 2000-12-27 | Sony Uk Ltd | Video data compression with scene change detection |
| EP1104970A1 (en) * | 1998-06-25 | 2001-06-06 | Hitachi, Ltd. | Method and device for converting number of frames of image signals |
| EP1134700A2 (en) * | 2000-03-14 | 2001-09-19 | Monolith Co., Ltd. | Image coding method and apparatus |
| EP1357756A1 (en) * | 2002-04-23 | 2003-10-29 | Monolith Co., Ltd. | Image coding method and apparatus, and image decoding method and apparatus |
| WO2005027526A1 (en) * | 2003-09-09 | 2005-03-24 | Apple Computer, Inc. | Video encoding method and scene cut detection method |
| EP1931145A1 (en) * | 2006-12-04 | 2008-06-11 | Hitachi, Ltd. | Frame rate conversion apparatus for video signal and display apparatus |
| EP2214409A1 (en) * | 2007-11-29 | 2010-08-04 | Panasonic Corporation | Reproduction device and reproduction method |
| US7889792B2 (en) | 2003-12-24 | 2011-02-15 | Apple Inc. | Method and system for video encoding using a variable number of B frames |
-
1992
- 1992-01-24 GB GB9201613A patent/GB2263602B/en not_active Expired - Fee Related
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6049354A (en) * | 1993-10-19 | 2000-04-11 | Canon Kabushiki Kaisha | Image shake-correcting system with selective image-shake correction |
| EP0649256A3 (en) * | 1993-10-19 | 1998-02-04 | Canon Kabushiki Kaisha | Motion compensation of a reproduced image signal |
| EP0649256A2 (en) * | 1993-10-19 | 1995-04-19 | Canon Kabushiki Kaisha | Motion compensation of a reproduced image signal |
| EP0710031A3 (en) * | 1994-10-26 | 1999-01-20 | Thomson Consumer Electronics, Inc. | System for coding a video signal in the presence of an image intensity gradient |
| CN1098599C (en) * | 1994-10-26 | 2003-01-08 | 汤姆森消费电子有限公司 | System for coding video signal in presence of image intensity gradient |
| GB2305569A (en) * | 1995-09-21 | 1997-04-09 | Innovision Res Ltd | Motion compensated interpolation |
| GB2305569B (en) * | 1995-09-21 | 1999-07-21 | Innovision Res Ltd | Motion compensated interpolation |
| EP0883298A2 (en) * | 1997-06-04 | 1998-12-09 | Hitachi, Ltd. | Conversion apparatus for image signals and TV receiver |
| EP0883298A3 (en) * | 1997-06-04 | 2000-03-29 | Hitachi, Ltd. | Conversion apparatus for image signals and TV receiver |
| EP0909092A3 (en) * | 1997-10-07 | 2000-06-07 | Hitachi, Ltd. | Method and apparatus for video signal conversion |
| EP0909092A2 (en) * | 1997-10-07 | 1999-04-14 | Hitachi, Ltd. | Method and apparatus for video signal conversion |
| EP1104970A4 (en) * | 1998-06-25 | 2005-08-03 | Hitachi Ltd | Method and device for converting number of frames of image signals |
| EP1104970A1 (en) * | 1998-06-25 | 2001-06-06 | Hitachi, Ltd. | Method and device for converting number of frames of image signals |
| EP1006723A3 (en) * | 1998-11-30 | 2001-11-14 | Fujitsu Limited | Apparatus and method for video frame rate conversion |
| US6778221B1 (en) | 1998-11-30 | 2004-08-17 | Fujitsu Limited | Apparatus and method for converting between different video formats |
| EP1006723A2 (en) * | 1998-11-30 | 2000-06-07 | Fujitsu Limited | Apparatus and method for video frame rate conversion |
| GB2351406A (en) * | 1999-06-21 | 2000-12-27 | Sony Uk Ltd | Video data compression with scene change detection |
| US7099511B2 (en) | 2000-03-14 | 2006-08-29 | Monolith Co., Ltd. | Method and apparatus for coding and decoding image data using critical points |
| EP1134700A3 (en) * | 2000-03-14 | 2004-04-28 | Monolith Co., Ltd. | Image coding method and apparatus |
| EP1134700A2 (en) * | 2000-03-14 | 2001-09-19 | Monolith Co., Ltd. | Image coding method and apparatus |
| EP1357756A1 (en) * | 2002-04-23 | 2003-10-29 | Monolith Co., Ltd. | Image coding method and apparatus, and image decoding method and apparatus |
| US7295612B2 (en) | 2003-09-09 | 2007-11-13 | Apple Inc. | Determining the number of unidirectional and bidirectional motion compensated frames to be encoded for a video sequence and detecting scene cuts in the video sequence |
| WO2005027526A1 (en) * | 2003-09-09 | 2005-03-24 | Apple Computer, Inc. | Video encoding method and scene cut detection method |
| EP1942678A1 (en) * | 2003-09-09 | 2008-07-09 | Apple Computer, Inc. | Video encoding method and scene cut detection method |
| US7856059B2 (en) | 2003-09-09 | 2010-12-21 | Apple Inc. | Determining the number of unidirectional and bidirectional motion compensated frames to be encoded for a video sequence and detecting scene cuts in the video sequence |
| US7889792B2 (en) | 2003-12-24 | 2011-02-15 | Apple Inc. | Method and system for video encoding using a variable number of B frames |
| US8130834B2 (en) | 2003-12-24 | 2012-03-06 | Apple Inc. | Method and system for video encoding using a variable number of B frames |
| EP1931145A1 (en) * | 2006-12-04 | 2008-06-11 | Hitachi, Ltd. | Frame rate conversion apparatus for video signal and display apparatus |
| US8319887B2 (en) | 2006-12-04 | 2012-11-27 | Hitachi, Ltd. | Frame rate conversion apparatus for video signal and display apparatus |
| EP2214409A1 (en) * | 2007-11-29 | 2010-08-04 | Panasonic Corporation | Reproduction device and reproduction method |
| EP2214409A4 (en) * | 2007-11-29 | 2010-12-08 | Panasonic Corp | Reproduction device and reproduction method |
Also Published As
| Publication number | Publication date |
|---|---|
| GB2263602B (en) | 1995-06-21 |
| GB9201613D0 (en) | 1992-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5347312A (en) | Motion compensated video signal processing | |
| US5631706A (en) | Converter and method for converting video signals of interlace format to video signals of progressive format | |
| US5943099A (en) | Interlaced-to-progressive conversion apparatus and method using motion and spatial correlation | |
| US5365273A (en) | Method and device for film-mode detection | |
| US6606126B1 (en) | Deinterlacing method for video signals based on motion-compensated interpolation | |
| US20050249282A1 (en) | Film-mode detection in video sequences | |
| EP0549681B2 (en) | Video image processing | |
| JPH04229795A (en) | Video system converter correcting movement | |
| GB2263602A (en) | Motion compensated video signal processing | |
| US8144248B2 (en) | Scan conversion apparatus | |
| US7023920B2 (en) | Facilitating motion estimation | |
| JP2003078926A (en) | Detector for telecine conversion method of video signal | |
| EP0561561B1 (en) | Digital video signal processing | |
| GB2277005A (en) | Detecting motion vectors in video signal processing; comparison with threshold. | |
| US5442409A (en) | Motion vector generation using interleaved subsets of correlation surface values | |
| JPH04334186A (en) | Interpolation device for scanning line | |
| GB2277004A (en) | Motion compensated video signal processing; motion/no motion flag | |
| JP2002369156A (en) | Video signal converter | |
| JP3871360B2 (en) | Video signal processing apparatus and method | |
| KR960012490B1 (en) | Video format converting apparatus and method for hdtv | |
| GB2308774A (en) | Selection of global motion vectors for video signal processing | |
| JP3121519B2 (en) | Motion interpolation method and motion interpolation circuit using motion vector, and motion vector detection method and motion vector detection circuit | |
| GB2281167A (en) | Motion compensated video signal processing | |
| JP2775688B2 (en) | Image signal processing device | |
| JP2762794B2 (en) | Scan line interpolator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 730A | Proceeding under section 30 patents act 1977 | ||
| PCNP | Patent ceased through non-payment of renewal fee |
Effective date: 20110124 |