WO2010004468A1

WO2010004468A1 - Reducing de-interlacing artifacts

Info

Publication number: WO2010004468A1
Application number: PCT/IB2009/052740
Authority: WO
Inventors: Matthijs C. Piek
Original assignee: Nxp B.V.
Priority date: 2008-07-11
Filing date: 2009-06-25
Publication date: 2010-01-14

Abstract

A video signal is processed to reduce de-interlacing artifacts. An input video signal comprises a plurality of de-interlaced frames (41, 42) which have been generated from an interlaced video signal having fields (11, 12) which occur at a first rate, the de-interlaced frames comprising an artifact (44) which occurs at a first frequency. A determination is made if motion is present between the de- interlaced frames (41, 42) of the input video signal. For at least an area of the input video signal for which there is no motion, or motion below a threshold value, frames are output at a second rate, higher than the first rate. The output frames are ordered such that the de-interlacing artifact (44) occurs at a second frequency which is higher than the first frequency. For an input video signal comprising odd frames and even frames, the step of ordering frames alternates an odd frame with an even frame at the second rate.

Description

DESCRIPTION

REDUCING DE-INTERLACING ARTIFACTS

This invention relates to processing a video signal which has been derived from an interlaced video signal.

Many types of video signal source are stored, or transmitted, in an interlaced manner. Interlacing is a technique which was used historically to reduce the amount of data which needed to be transmitted, while maintaining a high refresh rate, and is still used as part of legacy and emerging standards. An interlaced signal comprises consecutive odd and even fields. An odd field contains all odd lines of the video signal and an even field contains all even lines of the video signal.

Most Cathode ray tube (CRT) displays for television generate an image using an interlaced technique. However, many modern non-CRT displays such as Plasma Display Panels (PDPs) and Liquid Crystal Displays (LCD) require an input video signal in the form of a non-interlaced signal. In a non-interlaced signal, also known as a progressive scan signal, all of the lines of an image are present in each frame.

A de-interlacer can be used to convert an interlaced video signal into a progressive scan signal. An overview of de-interlacers can be found in G. de Haan and E. B. Bellers, "De-interlacing - An overview", Proceedings of the IEEE, vol. 86, no.9, pp. 1839-1857, September 1998. Figure 1 shows an example of a possible de-interlacing operation on an interlaced signal 10 comprising odd fields 11 and even fields 12. Each progressive scan output frame 21 , 22 is generated from a combination of fields of the interlaced signal 10. In this example, a pair of fields 11 , 12 of the interlaced signal 10 are combined 25 to form a progressive scan frame 21. Various mathematical operations can be used to combine the input fields, and more than two fields can be combined to generate each progressive scan frame. Many de-interlacers bias their output more towards the content of an odd field of the interlaced signal when generating an odd output frame and more towards the content of an even field of the interlaced signal when generating an even output frame. This can result in particular details appearing in certain de-interlaced frames, such as every odd frame, and can cause a visible flicker when the de-interlaced signal is displayed.

The present invention seeks to reduce the problem of artifacts in a de- interlaced video signal.

Accordingly, a first aspect of the present invention provides a method of processing a video signal to reduce de-interlacing artifacts comprising: receiving an input video signal comprising a plurality of de-interlaced frames which have been generated from an interlaced video signal having fields which occur at a first rate, the de-interlaced frames comprising an artifact which occurs at a first frequency; determining if motion is present between the de-interlaced frames of the input video signal; for at least an area of the input video signal for which there is no motion, or motion below a threshold value: outputting frames at a second rate, higher than the first rate; and, ordering output frames such that the de-interlacing artifact occurs at a second frequency which is higher than the first frequency.

The ordering can also be applied to selected areas of de-interlaced frames for which there is judged to be no motion, or motion less than a threshold value. The ordering can also be applied to entire de-interlaced frames. The threshold value can be fixed, or can be adjustable by a user.

It has been found that if no motion occurred between the odd and even fields of an interlaced signal, then it is possible to swap the de-interlaced frames which are generated from those fields arbitrarily in time without causing visual problems. Increasing the rate at which the frames is displayed, and re-ordering the de-interlaced frames such that an odd frame is always next to an even frame, the rate at which the detail flicker will occur is increased, to such a rate that it becomes unnoticeable, or much less annoying to a viewer.

Some of the most common de-interlacing artifacts are: detail or line flicker - lines or details differing between odd and even frames result in an unsteady image; jaggies - staircasing effects on diagonal or slanted edges which can be unstable between odd and even frames; loss of resolution - filtering out vertical detail can cause a blurring of an image; area flicker - the loss of resolution described above can be unstable between odd and even frames, causing large areas to flicker; double images - choosing information from the wrong field can cause a shadow image to be seen of moving objects. Many of these issues, especially detail or line flicker, can be reduced by using an embodiment of the invention.

Frame rate up-conversion can be performed before applying the frame reordering of this method. The input signal will comprise a sequence of de- interlaced frames which are already at the second (higher) rate, or at a frame rate which is higher than the first rate. Alternatively, frame rate up-conversion can be performed as part of this method. In this case, the input signal will comprise a sequence of frames at the first (lower) rate. The processed signal resulting from applying this method will be at the second (higher) rate with the content of at least part of certain frames re-ordered to reduce flicker. A processed signal resulting from this method can be applied to other processing stages, including other frame rate up-conversion stages.

Typically, the second rate is a frame rate which is double the first rate, although other multiples of the first rate are possible.

Further aspects of the invention provides apparatus for performing the method and software for performing the method. The software can be provided as computer-executable code which is tangibly embodied on an electronic memory device, hard disk, optical disk or any other machine-readable storage medium or it can be downloaded to a processing device via a network connection. Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

Figure 1 shows a technique for generating de-interlaced frames from fields of an interlaced video signal;

Figure 2 shows a video processing system in which the invention can be applied;

Figure 3 shows differences between de-interlaced frames which can cause flicker when the frames are displayed;

Figure 4 shows a sequence of the de-interlaced frames of Figure 3 and ways of presenting the frames at an up-converted frame rate;

Figure 5 shows an embodiment of the invention applied to a part of two de-interlaced frames;

Figure 6 shows a method of processing de-interlaced frames according to an embodiment of the present invention;

Figure 7 shows apparatus for performing the method of Figure 6.

Figure 2 shows a video processing system in which the invention can be used. Interlaced video data is retrieved from a source 30, such as a hard disk, memory or optical disc. The source can also be a live transmission received via a wired or wireless communication link. The interlaced video signal is output 31 to a de-interlacer 32. A de-interlacer 32 processes the fields of the interlaced signal to generate progressive scan video frames. The de-interlacer 32 and store 30 can form part of the same device 35, such as a set-top box, personal video recorder (PVR), optical disc player or media player. Alternately, the de- interlacer 32 can be a stand-alone device or can form part of another device in the processing chain between the source 30 and the display 38. De-interlaced frames are output 33 to a display processing module 36. Typically, the display processing module will form part of a display 38, although it could be provided as a separate unit in the processing chain between the source 30 and the display 38. Display processing module 36 includes artifact reduction processing module 37 which reduces the effects of artifacts, especially flicker, in the de-interlaced signal.

Figure 3 shows two consecutive frames 41 , 42 of a de-interlaced video signal. The terms "odd frame" and "even frame" are used in this description. Frame 41 is termed an "odd frame" as it has been generated by taking the odd field of an interlaced signal as the current field. Frame 42 is termed an "even frame" as it has been generated by taking the even field of an interlaced signal as the current field. A possible process of generating de-interlaced frames from an interlaced signal has been shown in Figure 1 , and various other de-interlacing techniques can be used. Artifacts can occur in certain de-interlaced frames. In Figure 2, an artifact is shown in the form of a horizontal line 44 which is only present in the odd frame 41 , and not in the even frame 42. Artifacts can take various forms, and can include features, or areas of lower resolution, which differ between odd frames and even frames.

Figure 4 now shows the effect of artifact 44 in a sequence of video frames. The top line of Figure 4, shows a de-interlaced signal 51 comprising the odd frames 41 and even frames 42 of Figure 3 at a first frame rate. Typically the first frame rate will be 50Hz or 60Hz. Detail 44 appears in every odd frame 41 and disappears in every even frame 42. This causes a flickering image of feature 44 to occur at a rate of one half of the frame rate. So, for a 60Hz frame rate, the artifact 44 flickers at a rate of 30Hz, which can be noticeable to a viewer.

The second line of Figure 4 shows the effect of increasing the frame rate of de-interlaced signal 51 to generate a signal 52 at a higher frame rate. Large screen displays typically increase the frame rate to increase the refresh rate of the display, which can help to reduce overall flicker of the displayed image and motion blur. In this example, the frame rate has been doubled (e.g. from 60Hz to 120Hz). A popular way to increase the frame rate is to repeat each frame. So, odd frame 41 is displayed twice and then even frame 42 is displayed twice. It can be seen that even though the frame rate has been increased, the detail 44 appears at the same rate as in signal 51 (e.g. 30Hz). The final line of Figure 4 shows a method in accordance with an embodiment of the present invention. The frame rate of the de-interlaced signal 51 is increased to generate a signal 53 at a higher frame rate. Frames at the higher frame rate are re-ordered such that odd frames 41 and even frames 42 are alternately presented at the higher frame rate. It can be seen that detail 44 now appears at half of the higher frame rate of signal 53. With a higher frame rate of 120Hz, the detail 44 now flickers at a rate of 60Hz, which is high enough to be unnoticeable, or much less intrusive, to a viewer.

The technique shown for signal 53 in Figure 4 is most effective if there is no motion between the odd and even fields of the interlaced video signal that were used to generate the odd and even de-interlaced frames. When there is no motion between frames, the technique can be applied to entire frames, as shown in Figure 4 for signal 53. Even where there is motion in some areas of the video frames, the technique can still be applied to those areas of the frames which do not have any motion. Figure 5 shows a sequence of four frames at the higher frame rate. Because there is motion between the frames, the frames are presented in a conventional manner. Odd frame 41 is presented twice and then the even frame 42 is presented twice. Block 50 of the frames does not contain any motion between frames. The content within block 50 is presented in the new manner. The content of block 50 for the odd de-interlaced frame is alternated with the content of block 50 for the even de-interlaced frame. The overall frame comprises a block 50 which alternately carries the odd and even frames and the remainder of each frame is presented in a frame duplicated manner.

The alternating presentation can be used where there is no motion between frames. Typically, conditions which give rise to no motion between frames are: (i) the odd field and even field of the source interlaced signal were captured at different times, and no motion occurred between the capturing of the fields; or (ii) the odd and even field of the source interlaced signal were captured at the same time.

Figure 5 shows the technique applied to a single block 50 where motion is not present. The technique can be applied to multiple blocks, of any shape (i.e. not only square/rectangular blocks). For example, a video signal may convey a still scene (i.e. camera capturing the scene was still with respect to the scene) with a person moving in the centre of the frame. In this case, only the centre of the frame is likely to convey any motion, with the background surrounding the person remaining still. The alternating presentation technique can be applied to all of the background areas of the frames.

Figures 4 and 5 shows a simple technique for increasing the frame rate (also known as frame rate up-conversion) where each source frame is repeated. This is a computationally simple technique. There are various other ways in which the frame rate can be increased, such as motion compensation, dynamic field insertion, grey field insertion. In a motion compensated technique a new frame is calculated by interpolating between frames, to give a smoother transition between frames. The method of alternating non-moving areas between frames can be applied to many of these other methods of frame rate conversion.

Figure 6 shows an overview of an embodiment of the processing method. Firstly, at step 71 two de-interlaced video frames are received at the lower frame rate (e.g. 50Hz or 60Hz). The two frames are an odd frame and an even frame, and have been generated from interlaced fields as previously described. At step 72 the method determines if there is motion between the two frames. If motion is not detected, then the method proceeds to step 74 and entire frames can be presented in an alternating manner at the up-converted frame rate (see signal 53, Figure 4). Step 75 determines if there is motion in all areas of the frame. If some areas (e.g. blocks) of the frame remain still between frames then it is possible to apply the new presentation technique to those still areas of the signal. At step 76 areas of the frames without motion are presented in an alternating manner at the up-converted frame rate, as shown in Figure 5, while other areas of the frames which contain motion are presented in a conventional manner, by presenting the first frame before presenting the second frame. Finally, if motion is present across all of the frames, the signal is processed as normal.

Motion can occur in various ways. Generally, motion can be global (i.e. affecting an entire frame) or local. A range of motion detection algorithms will be known to the skilled person and do not need to be described in more detail. An overview of different techniques is given in "Digital Video Post Processing", Gerard de Haan, Royal Philips Electronics 2006. Global motion can result from a panning or tilting camera movement, or tracking or zooming of the camera, as a scene is captured. Local motion typically involves a moving object within a captured scene. Various motion detection algorithms are known which can detect local motion. Typically, these use a form of block-matching algorithm which compares a block at a first position within a first image with a same sized block in different positions, offset with respect to the first position, in a second image. A computation such as Sum of Absolute Differences (SAD), or similar algorithm, is performed for each possible offset position to determine the amount of motion, if any, and the direction of motion.

Some types of video data coding include supplementary data which indicates if there is motion in the frames. Where additional video frames have been generated at some point in the processing chain to supplement frames at an original frame rate, the additional frames may be flagged as additional frames. In either case, it is possible to use any supplementary data to assist the motion detection process on frames of the input video signal.

It is preferable to output an alternating series of odd and even frames at the higher frame rate only in areas of the frames where there is no motion. It is also possible to output an alternating series of odd and even frames at the higher frame rate for areas of the frames with motion below a particular threshold value. However, this is less desirable as it is likely to result in other motion-related artifacts becoming visible.

Figure 7 shows an embodiment of apparatus for performing the method described above. An input 80 receives de-interlaced frames of a video signal at the lower frame rate (e.g. 50Hz or 60Hz). A frame delay stores a previous frame while a current frame is received. The current frame (from input 80) and the previous frame (from frame delay 83) are input to a motion estimator or motion detector 85. The motion estimator determines if there is any motion between the frames, and can determine which areas/blocks of the frames are locally affected by motion. This information is forwarded to a mixer 87 in the form of a gain map. A switch 84 is clocked at the higher frame rate (e.g. 100Hz or 120Hz) and switches between the current frame and the previous frame at the higher frame rate. Any other frame rate processing which is required is performed by unit 86. In an embodiment of the invention, unit 86 performs frame rate processing (e.g. up-conversion of the frame rate to 100Hz or 120Hz) as normal for entire frames. A mixer 87 receives an alternating sequence of the current frame and previous frame from switch 84. Mixer 87 also receives a processed form of the current frame from unit 86. Mixer 87 uses the gain map received from motion estimator 85 to decide which areas of the output frame should be taken from the current frame (received from unit 86) and which areas of the output frame should be taken from the alternating sequence of the current and previous frames (received from switch 84). In this manner, the areas of the video signal with no motion are presented in alternating form (output of switch 86) and the areas of the video signal with motion are presented in conventional form. In an apparatus which performs motion compensated frame rate up-conversion, motion estimation is required to perform the frame rate up-conversion. The functionality of the motion estimator 85 can be combined, at least in part, with any motion estimation required for frame rate up-conversion, thereby reducing the amount of additional functionality required to implement the method.

The apparatus shown in Figure 7 can be adapted to receive signals at input 80 having a higher frame rate (e.g. at 100Hz/120Hz). It is advantageous for the frame delay 83 to store a frame for two frame periods so that the apparatus can compare, and switch between, an odd frame and an even frame in the same manner as described above for receiving a signal at the lower frame rate. The input signals to the apparatus may have already been up-converted to the higher frame rate by an earlier processing stage. In a first embodiment, the apparatus detects areas of the frames which have no motion and re-orders the content of the frames for these areas. No additional frame rate processing is performed. In another embodiment, an additional stage of frame rate up-conversion is performed and combined with the re-ordering of entire frames, or areas of frames with no motion. This may further reduce the flicker resulting from de-interlacing artifacts.

Although not shown in Figure 7, additional frame delays may be required for the purpose of motion compensated frame rate conversion to determine if there was motion at the original capture time.

The apparatus can be incorporated into any video processing block which outputs at a higher frame rate (e.g. 100/120Hz), such as a timing converter (TCON) of a display, or a separate frame rate conversion integrated circuit with integrated frame rate conversion functionality. Also it could be added downstream of any apparatus having frame rate conversion functionality.

In the illustrated frame sequences of Figure 4 a number of frames contain the same de-interlacing artifact 44. It is possible for the same artifact to be present in a long sequence of frames or, taken to the opposite extreme, it is also possible for just two output frames to be swapped at the higher frame rate, e.g. replacing the motion-less even frame contents by odd frame contents.

The values of field and frame rates used in this description are illustrative of commonly used rates and are not limiting.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The words "comprising" and "including" do not exclude the presence of other elements or steps than those listed in the claim. Where the system/device/apparatus claims recite several means, several of these means can be embodied by one and the same item of hardware.

The various illustrative logical blocks, modules, circuits, and algorithm steps described above may be implemented as electronic hardware, as software modules executed by a processor, or as combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the described functions. A general- purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.

In the description above, and with reference to the Figures, there is described a method and apparatus for reducing de-interlacing artifacts. An input video signal comprises a plurality of de-interlaced frames 41 , 42 which have been generated from an interlaced video signal having fields 11 , 12 which occur at a first rate, the de-interlaced frames comprising an artifact 44 which occurs at a first frequency. A determination is made if motion is present between the de- interlaced frames 41 , 42 of the input video signal. For at least an area of the input video signal for which there is no motion, or motion below a threshold value, frames are output at a second rate, higher than the first rate. The output frames are ordered such that the de-interlacing artifacts occur at a second frequency which is higher than the first frequency. For an input video signal comprising odd frames and even frames, the step of ordering frames alternates an odd frame with an even frame at the second rate.

Claims

1. A method of processing a video signal to reduce de-interlacing artifacts comprising: receiving an input video signal comprising a plurality of de-interlaced frames (41 , 42) which have been generated from an interlaced video signal having fields which occur at a first rate, the de-interlaced frames comprising an artifact (44) which occurs at a first frequency; determining if motion is present between the de-interlaced frames (41 , 42) of the input video signal; for at least an area of the input video signal for which there is no motion, or motion below a threshold value: outputting frames at a second rate, higher than the first rate; and, ordering (53) output frames such that the de-interlacing artifact (44) occurs at a second frequency which is higher than the first frequency.

2. A method according to claim 1 wherein the input video signal comprises odd frames and even frames, and the step of ordering frames alternates an odd frame with an even frame at the second rate.

3. A method according to claim 1 or 2 wherein the input video signal comprises odd frames and even frames and wherein for an area of the input video signal for which there is motion, or motion above a threshold value, the method comprises outputting frames at the second rate, the output frames comprising N odd frames having substantially the same content followed by N even frames having substantially the same content, with N>1.

4. A method according to any one of the preceding claims wherein the input video signal comprises de-interlaced frames at a rate which is lower than the second rate, and the method comprises up-converting the frame rate to the second rate.

5. A method according to any one of claims 1 to 3 wherein the input video signal comprises de-interlaced frames at the second rate.

6. A method according to any one of the preceding claims wherein the step of determining if motion is present comprises at least one of: examining a sequence of frames; examining supplementary data accompanying the input video signal.

7. Apparatus for processing a video signal to reduce de-interlacing artifacts comprising: an input (80) for receiving a video signal comprising a plurality of de- interlaced frames (41 , 42) which have been generated from an interlaced video signal having fields which occur at a first rate, the de-interlaced frames (41 , 42) comprising an artifact (44) which occurs at a first frequency; a motion estimator (85) for determining if motion is present between the de-interlaced frames of the input video signal; and a processor which is arranged to, for at least an area of the input video signal for which there is no motion, or motion below a threshold value: output frames at a second rate, higher than the first rate; and, order output frames such that the de-interlacing artifact (44) occurs at a second frequency which is higher than the first frequency.

8. Apparatus according to claim 7 wherein the input video signal comprises odd frames and even frames, and the processor is arranged to order frames such that an odd frame is alternated with an even frame at the second rate.

9. Apparatus according to claim 7 or 8 wherein the input video signal comprises odd frames and even frames and wherein for an area of the input video signal for which there is motion, or motion above a threshold value, the processor is arranged to output frames at the second rate, the output frames comprising N odd frames having substantially the same content followed by N even frames having substantially the same content, with N>1.

10. Apparatus according to any one of claims 7 to 9 wherein the input video signal comprises de-interlaced frames at a rate which is lower than the second rate, and the processor is arranged to up-convert the frame rate to the second rate.

11. Apparatus according to any one of claims 7 to 9 wherein the input video signal comprises de-interlaced frames at the second rate.

12. Apparatus according to any one of claims 7 to 1 1 wherein the motion estimator (85) is arranged to determine if motion is present by at least one of: examining a sequence of frames; examining supplementary data accompanying the input video signal.

13. A de-interlaced video signal which has been subject to the processing method according to any one of claims 1 to 6.

14. Software for performing the method according to any one of claims 1 to 6.