WO2002063562A2 - 3-d recursive vector estimation for video enhancement - Google Patents
3-d recursive vector estimation for video enhancement Download PDFInfo
- Publication number
- WO2002063562A2 WO2002063562A2 PCT/IB2002/000275 IB0200275W WO02063562A2 WO 2002063562 A2 WO2002063562 A2 WO 2002063562A2 IB 0200275 W IB0200275 W IB 0200275W WO 02063562 A2 WO02063562 A2 WO 02063562A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- pixel region
- enhancement
- candidate
- enhanced
- spatio
- Prior art date
Links
- 239000013598 vector Substances 0.000 title claims abstract description 80
- 230000002708 enhancing effect Effects 0.000 claims abstract description 18
- 238000000034 method Methods 0.000 claims description 20
- 230000002123 temporal effect Effects 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims 4
- 238000006243 chemical reaction Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 11
- 230000033001 locomotion Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 9
- 230000015654 memory Effects 0.000 description 7
- 230000006872 improvement Effects 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 230000001052 transient effect Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003628 erosive effect Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
- H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Definitions
- the present invention is directed, in general, to video enhancement systems and, more specifically, to maintaining spatio-temporal consistency during video enhancement.
- Frequency peaking involves linear boosting or "peaking" of selected spatial frequencies within the image, often with a bandpass or highpass filter to enhances the associated spatial frequencies and with adaptive control to avoid “unnaturalness” relating to, for example, peaking large and steep edges.
- luminance transient improvement preserves the magnitude of the edge but increases the steepness of the edge, "pulling" samples near the edge on both sides towards the edge.
- Existing edge enhancement algorithms enhance the sharpness of an image based on the spatial information of the original image, often utilizing control parameters determined by a small spatial neighborhood of a given pixel position.
- a primary object of the present invention to provide, for use in a video signal processor, a technique for enhancing video information which evaluates candidate vectors of enhancement algorithms utilizing an error function biased towards spatio-temporal consistency with a penalty function.
- the penalty function increases with the distance— both spatial and temporal— of the subject block from the block for which the candidate vector was optimal. Enhancements are therefore gradual across both space and time and the enhanced video information is intrinsically free of perceptible artifacts.
- Fig. 1 depicts a system in which video enhancement with spatio-temporal consistency is implemented according to one embodiment of the present invention
- Fig. 2 illustrates in greater detail a system for video enhancement with spatio- temporal consistency according to one embodiment of the present invention
- Fig. 3 illustrates a logical organization of video information for video enhancement with spatio-temporal consistency according to one embodiment of the present invention
- Fig. 4 is a high level flow chart for a process of video enhancement with spatio-temporal consistency according to one embodiment of the present invention
- Fig. 5 is an illustration of displacement of a moving object from an expected position as a result of field rate conversion through field repetition;
- Figs. 6 A and 6B are comparative illustrations for spatial resolution enhancement.
- Fig. 1 depicts a system in which video enhancement with spatio-temporal consistency is implemented according to one embodiment of the present invention.
- System 100 includes a receiver 101, which in the exemplary embodiment is a high definition digital television (HDTV) large-screen or wide-screen television receiver.
- receiver 101 may be an intermediate transceiver or any other device employed to receive or transceive video signals, as for example a transceiver retransmitting video information for reception by a high definition television.
- receiver 101 includes a video enhancement mechanism as described in further detail below.
- Receiver 101 includes an input 102 for receiving video signals and may optionally include an output 103 for transmitting enhanced video signals to another device.
- Nideo signal processor 201 includes an enhancement vector estimator 202 and enhancement processor 203 which perform the video enhancement processing.
- Nideo signal processor 201 in the exemplary embodiment is the device from which the enhanced video output is transmitted either to display 104 or to a storage medium (not shown).
- Enhancement processor 203 performs the processing on received video signals required to enhance the video for display.
- Image or video enhancement is a broad area which may be roughly divided into three categories: restoration of "lost" (image/video) information; elimination of artifacts; and enhancement of selected image/video characteristics.
- restoration of "lost" (image/video) information is a broad area which may be roughly divided into three categories: restoration of "lost" (image/video) information; elimination of artifacts; and enhancement of selected image/video characteristics.
- the present invention is not limited to any particular category of video enhancement, for the purposes of simplicity resolution enhancement, which falls within the third category, will be utilized to describe and explain the invention. Nonetheless, those skilled in the art will understand that the invention may be readily adapted or extended to video enhancements other than resolution enhancement and falling within any of the three categories listed.
- Enhancement processor 203 together with enhancement vector estimator 202 in the exemplary embodiment, performs spatial resolution enhancement on the video information received.
- the technique for estimation of enhancement vectors according to the present invention is similar to the recursive search block matching motion estimation process described in the references identified above.
- Fig. 3 illustrates a logical organization of video information for video enhancement with spatio-temporal consistency according to one embodiment of the present invention.
- the organization depicted is employed for block enhancement by video signal processor 201 depicted in Fig. 2.
- the video information to be enhanced includes a plurality of successive pictures (which may be either fields or frames) to be displayed in sequence at a predefined rate.
- "Successive,” as used herein, refers to a subject picture being in consecutive series with another picture within the sequence, without regard to whether the subject picture is before or after the other picture within the video information.
- a portion of the sequence of pictures, n-2, n-1, n, n+1 and n+2 is shown in Fig. 3.
- Each picture comprises a two- dimensional array of pixels having coordinates (x,y) from the lower left corner of the
- Block enhancement units 206a-206n within video signal processor 201 enhance the received video information on a per block basis.
- spatial resolution enhancement will be employed to explain the present invention. Specifically, an increase in the spatial resolution of the incoming video by a factor of two in both spatial dimensions of the fields will be employed to describe the present invention.
- An initial estimate of higher spatial resolution video information G(x, ⁇ ) based on the lower resolution video information F(x, ) may be initially created by a simple spatial up-conversion—that is, a sample-rate conversion interpolation filter within block enhancement units 206a-206n is employed to obtain a higher resolution image.
- W, () within the above equation indicates enhancement of the image data quality (where spatial resolution has already been enhanced by sample-rate conversion) by an algorithm i within a set of algorithms.
- W 0 (F(x,n)) could be the image data after frequency peaking while W, (F(x, n)) may be the result after luminance transient improvement.
- the penalty P ⁇ within the error function given above is a monotonic decreasing function of the norm of the enhancement vector V , introducing a large penalty for small coefficients and a small penalty for large coefficients.
- the penalty P 2 is employed to bias the enhancement vector V towards a spatial-temporally consistent solution since this penalty depends on the selected enhancement vector candidate C . Accordingly, the value of penalty P 2 is selected from a predefined list of penalty values which are optimized for the application.
- Each enhancement vector candidate C is preferably selected from enhancement vectors previously determined to produce the smallest error function values for blocks within a spatio-temporal neighborhood around the block B(X) being processed.
- a "Y-prediction" estimator for recursive search block matching motion estimation in which spatial prediction vector candidates C sp and C SP2 are the vectors selected for blocks one block dimension above and to either side of and within the same field as the subject block B(X) while a temporal prediction candidate
- C TP is the vector selected for a block two blocks directly below and within the previous field n -1 from the field n containing the subject block B(X) .
- Selection of candidate enhancement vectors from the enhancement vectors which produced optimal results within the spatio-temporal neighborhood of the subject block B( ⁇ ) speeds the process of determining the best enhancement (the enhancement vector which produces the smallest error, or other suitable criteria for enhancement results, for the subject block B(X) ) since it is very likely that enhancement(s) similar to those producing the best results for other blocks within the neighborhood of the subject block B(X) will produce the best results for the subject block B(X) .
- all possible candidate vectors of enhancement algorithms may be tested for each block.
- the set of candidate vectors employed may change during processing of the video information, with, for example, all possible candidate vectors being tested for the first few fields of the video information and then a smaller subset of candidate vectors being employed for remaining fields, or with the selection of candidate vectors being otherwise refined as the video information is processed.
- one candidate is always updated with a random update vector.
- Several candidates may compete with each other, with the candidate yielding the smallest error ⁇ (C,X,n) being selected as the enhancement vector for the data within the subject block B(X) .
- a block within a current field of the received video information is first selected (step 402) and a simple enhancement, in this case sample rate conversion, is performed.
- the block is also enhanced utilizing each of a plurality of selected candidate enhancement vectors consisting one or more enhancement algorithms employed jointly or individually, such as frequency peaking and luminance transient improvement.
- An error function value where the error function includes a bias towards spatio-temporal consistency, is then computed for each candidate enhancement vector (step 404) and the enhancement corresponding to the candidate vector having the lowest error function value is selected (step 405) for display as part of the enhanced field.
- a determination as to whether all blocks within the current field have been processed is then made, followed by selection and processing of a next block within the current field (step 407) if additional blocks remain and initiation of processing on the next field (step 408) if the current field has been completely processed. Once initiated, the process proceeds until interrupted by an external influence, such as the receiver being turned off or the reception of video information being interrupted.
- machine usable mediums include: nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), recordable type mediums such as floppy disks, hard disk drives and compact disc read only memories (CD- ROMs) or digital versatile discs (DNDs), and transmission type mediums such as digital and analog communication links.
- ROMs read only memories
- EEPROMs electrically programmable read only memories
- CD- ROMs compact disc read only memories
- DNDs digital versatile discs
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Television Systems (AREA)
- Image Processing (AREA)
- Picture Signal Circuits (AREA)
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP02716230A EP1360648A2 (en) | 2001-02-08 | 2002-01-28 | Spatio-temporal video enhancement |
KR1020027013488A KR20020087128A (en) | 2001-02-08 | 2002-01-28 | 3-D Recursive vector estimation for video enhancement |
JP2002563430A JP2004519145A (en) | 2001-02-08 | 2002-01-28 | Evaluation of 3D Inductive Vectors for Video Enhancement |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US26725601P | 2001-02-08 | 2001-02-08 | |
US60/267,256 | 2001-02-08 | ||
US09/840,817 | 2001-04-24 | ||
US09/840,817 US7042945B2 (en) | 2001-04-24 | 2001-04-24 | 3-D recursive vector estimation for video enhancement |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2002063562A2 true WO2002063562A2 (en) | 2002-08-15 |
WO2002063562A3 WO2002063562A3 (en) | 2002-10-17 |
Family
ID=26952332
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2002/000275 WO2002063562A2 (en) | 2001-02-08 | 2002-01-28 | 3-d recursive vector estimation for video enhancement |
Country Status (5)
Country | Link |
---|---|
EP (1) | EP1360648A2 (en) |
JP (1) | JP2004519145A (en) |
KR (1) | KR20020087128A (en) |
CN (1) | CN1460230A (en) |
WO (1) | WO2002063562A2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2004073313A1 (en) * | 2003-02-13 | 2004-08-26 | Koninklijke Philips Electronics N.V. | Spatio-temporal up-conversion |
WO2007089803A2 (en) * | 2006-01-31 | 2007-08-09 | Thomson Licensing | Methods and apparatus for edge-based spatio-temporal filtering |
-
2002
- 2002-01-28 EP EP02716230A patent/EP1360648A2/en not_active Withdrawn
- 2002-01-28 CN CN02801065A patent/CN1460230A/en active Pending
- 2002-01-28 JP JP2002563430A patent/JP2004519145A/en not_active Withdrawn
- 2002-01-28 KR KR1020027013488A patent/KR20020087128A/en not_active Application Discontinuation
- 2002-01-28 WO PCT/IB2002/000275 patent/WO2002063562A2/en not_active Application Discontinuation
Non-Patent Citations (5)
Title |
---|
AUFRICHTIG R ET AL: "Spatio-temporal X-ray fluoroscopy filtering using object detection" PROCEEDINGS OF THE COMPUTERS IN CARDIOLOGY CONFERENCE. LONDON, SEPT. 5 - 8, 1993, LOS ALAMITOS, IEEE COMP. SOC. PRESS, US, 5 September 1993 (1993-09-05), pages 587-590, XP010128831 ISBN: 0-8186-5470-8 * |
BORMAN S ET AL: "Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors" IMAGE PROCESSING, 1999. ICIP 99. PROCEEDINGS. 1999 INTERNATIONAL CONFERENCE ON KOBE, JAPAN 24-28 OCT. 1999, PISCATAWAY, NJ, USA,IEEE, US, 24 October 1999 (1999-10-24), pages 469-473, XP010368835 ISBN: 0-7803-5467-2 * |
BRAILEAN J C ET AL: "SIMULTANEOUS RECURSIVE DISPLACEMENT ESTIMATION AND RESTORATION OF NOISY-BLURRED IMAGE SEQUENCES" IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE INC. NEW YORK, US, vol. 4, no. 9, 1 September 1995 (1995-09-01), pages 1236-1251, XP000533956 ISSN: 1057-7149 * |
HAAN DE G ET AL: "TRUE-MOTION ESTIMATION WITH 3-D RECURSIVE SEARCH BLOCK MATCHING" IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE INC. NEW YORK, US, vol. 3, no. 5, 1 October 1993 (1993-10-01), pages 368-379, XP000414663 ISSN: 1051-8215 cited in the application * |
KLEIHORST R P ET AL: "NOISE REDUCTION OF IMAGE SEQUENCES USING MOTION COMPENSATION AND SIGNAL DECOMPOSITION" IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE INC. NEW YORK, US, vol. 4, no. 3, 1 March 1995 (1995-03-01), pages 274-284, XP000501902 ISSN: 1057-7149 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2004073313A1 (en) * | 2003-02-13 | 2004-08-26 | Koninklijke Philips Electronics N.V. | Spatio-temporal up-conversion |
WO2007089803A2 (en) * | 2006-01-31 | 2007-08-09 | Thomson Licensing | Methods and apparatus for edge-based spatio-temporal filtering |
WO2007089803A3 (en) * | 2006-01-31 | 2007-10-11 | Thomson Licensing | Methods and apparatus for edge-based spatio-temporal filtering |
US8135234B2 (en) | 2006-01-31 | 2012-03-13 | Thomson Licensing | Method and apparatus for edge-based spatio-temporal filtering |
Also Published As
Publication number | Publication date |
---|---|
WO2002063562A3 (en) | 2002-10-17 |
CN1460230A (en) | 2003-12-03 |
KR20020087128A (en) | 2002-11-21 |
EP1360648A2 (en) | 2003-11-12 |
JP2004519145A (en) | 2004-06-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6452639B1 (en) | Raster scan conversion system for interpolating interlaced signals | |
Jeon et al. | Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure | |
US7034892B2 (en) | Spatio-temporal filter unit and image display apparatus comprising such a spatio-temporal filter unit | |
KR970009469B1 (en) | Interlace/sequential scan conversion apparatus and method for facilitating double smoothing function | |
US6377621B2 (en) | Motion compensated interpolation | |
US7720150B2 (en) | Pixel data selection device for motion compensated interpolation and method thereof | |
US6810081B2 (en) | Method for improving accuracy of block based motion compensation | |
US20110255004A1 (en) | High definition frame rate conversion | |
US7489350B2 (en) | Unit for and method of sharpness enhancement | |
US6847406B2 (en) | High quality, cost-effective film-to-video converter for high definition television | |
GB2337391A (en) | Interlaced to progressive scanning conversion with edge enhancement by vertical temporal interpolation | |
JP5133038B2 (en) | Image restoration method and image restoration apparatus | |
US7042945B2 (en) | 3-D recursive vector estimation for video enhancement | |
GB2405047A (en) | De-interlacing algorithm responsive to edge pattern | |
JPH01245684A (en) | Picture information transmitting system | |
EP0801862B1 (en) | Data filtering | |
WO2002063562A2 (en) | 3-d recursive vector estimation for video enhancement | |
JP3745425B2 (en) | Motion vector detection method and adaptive switching prefilter for motion vector detection | |
US20060038918A1 (en) | Unit for and method of image conversion | |
JPH06334909A (en) | Noise reducing device | |
JPH0447785A (en) | Television receiver |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A2 Designated state(s): CN JP KR |
|
AL | Designated countries for regional patents |
Kind code of ref document: A2 Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2002716230 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020027013488 Country of ref document: KR |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
AK | Designated states |
Kind code of ref document: A3 Designated state(s): CN JP KR |
|
AL | Designated countries for regional patents |
Kind code of ref document: A3 Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR |
|
WWP | Wipo information: published in national office |
Ref document number: 1020027013488 Country of ref document: KR |
|
WWE | Wipo information: entry into national phase |
Ref document number: 028010655 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2002563430 Country of ref document: JP |
|
WWP | Wipo information: published in national office |
Ref document number: 2002716230 Country of ref document: EP |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 2002716230 Country of ref document: EP |