US20140176794A1 - Image processing apparatus, image processing method, and program - Google Patents
Image processing apparatus, image processing method, and program Download PDFInfo
- Publication number
- US20140176794A1 US20140176794A1 US14/101,443 US201314101443A US2014176794A1 US 20140176794 A1 US20140176794 A1 US 20140176794A1 US 201314101443 A US201314101443 A US 201314101443A US 2014176794 A1 US2014176794 A1 US 2014176794A1
- Authority
- US
- United States
- Prior art keywords
- image
- frequency area
- unit
- original images
- difference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims description 9
- 238000000605 extraction Methods 0.000 claims abstract description 55
- 230000008859 change Effects 0.000 claims abstract description 18
- 239000013598 vector Substances 0.000 claims description 157
- 238000001514 detection method Methods 0.000 claims description 63
- 230000014509 gene expression Effects 0.000 description 64
- 238000010586 diagram Methods 0.000 description 46
- 239000000203 mixture Substances 0.000 description 32
- KDYFGRWQOYBRFD-UHFFFAOYSA-N succinic acid Chemical compound OC(=O)CCC(O)=O KDYFGRWQOYBRFD-UHFFFAOYSA-N 0.000 description 17
- 230000015654 memory Effects 0.000 description 16
- 238000006243 chemical reaction Methods 0.000 description 15
- 238000000034 method Methods 0.000 description 14
- 230000000694 effects Effects 0.000 description 11
- 239000000284 extract Substances 0.000 description 5
- 230000010365 information processing Effects 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000002238 attenuated effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0127—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
- H04N7/0132—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
- H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
Definitions
- the present disclosure relates to an image processing apparatus, an image processing method, and a program causing a computer to execute the image processing method. Specifically, the present disclosure relates to an image processing apparatus and an image processing method with which the number of frames per unit time in a moving image is changed and to a program causing a computer to execute the image processing method.
- a frame rate conversion to change the number of frames per unit time, i.e., a frame rate, in a moving image is performed in some cases.
- a frame rate conversion to increase a frame rate which is called an up-conversion, is performed.
- a frame interpolation method in which an interpolated frame is generated from original frames and inserted between the original frames is widely used.
- motion compensation processing to detect the motion of an object in the original frame and generate an interpolated frame based on the motion is performed. This allows a natural interpolation to be performed in a moving image including a moving object.
- a method of detecting the motion of an object on a block-to-block basis or a pixel-to-pixel basis is widely used, but the method may cause a possibility that the motion detection fails in some blocks or pixels.
- the object whose motion has failed to be detected is a high-frequency component, e.g., a character telop
- a defect easily occurs in an image with a part at which the detection has failed.
- an object including a high-frequency component has a clear boundary with the background, blurs occur at the part at which the detection has failed.
- an object including a low-frequency component has a blurred boundary, blurs are not liable to occur if the motion detection fails.
- the image processing apparatus detects an area including a high-frequency component with motion to be an area for character telop and changes an interpolation method for such an area (see, for example, Japanese Patent Application Laid-open No. 2009-296284).
- the image processing apparatus detects motion of the entire area in the area of the character telop and detects motion on a block-to-block basis or a pixel-to-pixel basis in the other areas.
- the entire area of the character telop in the original frame is displaced, and in areas other than the character telop, the original frame is displaced for each block or pixel, to generate an interpolated frame.
- the entire area of the character telop is displaced, so that no blurs occur in the area.
- the image quality of the moving image after the frame rate conversion may deteriorate.
- the image processing apparatus performs different types of processing on the area of the character telop and on the other areas, and hence there is a possibility that the boundaries between those areas look unnatural.
- the present disclosure has been made in view of the circumstances as described above, and it is desirable to improve the image quality of a moving image whose frame rate is converted.
- an image processing apparatus includes a normal interpolated image generation unit, a high-frequency area extraction unit, a high-frequency area interpolated image generation unit, and a combination unit.
- the normal interpolated image generation unit is configured to generate an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images.
- the high-frequency area extraction unit is configured to extract a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images.
- the high-frequency area interpolated image generation unit is configured to generate an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images.
- the combination unit is configured to execute combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
- This provides effects of generating the normal interpolated image based on each of the plurality of original images, generating the high-frequency area interpolated image based on the change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images, and combining the normal interpolated image and the high-frequency area interpolated image.
- the image processing apparatus may further include a vector detection unit configured to detect a vector indicating a direction and a distance in which a position of the high-frequency area changes within a predetermined period of time on the time series, and the high-frequency area interpolated image generation unit may be configured to generate the high-frequency area interpolated image based on the vector and each of the plurality of original images. This provides an effect of generating the high-frequency area interpolated image based on the vector indicating the direction and the distance in which the position of the high-frequency area changes and on each of the plurality of original images.
- the image processing apparatus may further include a combination ratio determination unit configured to determine a ratio at which the normal interpolated image and the high-frequency area interpolated image are combined, for each pixel in accordance with each pixel value of pixels in the high-frequency area, and the combination unit may be configured to execute the combining processing according to the ratio. This provides an effect of executing the combining processing according to the ratio determined for each pixel in accordance with each pixel value of the pixels in the high-frequency area.
- the high-frequency area extraction unit may include a filter unit, a difference detection unit, and an increase unit.
- the filter unit is configured to extract the high-frequency area in each of the plurality of original images.
- the difference detection unit is configured to detect, for each pixel, a difference in pixel value between two adjacent original images of the plurality of original images.
- the increase unit is configured to increase each pixel value of the pixels in the high-frequency area in each of the two adjacent original images in accordance with the difference, and supply the increased pixel value to the vector detection unit and the combination ratio determination unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference.
- the high-frequency area extraction unit may include a first difference coring processing unit configured to execute first difference coring processing to remove a difference smaller than a first difference threshold from the detected differences, and the increase unit may be configured to increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the first difference coring processing is performed, and supply the increased pixel value to the vector detection unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference on which the first difference coring processing is performed.
- the high-frequency area extraction unit may include a second difference coring processing unit configured to execute second difference coring processing to remove a difference smaller than a second difference threshold from the detected differences, and the increase unit may be configured to increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the second difference coring processing is performed, and supply the increased pixel value to the combination ratio determination unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference on which the second difference coring processing is performed.
- the high-frequency area extraction unit may include a first pixel coring processing unit configured to execute first pixel coring processing to remove a pixel with a pixel value smaller than a first pixel value threshold in the extracted high-frequency area, and the increase unit may be configured to increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the first pixel coring processing is performed, and supply the increased pixel value to the vector detection unit. This provides an effect of increasing, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the first pixel coring processing is performed.
- the high-frequency area extraction unit may include a second pixel coring processing unit configured to execute second pixel coring processing to remove a pixel with a pixel value smaller than a second pixel value threshold in the extracted high-frequency area
- the increase unit may be configured to increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the second pixel coring processing is performed, and supply the increased pixel value to the combination ratio determination unit. This provides an effect of increasing, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the second pixel coring processing is performed.
- each of the plurality of original images may include a plurality of blocks each having a predetermined shape
- the normal interpolated image generation unit may be configured to generate the normal interpolated image based on a change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images. This provides an effect of generating a normal interpolated image based on the change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images.
- the image processing apparatus may further include a selection unit configured to select the combined image by the combination unit and the plurality of original images in order of the time series and output the selected image. This provides an effect of selecting the combined image and the plurality of original images in order of the time series and outputting the selected image.
- FIG. 1 is a block diagram showing a configuration example of an information processing system in a first embodiment
- FIG. 2 is a block diagram showing a configuration example of an image processing apparatus in the first embodiment
- FIG. 3 is a block diagram showing a configuration example of an interpolated image generation unit in the first embodiment
- FIGS. 4A , 4 B, and 4 C are each a diagram for describing a method of detecting a local motion vector in the first embodiment
- FIG. 5 is a diagram showing an example of the local motion vector in the first embodiment
- FIG. 6 is a block diagram showing a configuration example of a normal interpolated image generation unit in the first embodiment
- FIGS. 7A , 7 B, and 7 C are each a diagram for describing a method of generating a normal interpolated image in the first embodiment
- FIG. 8 is a block diagram showing a configuration example of a high-frequency area extraction unit in the first embodiment
- FIG. 9 is a block diagram showing a configuration example of a difference detection unit in the first embodiment.
- FIG. 10 is a block diagram showing a configuration example of a high-frequency area interpolated image generation unit in the first embodiment
- FIG. 11 is a block diagram showing a configuration example of a combination ratio determination unit in the first embodiment
- FIG. 12 is a block diagram showing a configuration example of a combination unit in the first embodiment
- FIG. 13 is a diagram showing an example of the normal interpolated image in the first embodiment
- FIG. 14 is a diagram showing an example of a high-frequency area interpolated image in the first embodiment
- FIG. 15 is a diagram showing an example of combination ratio data in the first embodiment
- FIG. 16 is a diagram showing an example of a combined interpolated image in the first embodiment
- FIG. 17 is a flowchart showing an example of an operation of the image processing apparatus in the first embodiment
- FIG. 18 is a flowchart showing an example of interpolated image generation processing in the first embodiment
- FIG. 19 is a block diagram showing a configuration example of an interpolated image generation unit in a modified example of the first embodiment
- FIG. 20 is a block diagram showing a configuration example of a normal interpolated image generation unit in the modified example of the first embodiment
- FIG. 21 is a diagram showing an example of a normal interpolated image in the modified example of the first embodiment
- FIG. 22 is a block diagram showing a configuration example of an interpolated image generation unit in a second embodiment
- FIG. 23 is a block diagram showing a configuration example of a high-frequency area extraction unit in the second embodiment.
- FIG. 24 is a block diagram showing a configuration example of a high-frequency area extraction unit in a modified example of the second embodiment.
- Second Embodiment Example of Combining Normal Interpolated Image and High-Frequency Area Interpolated Image Generated from Difference Obtained After Coring Processing
- FIG. 1 is a block diagram showing a configuration example of an information processing system in a first embodiment.
- the information processing system includes a moving image supply apparatus 100 , an image processing apparatus 200 , and a display apparatus 400 .
- the moving image supply apparatus 100 supplies an input signal, which contains a moving image, to the image processing apparatus 200 via a signal line 109 .
- the moving image contains a plurality of frames (images) to be reproduced in a time-series order.
- each of the images in the input signal is referred to as an “original image”.
- the moving image supply apparatus 100 receives a broadcast wave providing a moving image and acquires the moving image from the broadcast wave.
- the frame rate of the moving image is represented by hertz (Hz)
- the frame rate of the moving image in the input signal is 60 hertz, for example.
- the moving image supply apparatus 100 may acquire a moving image from a source other than the broadcast wave.
- the moving image supply apparatus 100 may acquire the moving image by reading out a moving image recorded on a recording medium such as a DVD (Digital Versatile Disk) or recorded in a storage device such as an HDD (Hard Disk Drive).
- a recording medium such as a DVD (Digital Versatile Disk) or recorded in a storage device such as an HDD (Hard Disk Drive).
- HDD Hard Disk Drive
- the image processing apparatus 200 converts the frame rate of the moving image in the input signal and outputs an output signal, which contains the moving image whose frame rate is converted, to the display apparatus 400 via a signal line 209 .
- the frame rate of the moving image in the output signal is 120 hertz, for example.
- a ratio of the frame rate of the input signal to the frame rate of the output signal is 1:m (m is a real number larger than 1) in a frame rate conversion
- mX speed frame rate conversion For example, the conversion from 60 hertz to 120 hertz is a double-speed frame rate conversion.
- the image processing apparatus 200 may perform a frame rate conversion such as a quadruple-speed frame rate conversion from 30 hertz to 120 hertz, in addition to the double-speed frame rate conversion.
- the display apparatus 400 reproduces and displays the moving image contained in the output signal.
- FIG. 2 is a block diagram showing a configuration example of the image processing apparatus 200 in the first embodiment.
- the image processing apparatus 200 includes image memories 210 , 220 , and 230 , a selector 240 , and an interpolated image generation unit 300 .
- the image memories 210 and 230 each store an original image F n contained in the input signal.
- n is an integer indicating the order in which original images are reproduced.
- the interpolated image generation unit 300 and the selector 240 read out the original images stored in the image memories 210 and 230 , respectively, when the next original image is input, so that an original image delayed by one image can be acquired.
- the interpolated image generation unit 300 generates, based on the original images F n and F n-1 , an image to be interpolated between those original images F n and F n-1 .
- the generated image is a combined interpolated image MC_mix.
- the interpolated image generation unit 300 acquires the original image F n from the input signal and acquires the delayed original image F n-1 from the image memory 210 , to generate a combined interpolated image MC_mix based on those original images and cause the image memory 220 to store the resultant image MC_mix.
- the image memory 220 is for storing the combined interpolated image MC_mix.
- the selector 240 selects and outputs the combined interpolated image MC_mix stored in the image memory 220 and the original image stored in the image memory 230 in a time-series order.
- FIG. 3 is a block diagram showing a configuration example of the interpolated image generation unit 300 in the first embodiment.
- the interpolated image generation unit 300 includes a local motion vector detection unit 310 , a normal interpolated image generation unit 320 , a high-frequency area extraction unit 330 , a global motion vector detection unit 340 , a high-frequency area interpolated image generation unit 350 , a combination ratio determination unit 360 , and a combination unit 370 .
- the local motion vector detection unit 310 divides the original image into a plurality of blocks each having a predetermined shape and detects a vector, i.e., a local motion vector, which indicates a direction and a distance in which each of those blocks moves within a certain period of time. Specifically, the local motion vector detection unit 310 handles the original image as an image formed of square blocks each having 8 ⁇ 8 pixels, for example. The local motion vector detection unit 310 selects any one of the blocks of an interpolated image to use the block as a target block.
- a vector i.e., a local motion vector
- the local motion vector detection unit 310 detects a block of the original image, which corresponds to the target block of the interpolated image, and then sets a search range W for an area in the original image, which falls within a certain range from the block of the original image.
- the search range W is a range used for searching for a local motion vector by block matching. For example, in the original image, the search range W is set for the target block and a predetermined number of blocks around the target block.
- a vector that is a candidate when a local motion vector is searched for is referred to as a candidate motion vector mv.
- the candidate motion vector my can be divided into v n and v n-1 by Expressions 1 and 2 below.
- mvX represents an x component of the candidate motion vector mv
- mvY represents a y component of the candidate motion vector mv.
- Relpos represents an interpolation position parameter that indicates an interpolation position.
- the interpolation position is a reproduction position of each combined interpolated image MC_mix in the output signal on a time axis on which the moving image is reproduced.
- the interpolation position parameter or Relpos is a value obtained by dividing a time between the original image and the interpolated image by a time between two adjacent original images, for example.
- the time between the original image and the interpolated image is a half of the time between the two adjacent original images, and the interpolation position parameter of Relpos is “0.50”.
- the interpolation position parameter of Relpos of each of the interpolated images is “0.25”, “0.50”, or “0.75”.
- a pixel value “succ” of a pixel corresponding to an end point of the candidate motion vector my in the original image F n is expressed by Expression 3 below.
- a pixel value “prev” of a pixel corresponding to a start point of the candidate motion vector my in the original image F n-1 is expressed by Expression 4 below.
- F n (x, y) represents a pixel value of the pixel at coordinates (x, y) in the original image F n .
- F n-1 (x, y) represents a pixel value of the pixel at coordinates (x, y) in the original image F n-1 .
- the local motion vector detection unit 310 detects a candidate motion vector extending from one of the two blocks having the highest correlation to the other block and sets the detected vector as a local motion vector. Specifically, the local motion vector detection unit 310 detects a candidate motion vector my that satisfies Expressions 5 and 6 and sets the detected candidate motion vector my as a local motion vector mv L .
- E SAD represents the sum of absolute difference values of values of corresponding pixels of the two blocks and is used as an evaluation value indicating the level of the correlation between those blocks.
- B represents the target block.
- minE SAD represents a minimum value of E SAD .
- argminE SAD (mv) represents a function that returns my obtained when E SAD takes a minimum value, that is, when the correlation is highest.
- the processing of evaluating the level of the correlation based on the sum of absolute difference values of pixel values is referred to as SAD (Sum of Absolute Difference estimation).
- SAD Sum of Absolute Difference estimation
- the local motion vector detection unit 310 can also obtain the level of the correlation by processing other than the SAD.
- the local motion vector detection unit 310 may use MPC (Maximum matching Pixel Count).
- MPC Maximum matching Pixel Count
- the MPC the number of pixels at which an absolute difference value of pixel values is equal to or smaller than a predetermined matching threshold is counted to evaluate the correlation level based on the count value.
- a candidate motion vector obtained when the count value is maximum is detected as the local motion vector.
- the local motion vector detection unit 310 obtains the local motion vector mv L for each of all the blocks of the interpolated image and obtains a forward motion vector v n — L and a backward motion vector v n-1 — L by using Expressions 7 and 8.
- mvX L is an x component of the local motion vector mv L and mvY L is a y component of the local motion vector mv L .
- the local motion vector detection unit 310 supplies the forward motion vector v n — L and the backward motion vector v n-1 — L , of each block to the normal interpolated image generation unit 320 .
- the normal interpolated image generation unit 320 generates an image to be interpolated between the original images F n and F n-1 , which is a normal interpolated image, based on the local motion vector mv L and the original images F n and F n-1 .
- the normal interpolated image generation unit 320 receives the forward motion vector v n — L and the backward motion vector v n-1 — L from the local motion vector detection unit 310 .
- the normal interpolated image generation unit 320 substitutes the received vectors into Expressions 9 and 10 below to obtain the pixel values “succ” and “prev” in the original images F n and F n-1 .
- the normal interpolated image generation unit 320 substitutes the pixel values acquired by Expressions 9 and 10 into Expression 11 below to calculate a pixel value “out”.
- the normal interpolated image generation unit 320 supplies an image formed of the pixel values “out”, which is a normal interpolated image MC_N, to the combination unit 370 .
- the high-frequency area extraction unit 330 extracts an area having a spatial frequency higher than a predetermined value from each of the original images F n and F n-1 , and sets the extracted areas as high-frequency areas HP n and HP n-1 . Further, the high-frequency area extraction unit 330 detects a difference value k in pixel value between the original images F n and F n-1 for each pixel. Subsequently, the high-frequency area extraction unit 330 amplifies each pixel value of the extracted high-frequency areas HP n and HP n-1 in accordance with the difference value k, to generate high-frequency areas HPK n and HPK n-1 . The high-frequency area extraction unit 330 supplies those high-frequency areas HPK n and HPK n-1 to the global motion vector detection unit 340 and the combination ratio determination unit 360 .
- the global motion vector detection unit 340 detects a vector, i.e., a global motion vector, which indicates a direction and a distance in which the high-frequency area moves within a certain period of time.
- the global motion vector detection unit 340 receives the high-frequency areas HPK n and HPK n-1 from the high-frequency area extraction unit 330 and detects the vector indicating the moving direction of the whole of those areas, to set the detected vector as a global motion vector mv G .
- the global motion vector detection unit 340 sets the search range W for the whole of the high-frequency areas HPK n and HPK n-1 and detects a candidate motion vector that satisfies Expressions 6 and 12 below as the global motion vector mv G .
- MC represents the entire interpolated image.
- the global motion vector detection unit 340 substitutes the obtained global motion vector mv G into Expressions 13 and 14 below and obtains a forward motion vector V n — G and a backward motion vector V n-1 — G .
- mvX G represents an x component of the global motion vector mv G and mvY G represents a y component of the global motion vector mv G .
- the global motion vector detection unit 340 supplies the forward motion vector V n — c and the backward motion vector V n-1 — G to the high-frequency area interpolated image generation unit 350 and the combination ratio determination unit 360 .
- the global motion vector detection unit 340 is an example of a vector detection unit in the section “What is claimed is”.
- the high-frequency area interpolated image generation unit 350 generates an image to be interpolated between the original images F n and F n-1 , which is a high-frequency area interpolated image, based on the global motion vector mv G and the original images F n and F n-1 .
- the high-frequency area interpolated image generation unit 350 receives the forward motion vector v n — G and the backward motion vector V n-1 — G from the global motion vector detection unit 340 .
- the high-frequency area interpolated image generation unit 350 substitutes the received vectors into Expressions 15 and 16 below to obtain the pixel values “succ” and “prev” in the original images F n and F n-1 .
- the high-frequency area interpolated image generation unit 350 substitutes the pixel values acquired by Expressions 15 and 16 into Expression 11 to calculate a pixel value “out”.
- the high-frequency area interpolated image generation unit 350 supplies an image formed of the pixel values “out”, which is a high-frequency area interpolated image MC_HP n to the combination unit 370 .
- the combination ratio determination unit 360 determines a combination ratio of the normal interpolated image MC_N and the high-frequency area interpolated image MC_HP.
- the combination ratio determination unit 360 receives the high-frequency areas HPK n and HPK n-1 from the high-frequency area extraction unit 330 and combines the high-frequency areas HPK n and HPK n-1 to generate combination ratio data MIX.
- the combination ratio data MIX is image data that indicates the combination ratio of each pixel by pixel values.
- the combination ratio determination unit 360 supplies the combination ratio data MIX to the combination unit 370 .
- the combination unit 370 combines the normal interpolated image MC_N and the high-frequency area interpolated image MC_HP based on the combination ratio data MIX.
- the combination unit 370 causes the image memory 220 to store the combined image as the combined interpolated image MC_mix.
- FIGS. 4A , 4 B, and 4 C are each a diagram for describing a method of detecting a local motion vector in the first embodiment.
- FIG. 4A is an example of an original image 510 .
- FIG. 4B is an example of an original image 520 subsequent to the original image 510 .
- a block located at the same position as a block 511 in the original image 510 is selected as a target block.
- the block 511 and a predetermined number of blocks around the block 511 are set as a search range 512 .
- a block 521 corresponding to the block 511 and a predetermined number of blocks around the block 521 are set as a search range 522 .
- block matching between the blocks in the search range 512 and the blocks in the search range 522 is performed by using Expressions 5 and 6 and the like.
- a block 513 on the upper right of the search range 512 and a block 523 on the lower left of the search range 522 have the highest correlation. From this result, the vector indicating a direction and a distance from the block 513 to the block 523 is detected as a local motion vector 524 .
- FIG. 4C is a diagram showing an example of the forward motion vector and the backward motion vector.
- the local motion vector 524 is divided into the forward motion vector V n — L and the backward motion vector V n-1 — L by Expressions 7 and 8.
- FIG. 5 is a diagram showing an example of the local motion vector in the first embodiment.
- the local motion vector is obtained for each block, and each local motion vector is divided into the forward motion vector V n — L and the backward motion vector V n-1 — L .
- the forward motion vector V n — L and the backward motion vector V n-1 — L are obtained for each block.
- this vector is divided into a forward motion vector (—4, 0) and a backward motion vector (4, 0) by Expressions 1 and 2.
- FIG. 6 is a block diagram showing a configuration example of the normal interpolated image generation unit 320 in the first embodiment.
- the normal interpolated image generation unit 320 includes a forward motion compensation unit 321 , a backward motion compensation unit 322 , an adder 323 , and a multiplier 324 .
- the forward motion compensation unit 321 executes forward motion compensation processing to acquire each pixel value of coordinates (x, y) in the normal interpolated image based on the forward motion vector v n — L . Specifically, the pixel value “succ” in the original image F n , which is obtained by Expression 9, is acquired as a pixel value at the coordinates (x, y) in the normal interpolated image. The forward motion compensation unit 321 supplies the acquired pixel value to the adder 323 .
- the backward motion compensation unit 322 executes backward motion compensation processing to acquire each pixel value of coordinates (x, y) in the normal interpolated image based on the backward motion vector V n-1 — L .
- the pixel value “prev” in the original image F n-1 which is obtained by Expression 10, is acquired as a pixel value at the coordinates (x, y) in the normal interpolated image.
- the backward motion compensation unit 322 supplies the acquired pixel value to the adder 323 .
- the adder 323 adds the input values.
- the adder 323 receives an input of the pixel value “succ” acquired by the forward motion compensation unit 321 and an input of the pixel value “prev” acquired by the backward motion compensation unit 322 .
- the adder 323 adds those input values and supplies the resultant value to the multiplier 324 .
- the multiplier 324 multiplies the input value by a predetermined value.
- the multiplier 324 receives an input of the additional value supplied from the adder 323 .
- the multiplier 324 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out” in the normal interpolated image MC_N, to the combination unit 370 .
- the adder 323 and the multiplier 324 implement an arithmetic operation of Expression 11 when the interpolation position parameter of Relpos is 0.5.
- the normal interpolated image generation unit 320 executes both of the forward motion compensation processing and the backward motion compensation processing, but it may execute any one of the forward motion compensation processing and the backward motion compensation processing.
- the normal interpolated image generation unit 320 includes one of the forward motion compensation unit 321 and the backward motion compensation unit 322 and does not include the other motion compensation unit, the adder 323 , and the multiplier 324 . Subsequently, a pixel value generated by any one of the forward motion compensation unit 321 and the backward motion compensation unit 322 is output without change to the combination unit 370 , to serve as the pixel value of the normal interpolated image.
- FIGS. 7A , 7 B, and 7 C are each a diagram for describing a method of generating a normal interpolated image in the first embodiment.
- FIG. 7A is an example of a forward motion compensation image 530 generated by the forward motion compensation unit 321 .
- the forward motion compensation unit 321 uses Expression 9 to acquire, from the original image F n , a pixel value “succ” of a pixel that is located at a position to which a pixel at coordinates (x, y) in a target block 541 of the forward motion compensation image 530 is moved along the forward motion vector V n — L .
- This pixel value “succ” is used as a pixel value of the pixel at coordinates (x, y) in the target block 541 .
- the forward motion compensation image 530 is generated using a pixel obtained by moving the pixel of the original image F n according to the forward motion vector V n — L .
- FIG. 7B is an example of a backward motion compensation image 540 generated by the backward motion compensation unit 322 .
- the backward motion compensation unit 322 uses Expression 10 to acquire, from the original image F n-1 , a pixel value “prev” of a pixel that is located at a position to which a pixel at coordinates (x, y) in a target block 531 of the backward motion compensation image 540 is moved along the backward motion vector V n-1 — L .
- This pixel value “prev” is used as a pixel value of the pixel at coordinates (x, y) in the target block 541 .
- the backward motion compensation image 540 is generated using a pixel obtained by moving the pixel of the original image F n-1 according to the backward motion vector V n-1 — L .
- FIG. 7C is a diagram showing an example of a normal interpolated image 550 generated by combining the forward motion compensation image 530 shown in FIG. 7A and the backward motion compensation image 540 shown in FIG. 7B based on Expression 11.
- the forward motion vector V n — L and the backward motion vector V n-1 — L are vectors obtained when the local motion vector mv L is divided. Consequently, images in which pixels are moved according to those forward motion vector V n and backward motion vector V n-1 are combined to each other, and a block located between the blocks moved according to the local motion vector mv L is interpolated.
- FIG. 8 is a block diagram showing a configuration example of the high-frequency area extraction unit 330 in the first embodiment.
- the high-frequency area extraction unit 330 includes a low-pass filter 331 , a subtracter 332 , a multiplier 333 , an image memory 334 , and a difference detection unit 335 .
- the low-pass filter 331 extracts a low-frequency area LP n from the original image F n , the low-frequency area LP n having a spatial frequency lower than a predetermined value.
- the low-pass filter 331 is achieved by a FIR (Finite Impulse Response) filter, for example.
- the low-pass filter 331 supplies a pixel value of each pixel in the extracted low-frequency area LP n to the subtracter 332 .
- the subtracter 332 subtracts one of two input values from the other value.
- the subtracter 332 sequentially receives an input of the pixel value in the low-frequency area LP n extracted by the low-pass filter 331 and an input of the pixel value in the original image F n .
- the subtracter 332 subtracts the pixel value in the low-frequency area LP n from the pixel value in the original image F n and supplies the resultant value, which is a pixel value of the high-frequency area HP n , to the multiplier 333 .
- the high-frequency area extraction unit 330 extracts the high-frequency area HP n by the low-pass filter 331 and the subtracter 332 , but the high-frequency area extraction unit 330 may include a high-pass filter instead of the low-pass filter 331 and the subtracter 332 and extract the high-frequency area HP n by performing high-pass filtering on the original image F n . Further, the low-pass filter 331 and the subtracter 332 are examples of a filter unit in the section “What is claimed is”.
- the difference detection unit 335 detects a difference in pixel value between the original images F n and F n-1 for each pixel.
- the difference detection unit 335 detects a difference value k for each pixel.
- the difference detection unit 335 generates difference data K formed of those difference values k.
- the difference detection unit 335 sequentially supplies each of the difference values k in the difference data K to the multiplier 333 .
- the multiplier 333 multiplies the input values.
- the multiplier 333 receives an input of the difference value k in the difference data K n and an input of the pixel value in the high-frequency area HP n .
- the multiplier 333 multiplies those input values and supplies the resultant value, which is a pixel value in the high-frequency area HPK n , to the global motion vector detection unit 340 and the combination ratio determination unit 360 . Further, the multiplier 333 stores the multiplied value in the image memory 334 . Note that the multiplier 333 is an example of an increase unit in the section “What is claimed is”.
- the multiplier 333 multiplies the pixel value of the high-frequency area HP n by the difference value k, so that the pixel value of the high-frequency area that moves between the original images F n and F n-1 is amplified. Consequently, the global motion vector detection unit 340 can easily detect a vector, which indicates a moving direction of the high-frequency area with motion, as a global motion vector. Further, since the pixel value of the high-frequency area with motion is amplified, the combination ratio of the high-frequency area with motion can be preferentially increased.
- the image memory 334 stores the high-frequency area.
- the global motion vector detection unit 340 and the combination ratio determination unit 360 read out the high-frequency area stored in the image memory 334 when the next high-frequency area is generated, so that a delayed high-frequency area HPK n-1 can be acquired.
- FIG. 9 is a block diagram showing a configuration example of the difference detection unit 335 in the first embodiment.
- the difference detection unit 335 includes a subtracter 336 , an absolute value acquisition unit 337 , and a difference adjustment filter 338 .
- the subtracter 336 subtracts one of the two input values from the other value.
- the subtracter 336 sequentially receives an input of a pixel value of a pixel in the original image F n and an input of a pixel value of a pixel in the original image F n-1 .
- the subtracter 336 subtracts one of those input values from the other value and supplies the resultant value, i.e., a difference of the pixel values, to the absolute value acquisition unit 337 .
- the absolute value acquisition unit 337 acquires an absolute value of the input value.
- the absolute value acquisition unit 337 sequentially receives an input of the difference supplied from the subtracter 336 .
- the absolute value acquisition unit 337 supplies data formed of absolute values diff of the differences, which serves as difference absolute value data DIF n to the difference adjustment filter 338 .
- the difference adjustment filter 338 adjusts each absolute value diff in the difference absolute value data DIF. For example, each time an absolute value is input, the difference adjustment filter 338 uses the following expression to calculate a value obtained by amplifying the absolute value, and sets the resultant value as a difference value k. The difference adjustment filter 338 outputs data formed of those difference values k, as difference data K, to the multiplier 333 .
- Th d1 is a threshold larger than 0 and smaller than Th d2
- Th d2 is a threshold larger than Th d1 and smaller than 255.
- Expression 17 assumes that the pixel value has a value expressed in 8 bits, specifically, a value ranging from 0 to 255, but the number of bits assigned to the pixel value is not limited to 8 bits.
- the pixel value may be expressed in 10 bits. In such a case, in Expression 17, “255” is replaced with “1023”.
- the difference adjustment filter 338 amplifies the absolute value, so that the pixel value of the high-frequency area with motion is increased. Consequently, the global motion vector can be easily detected and the combination ratio of the high-frequency area with motion is preferentially set to be high.
- the difference adjustment filter 338 amplifies the absolute value of the difference, but the difference adjustment filter 338 may output the absolute value as the difference value k without performing amplification. Further, the difference adjustment filter 338 may attenuate the absolute value.
- the absolute value of the difference is extremely large, there is a possibility that motion of a noise component is erroneously detected as a global motion vector. In this case, the attenuation of the absolute value suppresses an occurrence of a false detection.
- the absolute value of the difference is extremely large, there is a possibility that an inappropriate combination ratio is determined based on the noise component. In this case, the attenuation of the absolute value allows an appropriate combination ratio to be determined.
- FIG. 10 is a block diagram showing a configuration example of the high-frequency area interpolated image generation unit 350 in the first embodiment.
- the high-frequency area interpolated image generation unit 350 includes a forward motion compensation unit 351 , a backward motion compensation unit 352 , an adder 353 , and a multiplier 354 .
- the configuration of the forward motion compensation unit 351 is the same as that of the forward motion compensation unit 321 in the normal interpolated image generation unit 320 except that the forward motion compensation processing is executed based on the forward motion vector v n — G instead of the forward motion vector V n — L .
- the forward motion compensation unit 351 supplies each pixel value “succ” acquired by the forward motion compensation processing to the adder 353 .
- the configuration of the backward motion compensation unit 352 is the same as that of the backward motion compensation unit 322 in the normal interpolated image generation unit 320 except that the backward motion compensation processing is executed based on the backward motion vector v n-1 — G instead of the backward motion vector V n-1 — L .
- the backward motion compensation unit 352 supplies each pixel value “prev” acquired by the backward motion compensation processing to the adder 353 .
- the adder 353 adds the input values.
- the adder 353 receives an input of the pixel value “succ” acquired by the forward motion compensation unit 351 and an input of the pixel value “prev” acquired by the backward motion compensation unit 352 .
- the adder 353 adds those input values and supplies the resultant value to the multiplier 354 .
- the multiplier 354 multiplies the input value by a predetermined value.
- the multiplier 354 receives an input of the additional value supplied from the adder 353 .
- the multiplier 354 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out” in the high-frequency area interpolated image MC_HP, to the combination unit 370 .
- the high-frequency area interpolated image generation unit 350 executes both of the forward motion compensation processing and the backward motion compensation processing, but it may execute any one of the forward motion compensation processing and the backward motion compensation processing.
- the high-frequency area interpolated image generation unit 350 includes one of the forward motion compensation unit 351 and the backward motion compensation unit 352 and does not include the other motion compensation unit, the adder 353 , and the multiplier 354 . Subsequently, a pixel value generated by any one of the forward motion compensation unit 351 and the backward motion compensation unit 352 is output without change to the combination unit 370 , to serve as the pixel value of the normal interpolated image.
- FIG. 11 is a block diagram showing a configuration example of the combination ratio determination unit 360 in the first embodiment.
- the combination ratio determination unit 360 includes a forward motion compensation unit 361 , a backward motion compensation unit 362 , a multiplier 363 , a gain adjustment unit 364 , and an interpolation filter 365 .
- the configuration of the forward motion compensation unit 361 is the same as that of the forward motion compensation unit 351 in the high-frequency area interpolated image generation unit 350 except that the forward motion compensation processing is executed based on the high-frequency area HPK n instead of the original image F n .
- the forward motion compensation unit 361 supplies each pixel value “succ” acquired by the forward motion compensation processing to the multiplier 363 .
- the configuration of the backward motion compensation unit 362 is the same as that of the backward motion compensation unit 352 in the high-frequency area interpolated image generation unit 350 except that the backward motion compensation processing is executed based on the high-frequency area HPK n-1 instead of the original image F n-1 .
- the backward motion compensation unit 362 supplies each pixel value “prev” acquired by the backward motion compensation processing to the multiplier 363 .
- the multiplier 363 multiplies the input values.
- the multiplier 363 receives an input of the pixel value “succ” acquired by the forward motion compensation unit 361 and an input of the pixel value “prev” acquired by the backward motion compensation unit 362 .
- the multiplier 363 multiplies those input values and supplies the resultant value to the gain adjustment unit 364 .
- the multiplier 363 multiplies the pixel value “succ” and the pixel value “prev”, so that the pixel value of the high-frequency area is amplified more than when the input values are added. Consequently, the combination ratio of the high-frequency area is preferentially set to be high.
- the gain adjustment unit 364 adjusts a gain and attenuates the pixel value supplied from the multiplier 363 .
- the gain adjustment unit 364 attenuates the pixel value from the multiplier 363 such that the value falls within a certain range through the gain adjustment. For example, when a pixel value has the data size of 8 bits and such pixel values are multiplied, the data size of the resultant value after the multiplication is 16 bits. The multiplication result is attenuated such that its data size is changed to 8 bits.
- the gain adjustment unit 364 supplies the adjusted pixel value to the interpolation filter 365 .
- the interpolation filter 365 interpolates, in an image MC_M formed of the pixel values generated by the gain adjustment unit 364 , a portion where the edge interrupts, to generate the combination ratio data MIX.
- the interpolation filter 365 uses the following expression to perform interpolation, for example.
- the interpolation filter 365 supplies the generated combination ratio data MIX to the combination unit 370 .
- MIX( x,y ) max ⁇ MC — M ( x ⁇ 1, y ) MC — M ( x,y ) MC — M ( x+ 1, y ) ⁇ Expression 18
- MC_M(x, y) represents a pixel value of a pixel at coordinates (x, y) in the image MC_M.
- MIX(x, y) represents a pixel value of a pixel at coordinates (x, y) in the combination ratio data MIX.
- max( ) represents a maximum value of the pixel value in parentheses.
- FIG. 12 is a block diagram showing a configuration example of the combination unit 370 in the first embodiment.
- the combination unit 370 includes a level adjustment unit 371 , a multiplier 372 , a subtracter 373 , a multiplier 374 , and an adder 375 .
- the level adjustment unit 371 adjusts the pixel value in the combination ratio data MIX so as to fall within the range from 0 to 1. For example, when the pixel value takes a value within the range from 0 to 255, the level adjustment unit 371 divides the pixel value by 255 to adjust the resultant value to a value falling within the range from 0 to 1. The level adjustment unit 371 sequentially supplies each of the adjusted values, which is a combination ratio mix, to the subtracter 373 and the multiplier 372 .
- the multiplier 372 multiplies the input values.
- the multiplier 372 receives an input of the combination ratio mix supplied from the level adjustment unit 371 and an input of the pixel value in the high-frequency area interpolated image MC_HP.
- the multiplier 372 multiplies those input values and supplies the resultant value to the adder 375 .
- the subtracter 373 subtracts the input value from a predetermined value.
- the subtracter 373 receives an input of the combination ratio mix supplied from the level adjustment unit 371 .
- the subtracter 373 subtracts the combination ratio mix from 1.0 and supplies the resultant value (1.0-mix) to the multiplier 374 .
- the multiplier 374 multiplies the input values.
- the multiplier 374 receives an input of the value (1.0-mix) supplied from the subtracter 373 and an input of the pixel value in the normal interpolated image MC_N.
- the multiplier 374 multiplies those input values and supplies the resultant value to the adder 375 .
- the adder 375 adds the input values.
- the adder 375 receives inputs of the multiplication results of the respective multipliers 372 and 374 .
- the adder 375 adds those input values and stores, in the image memory 220 , the resultant value serving as a pixel value in the combined interpolated image MC_mix.
- MC_mix(x, y) represents a pixel value of a pixel at coordinates (x, y) in the combined interpolated image MC_mix.
- MC_N(x, y) represents a pixel value of a pixel at coordinates (x, y) in the normal interpolated image MC_N.
- MC_HP(x, y) represents a pixel value of a pixel at coordinates (x, y) in the high-frequency area interpolated image MC_HP.
- mix represents a combination ratio generated by the level adjustment unit 371 based on the pixel value at coordinates (x, y) in the combination ratio data MIX.
- FIG. 13 is a diagram showing an example of a normal interpolated image 610 in the first embodiment.
- the image processing apparatus 200 receives an input of a moving image in which a character telop, which moves in a horizontal direction with the lapse of time, is displayed on a lower portion of a screen.
- the image processing apparatus 200 generates the normal interpolated image 610 from the original images F n and F n-1 in the moving image.
- the normal interpolated image 610 includes a character telop 611 in its lower portion. Note that characters are also displayed above the character telop 611 , but those character portions do not move with the lapse of time.
- the image processing apparatus 200 detects a local motion vector for each block and moves the block in the original image based on the local motion vector, to generate the normal interpolated image 610 .
- the area of the character telop the entire area moves in the horizontal direction, and thus all the values of the local motion vectors in the character telop are thought to be the same.
- a movement amount of the block for which the local motion vector detection has failed is different from an actual movement amount of the character telop, and thus a blur occurs in a boundary between the block for which the detection has failed and a block for which the detection has not failed.
- blurs occur at a plurality of portions such as blurs 612 and 613 .
- FIG. 14 is a diagram showing an example of a high-frequency area interpolated image 620 in the first embodiment.
- the image processing apparatus 200 moves the entire original image according to the global motion vectors each indicating the movement of a character telop 621 and generates the high-frequency area interpolated image 620 , so that no blurs occur in the character telop 621 .
- the entire original image is moved according to the global motion vectors irrespective of no movements at portions other than the character telop 621 , blurs occur at the portions other than the character telop 621 .
- FIG. 15 is a diagram showing an example of combination ratio data 630 in the first embodiment.
- the combination ratio data 630 is image data including, in the high-frequency area with motion, a pixel whose pixel value is amplified. For that reason, a pixel value of the character telop portion extracted as a high-frequency area with motion is higher than those of the other portions. Consequently, the value of the combination ratio mix in the character telop portion is set to be higher than those of the other portions.
- FIG. 16 is a diagram showing an example of a combined interpolated image 640 in the first embodiment.
- the image processing apparatus 200 combines the normal interpolated image 610 and the high-frequency area interpolated image 620 based on the combination ratio data 630 .
- the combination ratio of the character telop portion is set to be higher. Consequently, a combined interpolated image 640 in which the ratio of the high-frequency area interpolated image 620 is higher in a character telop 631 and the ratio of the normal interpolated image 610 is higher in the other portions is generated.
- FIG. 17 is a flowchart showing an example of an operation of the image processing apparatus 200 in the first embodiment. This operation is started when the image processing apparatus 200 executes the processing to convert the frame rate, for example.
- the image processing apparatus 200 determines whether an original image is input or not (Step S 910 ). When an original image is input (Step S 910 : Yes), the image processing apparatus 200 executes interpolated image generation processing to generate a combined interpolated image (Step S 920 ). Subsequently, the image processing apparatus 200 outputs the interpolated image and the original image in a time-series order (Step S 930 ). When an original image is not input (Step S 910 : No), or after the processing of Step S 930 is performed, the image processing apparatus 200 returns to the processing of Step S 910 .
- FIG. 18 is a flowchart showing an example of the interpolated image generation processing in the first embodiment.
- the interpolated image generation unit 300 of the image processing apparatus 200 detects a local motion vector for each block (Step S 921 ). Subsequently, the interpolated image generation unit 300 generates a normal interpolated image from original images based on the local motion vector (Step S 922 ).
- the interpolated image generation unit 300 extracts a high-frequency area from the original images (Step S 923 ). Subsequently, the interpolated image generation unit 300 detects a motion vector of the high-frequency area as a global motion vector (Step S 924 ). The interpolated image generation unit 300 generates a high-frequency area interpolated image from the original images based on the global motion vector (Step S 925 ).
- the interpolated image generation unit 300 determines a combination ratio for each pixel based on the high-frequency areas (Step S 926 ). Subsequently, the interpolated image generation unit 300 combines the normal interpolated image and the high-frequency area interpolated image according to the combination ratio (Step S 927 ). After Step S 927 , the interpolated image generation unit 300 terminates the interpolated image generation processing.
- the image processing apparatus 200 combines the high-frequency area interpolated image and the normal interpolated image that are generated based on the change in position of the high-frequency area, to generate an interpolated image with no blurs in the high-frequency area. Additionally, the image processing apparatus 200 combines the normal interpolated image and the high-frequency area interpolated image, to generate an interpolated image in which a boundary between the high-frequency area and another area looks natural. Such an interpolated image is interpolated between the original images. This allows an improvement of the image quality of a moving image whose frame rate is converted.
- the image processing apparatus 200 can also generate the normal interpolated image without detecting the local motion vector.
- the image processing apparatus 200 in a modified example is different from that of the first embodiment in that the local motion vector is not detected.
- FIG. 19 is a block diagram showing a configuration example of the interpolated image generation unit 300 in the modified example of the first embodiment.
- the interpolated image generation unit 300 in the modified example is different from that of the first embodiment in that a normal interpolated image generation unit 325 is provided instead of the local motion vector detection unit 310 and the normal interpolated image generation unit 320 .
- the normal interpolated image generation unit 325 combines the original images F n and F n-1 to generate the normal interpolated image MC_N.
- FIG. 20 is a block diagram showing a configuration example of the normal interpolated image generation unit 325 in the modified example of the first embodiment.
- the normal interpolated image generation unit 325 includes an adder 323 and a multiplier 324 .
- the adder 323 adds the input values.
- the adder 323 receives an input of a pixel value in the original image F n and an input of a pixel value in the original image F n-1 .
- the adder 323 adds those input values and supplies the resultant value to the multiplier 324 .
- the multiplier 324 multiplies the input value by a predetermined value.
- the multiplier 324 receives an input of the additional value supplied from the adder 323 .
- the multiplier 324 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out”, to the combination unit 370 .
- the normal interpolated image generation unit 325 generates a normal interpolated image without executing the motion compensation processing.
- the normal interpolated image generation unit 325 combines the original images F n and F n-1 to generate the normal interpolated image, but it may use one of the original images without change as the normal interpolated image.
- FIG. 21 is a diagram showing an example of a normal interpolated image 660 in the modified example of the first embodiment.
- the normal interpolated image 660 is an image generated without performing a motion compensation, and hence blurs occur in a portion of a character telop 661 that moves in the horizontal direction.
- the normal interpolated image 660 is combined with a high-frequency area interpolated image having no blurs in the character telop. This suppresses the occurrence of blurs in a combined interpolated image that is to be eventually generated.
- the image processing apparatus 200 amplifies the pixel value of the pixel in the high-frequency area according to the difference.
- the difference may include a noise component and it is desirable to execute coring processing to remove the noise component before the amplification.
- the image processing apparatus 200 of the second embodiment is different from that of the first embodiment in that the coring processing is performed on the difference.
- FIG. 22 is a block diagram showing a configuration example of the interpolated image generation unit 300 in the second embodiment.
- the interpolated image generation unit 300 in the second embodiment is different from that of the first embodiment in that a high-frequency area extraction unit 380 is provided instead of the high-frequency area extraction unit 330 .
- the high-frequency area extraction unit 380 executes the coring processing to remove a value smaller than a difference threshold Th k1 from the difference values k, to generate a difference value k1. Additionally, the high-frequency area extraction unit 380 executes the coring processing to remove a value smaller than a difference threshold Th k2 from the difference values k, to generate a difference value k2.
- the high-frequency area extraction unit 380 amplifies the pixel values of the high-frequency areas HP n and HP n-1 in accordance with the difference value k1, to generate high-frequency areas HPK1 n and HPK1 n-1 . Additionally, the high-frequency area extraction unit 380 amplifies the pixel values of the high-frequency areas HP n and HP n-1 in accordance with the difference value k2, to generate high-frequency areas HPK2 n and HPK2 n-1 .
- the high-frequency area extraction unit 380 supplies the high-frequency areas HPK1 n and HPK1 n-1 to the global motion vector detection unit 340 and supplies the high-frequency areas HPK2 n and HPK2 n-1 to the combination ratio determination unit 360 .
- FIG. 23 is a block diagram showing a configuration example of the high-frequency area extraction unit 380 in the second embodiment.
- the high-frequency area extraction unit 380 is different from the high-frequency area extraction unit 330 in that coring processing units 381 and 382 , a multiplier 383 , and an image memory 384 are additionally provided.
- the coring processing unit 381 executes the coring processing to remove the difference value k smaller than the difference threshold Th k1 .
- the coring processing unit 381 uses the following expression to obtain the difference value k1 from the difference values k in the difference data K.
- the coring processing unit 381 supplies difference data K1 formed of the difference value k1 to the multiplier 333 .
- the coring processing unit 381 is an example of a first difference coring processing unit in the section “What is claimed is”.
- the multiplier 333 multiplies the pixel value in the high-frequency area HP n and the difference value k1 in the difference data K1.
- Data of the multiplication result, which serves as the high-frequency area HPK1 n is supplied to the global motion vector detection unit 340 .
- the coring processing unit 382 executes the coring processing to remove the difference value k smaller than the difference threshold Th k2 .
- the coring processing unit 382 uses the following expression to obtain the difference value k2 from the difference values k in the difference data K.
- the coring processing unit 382 supplies difference data K2 formed of the difference value k2 to the multiplier 383 .
- the coring processing unit 382 is an example of a second difference coring processing unit in the section “What is claimed is”.
- the multiplier 383 multiplies the input values.
- the multiplier 383 receives an input of the pixel value in the high-frequency area HP n and an input of the difference value k2 in the difference data K2.
- the multiplier 383 multiplies those input values, supplies data of the multiplication result, which serves as the high-frequency area HPK2 n , to the combination ratio determination unit 360 , and causes the image memory 384 to store the data.
- the high-frequency area extraction unit 380 executes both of the coring processing using the difference threshold Th k1 and the coring processing using the difference threshold Th k2 , but it may execute any one of the processing.
- the image processing apparatus 200 executes the coring processing to remove the difference smaller than the difference threshold, to remove the noise component from the difference. This allows the global motion vector to be easily detected and an appropriate combination ratio to be obtained.
- the image processing apparatus 200 performs the coring processing on the difference, but it may perform the coring processing on the high-frequency area.
- the image processing apparatus 200 in the modified example is different from that of the second embodiment in that the coring processing is performed on the high-frequency area.
- FIG. 24 is a block diagram showing a configuration example of the high-frequency area extraction unit 380 in the modified example of the second embodiment.
- the high-frequency area extraction unit 380 in the modified example is different from that of the second embodiment in that coring processing units 385 and 386 are provided instead of the coring processing units 381 and 382 .
- the coring processing unit 385 executes coring processing to remove a pixel having a pixel value smaller than a pixel threshold Th p1 in the high-frequency area HP n .
- the coring processing unit 385 sequentially supplies the pixel value in the high-frequency area HP n , which is obtained after the coring processing, to the multiplier 333 .
- the multiplier 333 multiplies the difference value k and the pixel value supplied from the coring processing unit 385 .
- the coring processing unit 386 executes coring processing to remove a pixel having a pixel value smaller than a pixel threshold Th p2 in the high-frequency area HP n .
- the coring processing unit 386 sequentially supplies the pixel value in the high-frequency area HP n , which is obtained after the coring processing, to the multiplier 383 .
- the multiplier 383 multiplies the difference value k and the pixel value supplied from the coring processing unit 386 .
- the high-frequency area extraction unit 380 executes both of the coring processing using the pixel threshold Th p1 and the coring processing using the pixel threshold Th p2 , but it may execute any one of the processing.
- the coring processing unit 385 is an example of a first pixel coring processing unit in the section “What is claimed is”.
- the coring processing unit 386 is an example of a second pixel coring processing unit in the section “What is claimed is”.
- the image processing apparatus 200 performs the coring processing on the high-frequency area HP n to remove a noise component in the high-frequency area HP n .
- processing procedures described in the above embodiments may be considered as a method including a series of those procedures, or may be considered as a program causing a computer to execute the series of those procedures or as a recording medium storing the program.
- the recording medium include a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disk), a memory card, and a blu-ray disc (Blu-ray Disc (registered trademark)).
- An image processing apparatus including:
- a normal interpolated image generation unit configured to generate an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- a high-frequency area extraction unit configured to extract a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images
- a high-frequency area interpolated image generation unit configured to generate an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images;
- a combination unit configured to execute combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
- the image processing apparatus further including a vector detection unit configured to detect a vector indicating a direction and a distance in which a position of the high-frequency area changes within a certain period of time on the time series, in which
- the high-frequency area interpolated image generation unit is configured to generate the high-frequency area interpolated image based on the vector and each of the plurality of original images.
- the image processing apparatus further including a combination ratio determination unit configured to determine a ratio at which the normal interpolated image and the high-frequency area interpolated image are combined, for each pixel in accordance with each pixel value of pixels in the high-frequency area, in which
- the combination unit is configured to execute the combining processing according to the ratio.
- the high-frequency area extraction unit includes
- the high-frequency area extraction unit includes a first difference coring processing unit configured to execute first difference coring processing to remove a difference smaller than a first difference threshold from the detected differences, and
- the increase unit is configured to
- the high-frequency area extraction unit includes a second difference coring processing unit configured to execute second difference coring processing to remove a difference smaller than a second difference threshold from the detected differences, and
- the increase unit is configured to
- the high-frequency area extraction unit includes a first pixel coring processing unit configured to execute first pixel coring processing to remove a pixel with a pixel value smaller than a first pixel value threshold in the extracted high-frequency area, and
- the increase unit is configured to
- the high-frequency area extraction unit includes a second pixel coring processing unit configured to execute second pixel coring processing to remove a pixel with a pixel value smaller than a second pixel value threshold in the extracted high-frequency area, and
- the increase unit is configured to
- each of the plurality of original images includes a plurality of blocks each having a predetermined shape
- the normal interpolated image generation unit is configured to generate the normal interpolated image based on a change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images.
- An image processing method including:
- a normal interpolated image generation unit generating, by a normal interpolated image generation unit, an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- a high-frequency area interpolated image generation unit generating, by a high-frequency area interpolated image generation unit, an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images;
- a combination unit executing, by a combination unit, combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
- a normal interpolated image generation unit generating, by a normal interpolated image generation unit, an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- a high-frequency area interpolated image generation unit generating, by a high-frequency area interpolated image generation unit, an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images;
- a combination unit executing, by a combination unit, combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Television Systems (AREA)
- Image Processing (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
Description
- This application claims the benefit of Japanese Priority Patent Application JP 2012-277561 filed Dec. 20, 2012, the entire contents of which are incorporated herein by reference.
- The present disclosure relates to an image processing apparatus, an image processing method, and a program causing a computer to execute the image processing method. Specifically, the present disclosure relates to an image processing apparatus and an image processing method with which the number of frames per unit time in a moving image is changed and to a program causing a computer to execute the image processing method.
- In image processing apparatuses in related art such as television receivers, a frame rate conversion to change the number of frames per unit time, i.e., a frame rate, in a moving image is performed in some cases. For example, in order to smoothen the motion of an object in a moving image, a frame rate conversion to increase a frame rate, which is called an up-conversion, is performed.
- In such an up-conversion, a frame interpolation method in which an interpolated frame is generated from original frames and inserted between the original frames is widely used. In the frame interpolation method, motion compensation processing to detect the motion of an object in the original frame and generate an interpolated frame based on the motion is performed. This allows a natural interpolation to be performed in a moving image including a moving object.
- In performing the motion compensation processing, a method of detecting the motion of an object on a block-to-block basis or a pixel-to-pixel basis is widely used, but the method may cause a possibility that the motion detection fails in some blocks or pixels. If the object whose motion has failed to be detected is a high-frequency component, e.g., a character telop, a defect easily occurs in an image with a part at which the detection has failed. For example, since an object including a high-frequency component has a clear boundary with the background, blurs occur at the part at which the detection has failed. On the other hand, since an object including a low-frequency component has a blurred boundary, blurs are not liable to occur if the motion detection fails.
- To suppress the defect involving the failure of the motion detection, the following image processing apparatus is provided. The image processing apparatus detects an area including a high-frequency component with motion to be an area for character telop and changes an interpolation method for such an area (see, for example, Japanese Patent Application Laid-open No. 2009-296284). The image processing apparatus detects motion of the entire area in the area of the character telop and detects motion on a block-to-block basis or a pixel-to-pixel basis in the other areas. Subsequently, along the direction of the detected motion, the entire area of the character telop in the original frame is displaced, and in areas other than the character telop, the original frame is displaced for each block or pixel, to generate an interpolated frame. The entire area of the character telop is displaced, so that no blurs occur in the area.
- In the above-mentioned technique in related art, however, the image quality of the moving image after the frame rate conversion may deteriorate. Specifically, the image processing apparatus performs different types of processing on the area of the character telop and on the other areas, and hence there is a possibility that the boundaries between those areas look unnatural.
- This leads to a possibility that the image quality deteriorates.
- The present disclosure has been made in view of the circumstances as described above, and it is desirable to improve the image quality of a moving image whose frame rate is converted.
- According to an embodiment of the present disclosure, there are provided an image processing apparatus, an image processing method therefor, and a program causing a computer to execute the image processing method. The image processing apparatus includes a normal interpolated image generation unit, a high-frequency area extraction unit, a high-frequency area interpolated image generation unit, and a combination unit. The normal interpolated image generation unit is configured to generate an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images. The high-frequency area extraction unit is configured to extract a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images. The high-frequency area interpolated image generation unit is configured to generate an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images. The combination unit is configured to execute combining processing to combine the normal interpolated image and the high-frequency area interpolated image. This provides effects of generating the normal interpolated image based on each of the plurality of original images, generating the high-frequency area interpolated image based on the change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images, and combining the normal interpolated image and the high-frequency area interpolated image.
- Further, in the embodiment of the present disclosure, the image processing apparatus may further include a vector detection unit configured to detect a vector indicating a direction and a distance in which a position of the high-frequency area changes within a predetermined period of time on the time series, and the high-frequency area interpolated image generation unit may be configured to generate the high-frequency area interpolated image based on the vector and each of the plurality of original images. This provides an effect of generating the high-frequency area interpolated image based on the vector indicating the direction and the distance in which the position of the high-frequency area changes and on each of the plurality of original images.
- Further, in the embodiment of the present disclosure, the image processing apparatus may further include a combination ratio determination unit configured to determine a ratio at which the normal interpolated image and the high-frequency area interpolated image are combined, for each pixel in accordance with each pixel value of pixels in the high-frequency area, and the combination unit may be configured to execute the combining processing according to the ratio. This provides an effect of executing the combining processing according to the ratio determined for each pixel in accordance with each pixel value of the pixels in the high-frequency area.
- Further, in the embodiment of the present disclosure, the high-frequency area extraction unit may include a filter unit, a difference detection unit, and an increase unit. The filter unit is configured to extract the high-frequency area in each of the plurality of original images. The difference detection unit is configured to detect, for each pixel, a difference in pixel value between two adjacent original images of the plurality of original images. The increase unit is configured to increase each pixel value of the pixels in the high-frequency area in each of the two adjacent original images in accordance with the difference, and supply the increased pixel value to the vector detection unit and the combination ratio determination unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference.
- Further, in the embodiment of the present disclosure, the high-frequency area extraction unit may include a first difference coring processing unit configured to execute first difference coring processing to remove a difference smaller than a first difference threshold from the detected differences, and the increase unit may be configured to increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the first difference coring processing is performed, and supply the increased pixel value to the vector detection unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference on which the first difference coring processing is performed.
- Further, in the embodiment of the present disclosure, the high-frequency area extraction unit may include a second difference coring processing unit configured to execute second difference coring processing to remove a difference smaller than a second difference threshold from the detected differences, and the increase unit may be configured to increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the second difference coring processing is performed, and supply the increased pixel value to the combination ratio determination unit. This provides an effect of increasing each pixel value of the pixels in the high-frequency area in accordance with the difference on which the second difference coring processing is performed.
- Further, in the embodiment of the present disclosure, the high-frequency area extraction unit may include a first pixel coring processing unit configured to execute first pixel coring processing to remove a pixel with a pixel value smaller than a first pixel value threshold in the extracted high-frequency area, and the increase unit may be configured to increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the first pixel coring processing is performed, and supply the increased pixel value to the vector detection unit. This provides an effect of increasing, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the first pixel coring processing is performed.
- Further, in the embodiment of the present disclosure, the high-frequency area extraction unit may include a second pixel coring processing unit configured to execute second pixel coring processing to remove a pixel with a pixel value smaller than a second pixel value threshold in the extracted high-frequency area, and the increase unit may be configured to increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the second pixel coring processing is performed, and supply the increased pixel value to the combination ratio determination unit. This provides an effect of increasing, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the second pixel coring processing is performed.
- Further, in the embodiment of the present disclosure, each of the plurality of original images may include a plurality of blocks each having a predetermined shape, and the normal interpolated image generation unit may be configured to generate the normal interpolated image based on a change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images. This provides an effect of generating a normal interpolated image based on the change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images.
- Further, in the embodiment of the present disclosure, the image processing apparatus may further include a selection unit configured to select the combined image by the combination unit and the plurality of original images in order of the time series and output the selected image. This provides an effect of selecting the combined image and the plurality of original images in order of the time series and outputting the selected image.
- According to the present disclosure, it is possible to produce an excellent effect of improving the image quality of a moving image whose frame rate is converted.
- These and other objects, features and advantages of the present disclosure will become more apparent in light of the following detailed description of best mode embodiments thereof, as illustrated in the accompanying drawings.
-
FIG. 1 is a block diagram showing a configuration example of an information processing system in a first embodiment; -
FIG. 2 is a block diagram showing a configuration example of an image processing apparatus in the first embodiment; -
FIG. 3 is a block diagram showing a configuration example of an interpolated image generation unit in the first embodiment; -
FIGS. 4A , 4B, and 4C are each a diagram for describing a method of detecting a local motion vector in the first embodiment; -
FIG. 5 is a diagram showing an example of the local motion vector in the first embodiment; -
FIG. 6 is a block diagram showing a configuration example of a normal interpolated image generation unit in the first embodiment; -
FIGS. 7A , 7B, and 7C are each a diagram for describing a method of generating a normal interpolated image in the first embodiment; -
FIG. 8 is a block diagram showing a configuration example of a high-frequency area extraction unit in the first embodiment; -
FIG. 9 is a block diagram showing a configuration example of a difference detection unit in the first embodiment; -
FIG. 10 is a block diagram showing a configuration example of a high-frequency area interpolated image generation unit in the first embodiment; -
FIG. 11 is a block diagram showing a configuration example of a combination ratio determination unit in the first embodiment; -
FIG. 12 is a block diagram showing a configuration example of a combination unit in the first embodiment; -
FIG. 13 is a diagram showing an example of the normal interpolated image in the first embodiment; -
FIG. 14 is a diagram showing an example of a high-frequency area interpolated image in the first embodiment; -
FIG. 15 is a diagram showing an example of combination ratio data in the first embodiment; -
FIG. 16 is a diagram showing an example of a combined interpolated image in the first embodiment; -
FIG. 17 is a flowchart showing an example of an operation of the image processing apparatus in the first embodiment; -
FIG. 18 is a flowchart showing an example of interpolated image generation processing in the first embodiment; -
FIG. 19 is a block diagram showing a configuration example of an interpolated image generation unit in a modified example of the first embodiment; -
FIG. 20 is a block diagram showing a configuration example of a normal interpolated image generation unit in the modified example of the first embodiment; -
FIG. 21 is a diagram showing an example of a normal interpolated image in the modified example of the first embodiment; -
FIG. 22 is a block diagram showing a configuration example of an interpolated image generation unit in a second embodiment; -
FIG. 23 is a block diagram showing a configuration example of a high-frequency area extraction unit in the second embodiment; and -
FIG. 24 is a block diagram showing a configuration example of a high-frequency area extraction unit in a modified example of the second embodiment. - Hereinafter, modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described. Description will be given in the following order.
- 1. First Embodiment (Example of Combining Normal Interpolated Image and High-Frequency Area Interpolated Image)
- 2. Second Embodiment (Example of Combining Normal Interpolated Image and High-Frequency Area Interpolated Image Generated from Difference Obtained After Coring Processing)
-
FIG. 1 is a block diagram showing a configuration example of an information processing system in a first embodiment. The information processing system includes a movingimage supply apparatus 100, animage processing apparatus 200, and adisplay apparatus 400. - The moving
image supply apparatus 100 supplies an input signal, which contains a moving image, to theimage processing apparatus 200 via asignal line 109. The moving image contains a plurality of frames (images) to be reproduced in a time-series order. Hereinafter, each of the images in the input signal is referred to as an “original image”. For example, the movingimage supply apparatus 100 receives a broadcast wave providing a moving image and acquires the moving image from the broadcast wave. When the frame rate of the moving image is represented by hertz (Hz), the frame rate of the moving image in the input signal is 60 hertz, for example. - Note that the moving
image supply apparatus 100 may acquire a moving image from a source other than the broadcast wave. For example, the movingimage supply apparatus 100 may acquire the moving image by reading out a moving image recorded on a recording medium such as a DVD (Digital Versatile Disk) or recorded in a storage device such as an HDD (Hard Disk Drive). - The
image processing apparatus 200 converts the frame rate of the moving image in the input signal and outputs an output signal, which contains the moving image whose frame rate is converted, to thedisplay apparatus 400 via asignal line 209. The frame rate of the moving image in the output signal is 120 hertz, for example. - In general, when a ratio of the frame rate of the input signal to the frame rate of the output signal is 1:m (m is a real number larger than 1) in a frame rate conversion, such a conversion is referred to as an mX speed frame rate conversion. For example, the conversion from 60 hertz to 120 hertz is a double-speed frame rate conversion. Note that the
image processing apparatus 200 may perform a frame rate conversion such as a quadruple-speed frame rate conversion from 30 hertz to 120 hertz, in addition to the double-speed frame rate conversion. - The
display apparatus 400 reproduces and displays the moving image contained in the output signal. - (Configuration Example of Image Processing Apparatus)
-
FIG. 2 is a block diagram showing a configuration example of theimage processing apparatus 200 in the first embodiment. Theimage processing apparatus 200 includesimage memories selector 240, and an interpolatedimage generation unit 300. - The
image memories image generation unit 300 and theselector 240 read out the original images stored in theimage memories - The interpolated
image generation unit 300 generates, based on the original images Fn and Fn-1, an image to be interpolated between those original images Fn and Fn-1. The generated image is a combined interpolated image MC_mix. The interpolatedimage generation unit 300 acquires the original image Fn from the input signal and acquires the delayed original image Fn-1 from theimage memory 210, to generate a combined interpolated image MC_mix based on those original images and cause theimage memory 220 to store the resultant image MC_mix. Theimage memory 220 is for storing the combined interpolated image MC_mix. - The
selector 240 selects and outputs the combined interpolated image MC_mix stored in theimage memory 220 and the original image stored in theimage memory 230 in a time-series order. - (Configuration Example of Interpolated Image Generation Unit)
-
FIG. 3 is a block diagram showing a configuration example of the interpolatedimage generation unit 300 in the first embodiment. The interpolatedimage generation unit 300 includes a local motionvector detection unit 310, a normal interpolatedimage generation unit 320, a high-frequencyarea extraction unit 330, a global motionvector detection unit 340, a high-frequency area interpolatedimage generation unit 350, a combinationratio determination unit 360, and acombination unit 370. - The local motion
vector detection unit 310 divides the original image into a plurality of blocks each having a predetermined shape and detects a vector, i.e., a local motion vector, which indicates a direction and a distance in which each of those blocks moves within a certain period of time. Specifically, the local motionvector detection unit 310 handles the original image as an image formed of square blocks each having 8×8 pixels, for example. The local motionvector detection unit 310 selects any one of the blocks of an interpolated image to use the block as a target block. The local motionvector detection unit 310 detects a block of the original image, which corresponds to the target block of the interpolated image, and then sets a search range W for an area in the original image, which falls within a certain range from the block of the original image. The search range W is a range used for searching for a local motion vector by block matching. For example, in the original image, the search range W is set for the target block and a predetermined number of blocks around the target block. - Here, a vector that is a candidate when a local motion vector is searched for is referred to as a candidate motion vector mv. The candidate motion vector my can be divided into vn and vn-1 by
Expressions 1 and 2 below. -
- In
Expressions 1 and 2, mvX represents an x component of the candidate motion vector mv, and mvY represents a y component of the candidate motion vector mv. Further, inExpressions 1 and 2, Relpos represents an interpolation position parameter that indicates an interpolation position. Here, the interpolation position is a reproduction position of each combined interpolated image MC_mix in the output signal on a time axis on which the moving image is reproduced. The interpolation position parameter or Relpos is a value obtained by dividing a time between the original image and the interpolated image by a time between two adjacent original images, for example. In the double-speed frame rate conversion, the time between the original image and the interpolated image is a half of the time between the two adjacent original images, and the interpolation position parameter of Relpos is “0.50”. Additionally, in the quadruple-speed frame rate conversion, three interpolated images are inserted between two adjacent original images, and the time between one of the original images and each interpolated image is ¼, ½, or ¾ of the time between the two original images. Consequently, the interpolation position parameter of Relpos of each of the interpolated images is “0.25”, “0.50”, or “0.75”. - When the candidate motion vector my passes through a pixel at coordinates (x, y) in the target block, a pixel value “succ” of a pixel corresponding to an end point of the candidate motion vector my in the original image Fn is expressed by Expression 3 below. Further, a pixel value “prev” of a pixel corresponding to a start point of the candidate motion vector my in the original image Fn-1 is expressed by Expression 4 below.
-
succ=F n(x+v x— n ,y+v y— n) Expression 3 -
prev=F n-1(x+v x— n-1 ,y+v y— n-1) Expression 4 - In Expression 3, Fn(x, y) represents a pixel value of the pixel at coordinates (x, y) in the original image Fn. In Expression 4, Fn-1 (x, y) represents a pixel value of the pixel at coordinates (x, y) in the original image Fn-1.
- Within the search range W, statistics of differences between the pixel values “prev” and the pixel values “succ” are obtained by Expressions 3 and 4, to obtain two blocks having the highest correlation within the search range W. The local motion
vector detection unit 310 detects a candidate motion vector extending from one of the two blocks having the highest correlation to the other block and sets the detected vector as a local motion vector. Specifically, the local motionvector detection unit 310 detects a candidate motion vector my that satisfies Expressions 5 and 6 and sets the detected candidate motion vector my as a local motion vector mvL. -
- In Expression 5, x and y represent coordinates of a pixel in the target block. ESAD represents the sum of absolute difference values of values of corresponding pixels of the two blocks and is used as an evaluation value indicating the level of the correlation between those blocks. Further, B represents the target block. In Expression 6, minESAD represents a minimum value of ESAD. argminESAD(mv) represents a function that returns my obtained when ESAD takes a minimum value, that is, when the correlation is highest.
- As exemplified in Expressions 5 and 6, the processing of evaluating the level of the correlation based on the sum of absolute difference values of pixel values is referred to as SAD (Sum of Absolute Difference estimation). Note that the local motion
vector detection unit 310 can also obtain the level of the correlation by processing other than the SAD. For example, the local motionvector detection unit 310 may use MPC (Maximum matching Pixel Count). In the MPC, the number of pixels at which an absolute difference value of pixel values is equal to or smaller than a predetermined matching threshold is counted to evaluate the correlation level based on the count value. In the MPC, a candidate motion vector obtained when the count value is maximum is detected as the local motion vector. - The local motion
vector detection unit 310 obtains the local motion vector mvL for each of all the blocks of the interpolated image and obtains a forward motion vector vn— L and a backward motion vector vn-1— L by usingExpressions 7 and 8. -
- In
Expressions 7 and 8, mvXL is an x component of the local motion vector mvL and mvYL is a y component of the local motion vector mvL. - The local motion
vector detection unit 310 supplies the forward motion vector vn— L and the backward motion vector vn-1— L, of each block to the normal interpolatedimage generation unit 320. - The normal interpolated
image generation unit 320 generates an image to be interpolated between the original images Fn and Fn-1, which is a normal interpolated image, based on the local motion vector mvL and the original images Fn and Fn-1. The normal interpolatedimage generation unit 320 receives the forward motion vector vn— L and the backward motion vector vn-1— L from the local motionvector detection unit 310. The normal interpolatedimage generation unit 320 substitutes the received vectors into Expressions 9 and 10 below to obtain the pixel values “succ” and “prev” in the original images Fn and Fn-1. -
succ=F n(x+v x— n— L ,y+v y— n— L) Expression 9 -
prev=F n-1(x+v x— n-1— L ,y+v y— n-1— L) Expression 10 - The normal interpolated
image generation unit 320 substitutes the pixel values acquired by Expressions 9 and 10 intoExpression 11 below to calculate a pixel value “out”. -
out=prev·(1−Relpos)+succ·Relpos Expression 11 - The normal interpolated
image generation unit 320 supplies an image formed of the pixel values “out”, which is a normal interpolated image MC_N, to thecombination unit 370. - The high-frequency
area extraction unit 330 extracts an area having a spatial frequency higher than a predetermined value from each of the original images Fn and Fn-1, and sets the extracted areas as high-frequency areas HPn and HPn-1. Further, the high-frequencyarea extraction unit 330 detects a difference value k in pixel value between the original images Fn and Fn-1 for each pixel. Subsequently, the high-frequencyarea extraction unit 330 amplifies each pixel value of the extracted high-frequency areas HPn and HPn-1 in accordance with the difference value k, to generate high-frequency areas HPKn and HPKn-1. The high-frequencyarea extraction unit 330 supplies those high-frequency areas HPKn and HPKn-1 to the global motionvector detection unit 340 and the combinationratio determination unit 360. - The global motion
vector detection unit 340 detects a vector, i.e., a global motion vector, which indicates a direction and a distance in which the high-frequency area moves within a certain period of time. The global motionvector detection unit 340 receives the high-frequency areas HPKn and HPKn-1 from the high-frequencyarea extraction unit 330 and detects the vector indicating the moving direction of the whole of those areas, to set the detected vector as a global motion vector mvG. Specifically, the global motionvector detection unit 340 sets the search range W for the whole of the high-frequency areas HPKn and HPKn-1 and detects a candidate motion vector that satisfies Expressions 6 and 12 below as the global motion vector mvG. -
- In Expression 12, MC represents the entire interpolated image.
- The global motion
vector detection unit 340 substitutes the obtained global motion vector mvG into Expressions 13 and 14 below and obtains a forward motion vector Vn— G and a backward motion vector Vn-1— G. -
- In Expressions 13 and 14, mvXG represents an x component of the global motion vector mvG and mvYG represents a y component of the global motion vector mvG.
- The global motion
vector detection unit 340 supplies the forward motion vector Vn— c and the backward motion vector Vn-1— G to the high-frequency area interpolatedimage generation unit 350 and the combinationratio determination unit 360. Note that the global motionvector detection unit 340 is an example of a vector detection unit in the section “What is claimed is”. - The high-frequency area interpolated
image generation unit 350 generates an image to be interpolated between the original images Fn and Fn-1, which is a high-frequency area interpolated image, based on the global motion vector mvG and the original images Fn and Fn-1. The high-frequency area interpolatedimage generation unit 350 receives the forward motion vector vn— G and the backward motion vector Vn-1— G from the global motionvector detection unit 340. The high-frequency area interpolatedimage generation unit 350 substitutes the received vectors into Expressions 15 and 16 below to obtain the pixel values “succ” and “prev” in the original images Fn and Fn-1. -
succ=F n(x+v x— n— G ,y+v y— n— G) Expression 15 -
prev=F n-1(x+v x— n-1— G ,y+v y— n-1— G) Expression 16 - The high-frequency area interpolated
image generation unit 350 substitutes the pixel values acquired by Expressions 15 and 16 intoExpression 11 to calculate a pixel value “out”. The high-frequency area interpolatedimage generation unit 350 supplies an image formed of the pixel values “out”, which is a high-frequency area interpolated image MC_HPn to thecombination unit 370. - The combination
ratio determination unit 360 determines a combination ratio of the normal interpolated image MC_N and the high-frequency area interpolated image MC_HP. The combinationratio determination unit 360 receives the high-frequency areas HPKn and HPKn-1 from the high-frequencyarea extraction unit 330 and combines the high-frequency areas HPKn and HPKn-1 to generate combination ratio data MIX. The combination ratio data MIX is image data that indicates the combination ratio of each pixel by pixel values. The combinationratio determination unit 360 supplies the combination ratio data MIX to thecombination unit 370. - The
combination unit 370 combines the normal interpolated image MC_N and the high-frequency area interpolated image MC_HP based on the combination ratio data MIX. Thecombination unit 370 causes theimage memory 220 to store the combined image as the combined interpolated image MC_mix. -
FIGS. 4A , 4B, and 4C are each a diagram for describing a method of detecting a local motion vector in the first embodiment.FIG. 4A is an example of anoriginal image 510.FIG. 4B is an example of anoriginal image 520 subsequent to theoriginal image 510. - In the normal interpolated image, it is assumed that a block located at the same position as a
block 511 in theoriginal image 510 is selected as a target block. In this case, in theoriginal image 510, theblock 511 and a predetermined number of blocks around the block 511 (for example, 8 blocks) are set as asearch range 512. Further, also in theoriginal image 520, ablock 521 corresponding to theblock 511 and a predetermined number of blocks around the block 521 (for example, 8 blocks) are set as asearch range 522. Subsequently, block matching between the blocks in thesearch range 512 and the blocks in thesearch range 522 is performed by using Expressions 5 and 6 and the like. - It is assumed that the following result is obtained in the block matching: a
block 513 on the upper right of thesearch range 512 and ablock 523 on the lower left of thesearch range 522 have the highest correlation. From this result, the vector indicating a direction and a distance from theblock 513 to theblock 523 is detected as alocal motion vector 524. -
FIG. 4C is a diagram showing an example of the forward motion vector and the backward motion vector. Thelocal motion vector 524 is divided into the forward motion vector Vn— L and the backward motion vector Vn-1— L byExpressions 7 and 8. -
FIG. 5 is a diagram showing an example of the local motion vector in the first embodiment. The local motion vector is obtained for each block, and each local motion vector is divided into the forward motion vector Vn— L and the backward motion vector Vn-1— L. As a result, as exemplified inFIG. 5 , the forward motion vector Vn— L and the backward motion vector Vn-1— L are obtained for each block. For example, when a local motion vector (8, 0) is detected in a block B10 in the normal interpolated image, this vector is divided into a forward motion vector (—4, 0) and a backward motion vector (4, 0) byExpressions 1 and 2. - (Configuration Example of Normal Interpolated Image Generation Unit)
-
FIG. 6 is a block diagram showing a configuration example of the normal interpolatedimage generation unit 320 in the first embodiment. The normal interpolatedimage generation unit 320 includes a forwardmotion compensation unit 321, a backwardmotion compensation unit 322, anadder 323, and amultiplier 324. - The forward
motion compensation unit 321 executes forward motion compensation processing to acquire each pixel value of coordinates (x, y) in the normal interpolated image based on the forward motion vector vn— L. Specifically, the pixel value “succ” in the original image Fn, which is obtained by Expression 9, is acquired as a pixel value at the coordinates (x, y) in the normal interpolated image. The forwardmotion compensation unit 321 supplies the acquired pixel value to theadder 323. - The backward
motion compensation unit 322 executes backward motion compensation processing to acquire each pixel value of coordinates (x, y) in the normal interpolated image based on the backward motion vector Vn-1— L. Specifically, the pixel value “prev” in the original image Fn-1, which is obtained by Expression 10, is acquired as a pixel value at the coordinates (x, y) in the normal interpolated image. The backwardmotion compensation unit 322 supplies the acquired pixel value to theadder 323. - The
adder 323 adds the input values. Theadder 323 receives an input of the pixel value “succ” acquired by the forwardmotion compensation unit 321 and an input of the pixel value “prev” acquired by the backwardmotion compensation unit 322. Theadder 323 adds those input values and supplies the resultant value to themultiplier 324. - The
multiplier 324 multiplies the input value by a predetermined value. Themultiplier 324 receives an input of the additional value supplied from theadder 323. Themultiplier 324 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out” in the normal interpolated image MC_N, to thecombination unit 370. - The
adder 323 and themultiplier 324 implement an arithmetic operation ofExpression 11 when the interpolation position parameter of Relpos is 0.5. - Note that the normal interpolated
image generation unit 320 executes both of the forward motion compensation processing and the backward motion compensation processing, but it may execute any one of the forward motion compensation processing and the backward motion compensation processing. In this case, the normal interpolatedimage generation unit 320 includes one of the forwardmotion compensation unit 321 and the backwardmotion compensation unit 322 and does not include the other motion compensation unit, theadder 323, and themultiplier 324. Subsequently, a pixel value generated by any one of the forwardmotion compensation unit 321 and the backwardmotion compensation unit 322 is output without change to thecombination unit 370, to serve as the pixel value of the normal interpolated image. -
FIGS. 7A , 7B, and 7C are each a diagram for describing a method of generating a normal interpolated image in the first embodiment.FIG. 7A is an example of a forwardmotion compensation image 530 generated by the forwardmotion compensation unit 321. The forwardmotion compensation unit 321 uses Expression 9 to acquire, from the original image Fn, a pixel value “succ” of a pixel that is located at a position to which a pixel at coordinates (x, y) in a target block 541 of the forwardmotion compensation image 530 is moved along the forward motion vector Vn— L. This pixel value “succ” is used as a pixel value of the pixel at coordinates (x, y) in the target block 541. In other words, the forwardmotion compensation image 530 is generated using a pixel obtained by moving the pixel of the original image Fn according to the forward motion vector Vn— L. -
FIG. 7B is an example of a backwardmotion compensation image 540 generated by the backwardmotion compensation unit 322. The backwardmotion compensation unit 322 uses Expression 10 to acquire, from the original image Fn-1, a pixel value “prev” of a pixel that is located at a position to which a pixel at coordinates (x, y) in atarget block 531 of the backwardmotion compensation image 540 is moved along the backward motion vector Vn-1— L. This pixel value “prev” is used as a pixel value of the pixel at coordinates (x, y) in the target block 541. In other words, the backwardmotion compensation image 540 is generated using a pixel obtained by moving the pixel of the original image Fn-1 according to the backward motion vector Vn-1— L. -
FIG. 7C is a diagram showing an example of a normalinterpolated image 550 generated by combining the forwardmotion compensation image 530 shown inFIG. 7A and the backwardmotion compensation image 540 shown inFIG. 7B based onExpression 11. The forward motion vector Vn— L and the backward motion vector Vn-1— L are vectors obtained when the local motion vector mvL is divided. Consequently, images in which pixels are moved according to those forward motion vector Vn and backward motion vector Vn-1 are combined to each other, and a block located between the blocks moved according to the local motion vector mvL is interpolated. - (Configuration Example of High-Frequency Area Extraction Unit)
-
FIG. 8 is a block diagram showing a configuration example of the high-frequencyarea extraction unit 330 in the first embodiment. The high-frequencyarea extraction unit 330 includes a low-pass filter 331, asubtracter 332, amultiplier 333, animage memory 334, and adifference detection unit 335. - The low-
pass filter 331 extracts a low-frequency area LPn from the original image Fn, the low-frequency area LPn having a spatial frequency lower than a predetermined value. The low-pass filter 331 is achieved by a FIR (Finite Impulse Response) filter, for example. The low-pass filter 331 supplies a pixel value of each pixel in the extracted low-frequency area LPn to thesubtracter 332. - The
subtracter 332 subtracts one of two input values from the other value. Thesubtracter 332 sequentially receives an input of the pixel value in the low-frequency area LPn extracted by the low-pass filter 331 and an input of the pixel value in the original image Fn. Thesubtracter 332 subtracts the pixel value in the low-frequency area LPn from the pixel value in the original image Fn and supplies the resultant value, which is a pixel value of the high-frequency area HPn, to themultiplier 333. - Note that the high-frequency
area extraction unit 330 extracts the high-frequency area HPn by the low-pass filter 331 and thesubtracter 332, but the high-frequencyarea extraction unit 330 may include a high-pass filter instead of the low-pass filter 331 and thesubtracter 332 and extract the high-frequency area HPn by performing high-pass filtering on the original image Fn. Further, the low-pass filter 331 and thesubtracter 332 are examples of a filter unit in the section “What is claimed is”. - The
difference detection unit 335 detects a difference in pixel value between the original images Fn and Fn-1 for each pixel. Thedifference detection unit 335 detects a difference value k for each pixel. Thedifference detection unit 335 generates difference data K formed of those difference values k. Thedifference detection unit 335 sequentially supplies each of the difference values k in the difference data K to themultiplier 333. - The
multiplier 333 multiplies the input values. Themultiplier 333 receives an input of the difference value k in the difference data Kn and an input of the pixel value in the high-frequency area HPn. Themultiplier 333 multiplies those input values and supplies the resultant value, which is a pixel value in the high-frequency area HPKn, to the global motionvector detection unit 340 and the combinationratio determination unit 360. Further, themultiplier 333 stores the multiplied value in theimage memory 334. Note that themultiplier 333 is an example of an increase unit in the section “What is claimed is”. - The
multiplier 333 multiplies the pixel value of the high-frequency area HPn by the difference value k, so that the pixel value of the high-frequency area that moves between the original images Fn and Fn-1 is amplified. Consequently, the global motionvector detection unit 340 can easily detect a vector, which indicates a moving direction of the high-frequency area with motion, as a global motion vector. Further, since the pixel value of the high-frequency area with motion is amplified, the combination ratio of the high-frequency area with motion can be preferentially increased. - The
image memory 334 stores the high-frequency area. The global motionvector detection unit 340 and the combinationratio determination unit 360 read out the high-frequency area stored in theimage memory 334 when the next high-frequency area is generated, so that a delayed high-frequency area HPKn-1 can be acquired. - (Configuration Example of Difference Detection Unit)
-
FIG. 9 is a block diagram showing a configuration example of thedifference detection unit 335 in the first embodiment. Thedifference detection unit 335 includes asubtracter 336, an absolutevalue acquisition unit 337, and adifference adjustment filter 338. - The
subtracter 336 subtracts one of the two input values from the other value. Thesubtracter 336 sequentially receives an input of a pixel value of a pixel in the original image Fn and an input of a pixel value of a pixel in the original image Fn-1. Thesubtracter 336 subtracts one of those input values from the other value and supplies the resultant value, i.e., a difference of the pixel values, to the absolutevalue acquisition unit 337. - The absolute
value acquisition unit 337 acquires an absolute value of the input value. The absolutevalue acquisition unit 337 sequentially receives an input of the difference supplied from thesubtracter 336. The absolutevalue acquisition unit 337 supplies data formed of absolute values diff of the differences, which serves as difference absolute value data DIFn to thedifference adjustment filter 338. - The
difference adjustment filter 338 adjusts each absolute value diff in the difference absolute value data DIF. For example, each time an absolute value is input, thedifference adjustment filter 338 uses the following expression to calculate a value obtained by amplifying the absolute value, and sets the resultant value as a difference value k. Thedifference adjustment filter 338 outputs data formed of those difference values k, as difference data K, to themultiplier 333. -
- In Expression 17, “a” and “b” are factors, each of which is a real number. Further, Thd1 is a threshold larger than 0 and smaller than Thd2, and Thd2 is a threshold larger than Thd1 and smaller than 255.
- Note that Expression 17 assumes that the pixel value has a value expressed in 8 bits, specifically, a value ranging from 0 to 255, but the number of bits assigned to the pixel value is not limited to 8 bits. For example, the pixel value may be expressed in 10 bits. In such a case, in Expression 17, “255” is replaced with “1023”.
- The
difference adjustment filter 338 amplifies the absolute value, so that the pixel value of the high-frequency area with motion is increased. Consequently, the global motion vector can be easily detected and the combination ratio of the high-frequency area with motion is preferentially set to be high. - Note that the
difference adjustment filter 338 amplifies the absolute value of the difference, but thedifference adjustment filter 338 may output the absolute value as the difference value k without performing amplification. Further, thedifference adjustment filter 338 may attenuate the absolute value. When the absolute value of the difference is extremely large, there is a possibility that motion of a noise component is erroneously detected as a global motion vector. In this case, the attenuation of the absolute value suppresses an occurrence of a false detection. Additionally, when the absolute value of the difference is extremely large, there is a possibility that an inappropriate combination ratio is determined based on the noise component. In this case, the attenuation of the absolute value allows an appropriate combination ratio to be determined. - (Configuration Example of High-Frequency Area Interpolated Image Generation Unit)
-
FIG. 10 is a block diagram showing a configuration example of the high-frequency area interpolatedimage generation unit 350 in the first embodiment. The high-frequency area interpolatedimage generation unit 350 includes a forwardmotion compensation unit 351, a backwardmotion compensation unit 352, anadder 353, and amultiplier 354. - The configuration of the forward
motion compensation unit 351 is the same as that of the forwardmotion compensation unit 321 in the normal interpolatedimage generation unit 320 except that the forward motion compensation processing is executed based on the forward motion vector vn— G instead of the forward motion vector Vn— L. The forwardmotion compensation unit 351 supplies each pixel value “succ” acquired by the forward motion compensation processing to theadder 353. - The configuration of the backward
motion compensation unit 352 is the same as that of the backwardmotion compensation unit 322 in the normal interpolatedimage generation unit 320 except that the backward motion compensation processing is executed based on the backward motion vector vn-1— G instead of the backward motion vector Vn-1— L. The backwardmotion compensation unit 352 supplies each pixel value “prev” acquired by the backward motion compensation processing to theadder 353. - The
adder 353 adds the input values. Theadder 353 receives an input of the pixel value “succ” acquired by the forwardmotion compensation unit 351 and an input of the pixel value “prev” acquired by the backwardmotion compensation unit 352. Theadder 353 adds those input values and supplies the resultant value to themultiplier 354. - The
multiplier 354 multiplies the input value by a predetermined value. Themultiplier 354 receives an input of the additional value supplied from theadder 353. Themultiplier 354 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out” in the high-frequency area interpolated image MC_HP, to thecombination unit 370. - Note that the high-frequency area interpolated
image generation unit 350 executes both of the forward motion compensation processing and the backward motion compensation processing, but it may execute any one of the forward motion compensation processing and the backward motion compensation processing. In this case, the high-frequency area interpolatedimage generation unit 350 includes one of the forwardmotion compensation unit 351 and the backwardmotion compensation unit 352 and does not include the other motion compensation unit, theadder 353, and themultiplier 354. Subsequently, a pixel value generated by any one of the forwardmotion compensation unit 351 and the backwardmotion compensation unit 352 is output without change to thecombination unit 370, to serve as the pixel value of the normal interpolated image. - (Configuration Example of Combination Ratio Determination Unit)
-
FIG. 11 is a block diagram showing a configuration example of the combinationratio determination unit 360 in the first embodiment. The combinationratio determination unit 360 includes a forwardmotion compensation unit 361, a backwardmotion compensation unit 362, amultiplier 363, again adjustment unit 364, and aninterpolation filter 365. - The configuration of the forward
motion compensation unit 361 is the same as that of the forwardmotion compensation unit 351 in the high-frequency area interpolatedimage generation unit 350 except that the forward motion compensation processing is executed based on the high-frequency area HPKn instead of the original image Fn. The forwardmotion compensation unit 361 supplies each pixel value “succ” acquired by the forward motion compensation processing to themultiplier 363. - The configuration of the backward
motion compensation unit 362 is the same as that of the backwardmotion compensation unit 352 in the high-frequency area interpolatedimage generation unit 350 except that the backward motion compensation processing is executed based on the high-frequency area HPKn-1 instead of the original image Fn-1. The backwardmotion compensation unit 362 supplies each pixel value “prev” acquired by the backward motion compensation processing to themultiplier 363. - The
multiplier 363 multiplies the input values. Themultiplier 363 receives an input of the pixel value “succ” acquired by the forwardmotion compensation unit 361 and an input of the pixel value “prev” acquired by the backwardmotion compensation unit 362. Themultiplier 363 multiplies those input values and supplies the resultant value to thegain adjustment unit 364. - The
multiplier 363 multiplies the pixel value “succ” and the pixel value “prev”, so that the pixel value of the high-frequency area is amplified more than when the input values are added. Consequently, the combination ratio of the high-frequency area is preferentially set to be high. - The
gain adjustment unit 364 adjusts a gain and attenuates the pixel value supplied from themultiplier 363. Thegain adjustment unit 364 attenuates the pixel value from themultiplier 363 such that the value falls within a certain range through the gain adjustment. For example, when a pixel value has the data size of 8 bits and such pixel values are multiplied, the data size of the resultant value after the multiplication is 16 bits. The multiplication result is attenuated such that its data size is changed to 8 bits. Thegain adjustment unit 364 supplies the adjusted pixel value to theinterpolation filter 365. - The
interpolation filter 365 interpolates, in an image MC_M formed of the pixel values generated by thegain adjustment unit 364, a portion where the edge interrupts, to generate the combination ratio data MIX. Theinterpolation filter 365 uses the following expression to perform interpolation, for example. Theinterpolation filter 365 supplies the generated combination ratio data MIX to thecombination unit 370. -
MIX(x,y)=max{MC — M(x−1,y)MC — M(x,y)MC — M(x+1,y)} Expression 18 - In Expression 18, MC_M(x, y) represents a pixel value of a pixel at coordinates (x, y) in the image MC_M. MIX(x, y) represents a pixel value of a pixel at coordinates (x, y) in the combination ratio data MIX. max( ) represents a maximum value of the pixel value in parentheses.
- (Configuration Example of Combination Unit)
-
FIG. 12 is a block diagram showing a configuration example of thecombination unit 370 in the first embodiment. Thecombination unit 370 includes a level adjustment unit 371, a multiplier 372, a subtracter 373, a multiplier 374, and an adder 375. - The level adjustment unit 371 adjusts the pixel value in the combination ratio data MIX so as to fall within the range from 0 to 1. For example, when the pixel value takes a value within the range from 0 to 255, the level adjustment unit 371 divides the pixel value by 255 to adjust the resultant value to a value falling within the range from 0 to 1. The level adjustment unit 371 sequentially supplies each of the adjusted values, which is a combination ratio mix, to the subtracter 373 and the multiplier 372.
- The multiplier 372 multiplies the input values. The multiplier 372 receives an input of the combination ratio mix supplied from the level adjustment unit 371 and an input of the pixel value in the high-frequency area interpolated image MC_HP. The multiplier 372 multiplies those input values and supplies the resultant value to the adder 375.
- The subtracter 373 subtracts the input value from a predetermined value. The subtracter 373 receives an input of the combination ratio mix supplied from the level adjustment unit 371. The subtracter 373 subtracts the combination ratio mix from 1.0 and supplies the resultant value (1.0-mix) to the multiplier 374.
- The multiplier 374 multiplies the input values. The multiplier 374 receives an input of the value (1.0-mix) supplied from the subtracter 373 and an input of the pixel value in the normal interpolated image MC_N. The multiplier 374 multiplies those input values and supplies the resultant value to the adder 375.
- The adder 375 adds the input values. The adder 375 receives inputs of the multiplication results of the respective multipliers 372 and 374. The adder 375 adds those input values and stores, in the
image memory 220, the resultant value serving as a pixel value in the combined interpolated image MC_mix. - To sum it up, the multiplier 372, the subtracter 373, the multiplier 374, and the adder 375 execute the following arithmetic operation of Expression 19 below.
-
MC_mix(x,y)=MC — N(x,y)×(1.0−mix)+MC — HP(x,y)×mix Expression 19 - In Expression 19, MC_mix(x, y) represents a pixel value of a pixel at coordinates (x, y) in the combined interpolated image MC_mix. MC_N(x, y) represents a pixel value of a pixel at coordinates (x, y) in the normal interpolated image MC_N. MC_HP(x, y) represents a pixel value of a pixel at coordinates (x, y) in the high-frequency area interpolated image MC_HP. mix represents a combination ratio generated by the level adjustment unit 371 based on the pixel value at coordinates (x, y) in the combination ratio data MIX.
-
FIG. 13 is a diagram showing an example of a normalinterpolated image 610 in the first embodiment. Here, it is assumed that theimage processing apparatus 200 receives an input of a moving image in which a character telop, which moves in a horizontal direction with the lapse of time, is displayed on a lower portion of a screen. Theimage processing apparatus 200 generates the normalinterpolated image 610 from the original images Fn and Fn-1 in the moving image. The normalinterpolated image 610 includes acharacter telop 611 in its lower portion. Note that characters are also displayed above thecharacter telop 611, but those character portions do not move with the lapse of time. - As described above, the
image processing apparatus 200 detects a local motion vector for each block and moves the block in the original image based on the local motion vector, to generate the normalinterpolated image 610. As to the area of the character telop, the entire area moves in the horizontal direction, and thus all the values of the local motion vectors in the character telop are thought to be the same. - When a lot of blocks exist in the character telop, however, there is a case where a vector with a value, which is different from values of local motion vectors of adjacent blocks, may be detected from some blocks in the character telop. In this case, the local motion vector of such a block does not have a correct value, and it is evaluated that the local motion vector detection has failed.
- A movement amount of the block for which the local motion vector detection has failed is different from an actual movement amount of the character telop, and thus a blur occurs in a boundary between the block for which the detection has failed and a block for which the detection has not failed. For example, in the
character telop 611 in the normalinterpolated image 610, blurs occur at a plurality of portions such asblurs -
FIG. 14 is a diagram showing an example of a high-frequency area interpolatedimage 620 in the first embodiment. Theimage processing apparatus 200 moves the entire original image according to the global motion vectors each indicating the movement of acharacter telop 621 and generates the high-frequency area interpolatedimage 620, so that no blurs occur in thecharacter telop 621. On the other hand, since the entire original image is moved according to the global motion vectors irrespective of no movements at portions other than thecharacter telop 621, blurs occur at the portions other than thecharacter telop 621. -
FIG. 15 is a diagram showing an example ofcombination ratio data 630 in the first embodiment. Thecombination ratio data 630 is image data including, in the high-frequency area with motion, a pixel whose pixel value is amplified. For that reason, a pixel value of the character telop portion extracted as a high-frequency area with motion is higher than those of the other portions. Consequently, the value of the combination ratio mix in the character telop portion is set to be higher than those of the other portions. -
FIG. 16 is a diagram showing an example of a combined interpolatedimage 640 in the first embodiment. Theimage processing apparatus 200 combines the normalinterpolated image 610 and the high-frequency area interpolatedimage 620 based on thecombination ratio data 630. In thecombination ratio data 630, the combination ratio of the character telop portion is set to be higher. Consequently, a combined interpolatedimage 640 in which the ratio of the high-frequency area interpolatedimage 620 is higher in acharacter telop 631 and the ratio of the normalinterpolated image 610 is higher in the other portions is generated. No blurs occur in the character telop in the high-frequency area interpolatedimage 620, and no blurs occur in portions other than the character telop in the normalinterpolated image 610. Consequently, no blurs occur as a whole in the combined interpolatedimage 640, which is obtained by combining the images with a high ratio of the portions having no blurs. The combined interpolatedimage 640 is interpolated in the original image. This allows an improvement of the image quality of a moving image whose frame rate is converted. - (Operation Example of Image Processing Apparatus)
-
FIG. 17 is a flowchart showing an example of an operation of theimage processing apparatus 200 in the first embodiment. This operation is started when theimage processing apparatus 200 executes the processing to convert the frame rate, for example. - The
image processing apparatus 200 determines whether an original image is input or not (Step S910). When an original image is input (Step S910: Yes), theimage processing apparatus 200 executes interpolated image generation processing to generate a combined interpolated image (Step S920). Subsequently, theimage processing apparatus 200 outputs the interpolated image and the original image in a time-series order (Step S930). When an original image is not input (Step S910: No), or after the processing of Step S930 is performed, theimage processing apparatus 200 returns to the processing of Step S910. -
FIG. 18 is a flowchart showing an example of the interpolated image generation processing in the first embodiment. The interpolatedimage generation unit 300 of theimage processing apparatus 200 detects a local motion vector for each block (Step S921). Subsequently, the interpolatedimage generation unit 300 generates a normal interpolated image from original images based on the local motion vector (Step S922). - The interpolated
image generation unit 300 extracts a high-frequency area from the original images (Step S923). Subsequently, the interpolatedimage generation unit 300 detects a motion vector of the high-frequency area as a global motion vector (Step S924). The interpolatedimage generation unit 300 generates a high-frequency area interpolated image from the original images based on the global motion vector (Step S925). - The interpolated
image generation unit 300 determines a combination ratio for each pixel based on the high-frequency areas (Step S926). Subsequently, the interpolatedimage generation unit 300 combines the normal interpolated image and the high-frequency area interpolated image according to the combination ratio (Step S927). After Step S927, the interpolatedimage generation unit 300 terminates the interpolated image generation processing. - As described above, according to the first embodiment, the
image processing apparatus 200 combines the high-frequency area interpolated image and the normal interpolated image that are generated based on the change in position of the high-frequency area, to generate an interpolated image with no blurs in the high-frequency area. Additionally, theimage processing apparatus 200 combines the normal interpolated image and the high-frequency area interpolated image, to generate an interpolated image in which a boundary between the high-frequency area and another area looks natural. Such an interpolated image is interpolated between the original images. This allows an improvement of the image quality of a moving image whose frame rate is converted. - Although the local motion vector is detected to generate the normal interpolated image in the first embodiment, the
image processing apparatus 200 can also generate the normal interpolated image without detecting the local motion vector. Theimage processing apparatus 200 in a modified example is different from that of the first embodiment in that the local motion vector is not detected. - (Configuration Example of Interpolated Image Generation Unit)
-
FIG. 19 is a block diagram showing a configuration example of the interpolatedimage generation unit 300 in the modified example of the first embodiment. The interpolatedimage generation unit 300 in the modified example is different from that of the first embodiment in that a normal interpolatedimage generation unit 325 is provided instead of the local motionvector detection unit 310 and the normal interpolatedimage generation unit 320. - The normal interpolated
image generation unit 325 combines the original images Fn and Fn-1 to generate the normal interpolated image MC_N. -
FIG. 20 is a block diagram showing a configuration example of the normal interpolatedimage generation unit 325 in the modified example of the first embodiment. The normal interpolatedimage generation unit 325 includes anadder 323 and amultiplier 324. - The
adder 323 adds the input values. Theadder 323 receives an input of a pixel value in the original image Fn and an input of a pixel value in the original image Fn-1. Theadder 323 adds those input values and supplies the resultant value to themultiplier 324. - The
multiplier 324 multiplies the input value by a predetermined value. Themultiplier 324 receives an input of the additional value supplied from theadder 323. Themultiplier 324 multiplies the input value by 0.5 and supplies the resultant value, which is the pixel value “out”, to thecombination unit 370. In such a manner, the normal interpolatedimage generation unit 325 generates a normal interpolated image without executing the motion compensation processing. - Note that the normal interpolated
image generation unit 325 combines the original images Fn and Fn-1 to generate the normal interpolated image, but it may use one of the original images without change as the normal interpolated image. -
FIG. 21 is a diagram showing an example of a normalinterpolated image 660 in the modified example of the first embodiment. The normalinterpolated image 660 is an image generated without performing a motion compensation, and hence blurs occur in a portion of acharacter telop 661 that moves in the horizontal direction. The normalinterpolated image 660 is combined with a high-frequency area interpolated image having no blurs in the character telop. This suppresses the occurrence of blurs in a combined interpolated image that is to be eventually generated. - In the first embodiment, the
image processing apparatus 200 amplifies the pixel value of the pixel in the high-frequency area according to the difference. However, the difference may include a noise component and it is desirable to execute coring processing to remove the noise component before the amplification. Theimage processing apparatus 200 of the second embodiment is different from that of the first embodiment in that the coring processing is performed on the difference. - (Configuration Example of Interpolated Image Generation Unit)
-
FIG. 22 is a block diagram showing a configuration example of the interpolatedimage generation unit 300 in the second embodiment. The interpolatedimage generation unit 300 in the second embodiment is different from that of the first embodiment in that a high-frequencyarea extraction unit 380 is provided instead of the high-frequencyarea extraction unit 330. - The high-frequency
area extraction unit 380 executes the coring processing to remove a value smaller than a difference threshold Thk1 from the difference values k, to generate a difference value k1. Additionally, the high-frequencyarea extraction unit 380 executes the coring processing to remove a value smaller than a difference threshold Thk2 from the difference values k, to generate a difference value k2. The high-frequencyarea extraction unit 380 amplifies the pixel values of the high-frequency areas HPn and HPn-1 in accordance with the difference value k1, to generate high-frequency areas HPK1n and HPK1n-1. Additionally, the high-frequencyarea extraction unit 380 amplifies the pixel values of the high-frequency areas HPn and HPn-1 in accordance with the difference value k2, to generate high-frequency areas HPK2n and HPK2n-1. - The high-frequency
area extraction unit 380 supplies the high-frequency areas HPK1n and HPK1n-1 to the global motionvector detection unit 340 and supplies the high-frequency areas HPK2n and HPK2n-1 to the combinationratio determination unit 360. - (Configuration Example of High-Frequency Area Extraction Unit)
-
FIG. 23 is a block diagram showing a configuration example of the high-frequencyarea extraction unit 380 in the second embodiment. The high-frequencyarea extraction unit 380 is different from the high-frequencyarea extraction unit 330 in thatcoring processing units multiplier 383, and animage memory 384 are additionally provided. - The
coring processing unit 381 executes the coring processing to remove the difference value k smaller than the difference threshold Thk1. For example, thecoring processing unit 381 uses the following expression to obtain the difference value k1 from the difference values k in the difference data K. Thecoring processing unit 381 supplies difference data K1 formed of the difference value k1 to themultiplier 333. Note that thecoring processing unit 381 is an example of a first difference coring processing unit in the section “What is claimed is”. -
- The
multiplier 333 multiplies the pixel value in the high-frequency area HPn and the difference value k1 in the difference data K1. Data of the multiplication result, which serves as the high-frequency area HPK1n, is supplied to the global motionvector detection unit 340. - The
coring processing unit 382 executes the coring processing to remove the difference value k smaller than the difference threshold Thk2. For example, thecoring processing unit 382 uses the following expression to obtain the difference value k2 from the difference values k in the difference data K. Thecoring processing unit 382 supplies difference data K2 formed of the difference value k2 to themultiplier 383. Note that thecoring processing unit 382 is an example of a second difference coring processing unit in the section “What is claimed is”. -
- The
multiplier 383 multiplies the input values. Themultiplier 383 receives an input of the pixel value in the high-frequency area HPn and an input of the difference value k2 in the difference data K2. Themultiplier 383 multiplies those input values, supplies data of the multiplication result, which serves as the high-frequency area HPK2n, to the combinationratio determination unit 360, and causes theimage memory 384 to store the data. - Note that the high-frequency
area extraction unit 380 executes both of the coring processing using the difference threshold Thk1 and the coring processing using the difference threshold Thk2, but it may execute any one of the processing. - As described above, according to the second embodiment, the
image processing apparatus 200 executes the coring processing to remove the difference smaller than the difference threshold, to remove the noise component from the difference. This allows the global motion vector to be easily detected and an appropriate combination ratio to be obtained. - In the second embodiment, the
image processing apparatus 200 performs the coring processing on the difference, but it may perform the coring processing on the high-frequency area. Theimage processing apparatus 200 in the modified example is different from that of the second embodiment in that the coring processing is performed on the high-frequency area. - (Configuration Example of High-Frequency Area Extraction Unit)
-
FIG. 24 is a block diagram showing a configuration example of the high-frequencyarea extraction unit 380 in the modified example of the second embodiment. The high-frequencyarea extraction unit 380 in the modified example is different from that of the second embodiment in thatcoring processing units coring processing units - The
coring processing unit 385 executes coring processing to remove a pixel having a pixel value smaller than a pixel threshold Thp1 in the high-frequency area HPn. Thecoring processing unit 385 sequentially supplies the pixel value in the high-frequency area HPn, which is obtained after the coring processing, to themultiplier 333. Themultiplier 333 multiplies the difference value k and the pixel value supplied from thecoring processing unit 385. - The
coring processing unit 386 executes coring processing to remove a pixel having a pixel value smaller than a pixel threshold Thp2 in the high-frequency area HPn. Thecoring processing unit 386 sequentially supplies the pixel value in the high-frequency area HPn, which is obtained after the coring processing, to themultiplier 383. Themultiplier 383 multiplies the difference value k and the pixel value supplied from thecoring processing unit 386. - Note that the high-frequency
area extraction unit 380 executes both of the coring processing using the pixel threshold Thp1 and the coring processing using the pixel threshold Thp2, but it may execute any one of the processing. Further, thecoring processing unit 385 is an example of a first pixel coring processing unit in the section “What is claimed is”. Furthermore, thecoring processing unit 386 is an example of a second pixel coring processing unit in the section “What is claimed is”. - As described above, according to the modified example, the
image processing apparatus 200 performs the coring processing on the high-frequency area HPn to remove a noise component in the high-frequency area HPn. - Note that the embodiments described above are merely examples for embodying the present disclosure, and the matters in the embodiments have correlations with the matters specifying the present disclosure in the section “What is claimed is”. Similarly, the matters specifying the present disclosure in the section “What is claimed is” have correlations with the matters in the embodiments of the present disclosure, which are denoted by the same names as those in the section “What is claimed is”. Note that the present disclosure is not limited to the embodiments and can be embodied by variously modifying the embodiments without departing from the gist of the present disclosure.
- Additionally, the processing procedures described in the above embodiments may be considered as a method including a series of those procedures, or may be considered as a program causing a computer to execute the series of those procedures or as a recording medium storing the program. Examples of the recording medium include a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disk), a memory card, and a blu-ray disc (Blu-ray Disc (registered trademark)).
- Note that the present disclosure can also take the following configurations.
- (1) An image processing apparatus, including:
- a normal interpolated image generation unit configured to generate an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- a high-frequency area extraction unit configured to extract a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images;
- a high-frequency area interpolated image generation unit configured to generate an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images; and
- a combination unit configured to execute combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
- (2) The image processing apparatus according to (1), further including a vector detection unit configured to detect a vector indicating a direction and a distance in which a position of the high-frequency area changes within a certain period of time on the time series, in which
- the high-frequency area interpolated image generation unit is configured to generate the high-frequency area interpolated image based on the vector and each of the plurality of original images.
- (3) The image processing apparatus according to (2), further including a combination ratio determination unit configured to determine a ratio at which the normal interpolated image and the high-frequency area interpolated image are combined, for each pixel in accordance with each pixel value of pixels in the high-frequency area, in which
- the combination unit is configured to execute the combining processing according to the ratio.
- (4) The image processing apparatus according to (3), in which
- the high-frequency area extraction unit includes
-
- a filter unit configured to extract the high-frequency area in each of the plurality of original images,
- a difference detection unit configured to detect, for each pixel, a difference in pixel value between two adjacent original images of the plurality of original images, and
- an increase unit configured to
- increase each pixel value of the pixels in the high-frequency area in each of the two adjacent original images in accordance with the difference, and
- supply the increased pixel value to the vector detection unit and the combination ratio determination unit.
(5) The image processing apparatus according to (4), in which
- the high-frequency area extraction unit includes a first difference coring processing unit configured to execute first difference coring processing to remove a difference smaller than a first difference threshold from the detected differences, and
- the increase unit is configured to
-
- increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the first difference coring processing is performed, and
- supply the increased pixel value to the vector detection unit.
(6) The image processing apparatus according to (4) or (5), in which
- the high-frequency area extraction unit includes a second difference coring processing unit configured to execute second difference coring processing to remove a difference smaller than a second difference threshold from the detected differences, and
- the increase unit is configured to
-
- increase each pixel value of the pixels in the high-frequency area in accordance with the difference on which the second difference coring processing is performed, and
- supply the increased pixel value to the combination ratio determination unit.
(7) The image processing apparatus according to any one of (4) to (6), in which
- the high-frequency area extraction unit includes a first pixel coring processing unit configured to execute first pixel coring processing to remove a pixel with a pixel value smaller than a first pixel value threshold in the extracted high-frequency area, and
- the increase unit is configured to
-
- increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the first pixel coring processing is performed, and
- supply the increased pixel value to the vector detection unit.
(8) The image processing apparatus according to any one of (4) to (7), in which
- the high-frequency area extraction unit includes a second pixel coring processing unit configured to execute second pixel coring processing to remove a pixel with a pixel value smaller than a second pixel value threshold in the extracted high-frequency area, and
- the increase unit is configured to
-
- increase, in accordance with the difference, each pixel value of the pixels in the high-frequency area on which the second pixel coring processing is performed, and
- supply the increased pixel value to the combination ratio determination unit.
(9) The image processing apparatus according to any one of (1) to (8), in which
- each of the plurality of original images includes a plurality of blocks each having a predetermined shape, and
- the normal interpolated image generation unit is configured to generate the normal interpolated image based on a change in position of each of the plurality of blocks along with an elapse of time on the time series and on each of the plurality of original images.
- (10) The image processing apparatus according to any one of (1) to (9), further including a selection unit configured to select the combined image by the combination unit and the plurality of original images in order of the time series and output the selected image.
(11) An image processing method, including: - generating, by a normal interpolated image generation unit, an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- extracting, by a high-frequency area extraction unit, a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images;
- generating, by a high-frequency area interpolated image generation unit, an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images; and
- executing, by a combination unit, combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
- (12) A program causing a computer to execute:
- generating, by a normal interpolated image generation unit, an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images;
- extracting, by a high-frequency area extraction unit, a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images;
- generating, by a high-frequency area interpolated image generation unit, an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images; and
- executing, by a combination unit, combining processing to combine the normal interpolated image and the high-frequency area interpolated image.
Claims (12)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012277561A JP2014123787A (en) | 2012-12-20 | 2012-12-20 | Image processing apparatus, image processing method and program |
JP2012-277561 | 2012-12-20 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140176794A1 true US20140176794A1 (en) | 2014-06-26 |
US8792053B2 US8792053B2 (en) | 2014-07-29 |
Family
ID=50957465
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/101,443 Expired - Fee Related US8792053B2 (en) | 2012-12-20 | 2013-12-10 | Image processing apparatus, image processing method, and program |
Country Status (3)
Country | Link |
---|---|
US (1) | US8792053B2 (en) |
JP (1) | JP2014123787A (en) |
CN (1) | CN103888775A (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140092109A1 (en) * | 2012-09-28 | 2014-04-03 | Nvidia Corporation | Computer system and method for gpu driver-generated interpolated frames |
US20180020229A1 (en) * | 2016-07-14 | 2018-01-18 | Sharp Laboratories Of America, Inc. | Computationally efficient motion compensated frame rate conversion system |
US10026155B1 (en) * | 2017-05-30 | 2018-07-17 | Olympus Corporation | Image-processing apparatus |
US11425331B1 (en) * | 2021-04-21 | 2022-08-23 | Terawins, Inc. | Video-image-interpolation apparatus and method for adaptive motion-compensated frame interpolation |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110059685B (en) * | 2019-04-26 | 2022-10-21 | 腾讯科技(深圳)有限公司 | Character area detection method, device and storage medium |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS6049339A (en) * | 1983-08-30 | 1985-03-18 | Dainippon Screen Mfg Co Ltd | Editing device for copy image |
JP2003069998A (en) * | 2001-08-28 | 2003-03-07 | Canon Inc | Image processor and image processing method |
JP4225795B2 (en) * | 2003-01-22 | 2009-02-18 | オリンパス株式会社 | Imaging system, image processing program |
EP3217660B1 (en) * | 2007-06-15 | 2018-10-31 | Panasonic Intellectual Property Management Co., Ltd. | Image processing device |
JP4535114B2 (en) * | 2007-10-26 | 2010-09-01 | ソニー株式会社 | Imaging apparatus and imaging method, display control apparatus and display control method, and program |
WO2009145201A1 (en) * | 2008-05-27 | 2009-12-03 | 三洋電機株式会社 | Image processing device, image processing method, and imaging device |
JP5188272B2 (en) | 2008-06-05 | 2013-04-24 | 株式会社日立製作所 | Video processing apparatus and video display apparatus |
JP2010028722A (en) * | 2008-07-24 | 2010-02-04 | Sanyo Electric Co Ltd | Imaging apparatus and image processing apparatus |
JP2010063088A (en) * | 2008-08-08 | 2010-03-18 | Sanyo Electric Co Ltd | Imaging apparatus |
JP5144456B2 (en) * | 2008-10-09 | 2013-02-13 | 富士フイルム株式会社 | Image processing apparatus and method, image reproducing apparatus and method, and program |
KR101548285B1 (en) * | 2009-01-20 | 2015-08-31 | 삼성전자주식회사 | Apparatus and method for obtaining high resolution image |
JP2011151345A (en) * | 2009-12-25 | 2011-08-04 | Hoya Corp | Imaging element and imaging apparatus |
JP2011134204A (en) * | 2009-12-25 | 2011-07-07 | Sony Corp | Image processing device, image processing method and program |
JP5366855B2 (en) * | 2010-02-16 | 2013-12-11 | 富士フイルム株式会社 | Image processing method and apparatus, and program |
-
2012
- 2012-12-20 JP JP2012277561A patent/JP2014123787A/en active Pending
-
2013
- 2013-12-10 US US14/101,443 patent/US8792053B2/en not_active Expired - Fee Related
- 2013-12-12 CN CN201310682585.5A patent/CN103888775A/en active Pending
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140092109A1 (en) * | 2012-09-28 | 2014-04-03 | Nvidia Corporation | Computer system and method for gpu driver-generated interpolated frames |
US20180020229A1 (en) * | 2016-07-14 | 2018-01-18 | Sharp Laboratories Of America, Inc. | Computationally efficient motion compensated frame rate conversion system |
US10026155B1 (en) * | 2017-05-30 | 2018-07-17 | Olympus Corporation | Image-processing apparatus |
US11425331B1 (en) * | 2021-04-21 | 2022-08-23 | Terawins, Inc. | Video-image-interpolation apparatus and method for adaptive motion-compensated frame interpolation |
Also Published As
Publication number | Publication date |
---|---|
CN103888775A (en) | 2014-06-25 |
US8792053B2 (en) | 2014-07-29 |
JP2014123787A (en) | 2014-07-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8792053B2 (en) | Image processing apparatus, image processing method, and program | |
US10127644B2 (en) | Generating synthetic video frames using optical flow | |
US20130301951A1 (en) | Method and apparatus for removing image noise | |
JPWO2006025396A1 (en) | Image processing apparatus and image processing program | |
JP2005176381A (en) | Adaptive motion compensated interpolating method and apparatus | |
US20100060798A1 (en) | Video signal processing device, video signal processing method, and video signal processing program | |
US20180091871A1 (en) | Apparatus for embedding digital watermark and method for embedding digital watermark | |
KR101805625B1 (en) | Method and appratus for converting image rate using image scaling | |
US8244055B2 (en) | Image processing apparatus and method, and program | |
JP5490236B2 (en) | Image processing apparatus and method, image display apparatus and method | |
US20120308080A1 (en) | Image processing apparatus, image processing method, and program | |
US8385430B2 (en) | Video signal processing apparatus and video signal processing method | |
US8139151B2 (en) | Moving image processing apparatus, control method thereof, and program | |
US20090310679A1 (en) | Video processing apparatus and methods | |
JP5962278B2 (en) | Image processing program, image processing apparatus, and image processing method | |
JP2005150903A (en) | Image processing apparatus, noise elimination method, and noise elimination program | |
JP2002223374A (en) | Device and method for removing noise | |
US8224120B2 (en) | Image signal processing apparatus and image signal processing method | |
JP5015089B2 (en) | Frame rate conversion device, frame rate conversion method, television receiver, frame rate conversion program, and recording medium recording the program | |
US20070222890A1 (en) | Picture processing circuit and picture processing method | |
US8411200B2 (en) | Video signal processing device, method, and non-transitory computer readable medium storing image processing program capable of producing an appropriate interpolation frame | |
JP5147655B2 (en) | Video signal processing device and video display device | |
JP2007028251A (en) | Image processing apparatus, image processing method, program, and recording medium | |
US20120308137A1 (en) | Image processing apparatus, image processing method, and program | |
JP2015079329A (en) | Image processor, image processing method and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: SONY CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FUJISHIRO, SHIGEO;SUZUKI, YOSHITO;OZEKI, EIJI;AND OTHERS;SIGNING DATES FROM 20131107 TO 20131124;REEL/FRAME:031786/0933 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SATURN LICENSING LLC, NEW YORK Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SONY CORPORATION;REEL/FRAME:043177/0794 Effective date: 20170613 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551) Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20220729 |