WO2015107880A1 - Image processing device, image processing method, and recording medium containing program - Google Patents
Image processing device, image processing method, and recording medium containing program Download PDFInfo
- Publication number
- WO2015107880A1 WO2015107880A1 PCT/JP2015/000087 JP2015000087W WO2015107880A1 WO 2015107880 A1 WO2015107880 A1 WO 2015107880A1 JP 2015000087 W JP2015000087 W JP 2015000087W WO 2015107880 A1 WO2015107880 A1 WO 2015107880A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame image
- image
- frame
- luminance
- interest
- Prior art date
Links
- 238000012545 processing Methods 0.000 title claims abstract description 54
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 239000000203 mixture Substances 0.000 claims description 18
- 230000002194 synthesizing effect Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 abstract description 44
- 230000015572 biosynthetic process Effects 0.000 abstract description 17
- 238000003786 synthesis reaction Methods 0.000 abstract description 17
- 230000001939 inductive effect Effects 0.000 abstract 1
- 201000003040 photosensitive epilepsy Diseases 0.000 abstract 1
- 201000005070 reflex epilepsy Diseases 0.000 abstract 1
- 235000019557 luminance Nutrition 0.000 description 150
- 238000000034 method Methods 0.000 description 39
- 230000004397 blinking Effects 0.000 description 30
- 238000012937 correction Methods 0.000 description 19
- 206010034972 Photosensitivity reaction Diseases 0.000 description 17
- 230000036211 photosensitivity Effects 0.000 description 17
- 238000004458 analytical method Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 13
- 206010010904 Convulsion Diseases 0.000 description 12
- 238000004364 calculation method Methods 0.000 description 7
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000003860 storage Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008094 contradictory effect Effects 0.000 description 4
- 238000009826 distribution Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 241000098700 Sarcocheilichthys parvus Species 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 229920006395 saturated elastomer Polymers 0.000 description 2
- 208000024891 symptom Diseases 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 206010028813 Nausea Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 230000036461 convulsion Effects 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 206010015037 epilepsy Diseases 0.000 description 1
- 230000001667 episodic effect Effects 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008693 nausea Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/147—Scene change detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
- H04N5/57—Control of contrast or brightness
Definitions
- the present invention relates to an image processing apparatus, an image processing method, and a recording medium on which a program is recorded.
- Photosensitivity seizures are abnormal responses to light stimuli that jump into the eye, with symptoms resembling epilepsy such as convulsions and impaired consciousness.
- Non- Patent Document 1 In order to prevent such influence from affecting viewers, attempts have been made to suppress the distribution of video content that affects the human body.
- ITU International Telecommunications Union
- Non-Patent Document 2 In Japan, the Japan Broadcasting Corporation and the Japan Broadcasting Corporation have established guidelines for the production of animations in particular, and are demanding compliance with those involved in broadcasting (Non-Patent Document 2).
- the current shooting environment includes many flickers (for example, general flashes with large brightness changes) and large changes in saturation (for example, red flashes) that can trigger photosensitivity attacks.
- flickers for example, general flashes with large brightness changes
- saturation for example, red flashes
- means for alerting the viewer with a telop or the like is taken.
- One of the shooting environments, including many flickers that can trigger photosensitivity attacks in news footage is a press conference. In a press conference, a short-time bright area is repeatedly generated by a flash emitted from a news photographer at the time of shooting, and many blinks are repeated in the video content.
- a liquid crystal display detects a scene (image) that induces a photosensitivity seizure, and reduces the luminance of the backlight unit with respect to the scene (image) that induces the photosensitivity seizure, thereby allowing viewing and listening. Prevent the effects of photosensitivity attacks on the elderly.
- ⁇ Flickering large brightness change
- large change in saturation that can induce photosensitivity seizures may occur not only in the entire image but also in some areas of the video. Since the related method corrects the entire image uniformly without making these determinations, a decrease in contrast or brightness occurs in an area that does not need to be corrected originally and the luminance or saturation does not change significantly. The image quality may deteriorate.
- an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of generating a video in which fluctuations in luminance or saturation are suppressed with respect to a video that may induce a photosensitivity seizure. Is to provide.
- the present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It has a determination part which determines whether it is a frame image containing a change area.
- a frame generation unit that generates an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image.
- the image processing apparatus includes a frame image synthesis unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolation frame image based on a determination result of the determination unit.
- the present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It is determined whether the frame image includes a fluctuation region. Furthermore, an interpolated frame image that is an attention frame image without the fluctuation region is generated based on a frame image other than the attention frame image. Further, the image processing method generates an output frame image obtained by combining the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
- the present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It is determined whether the frame image includes a fluctuation region. Furthermore, an interpolated frame image that is an attention frame image without the fluctuation region is generated based on a frame image other than the attention frame image. The program further causes the computer to generate an output frame image obtained by combining the frame image of interest and the interpolated frame image based on the determination result of the determination unit.
- the present invention it is possible to generate an image in which variation in luminance or saturation is suppressed with respect to an image that may induce a photosensitivity seizure.
- FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment.
- FIG. 2 is a schematic diagram showing a rectangular area luminance calculation method.
- FIG. 3 is a schematic diagram illustrating a method of selecting a frame image that does not include a fluctuation region.
- FIG. 4 is a schematic diagram for explaining a case where the correction becomes unstable.
- FIG. 5 is a graph showing an example of a method for setting the value of the rate of change in local area luminance in the output frame image.
- FIG. 6 is a flowchart showing the operation of the image processing apparatus according to the first embodiment.
- FIG. 7 is a block diagram of an image processing apparatus according to the second embodiment.
- FIG. 8 is a flowchart showing the operation of the image processing apparatus according to the second embodiment.
- FIG. 9 is a block diagram of an image processing apparatus according to the third embodiment of the present invention.
- FIG. 10 is a diagram showing a hardware configuration in which the image processing apparatus according to the first to third embodiments is realized by
- FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment of the present invention.
- the image processing apparatus includes a flash frame determination unit 11, a frame interpolation unit 12, and a frame image synthesis unit 13. Then, the image processing apparatus according to the first embodiment inputs a group of frame images from time (t ⁇ m) to time (t + m) configured by a preset number of frames 2m + 1. Further, the image processing apparatus is a frame image including a variation region caused by blinking (a large variation in luminance) due to flashing or the like, in which the attention frame image that is the input frame image at time t may induce a photosensitivity seizure. Determine if there is.
- the image processing apparatus When the variation area is included, the image processing apparatus generates an output frame image in which the influence of blinking is reduced (luminance variation is suppressed) by appropriately suppressing the variation area by the correction process.
- the fluctuation region means a bright region that is brighter (brightness increases) in the frame image of interest than in other frame images.
- the flash frame determination unit 11 receives an input frame image group as an input, and whether the frame image of interest (the input frame image at time t) is a frame image including a variation area due to blinking (a large variation in luminance) that affects the viewer. Determine.
- One method for determining whether a target frame image is a frame including a variable region is a method of using the rate of change in local region luminance between the target frame image and another input frame image.
- Local area luminance represents the luminance value of the pixel peripheral area in each pixel of the input frame image group.
- a method of calculating the local area luminance first, for each pixel of the input frame image group, color information expressed in RGB (Red-Green-Blue) or the like is converted into luminance information representing brightness, and then converted. There is a method of calculating the luminance value of the pixel peripheral region by performing smoothing processing on the periphery of the target pixel with respect to the luminance information.
- YUV luminance component (Y) and color difference components (U and V)
- YCbCr or YPbPr luminance which is a color system used for broadcasting
- YCbCr or YPbPr luminance which is a color system used for broadcasting
- YPbPr luminance which is a color system used for broadcasting
- YCbCr or YPbPr luminance which is a color system used for broadcasting
- an average value is obtained around the pixel of interest (x + i, y + j), the pixel region (2p + 1) * (2q + 1) represented by q pixels above and below, and p pixels left and right respectively.
- the local region luminance l t (x, y) at the pixel position (x, y) of the target frame image (input frame image at time t) is the luminance information Y of the target frame image (input frame image at time t).
- t it can be expressed as in equation (1).
- Expression (2) a weighted average using a preset weight w may be used.
- the change rate of the local area luminance represents a fluctuation ratio of the local area luminance between the pixel of the target frame image and the pixel of another input frame image at the same position.
- the local region luminance fluctuation ratio r t ⁇ t + k (x, y) between the frame of interest that is the input frame at the time t at the pixel position (x, y) and the frame at the time (t + k) in the input frame group is expressed by Equation (4). It is calculated as follows.
- the flash frame determination unit 11 determines, based on the local region luminance fluctuation ratio, whether the region of interest includes an area that is brighter than other frame images. As a result, when the frame image of interest includes a region that is significantly brighter (brightness is increased) compared to other frame images that are temporally before and after, the frame of interest image blinks (has a higher luminance). It is determined that the frame image includes a variation region due to (change).
- the area ratio ⁇ (the ratio of the area where light blinks in the display area) is used. This is a method of determining whether or not the local area luminance fluctuation ratio r t ⁇ t + k exceeding the threshold value ⁇ exceeds the area ratio ⁇ .
- the flash frame determination unit 11 determines the flag flag t ⁇ t + k. Is 1. Otherwise, the flash frame determination unit 11 sets the determination flag flag t ⁇ t + k to 0. A determination flag is calculated for the combination of the target frame image and another predetermined input frame image. If there are frame images with the determination flag set to 1 at each of the preceding and following times of the target frame image, the target frame image It is determined that the frame image includes a variation area due to blinking.
- the flash frame determination unit 11 may use a method using a variation ratio of the luminance of the rectangular area as another method for determining whether the frame image of interest is a frame image including a fluctuation area due to blinking of light.
- the rectangular area luminance represents an average luminance value for each rectangular area designated in advance in each frame image.
- the rectangular area luminance in a rectangular area of 10 ⁇ 10 blocks is obtained by dividing an input frame image into rectangular areas of 10 ⁇ 10 blocks as shown in FIG.
- the index of the luminance value the Y value of the YUV color system, the Y value of the XYZ color system, the V value of the HSV color system, and the like are used as in the local area luminance calculation.
- the variation ratio of the rectangular area luminance represents the variation ratio of the rectangular area luminance of the target block of the target frame image and the rectangular area luminance of the block of another input frame image at the same position.
- R t ⁇ t + k (i, j) is calculated as shown in Equation (5).
- the determination using the rectangular region luminance variation ratio is performed by adding another frame image to the attention frame image in the combination of the attention frame image at time t and another predetermined input frame image. It is determined whether a region that is larger and brighter is included. A value of the determination flag is set, and when there is a frame image with the determination flag being 1 at each of the preceding and following times of the target frame image, it is determined that the target frame is a frame image including a change area due to light blinking.
- the threshold luminance ⁇ and the area rate ⁇ of the luminance fluctuation rate set in advance are used, and the luminance fluctuation ratio of the rectangular area exceeding the threshold ⁇ is the area rate ⁇ .
- the flash frame determination unit 11 outputs analysis information as a flash determination result.
- the analysis information includes a determination flag between the frame image of interest and another input frame image.
- the flash frame determination unit 11 can calculate a determination flag between input frame images other than the target frame image, similarly to the analysis information.
- the calculated determination flag may be output as auxiliary information together with the analysis information.
- the frame interpolation unit 12 inputs the input frame image group other than the target frame image and the analysis information output from the flash frame determination unit 11. When it is determined that the target frame image is a frame image including a fluctuation region due to the blinking of light, the frame interpolation unit 12 uses the input frame image group other than the attention frame image as the attention that has no fluctuation region due to the blinking of light. A frame image is generated and output as an interpolated frame image.
- the frame interpolation unit 12 first selects a frame image to be used for generating an interpolation frame image and synthesizes them to generate an interpolation frame image.
- a method of selecting a frame image there is a method of selecting a frame image that does not include a fluctuating region due to blinking of light before and after the closest to the frame image of interest.
- a specific selection method there is a method using a determination flag calculated by the flash frame determination unit 11. In this method, out of the frame images whose determination flag is 1, the respective frames before and after the closest to the target frame image are selected. For example, with respect to the frame image at time t including the fluctuation region shown in FIG.
- the selection of the use frame image may be corrected by using a determination flag between input frame images other than the target frame image which is input auxiliary information.
- the correction is assumed, for example, when a frame image at time (t + k) is selected as a use frame in selection using a determination flag between a frame image of interest and another frame image.
- the selected frame image is changed to time (t + k + 1).
- one of the methods for generating the interpolated frame image I es ⁇ t at time t is the input frame image I t + e at time (t + e) and using the input frame image I t-f of the time (t-f), is calculated by the equation (6).
- estimates a motion vector of each pixel of the input frame image I t + e and I t-f, the motion compensated interpolation, the method of generating an interpolation frame image I es-t It may be used.
- FIG. 4 is a schematic diagram for explaining the reason why the correction becomes unstable.
- a case where five frames from time (t ⁇ 2) to time (t + 2) are set as an input frame image group, and a case where a frame image including a fluctuating region continuous for three frames is included in the video sequence. Assumed.
- the input frame image group is case2 in FIG. 4, blinking can be detected and corrected.
- the frame image synthesis unit 13 synthesizes the target frame image and the interpolated frame image based on the determination result of the luminance variation in the target frame image (input frame image at time t), and outputs an output frame image in which blinking due to flash or the like is suppressed. Generate and output.
- the determination flag in the target frame image at time t is used. That is, when the determination flag is 0, the frame image synthesis unit 13 outputs the input frame image of interest (the input frame image at time t) as an output frame image.
- the determination flag is 1, the frame image synthesis unit 13 And the interpolated frame image are combined and output as an output frame image.
- the frame image composition unit 13 calculates a composition ratio in each pixel when it is determined that the frame image of interest is an image including a fluctuation region due to blinking of light and an interpolated frame image is generated. An output image is generated by processing. In other cases, the frame image synthesis unit 13 sets the input frame image of interest as an output image. Given the composition ratio u (x, y) at the pixel of interest at pixel position (x, y) and the value of the interpolated frame image I t-es (x, y) at time t, the value of the output image at that position I t-out (x, y) is calculated as shown in Equation (7).
- the composition ratio there is a method using a change rate of local area luminance between a target frame image and an interpolated frame image.
- the local area luminance change rate rt -es between the target frame image and the interpolated frame image can be calculated using the same method as the local area luminance change rate of the flash frame determination unit 11.
- the composition ratio u (x, y) at the pixel of interest at the pixel position (x, y) is determined by changing the local area luminance change rate r t-es (x, y) at the same position to a preset r t-es (
- the local area luminance change rate value r tar (x, y) in the output frame image corresponding to the value of x, y) is used.
- the local area luminance change rate in the output frame image can be calculated as in equation (8) so that r tar (x, y).
- a method of setting the local area luminance change rate value r tar in the output frame image as shown in the graph of FIG.
- r tar r t-es is set for a small value r t-es .
- r tar is set to a set maximum value and not larger than that value.
- the composition ratio there is also a method using the change rate of the luminance of the rectangular area. Specifically, the rectangular area luminance of the output image corresponding to the change rate R t-es of the rectangular area luminance calculated using the same method as that of the flash frame determination unit 11 and a preset value of R t-es.
- the composite ratio U for each rectangular area is calculated from the rate of change of.
- the synthesis ratio u for each pixel can be obtained from the calculated synthesis ratio ⁇ for each rectangular area using linear interpolation or bicubic interpolation.
- the flash frame determination unit 11 determines whether or not the frame image of interest at time t is a frame image that includes a fluctuating region due to flashing of light by flash or the like that may induce a photosensitivity attack (S11).
- the frame interpolation unit 12 is closest to the target frame image based on the input frame image group other than the target frame based on the analysis information that is the determination result of the luminance variation in the target frame image (the input frame image at time t). A frame image that does not include a fluctuation region due to blinking of the front and rear lights is selected. That is, the frame interpolation unit 12 receives the analysis information from the flash frame determination unit 11, and selects the input frame images before and after the closest to the target frame image from among the input frame images whose analysis information determination flag is 1. . Further, the frame interpolating unit 12 generates an interpolated frame image by using the selected frame images at the previous and subsequent times as a target frame image having no fluctuation region due to blinking of light (S12).
- the frame image composition unit 13 receives a determination result including analysis information from the flash frame determination unit 11. Subsequently, the frame image synthesis unit 13 synthesizes the attention frame image that is the input frame image at time t and the interpolation frame image based on the determination result of the luminance variation in the attention frame image (input frame image at time t), An output image in which blinking due to flash or the like is suppressed is generated and output (S13). That is, when the determination flag is 0, the input frame image of interest (the input frame image at time t) is output as an output frame image. When the determination flag is 1, the frame of interest image and the interpolation frame image are combined. To output as an output frame image.
- the effect of the first embodiment can generate an image in which fluctuations in luminance are suppressed with respect to an image including a large luminance change that may induce a photosensitivity seizure.
- the reason for this is that, for the frame image of interest, a frame image with no luminance change estimated from other frame images is synthesized while changing the synthesis weight for each pixel, thereby correcting mainly areas with large luminance changes, This is because information lost due to overexposure can be restored.
- the present invention is not limited to this.
- the present invention can be similarly applied to the case where the fluctuation region is a dark region that is darker (the luminance is smaller) in the frame image of interest than the other frame images.
- luminance variation that is, a general flash has been described, but the present invention can be similarly applied to saturation variation such as red flash.
- FIG. 7 is a block diagram of an image processing apparatus according to the second embodiment of the present invention.
- the image processing apparatus includes a flash frame determination unit 21, a frame interpolation unit 22, and a frame image synthesis unit 23.
- a group of frame images from time (t ⁇ m) to time (t + m) composed of a preset number of frames m is input and blinking is performed.
- An output frame image with reduced influence is generated.
- the frame image composition unit 13 calculates the composition ratio
- the local area luminance variation ratio and the rectangular area luminance variation ratio between the target frame image and the interpolation frame image are calculated. Need to be calculated, and the processing amount becomes relatively large.
- the image processing apparatus according to the second embodiment of the present invention is based on a local area luminance variation rate or a rectangular region luminance variation ratio between input frame images. By interpolating the variation rate of the local region luminance and the variation rate of the rectangular region luminance, the amount of calculation for calculating these is suppressed.
- the flash frame determination unit 21 includes an input frame image group as an input, and includes a fluctuation region due to blinking that causes the frame image of interest at time t to harm the viewer. It is determined whether it is a frame image.
- the flash frame determination unit 21 calculates the local region luminance variation ratio and the rectangular region luminance variation ratio, and calculates the calculated local region luminance variation ratio and the rectangle. The only difference is that the region luminance fluctuation ratio is output.
- the flash frame determination unit 21 localizes the frame image of interest at the time t at the pixel position (x, y) and the frame image at the time (t + k) in the input frame image group.
- the area luminance fluctuation ratio r ′ t ⁇ t + k (x, y) is calculated as in Expression (9).
- the calculation formula of Formula (9) differs from the calculation formula of Formula (4) in that the denominator uses the local region luminance of the target frame image.
- the rectangular area luminance variation ratio is used for flash frame determination, the rectangular area luminance of the target frame at time t at the block position (i, j) and the rectangular area of the frame at time (t + k) in the input frame group.
- the luminance variation ratio R ′ t ⁇ t + k (i, j) is calculated as in Expression (10).
- ⁇ is set as the threshold value of the luminance variation rate.
- the threshold value ⁇ ′ of the luminance variation rate in the flash frame determination unit 21 may be set as shown in Expression (11).
- the frame interpolation unit 22 has the same function as the frame interpolation unit 12 of the first embodiment.
- the frame interpolation unit 22 generates an attention frame image that does not have a fluctuation region due to blinking of light, and at the same time, based on the input region luminance fluctuation rate, a region luminance between the attention frame image and the generated interpolation frame image Is generated and output as the interpolated area luminance fluctuation rate.
- the frame interpolation unit 22 calculates the local area luminance fluctuation rate rt -es between the target frame image and the generated interpolated frame image.
- the frame interpolation unit 22 calculates the rectangular area luminance fluctuation rate R t-es between the frame image of interest and the generated interpolated frame image.
- the frame interpolation unit 22 receives analysis information from the flash frame determination unit 21. Similar to the frame interpolation unit 12, the frame interpolation unit 22, based on the analysis information that is the determination result of the luminance variation in the target frame image (input frame image at time t), the light before and after the closest to the target frame image. An interpolated frame image is generated as a focused frame image that does not include a fluctuation region due to blinking. That is, the frame interpolation unit 22 selects the input frame images before and after the closest to the target frame image from among the input frame images whose analysis information determination flag is 1.
- the frame interpolation unit 22 generates an interpolated frame image by using the selected frame images at the previous and subsequent times as a target frame image having no fluctuation region due to the blinking of light. Further, the frame interpolation unit 22 synthesizes the input region luminance fluctuation rate by the same processing as that of the interpolation frame image, and calculates the composite region luminance fluctuation rate. For example, when the local area luminance fluctuation rate is used for flash frame determination and the equation (6) is used for interpolation frame image calculation, the combined local area luminance fluctuation rate r ′ t at the pixel position (x, y). -Es is calculated as shown in Equation (12).
- the frame interpolation unit 22 calculates the interpolation area luminance fluctuation rate to be output by the calculation formula of Formula (14) or Formula (15).
- the frame image composition unit 23 receives the determination result from the flash frame determination unit 21. Similar to the frame image synthesis unit 13 of the first embodiment, the frame image synthesis unit 23 synthesizes the target frame image and the interpolated frame image based on the analysis information included in the determination result of the target frame image, and performs flash Generate and output an output image that suppresses blinking due to the above.
- the frame image composition unit 23 calculates the interpolated area luminance fluctuation calculated by the frame interpolation unit 22 for calculating the composition weight of the target frame image and the interpolated frame image. It is different in using rate.
- the flash frame determination unit 21 determines whether or not the frame image of interest at time t is a frame image that includes a variation region due to blinking of light due to flash or the like that may induce a photosensitivity seizure. Based on the determination (S21).
- the frame interpolation unit 22 generates an interpolated frame image that is an attention frame image that does not have a fluctuation region due to blinking of light based on the input frame image group other than the attention frame (S22).
- the frame interpolation unit 22 calculates a region luminance fluctuation rate between the target frame image and the interpolation frame image (S23).
- the frame image synthesis unit 23 synthesizes the target frame image and the interpolation frame image, and generates and outputs an output image in which blinking due to flash or the like is suppressed (S24).
- FIG. 9 is a block diagram of an image processing apparatus according to the third embodiment of the present invention.
- the image processing apparatus includes an acquisition unit 30 and a correction unit 31.
- the acquisition unit 30 acquires the luminance change rate in the time direction of the video and a predetermined ratio in the entire area of the frame image.
- the rate of change in luminance in the time direction is, for example, the threshold value ⁇ of the luminance variation rate described above.
- the acquisition unit 30 acquires these values from the outside and holds them.
- the correction unit 31 inputs an image having a luminance change that exceeds the change rate at the above-described ratio, and the ratio or the change rate of the luminance between the frames forming the video is less than the acquired change rate or change rate of the luminance.
- the luminance of at least a partial region of at least a part of the frame forming the video is corrected. Specifically, by identifying a region of a frame that exceeds the rate or rate of change in luminance, and replacing the luminance of that region with, for example, the luminance of a pixel in the same position as the region of the other frame, As a whole, an image with no change in luminance exceeding the rate of change at the rate described above is generated.
- the luminance of the region may be replaced with an average value or intermediate luminance of the luminance of the pixels in the region at the same position as the region of the previous and subsequent frames.
- Blinks that can induce photosensitivity seizures may occur not only in the entire frame, but also in some areas of the frame.
- the related methods so far correct the entire frame uniformly without discriminating the blinking location.
- the image processing apparatus according to the present embodiment can suppress performing correction more than necessary for an area that does not need to be corrected and does not blink.
- the embodiment according to the present invention can be applied to a video editing system for editing video recorded on a hard disk or the like.
- the present invention can be applied to a video camera, a display terminal, or the like.
- FIG. 10 is a diagram illustrating a hardware configuration in which the image processing apparatus according to the first to third embodiments is realized by a computer apparatus.
- the image processing apparatus includes a CPU (Central Processing Unit) 91, a communication I / F (communication interface) 92 for network connection, a memory 93, and a storage device 94 such as a hard disk for storing programs. Including.
- CPU Central Processing Unit
- I / F communication interface
- storage device 94 such as a hard disk for storing programs.
- the CPU 91 of the image processing apparatus is connected to an input device 95 and an output device 96 via a system bus 97.
- the CPU 91 operates the operating system to control each functional unit in the first to third embodiments. Further, the CPU 91 reads out programs and data from the recording medium mounted on the drive device to the memory 93, for example.
- the CPU 91 has a function of processing an input information signal, for example, and executes various functions based on a program.
- the storage device 94 is, for example, an optical disk, a flexible disk, a magnetic optical disk, an external hard disk, or a semiconductor memory. A part of the storage medium of the storage device 94 is a nonvolatile storage device, and stores a program therein.
- the program is connected to a communication network.
- the input device 95 is realized by, for example, a mouse, a keyboard, a built-in key button, a card slot, or a touch panel, and is used for an input operation.
- the output device 96 is realized by a display, for example, and is used for outputting and confirming information processed by the CPU 91.
- the first to third embodiments are realized by the hardware configuration shown in FIG.
- the means for realizing each unit included in the image processing apparatus is not particularly limited. That is, each realization means may be realized by one physically coupled device, or two or more physically separated devices may be connected by wire or wirelessly and realized by these plural devices. Good.
- a determination unit that determines whether the frame image includes a variable region;
- a frame generation unit that generates an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image;
- An image processing apparatus comprising: a frame image synthesis unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
- Additional remark 2 The image processing apparatus of Additional remark 1 which selects the frame image of each time before and after which adjoins temporally which does not contain the said fluctuation
- the fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
- luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness
- An image processing apparatus according to 1.
- a frame image of interest in a video whose luminance or saturation changes in a contradictory manner is compared with the frame images before and after the image, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold value. Determine whether it is a frame image that includes a variable region, Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated, An image processing method for generating an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
- Additional remark 8 The image processing method of Additional remark 7 which selects the frame image of each time before and behind which does not contain the said fluctuation
- the fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
- luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness
- luminance or saturation changes contrary is compared with the frame image before and behind it, and the variation
- Additional remark 14 The program of Additional remark 13 which selects the frame image of each time before and after which adjoins temporally which does not contain the said fluctuation
- the fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
- Supplementary note 17 calculates the fluctuation ratio of the local area luminance between the frame image and the interpolation frame image using the fluctuation ratio of the local area luminance between the frame image of interest and the frame image selected to generate the interpolation frame image. The program described in.
- An acquisition unit that acquires a change rate of luminance in a time direction of a video and a predetermined ratio in the entire region of the frame image;
- the video having a change in luminance that exceeds the rate of change at the rate, the rate or change rate of the luminance between each frame forming the video is less than the acquired rate of change of luminance or the rate of change.
- An image processing apparatus comprising: a correction unit that corrects luminance of at least a part of a region of a part of the frame.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Transforming Electric Information Into Light Information (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Controls And Circuits For Display Device (AREA)
- Television Systems (AREA)
Abstract
The present invention provides an image processing device, an image processing method, and a program which are capable of generating, for a video having the possibility of inducing photosensitive epilepsy, a video in which a change in luminance or saturation is suppressed. The present invention is an image processing device having: a determination unit for comparing a frame image of interest and frame images preceding and following the frame image of interest in a video in which luminance or saturation changes reciprocally, and determining whether the frame image of interest includes a change area in which a luminance or saturation change rate exceeds a prescribed threshold; a frame generation unit for generating, on the basis of frame images other than the frame image of interest, an interpolated frame image which is a frame image of interest in which the change area is nonexistent; and a frame image synthesis unit for generating, on the basis of the result of determination by the determination means, an outputted frame image in which the frame image of interest and the interpolated frame image are synthesized.
Description
本発明は、画像処理装置、画像処理方法及びプログラムを記録した記録媒体に関する。
The present invention relates to an image processing apparatus, an image processing method, and a recording medium on which a program is recorded.
映像コンテンツの中には、視聴者にめまいや吐き気などの症状を伴う光過敏性発作を発症させるものがある。光過敏性発作は、視覚に飛び込んだ光刺激に対する異常反応であり、痙攣や意識障害などの癲癇に似た症状を伴う。
Some video content causes viewers to develop photosensitivity attacks with symptoms such as dizziness and nausea. Photosensitivity seizures are abnormal responses to light stimuli that jump into the eye, with symptoms resembling epilepsy such as convulsions and impaired consciousness.
このような影響が視聴者に及ぶことを防ぐために、人体に影響を与える映像コンテンツの配信の抑制が社会的に試みられている。例えば、国際電気通信連合(International Telecommunication Union;ITU)は、映像コンテンツ製作者に対し映像コンテンツは光過敏性発作を発生させる危険性があることを周知するよう映像配信組織に勧告している(非特許文献1)。また、日本においては、日本放送協会と日本民間放送連盟が、特にアニメ―ションの製作に関しガイドラインを制定し、放送に携わるものに遵守するよう求めている(非特許文献2)。
In order to prevent such influence from affecting viewers, attempts have been made to suppress the distribution of video content that affects the human body. For example, the International Telecommunications Union (ITU) recommends video distribution organizations to inform video content producers that video content is at risk of causing photosensitivity attacks (non- Patent Document 1). In Japan, the Japan Broadcasting Corporation and the Japan Broadcasting Corporation have established guidelines for the production of animations in particular, and are demanding compliance with those involved in broadcasting (Non-Patent Document 2).
しかし、報道映像のような速報性が求められる映像コンテンツを生放送で放送する際に、人体に影響を与える映像コンテンツの配信自体を抑制することが困難な場合がある。このため現状では、撮影環境が光過敏性発作を誘発する可能性のある多くの明滅(例えば、大きく輝度が変化する一般閃光)や、彩度の大きな変化(例えば、赤色閃光)を含んでいる場合、テロップ等で視聴者に注意を喚起する手段がとられている。報道映像において、光過敏性発作を誘発する可能性のある多くの明滅を含む撮影環境の一つに、記者会見等がある。記者会見では、報道カメラマンから撮影時に発せられるフラッシュによる短時間の明領域が繰り返し発生し、映像コンテンツ内に多くの明滅が繰り返されることになる。
However, it may be difficult to suppress the distribution itself of video content that affects the human body when broadcasting live video content that requires rapid reporting, such as news footage. For this reason, the current shooting environment includes many flickers (for example, general flashes with large brightness changes) and large changes in saturation (for example, red flashes) that can trigger photosensitivity attacks. In such a case, means for alerting the viewer with a telop or the like is taken. One of the shooting environments, including many flickers that can trigger photosensitivity attacks in news footage, is a press conference. In a press conference, a short-time bright area is repeatedly generated by a flash emitted from a news photographer at the time of shooting, and many blinks are repeated in the video content.
これまで、配信を抑止できなかった人体に影響を与える映像コンテンツを自動的に検出し、補正する方法が提案されている。
[To date, a method has been proposed for automatically detecting and correcting video content that affects the human body, for which distribution could not be suppressed.
特許文献1では、液晶ディスプレイにおいて、光過敏性発作を誘起するシーン(画像)を検出し、光過敏性発作を誘起するシーン(画像)に対し、バックライトユニットの輝度を低下させることによって、視聴者への光過敏性発作の影響を未然に防止する。
In Patent Document 1, a liquid crystal display detects a scene (image) that induces a photosensitivity seizure, and reduces the luminance of the backlight unit with respect to the scene (image) that induces the photosensitivity seizure, thereby allowing viewing and listening. Prevent the effects of photosensitivity attacks on the elderly.
特許文献2では、第nフレーム画像と第(n+1)フレーム画像のヒストグラム比較結果に基づいて、ガンマ補正またはトーンカーブ補正による第(n+1)フレーム画像のダイナミックレンジを狭める補正をすることによって、強い明滅(輝度変化)を緩和し、眼精疲労または体調不良を低減させる。
In Patent Document 2, strong blinking is performed by performing correction for narrowing the dynamic range of the (n + 1) th frame image by gamma correction or tone curve correction based on the histogram comparison result between the nth frame image and the (n + 1) th frame image. Reducing (luminance change) and reducing eye strain or poor physical condition.
しかしながら、関連する技術には、下記の課題がある。
However, related technologies have the following problems.
光過敏性発作を誘起する可能性のある明滅(大きな輝度変化)や彩度の大きな変化は、画像全体だけではなく、映像の一部領域に発生する場合がある。関連する手法は、これらの判別を行わず画像全体一律に補正するため、本来補正する必要のない、輝度や彩度が大きく変化していない領域においてコントラストや明度の低下が発生し、その領域の画質が劣化する場合がある。
明 Flickering (large brightness change) and large change in saturation that can induce photosensitivity seizures may occur not only in the entire image but also in some areas of the video. Since the related method corrects the entire image uniformly without making these determinations, a decrease in contrast or brightness occurs in an area that does not need to be corrected originally and the luminance or saturation does not change significantly. The image quality may deteriorate.
また、フラッシュ等による明滅の場合には、フラッシュの当てられた領域の画素の一部の色情報がカメラのダイナミックレンジを超えて飽和している場合がある。飽和した画素については、もともとの情報が失われている。このため、そのフレーム画像のみを用いた補正処理のみでは、過度の彩度を持つ画素を発生させる場合や、過少の彩度を持つ画素を発生させる場合があり、色味の変動を抑制することができない。その結果、そのフレーム画像のみを用いた補正処理のみでは、自然な輝度又は彩度の変動の緩和を実現することは困難である。
Also, in the case of blinking due to flash or the like, color information of a part of pixels in the area where the flash is applied may be saturated beyond the dynamic range of the camera. For saturated pixels, the original information is lost. For this reason, only correction processing using only the frame image may generate pixels with excessive saturation or may generate pixels with insufficient saturation, thereby suppressing variation in color. I can't. As a result, it is difficult to realize natural relaxation of fluctuations in luminance or saturation with only correction processing using only the frame image.
そこで、本発明の目的は、光過敏性発作を誘起する可能性のある映像に対して、輝度又は彩度の変動が抑制された映像を生成することができる画像処理装置、画像処理方法及びプログラムを提供することにある。
Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of generating a video in which fluctuations in luminance or saturation are suppressed with respect to a video that may induce a photosensitivity seizure. Is to provide.
本発明は、輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定する判定部を有する。更に、前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成するフレーム生成部を有する。更に、前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成するフレーム画像合成部と有する画像処理装置である。
The present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It has a determination part which determines whether it is a frame image containing a change area. In addition, a frame generation unit that generates an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image. Furthermore, the image processing apparatus includes a frame image synthesis unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolation frame image based on a determination result of the determination unit.
本発明は、輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定する。更に、前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成する。更に、前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する画像処理方法である。
The present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It is determined whether the frame image includes a fluctuation region. Furthermore, an interpolated frame image that is an attention frame image without the fluctuation region is generated based on a frame image other than the attention frame image. Further, the image processing method generates an output frame image obtained by combining the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
本発明は、輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定する。更に、前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成する。更に、前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する、ことをコンピュータに実行させるプログラムである。
The present invention compares a frame image of interest and a frame image before and after the image in which the luminance or saturation changes in a contradictory manner, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold. It is determined whether the frame image includes a fluctuation region. Furthermore, an interpolated frame image that is an attention frame image without the fluctuation region is generated based on a frame image other than the attention frame image. The program further causes the computer to generate an output frame image obtained by combining the frame image of interest and the interpolated frame image based on the determination result of the determination unit.
本発明によれば、光過敏性発作を誘起する可能性のある映像に対して、輝度又は彩度の変動が抑制された映像を生成することができる。
According to the present invention, it is possible to generate an image in which variation in luminance or saturation is suppressed with respect to an image that may induce a photosensitivity seizure.
本発明の実施の形態について図面を参照して説明する。
Embodiments of the present invention will be described with reference to the drawings.
(第1の実施の形態)
図1は、本発明による第1の実施の形態における画像処理装置のブロック図である。 (First embodiment)
FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment of the present invention.
図1は、本発明による第1の実施の形態における画像処理装置のブロック図である。 (First embodiment)
FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment of the present invention.
第1の実施の形態の画像処理装置は、フラッシュフレーム判定部11とフレーム補間部12とフレーム画像合成部13とを有する。そして、第1の実施の形態の画像処理装置は、あらかじめ設定されたフレーム数2m+1で構成される時刻(t-m)から時刻(t+m)までのフレーム画像群を入力する。更に、画像処理装置は、時刻tの入力フレーム画像である注目フレーム画像が、光過敏性発作を誘発する可能性のある、フラッシュ等による明滅(輝度の大きな変動)による変動領域を含むフレーム画像であるかを判定する。変動領域が含まれる場合、画像処理装置は、補正処理によって変動領域を適切に抑制することで明滅の影響を低減させた(輝度の変動が抑制された)出力フレーム画像を生成する。尚、以下の説明において、変動領域は、注目フレーム画像において他フレーム画像より大きく明るくなる(輝度が大きくなる)明領域を意味するものとする。
The image processing apparatus according to the first embodiment includes a flash frame determination unit 11, a frame interpolation unit 12, and a frame image synthesis unit 13. Then, the image processing apparatus according to the first embodiment inputs a group of frame images from time (t−m) to time (t + m) configured by a preset number of frames 2m + 1. Further, the image processing apparatus is a frame image including a variation region caused by blinking (a large variation in luminance) due to flashing or the like, in which the attention frame image that is the input frame image at time t may induce a photosensitivity seizure. Determine if there is. When the variation area is included, the image processing apparatus generates an output frame image in which the influence of blinking is reduced (luminance variation is suppressed) by appropriately suppressing the variation area by the correction process. In the following description, the fluctuation region means a bright region that is brighter (brightness increases) in the frame image of interest than in other frame images.
フラッシュフレーム判定部11は、入力フレーム画像群を入力とし、注目フレーム画像(時刻tの入力フレーム画像)が視聴者に影響を与える明滅(輝度の大きな変動)による変動領域を含むフレーム画像であるかを判定する。
The flash frame determination unit 11 receives an input frame image group as an input, and whether the frame image of interest (the input frame image at time t) is a frame image including a variation area due to blinking (a large variation in luminance) that affects the viewer. Determine.
注目フレーム画像が変動領域を含むフレームであるかの判別方法の一つに、注目フレーム画像と他の入力フレーム画像との間の局所領域輝度の変化率を用いる方法がある。
One method for determining whether a target frame image is a frame including a variable region is a method of using the rate of change in local region luminance between the target frame image and another input frame image.
局所領域輝度は、入力フレーム画像群の各画素における、画素周辺領域の輝度値を表す。局所領域輝度の算出方法として、まず、入力フレーム画像群の各画素について、RGB(Red-Green-Blue)などで表記される色情報から明るさを表す輝度情報に変換し、その後、変換された輝度情報に対して、注目画素周辺に対し平滑化処理を施すことで、画素周辺領域の輝度値を算出する方法がある。
Local area luminance represents the luminance value of the pixel peripheral area in each pixel of the input frame image group. As a method of calculating the local area luminance, first, for each pixel of the input frame image group, color information expressed in RGB (Red-Green-Blue) or the like is converted into luminance information representing brightness, and then converted. There is a method of calculating the luminance value of the pixel peripheral region by performing smoothing processing on the periphery of the target pixel with respect to the luminance information.
色情報から輝度情報への変換方法として、放送用に用いられる表色系であるYUV(luminance component (Y) and color difference components (U and V)),YCbCr又はYPbPrの輝度を表すY値を算出する方法や、XYZ表色系のY値を算出する方法がある。HSV(Hue-Saturation-Value)表色系のV値等、他の輝度を表す指標に変換してもよい。また、入力フレーム画像群が、表示のために予めガンマ補正が施されている場合には、輝度情報への変換の前に、逆ガンマ補正により正しい色情報に変換してもよい。
As a conversion method from color information to luminance information, YUV (luminance component (Y) and color difference components (U and V)), YCbCr or YPbPr luminance, which is a color system used for broadcasting, is calculated. And a method of calculating the Y value of the XYZ color system. You may convert into the index | index showing other luminances, such as V value of HSV (Hue-Saturation-Value) color system. Further, when the input frame image group is previously subjected to gamma correction for display, it may be converted to correct color information by inverse gamma correction before conversion to luminance information.
平滑化処理の方式として、注目画素周辺(x+i,y+j)と、その上と下それぞれq画素分、その左と右それぞれp画素分で表される画素領域(2p+1)*(2q+1)で平均値を取る方法がある。この場合、注目フレーム画像(時刻tの入力フレーム画像)の画素位置(x,y)における局所領域輝度lt(x,y)は、注目フレーム画像(時刻tの入力フレーム画像)の輝度情報Ytを用いて、式(1)のように表すことができる。
また、式(2)のように、あらかじめ設定された重みwを用いた重み付き平均を用いてもよい。
重みの設定方法として、設定されたパラメタσを用いて式(3)で算出されるガウシアン重みを用いる方法がある。
局所領域輝度の変化率は、注目フレーム画像の画素と、同位置の他の入力フレーム画像の画素との間の局所領域輝度の変動比率を表す。画素位置(x,y)における時刻tにおける入力フレームである注目フレームと入力フレーム群内の時刻(t+k)のフレームの局所領域輝度変動比率rt-t+k (x,y)は、式(4)のように算出される。
フラッシュフレーム判定部11は、局所領域輝度変動比率をもとに、注目フレーム画像に他フレーム画像より大きく明るくなる領域が含まれているかどうかを判定する。その結果、注目フレーム画像が、時間的に前後にある他フレーム画像と比較して、大きく明るくなる(輝度が大きくなる)領域が含まれている場合には、注目フレーム画像は明滅(輝度の大きな変化)による変動領域を含むフレーム画像であると判定される。 As a smoothing method, an average value is obtained around the pixel of interest (x + i, y + j), the pixel region (2p + 1) * (2q + 1) represented by q pixels above and below, and p pixels left and right respectively. There is a way. In this case, the local region luminance l t (x, y) at the pixel position (x, y) of the target frame image (input frame image at time t) is the luminance information Y of the target frame image (input frame image at time t). Using t , it can be expressed as in equation (1).
Further, as in Expression (2), a weighted average using a preset weight w may be used.
As a method for setting the weight, there is a method using the Gaussian weight calculated by Expression (3) using the set parameter σ.
The change rate of the local area luminance represents a fluctuation ratio of the local area luminance between the pixel of the target frame image and the pixel of another input frame image at the same position. The local region luminance fluctuation ratio r t−t + k (x, y) between the frame of interest that is the input frame at the time t at the pixel position (x, y) and the frame at the time (t + k) in the input frame group is expressed by Equation (4). It is calculated as follows.
The flashframe determination unit 11 determines, based on the local region luminance fluctuation ratio, whether the region of interest includes an area that is brighter than other frame images. As a result, when the frame image of interest includes a region that is significantly brighter (brightness is increased) compared to other frame images that are temporally before and after, the frame of interest image blinks (has a higher luminance). It is determined that the frame image includes a variation region due to (change).
また、式(2)のように、あらかじめ設定された重みwを用いた重み付き平均を用いてもよい。
重みの設定方法として、設定されたパラメタσを用いて式(3)で算出されるガウシアン重みを用いる方法がある。
局所領域輝度の変化率は、注目フレーム画像の画素と、同位置の他の入力フレーム画像の画素との間の局所領域輝度の変動比率を表す。画素位置(x,y)における時刻tにおける入力フレームである注目フレームと入力フレーム群内の時刻(t+k)のフレームの局所領域輝度変動比率rt-t+k (x,y)は、式(4)のように算出される。
フラッシュフレーム判定部11は、局所領域輝度変動比率をもとに、注目フレーム画像に他フレーム画像より大きく明るくなる領域が含まれているかどうかを判定する。その結果、注目フレーム画像が、時間的に前後にある他フレーム画像と比較して、大きく明るくなる(輝度が大きくなる)領域が含まれている場合には、注目フレーム画像は明滅(輝度の大きな変化)による変動領域を含むフレーム画像であると判定される。 As a smoothing method, an average value is obtained around the pixel of interest (x + i, y + j), the pixel region (2p + 1) * (2q + 1) represented by q pixels above and below, and p pixels left and right respectively. There is a way. In this case, the local region luminance l t (x, y) at the pixel position (x, y) of the target frame image (input frame image at time t) is the luminance information Y of the target frame image (input frame image at time t). Using t , it can be expressed as in equation (1).
Further, as in Expression (2), a weighted average using a preset weight w may be used.
As a method for setting the weight, there is a method using the Gaussian weight calculated by Expression (3) using the set parameter σ.
The change rate of the local area luminance represents a fluctuation ratio of the local area luminance between the pixel of the target frame image and the pixel of another input frame image at the same position. The local region luminance fluctuation ratio r t−t + k (x, y) between the frame of interest that is the input frame at the time t at the pixel position (x, y) and the frame at the time (t + k) in the input frame group is expressed by Equation (4). It is calculated as follows.
The flash
注目フレーム画像(時刻tの入力フレーム画像)が入力フレーム画像群内の時刻(t+k)のフレーム画像より大きく明るくなる領域があるか否かの判定には、予め設定した輝度変動率の閾値αと面積率β(表示領域のうち光の明滅が発生する面積の割合)を用いる。これは、閾値αを超える局所領域輝度変動比率rt-t+kが面積率βを超えるか否かによって判定する方法である。この方法では、例えば、日本放送協会と日本民間放送連盟によるガイドラインの基準の一つである、「点滅が同時に起こる面積が画面1/4を超え、かつ、輝度変化が10%以上」であるかを検知するためには、α=0.1、β=0.25を設定すればよい。
In order to determine whether or not there is a region in which the frame image of interest (the input frame image at time t) is larger and brighter than the frame image at time (t + k) in the input frame image group, The area ratio β (the ratio of the area where light blinks in the display area) is used. This is a method of determining whether or not the local area luminance fluctuation ratio r t−t + k exceeding the threshold value α exceeds the area ratio β. In this method, for example, is one of the guidelines of the guidelines of the Japan Broadcasting Corporation and the Japan Broadcasting Corporation, “whether the area where the flashing occurs simultaneously exceeds 1/4 of the screen and the luminance change is 10% or more” Can be detected by setting α = 0.1 and β = 0.25.
フラッシュフレーム判定部11は、注目フレーム画像(時刻tの入力フレーム画像)が入力フレーム画像群内の時刻(t+k)のフレームより大きく明るくなる領域があると判定された場合、判定フラグflagt-t+kを1とする。そうでない場合には、フラッシュフレーム判定部11は、判定フラグflagt-t+kを0とする。そして、注目フレーム画像と他の所定の入力フレーム画像との組み合わせにおいて、判定フラグを算出し、注目フレーム画像の前後時刻それぞれに判定フラグが1となるフレーム画像が存在する場合、注目フレーム画像が光の明滅による変動領域を含むフレーム画像であると判定する。
When it is determined that there is an area where the frame image of interest (the input frame image at time t) is brighter than the frame at time (t + k) in the input frame image group, the flash frame determination unit 11 determines the flag flag t−t + k. Is 1. Otherwise, the flash frame determination unit 11 sets the determination flag flag t−t + k to 0. A determination flag is calculated for the combination of the target frame image and another predetermined input frame image. If there are frame images with the determination flag set to 1 at each of the preceding and following times of the target frame image, the target frame image It is determined that the frame image includes a variation area due to blinking.
フラッシュフレーム判定部11は、注目フレーム画像が光の明滅による変動領域を含むフレーム画像であるか判別する別の方法として、矩形領域輝度の変動比率を用いる方法を利用してもよい。矩形領域輝度は、各フレーム画像におけるあらかじめ指定された矩形領域毎の平均輝度値を表す。例えば、10×10ブロックの矩形領域での矩形領域輝度は、図2のように、入力フレーム画像を10×10ブロックの矩形領域に分割し、それぞれの領域について平均輝度値を表す。輝度値の指標は、局所領域輝度算出と同様に、YUV表色系のY値、XYZ表色系のY値、HSV表色系のV値等を用いる。
The flash frame determination unit 11 may use a method using a variation ratio of the luminance of the rectangular area as another method for determining whether the frame image of interest is a frame image including a fluctuation area due to blinking of light. The rectangular area luminance represents an average luminance value for each rectangular area designated in advance in each frame image. For example, the rectangular area luminance in a rectangular area of 10 × 10 blocks is obtained by dividing an input frame image into rectangular areas of 10 × 10 blocks as shown in FIG. As the index of the luminance value, the Y value of the YUV color system, the Y value of the XYZ color system, the V value of the HSV color system, and the like are used as in the local area luminance calculation.
矩形領域輝度の変動比率は、注目フレーム画像の注目ブロックの矩形領域輝度と、同位置の他の入力フレーム画像のブロックの矩形領域輝度の変動比率を表す。ブロック位置(i,j)における時刻tにおける注目フレームの矩形領域輝度Lt(i,j)と入力フレーム群内の時刻(t+k)のフレームの矩形領域輝度Lt+k(i,j)の変動比率Rt-t+k(i,j)は、式(5)のように算出される。
矩形領域輝度変動比率を用いた判定は、局所領域輝度変動比率を用いた判定と同様に、時刻tにおける注目フレーム画像と他の所定の入力フレーム画像との組み合わせにおいて、注目フレーム画像に他フレーム画像より大きく明るくなる領域が含まれているか判定する。判定フラグの値を設定しておき、注目フレーム画像の前後時刻それぞれに判定フラグが1となるフレーム画像が存在する場合、注目フレームが光の明滅による変動領域を含むフレーム画像であると判定する。 The variation ratio of the rectangular area luminance represents the variation ratio of the rectangular area luminance of the target block of the target frame image and the rectangular area luminance of the block of another input frame image at the same position. Variation ratio between the rectangular area luminance L t (i, j) of the frame of interest at time t at the block position (i, j) and the rectangular area luminance L t + k (i, j) of the frame at time (t + k) in the input frame group. R t−t + k (i, j) is calculated as shown in Equation (5).
Similar to the determination using the local region luminance variation ratio, the determination using the rectangular region luminance variation ratio is performed by adding another frame image to the attention frame image in the combination of the attention frame image at time t and another predetermined input frame image. It is determined whether a region that is larger and brighter is included. A value of the determination flag is set, and when there is a frame image with the determination flag being 1 at each of the preceding and following times of the target frame image, it is determined that the target frame is a frame image including a change area due to light blinking.
矩形領域輝度変動比率を用いた判定は、局所領域輝度変動比率を用いた判定と同様に、時刻tにおける注目フレーム画像と他の所定の入力フレーム画像との組み合わせにおいて、注目フレーム画像に他フレーム画像より大きく明るくなる領域が含まれているか判定する。判定フラグの値を設定しておき、注目フレーム画像の前後時刻それぞれに判定フラグが1となるフレーム画像が存在する場合、注目フレームが光の明滅による変動領域を含むフレーム画像であると判定する。 The variation ratio of the rectangular area luminance represents the variation ratio of the rectangular area luminance of the target block of the target frame image and the rectangular area luminance of the block of another input frame image at the same position. Variation ratio between the rectangular area luminance L t (i, j) of the frame of interest at time t at the block position (i, j) and the rectangular area luminance L t + k (i, j) of the frame at time (t + k) in the input frame group. R t−t + k (i, j) is calculated as shown in Equation (5).
Similar to the determination using the local region luminance variation ratio, the determination using the rectangular region luminance variation ratio is performed by adding another frame image to the attention frame image in the combination of the attention frame image at time t and another predetermined input frame image. It is determined whether a region that is larger and brighter is included. A value of the determination flag is set, and when there is a frame image with the determination flag being 1 at each of the preceding and following times of the target frame image, it is determined that the target frame is a frame image including a change area due to light blinking.
判定フラグの値の設定方法に、局所輝度変動率を用いる場合と同様に、あらかじめ設定した輝度変動率の閾値αと面積率βを用いて、閾値αを超える矩形領域輝度変動比率が面積率βを超えるか否かによって、1または0を設定する方法がある。
As in the case of using the local luminance fluctuation rate in the determination flag value setting method, the threshold luminance α and the area rate β of the luminance fluctuation rate set in advance are used, and the luminance fluctuation ratio of the rectangular area exceeding the threshold α is the area rate β. There is a method of setting 1 or 0 depending on whether or not.
フラッシュフレーム判定部11は、フラッシュ判定結果として解析情報を出力する。その解析情報には、注目フレーム画像と他の入力フレーム画像間との判定フラグが含まれる。また、フラッシュフレーム判定部11は、解析情報と同様に、注目フレーム画像以外の入力フレーム画像間同士での判定フラグを算出できる。算出された判定フラグを補助情報として、解析情報と共に出力してもよい。
The flash frame determination unit 11 outputs analysis information as a flash determination result. The analysis information includes a determination flag between the frame image of interest and another input frame image. Moreover, the flash frame determination unit 11 can calculate a determination flag between input frame images other than the target frame image, similarly to the analysis information. The calculated determination flag may be output as auxiliary information together with the analysis information.
フレーム補間部12は、注目フレーム画像以外の入力フレーム画像群とフラッシュフレーム判定部11から出力された解析情報を入力する。注目フレーム画像が光の明滅による変動領域を含むフレーム画像と判定された場合に、フレーム補間部12は、注目フレーム画像以外の入力フレーム画像群をもとに、光の明滅による変動領域がない注目フレーム画像を生成し、補間フレーム画像として出力する。
The frame interpolation unit 12 inputs the input frame image group other than the target frame image and the analysis information output from the flash frame determination unit 11. When it is determined that the target frame image is a frame image including a fluctuation region due to the blinking of light, the frame interpolation unit 12 uses the input frame image group other than the attention frame image as the attention that has no fluctuation region due to the blinking of light. A frame image is generated and output as an interpolated frame image.
フレーム補間部12は、まず、補間フレーム画像生成のために利用するフレーム画像を選択し、それらを合成することによって、補間フレーム画像を生成する。
The frame interpolation unit 12 first selects a frame image to be used for generating an interpolation frame image and synthesizes them to generate an interpolation frame image.
フレーム画像の選択方法として、注目フレーム画像に最も近接する前後それぞれの光の明滅による変動領域を含まないフレーム画像を選択方法がある。具体的な選択方法として、フラッシュフレーム判定部11で算出された判定フラグを用いる方法がある。この方法では、判定フラグが1となるフレーム画像のうち、注目フレーム画像に最も近い前後それぞれのフレームが選択される。例えば、図3に示す変動領域を含む時刻tのフレーム画像に対して、case1の場合は、時刻(t-1)と時刻(t+1)のフレーム画像、case2の場合は、時刻(t-2)と時刻(t+1)のフレーム画像、case3の場合は、時刻(t-1)と時刻(t+2)のフレーム画像、case4の場合は、時刻(t-2)と時刻(t+2)のフレーム画像が選択される。
As a method of selecting a frame image, there is a method of selecting a frame image that does not include a fluctuating region due to blinking of light before and after the closest to the frame image of interest. As a specific selection method, there is a method using a determination flag calculated by the flash frame determination unit 11. In this method, out of the frame images whose determination flag is 1, the respective frames before and after the closest to the target frame image are selected. For example, with respect to the frame image at time t including the fluctuation region shown in FIG. 3, in case 1, the frame image at time (t-1) and time (t + 1), and in case 2, the time (t-2) Frame image at time (t + 1) and time (t + 1), and in case 3, frame image at time (t-1) and time (t + 2), and in case 4, frame image at time (t-2) and time (t + 2) are selected. Is done.
また、利用フレーム画像の選択は、入力された補助情報である注目フレーム画像以外の入力フレーム画像間同士での判定フラグを用いて修正してもよい。修正は、例えば、注目フレーム画像と他フレーム画像との判定フラグを用いた選択で利用フレームとして、時刻(t+k)のフレーム画像が選択された場合を想定する。注目フレーム画像(時刻tの入力フレーム画像)と時刻(t+k+1)のフレーム画像の判定フラグflagt-t+kおよび時刻(t+k+1)のフレーム画像と時刻(t+k)のフレーム画像の判定フラグflagt+k+1-t+kの値が共に1の場合がある。このとき、時刻(t+k+1)のフレーム画像と時刻(t+k)のフレーム画像との間にも大きな輝度変化があると考えられるため、選択フレーム画像が時刻(t+k+1)に変更される。
In addition, the selection of the use frame image may be corrected by using a determination flag between input frame images other than the target frame image which is input auxiliary information. The correction is assumed, for example, when a frame image at time (t + k) is selected as a use frame in selection using a determination flag between a frame image of interest and another frame image. Determination flag flag t−t + k of the frame image of interest (input frame image at time t) and frame image at time (t + k + 1) and determination flag flag t + k + 1−t + k of frame image at time (t + k + 1) and frame image at time (t + k) Both values may be 1. At this time, since it is considered that there is a large luminance change between the frame image at time (t + k + 1) and the frame image at time (t + k), the selected frame image is changed to time (t + k + 1).
選択された前後時刻のフレーム画像から補間フレーム画像を生成する方式として次の方式がある。選択されたフレーム画像が時刻(t+e)と時刻(t-f)の場合、時刻tにおける補間フレーム画像Ies-tを生成する方法の一つに、時刻(t+e)の入力フレーム画像It+eと時刻(t-f)の入力フレーム画像It-fを用いて、式(6)のように算出する。
また、補間フレーム画像を生成する別の方式として、入力フレーム画像It+eとIt-fの各画素の動きベクトルを推定し、動き補償補間により、補間フレーム画像Ies-tを生成する方法を用いてもよい。 As a method for generating an interpolated frame image from selected frame images at previous and subsequent times, there is the following method. When the selected frame image is time (t + e) and time (t−f), one of the methods for generating the interpolated frame image I es−t at time t is the input frame image I t + e at time (t + e) and using the input frame image I t-f of the time (t-f), is calculated by the equation (6).
As another method of generating an interpolation frame image, estimates a motion vector of each pixel of the input frame image I t + e and I t-f, the motion compensated interpolation, the method of generating an interpolation frame image I es-t It may be used.
また、補間フレーム画像を生成する別の方式として、入力フレーム画像It+eとIt-fの各画素の動きベクトルを推定し、動き補償補間により、補間フレーム画像Ies-tを生成する方法を用いてもよい。 As a method for generating an interpolated frame image from selected frame images at previous and subsequent times, there is the following method. When the selected frame image is time (t + e) and time (t−f), one of the methods for generating the interpolated frame image I es−t at time t is the input frame image I t + e at time (t + e) and using the input frame image I t-f of the time (t-f), is calculated by the equation (6).
As another method of generating an interpolation frame image, estimates a motion vector of each pixel of the input frame image I t + e and I t-f, the motion compensated interpolation, the method of generating an interpolation frame image I es-t It may be used.
また、フレーム補間部12は、入力フレーム画像群が時刻(t-m)から時刻(t+m)の(2m+1)フレームである場合、選択されたフレーム画像の時間的距離がmを超える場合、補正が不安定になるため、補間フレーム画像の生成処理を打ち切る機構を設けてもいい。図4は、補正が不安定になる理由を説明するための模式図である。模式図では、時刻(t-2)から時刻(t+2)の5フレームを入力フレーム画像群とする場合で、映像シーケンスの中に3フレーム連続する変動領域を含むフレーム画像が含まれている場合を想定している。入力フレーム画像群が図4のcase2の場合、明滅を検知し補正することが可能であるが、case1およびcase3の場合には、明滅を検知することはできないため、補正処理を実行することができない。そのため、補正の結果、3フレームで1度の明滅が、1フレームの2度の明滅になり、フラッシュ低減の効果を得ることはできない。このような状況が発生するのは、入力フレーム画像群が時刻(t-m)から時刻(t+m)の(2m+1)フレームであるとき、選択されたフレーム画像の時間的距離がmを超える場合であり、このような場合には、効果を得ることはできないため処理を打ち切ってもよい。その際には、フレーム補間部12の出力として、補間フレーム画像生成を打ち切ったとの情報を出力する。
In addition, when the input frame image group is a (2m + 1) frame from time (t−m) to time (t + m), the frame interpolation unit 12 performs correction when the temporal distance of the selected frame image exceeds m. Since it becomes unstable, a mechanism for canceling the generation processing of the interpolated frame image may be provided. FIG. 4 is a schematic diagram for explaining the reason why the correction becomes unstable. In the schematic diagram, a case where five frames from time (t−2) to time (t + 2) are set as an input frame image group, and a case where a frame image including a fluctuating region continuous for three frames is included in the video sequence. Assumed. When the input frame image group is case2 in FIG. 4, blinking can be detected and corrected. However, in case1 and case3, blinking cannot be detected, and thus correction processing cannot be executed. . Therefore, as a result of the correction, blinking once in three frames turns into blinking twice in one frame, and an effect of reducing flash cannot be obtained. Such a situation occurs when the input frame image group is a (2m + 1) frame from time (t−m) to time (t + m) and the temporal distance of the selected frame image exceeds m. Yes, in such a case, the effect may not be obtained and the processing may be aborted. At that time, information indicating that the interpolation frame image generation has been terminated is output as the output of the frame interpolation unit 12.
フレーム画像合成部13は、注目フレーム画像(時刻tの入力フレーム画像)における輝度変動の判定結果に基づき、注目フレーム画像と補間フレーム画像とを合成し、フラッシュ等による明滅を抑制した出力フレーム画像を生成し出力する。判定結果は、時刻tの注目フレーム画像における判定フラグが使われる。すなわち、フレーム画像合成部13は、判定フラグが0の場合は、入力された注目フレーム画像(時刻tの入力フレーム画像)を出力フレーム画像として出力し、判定フラグが1の場合は、注目フレーム画像と補間フレーム画像を合成して出力フレーム画像として出力する。
The frame image synthesis unit 13 synthesizes the target frame image and the interpolated frame image based on the determination result of the luminance variation in the target frame image (input frame image at time t), and outputs an output frame image in which blinking due to flash or the like is suppressed. Generate and output. As the determination result, the determination flag in the target frame image at time t is used. That is, when the determination flag is 0, the frame image synthesis unit 13 outputs the input frame image of interest (the input frame image at time t) as an output frame image. When the determination flag is 1, the frame image synthesis unit 13 And the interpolated frame image are combined and output as an output frame image.
フレーム画像合成部13は、注目フレーム画像が光の明滅による変動領域が含まれた画像であると判別され、かつ、補間フレーム画像が生成された場合に、各画素における合成比率を算出し、合成処理により出力画像を生成する。その他の場合には、フレーム画像合成部13は、入力された注目フレーム画像を出力画像とする。画素位置(x,y)の注目画素における合成比率u(x,y)、時刻tにおける補間フレーム画像の値It-es(x,y)が与えられたとき、同位置における出力画像の値It-out(x,y)は、式(7)のように算出される。
合成比率の算出方法として、注目フレーム画像と補間フレーム画像との間の局所領域輝度の変化率を用いる方法がある。注目フレーム画像と補間フレーム画像との間の局所領域輝度の変化率rt-esは、フラッシュフレーム判定部11の局所領域輝度の変化率と同様に方式を用いて算出することができる。画素位置(x,y)の注目画素における合成比率u(x,y)は、同位置における局所領域輝度の変化率rt-es(x,y)を、あらかじめ設定されたrt-es(x,y)の値に対応した出力フレーム画像における局所領域輝度の変化率の値rtar(x,y)を用いる。これにより、出力フレーム画像における局所領域輝度の変化率がrtar(x,y)になるよう、式(8)のように算出することができる。
出力フレーム画像における局所領域輝度の変化率の値rtarの設定方法の一例として、図5のグラフのように、ある程度小さい値のrt-esに対してはrtar=rt-esとし、大きな値のrt-esに対してはrtarを設定された最大値にし、その値より大きくならないようにする方法がある。 The frameimage composition unit 13 calculates a composition ratio in each pixel when it is determined that the frame image of interest is an image including a fluctuation region due to blinking of light and an interpolated frame image is generated. An output image is generated by processing. In other cases, the frame image synthesis unit 13 sets the input frame image of interest as an output image. Given the composition ratio u (x, y) at the pixel of interest at pixel position (x, y) and the value of the interpolated frame image I t-es (x, y) at time t, the value of the output image at that position I t-out (x, y) is calculated as shown in Equation (7).
As a method for calculating the composition ratio, there is a method using a change rate of local area luminance between a target frame image and an interpolated frame image. The local area luminance change rate rt -es between the target frame image and the interpolated frame image can be calculated using the same method as the local area luminance change rate of the flashframe determination unit 11. The composition ratio u (x, y) at the pixel of interest at the pixel position (x, y) is determined by changing the local area luminance change rate r t-es (x, y) at the same position to a preset r t-es ( The local area luminance change rate value r tar (x, y) in the output frame image corresponding to the value of x, y) is used. As a result, the local area luminance change rate in the output frame image can be calculated as in equation (8) so that r tar (x, y).
As an example of a method of setting the local area luminance change rate value r tar in the output frame image, as shown in the graph of FIG. 5, r tar = r t-es is set for a small value r t-es , For a large value of rt -es, there is a method in which r tar is set to a set maximum value and not larger than that value.
合成比率の算出方法として、注目フレーム画像と補間フレーム画像との間の局所領域輝度の変化率を用いる方法がある。注目フレーム画像と補間フレーム画像との間の局所領域輝度の変化率rt-esは、フラッシュフレーム判定部11の局所領域輝度の変化率と同様に方式を用いて算出することができる。画素位置(x,y)の注目画素における合成比率u(x,y)は、同位置における局所領域輝度の変化率rt-es(x,y)を、あらかじめ設定されたrt-es(x,y)の値に対応した出力フレーム画像における局所領域輝度の変化率の値rtar(x,y)を用いる。これにより、出力フレーム画像における局所領域輝度の変化率がrtar(x,y)になるよう、式(8)のように算出することができる。
出力フレーム画像における局所領域輝度の変化率の値rtarの設定方法の一例として、図5のグラフのように、ある程度小さい値のrt-esに対してはrtar=rt-esとし、大きな値のrt-esに対してはrtarを設定された最大値にし、その値より大きくならないようにする方法がある。 The frame
As a method for calculating the composition ratio, there is a method using a change rate of local area luminance between a target frame image and an interpolated frame image. The local area luminance change rate rt -es between the target frame image and the interpolated frame image can be calculated using the same method as the local area luminance change rate of the flash
As an example of a method of setting the local area luminance change rate value r tar in the output frame image, as shown in the graph of FIG. 5, r tar = r t-es is set for a small value r t-es , For a large value of rt -es, there is a method in which r tar is set to a set maximum value and not larger than that value.
合成比率の算出方法の別の方法として、矩形領域輝度の変化率を用いる方法もある。具体的には、フラッシュフレーム判定部11と同様の方式を用いて算出された矩形領域輝度の変化率Rt-esと予め設定されたRt-esの値に対応する出力画像の矩形領域輝度の変化率から矩形領域毎の合成比率Uを算出する。算出された矩形領域毎の合成比率βから線形補間や双三次補間を用いて、画素毎の合成比率uを求めることができる。
As another method of calculating the composition ratio, there is also a method using the change rate of the luminance of the rectangular area. Specifically, the rectangular area luminance of the output image corresponding to the change rate R t-es of the rectangular area luminance calculated using the same method as that of the flash frame determination unit 11 and a preset value of R t-es. The composite ratio U for each rectangular area is calculated from the rate of change of. The synthesis ratio u for each pixel can be obtained from the calculated synthesis ratio β for each rectangular area using linear interpolation or bicubic interpolation.
次に、図1および図6のフローチャートを参照して、本実施の形態の動作を説明する。
Next, the operation of the present embodiment will be described with reference to the flowcharts of FIGS.
フラッシュフレーム判定部11は、時刻tの注目フレーム画像が、光過敏性発作を誘発する可能性のある、フラッシュ等による光の明滅による変動領域を含むフレーム画像であるかを判定する(S11)。
The flash frame determination unit 11 determines whether or not the frame image of interest at time t is a frame image that includes a fluctuating region due to flashing of light by flash or the like that may induce a photosensitivity attack (S11).
フレーム補間部12は、注目フレーム画像(時刻tの入力フレーム画像)における輝度変動の判定結果である解析情報に基づき、注目フレーム以外の入力フレーム画像群をもとに、注目フレーム画像に最も近接する前後それぞれの光の明滅による変動領域を含まないフレーム画像を選択する。すなわち、フレーム補間部12は、フラッシュフレーム判定部11からの解析情報を受付け、解析情報の判定フラグが1となる入力フレーム画像のうち、注目フレーム画像に最も近い前後それぞれの入力フレーム画像を選択する。更に、フレーム補間部12は、選択された前後時刻のフレーム画像を光の明滅による変動領域がない注目フレーム画像として、補間フレーム画像を生成する(S12)。
The frame interpolation unit 12 is closest to the target frame image based on the input frame image group other than the target frame based on the analysis information that is the determination result of the luminance variation in the target frame image (the input frame image at time t). A frame image that does not include a fluctuation region due to blinking of the front and rear lights is selected. That is, the frame interpolation unit 12 receives the analysis information from the flash frame determination unit 11, and selects the input frame images before and after the closest to the target frame image from among the input frame images whose analysis information determination flag is 1. . Further, the frame interpolating unit 12 generates an interpolated frame image by using the selected frame images at the previous and subsequent times as a target frame image having no fluctuation region due to blinking of light (S12).
フレーム画像合成部13は、フラッシュフレーム判定部11から解析情報を含む判定結果を受付ける。続いて、フレーム画像合成部13は、注目フレーム画像(時刻tの入力フレーム画像)における輝度変動の判定結果に基づき、時刻tの入力フレーム画像である注目フレーム画像と補間フレーム画像とを合成し、フラッシュ等による明滅を抑制した出力画像を生成し出力する(S13)。すなわち、判定フラグが0の場合は、入力された注目フレーム画像(時刻tの入力フレーム画像)を出力フレーム画像として出力し、判定フラグが1の場合は、注目フレーム画像と補間フレーム画像を合成して出力フレーム画像として出力する。
The frame image composition unit 13 receives a determination result including analysis information from the flash frame determination unit 11. Subsequently, the frame image synthesis unit 13 synthesizes the attention frame image that is the input frame image at time t and the interpolation frame image based on the determination result of the luminance variation in the attention frame image (input frame image at time t), An output image in which blinking due to flash or the like is suppressed is generated and output (S13). That is, when the determination flag is 0, the input frame image of interest (the input frame image at time t) is output as an output frame image. When the determination flag is 1, the frame of interest image and the interpolation frame image are combined. To output as an output frame image.
第1の実施の形態の効果は、光過敏性発作を誘起する可能性のある、大きな輝度変化を含む映像に対して、輝度の変動が抑制された映像を生成することができる。その理由は、注目フレーム画像について、他のフレーム画像から推定される輝度変化のないフレーム画像を、画素毎に合成重みを変化させながら合成することで、主として大きな輝度変化のある領域を補正し、かつ白とび等で失われた情報を復元できるからである。
The effect of the first embodiment can generate an image in which fluctuations in luminance are suppressed with respect to an image including a large luminance change that may induce a photosensitivity seizure. The reason for this is that, for the frame image of interest, a frame image with no luminance change estimated from other frame images is synthesized while changing the synthesis weight for each pixel, thereby correcting mainly areas with large luminance changes, This is because information lost due to overexposure can be restored.
尚、上述した実施の形態では、変動領域が注目フレーム画像において他フレーム画像より大きく明るくなる(輝度が大きくなる)明領域の例を説明したが、これに限られるものではない。例えば、上述した実施の形態では、変動領域が注目フレーム画像において他フレーム画像より大きく暗くなる(輝度が小さく)暗領域である場合も、同様に適用することができる。
In the above-described embodiment, an example of a bright area in which the fluctuation area is larger and brighter than other frame images in the attention frame image has been described. However, the present invention is not limited to this. For example, in the above-described embodiment, the present invention can be similarly applied to the case where the fluctuation region is a dark region that is darker (the luminance is smaller) in the frame image of interest than the other frame images.
更に、上述した実施の形態では、輝度の変動、すなわち、一般閃光についての例を説明したが、赤色閃光等の彩度の変動についても同様に適用することができる。
Furthermore, in the above-described embodiment, an example of luminance variation, that is, a general flash has been described, but the present invention can be similarly applied to saturation variation such as red flash.
(第2の実施の形態)
図7は、本発明による第2の実施の形態における画像処理装置のブロック図である。 (Second Embodiment)
FIG. 7 is a block diagram of an image processing apparatus according to the second embodiment of the present invention.
図7は、本発明による第2の実施の形態における画像処理装置のブロック図である。 (Second Embodiment)
FIG. 7 is a block diagram of an image processing apparatus according to the second embodiment of the present invention.
第2の実施の形態の画像処理装置は、フラッシュフレーム判定部21とフレーム補間部22とフレーム画像合成部23とを有する。本発明による第1の実施の形態の画像処理装置と同様に、あらかじめ設定されたフレーム数mで構成される時刻(t-m)から時刻(t+m)までのフレーム画像群を入力とし、明滅の影響を低減させた出力フレーム画像を生成する。
The image processing apparatus according to the second embodiment includes a flash frame determination unit 21, a frame interpolation unit 22, and a frame image synthesis unit 23. As in the image processing apparatus according to the first embodiment of the present invention, a group of frame images from time (t−m) to time (t + m) composed of a preset number of frames m is input and blinking is performed. An output frame image with reduced influence is generated.
第1の実施の形態の画像処理装置では、フレーム画像合成部13において合成比率を算出する際に、注目フレーム画像と補間フレーム画像との間の局所領域輝度の変動率や矩形領域輝度の変動比率を算出する必要があり、比較的処理量が大きくなる。本発明による第2の実施の形態の画像処理装置は、入力フレーム画像間での局所領域輝度の変動率や矩形領域輝度の変動比率をもとに、注目フレーム画像と補間フレーム画像との間の局所領域輝度の変動率や矩形領域輝度の変動比率を補間することで、これらを算出するための計算量を抑制する。
In the image processing apparatus according to the first embodiment, when the frame image composition unit 13 calculates the composition ratio, the local area luminance variation ratio and the rectangular area luminance variation ratio between the target frame image and the interpolation frame image are calculated. Need to be calculated, and the processing amount becomes relatively large. The image processing apparatus according to the second embodiment of the present invention is based on a local area luminance variation rate or a rectangular region luminance variation ratio between input frame images. By interpolating the variation rate of the local region luminance and the variation rate of the rectangular region luminance, the amount of calculation for calculating these is suppressed.
フラッシュフレーム判定部21は、第1の実施の形態におけるフラッシュフレーム判定部11と同様に、入力フレーム画像群を入力とし、時刻tの注目フレーム画像が視聴者に危害を及ぼす明滅による変動領域を含むフレーム画像であるかの判定を行う。フラッシュフレーム判定部21は、第1の実施の形態におけるフラッシュフレーム判定部11と比較して、局所領域輝度変動比率や矩形領域輝度変動比率の算出方式と、算出された局所領域輝度変動比率や矩形領域輝度変動比率を出力する点のみで異なる。
Similarly to the flash frame determination unit 11 in the first embodiment, the flash frame determination unit 21 includes an input frame image group as an input, and includes a fluctuation region due to blinking that causes the frame image of interest at time t to harm the viewer. It is determined whether it is a frame image. Compared with the flash frame determination unit 11 in the first embodiment, the flash frame determination unit 21 calculates the local region luminance variation ratio and the rectangular region luminance variation ratio, and calculates the calculated local region luminance variation ratio and the rectangle. The only difference is that the region luminance fluctuation ratio is output.
フラッシュフレーム判定部21は、フラッシュフレーム判定に局所領域輝度変動比率を用いる場合、画素位置(x,y)における時刻tにおける注目フレーム画像と入力フレーム画像群内の時刻(t+k)のフレーム画像の局所領域輝度変動比率r’t-t+k(x,y)を式(9)のように算出する。式(9)の算出式は、式(4)の算出式と比較して、分母が注目フレーム画像の局所領域輝度を用いている点で異なる。
同様に、フラッシュフレーム判定に矩形領域輝度変動比率を用いる場合には、ブロック位置(i,j)における時刻tにおける注目フレームの矩形領域輝度と入力フレーム群内の時刻(t+k)のフレームの矩形領域輝度の変動比率R’t-t+k(i,j)は、式(10)のように算出される。
第1の実施の形態におけるフラッシュフレーム判定部11では、輝度変動率の閾値としてαを設定している。第2の実施の形態で、同一の判定を実施するためには、フラッシュフレーム判定部21における輝度変動率の閾値α’を式(11)のように設定すればよい。
フレーム補間部22は、第1の実施の形態のフレーム補間部12と同様の機能を有する。フレーム補間部22は、光の明滅による変動領域がない注目フレーム画像を生成すると同時に、入力された領域輝度変動率をもとに、注目フレーム画像と生成された補間フレーム画像との間の領域輝度の変動率を生成し、補間領域輝度変動率として出力する。フレーム補間部22は、フラッシュフレーム判定に局所領域輝度変動率が用いられた場合、注目フレーム画像と生成された補間フレーム画像との間の局所領域輝度変動率rt-esを算出する。また、フレーム補間部22は、フラッシュフレーム判定に矩形領域輝度変動率が用いられた場合、注目フレーム画像と生成された補間フレーム画像との間の矩形領域輝度変動率Rt-esを算出する。 When the local region luminance fluctuation ratio is used for the flash frame determination, the flashframe determination unit 21 localizes the frame image of interest at the time t at the pixel position (x, y) and the frame image at the time (t + k) in the input frame image group. The area luminance fluctuation ratio r ′ t−t + k (x, y) is calculated as in Expression (9). The calculation formula of Formula (9) differs from the calculation formula of Formula (4) in that the denominator uses the local region luminance of the target frame image.
Similarly, when the rectangular area luminance variation ratio is used for flash frame determination, the rectangular area luminance of the target frame at time t at the block position (i, j) and the rectangular area of the frame at time (t + k) in the input frame group. The luminance variation ratio R ′ t−t + k (i, j) is calculated as in Expression (10).
In the flashframe determination unit 11 in the first embodiment, α is set as the threshold value of the luminance variation rate. In the second embodiment, in order to perform the same determination, the threshold value α ′ of the luminance variation rate in the flash frame determination unit 21 may be set as shown in Expression (11).
Theframe interpolation unit 22 has the same function as the frame interpolation unit 12 of the first embodiment. The frame interpolation unit 22 generates an attention frame image that does not have a fluctuation region due to blinking of light, and at the same time, based on the input region luminance fluctuation rate, a region luminance between the attention frame image and the generated interpolation frame image Is generated and output as the interpolated area luminance fluctuation rate. When the local area luminance fluctuation rate is used for the flash frame determination, the frame interpolation unit 22 calculates the local area luminance fluctuation rate rt -es between the target frame image and the generated interpolated frame image. In addition, when the rectangular area luminance fluctuation rate is used for the flash frame determination, the frame interpolation unit 22 calculates the rectangular area luminance fluctuation rate R t-es between the frame image of interest and the generated interpolated frame image.
同様に、フラッシュフレーム判定に矩形領域輝度変動比率を用いる場合には、ブロック位置(i,j)における時刻tにおける注目フレームの矩形領域輝度と入力フレーム群内の時刻(t+k)のフレームの矩形領域輝度の変動比率R’t-t+k(i,j)は、式(10)のように算出される。
第1の実施の形態におけるフラッシュフレーム判定部11では、輝度変動率の閾値としてαを設定している。第2の実施の形態で、同一の判定を実施するためには、フラッシュフレーム判定部21における輝度変動率の閾値α’を式(11)のように設定すればよい。
フレーム補間部22は、第1の実施の形態のフレーム補間部12と同様の機能を有する。フレーム補間部22は、光の明滅による変動領域がない注目フレーム画像を生成すると同時に、入力された領域輝度変動率をもとに、注目フレーム画像と生成された補間フレーム画像との間の領域輝度の変動率を生成し、補間領域輝度変動率として出力する。フレーム補間部22は、フラッシュフレーム判定に局所領域輝度変動率が用いられた場合、注目フレーム画像と生成された補間フレーム画像との間の局所領域輝度変動率rt-esを算出する。また、フレーム補間部22は、フラッシュフレーム判定に矩形領域輝度変動率が用いられた場合、注目フレーム画像と生成された補間フレーム画像との間の矩形領域輝度変動率Rt-esを算出する。 When the local region luminance fluctuation ratio is used for the flash frame determination, the flash
Similarly, when the rectangular area luminance variation ratio is used for flash frame determination, the rectangular area luminance of the target frame at time t at the block position (i, j) and the rectangular area of the frame at time (t + k) in the input frame group. The luminance variation ratio R ′ t−t + k (i, j) is calculated as in Expression (10).
In the flash
The
フレーム補間部22は、フラッシュフレーム判定部21から解析情報を受付ける。フレーム補間部22は、フレーム補間部12と同様に、注目フレーム画像(時刻tの入力フレーム画像)における輝度変動の判定結果である解析情報に基づき、注目フレーム画像に最も近接する前後それぞれの光の明滅による変動領域を含まない注目フレーム画像として、補間フレーム画像を生成する。すなわち、フレーム補間部22は、解析情報の判定フラグが1となる入力フレーム画像のうち、注目フレーム画像に最も近い前後それぞれの入力フレーム画像を選択する。更に、フレーム補間部22は、選択された前後時刻のフレーム画像を光の明滅による変動領域がない注目フレーム画像として、補間フレーム画像を生成する。さらに、フレーム補間部22は、補間フレーム画像と同様の処理にて、入力された領域輝度変動率を合成し、合成領域輝度変動率を算出する。例えば、フラッシュフレーム判定に局所領域輝度変動率が用いられており、補間フレーム画像算出に式(6)が用いられている場合、画素位置(x,y)における合成局所領域輝度変動率r’t-esは、式(12)のように算出される。また、フラッシュフレーム判定に局所領域輝度変動率が用いられている場合には、ブロック位置(i,j)における合成矩形領域輝度変動率R’t-esは、式(13)のように算出される。そして、フレーム補間部22は、式(14)または式(15)の算出式によって、出力されるべき補間領域輝度変動率を算出する。
フレーム画像合成部23は、フラッシュフレーム判定部21から判定結果を受付ける。フレーム画像合成部23は、第1の実施の形態のフレーム画像合成部13と同様に、注目フレーム画像の判定結果に含まれる解析情報に基づき、注目フレーム画像と補間フレーム画像とを合成し、フラッシュ等による明滅を抑制した出力画像を生成し出力する。フレーム画像合成部23は、第1の実施の形態のフレーム画像合成部13と比較して、注目フレーム画像と補間フレーム画像との合成重み算出に、フレーム補間部22で算出された補間領域輝度変動率を用いる点で異なる。 Theframe interpolation unit 22 receives analysis information from the flash frame determination unit 21. Similar to the frame interpolation unit 12, the frame interpolation unit 22, based on the analysis information that is the determination result of the luminance variation in the target frame image (input frame image at time t), the light before and after the closest to the target frame image. An interpolated frame image is generated as a focused frame image that does not include a fluctuation region due to blinking. That is, the frame interpolation unit 22 selects the input frame images before and after the closest to the target frame image from among the input frame images whose analysis information determination flag is 1. Further, the frame interpolation unit 22 generates an interpolated frame image by using the selected frame images at the previous and subsequent times as a target frame image having no fluctuation region due to the blinking of light. Further, the frame interpolation unit 22 synthesizes the input region luminance fluctuation rate by the same processing as that of the interpolation frame image, and calculates the composite region luminance fluctuation rate. For example, when the local area luminance fluctuation rate is used for flash frame determination and the equation (6) is used for interpolation frame image calculation, the combined local area luminance fluctuation rate r ′ t at the pixel position (x, y). -Es is calculated as shown in Equation (12). When the local area luminance fluctuation rate is used for the flash frame determination, the combined rectangular area luminance fluctuation rate R ′ tes at the block position (i, j) is calculated as shown in Expression (13). The Then, the frame interpolation unit 22 calculates the interpolation area luminance fluctuation rate to be output by the calculation formula of Formula (14) or Formula (15).
The frameimage composition unit 23 receives the determination result from the flash frame determination unit 21. Similar to the frame image synthesis unit 13 of the first embodiment, the frame image synthesis unit 23 synthesizes the target frame image and the interpolated frame image based on the analysis information included in the determination result of the target frame image, and performs flash Generate and output an output image that suppresses blinking due to the above. Compared with the frame image composition unit 13 of the first embodiment, the frame image composition unit 23 calculates the interpolated area luminance fluctuation calculated by the frame interpolation unit 22 for calculating the composition weight of the target frame image and the interpolated frame image. It is different in using rate.
フレーム画像合成部23は、フラッシュフレーム判定部21から判定結果を受付ける。フレーム画像合成部23は、第1の実施の形態のフレーム画像合成部13と同様に、注目フレーム画像の判定結果に含まれる解析情報に基づき、注目フレーム画像と補間フレーム画像とを合成し、フラッシュ等による明滅を抑制した出力画像を生成し出力する。フレーム画像合成部23は、第1の実施の形態のフレーム画像合成部13と比較して、注目フレーム画像と補間フレーム画像との合成重み算出に、フレーム補間部22で算出された補間領域輝度変動率を用いる点で異なる。 The
The frame
次に、図7および図8のフローチャートを参照して、本実施の形態の動作を説明する。
Next, the operation of the present embodiment will be described with reference to the flowcharts of FIGS.
フラッシュフレーム判定部21は、時刻tの注目フレーム画像が、光過敏性発作を誘発する可能性のある、フラッシュ等による光の明滅による変動領域を含むフレーム画像であるかを、領域輝度変動率に基づいて判定する(S21)。
The flash frame determination unit 21 determines whether or not the frame image of interest at time t is a frame image that includes a variation region due to blinking of light due to flash or the like that may induce a photosensitivity seizure. Based on the determination (S21).
フレーム補間部22は、注目フレーム以外の入力フレーム画像群をもとに、光の明滅による変動領域がない注目フレーム画像である補間フレーム画像を生成する(S22)。
The frame interpolation unit 22 generates an interpolated frame image that is an attention frame image that does not have a fluctuation region due to blinking of light based on the input frame image group other than the attention frame (S22).
フレーム補間部22は、注目フレーム画像と補間フレーム画像との間の領域輝度変動率を算出する(S23)。
The frame interpolation unit 22 calculates a region luminance fluctuation rate between the target frame image and the interpolation frame image (S23).
フレーム画像合成部23は、注目フレーム画像と補間フレーム画像とを合成し、フラッシュ等による明滅を抑制した出力画像を生成し出力する(S24)。
The frame image synthesis unit 23 synthesizes the target frame image and the interpolation frame image, and generates and outputs an output image in which blinking due to flash or the like is suppressed (S24).
(第3の実施の形態)
図9は、本発明の第3の実施の形態における画像処理装置のブロック図である。 (Third embodiment)
FIG. 9 is a block diagram of an image processing apparatus according to the third embodiment of the present invention.
図9は、本発明の第3の実施の形態における画像処理装置のブロック図である。 (Third embodiment)
FIG. 9 is a block diagram of an image processing apparatus according to the third embodiment of the present invention.
第3の実施の形態の画像処理装置は、取得部30と補正部31とを有する。
The image processing apparatus according to the third embodiment includes an acquisition unit 30 and a correction unit 31.
取得部30は、映像の時間方向の輝度の変化率と、フレーム画像の全領域における所定の割合とを取得する。ここで、時間方向の輝度の変化率は、例えば、上述した輝度変動率の閾値αである。また、フレーム画像の全領域における所定の割合は、例えば、上述した面積率βであり、フレーム画像全体に占める光の明滅が発生する面積の割合である。更に、具体的には、日本放送協会と日本民間放送連盟によるガイドラインの基準に基づけば、α=0.1、β=0.25である。取得部30は、これらの値を外部から取得し、保持しておく。
The acquisition unit 30 acquires the luminance change rate in the time direction of the video and a predetermined ratio in the entire area of the frame image. Here, the rate of change in luminance in the time direction is, for example, the threshold value α of the luminance variation rate described above. Further, the predetermined ratio in the entire region of the frame image is, for example, the above-described area ratio β, which is the ratio of the area where light blinks in the entire frame image. More specifically, α = 0.1 and β = 0.25 based on the standard of the guidelines by the Japan Broadcasting Corporation and the Japan Broadcasting Corporation. The acquisition unit 30 acquires these values from the outside and holds them.
補正部31は、上述した割合で変化率を上回る輝度の変化がある映像を入力し、映像をなす各フレーム間における割合又は輝度の変化率が、取得した輝度の変化率又は変化率を下回るように、映像をなす少なくとも一部のフレームの一部の領域の輝度を補正する。具体的には、割合又は輝度の変化率を超えるフレームの領域を特定し、その領域の輝度を、例えば他のフレームの当該領域と同一の位置の領域にある画素の輝度で置き換えることにより、映像全体として、上述した割合で変化率を上回る輝度の変化がない映像を生成する。なお、領域の輝度は、前後フレームの当該領域と同一の位置の領域にある画素の輝度との平均値又は中間値の輝度に置き換えてもよい。
The correction unit 31 inputs an image having a luminance change that exceeds the change rate at the above-described ratio, and the ratio or the change rate of the luminance between the frames forming the video is less than the acquired change rate or change rate of the luminance. In addition, the luminance of at least a partial region of at least a part of the frame forming the video is corrected. Specifically, by identifying a region of a frame that exceeds the rate or rate of change in luminance, and replacing the luminance of that region with, for example, the luminance of a pixel in the same position as the region of the other frame, As a whole, an image with no change in luminance exceeding the rate of change at the rate described above is generated. Note that the luminance of the region may be replaced with an average value or intermediate luminance of the luminance of the pixels in the region at the same position as the region of the previous and subsequent frames.
光過敏性発作を誘起する可能性のある明滅は、フレーム全体だけではなく、フレームの一部領域に発生する場合がある。関連するこれまでの手法は、明滅の発生個所の判別は行わずフレーム全体について一律に補正している。本実施の形態における画像処理装置は、本来補正する必要のない、明滅の発生していない領域について、必要以上の補正を行うことを抑制することができる。
• Blinks that can induce photosensitivity seizures may occur not only in the entire frame, but also in some areas of the frame. The related methods so far correct the entire frame uniformly without discriminating the blinking location. The image processing apparatus according to the present embodiment can suppress performing correction more than necessary for an area that does not need to be corrected and does not blink.
本発明による実施の形態は、ハードディスク等に記録されている映像を編集する映像編集システムに適用することができる。また、メモリ上で保持されたフレーム画像群を用いることで、ビデオカメラやディスプレイ端末等にも適用することができる。
The embodiment according to the present invention can be applied to a video editing system for editing video recorded on a hard disk or the like. In addition, by using a group of frame images held on a memory, the present invention can be applied to a video camera, a display terminal, or the like.
また、上述した説明からも明らかなように、各部をハードウェアで構成することも可能であるが、コンピュータプログラムにより実現することも可能である。この場合、プログラムメモリに格納されているプログラムで動作するプロセッサによって、上述した各実施の形態と同様の機能、動作を実現させる。また、上述した実施の形態の一部の機能のみをコンピュータプログラムにより実現することも可能である。
図10は、第1~3の実施の形態における画像処理装置をコンピュータ装置で実現したハードウェア構成を示す図である。
図10に示すように、画像処理装置は、CPU(Central Processing Unit)91、ネットワーク接続用の通信I/F(通信インターフェース)92、メモリ93、及び、プログラムを格納するハードディスク等の記憶装置94を含む。また、画像処理装置のCPU91は、システムバス97を介して入力装置95及び、出力装置96に接続されている。
CPU91は、オペレーティングシステムを動作させて第1~3の実施の形態の各機能部を制御する。またCPU91は、例えば、ドライブ装置に装着された記録媒体からメモリ93にプログラムやデータを読み出す。
また、CPU91は、例えば、入力される情報信号を処理する機能を有し、プログラムに基づいて各種機能の処理を実行する。
記憶装置94は、例えば、光ディスク、フレキシブルディスク、磁気光ディスク、外付けハードディスク、又は半導体メモリ等である。記憶装置94の一部の記憶媒体は、不揮発性記憶装置であり、そこにプログラムを記憶する。また、プログラムは、通信網に接続されている。図示しない外部コンピュータからダウンロードされてもよい。
入力装置95は、例えば、マウス、キーボード、内蔵のキーボタン、カード取込口、又は、タッチパネルなどで実現され、入力操作に用いられる。
出力装置96は、例えば、ディスプレイで実現され、CPU91により処理された情報等を出力して確認するために用いられる。
以上のように、第1~3の実施の形態は、図10に示されるハードウェア構成によって実現される。但し、画像処理装置が備える各部の実現手段は、特に限定されない。すなわち、各実現手段は、物理的に結合した一つの装置により実現されてもよいし、物理的に分離した二つ以上の装置を有線又は無線で接続し、これら複数の装置により実現してもよい。 Further, as is apparent from the above description, each unit can be configured by hardware, but can also be realized by a computer program. In this case, functions and operations similar to those of the above-described embodiments are realized by a processor that operates according to a program stored in the program memory. In addition, only some functions of the above-described embodiments can be realized by a computer program.
FIG. 10 is a diagram illustrating a hardware configuration in which the image processing apparatus according to the first to third embodiments is realized by a computer apparatus.
As shown in FIG. 10, the image processing apparatus includes a CPU (Central Processing Unit) 91, a communication I / F (communication interface) 92 for network connection, amemory 93, and a storage device 94 such as a hard disk for storing programs. Including. The CPU 91 of the image processing apparatus is connected to an input device 95 and an output device 96 via a system bus 97.
TheCPU 91 operates the operating system to control each functional unit in the first to third embodiments. Further, the CPU 91 reads out programs and data from the recording medium mounted on the drive device to the memory 93, for example.
TheCPU 91 has a function of processing an input information signal, for example, and executes various functions based on a program.
Thestorage device 94 is, for example, an optical disk, a flexible disk, a magnetic optical disk, an external hard disk, or a semiconductor memory. A part of the storage medium of the storage device 94 is a nonvolatile storage device, and stores a program therein. The program is connected to a communication network. It may be downloaded from an external computer (not shown).
Theinput device 95 is realized by, for example, a mouse, a keyboard, a built-in key button, a card slot, or a touch panel, and is used for an input operation.
Theoutput device 96 is realized by a display, for example, and is used for outputting and confirming information processed by the CPU 91.
As described above, the first to third embodiments are realized by the hardware configuration shown in FIG. However, the means for realizing each unit included in the image processing apparatus is not particularly limited. That is, each realization means may be realized by one physically coupled device, or two or more physically separated devices may be connected by wire or wirelessly and realized by these plural devices. Good.
図10は、第1~3の実施の形態における画像処理装置をコンピュータ装置で実現したハードウェア構成を示す図である。
図10に示すように、画像処理装置は、CPU(Central Processing Unit)91、ネットワーク接続用の通信I/F(通信インターフェース)92、メモリ93、及び、プログラムを格納するハードディスク等の記憶装置94を含む。また、画像処理装置のCPU91は、システムバス97を介して入力装置95及び、出力装置96に接続されている。
CPU91は、オペレーティングシステムを動作させて第1~3の実施の形態の各機能部を制御する。またCPU91は、例えば、ドライブ装置に装着された記録媒体からメモリ93にプログラムやデータを読み出す。
また、CPU91は、例えば、入力される情報信号を処理する機能を有し、プログラムに基づいて各種機能の処理を実行する。
記憶装置94は、例えば、光ディスク、フレキシブルディスク、磁気光ディスク、外付けハードディスク、又は半導体メモリ等である。記憶装置94の一部の記憶媒体は、不揮発性記憶装置であり、そこにプログラムを記憶する。また、プログラムは、通信網に接続されている。図示しない外部コンピュータからダウンロードされてもよい。
入力装置95は、例えば、マウス、キーボード、内蔵のキーボタン、カード取込口、又は、タッチパネルなどで実現され、入力操作に用いられる。
出力装置96は、例えば、ディスプレイで実現され、CPU91により処理された情報等を出力して確認するために用いられる。
以上のように、第1~3の実施の形態は、図10に示されるハードウェア構成によって実現される。但し、画像処理装置が備える各部の実現手段は、特に限定されない。すなわち、各実現手段は、物理的に結合した一つの装置により実現されてもよいし、物理的に分離した二つ以上の装置を有線又は無線で接続し、これら複数の装置により実現してもよい。 Further, as is apparent from the above description, each unit can be configured by hardware, but can also be realized by a computer program. In this case, functions and operations similar to those of the above-described embodiments are realized by a processor that operates according to a program stored in the program memory. In addition, only some functions of the above-described embodiments can be realized by a computer program.
FIG. 10 is a diagram illustrating a hardware configuration in which the image processing apparatus according to the first to third embodiments is realized by a computer apparatus.
As shown in FIG. 10, the image processing apparatus includes a CPU (Central Processing Unit) 91, a communication I / F (communication interface) 92 for network connection, a
The
The
The
The
The
As described above, the first to third embodiments are realized by the hardware configuration shown in FIG. However, the means for realizing each unit included in the image processing apparatus is not particularly limited. That is, each realization means may be realized by one physically coupled device, or two or more physically separated devices may be connected by wire or wirelessly and realized by these plural devices. Good.
尚、上記の実施形態の一部又は全部は、以下の付記のようにも記載されうるが、以下には限られない。
In addition, although a part or all of said embodiment can be described also as the following additional remarks, it is not restricted to the following.
(付記1) 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定する判定部と、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成するフレーム生成部と、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成するフレーム画像合成部と
有する画像処理装置。 (Additional remark 1) The attention frame image in the image | video with which brightness | luminance or saturation changes contradictingly and the frame image before and behind that are compared, The fluctuation | variation rate of the said brightness | luminance or saturation of the said attention frame image exceeded the predetermined threshold value A determination unit that determines whether the frame image includes a variable region;
A frame generation unit that generates an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image;
An image processing apparatus comprising: a frame image synthesis unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成するフレーム生成部と、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成するフレーム画像合成部と
有する画像処理装置。 (Additional remark 1) The attention frame image in the image | video with which brightness | luminance or saturation changes contradictingly and the frame image before and behind that are compared, The fluctuation | variation rate of the said brightness | luminance or saturation of the said attention frame image exceeded the predetermined threshold value A determination unit that determines whether the frame image includes a variable region;
A frame generation unit that generates an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image;
An image processing apparatus comprising: a frame image synthesis unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
(付記2) 前記補間フレーム画像の生成に用いるフレーム画像として、前記変動領域を含まない時間的に近接する前後それぞれの時刻のフレーム画像を選択する
付記1に記載の画像処理装置。 (Additional remark 2) The image processing apparatus ofAdditional remark 1 which selects the frame image of each time before and after which adjoins temporally which does not contain the said fluctuation | variation area | region as a frame image used for the production | generation of the said interpolation frame image.
付記1に記載の画像処理装置。 (Additional remark 2) The image processing apparatus of
(付記3) 前記注目フレーム画像と前記補間フレーム画像との変動比率に応じて、前記注目フレーム画像と前記補間フレーム画像とを合成するための合成比率を変化させる、
付記1又は付記2に記載の画像処理装置。 (Supplementary Note 3) According to a variation ratio between the attention frame image and the interpolation frame image, a composition ratio for combining the attention frame image and the interpolation frame image is changed.
The image processing apparatus according toappendix 1 or appendix 2.
付記1又は付記2に記載の画像処理装置。 (Supplementary Note 3) According to a variation ratio between the attention frame image and the interpolation frame image, a composition ratio for combining the attention frame image and the interpolation frame image is changed.
The image processing apparatus according to
(付記4) 前記変動比率は、画素毎の画素周辺の輝度の重み付き平均値である局所領域輝度の変動比率である、
付記3に記載の画像処理装置。 (Supplementary Note 4) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The image processing apparatus according to appendix 3.
付記3に記載の画像処理装置。 (Supplementary Note 4) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The image processing apparatus according to appendix 3.
(付記5) 前記注目フレーム画像と他のフレーム画像の局所領域輝度の変動比率に基づき、前記補間フレーム画像の生成に用いるフレーム画像の選択する、
付記4に記載の画像処理装置。 (Supplementary Note 5) Based on the fluctuation ratio of the local area luminance between the frame image of interest and the other frame images, a frame image used for generation of the interpolation frame image is selected.
The image processing apparatus according to appendix 4.
付記4に記載の画像処理装置。 (Supplementary Note 5) Based on the fluctuation ratio of the local area luminance between the frame image of interest and the other frame images, a frame image used for generation of the interpolation frame image is selected.
The image processing apparatus according to appendix 4.
(付記6) フレーム画像と補間フレーム画像の局所領域輝度の変動比率を、注目フレーム画像と補間フレーム画像生成のために選択されたフレーム画像との局所領域輝度の変動比率を用いて算出する
付記5に記載の画像処理装置。 (Additional remark 6) The fluctuation ratio of the local area brightness | luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness | luminance of an attention frame image and the frame image selected for interpolation frame image generation. An image processing apparatus according to 1.
付記5に記載の画像処理装置。 (Additional remark 6) The fluctuation ratio of the local area brightness | luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness | luminance of an attention frame image and the frame image selected for interpolation frame image generation. An image processing apparatus according to 1.
(付記7) 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定し、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する
画像処理方法。 (Supplementary Note 7) A frame image of interest in a video whose luminance or saturation changes in a contradictory manner is compared with the frame images before and after the image, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold value. Determine whether it is a frame image that includes a variable region,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
An image processing method for generating an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する
画像処理方法。 (Supplementary Note 7) A frame image of interest in a video whose luminance or saturation changes in a contradictory manner is compared with the frame images before and after the image, and the variation rate of the luminance or saturation of the frame of interest exceeds a predetermined threshold value. Determine whether it is a frame image that includes a variable region,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
An image processing method for generating an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit.
(付記8) 前記補間フレーム画像の生成に用いるフレーム画像として、前記変動領域を含まない時間的に近接する前後それぞれの時刻のフレーム画像を選択する
付記7に記載の画像処理方法。 (Additional remark 8) The image processing method of Additional remark 7 which selects the frame image of each time before and behind which does not contain the said fluctuation | variation area | region as the frame image used for the production | generation of the said interpolation frame image.
付記7に記載の画像処理方法。 (Additional remark 8) The image processing method of Additional remark 7 which selects the frame image of each time before and behind which does not contain the said fluctuation | variation area | region as the frame image used for the production | generation of the said interpolation frame image.
(付記9) 前記注目フレーム画像と前記補間フレーム画像との変動比率に応じて、前記注目フレーム画像と前記補間フレーム画像とを合成するための合成比率を変化させる、
付記7又は付記8に記載の画像処理方法。 (Supplementary note 9) According to a variation ratio between the attention frame image and the interpolation frame image, a composition ratio for combining the attention frame image and the interpolation frame image is changed.
The image processing method according to appendix 7 or appendix 8.
付記7又は付記8に記載の画像処理方法。 (Supplementary note 9) According to a variation ratio between the attention frame image and the interpolation frame image, a composition ratio for combining the attention frame image and the interpolation frame image is changed.
The image processing method according to appendix 7 or appendix 8.
(付記10) 前記変動比率は、画素毎の画素周辺の輝度の重み付き平均値である局所領域輝度の変動比率である、
付記9に記載の画像処理方法。 (Supplementary Note 10) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The image processing method according to appendix 9.
付記9に記載の画像処理方法。 (Supplementary Note 10) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The image processing method according to appendix 9.
(付記11) 前記注目フレーム画像と他のフレーム画像の局所領域輝度の変動比率に基づき、前記補間フレーム画像の生成に用いるフレーム画像の選択する、
付記10に記載の画像処理方法。 (Additional remark 11) Based on the fluctuation ratio of the local area brightness | luminance of the said attention frame image and another frame image, the frame image used for the production | generation of the said interpolation frame image is selected.
The image processing method according to attachment 10.
付記10に記載の画像処理方法。 (Additional remark 11) Based on the fluctuation ratio of the local area brightness | luminance of the said attention frame image and another frame image, the frame image used for the production | generation of the said interpolation frame image is selected.
The image processing method according to attachment 10.
(付記12) フレーム画像と補間フレーム画像の局所領域輝度の変動比率を、注目フレーム画像と補間フレーム画像生成のために選択されたフレーム画像との局所領域輝度の変動比率を用いて算出する
付記11に記載の画像処理方法。 (Additional remark 12) The fluctuation ratio of the local area brightness | luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness | luminance of an attention frame image and the frame image selected for the interpolation frame image generation. An image processing method described in 1.
付記11に記載の画像処理方法。 (Additional remark 12) The fluctuation ratio of the local area brightness | luminance of a frame image and an interpolation frame image is calculated using the fluctuation ratio of the local area brightness | luminance of an attention frame image and the frame image selected for the interpolation frame image generation. An image processing method described in 1.
(付記13) 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定し、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する、
ことをコンピュータに実行させるプログラム。 (Additional remark 13) The attention frame image in the image | video with which brightness | luminance or saturation changes contrary is compared with the frame image before and behind it, and the variation | change_rate of the said brightness | luminance or saturation of the said attention frame image exceeded the predetermined threshold value Determine whether it is a frame image that includes a variable region,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
Based on the determination result of the determination unit, an output frame image is generated by combining the frame image of interest and the interpolation frame image.
A program that causes a computer to execute.
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する、
ことをコンピュータに実行させるプログラム。 (Additional remark 13) The attention frame image in the image | video with which brightness | luminance or saturation changes contrary is compared with the frame image before and behind it, and the variation | change_rate of the said brightness | luminance or saturation of the said attention frame image exceeded the predetermined threshold value Determine whether it is a frame image that includes a variable region,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
Based on the determination result of the determination unit, an output frame image is generated by combining the frame image of interest and the interpolation frame image.
A program that causes a computer to execute.
(付記14) 前記補間フレーム画像の生成に用いるフレーム画像として、前記変動領域を含まない時間的に近接する前後それぞれの時刻のフレーム画像を選択する
付記13に記載のプログラム。 (Additional remark 14) The program ofAdditional remark 13 which selects the frame image of each time before and after which adjoins temporally which does not contain the said fluctuation | variation area | region as a frame image used for the production | generation of the said interpolation frame image.
付記13に記載のプログラム。 (Additional remark 14) The program of
(付記15) 前記注目フレーム画像と前記補間フレーム画像との変動比率に応じて、前記注目フレーム画像と前記補間フレーム画像とを合成するための合成比率を変化させる、
付記13又は付記14に記載のプログラム。 (Supplementary Note 15) A synthesis ratio for synthesizing the target frame image and the interpolation frame image is changed according to a variation ratio between the target frame image and the interpolation frame image.
The program according toSupplementary Note 13 or Supplementary Note 14.
付記13又は付記14に記載のプログラム。 (Supplementary Note 15) A synthesis ratio for synthesizing the target frame image and the interpolation frame image is changed according to a variation ratio between the target frame image and the interpolation frame image.
The program according to
(付記16) 前記変動比率は、画素毎の画素周辺の輝度の重み付き平均値である局所領域輝度の変動比率である、
付記15に記載のプログラム。 (Supplementary Note 16) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The program according to appendix 15.
付記15に記載のプログラム。 (Supplementary Note 16) The fluctuation ratio is a fluctuation ratio of local area luminance that is a weighted average value of luminance around each pixel.
The program according to appendix 15.
(付記17) 前記注目フレーム画像と他のフレーム画像の局所領域輝度の変動比率に基づき、前記補間フレーム画像の生成に用いるフレーム画像の選択する、
付記16に記載のプログラム。 (Additional remark 17) Based on the fluctuation ratio of the local area brightness | luminance of the said attention frame image and another frame image, the frame image used for the production | generation of the said interpolation frame image is selected.
The program according to appendix 16.
付記16に記載のプログラム。 (Additional remark 17) Based on the fluctuation ratio of the local area brightness | luminance of the said attention frame image and another frame image, the frame image used for the production | generation of the said interpolation frame image is selected.
The program according to appendix 16.
(付記18) フレーム画像と補間フレーム画像の局所領域輝度の変動比率を、注目フレーム画像と補間フレーム画像生成のために選択されたフレーム画像との局所領域輝度の変動比率を用いて算出する
付記17に記載のプログラム。 (Supplementary note 18) Supplementary note 17 calculates the fluctuation ratio of the local area luminance between the frame image and the interpolation frame image using the fluctuation ratio of the local area luminance between the frame image of interest and the frame image selected to generate the interpolation frame image. The program described in.
付記17に記載のプログラム。 (Supplementary note 18) Supplementary note 17 calculates the fluctuation ratio of the local area luminance between the frame image and the interpolation frame image using the fluctuation ratio of the local area luminance between the frame image of interest and the frame image selected to generate the interpolation frame image. The program described in.
(付記19) 映像の時間方向の輝度の変化率と、フレーム画像の全領域における所定の割合とを取得する取得部と、
前記割合で前記変化率を上回る輝度の変化がある映像を、前記映像をなす各フレーム間における割合又は輝度の変化率が、前記取得した輝度の変化率又は変化率を下回るように、前記映像をなす少なくとも一部のフレームの一部の領域の輝度を補正する補正部と
を備える画像処理装置。 (Supplementary Note 19) An acquisition unit that acquires a change rate of luminance in a time direction of a video and a predetermined ratio in the entire region of the frame image;
The video having a change in luminance that exceeds the rate of change at the rate, the rate or change rate of the luminance between each frame forming the video is less than the acquired rate of change of luminance or the rate of change. An image processing apparatus comprising: a correction unit that corrects luminance of at least a part of a region of a part of the frame.
前記割合で前記変化率を上回る輝度の変化がある映像を、前記映像をなす各フレーム間における割合又は輝度の変化率が、前記取得した輝度の変化率又は変化率を下回るように、前記映像をなす少なくとも一部のフレームの一部の領域の輝度を補正する補正部と
を備える画像処理装置。 (Supplementary Note 19) An acquisition unit that acquires a change rate of luminance in a time direction of a video and a predetermined ratio in the entire region of the frame image;
The video having a change in luminance that exceeds the rate of change at the rate, the rate or change rate of the luminance between each frame forming the video is less than the acquired rate of change of luminance or the rate of change. An image processing apparatus comprising: a correction unit that corrects luminance of at least a part of a region of a part of the frame.
以上好ましい実施の形態をあげて本発明を説明したが、本発明は必ずしも上記実施の形態に限定されるものではなく、その技術的思想の範囲内において様々に変形し実施することが出来る。
この出願は、2014年1月14日に出願された日本出願特願2014-003880号を基礎とする優先権を主張し、その開示の全てをここに取り込む。 Although the present invention has been described with reference to the preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and various modifications can be made within the scope of the technical idea.
This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2014-003880 for which it applied on January 14, 2014, and takes in those the indications of all here.
この出願は、2014年1月14日に出願された日本出願特願2014-003880号を基礎とする優先権を主張し、その開示の全てをここに取り込む。 Although the present invention has been described with reference to the preferred embodiments, the present invention is not necessarily limited to the above-described embodiments, and various modifications can be made within the scope of the technical idea.
This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2014-003880 for which it applied on January 14, 2014, and takes in those the indications of all here.
11 フラッシュフレーム判定部
12 フレーム補間部
13 フレーム画像合成部
21 フラッシュフレーム判定部
22 フレーム補間部
23 フレーム画像合成部
30 取得部
31 補正部
91 CPU
92 通信I/F(通信インターフェース)
93 メモリ
94 記憶装置
95 入力装置
96 出力装置
97 システムバス DESCRIPTION OFSYMBOLS 11 Flash frame determination part 12 Frame interpolation part 13 Frame image composition part 21 Flash frame determination part 22 Frame interpolation part 23 Frame image composition part 30 Acquisition part 31 Correction part 91 CPU
92 Communication I / F (communication interface)
93Memory 94 Storage Device 95 Input Device 96 Output Device 97 System Bus
12 フレーム補間部
13 フレーム画像合成部
21 フラッシュフレーム判定部
22 フレーム補間部
23 フレーム画像合成部
30 取得部
31 補正部
91 CPU
92 通信I/F(通信インターフェース)
93 メモリ
94 記憶装置
95 入力装置
96 出力装置
97 システムバス DESCRIPTION OF
92 Communication I / F (communication interface)
93
Claims (9)
- 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定する判定手段と、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成するフレーム生成手段と、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成するフレーム画像合成手段と
を有する画像処理装置。 Comparing the frame image of interest in a video in which luminance or saturation changes in contradiction and the frame images before and after the image, the frame of interest image includes a variation region in which the variation rate of the luminance or saturation exceeds a predetermined threshold Determining means for determining whether the image is a frame image;
Frame generating means for generating an interpolated frame image that is an attention frame image without the fluctuation region based on a frame image other than the attention frame image;
An image processing apparatus comprising: a frame image synthesizing unit that generates an output frame image obtained by synthesizing the frame image of interest and the interpolated frame image based on a determination result of the determination unit. - 前記判定手段は、
輝度又は彩度の変動率が所定の閾値を超える変動領域の面積と表示領域の面積との割合が所定の閾値を超えるか否かで前記変動領域を含むフレーム画像であるかを判定する、
請求項1に記載の画像処理装置。 The determination means includes
It is determined whether the frame image includes the variation area by determining whether or not the ratio of the area of the variation area and the area of the display area where the variation rate of luminance or saturation exceeds a predetermined threshold exceeds a predetermined threshold.
The image processing apparatus according to claim 1. - 前記補間フレーム画像の生成に用いるフレーム画像として、前記変動領域を含まない時間的に近接する前後それぞれの時刻のフレーム画像を選択する、
請求項1又は2に記載の画像処理装置。 As frame images used to generate the interpolated frame image, select frame images at respective times before and after being close in time that do not include the fluctuation region.
The image processing apparatus according to claim 1. - 前記注目フレーム画像と前記補間フレーム画像との変動比率に応じて、前記注目フレーム画像と前記補間フレーム画像とを合成するための合成比率を変える、
請求項1又は2に記載の画像処理装置。 According to a variation ratio between the attention frame image and the interpolation frame image, a composition ratio for combining the attention frame image and the interpolation frame image is changed.
The image processing apparatus according to claim 1. - 前記変動比率は、画素毎の画素周辺の輝度の重み付き平均値である局所領域輝度の変動比率である、
請求項4に記載の画像処理装置。 The variation ratio is a variation ratio of local area luminance that is a weighted average value of luminance around the pixel for each pixel.
The image processing apparatus according to claim 4. - 前記注目フレーム画像と他のフレーム画像の局所領域輝度の変動比率に基づき、前記補間フレーム画像の生成に用いるフレーム画像の選択する、
請求項5に記載の画像処理装置。 Selecting a frame image to be used for generating the interpolated frame image based on a variation ratio of local region luminance between the frame image of interest and another frame image;
The image processing apparatus according to claim 5. - フレーム画像と補間フレーム画像の局所領域輝度の変動比率を、注目フレーム画像と補間フレーム画像生成のために選択されたフレーム画像との局所領域輝度の変動比率を用いて算出する、
請求項6に記載の画像処理装置。 Calculating a fluctuation ratio of the local area luminance between the frame image and the interpolated frame image using a fluctuation ratio of the local area luminance between the frame image of interest and the frame image selected for generating the interpolated frame image;
The image processing apparatus according to claim 6. - 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定し、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する、
画像処理方法。 Comparing the frame image of interest in a video in which luminance or saturation changes in contradiction and the frame images before and after the image, the frame of interest image includes a variation region in which the variation rate of the luminance or saturation exceeds a predetermined threshold Determine whether it is a frame image,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
Based on the determination result of the determination unit, an output frame image is generated by combining the frame image of interest and the interpolation frame image.
Image processing method. - 輝度又は彩度が相反して変化する映像における注目フレーム画像とその前後のフレーム画像とを比較し、前記注目フレーム画像が前記輝度又は彩度の変動率が所定の閾値を超えた変動領域を含むフレーム画像であるかを判定し、
前記注目フレーム画像以外のフレーム画像に基づいて、前記変動領域がない注目フレーム画像である補間フレーム画像を生成し、
前記判定手段の判定結果に基づき、前記注目フレーム画像と前記補間フレーム画像とを合成した出力フレーム画像を生成する、
ことをコンピュータに実行させるプログラムを記録した記録媒体。 Comparing the frame image of interest in a video in which luminance or saturation changes in contradiction and the frame images before and after the image, the frame of interest image includes a variation region in which the variation rate of the luminance or saturation exceeds a predetermined threshold Determine whether it is a frame image,
Based on a frame image other than the target frame image, an interpolated frame image that is a target frame image without the fluctuation region is generated,
Based on the determination result of the determination unit, an output frame image is generated by combining the frame image of interest and the interpolation frame image.
The recording medium which recorded the program which makes a computer perform this.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2015557764A JP6582994B2 (en) | 2014-01-14 | 2015-01-09 | Image processing apparatus, image processing method, and program |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014003880 | 2014-01-14 | ||
JP2014-003880 | 2014-01-14 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015107880A1 true WO2015107880A1 (en) | 2015-07-23 |
Family
ID=53542777
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2015/000087 WO2015107880A1 (en) | 2014-01-14 | 2015-01-09 | Image processing device, image processing method, and recording medium containing program |
Country Status (2)
Country | Link |
---|---|
JP (1) | JP6582994B2 (en) |
WO (1) | WO2015107880A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10742923B2 (en) | 2018-10-25 | 2020-08-11 | International Business Machines Corporation | Detection of photosensitive triggers in video content |
CN115174969A (en) * | 2022-07-01 | 2022-10-11 | 抖音视界(北京)有限公司 | Video pushing method and device and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000115778A (en) * | 1998-09-25 | 2000-04-21 | Sarnoff Corp | Detection and coding of flash frame in video data |
JP2004242055A (en) * | 2003-02-06 | 2004-08-26 | Sony Corp | Signal processing apparatus and method, recording medium, and program |
JP2006180470A (en) * | 2004-11-26 | 2006-07-06 | Canon Inc | Image processing apparatus and image processing method |
JP2008124956A (en) * | 2006-11-15 | 2008-05-29 | Nippon Hoso Kyokai <Nhk> | Motion vector correcting device, motion vector correction program, interpolation frame generating device and video correcting device |
-
2015
- 2015-01-09 WO PCT/JP2015/000087 patent/WO2015107880A1/en active Application Filing
- 2015-01-09 JP JP2015557764A patent/JP6582994B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000115778A (en) * | 1998-09-25 | 2000-04-21 | Sarnoff Corp | Detection and coding of flash frame in video data |
JP2004242055A (en) * | 2003-02-06 | 2004-08-26 | Sony Corp | Signal processing apparatus and method, recording medium, and program |
JP2006180470A (en) * | 2004-11-26 | 2006-07-06 | Canon Inc | Image processing apparatus and image processing method |
JP2008124956A (en) * | 2006-11-15 | 2008-05-29 | Nippon Hoso Kyokai <Nhk> | Motion vector correcting device, motion vector correction program, interpolation frame generating device and video correcting device |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10742923B2 (en) | 2018-10-25 | 2020-08-11 | International Business Machines Corporation | Detection of photosensitive triggers in video content |
US11153526B2 (en) | 2018-10-25 | 2021-10-19 | International Business Machines Corporation | Detection of photosensitive triggers in video content |
CN115174969A (en) * | 2022-07-01 | 2022-10-11 | 抖音视界(北京)有限公司 | Video pushing method and device and storage medium |
Also Published As
Publication number | Publication date |
---|---|
JP6582994B2 (en) | 2019-10-02 |
JPWO2015107880A1 (en) | 2017-03-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10397486B2 (en) | Image capture apparatus and method executed by image capture apparatus | |
CN105745914B (en) | Method and system for inverse tone mapping (ITM) | |
JP4783985B2 (en) | Video processing apparatus, video display apparatus, video processing method used therefor, and program thereof | |
US10459318B2 (en) | Image display apparatus method and storage medium for modifying display luminance | |
US9672603B2 (en) | Image processing apparatus, image processing method, display apparatus, and control method for display apparatus for generating and displaying a combined image of a high-dynamic-range image and a low-dynamic-range image | |
US20180033154A1 (en) | Image processing apparatus, imaging apparatus, image processing method, and storage medium storing image processing program | |
JP5089783B2 (en) | Image processing apparatus and control method thereof | |
WO2016157838A1 (en) | Signal processing device, display device, signal processing method, and program | |
US20180025700A1 (en) | Image processing apparatus and display apparatus | |
US9672650B2 (en) | Image processing apparatus, image processing method, and storage medium | |
JPWO2019146226A1 (en) | Image processing device, output information control method, and program | |
JP2018061130A (en) | Image processing device, image processing method, and program | |
WO2017159182A1 (en) | Display control device, display apparatus, television receiver, control method for display control device, control program, and recording medium | |
WO2020031742A1 (en) | Image processing device, image processing method, and program | |
US7894667B2 (en) | Image processing method and apparatus thereof | |
JP6582994B2 (en) | Image processing apparatus, image processing method, and program | |
US10205922B2 (en) | Display control apparatus, method of controlling the same, and non-transitory computer-readable storage medium | |
JP5365881B2 (en) | Image processing apparatus and image processing method | |
JP2009290828A (en) | Image processor, and image processing method | |
KR20150040559A (en) | Apparatus for Improving Image Quality and Computer-Readable Recording Medium with Program Therefor | |
JP6708131B2 (en) | Video processing device, video processing method and program | |
JP2008123472A (en) | Video signal processor | |
JP2020030262A (en) | Image display device and control method thereof | |
US20240031545A1 (en) | Information processing apparatus, information processing method, and storage medium | |
JP6693269B2 (en) | Video processing device, video processing method and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15737513 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2015557764 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 15737513 Country of ref document: EP Kind code of ref document: A1 |