US20210049427A1 - Image processing apparatus, image processing method, and image processing system - Google Patents
Image processing apparatus, image processing method, and image processing system Download PDFInfo
- Publication number
- US20210049427A1 US20210049427A1 US16/966,473 US201816966473A US2021049427A1 US 20210049427 A1 US20210049427 A1 US 20210049427A1 US 201816966473 A US201816966473 A US 201816966473A US 2021049427 A1 US2021049427 A1 US 2021049427A1
- Authority
- US
- United States
- Prior art keywords
- image
- mode
- fusion
- black
- processing apparatus
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 61
- 238000003672 processing method Methods 0.000 title claims description 5
- 230000004927 fusion Effects 0.000 claims abstract description 31
- 238000007499 fusion processing Methods 0.000 claims abstract description 23
- 230000035945 sensitivity Effects 0.000 claims description 56
- 230000007423 decrease Effects 0.000 claims description 11
- 230000007704 transition Effects 0.000 description 36
- 230000008859 change Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 13
- 230000000694 effects Effects 0.000 description 12
- 238000000034 method Methods 0.000 description 8
- 238000007781 pre-processing Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000006872 improvement Effects 0.000 description 4
- 230000001965 increasing effect Effects 0.000 description 4
- 230000010365 information processing Effects 0.000 description 3
- 230000004888 barrier function Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000002950 deficient Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001151 other effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
Images
Classifications
-
- G06K9/6293—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
- G06F18/256—Fusion techniques of classification results, e.g. of results related to same input data of results relating to different input data, e.g. multimodal recognition
-
- G06K9/6289—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present disclosure relates to an image processing apparatus, an image processing method, and an image processing system.
- Patent Document 1 describes that a captured image generated by a camera removable from an information processing terminal is supplied to the information processing terminal by wireless communication.
- Patent Document 2 described below describes that a plurality of image capturing units is provided to simultaneously generate a plurality of images having different image qualities, e.g., images having a first angle of view and a second angle of view narrower than the second angle of view.
- a difference occurs in a viewpoint position between a mode based on the black-and-white image and a mode based on the color image. Furthermore, in this case, in the mode based on the black-and-white image and the mode based on the color image, a difference occurs in luminance and color shade due to a difference in characteristics of the respective image capturing devices or the like.
- an image processing apparatus including: a fusion processing unit that fuses a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and a fusion mode determination unit that determines a mode of the fusion in accordance with the value of a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
- an image processing method including: fusing a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and determining a mode of the fusion in accordance with the value of a predetermined variable and setting, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- an image processing system including: a first image capturing apparatus that captures a black-and-white image; a second image capturing apparatus that captures a color image; and an image processing apparatus including a fusion processing unit that fuses image information of the black-and-white image and image information of the color image, and a fusion mode determination unit that determines a mode of the fusion in accordance with a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
- FIG. 1 is a schematic diagram showing a schematic configuration of an image capturing apparatus according to an embodiment of the present disclosure.
- FIG. 2 is a schematic diagram showing an example of a fusion mode.
- FIG. 3 is a schematic diagram showing an effect (S/N ratio of luminance, sense of resolution) in a case where a black-and-white image and a color image are fused and the degree of failure of the fused image for each of mode A, mode B, and mode C.
- FIG. 4 is a schematic diagram showing alignment processing performed in the mode B and the mode C.
- FIG. 5 is a schematic diagram showing parallax and occlusion between a color image and a black-and-white image due to a difference in position between an image capturing unit 100 and an image capturing unit 110 .
- FIG. 6 is a schematic diagram showing mode transition in accordance with ISO sensitivity.
- FIG. 7 is a schematic diagram showing the configuration of an image processing system in which an image capturing apparatus that captures a color image and an image capturing apparatus that captures a black-and-white image are configured separately from an image processing apparatus.
- the image processing apparatus 1000 is included in an image capturing apparatus such as a digital camera and an electronic apparatus such as a smartphone and a tablet terminal.
- the image processing apparatus 1000 has the two-lens image capturing units 100 and 110 .
- Each of the image capturing units 100 and 110 includes an image capturing device and an optical system such as an optical lens.
- the optical system may include a zoom optical system and a focus optical system, and may change the zoom magnification and the focus position.
- the image capturing device included in each of the image capturing unit 100 and the image capturing unit 110 is constituted of a CMOS image sensor and the like, and performs photoelectric conversion of light captured by a lens to generate image data of a captured image.
- the image capturing device of the image capturing unit 100 captures a color image.
- the image capturing device of the image capturing unit 110 captures a black-and-white (B/W) image.
- the image capturing unit 100 and the image capturing unit 110 can capture a still image and a moving image.
- the image capturing device of the image capturing unit 100 that captures a color image uses a color filter of Bayer array.
- the Bayer array two pixels at diagonal positions are green (G) pixels in a pixel unit of 2 ⁇ 2 pixels, and the remaining pixels are a red (R) pixel and a blue (B) pixel. That is, the image capturing device of the image capturing unit 100 is constituted by color pixels, each pixel of which outputs an electric signal based on the incident light amount of any one color component of red, blue, and green, and generates image data of a color image in which each pixel indicates any of the three primary color (RGB) components.
- the image data of the Bayer array generates full-color image data by interpolating and generating pixels between the same colors by demosaic processing, and hence the substantial sense of resolution is deteriorated.
- artifacts such as moire and maze noise may be generated.
- the image capturing device of the image capturing unit 110 all pixels are constituted by white (W) pixels that output an electric signal based on the incident light amount in the entire wavelength region of visible light. Accordingly, the image capturing device of the image capturing unit 110 generates image data of a black-and-white image.
- the image capturing device of the image capturing unit 110 is formed only of white (W) pixels, it is not required to perform demosaic processing, and it is possible to obtain image data having a higher sense of resolution and less artifacts than those by the image capturing device of the image capturing unit 100 .
- the image processing apparatus 1000 of the present embodiment fuses images by a plurality of cameras. Specifically, a black-and-white image obtained from the high-sensitivity black-and-white image capturing device and a color image obtained from the image capturing device in the Bayer array are fused to acquire a high-sensitivity image.
- the image processing apparatus 1000 includes, in addition to the two-lens image capturing units 100 and 110 , a pre-processing units 200 and 210 , a fusion processing unit 300 , an exposure control unit 400 , a fusion mode determination unit 500 , and a post-processing unit 600 .
- the pre-processing unit 200 performs correction processing such as lens distortion correction and defective pixel correction on the image data of the color image captured by the image capturing unit 100 .
- the pre-processing unit 200 outputs the corrected color image data to the fusion processing unit 300 .
- the pre-processing unit 210 performs correction processing such as lens distortion correction and defective pixel correction on the image data of the black-and-white image captured by the image capturing unit 110 .
- the pre-processing unit 210 outputs the corrected black-and-white image data to the fusion processing unit 300 .
- the exposure control unit 400 not only performs overall exposure control of the image but also determines the ISO sensitivity on the basis of the brightness of the subject image and outputs it to the fusion mode determination unit 500 .
- the ISO sensitivity is an index indicating the brightness of the subject image, and is an index indicating how much the image processing apparatus 1000 amplifies the light entering from the lenses of the image capturing units 100 and 110 .
- ISO 200 indicates that the sensitivity is twice as high as that of ISO100, and in a case of ISO 200, it is possible to obtain an image (photograph) of the same brightness even in a place where the amount of light is half that in a case of ISO 100.
- the exposure control unit 400 determines ISO sensitivity for capturing an image with proper exposure on the basis of the statistical information (detection value) obtained by the pre-processing unit 200 .
- the ISO sensitivity is determined so that the darker the subject image is, the higher the ISO sensitivity becomes.
- the exposure control unit 400 can also determine the brightness of the subject image on the basis of the black-and-white image obtained from the image capturing unit 110 , and determine the ISO sensitivity.
- the exposure control unit 400 can also determine shutter speed (F-number) for capturing an image with proper exposure on the basis of the statistical information (detection value) obtained by the pre-processing unit 200 .
- the fusion processing unit 300 fuses (integrates) the color image captured by the image capturing unit 100 and the black-and-white image captured by the image capturing unit 110 .
- the fusion mode determination unit 500 changes a fusion mode on the basis of ISO sensitivity when the fusion processing unit 300 fuses the black-and-white image and the color image.
- the fusion processing unit 300 performs fusion processing of the color image and the black-and-white image on the basis of the fusion mode.
- the fusion mode determination unit 500 may fix the mode during video recording to the mode at the time of video recording start on the basis of information (video recording start information) indicating that video recording has been started.
- FIG. 2 is a schematic diagram showing an example of the fusion mode.
- FIG. 2 shows three modes of the mode A, the mode B, and the mode C.
- the mode A is a mode in which capturing an image is performed only by a single lens using an image capturing device in the Bayer array included in the image capturing unit 100 . That is, in the mode A, fusion of the color image and the black-and-white image is not performed.
- the mode B is a mode in which the S/N ratio of an image is improved by adding the luminance of a black-and-white image captured by the image capturing unit 100 having a high similarity to the color image on the basis of the color image captured by the image capturing unit 110 .
- the mode C is a mode in which the S/N ratio of an image is improved by adding a color component of a color captured image on the basis of the luminance of a black-and-white image. For example, in a case where a dark subject is captured by the image capturing unit 100 and the image capturing unit 110 , the S/N ratio of the black-and-white image captured by the image capturing unit 110 is better than that of the color image captured by the image capturing unit 100 . Accordingly, use of the luminance of the black-and-white image allows the S/N ratio to be improved, and even a dark subject to be captured as a bright image.
- mode transition is performed such as the mode A-->the mode B-->the mode C. Even in a case where the luminance of the subject is low, the S/N ratio of the luminance can be improved by performing the fusion processing by the mode B and the mode C.
- FIG. 3 is a schematic diagram showing an effect (S/N ratio, sense of resolution, and the degree of failure of the fused image) in a case where a color image and a black-and-white image are fused for each of mode A, mode B, and mode C.
- S/N ratio, sense of resolution, and the degree of failure of the fused image In a case where the subject is dark and the ISO sensitivity is increased, the image captured in the mode A becomes dark, and the S/N ratio and the sense of resolution decrease. In such a case, the S/N ratio and the sense of resolution can be improved by using the mode B and the mode C.
- the S/N ratio is improved as compared with the mode A, but the improvement effect is small.
- the mode B has the same sense of resolution as the mode A has.
- the degree of occurrence of image data errors and signal distortion (hereafter referred to as an artifact) is also small, and hence failure of the image hardly occurs.
- the mode C is much larger in improvement effect on the S/N ratio than the mode B is.
- the sense of resolution is also improved as compared with the mode A.
- the mode C the degree of occurrence of an artifact is also increased, and the failure of an image becomes larger than that in the mode B.
- the mode C is based on the black-and-white image, a viewpoint movement corresponding to the parallax is generated with respect to the mode A and the mode B.
- the occurrence factors of artifacts include, for example, that the parallax between the color image and the black-and-white image is so large that the parallax exceeds the difference amount with which the both can be fused, that the fusion processing cannot be performed because there is no information regarding occlusion and a wrong color is fused, that the calculation of the position vector using the luminance information fails and the color is not fused at an appropriate position, and the like. In either case, a color difference occurs due to the occurrence of artifacts.
- the mode C is high in the improvement effects of the S/N ratio and the sense of resolution, but the failure of the image also becomes large, and thus it can be said to be a high-risk, high-return mode.
- the mode B is low in the improvement effects of the S/N ratio and the sense of resolution, but the failure of the image is small, and thus it can be said to be a low-risk, low-return mode.
- the more the ISO sensitivity increases the more the mode becomes high-risk, high-return.
- FIG. 4 is a schematic diagram showing the alignment processing performed in the mode B and the mode C.
- the positions of the image capturing unit 100 and the image capturing unit 110 constituting a two-lens module do not fully coincide with each other. Therefore, parallax occurs between a black-and-white image 810 and a color image 800 .
- the position of the black-and-white image 810 is aligned to the color image 800 on the basis of the color image 800 .
- the position of the color image 800 is aligned to the black-and-white image on the basis of the black-and-white image 810 . Therefore, when the mode transitions between the mode B and the mode C, the position of the viewpoint is moved by the amount of the parallax.
- the fusion processing unit 300 performs fusion processing for the color image and the black-and-white image having been aligned.
- FIG. 5 is a schematic diagram showing parallax 820 and occlusion 830 between the color image 800 and the black-and-white image 810 due to the difference in position between the image capturing unit 100 and the image capturing unit 110 . It is to be noted that the upper part of FIG. 5 shows the image capturing unit 100 , the image capturing unit 110 , and the subject 1500 viewed from above.
- the parallax 820 and the occlusion 830 occur in a case where an image of the subject 1500 is captured by the image capturing unit 100 and the image capturing unit 110 .
- the amounts of the parallax 820 and the occlusion 830 change in accordance with the distance from the image capturing units 100 and 110 to the subject 1500 .
- the amounts of the parallax 820 and the occlusion 830 decrease as the distance to the subject 1500 becomes longer, and increase as the distance to the subject 1500 becomes shorter.
- the mode transitions on the basis of the ISO sensitivity set in accordance with the brightness of the subject 1500 .
- a change in the viewpoint occurs because the position is based on the color image 800 in the mode A and the mode B, and the position is based on the black-and-white image 810 in the mode C.
- the mode B since the luminance of the black-and-white image is added on the basis of the color of the color image, a difference between the luminance and the color is unlikely to occur.
- the mode C since the color of the color image is put on the basis of the luminance of the black-and-white image, a color difference with respect to the luminance may occur. Accordingly, when transition is performed between the mode B and the mode C, the luminance and color shade may change or the S/N may change. Accordingly, there is a possibility that the artifact will be conspicuous due to the mode transition.
- the threshold value for determining the mode is set in accordance with each mode. More specifically, the threshold value of the ISO sensitivity for mode transition is set to a value different between a case where the ISO sensitivity increases and a case where the ISO sensitivity decreases, and in a case where the ISO sensitivity is a value near the boundary of mode switching, frequent switching of the mode is suppressed.
- FIG. 6 is a schematic diagram showing mode transition in accordance with ISO sensitivity.
- a threshold value a of the ISO sensitivity at the time of transitioning from the mode A to the mode B and a threshold value c of the ISO sensitivity at the time of transitioning from the mode A or the mode B to the mode C are set.
- the ISO sensitivity becomes larger than the threshold value a the transition from the mode A to the mode B is performed.
- the ISO sensitivity becomes larger than the threshold value c the transition from the mode A or B to the mode C is performed.
- a threshold value d of the ISO sensitivity at the time of transitioning from the mode C to the mode B and a threshold value b of the ISO sensitivity at the time of transitioning from the mode B or C to the mode A are set.
- the threshold value d the transition from the mode C to the mode B is performed.
- the threshold value b the transition from the mode B or C to the mode A is performed.
- the threshold value a of the ISO sensitivity at the time of transitioning from the mode A to the mode B is larger than the threshold value b in a case of transitioning from the mode B or C to the mode A.
- the threshold value c of the ISO sensitivity at the time of transitioning from the mode A or B to the mode C is larger than the threshold value d in a case of transitioning from the mode C to the mode B.
- the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases. If the threshold value of the ISO sensitivity at the time of switching the modes is set to be the same for the case where the ISO sensitivity increases and the case where it decreases, in a case where the ISO sensitivity determined on the basis of the luminance of the subject is near the threshold value of switching between the mode B and the mode C for example, there is a possibility that the transition between the mode B and the mode C is frequently performed in accordance with the increase or decrease of the ISO sensitivity.
- the transition accompanied by a change visually perceivable by the user may be frequently repeated near the switching threshold value of the modulation.
- a viewpoint change, a luminance change, a color shade change, a change in the S/N ratio, and the like occur, thereby giving the user a sense of strangeness.
- the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases, the mode can be determined by hysteresis control.
- the ISO sensitivity is a value near the boundary of mode transition, it is possible to suppress the mode from frequently switching, and it is possible to suppress giving the user a sense of strangeness.
- the mode is switched on the basis of the ISO sensitivity
- the mode can also be switched on the basis of other parameters depending on the image capturing environment.
- Other parameters include, for example, shutter speed, EV value, zoom position, focus position, and the like.
- the threshold value of the mode transition can be optimally set in accordance with the change in conspicuousness of image data errors and signal distortion (artifact) due to zooming.
- FIG. 7 is a schematic diagram showing the configuration of an image processing system 2000 in which an image capturing apparatus 1100 that captures a color image and an image capturing apparatus 1200 that captures a black-and-white image are configured separately from the image processing apparatus 1000 .
- each of the image capturing apparatus 1100 and the image capturing apparatus 1200 is communicable with the image processing apparatus 1000 wirelessly or by wire. It is to be noted that for the communication method, any method can be adopted and is not particularly limited. As shown in FIG. 7 , data of a color image captured by the image capturing apparatus 1100 is transmitted to the image processing apparatus 1000 . In addition, data of a black-and-white image captured by the image capturing apparatus 1200 is transmitted to the image processing apparatus 1000 . The processing in the image processing apparatus 1000 is performed similarly to the processing described with reference to FIG. 1 .
- the mode transition is performed in accordance with a modulation axis (ISO sensitivity) depending on the image capturing environment, it is possible to suppress mode transition from frequently occurring by individually setting the switching threshold value of modulation in accordance with the state of mode.
- the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases, mode switching is not frequently performed, and it is possible to suppress a sense of strangeness to the user.
- An image processing apparatus including:
- a fusion processing unit that fuses a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject
- a fusion mode determination unit that determines a mode of the fusion in accordance with the value of a predetermined variable and sets, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- the predetermined variable is a variable related to an environment at the time of capturing an image of the subject.
- the mode includes a first mode in which the color image is output from the fusion processing unit without fusing the color image and the black-and-white image, a second mode in which information of the black-and-white image is fused to information of the color image on the basis of the color image, and a third mode in which information of the color image is fused to information of the black-and-white image on the basis of the black-and-white image.
- An image processing method including:
- determining a mode of the fusion in accordance with the value of a predetermined variable and setting, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- An image processing system including:
- a first image capturing apparatus that captures a black-and-white image
- an image processing apparatus including a fusion processing unit that fuses image information of the black-and-white image and image information of the color image, and a fusion mode determination unit that determines a mode of the fusion in accordance with a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
Abstract
Description
- The present disclosure relates to an image processing apparatus, an image processing method, and an image processing system.
- Conventionally, in information processing terminals such as a portable smartphone, the image performance of the image capturing unit is lower than that of a single-lens reflex camera or the like due to a demand for miniaturization and thinning. Thus, for example,
Patent Document 1 described below describes that a captured image generated by a camera removable from an information processing terminal is supplied to the information processing terminal by wireless communication. Furthermore, Patent Document 2 described below describes that a plurality of image capturing units is provided to simultaneously generate a plurality of images having different image qualities, e.g., images having a first angle of view and a second angle of view narrower than the second angle of view. -
- Patent Document 1: Japanese Patent Application Laid-Open No. 2015-088824
- Patent Document 2: Japanese Patent Application Laid-Open No. 2013-219525
- As a method for enhancing the sensitivity of a digital camera or the like, there is a method for increasing the lens aperture or the cell size of an image capturing device. However, this method has a barrier in miniaturizing and thinning the image capturing device.
- On the other hand, in a case of assuming that a high-sensitivity image is acquired by fusing a black-and-white image obtained from a high-sensitivity black-and-white image capturing device and a color image obtained from an image capturing device in a Bayer array, a difference occurs in a viewpoint position between a mode based on the black-and-white image and a mode based on the color image. Furthermore, in this case, in the mode based on the black-and-white image and the mode based on the color image, a difference occurs in luminance and color shade due to a difference in characteristics of the respective image capturing devices or the like. Accordingly, when the mode transitions, there is a possibility that the subject image visually recognized by the user changes and gives the user a sense of strangeness. In particular, in a case of capturing a moving image, if these modes transition in real time, there is a high possibility of giving to the user a sense of strangeness.
- Therefore, it has been required to suppress a sense of strangeness caused by mode transition when images obtained from a plurality of image capturing devices are fused.
- According to the present disclosure, it is provided an image processing apparatus including: a fusion processing unit that fuses a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and a fusion mode determination unit that determines a mode of the fusion in accordance with the value of a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
- Furthermore, according to the present disclosure, it is provided an image processing method including: fusing a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and determining a mode of the fusion in accordance with the value of a predetermined variable and setting, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- Furthermore, according to the present disclosure, it is provided an image processing system including: a first image capturing apparatus that captures a black-and-white image; a second image capturing apparatus that captures a color image; and an image processing apparatus including a fusion processing unit that fuses image information of the black-and-white image and image information of the color image, and a fusion mode determination unit that determines a mode of the fusion in accordance with a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
- As described above, according to the present disclosure, it is possible to suppress a sense of strangeness caused by mode transition when images obtained from a plurality of image capturing devices are fused.
- It is to be noted that the above effects are not necessarily restrictive, and any of the effects shown in the present description or other effects that can be understood from the present description may be achieved in addition to or in place of the above effects.
-
FIG. 1 is a schematic diagram showing a schematic configuration of an image capturing apparatus according to an embodiment of the present disclosure. -
FIG. 2 is a schematic diagram showing an example of a fusion mode. -
FIG. 3 is a schematic diagram showing an effect (S/N ratio of luminance, sense of resolution) in a case where a black-and-white image and a color image are fused and the degree of failure of the fused image for each of mode A, mode B, and mode C. -
FIG. 4 is a schematic diagram showing alignment processing performed in the mode B and the mode C. -
FIG. 5 is a schematic diagram showing parallax and occlusion between a color image and a black-and-white image due to a difference in position between animage capturing unit 100 and animage capturing unit 110. -
FIG. 6 is a schematic diagram showing mode transition in accordance with ISO sensitivity. -
FIG. 7 is a schematic diagram showing the configuration of an image processing system in which an image capturing apparatus that captures a color image and an image capturing apparatus that captures a black-and-white image are configured separately from an image processing apparatus. - A preferred embodiment of the present disclosure will be described below in detail with reference to the accompanying drawings. It is to be noted that in the present description and drawings, components having substantially the same functional configuration are given the same reference numerals, and redundant description thereof is omitted.
- It is to be noted that the description will be given in the following order.
- 1. Configuration Example of Image Processing Apparatus
- 2. Specific Example of Fusion Mode
- 3. Regarding Switching Threshold Value for Mode Transition
- 4. Regarding Other Parameters for Mode Transition
- 5. Configuration Example of Image Processing System
- 1. Configuration Example of Image Processing Apparatus
- First, a schematic configuration of an image processing apparatus (image capturing apparatus) 1000 according to an embodiment of the present disclosure will be described with reference to
FIG. 1 . Theimage processing apparatus 1000 is included in an image capturing apparatus such as a digital camera and an electronic apparatus such as a smartphone and a tablet terminal. Theimage processing apparatus 1000 has the two-lensimage capturing units image capturing units - The image capturing device included in each of the
image capturing unit 100 and theimage capturing unit 110 is constituted of a CMOS image sensor and the like, and performs photoelectric conversion of light captured by a lens to generate image data of a captured image. The image capturing device of theimage capturing unit 100 captures a color image. On the other hand, the image capturing device of theimage capturing unit 110 captures a black-and-white (B/W) image. In the present embodiment, theimage capturing unit 100 and theimage capturing unit 110 can capture a still image and a moving image. - The image capturing device of the
image capturing unit 100 that captures a color image uses a color filter of Bayer array. In the Bayer array, two pixels at diagonal positions are green (G) pixels in a pixel unit of 2×2 pixels, and the remaining pixels are a red (R) pixel and a blue (B) pixel. That is, the image capturing device of theimage capturing unit 100 is constituted by color pixels, each pixel of which outputs an electric signal based on the incident light amount of any one color component of red, blue, and green, and generates image data of a color image in which each pixel indicates any of the three primary color (RGB) components. The image data of the Bayer array generates full-color image data by interpolating and generating pixels between the same colors by demosaic processing, and hence the substantial sense of resolution is deteriorated. In addition, artifacts such as moire and maze noise may be generated. - On the other hand, in the image capturing device of the
image capturing unit 110, all pixels are constituted by white (W) pixels that output an electric signal based on the incident light amount in the entire wavelength region of visible light. Accordingly, the image capturing device of theimage capturing unit 110 generates image data of a black-and-white image. In addition, since the image capturing device of theimage capturing unit 110 is formed only of white (W) pixels, it is not required to perform demosaic processing, and it is possible to obtain image data having a higher sense of resolution and less artifacts than those by the image capturing device of theimage capturing unit 100. - As described earlier, as a method for enhancing the sensitivity of a digital camera or the like, there is a method for increasing the lens aperture or the cell size of an image capturing device. However, this method has a barrier in miniaturizing and thinning the image capturing device. In order to enhance sensitivity while maintaining the miniaturization of the image capturing device, the
image processing apparatus 1000 of the present embodiment fuses images by a plurality of cameras. Specifically, a black-and-white image obtained from the high-sensitivity black-and-white image capturing device and a color image obtained from the image capturing device in the Bayer array are fused to acquire a high-sensitivity image. - As shown in
FIG. 1 , theimage processing apparatus 1000 includes, in addition to the two-lensimage capturing units pre-processing units fusion processing unit 300, anexposure control unit 400, a fusionmode determination unit 500, and apost-processing unit 600. - The
pre-processing unit 200 performs correction processing such as lens distortion correction and defective pixel correction on the image data of the color image captured by theimage capturing unit 100. Thepre-processing unit 200 outputs the corrected color image data to thefusion processing unit 300. - The
pre-processing unit 210 performs correction processing such as lens distortion correction and defective pixel correction on the image data of the black-and-white image captured by theimage capturing unit 110. Thepre-processing unit 210 outputs the corrected black-and-white image data to thefusion processing unit 300. - The
exposure control unit 400 not only performs overall exposure control of the image but also determines the ISO sensitivity on the basis of the brightness of the subject image and outputs it to the fusionmode determination unit 500. The ISO sensitivity is an index indicating the brightness of the subject image, and is an index indicating how much theimage processing apparatus 1000 amplifies the light entering from the lenses of theimage capturing units ISO 200 indicates that the sensitivity is twice as high as that of ISO100, and in a case ofISO 200, it is possible to obtain an image (photograph) of the same brightness even in a place where the amount of light is half that in a case ofISO 100. - Specifically, the
exposure control unit 400 determines ISO sensitivity for capturing an image with proper exposure on the basis of the statistical information (detection value) obtained by thepre-processing unit 200. At this time, the ISO sensitivity is determined so that the darker the subject image is, the higher the ISO sensitivity becomes. Theexposure control unit 400 can also determine the brightness of the subject image on the basis of the black-and-white image obtained from theimage capturing unit 110, and determine the ISO sensitivity. In addition, theexposure control unit 400 can also determine shutter speed (F-number) for capturing an image with proper exposure on the basis of the statistical information (detection value) obtained by thepre-processing unit 200. - The
fusion processing unit 300 fuses (integrates) the color image captured by theimage capturing unit 100 and the black-and-white image captured by theimage capturing unit 110. The fusionmode determination unit 500 changes a fusion mode on the basis of ISO sensitivity when thefusion processing unit 300 fuses the black-and-white image and the color image. Thefusion processing unit 300 performs fusion processing of the color image and the black-and-white image on the basis of the fusion mode. When video recording is started at the time of capturing a stand-by moving image, the fusionmode determination unit 500 may fix the mode during video recording to the mode at the time of video recording start on the basis of information (video recording start information) indicating that video recording has been started. - 2. Specific Example of Fusion Mode
-
FIG. 2 is a schematic diagram showing an example of the fusion mode.FIG. 2 shows three modes of the mode A, the mode B, and the mode C. The mode A is a mode in which capturing an image is performed only by a single lens using an image capturing device in the Bayer array included in theimage capturing unit 100. That is, in the mode A, fusion of the color image and the black-and-white image is not performed. - The mode B is a mode in which the S/N ratio of an image is improved by adding the luminance of a black-and-white image captured by the
image capturing unit 100 having a high similarity to the color image on the basis of the color image captured by theimage capturing unit 110. The mode C is a mode in which the S/N ratio of an image is improved by adding a color component of a color captured image on the basis of the luminance of a black-and-white image. For example, in a case where a dark subject is captured by theimage capturing unit 100 and theimage capturing unit 110, the S/N ratio of the black-and-white image captured by theimage capturing unit 110 is better than that of the color image captured by theimage capturing unit 100. Accordingly, use of the luminance of the black-and-white image allows the S/N ratio to be improved, and even a dark subject to be captured as a bright image. - In the present embodiment, on the basis of the ISO sensitivity determined by the
exposure control unit 400, as the ISO sensitivity is larger, mode transition is performed such as the mode A-->the mode B-->the mode C. Even in a case where the luminance of the subject is low, the S/N ratio of the luminance can be improved by performing the fusion processing by the mode B and the mode C. -
FIG. 3 is a schematic diagram showing an effect (S/N ratio, sense of resolution, and the degree of failure of the fused image) in a case where a color image and a black-and-white image are fused for each of mode A, mode B, and mode C. In a case where the subject is dark and the ISO sensitivity is increased, the image captured in the mode A becomes dark, and the S/N ratio and the sense of resolution decrease. In such a case, the S/N ratio and the sense of resolution can be improved by using the mode B and the mode C. - As shown in
FIG. 3 , in the mode B, the S/N ratio is improved as compared with the mode A, but the improvement effect is small. In addition, the mode B has the same sense of resolution as the mode A has. In addition, in the mode B, the degree of occurrence of image data errors and signal distortion (hereafter referred to as an artifact) is also small, and hence failure of the image hardly occurs. - The mode C is much larger in improvement effect on the S/N ratio than the mode B is. In addition, in the mode C, the sense of resolution is also improved as compared with the mode A. On the other hand, in the mode C, the degree of occurrence of an artifact is also increased, and the failure of an image becomes larger than that in the mode B. In addition, the mode C is based on the black-and-white image, a viewpoint movement corresponding to the parallax is generated with respect to the mode A and the mode B.
- It is to be noted that the occurrence factors of artifacts include, for example, that the parallax between the color image and the black-and-white image is so large that the parallax exceeds the difference amount with which the both can be fused, that the fusion processing cannot be performed because there is no information regarding occlusion and a wrong color is fused, that the calculation of the position vector using the luminance information fails and the color is not fused at an appropriate position, and the like. In either case, a color difference occurs due to the occurrence of artifacts.
- As described above, the mode C is high in the improvement effects of the S/N ratio and the sense of resolution, but the failure of the image also becomes large, and thus it can be said to be a high-risk, high-return mode. On the other hand, the mode B is low in the improvement effects of the S/N ratio and the sense of resolution, but the failure of the image is small, and thus it can be said to be a low-risk, low-return mode. In other words, the more the ISO sensitivity increases, the more the mode becomes high-risk, high-return.
-
FIG. 4 is a schematic diagram showing the alignment processing performed in the mode B and the mode C. When an image of a subject 1500 shown inFIG. 4 is captured, the positions of theimage capturing unit 100 and theimage capturing unit 110 constituting a two-lens module do not fully coincide with each other. Therefore, parallax occurs between a black-and-white image 810 and acolor image 800. In the mode B, the position of the black-and-white image 810 is aligned to thecolor image 800 on the basis of thecolor image 800. In the mode C, on the other hand, the position of thecolor image 800 is aligned to the black-and-white image on the basis of the black-and-white image 810. Therefore, when the mode transitions between the mode B and the mode C, the position of the viewpoint is moved by the amount of the parallax. Thefusion processing unit 300 performs fusion processing for the color image and the black-and-white image having been aligned. -
FIG. 5 is a schematicdiagram showing parallax 820 andocclusion 830 between thecolor image 800 and the black-and-white image 810 due to the difference in position between theimage capturing unit 100 and theimage capturing unit 110. It is to be noted that the upper part ofFIG. 5 shows theimage capturing unit 100, theimage capturing unit 110, and the subject 1500 viewed from above. - As shown in
FIG. 5 , in a case where an image of the subject 1500 is captured by theimage capturing unit 100 and theimage capturing unit 110, what are seen by theimage capturing unit 100 and theimage capturing unit 110 are different from each other, and theparallax 820 and theocclusion 830 occur. The amounts of theparallax 820 and theocclusion 830 change in accordance with the distance from theimage capturing units parallax 820 and theocclusion 830 decrease as the distance to the subject 1500 becomes longer, and increase as the distance to the subject 1500 becomes shorter. - As described above, in the present embodiment, the mode transitions on the basis of the ISO sensitivity set in accordance with the brightness of the subject 1500. On the other hand, when transition is performed between the mode A or the mode B and the mode C, a change in the viewpoint occurs because the position is based on the
color image 800 in the mode A and the mode B, and the position is based on the black-and-white image 810 in the mode C. - In addition, when transition is performed between the mode A or the mode B and the mode C, a change in color shade and a change in luminance due to a spectral sensitivity difference between the image capturing device of the
image capturing unit 100 and the image capturing device of theimage capturing unit 110 occur. Furthermore, when transition is performed between the mode A or the mode B and the mode C, a difference occurs in the S/N ratio of the luminance. - More specifically, in the mode B, since the luminance of the black-and-white image is added on the basis of the color of the color image, a difference between the luminance and the color is unlikely to occur. On the other hand, in the mode C, since the color of the color image is put on the basis of the luminance of the black-and-white image, a color difference with respect to the luminance may occur. Accordingly, when transition is performed between the mode B and the mode C, the luminance and color shade may change or the S/N may change. Accordingly, there is a possibility that the artifact will be conspicuous due to the mode transition.
FIG. 3 shows that when transition is performed between the mode B and mode C, a change in viewpoint, a change in luminance and color shade, and a change in S/N ratio occur. Similarly, also in a case where transition is performed between the mode A and the mode B, the luminance and the S/N ratio change. - As described above, at the time of mode transition, changes visually perceivable by the user, such as viewpoint movement, the angle of view, exposure, color shade, resolution, a sense of resolution, and noise, occur. Then, when these changes occur at the time of the mode transition, the user may have a sense of strangeness. In particular, when the ISO sensitivity defined on the basis of the brightness of the subject 1500 is near the boundary where two modes are switched, the mode is frequently switched between the two modes in accordance with a slight change in the brightness of the subject 1500, and there is a possibility that the user has a large sense of strangeness.
- In addition, these changes perceived by the user become noticeable particularly in a case where the user visually recognizes the moving image. When the mode is switched while the user is viewing the moving image, a change in viewpoint, a change in luminance and color shade, and a change in S/N ratio occur in real time, thereby giving a sense of strangeness to the user visually recognizing the moving image.
- 3. Regarding Switching Threshold Value for Mode Transition
- In the present embodiment, when the mode is transitioned on the basis of the ISO sensitivity, the threshold value for determining the mode is set in accordance with each mode. More specifically, the threshold value of the ISO sensitivity for mode transition is set to a value different between a case where the ISO sensitivity increases and a case where the ISO sensitivity decreases, and in a case where the ISO sensitivity is a value near the boundary of mode switching, frequent switching of the mode is suppressed.
-
FIG. 6 is a schematic diagram showing mode transition in accordance with ISO sensitivity. As shown inFIG. 6 , in a case where the ISO sensitivity increases, a threshold value a of the ISO sensitivity at the time of transitioning from the mode A to the mode B and a threshold value c of the ISO sensitivity at the time of transitioning from the mode A or the mode B to the mode C are set. When the ISO sensitivity becomes larger than the threshold value a, the transition from the mode A to the mode B is performed. In addition, when the ISO sensitivity becomes larger than the threshold value c, the transition from the mode A or B to the mode C is performed. - In addition, in a case where the ISO sensitivity decreases, a threshold value d of the ISO sensitivity at the time of transitioning from the mode C to the mode B and a threshold value b of the ISO sensitivity at the time of transitioning from the mode B or C to the mode A are set. When the ISO sensitivity becomes smaller than the threshold value d, the transition from the mode C to the mode B is performed. In addition, when the ISO sensitivity becomes smaller than the threshold value b, the transition from the mode B or C to the mode A is performed.
- As shown in
FIG. 6 , the threshold value a of the ISO sensitivity at the time of transitioning from the mode A to the mode B is larger than the threshold value b in a case of transitioning from the mode B or C to the mode A. In addition, the threshold value c of the ISO sensitivity at the time of transitioning from the mode A or B to the mode C is larger than the threshold value d in a case of transitioning from the mode C to the mode B. These threshold values are individually set for each of the modes A, B, and C at the boundary of the ISO sensitivity at both ends of the modes A, B, and C. That is, when the mode is transitioned on the basis of the ISO sensitivity, the threshold value for determining the mode is set in accordance with each mode. - As described above, in the present embodiment, the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases. If the threshold value of the ISO sensitivity at the time of switching the modes is set to be the same for the case where the ISO sensitivity increases and the case where it decreases, in a case where the ISO sensitivity determined on the basis of the luminance of the subject is near the threshold value of switching between the mode B and the mode C for example, there is a possibility that the transition between the mode B and the mode C is frequently performed in accordance with the increase or decrease of the ISO sensitivity. In other words, in a case where the mode transition is performed in accordance with a modulation axis (ISO sensitivity) depending on the image capturing environment, the transition accompanied by a change visually perceivable by the user may be frequently repeated near the switching threshold value of the modulation. In this case, as described above for example, when the mode transitions between the mode B and the mode C, a viewpoint change, a luminance change, a color shade change, a change in the S/N ratio, and the like occur, thereby giving the user a sense of strangeness.
- According to the present embodiment, since the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases, the mode can be determined by hysteresis control. Thus, even if the ISO sensitivity is a value near the boundary of mode transition, it is possible to suppress the mode from frequently switching, and it is possible to suppress giving the user a sense of strangeness.
- 4. Regarding Other Parameters for Mode Transition
- While in the above example, the mode is switched on the basis of the ISO sensitivity, the mode can also be switched on the basis of other parameters depending on the image capturing environment. Other parameters include, for example, shutter speed, EV value, zoom position, focus position, and the like. By switching the modes A to C in accordance with these parameters, it is possible to output an optimum image in accordance with the values of the parameters from the
image processing apparatus 1000. In addition, mode control may be performed by combining a plurality of parameters. - For example, by including a plurality of threshold values of mode transition by ISO sensitivity in accordance with the zoom position, the threshold value of the mode transition can be optimally set in accordance with the change in conspicuousness of image data errors and signal distortion (artifact) due to zooming.
- 5. Configuration Example of Image Processing System
- In the
image processing apparatus 1000 shown inFIG. 1 , theimage capturing unit 100 and theimage capturing unit 110 may be provided as an image capturing apparatus separate from theimage processing apparatus 1000.FIG. 7 is a schematic diagram showing the configuration of animage processing system 2000 in which animage capturing apparatus 1100 that captures a color image and animage capturing apparatus 1200 that captures a black-and-white image are configured separately from theimage processing apparatus 1000. - In
FIG. 7 , each of theimage capturing apparatus 1100 and theimage capturing apparatus 1200 is communicable with theimage processing apparatus 1000 wirelessly or by wire. It is to be noted that for the communication method, any method can be adopted and is not particularly limited. As shown inFIG. 7 , data of a color image captured by theimage capturing apparatus 1100 is transmitted to theimage processing apparatus 1000. In addition, data of a black-and-white image captured by theimage capturing apparatus 1200 is transmitted to theimage processing apparatus 1000. The processing in theimage processing apparatus 1000 is performed similarly to the processing described with reference toFIG. 1 . - As described above, according to the present embodiment, in a case where the mode transition is performed in accordance with a modulation axis (ISO sensitivity) depending on the image capturing environment, it is possible to suppress mode transition from frequently occurring by individually setting the switching threshold value of modulation in accordance with the state of mode. Specifically, since the threshold value of the ISO sensitivity at the time of switching the mode is set to a value different between the case where the ISO sensitivity increases and the case where it decreases, mode switching is not frequently performed, and it is possible to suppress a sense of strangeness to the user.
- While the preferred embodiment of the present disclosure has been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such an example. It is clear that a person ordinarily skilled in the art of the present disclosure can conceive of various modifications or variations within the scope of the technical idea set forth in the claims, and those modifications or variations are also understood to naturally fall within the technical scope of the present disclosure.
- In addition, the effects described in the present description are merely illustrative or exemplary and not restrictive. That is, the technology according to the present disclosure can achieve other effects apparent to those skilled in the art from the description in the present description in addition to or in place of the above effects.
- It is to be noted that the following configuration also falls within the technical scope of the present disclosure.
- (1) An image processing apparatus including:
- a fusion processing unit that fuses a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and
- a fusion mode determination unit that determines a mode of the fusion in accordance with the value of a predetermined variable and sets, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- (2) The image processing apparatus according to (1), in which the threshold value corresponding to a boundary where any mode and another mode are switched is different between a case where the variable increases and a case where the variable decreases.
- (3) The image processing apparatus according to (1) or (2), in which the fusion processing unit fuses the image information of a black-and-white image and the image information of a color image.
- (4) The image processing apparatus according to any of (1) to (3), in which a plurality of the image capturing devices captures a moving image of the subject.
- (5) The image processing apparatus according to any of (1) to (4), in which the predetermined variable is a variable related to an environment at the time of capturing an image of the subject.
- (6) The image processing apparatus according to any of (1) to (5), in which the predetermined variable is ISO sensitivity, shutter speed, EV value, zoom magnification, or focus position.
- (7) The image processing apparatus according to (3), in which the mode includes a first mode in which the color image is output from the fusion processing unit without fusing the color image and the black-and-white image, a second mode in which information of the black-and-white image is fused to information of the color image on the basis of the color image, and a third mode in which information of the color image is fused to information of the black-and-white image on the basis of the black-and-white image.
- (8) The image processing apparatus according to (7), in which in the second mode, luminance information of the black-and-white image is fused to information of the color image.
- (9) The image processing apparatus according to (7) or (8), in which in the third mode, color information of the color image is fused to information of the black-and-white image.
- (10) The image processing apparatus according to any of (1) to (9), in which when video recording is started at a time of capturing a stand-by moving image, the fusion mode determination unit fixes to the mode at the time of starting video recording.
- (11) An image processing method including:
- fusing a plurality of pieces of image information obtained from each of a plurality of image capturing devices that captures the same subject; and
- determining a mode of the fusion in accordance with the value of a predetermined variable and setting, in accordance with each of a plurality of the modes, the threshold value of the variable for determining the mode.
- (12) An image processing system including:
- a first image capturing apparatus that captures a black-and-white image;
- a second image capturing apparatus that captures a color image; and
- an image processing apparatus including a fusion processing unit that fuses image information of the black-and-white image and image information of the color image, and a fusion mode determination unit that determines a mode of the fusion in accordance with a predetermined variable and sets, in accordance with each of a plurality of the modes, a threshold value of the variable for determining the mode.
-
- 1000 Image capturing apparatus
- 100, 110 Image capturing unit
- 300 Fusion processing unit
- 500 Fusion mode determination unit
Claims (12)
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2018-019767 | 2018-02-07 | ||
JPJP2018-019767 | 2018-02-07 | ||
JP2018019767 | 2018-02-07 | ||
PCT/JP2018/045561 WO2019155757A1 (en) | 2018-02-07 | 2018-12-11 | Image processing device, image processing method, and image processing system |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210049427A1 true US20210049427A1 (en) | 2021-02-18 |
US11347978B2 US11347978B2 (en) | 2022-05-31 |
Family
ID=67548876
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/966,473 Active 2038-12-30 US11347978B2 (en) | 2018-02-07 | 2018-12-11 | Image processing apparatus, image processing method, and image processing system |
Country Status (4)
Country | Link |
---|---|
US (1) | US11347978B2 (en) |
EP (1) | EP3734961A4 (en) |
JP (1) | JP7052811B2 (en) |
WO (1) | WO2019155757A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113810601A (en) * | 2021-08-12 | 2021-12-17 | 荣耀终端有限公司 | Terminal image processing method and device and terminal equipment |
US11803949B2 (en) * | 2020-08-06 | 2023-10-31 | Apple Inc. | Image fusion architecture with multimode operations |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113642654B (en) * | 2021-08-16 | 2022-08-30 | 北京百度网讯科技有限公司 | Image feature fusion method and device, electronic equipment and storage medium |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2007267330A (en) * | 2006-03-30 | 2007-10-11 | Olympus Imaging Corp | Digital single-lens reflex camera |
WO2013079778A2 (en) * | 2011-12-02 | 2013-06-06 | Nokia Corporation | Method, apparatus and computer program product for capturing images |
JP2013219525A (en) | 2012-04-06 | 2013-10-24 | Nec Saitama Ltd | Imaging device, control method therefor, and program |
JP2014143457A (en) * | 2013-01-22 | 2014-08-07 | Canon Inc | HDR switching control method |
DE102013012810A1 (en) * | 2013-08-01 | 2015-02-05 | Connaught Electronics Ltd. | Method for activating and deactivating an image correction function, camera system and motor vehicle |
JP6201641B2 (en) | 2013-10-29 | 2017-09-27 | ソニー株式会社 | Information processing apparatus, imaging apparatus, imaging system, information processing apparatus control method, imaging apparatus control method, and program |
US9565361B2 (en) * | 2015-05-14 | 2017-02-07 | Altek Semiconductor Corp. | Image capturing device and hybrid image processing method thereof |
US9743015B2 (en) * | 2015-05-22 | 2017-08-22 | Samsung Electronics Co., Ltd. | Image capturing apparatus and method of controlling the same |
CN105049718A (en) * | 2015-07-06 | 2015-11-11 | 深圳市金立通信设备有限公司 | Image processing method and terminal |
WO2017044130A1 (en) * | 2015-09-11 | 2017-03-16 | The Ip Law Office Of Guy Levi, Llc | Multi-modal optoelectronic vision system and uses thereof |
JP2017061079A (en) * | 2015-09-24 | 2017-03-30 | キヤノン株式会社 | Image forming apparatus, and control method therefor |
KR102446442B1 (en) * | 2015-11-24 | 2022-09-23 | 삼성전자주식회사 | Digital photographing apparatus and the operating method for the same |
WO2018092378A1 (en) | 2016-11-17 | 2018-05-24 | シャープ株式会社 | Information processing device, information processing method, and program |
CN109937568B (en) * | 2016-11-17 | 2021-03-09 | 索尼公司 | Image processing apparatus, image processing method, and program |
JP2019129446A (en) | 2018-01-25 | 2019-08-01 | ソニーセミコンダクタソリューションズ株式会社 | Image processing device, image processing method, and program |
-
2018
- 2018-12-11 EP EP18905319.2A patent/EP3734961A4/en not_active Withdrawn
- 2018-12-11 WO PCT/JP2018/045561 patent/WO2019155757A1/en unknown
- 2018-12-11 US US16/966,473 patent/US11347978B2/en active Active
- 2018-12-11 JP JP2019570320A patent/JP7052811B2/en active Active
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11803949B2 (en) * | 2020-08-06 | 2023-10-31 | Apple Inc. | Image fusion architecture with multimode operations |
CN113810601A (en) * | 2021-08-12 | 2021-12-17 | 荣耀终端有限公司 | Terminal image processing method and device and terminal equipment |
Also Published As
Publication number | Publication date |
---|---|
EP3734961A1 (en) | 2020-11-04 |
WO2019155757A1 (en) | 2019-08-15 |
US11347978B2 (en) | 2022-05-31 |
EP3734961A4 (en) | 2021-06-09 |
JP7052811B2 (en) | 2022-04-12 |
JPWO2019155757A1 (en) | 2021-01-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10742889B2 (en) | Image photographing method, image photographing apparatus, and terminal | |
CN108028895B (en) | Calibration of defective image sensor elements | |
US8988585B2 (en) | Focus adjustment apparatus | |
US9282316B2 (en) | Stereoscopic imaging device and stereoscopic imaging method | |
US9699427B2 (en) | Imaging device, imaging method, and image processing device | |
US20150092992A1 (en) | Image processing device, image capturing apparatus, and image processing method | |
US10992854B2 (en) | Image processing apparatus, imaging apparatus, image processing method, and storage medium | |
KR101120966B1 (en) | A hand jitter reduction system for cameras | |
US11347978B2 (en) | Image processing apparatus, image processing method, and image processing system | |
US20180330529A1 (en) | Image processing apparatus, image processing method, and computer readable recording medium | |
US9247112B2 (en) | Imaging device and method for displaying electronic viewfinder | |
US9332199B2 (en) | Imaging device, image processing device, and image processing method | |
US9122125B2 (en) | Image-capturing apparatus and method of controlling the same | |
US20120194732A1 (en) | Photographing apparatus, display control method, and program | |
US20130188025A1 (en) | Stereoscopic imaging device and stereoscopic imaging method | |
US11805326B2 (en) | Image processing apparatus, control method thereof, and storage medium | |
CN116055907A (en) | Image processing apparatus, image capturing apparatus, image processing method, and storage medium | |
US9712752B2 (en) | Image processing device, imaging device, image processing method, and computer readable medium, capable of improving visibility of a boundary line between a first display image and a second display image for use in focus-checking | |
JP2011109411A (en) | Method and device for determining white balance correction coefficient, method and device for correcting white balance, and imaging apparatus | |
US9723207B2 (en) | Imaging device and control method therefor | |
KR20190139788A (en) | Methods and apparatus for capturing media using plurality of cameras in electronic device | |
JP6921567B2 (en) | Image processing device, imaging device, pixel abnormality detection method and program | |
JP2019197305A (en) | Image processing system and image processing method and program | |
JP2015228617A (en) | Apparatus and method for displaying focusing frame, and imaging apparatus | |
JP2014139738A (en) | Image processing apparatus, imaging apparatus, and image processing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: SONY CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GOTO, SHUICHI;ITO, KEISUKE;REEL/FRAME:054109/0431 Effective date: 20201002 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |