TW202211165A - Dual sensor imaging system and depth map calculation method thereof - Google Patents

Dual sensor imaging system and depth map calculation method thereof Download PDF

Info

Publication number
TW202211165A
TW202211165A TW109146922A TW109146922A TW202211165A TW 202211165 A TW202211165 A TW 202211165A TW 109146922 A TW109146922 A TW 109146922A TW 109146922 A TW109146922 A TW 109146922A TW 202211165 A TW202211165 A TW 202211165A
Authority
TW
Taiwan
Prior art keywords
image
infrared
color
images
sensor
Prior art date
Application number
TW109146922A
Other languages
Chinese (zh)
Other versions
TWI767484B (en
Inventor
彭詩淵
鄭書峻
黃旭鍊
李運錦
賴國銘
Original Assignee
聚晶半導體股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 聚晶半導體股份有限公司 filed Critical 聚晶半導體股份有限公司
Priority to US17/214,946 priority Critical patent/US11496660B2/en
Publication of TW202211165A publication Critical patent/TW202211165A/en
Application granted granted Critical
Publication of TWI767484B publication Critical patent/TWI767484B/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/743—Bracketing, i.e. taking a series of images with varying exposure conditions

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Cameras In General (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Traffic Control Systems (AREA)
  • Radar Systems Or Details Thereof (AREA)
  • Measurement Of Optical Distance (AREA)
  • Burglar Alarm Systems (AREA)
  • Air Bags (AREA)

Abstract

A dual sensor imaging system and a depth map calculation method thereof are provided. The dual sensor imaging system includes at least one color sensor, at least one infrared ray (IR) sensor, a storage device and a processor. The processor is configured to load and execute a computer program stored in the storage device to control the color sensor and the IR sensor to respectively capture multiple color images and multiple IR images using multiple exposure conditions under an imaging scene, adaptively select a combination of the color image and the IR image that are comparable to each other from the captured color images and IR images, and calculate a depth map of the imaging scene by using the selected color image and IR image.

Description

雙感測器攝像系統及其深度圖計算方法Dual-sensor camera system and its depth map calculation method

本發明是有關於一種攝像系統及方法，且特別是有關於一種雙感測器攝像系統及其深度圖計算方法。The present invention relates to a camera system and method, and more particularly, to a dual-sensor camera system and a depth map calculation method thereof.

相機的曝光條件（包括光圈、快門、感光度）會影響所拍攝影像的品質，因此許多相機在拍攝影像的過程中會自動調整曝光條件，以獲得清晰且明亮的影像。然而，在低光源或是背光等高反差的場景中，相機調整曝光條件的結果可能會產生雜訊過高或是部分區域過曝的結果，無法兼顧所有區域的影像品質。The camera's exposure conditions (including aperture, shutter, and sensitivity) affect the quality of the images captured, so many cameras automatically adjust exposure conditions during image capture to obtain clear and bright images. However, in scenes with high contrast such as low light source or backlight, the result of camera adjustment of exposure conditions may result in excessive noise or overexposure in some areas, and the image quality in all areas cannot be balanced.

對此，目前技術有採用一種新的影像感測器架構，其是利用紅外線（IR）感測器高光敏感度的特性，在影像感測器的色彩像素中穿插配置IR像素，以輔助亮度偵測。舉例來說，圖1是習知使用影像感測器擷取影像的示意圖。請參照圖1，習知的影像感測器10中除了配置有紅（R）、綠（G）、藍（B）等顏色像素外，還穿插配置有紅外線（I）像素。藉此，影像感測器10能夠將R、G、B顏色像素所擷取的色彩資訊12與I像素所擷取的亮度資訊14結合，而獲得色彩及亮度適中的影像16。In this regard, the current technology adopts a new image sensor architecture, which utilizes the characteristics of high light sensitivity of infrared (IR) sensors to intersperse and configure IR pixels among the color pixels of the image sensor to assist brightness detection. Measurement. For example, FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. Referring to FIG. 1 , in addition to red (R), green (G), blue (B) and other color pixels, the conventional image sensor 10 is also interspersed with infrared (I) pixels. Thereby, the image sensor 10 can combine the color information 12 captured by the R, G, and B color pixels with the luminance information 14 captured by the I pixel to obtain an image 16 with moderate color and brightness.

然而，在上述單一影像感測器的架構下，影像感測器中每個像素的曝光條件相同，因此只能選擇較適用於顏色像素或紅外線像素的曝光條件來擷取影像，結果仍無法有效地利用兩種像素的特性來改善所擷取影像的影像品質。However, in the above-mentioned single image sensor structure, the exposure conditions of each pixel in the image sensor are the same, so only the exposure conditions that are more suitable for color pixels or infrared pixels can be selected to capture images, and the result is still ineffective. The characteristics of the two pixels are used to improve the image quality of the captured image.

本發明提供一種雙感測器攝像系統及其深度圖計算方法，可精確地算出攝像場景的深度圖。The invention provides a dual-sensor camera system and a depth map calculation method thereof, which can accurately calculate the depth map of a camera scene.

本發明的雙感測器攝像系統包括至少一個色彩感測器、至少一個紅外線感測器、儲存裝置以及耦接所述色彩感測器、紅外光感測器及儲存裝置的處理器。所述處理器經配置以載入並執行儲存在儲存裝置中的電腦程式以：控制色彩感測器及紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像；從多張色彩影像及多張紅外線影像中適應性選擇出彼此可對比的色彩影像及紅外線影像的組合；以及使用所選擇的色彩影像及紅外線影像計算攝像場景的深度圖。The dual-sensor camera system of the present invention includes at least one color sensor, at least one infrared sensor, a storage device, and a processor coupled to the color sensor, the infrared light sensor, and the storage device. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the infrared sensor to respectively capture a plurality of color images and multiple infrared images; adaptively selecting a combination of color images and infrared images that are comparable to each other from the multiple color images and the multiple infrared images; and calculating a depth map of the camera scene using the selected color images and infrared images.

本發明的雙感測器攝像系統的深度圖計算方法，適用於包括至少一個色彩感測器、至少一個紅外線感測器及處理器的雙感測器攝像系統。所述方法包括下列步驟：控制色彩感測器及紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像；從多張色彩影像及多張紅外線影像中適應性選擇出彼此可對比的色彩影像及紅外線影像的組合；以及使用所選擇的色彩影像及紅外線影像計算攝像場景的深度圖。The depth map calculation method of the dual-sensor camera system of the present invention is suitable for a dual-sensor camera system comprising at least one color sensor, at least one infrared sensor and a processor. The method includes the following steps: controlling a color sensor and an infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; A combination of color images and infrared images that are comparable to each other is adaptively selected in the images; and a depth map of the camera scene is calculated using the selected color images and infrared images.

基於上述，本發明的雙感測器攝像系統及其深度圖計算方法，藉由在獨立配置的色彩感測器及紅外線感測器上採用適於當前攝像場景的不同曝光條件來擷取多張影像，並從中選擇出彼此可對比的色彩及紅外線影像來計算攝像場景的深度圖，藉此可精確地算出攝像場景的深度圖。Based on the above, the dual-sensor camera system and the depth map calculation method of the present invention capture multiple images by adopting different exposure conditions suitable for the current camera scene on the independently configured color sensor and infrared sensor. image, and select the color and infrared images that can be compared with each other to calculate the depth map of the shooting scene, so that the depth map of the shooting scene can be accurately calculated.

本發明實施例適用在獨立配置有色彩感測器及紅外線感測器的雙感測器攝像系統。其中，由於色彩感測器及紅外線感測器之間具有像差（parallex），其所擷取的色彩影像及紅外線影像可用以計算攝像場景的深度圖。針對色彩感測器所擷取的色彩影像可能會因為攝像場景中的光線反射、陰影、高反差等因素的影響而有過曝或曝光不足的情況，本發明實施例利用紅外線影像具有較佳的訊噪比（Signal to noise ratio，SNR）且包含較多的攝像場景的紋理細節的優點，使用紅外線影像所提供的紋理資訊來輔助缺陷區域的深度值的計算，從而可獲得精確的攝像場景的深度圖。The embodiments of the present invention are applicable to a dual-sensor camera system independently configured with a color sensor and an infrared sensor. The color image and the infrared image captured by the color sensor and the infrared sensor can be used to calculate the depth map of the camera scene due to the parallax between the color sensor and the infrared sensor. In view of the situation that the color image captured by the color sensor may be overexposed or underexposed due to the influence of light reflection, shadow, high contrast and other factors in the shooting scene, the embodiment of the present invention uses the infrared image to have better The signal-to-noise ratio (SNR) has the advantage of containing more texture details of the camera scene. The texture information provided by the infrared image is used to assist the calculation of the depth value of the defect area, so as to obtain accurate camera scenes. depth map.

圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。請參照圖2，本發明實施例的影像感測器20採用獨立配置色彩感測器22與紅外線（IR）感測器24的雙感測器架構，利用色彩感測器22與紅外線感測器24各自的特性，採用適於當前拍攝場景的多個曝光條件分別擷取多張影像，並從中選擇曝光條件適當的色彩影像22a與紅外線影像24a。在一些實施例中，透過影像融合的方式，可使用紅外線影像24a來補足色彩影像22a中缺乏的紋理細節，從而獲得色彩及紋理細節均佳的場景影像26。而在一些實施例中，則可使用色彩影像22a與紅外線影像24a來計算攝像場景的深度圖，並利用紅外線影像24a所提供的紋理細節來補償色彩影像中缺乏的紋理細節，並輔助計算缺陷區域的深度值。FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. Referring to FIG. 2 , the image sensor 20 of the embodiment of the present invention adopts a dual-sensor structure in which a color sensor 22 and an infrared (IR) sensor 24 are independently configured, and the color sensor 22 and the infrared sensor are used. 24 with their respective characteristics, a plurality of images are respectively captured using a plurality of exposure conditions suitable for the current shooting scene, and a color image 22a and an infrared image 24a with appropriate exposure conditions are selected from among them. In some embodiments, through image fusion, the infrared image 24a can be used to supplement the texture details lacking in the color image 22a, so as to obtain a scene image 26 with good color and texture details. In some embodiments, the color image 22a and the infrared image 24a can be used to calculate the depth map of the camera scene, and the texture details provided by the infrared image 24a can be used to compensate for the lack of texture details in the color image, and assist in the calculation of defect areas depth value.

圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。請參照圖3，本實施例的雙感測器攝像系統30可配置於手機、平板電腦、筆記型電腦、導航裝置、行車紀錄器、數位相機、數位攝影機等電子裝置中，用以提供攝像功能。雙感測器攝像系統30包括至少一個色彩感測器32、至少一個紅外線感測器34、儲存裝置36及處理器38，其功能分述如下：FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 , the dual-sensor camera system 30 of this embodiment can be configured in electronic devices such as mobile phones, tablet computers, notebook computers, navigation devices, driving recorders, digital cameras, digital cameras, etc., to provide a camera function . The dual-sensor camera system 30 includes at least one color sensor 32, at least one infrared sensor 34, a storage device 36 and a processor 38, and its functions are described as follows:

色彩感測器32例如包括電荷耦合元件（Charge Coupled Device，CCD）、互補性氧化金屬半導體（Complementary Metal-Oxide Semiconductor，CMOS）元件或其他種類的感光元件，而可感測光線強度以產生攝像場景的影像。色彩感測器32例如是紅綠藍（RGB）影像感測器，其中包括紅（R）、綠（G）、藍（B）顏色像素，用以擷取攝像場景中的紅光、綠光、藍光等色彩資訊，並將這些色彩資訊合成以生成攝像場景的色彩影像。The color sensor 32 includes, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) device, or other types of photosensitive devices, and can sense light intensity to generate a camera scene image. The color sensor 32 is, for example, a red-green-blue (RGB) image sensor, which includes red (R), green (G), and blue (B) color pixels for capturing red light and green light in the camera scene , blue light and other color information, and combine these color information to generate a color image of the camera scene.

紅外線感測器34例如包括CCD、CMOS元件或其他種類的感光元件，其經由調整感光元件的波長感測範圍，而能夠感測紅外光。紅外線感測器34例如是以上述感光元件作為像素來擷取攝像場景中的紅外光資訊，並將這些紅外光資訊合成以生成攝像場景的紅外線影像。The infrared sensor 34 includes, for example, a CCD, a CMOS element or other types of photosensitive elements, which can sense infrared light by adjusting the wavelength sensing range of the photosensitive element. The infrared sensor 34, for example, uses the above-mentioned photosensitive elements as pixels to capture infrared light information in the imaging scene, and synthesizes the infrared light information to generate an infrared image of the imaging scene.

儲存裝置36例如是任意型式的固定式或可移動式隨機存取記憶體（Random Access Memory，RAM）、唯讀記憶體（Read-Only Memory，ROM）、快閃記憶體（Flash memory）、硬碟或類似元件或上述元件的組合，而用以儲存可由處理器38執行的電腦程式。在一些實施例中，儲存裝置36例如還可儲存由色彩感測器32所擷取的色彩影像及紅外線感測器34所擷取的紅外線影像。The storage device 36 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard drive A disk or similar element, or a combination of the foregoing, for storing computer programs executable by the processor 38 . In some embodiments, the storage device 36 may also store, for example, the color image captured by the color sensor 32 and the infrared image captured by the infrared sensor 34 .

處理器38例如是中央處理單元（Central Processing Unit，CPU），或是其他可程式化之一般用途或特殊用途的微處理器（Microprocessor）、微控制器（Microcontroller）、數位訊號處理器（Digital Signal Processor，DSP）、可程式化控制器、特殊應用積體電路（Application Specific Integrated Circuits，ASIC）、可程式化邏輯裝置（Programmable Logic Device，PLD）或其他類似裝置或這些裝置的組合，本發明不在此限制。在本實施例中，處理器38可從儲存裝置36載入電腦程式，以執行本發明實施例的雙感測器攝像系統的深度圖計算方法。The processor 38 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessors), microcontrollers (Microcontrollers), and digital signal processors (Digital Signal Processors). Processor, DSP), programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, the present invention does not this limit. In this embodiment, the processor 38 can load a computer program from the storage device 36 to execute the depth map calculation method of the dual-sensor camera system according to the embodiment of the present invention.

圖4是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。請同時參照圖3及圖4，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的深度圖計算方法的詳細步驟。FIG. 4 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 4 at the same time, the method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and the following describes the depth map calculation method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S402中，由處理器38控制色彩感測器32及紅外線感測器34採用適用於所識別之攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像。In step S402 , the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for the identified shooting scene.

在一些實施例中，處理器38例如是以標準曝光條件中的曝光時間為基準，控制色彩感測器32及紅外線感測器34擷取曝光時間較短或較長的色彩影像，這些色彩影像彼此的曝光時間的差例如為介於-3至3的曝光值（Exposure Value，EV）中的任意值，在此不設限。舉例來說，若A影像比B影像亮一倍，則可將B影像的EV加1，以此類推，曝光值可以有小數（例如+0.3EV），在此不設限。In some embodiments, the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture color images with shorter or longer exposure times based on the exposure time in standard exposure conditions, for example. The difference between the exposure times of each other is, for example, any value in the exposure value (Exposure Value, EV) between -3 and 3, which is not limited here. For example, if the A image is twice as bright as the B image, the EV of the B image can be increased by 1, and so on, the exposure value can have a decimal (eg +0.3EV), which is not limited here.

在一些實施例中，處理器38例如是控制色彩感測器32及紅外線感測器34中的至少一者採用標準曝光條件來擷取攝像場景的至少一張標準影像，並使用這些標準影像來識別攝像場景。所述標準曝光條件例如包括採用既有測光技術所決定的光圈、快門、感光度等參數，而處理器38則根據在此曝光條件下所擷取之影像的色相（Hue）、明度（Value）、彩度（Chroma）、白平衡等影像參數的強弱或分佈來識別攝像場景，包括攝像場景的位置（室內或室外）、光源（高光源或低光源）、反差（高反差或低反差）、攝像物的種類（物品或人像）或狀態（動態或靜態）等。在其他實施例中，處理器38亦可採用定位方式來識別攝像場景或是直接接收使用者操作來設定攝像場景，在此不設限。In some embodiments, the processor 38 controls at least one of the color sensor 32 and the infrared sensor 34 to capture at least one standard image of the camera scene using standard exposure conditions, and uses these standard images to Identify the camera scene. The standard exposure conditions include, for example, parameters such as aperture, shutter, and sensitivity determined by using the existing light metering technology. , Chroma, white balance and other image parameters to identify the camera scene, including the location of the camera scene (indoor or outdoor), light source (high light source or low light source), contrast (high contrast or low contrast), The type (object or portrait) or state (dynamic or static) of the photographed object, etc. In other embodiments, the processor 38 may also use a positioning method to identify the camera scene or directly receive user operations to set the camera scene, which is not limited herein.

在步驟S404中，由處理器38從多張色彩影像及多張紅外線影像中適應性選擇出彼此可對比的色彩影像及紅外線影像的組合。在一些實施例中，處理器38例如會根據各張色彩影像的顏色細節和各張紅外線影像的紋理細節來選擇彼此可對比的色彩影像及紅外線影像的組合。在一些實施例中，處理器38則會以色彩影像或紅外線影像作為基準，來比較各張色彩影像和紅外線影像的影像直方圖，藉此確定彼此可對比的色彩影像及紅外線影像的組合。In step S404, the processor 38 adaptively selects a combination of color images and infrared images that are comparable to each other from the plurality of color images and the plurality of infrared images. In some embodiments, the processor 38 selects a combination of color images and infrared images that are comparable to each other, for example, according to color details of each color image and texture details of each infrared image. In some embodiments, the processor 38 uses the color image or the infrared image as a reference to compare the image histograms of each color image and the infrared image, thereby determining a combination of color images and infrared images that are comparable to each other.

在步驟S406中，由處理器38使用所選擇的述色彩影像及紅外線影像計算攝像場景的深度圖。在一些實施例中，處理器38例如會擷取所選擇的色彩影像及紅外線影像中特徵強健的多個特徵點，並根據色彩影像及紅外線影像中彼此相對應的特徵點的位置，計算攝像場景的深度圖。In step S406, the processor 38 calculates the depth map of the camera scene using the selected color image and infrared image. In some embodiments, the processor 38 captures, for example, a plurality of feature points with robust features in the selected color image and the infrared image, and calculates the camera scene according to the positions of the feature points corresponding to each other in the color image and the infrared image. depth map.

藉由上述方法，雙感測器攝像系統30可選擇出顏色細節較佳的色彩影像及紋理細節較佳的紅外線影像來計算攝像場景的深度圖，並利用紅外線影像來補償或取代色彩影像中所缺乏的紋理細節來計算深度值，藉此可精確地算出攝像場景的深度圖。Through the above method, the dual-sensor camera system 30 can select a color image with better color detail and an infrared image with better texture detail to calculate the depth map of the camera scene, and use the infrared image to compensate or replace all of the color images. The lack of texture details to calculate the depth value, which can accurately calculate the depth map of the camera scene.

在一些實施例中，處理器38例如會先根據各張色彩影像的顏色細節，選擇其中一張色彩影像作為基準影像，接著辨識基準影像中缺乏紋理細節的至少一個缺陷區域，然後再根據各張紅外線影像中對應於這些缺陷區域的影像的紋理細節，選擇其中一張紅外線影像作為可與基準影像彼此對比的影像，一同用於深度圖的計算。In some embodiments, the processor 38 first selects one of the color images as the reference image according to the color details of each color image, and then identifies at least one defective area lacking texture details in the reference image, and then selects one of the color images as the reference image according to the color details of each The texture details of the images corresponding to these defect areas in the infrared images are selected, and one of the infrared images is selected as an image that can be compared with the reference image, and is used for the calculation of the depth map together.

詳言之，基於色彩感測器32每次只能採用單一曝光條件擷取色彩影像，在攝像場景為低光源或高反差的情況下，每一張色彩影像都可能會出現高雜訊、過曝或曝光不足的區域（即上述的缺陷區域）。此時，處理器38即可利用紅外線感測器34高光敏感度的特性，針對上述的缺陷區域，從先前擷取的多張紅外線影像中，選擇具備該缺陷區域的紋理細節的紅外線影像，而可用以補足色彩影像中缺陷區域的紋理細節。To be more specific, because the color sensor 32 can only use a single exposure condition to capture color images at a time, when the shooting scene is low light source or high contrast, each color image may have high noise and excessive noise. Areas that are exposed or underexposed (i.e. the defective areas described above). At this time, the processor 38 can use the characteristics of the high light sensitivity of the infrared sensor 34 to select the infrared image with the texture details of the defect area from the multiple infrared images captured previously for the above-mentioned defective area, and Can be used to complement texture details in defective areas in color images.

圖5是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。請同時參照圖3及圖5，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的深度圖計算方法的詳細步驟。FIG. 5 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 5 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following is a description of the depth map calculation method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S502中，由處理器38從多張色彩影像中選擇能顯露出攝像場景的顏色細節的色彩影像作為基準影像。In step S502, the processor 38 selects a color image that can reveal the color details of the shooting scene from the plurality of color images as a reference image.

在一些實施例中，處理器38例如是根據各張色彩影像的顏色細節，從多張色彩影像中選擇顏色細節最多的色彩影像作為基準影像。所述顏色細節的多寡例如可由色彩影像中過曝或曝光不足區域的大小來決定。In some embodiments, the processor 38 selects the color image with the most color details from the plurality of color images as the reference image, for example, according to the color details of each color image. The amount of color detail can be determined, for example, by the size of the overexposed or underexposed areas in the color image.

詳細而言，過曝區域像素的顏色趨近白色、曝光不足區域像素的顏色趨近黑色，因此這些區域的顏色細節會較少。因此，若色彩影像中包括較多的這類區域，代表其顏色細節較少，處理器38據此即可判斷出哪一張色彩影像的顏色細節最多，而用以作為基準影像。在其他實施例中，處理器38也可依據各張色彩影像的對比度、飽和度或其他影像參數來分辨其顏色細節的多寡，在此不設限。In detail, the color of the pixels in the overexposed areas tends to be white, and the color of the pixels in the underexposed areas tends to be black, so the color details of these areas will be less. Therefore, if the color image includes more such regions, it means that the color details are less, and the processor 38 can determine which color image has the most color details accordingly, and use it as the reference image. In other embodiments, the processor 38 can also distinguish the color details of each color image according to the contrast, saturation or other image parameters, which is not limited herein.

在步驟S504中，由處理器38辨識基準影像中缺乏紋理細節的至少一個缺陷區域。所述的缺陷區域例如是上述的過曝區域或曝光不足區域，或是在低光源下所擷取的具較高雜訊的區域，在此不設限。In step S504, the processor 38 identifies at least one defect region in the reference image that lacks texture details. The defect area is, for example, the above-mentioned overexposed area or underexposed area, or an area with higher noise captured under low light source, which is not limited herein.

在步驟S506中，由處理器38根據各張紅外線影像中對應於所述缺陷區域的影像的紋理細節，選擇其中一張紅外線影像，而用以作為與基準影像彼此對比的組合。In step S506 , the processor 38 selects one of the infrared images according to the texture details of the images corresponding to the defective area in each of the infrared images, and uses it as a combination to be compared with the reference image.

在一些實施例中，處理器38例如是選擇對應於所述缺陷區域的影像的紋理細節最多的紅外線影像作為與基準影像彼此對比的組合。其中，處理器38例如是依據各張紅外線影像的對比度或其他影像參數來分辨其紋理細節的多寡，在此不設限。In some embodiments, the processor 38 selects, for example, the infrared image corresponding to the image of the defective area with the most texture detail as the combination to be compared with the reference image. Wherein, the processor 38, for example, determines the amount of texture details of each infrared image according to the contrast ratio or other image parameters, which is not limited herein.

在步驟S508中，由處理器38執行特徵擷取演算法，從基準影像及所選擇的紅外線影像中擷取特徵強健的多個特徵點。In step S508, the processor 38 executes a feature extraction algorithm to extract multiple feature points with robust features from the reference image and the selected infrared image.

在一些實施例中，所述的特徵擷取演算法例如是哈里斯邊角偵測（Harris corner detector）、海森仿射區域偵測（Hessian-affine region detector）、最大穩定極值區域（Maximally Stable Extremal Regions，MSER）、尺度不變特徵變換（Scale invariant feature transform，SIFT）或加速穩健特徵（Speeded up robust features，SURF），所述特徵點例如是影像中的邊緣或角落像素，在此不設限。在一些實施例中，處理器38還可根據所擷取特徵之間的對應關係將色彩影像及紅外線影像對齊。In some embodiments, the feature extraction algorithm is, for example, Harris corner detector, Hessian-affine region detector, Maximally stable extremum region Stable Extremal Regions, MSER), Scale invariant feature transform (SIFT) or Speeded up robust features (SURF), the feature points are, for example, edge or corner pixels in the image, not here set limits. In some embodiments, the processor 38 may also align the color image and the infrared image according to the correspondence between the captured features.

在步驟S510中，由處理器38根據基準影像及紅外線影像中彼此相對應的特徵點的位置，計算攝像場景的深度圖。In step S510, the processor 38 calculates the depth map of the imaging scene according to the positions of the feature points corresponding to each other in the reference image and the infrared image.

在一些實施例中，處理器38例如是直接計算基準影像及紅外線影像中相對應的各個像素的像差，並依據雙感測器攝像系統30的色彩感測器及紅外線感測器34拍攝影像時的焦距、兩個感測器的間距以及各個像素的像差，估測各個像素的深度。其中，處理器38例如是依據各個像素在基準影像及紅外線影像中的位置，計算各個像素在基準影像及紅外線影像之間的位移，以作為像差。In some embodiments, the processor 38 directly calculates the aberration of each pixel corresponding to the reference image and the infrared image, and captures the image according to the color sensor and the infrared sensor 34 of the dual-sensor camera system 30 . The focal length, the distance between the two sensors, and the aberration of each pixel are used to estimate the depth of each pixel. The processor 38, for example, calculates the displacement of each pixel between the reference image and the infrared image according to the position of each pixel in the reference image and the infrared image, as the aberration.

詳細而言，雙感測器攝像系統30所拍攝的基準影像及紅外線影像中相對應像素的像差是由焦距（決定影像大小）、感測器間距（決定影像重疊範圍）以及該像素對應物件與感測器間的距離（即深度值，決定影像中物件的大小）來決定，其中存在著某種比例關係，而記載此比例關係的關係表可藉由在雙感測器攝像系統30出廠前預先測試而得。因此，當使用者使用雙感測器攝像系統30拍攝影像，而處理器38在計算影像中各個像素的像差時，即可利用預先建立的關係表查詢而獲得各個像素的深度值。In detail, the aberration of the corresponding pixels in the reference image and the infrared image captured by the dual-sensor camera system 30 is determined by the focal length (determining the size of the image), the sensor distance (determining the overlapping range of the images), and the object corresponding to the pixel The distance between the sensor and the sensor (ie the depth value, which determines the size of the object in the image) is determined, there is a certain proportional relationship, and the relationship table recording this proportional relationship can be used in the dual-sensor camera system 30. pre-tested. Therefore, when the user uses the dual-sensor camera system 30 to capture an image, and the processor 38 calculates the aberration of each pixel in the image, the depth value of each pixel can be obtained by querying the pre-established relationship table.

藉由上述方法，雙感測器攝像系統30即可利用色彩影像及紅外線影像中相對應像素的位置關係來計算各個像素的深度值，從而獲得精確的攝像場景的深度圖。Through the above method, the dual-sensor camera system 30 can calculate the depth value of each pixel using the positional relationship of the corresponding pixels in the color image and the infrared image, thereby obtaining an accurate depth map of the camera scene.

舉例來說，圖6是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的範例。請參照圖6，本實施例是通過上述圖5的深度圖計算方法，選擇出顏色細節最多的色彩影像62作為基準影像，並針對色彩影像62中缺乏紋理細節的缺陷區域（例如人臉區域62a），從採用不同曝光條件擷取的多張紅外線影像中選擇出該缺陷區域的紋理細節最多的紅外線影像64，用以與色彩影像62進行對比，從而計算出精確的攝像場景的深度圖66。For example, FIG. 6 is an example of a depth map calculation method of a dual-sensor camera system according to an embodiment of the present invention. Referring to FIG. 6 , in this embodiment, the color image 62 with the most color details is selected as the reference image by the depth map calculation method in the above-mentioned FIG. ), and selects the infrared image 64 with the most texture detail in the defect area from the multiple infrared images captured under different exposure conditions, and compares it with the color image 62 to calculate an accurate depth map 66 of the camera scene.

在一些實施例中，處理器38例如會在使用者啟動即時顯示（live view）模式時，控制色彩感測器32拍攝多張色彩影像，以執行自動對焦（auto focus），藉此獲得所拍攝物體的焦距，並根據此焦距來決定可顯露出物體顏色細節最多的色彩影像。In some embodiments, the processor 38 controls the color sensor 32 to capture a plurality of color images, for example, when the user activates the live view mode, so as to perform auto focus, thereby obtaining the captured images. The focal length of the object, and based on this focal length, the color image that can reveal the most color details of the object is determined.

在即時顯示模式中，處理器38例如會以此可顯露出物體顏色細節最多的色彩影像對應的曝光時間為基準，控制色彩感測器32以較此曝光時間為長或短的多個曝光時間拍攝多張色彩影像，藉此監測攝像場景的環境變化。類似地，處理器38也可以可顯露出物體紋理細節最多的紅外線影像對應的曝光時間為基準，控制紅外線感測器34以較此曝光時間為長或短的多個曝光時間拍攝多張紅外線影像。最後，處理器38可從這些由色彩感測器32及紅外線感測器34所拍攝的影像中，選擇彼此最能夠對比的色彩影像及紅外線影像的組合，而用以計算攝像場景的深度圖。In the real-time display mode, the processor 38 uses, for example, the exposure time corresponding to the color image that can reveal the most color details of the object as a reference, and controls the color sensor 32 to use a plurality of exposure times that are longer or shorter than the exposure time. Capture multiple color images to monitor environmental changes in the camera scene. Similarly, the processor 38 can also control the infrared sensor 34 to capture multiple infrared images with multiple exposure times longer or shorter than the exposure time corresponding to the infrared image that can reveal the most texture details of the object as a reference. . Finally, the processor 38 can select a combination of the color image and the infrared image that is most comparable to each other from the images captured by the color sensor 32 and the infrared sensor 34 to calculate the depth map of the camera scene.

舉例來說，在一些實施例中，處理器38會計算這些色彩影像及紅外線影像中每張影像的影像直方圖，並選擇以色彩影像或紅外線影像作為基準，來比較各張色彩影像和紅外線影像的影像直方圖，藉此確定彼此最能夠對比的色彩影像及紅外線影像的組合，並用以計算攝像場景的深度圖。For example, in some embodiments, the processor 38 calculates an image histogram for each of the color images and the infrared image, and selects the color image or the infrared image as a benchmark to compare the color and infrared images. The image histogram is used to determine the combination of color image and infrared image that can best compare with each other, and used to calculate the depth map of the camera scene.

詳細而言，在一些實施例中，處理器38例如是選擇色彩影像其中之一（例如是選擇可顯露出物體顏色細節最多的色彩影像）作為基準影像，並選擇紅外線影像其中之一（例如是選擇可顯露出物體紋理細節最多的色彩影像）來與基準影像比較，而依據這些影像的影像直方圖判斷所選擇的紅外線影像的亮度是否高於基準影像的亮度。其中，若判斷結果為是，則處理器38會從紅外線感測器34預先擷取的多張紅外線影像中選擇曝光時間較所選擇的紅外線影像的曝光時間短的紅外線影像，或控制紅外線感測器34採用較所選擇的紅外線影像的曝光時間短的曝光時間擷取紅外線影像，用以作為與基準影像彼此對比的組合。反之，若判斷結果為否，則處理器38會從紅外線感測器34預先擷取的多張紅外線影像中選擇曝光時間較所選擇的紅外線影像的曝光時間長的紅外線影像，或控制紅外線感測器34採用較所選擇的紅外線影像的曝光時間長的曝光時間擷取紅外線影像，用以作為與基準影像彼此對比的組合。Specifically, in some embodiments, the processor 38 selects one of the color images (eg, selects the color image that can reveal the most color details of the object) as the reference image, and selects one of the infrared images (eg, is Select the color image that can reveal the most texture details of the object) to compare with the reference image, and judge whether the brightness of the selected infrared image is higher than the brightness of the reference image according to the image histogram of these images. Wherein, if the determination result is yes, the processor 38 selects an infrared image with a shorter exposure time than the selected infrared image from a plurality of infrared images pre-captured by the infrared sensor 34, or controls the infrared sensor The device 34 uses an exposure time shorter than the exposure time of the selected infrared image to capture the infrared image, which is used as a combination to be compared with the reference image. On the other hand, if the determination result is no, the processor 38 selects an infrared image with a longer exposure time than the selected infrared image from a plurality of infrared images pre-captured by the infrared sensor 34, or controls the infrared sensing The device 34 uses an exposure time longer than the exposure time of the selected infrared image to capture the infrared image, which is used as a combination to be compared with the reference image.

另一方面，在一些實施例中，處理器38例如是選擇紅外線影像其中之一（例如是選擇可顯露出物體紋理細節最多的色彩影像）作為基準影像，並選擇色彩影像其中之一（例如是選擇可顯露出物體顏色細節最多的色彩影像）來與基準影像比較，依據這些影像的影像直方圖，判斷所選擇的色彩影像的亮度是否高於基準影像的亮度。其中，若判斷結果為是，則處理器38會從色彩感測器32預先擷取的多張色彩影像中選擇曝光時間較所選擇的色彩影像的曝光時間短的色彩影像，或控制色彩感測器32採用較所選擇的色彩影像的曝光時間短的曝光時間擷取色彩影像，用以作為與基準影像彼此對比的組合。反之，若判斷結果為否，則處理器38會從色彩感測器32預先擷取的多張色彩影像中選擇曝光時間較所選擇的色彩影像的曝光時間長的色彩影像，或控制色彩感測器32採用較所選擇的色彩影像的曝光時間長的曝光時間擷取色彩影像，用以作為與基準影像彼此對比的組合。On the other hand, in some embodiments, the processor 38 selects one of the infrared images (eg, selects a color image that can reveal the most texture details of the object) as the reference image, and selects one of the color images (eg, is Select the color image that can reveal the most color details of the object) to compare with the reference image, and determine whether the brightness of the selected color image is higher than the brightness of the reference image according to the image histogram of these images. Wherein, if the determination result is yes, the processor 38 selects a color image with a shorter exposure time than the exposure time of the selected color image from the plurality of color images pre-captured by the color sensor 32, or controls the color sensing The controller 32 uses an exposure time shorter than the exposure time of the selected color image to capture the color image, which is used as a combination to be compared with the reference image. Conversely, if the determination result is no, the processor 38 selects a color image with a longer exposure time than the selected color image from the plurality of color images pre-captured by the color sensor 32, or controls the color sensing The controller 32 uses an exposure time longer than the exposure time of the selected color image to capture the color image, which is used as a combination to be compared with the reference image.

藉由上述方法，雙感測器攝像系統30即可從多張色彩影像及紅外線影像中適應性選擇出彼此最能夠對比的色彩影像及紅外線影像的組合，並用以算出精確的攝像場景的深度圖。Through the above method, the dual-sensor camera system 30 can adaptively select a combination of color images and infrared images that are most comparable to each other from a plurality of color images and infrared images, and use it to calculate an accurate depth map of the camera scene. .

在一些實施例中，即便是選擇彼此最能夠對比的色彩影像及紅外線影像的組合來計算攝像場景的深度圖，所選擇的色彩影像仍有可能會因為反射或色彩感測器32的動態範圍不足等因素，而具有許多缺乏顏色及/或紋理細節不足的缺陷區域，此即所謂的遮擋（occlusion）。在此情況下，可使用由紅外線影像提供的紋理細節作為參考依據，而從遮擋周圍像素的深度值來估測該遮擋的深度值。In some embodiments, even if the combination of the most comparable color image and infrared image is selected to calculate the depth map of the camera scene, the selected color image may still be due to reflection or insufficient dynamic range of the color sensor 32 and other factors, there are many defective areas lacking color and/or texture details, which is called occlusion. In this case, the texture details provided by the infrared image can be used as a reference to estimate the depth value of the occlusion from the depth values of the surrounding pixels of the occlusion.

詳細而言，圖7是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。請同時參照圖3及圖7，本實施例的方法適用於上述的雙感測器攝像系統30，並額外在雙感測器攝像系統30中配置如紅外線發光二極體（Light emitting diode，LED）等紅外線投射器（IR projector）（未繪示），用以加強所擷取的紅外線影像的紋理細節。以下即搭配雙感測器攝像系統30的各項元件說明本實施例的深度圖計算方法的詳細步驟。In detail, FIG. 7 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 7 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 , and additionally, an infrared light emitting diode (LED) is configured in the dual-sensor camera system 30 . ) and other IR projectors (not shown) to enhance the texture details of the captured infrared images. The following describes the detailed steps of the depth map calculation method of the present embodiment in conjunction with various elements of the dual-sensor camera system 30 .

在步驟S702中，由處理器38偵測所選擇的色彩影像中缺乏顏色細節或紋理細節的至少一個遮擋，並在步驟S704中，判斷是否偵測到遮擋。In step S702, the processor 38 detects at least one occlusion lacking color details or texture details in the selected color image, and in step S704, it is determined whether an occlusion is detected.

若在步驟S704中有偵測到遮擋，則在步驟S706中，處理器38會控制紅外線投射器投射紅外線至攝像場景，並控制紅外線感測器34擷取攝像場景的紅外線影像。其中，藉由投射紅外線至攝像場景，可增強紅外線感測器34所擷取的攝像場景中暗部區域的紋理細節，而用以輔助後續深度圖的計算。If blocking is detected in step S704, in step S706, the processor 38 controls the infrared projector to project infrared rays to the camera scene, and controls the infrared sensor 34 to capture the infrared image of the camera scene. Among them, by projecting infrared rays to the camera scene, the texture details of the dark area in the camera scene captured by the infrared sensor 34 can be enhanced, so as to assist the calculation of the subsequent depth map.

在步驟S708中，處理器38會根據紅外線感測器34所擷取的紅外線影像所提供的各個遮擋周圍的紋理細節，由各個遮擋周圍的多個像素的深度值決定遮擋的深度值。詳細而言，由於紅外線影像可提供遮擋周圍像素的精確的紋理細節，因此可利用與遮擋具有同質性（homogeneity）的周圍像素的深度值，來填補深度圖中空洞的深度值，使得深度圖中的空洞得以經由紅外線影像的輔助而填補正確的深度值。In step S708 , the processor 38 determines the depth value of the occlusion from the depth values of the pixels around each occlusion according to the texture details around each occlusion provided by the infrared image captured by the infrared sensor 34 . In detail, since the infrared image can provide accurate texture details of the occluded surrounding pixels, the depth values of the surrounding pixels with homogeneity with the occlusion can be used to fill in the depth values of the holes in the depth map, so that the depth map The holes are filled with the correct depth values with the aid of infrared imagery.

另一方面，若在步驟S704中沒有偵測到遮擋，則在步驟S710中，處理器38會根據基準影像及紅外線影像中彼此相對應的特徵點的位置，計算攝像場景的深度圖。此步驟與前述實施例的步驟S510相同或相似，故其詳細內容在此不再贅述。On the other hand, if no occlusion is detected in step S704, then in step S710, the processor 38 calculates the depth map of the camera scene according to the positions of the corresponding feature points in the reference image and the infrared image. This step is the same as or similar to step S510 in the foregoing embodiment, so the detailed content thereof will not be repeated here.

藉由上述方法，雙感測器攝像系統30可有效地填補所計算的深度圖中的空洞，從而獲得完整且精確的攝像場景的深度圖。Through the above method, the dual-sensor camera system 30 can effectively fill the holes in the calculated depth map, thereby obtaining a complete and accurate depth map of the camera scene.

綜上所述，本發明的雙感測器攝像系統及其深度圖計算方法藉由獨立配置色彩感測器與紅外線感測器，並採用適於當前拍攝場景的多個曝光條件分別擷取多張影像，從中選擇彼此可對比的色彩影像及紅外線影像來進行深度圖的計算，藉此可精確地算出各種攝像場景的深度圖。而藉由使用紅外線影像所提供的紋理細節來輔助計算深度圖中空洞的深度值，藉此可生成完整的攝像場景的深度圖。To sum up, the dual-sensor camera system and the depth map calculation method of the present invention configure the color sensor and the infrared sensor independently, and use multiple exposure conditions suitable for the current shooting scene to capture multiple A color image and an infrared image that are comparable to each other are selected to calculate the depth map, so that the depth map of various camera scenes can be accurately calculated. By using the texture details provided by the infrared image to assist in calculating the depth value of the holes in the depth map, a complete depth map of the camera scene can be generated.

10、20:影像感測器 12:色彩資訊 14:亮度資訊 16:影像 22:色彩感測器 22a、62:色彩影像 24:紅外線感測器 24a、64:紅外線影像 26:場景影像 30:雙感測器攝像系統 32:色彩感測器 34:紅外線感測器 36:儲存裝置 38:處理器 62a:人臉區域 66:深度圖 R、G、B、I:像素 S402~S406、S502~S510、S702~S710:步驟10, 20: Image sensor 12: Color Information 14: Brightness information 16: Video 22: Color sensor 22a, 62: Color image 24: Infrared sensor 24a, 64: Infrared imagery 26: Scene image 30: Dual-sensor camera system 32: Color Sensor 34: Infrared sensor 36: Storage Device 38: Processor 62a: face area 66: Depth Map R, G, B, I: pixels S402~S406, S502~S510, S702~S710: Steps

圖1是習知使用影像感測器擷取影像的示意圖。 圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。 圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。 圖4是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。 圖5是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。 圖6是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的範例。 圖7是依照本發明一實施例所繪示的雙感測器攝像系統的深度圖計算方法的流程圖。FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. FIG. 4 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. FIG. 5 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention. FIG. 6 is an example of a depth map calculation method of a dual-sensor camera system according to an embodiment of the present invention. 7 is a flowchart of a method for calculating a depth map of a dual-sensor camera system according to an embodiment of the present invention.

S402~S406:步驟S402~S406: Steps

Claims (20)

一種雙感測器攝像系統，包括： 至少一色彩感測器； 至少一紅外線感測器； 儲存裝置，儲存電腦程式；以及 處理器，耦接所述至少一色彩感測器、所述至少一紅外光感測器及所述儲存裝置，經配置以載入並執行所述電腦程式以： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像； 從所述多張色彩影像及所述多張紅外線影像中適應性選擇出彼此可對比的所述色彩影像及所述紅外線影像的組合；以及 使用所選擇的所述色彩影像及所述紅外線影像計算所述攝像場景的深度圖。A dual-sensor camera system comprising: at least one color sensor; at least one infrared sensor; storage devices to store computer programs; and A processor, coupled to the at least one color sensor, the at least one infrared light sensor, and the storage device, is configured to load and execute the computer program to: controlling the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a photographic scene; Adaptively selecting a combination of the color image and the infrared image that are comparable to each other from the plurality of color images and the plurality of infrared images; and A depth map of the camera scene is calculated using the selected color image and the infrared image. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 從所述多張色彩影像中選擇能顯露出所述攝像場景的顏色細節的色彩影像作為基準影像； 辨識所述基準影像中缺乏紋理細節的至少一缺陷區域；以及 從所述多張紅外線影像中選擇能顯露出所述缺陷區域的所述紋理細節的紅外線影像，用以作為與所述基準影像彼此對比的組合。The dual-sensor camera system of claim 1, wherein the processor comprises: selecting, from the plurality of color images, a color image that can reveal the color details of the camera scene as a reference image; identifying at least one defect region in the reference image that lacks texture detail; and An infrared image capable of revealing the texture details of the defect region is selected from the plurality of infrared images as a combination to be compared with the reference image. 如請求項2所述的雙感測器攝像系統，其中所述處理器更包括： 利用所述色彩影像執行自動對焦，以取得所述攝像場景中所拍攝物體的焦距，並根據所述焦距決定可顯露所述物體的所述顏色細節最多的色彩影像作為所述基準影像。The dual-sensor camera system of claim 2, wherein the processor further comprises: Auto-focus is performed by using the color image to obtain the focal length of the photographed object in the shooting scene, and according to the focal length, the color image that can reveal the most color details of the object is determined as the reference image. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 計算各所述色彩影像及各所述紅外線影像的影像直方圖；以及 選擇所述色彩影像其中之一或所述紅外線影像其中之一作為基準，比較各所述色彩影像及各所述紅外線影像的所述影像直方圖，以確定彼此可對比的所述色彩影像及所述紅外線影像的組合。The dual-sensor camera system of claim 1, wherein the processor comprises: calculating an image histogram for each of the color images and each of the infrared images; and selecting one of the color images or one of the infrared images as a reference, and comparing the image histograms of each of the color images and each of the infrared images to determine the color images and all the infrared images that are comparable to each other combination of infrared images. 如請求項4所述的雙感測器攝像系統，其中所述處理器包括： 選擇所述色彩影像其中之一作為基準影像，並選擇所述紅外線影像其中之一與所述基準影像比較，而依據所述影像直方圖判斷所選擇的所述紅外線影像的亮度是否高於所述基準影像的亮度； 若是，從所擷取的所述多張紅外線影像中選擇曝光時間較所選擇的所述紅外線影像的曝光時間短的紅外線影像，或控制所述至少一紅外線感測器採用較所選擇的所述紅外線影像的曝光時間短的曝光時間擷取紅外線影像，用以作為與所述基準影像彼此對比的組合；以及 若否，從所擷取的所述多張紅外線影像中選擇曝光時間較所選擇的所述紅外線影像的曝光時間長的紅外線影像，或控制所述至少一紅外線感測器採用較所選擇的所述紅外線影像的曝光時間長的曝光時間擷取紅外線影像，用以作為與所述基準影像彼此對比的組合。The dual-sensor camera system of claim 4, wherein the processor comprises: Selecting one of the color images as a reference image, selecting one of the infrared images to compare with the reference image, and determining whether the brightness of the selected infrared image is higher than the brightness of the selected infrared image according to the image histogram The brightness of the reference image; If so, select an infrared image with a shorter exposure time than the selected infrared image from the captured infrared images, or control the at least one infrared sensor to use the selected infrared image. capturing an infrared image with a short exposure time of the infrared image as a combination for comparison with the reference image; and If not, select an infrared image with a longer exposure time than the exposure time of the selected infrared image from the captured infrared images, or control the at least one infrared sensor to use a longer exposure time than the selected infrared image. The infrared image is captured with a long exposure time of the infrared image, and is used as a combination with the reference image to be compared with each other. 如請求項4所述的雙感測器攝像系統，其中所述處理器包括： 選擇所述紅外線影像其中之一作為基準影像，並選擇所述色彩影像其中之一與所述基準影像比較，而依據所述影像直方圖，判斷所選擇的所述色彩影像的亮度是否高於所述基準影像的亮度； 若是，從所擷取的所述多張色彩影像中選擇曝光時間較所選擇的所述色彩影像的曝光時間短的色彩影像，或控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間短的曝光時間擷取色彩影像，用以作為與所述基準影像彼此對比的組合；以及 若否，從所擷取的所述多張色彩影像中選擇曝光時間較所選擇的所述色彩影像的曝光時間長的色彩影像，或控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間長的曝光時間擷取色彩影像，用以作為與所述基準影像彼此對比的組合。The dual-sensor camera system of claim 4, wherein the processor comprises: One of the infrared images is selected as a reference image, and one of the color images is selected to be compared with the reference image, and according to the image histogram, it is determined whether the brightness of the selected color image is higher than the selected color image. the brightness of the reference image; If yes, select a color image with a shorter exposure time than the exposure time of the selected color image from the captured color images, or control the at least one color sensor to use the selected color image The exposure time of the color image is short to capture the color image as a combination with the reference image for comparison with each other; and If not, select a color image with a longer exposure time than the exposure time of the selected color image from the plurality of captured color images, or control the at least one color sensor to use a color image with a longer exposure time than the selected color image. The color image is captured with a long exposure time of the color image and used as a combination with the reference image to be compared with each other. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 偵測所選擇的所述色彩影像中缺乏顏色細節或紋理細節的至少一遮擋（occlusion）；以及 執行空洞填補（hole filling）演算法，以根據所選擇的所述紅外線影像所提供的各所述遮擋周圍的紋理細節，由各所述遮擋周圍多個像素的深度值決定所述遮擋的深度值。The dual-sensor camera system of claim 1, wherein the processor comprises: detecting at least one occlusion lacking color detail or texture detail in the selected color image; and Execute a hole filling algorithm to determine the depth value of the occlusion from the depth values of a plurality of pixels around the occlusion according to the texture details around the occlusion provided by the selected infrared image . 如請求項7所述的雙感測器攝像系統，其中所述雙感測器攝像系統更包括紅外線投射器（IR projector），所述處理器更包括： 在偵測到所述遮擋時，控制所述紅外線投射器投射紅外線至所述攝像場景，並控制所述至少一紅外線感測器擷取所述攝像場景的紅外線影像；以及 根據所擷取的所述紅外線影像所提供的各所述遮擋周圍的紋理細節，由各所述遮擋周圍的多個像素的深度值決定所述遮擋的深度值。The dual-sensor camera system of claim 7, wherein the dual-sensor camera system further comprises an infrared projector (IR projector), and the processor further comprises: When the blocking is detected, controlling the infrared projector to project infrared rays to the camera scene, and controlling the at least one infrared sensor to capture infrared images of the camera scene; and According to the texture details around each of the occlusions provided by the captured infrared image, the depth values of the occlusions are determined by the depth values of a plurality of pixels around each of the occlusions. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 擷取所選擇的所述色彩影像及所述紅外線影像中特徵強健的多個特徵點；以及 根據所述色彩影像及所述紅外線影像中彼此相對應的所述特徵點的位置，計算所述攝像場景的所述深度圖。The dual-sensor camera system of claim 1, wherein the processor comprises: capturing a plurality of feature points with robust features in the selected color image and the infrared image; and The depth map of the imaging scene is calculated according to the positions of the feature points corresponding to each other in the color image and the infrared image. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器中的至少一者採用標準曝光條件擷取所述攝像場景的至少一標準影像，並使用所述至少一標準影像識別所述攝像場景。The dual-sensor camera system of claim 1, wherein the processor comprises: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture at least one standard image of the camera scene using standard exposure conditions, and using the at least one standard image to identify the camera scene. 一種雙感測器攝像系統的深度圖計算方法，所述雙感測器攝像系統包括至少一色彩感測器、至少一紅外線感測器及處理器，所述方法包括下列步驟： 由所述處理器控制所述至少一色彩感測器及所述至少一紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像； 由所述處理器從所述多張色彩影像及所述多張紅外線影像中適應性選擇出彼此可對比的所述色彩影像及所述紅外線影像的組合；以及 由所述處理器使用所選擇的所述色彩影像及所述紅外線影像計算所述攝像場景的深度圖。A depth map calculation method for a dual-sensor camera system, the dual-sensor camera system includes at least one color sensor, at least one infrared sensor, and a processor, and the method includes the following steps: The processor controls the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; adaptively selecting, by the processor, a combination of the color image and the infrared image that are comparable to each other from the plurality of color images and the plurality of infrared images; and A depth map of the camera scene is calculated by the processor using the selected color image and the infrared image. 如請求項11所述的方法，其中從所述多張色彩影像及所述多張紅外線影像中適應性選擇出彼此可對比的所述色彩影像及所述紅外線影像的組合的步驟包括： 從所述多張色彩影像中選擇能顯露出所述攝像場景的顏色細節的色彩影像作為基準影像； 辨識所述基準影像中缺乏紋理細節的至少一缺陷區域；以及 從所述多張紅外線影像中選擇能顯露出所述缺陷區域的所述紋理細節的紅外線影像。The method of claim 11, wherein the step of adaptively selecting a combination of the color images and the infrared images that are comparable to each other from the plurality of color images and the plurality of infrared images comprises: selecting, from the plurality of color images, a color image that can reveal the color details of the camera scene as a reference image; identifying at least one defect region in the reference image that lacks texture detail; and An infrared image that reveals the texture detail of the defect area is selected from the plurality of infrared images. 如請求項12所述的方法，其中從所述多張色彩影像中選擇能顯露出所述攝像場景的顏色細節的色彩影像作為基準影像的步驟包括： 利用所述色彩影像執行自動對焦，以取得所述攝像場景中所拍攝物體的焦距，並根據所述焦距決定可顯露所述物體的所述顏色細節最多的色彩影像作為所述基準影像。The method of claim 12, wherein the step of selecting a color image from the plurality of color images that can reveal the color details of the camera scene as a reference image comprises: Auto-focus is performed by using the color image to obtain the focal length of the photographed object in the shooting scene, and according to the focal length, the color image that can reveal the most color details of the object is determined as the reference image. 如請求項11所述的方法，其中從所述多張色彩影像及所述多張紅外線影像中適應性選擇出彼此可對比的所述色彩影像及所述紅外線影像的組合的步驟包括： 計算各所述色彩影像及各所述紅外線影像的影像直方圖；以及 選擇所述色彩影像其中之一或所述紅外線影像其中之一作為基準，比較各所述色彩影像及各所述紅外線影像的所述影像直方圖，以確定彼此可對比的所述色彩影像及所述紅外線影像的組合。The method of claim 11, wherein the step of adaptively selecting a combination of the color images and the infrared images that are comparable to each other from the plurality of color images and the plurality of infrared images comprises: calculating an image histogram for each of the color images and each of the infrared images; and selecting one of the color images or one of the infrared images as a reference, and comparing the image histograms of each of the color images and each of the infrared images to determine the color images and all the infrared images that are comparable to each other combination of infrared images. 如請求項14所述的方法，其中比較各所述色彩影像及各所述紅外線影像的所述影像直方圖，以確定彼此可對比的所述色彩影像及所述紅外線影像的組合的步驟包括： 選擇所述色彩影像其中之一作為基準影像，並選擇所述紅外線影像其中之一與所述基準影像比較，而依據所述影像直方圖判斷所選擇的所述紅外線影像的亮度是否高於所述基準影像的亮度； 若是，從所擷取的所述多張紅外線影像中選擇曝光時間較所選擇的所述紅外線影像的曝光時間短的紅外線影像，或控制所述至少一紅外線感測器採用較所選擇的所述紅外線影像的曝光時間短的曝光時間擷取紅外線影像，用以作為與所述基準影像彼此對比的組合；以及 若否，從所擷取的所述多張紅外線影像中選擇曝光時間較所選擇的所述紅外線影像的曝光時間長的紅外線影像，或控制所述至少一紅外線感測器採用較所選擇的所述紅外線影像的曝光時間長的曝光時間擷取紅外線影像，用以作為與所述基準影像彼此對比的組合。The method of claim 14, wherein the step of comparing the image histograms of each of the color images and each of the infrared images to determine combinations of the color images and the infrared images that are comparable to each other comprises: Selecting one of the color images as a reference image, selecting one of the infrared images to compare with the reference image, and determining whether the brightness of the selected infrared image is higher than the brightness of the selected infrared image according to the image histogram The brightness of the reference image; If so, select an infrared image with a shorter exposure time than the selected infrared image from the captured infrared images, or control the at least one infrared sensor to use the selected infrared image. capturing an infrared image with a short exposure time of the infrared image as a combination for comparison with the reference image; and If not, select an infrared image with a longer exposure time than the exposure time of the selected infrared image from the captured infrared images, or control the at least one infrared sensor to use a longer exposure time than the selected infrared image. The infrared image is captured with a long exposure time of the infrared image, and is used as a combination with the reference image to be compared with each other. 如請求項14所述的方法，其中比較各所述色彩影像及各所述紅外線影像的所述影像直方圖，以確定彼此可對比的所述色彩影像及所述紅外線影像的組合的步驟包括： 選擇所述紅外線影像其中之一作為基準影像，並選擇所述色彩影像其中之一與所述基準影像比較，而依據所述影像直方圖，判斷所選擇的所述色彩影像的亮度是否高於所述基準影像的亮度； 若是，從所擷取的所述多張色彩影像中選擇曝光時間較所選擇的所述色彩影像的曝光時間短的色彩影像，或控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間短的曝光時間擷取色彩影像，用以作為與所述基準影像彼此對比的組合；以及 若否，從所擷取的所述多張色彩影像中選擇曝光時間較所選擇的所述色彩影像的曝光時間長的色彩影像，或控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間長的曝光時間擷取色彩影像，用以作為與所述基準影像彼此對比的組合。The method of claim 14, wherein the step of comparing the image histograms of each of the color images and each of the infrared images to determine combinations of the color images and the infrared images that are comparable to each other comprises: One of the infrared images is selected as a reference image, and one of the color images is selected to be compared with the reference image, and according to the image histogram, it is determined whether the brightness of the selected color image is higher than the selected color image. the brightness of the reference image; If yes, select a color image with a shorter exposure time than the exposure time of the selected color image from the captured color images, or control the at least one color sensor to use the selected color image The exposure time of the color image is short to capture the color image as a combination with the reference image for comparison with each other; and If not, select a color image with a longer exposure time than the exposure time of the selected color image from the plurality of captured color images, or control the at least one color sensor to use a color image with a longer exposure time than the selected color image. The color image is captured with a long exposure time of the color image and used as a combination with the reference image to be compared with each other. 如請求項11所述的方法，更包括： 偵測所選擇的所述色彩影像中缺乏顏色細節或紋理細節的至少一遮擋；以及 執行空洞填補演算法，以根據所選擇的所述紅外線影像所提供的各所述遮擋周圍的紋理細節，由各所述遮擋周圍多個像素的深度值決定所述遮擋的深度值。The method according to claim 11, further comprising: detecting at least one occlusion lacking color detail or texture detail in the selected color image; and A hole filling algorithm is executed to determine the depth value of the occlusion from the depth values of a plurality of pixels around each occlusion according to the texture details around each occlusion provided by the selected infrared image. 如請求項17所述的方法，其中所述雙感測器攝像系統更包括紅外線投射器，在偵測所選擇的所述色彩影像中缺乏顏色細節或紋理細節的至少一遮擋的步驟之後，所述方法更包括： 在偵測到所述遮擋時，控制所述紅外線投射器投射紅外線至所述攝像場景，並控制所述至少一紅外線感測器擷取所述攝像場景的紅外線影像；以及 根據所擷取的所述紅外線影像所提供的各所述遮擋周圍的紋理細節，由各所述遮擋周圍多個像素的深度值決定所述遮擋的深度值。The method of claim 17, wherein the dual-sensor camera system further comprises an infrared projector, after the step of detecting at least one occlusion lacking color detail or texture detail in the selected color image, the The method further includes: When the blocking is detected, controlling the infrared projector to project infrared rays to the camera scene, and controlling the at least one infrared sensor to capture infrared images of the camera scene; and According to the texture details around each of the occlusions provided by the captured infrared image, the depth values of the occlusions are determined by the depth values of a plurality of pixels around each of the occlusions. 如請求項11所述的方法，其中使用所選擇的所述色彩影像及所述紅外線影像計算所述攝像場景的深度圖的步驟包括： 擷取所選擇的所述色彩影像及所述紅外線影像中特徵強健的多個特徵點；以及 根據所述色彩影像及所述紅外線影像中彼此相對應的所述特徵點的位置，計算所述攝像場景的所述深度圖。The method of claim 11, wherein the step of calculating a depth map of the camera scene using the selected color image and the infrared image comprises: capturing a plurality of feature points with robust features in the selected color image and the infrared image; and The depth map of the imaging scene is calculated according to the positions of the feature points corresponding to each other in the color image and the infrared image. 如請求項11所述的方法，其中在控制所述至少一色彩感測器及所述至少一紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像的步驟之前，所述方法更包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器中的至少一者採用標準曝光條件擷取所述攝像場景的至少一標準影像，並使用所述至少一標準影像識別所述攝像場景。The method of claim 11, wherein the at least one color sensor and the at least one infrared sensor are controlled to capture a plurality of color images and a plurality of color images respectively using a plurality of exposure conditions suitable for a shooting scene Before the step of infrared imaging, the method further includes: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture at least one standard image of the camera scene using standard exposure conditions, and using the at least one standard image to identify the camera scene.
TW109146922A 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof TWI767484B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/214,946 US11496660B2 (en) 2020-09-04 2021-03-29 Dual sensor imaging system and depth map calculation method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063074477P 2020-09-04 2020-09-04
US63/074,477 2020-09-04

Publications (2)

Publication Number Publication Date
TW202211165A true TW202211165A (en) 2022-03-16
TWI767484B TWI767484B (en) 2022-06-11

Family

ID=80438521

Family Applications (5)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109146831A TWI797528B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof
TW109146764A TWI764484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof
TW109146922A TWI767484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof

Family Applications Before (4)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109146831A TWI797528B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof
TW109146764A TWI764484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof

Country Status (2)

Country Link
CN (5) CN114143418B (en)
TW (5) TWI767468B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116091341B (en) * 2022-12-15 2024-04-02 南京信息工程大学 Exposure difference enhancement method and device for low-light image

Family Cites Families (70)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004246252A (en) * 2003-02-17 2004-09-02 Takenaka Komuten Co Ltd Apparatus and method for collecting image information
JP2005091434A (en) * 2003-09-12 2005-04-07 Noritsu Koki Co Ltd Position adjusting method and image reader with damage compensation function using the same
JP4244018B2 (en) * 2004-03-25 2009-03-25 ノーリツ鋼機株式会社 Defective pixel correction method, program, and defective pixel correction system for implementing the method
JP4341680B2 (en) * 2007-01-22 2009-10-07 セイコーエプソン株式会社 projector
US9307212B2 (en) * 2007-03-05 2016-04-05 Fotonation Limited Tone mapping for low-light video frame enhancement
EP3876510A1 (en) * 2008-05-20 2021-09-08 FotoNation Limited Capturing and processing of images using monolithic camera array with heterogeneous imagers
US8866920B2 (en) * 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
CN101404060B (en) * 2008-11-10 2010-06-30 北京航空航天大学 Human face recognition method based on visible light and near-infrared Gabor information amalgamation
US8749635B2 (en) * 2009-06-03 2014-06-10 Flir Systems, Inc. Infrared camera systems and methods for dual sensor applications
WO2010104490A1 (en) * 2009-03-12 2010-09-16 Hewlett-Packard Development Company, L.P. Depth-sensing camera system
JP5670456B2 (en) * 2009-08-25 2015-02-18 アイピーリンク・リミテッド Reduce noise in color images
US8478123B2 (en) * 2011-01-25 2013-07-02 Aptina Imaging Corporation Imaging devices having arrays of image sensors and lenses with multiple aperture sizes
JP2013115679A (en) * 2011-11-30 2013-06-10 Fujitsu General Ltd Imaging apparatus
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
TW201401186A (en) * 2012-06-25 2014-01-01 Psp Security Co Ltd System and method for identifying human face
US20150245062A1 (en) * 2012-09-25 2015-08-27 Nippon Telegraph And Telephone Corporation Picture encoding method, picture decoding method, picture encoding apparatus, picture decoding apparatus, picture encoding program, picture decoding program and recording medium
KR102070778B1 (en) * 2012-11-23 2020-03-02 엘지전자 주식회사 Rgb-ir sensor with pixels array and apparatus and method for obtaining 3d image using the same
EP2936799B1 (en) * 2012-12-21 2018-10-17 Flir Systems, Inc. Time spaced infrared image enhancement
TWM458748U (en) * 2012-12-26 2013-08-01 Chunghwa Telecom Co Ltd Image type depth information retrieval device
JP6055681B2 (en) * 2013-01-10 2016-12-27 株式会社 日立産業制御ソリューションズ Imaging device
CN104661008B (en) * 2013-11-18 2017-10-31 深圳中兴力维技术有限公司 The treating method and apparatus that color image quality is lifted under low light conditions
CN104021548A (en) * 2014-05-16 2014-09-03 中国科学院西安光学精密机械研究所 Method for acquiring 4D scene information
US9516295B2 (en) * 2014-06-30 2016-12-06 Aquifi, Inc. Systems and methods for multi-channel imaging based on multiple exposure settings
JP6450107B2 (en) * 2014-08-05 2019-01-09 キヤノン株式会社 Image processing apparatus, image processing method, program, and storage medium
JP6597636B2 (en) * 2014-12-10 2019-10-30 ソニー株式会社 Imaging apparatus, imaging method, program, and image processing apparatus
JP6185213B2 (en) * 2015-03-31 2017-08-23 富士フイルム株式会社 Imaging apparatus, image processing method of imaging apparatus, and program
WO2016192437A1 (en) * 2015-06-05 2016-12-08 深圳奥比中光科技有限公司 3d image capturing apparatus and capturing method, and 3d image system
JP2017011634A (en) * 2015-06-26 2017-01-12 キヤノン株式会社 Imaging device, control method for the same and program
CN105049829B (en) * 2015-07-10 2018-12-25 上海图漾信息科技有限公司 Optical filter, imaging sensor, imaging device and 3-D imaging system
CN105069768B (en) * 2015-08-05 2017-12-29 武汉高德红外股份有限公司 A kind of visible images and infrared image fusion processing system and fusion method
US10523855B2 (en) * 2015-09-24 2019-12-31 Intel Corporation Infrared and visible light dual sensor imaging system
TW201721269A (en) * 2015-12-11 2017-06-16 宏碁股份有限公司 Automatic exposure system and auto exposure method thereof
JP2017112401A (en) * 2015-12-14 2017-06-22 ソニー株式会社 Imaging device, apparatus and method for image processing, and program
CN206117865U (en) * 2016-01-16 2017-04-19 上海图漾信息科技有限公司 Range data monitoring device
JP2017163297A (en) * 2016-03-09 2017-09-14 キヤノン株式会社 Imaging apparatus
KR101747603B1 (en) * 2016-05-11 2017-06-16 재단법인 다차원 스마트 아이티 융합시스템 연구단 Color night vision system and operation method thereof
CN106815826A (en) * 2016-12-27 2017-06-09 上海交通大学 Night vision image Color Fusion based on scene Recognition
CN108280807A (en) * 2017-01-05 2018-07-13 浙江舜宇智能光学技术有限公司 Monocular depth image collecting device and system and its image processing method
US11145077B2 (en) * 2017-02-06 2021-10-12 Photonic Sensors & Algorithms, S.L. Device and method for obtaining depth information from a scene
CN108419062B (en) * 2017-02-10 2020-10-02 杭州海康威视数字技术股份有限公司 Image fusion apparatus and image fusion method
CN109474770B (en) * 2017-09-07 2021-09-14 华为技术有限公司 Imaging device and imaging method
CN109712102B (en) * 2017-10-25 2020-11-27 杭州海康威视数字技术股份有限公司 Image fusion method and device and image acquisition equipment
CN107846537B (en) * 2017-11-08 2019-11-26 维沃移动通信有限公司 A kind of CCD camera assembly, image acquiring method and mobile terminal
CN112788249B (en) * 2017-12-20 2022-12-06 杭州海康威视数字技术股份有限公司 Image fusion method and device, electronic equipment and computer readable storage medium
US10748247B2 (en) * 2017-12-26 2020-08-18 Facebook, Inc. Computing high-resolution depth images using machine learning techniques
US10757320B2 (en) * 2017-12-28 2020-08-25 Waymo Llc Multiple operating modes to expand dynamic range
TWI661726B (en) * 2018-01-09 2019-06-01 呂官諭 Image sensor with enhanced image recognition and application
CN110136183B (en) * 2018-02-09 2021-05-18 华为技术有限公司 Image processing method and device and camera device
CN108965654B (en) * 2018-02-11 2020-12-25 浙江宇视科技有限公司 Double-spectrum camera system based on single sensor and image processing method
CN110572583A (en) * 2018-05-18 2019-12-13 杭州海康威视数字技术股份有限公司 method for shooting image and camera
CN108961195B (en) * 2018-06-06 2021-03-23 Oppo广东移动通信有限公司 Image processing method and device, image acquisition device, readable storage medium and computer equipment
JP6574878B2 (en) * 2018-07-19 2019-09-11 キヤノン株式会社 Image processing apparatus, image processing method, imaging apparatus, program, and storage medium
JP7254461B2 (en) * 2018-08-01 2023-04-10 キヤノン株式会社 IMAGING DEVICE, CONTROL METHOD, RECORDING MEDIUM, AND INFORMATION PROCESSING DEVICE
CN109035193A (en) * 2018-08-29 2018-12-18 成都臻识科技发展有限公司 A kind of image processing method and imaging processing system based on binocular solid camera
PL3852350T3 (en) * 2018-09-14 2024-06-10 Zhejiang Uniview Technologies Co., Ltd. Automatic exposure method and apparatus for dual-light image, and dual-light image camera and machine storage medium
JP2020052001A (en) * 2018-09-28 2020-04-02 パナソニックＩｐマネジメント株式会社 Depth acquisition device, depth acquisition method, and program
US11176694B2 (en) * 2018-10-19 2021-11-16 Samsung Electronics Co., Ltd Method and apparatus for active depth sensing and calibration method thereof
CN109636732B (en) * 2018-10-24 2023-06-23 深圳先进技术研究院 Hole repairing method of depth image and image processing device
CN110248105B (en) * 2018-12-10 2020-12-08 浙江大华技术股份有限公司 Image processing method, camera and computer storage medium
US11120536B2 (en) * 2018-12-12 2021-09-14 Samsung Electronics Co., Ltd Apparatus and method for determining image sharpness
WO2020168465A1 (en) * 2019-02-19 2020-08-27 华为技术有限公司 Image processing device and method
US10972649B2 (en) * 2019-02-27 2021-04-06 X Development Llc Infrared and visible imaging system for device identification and tracking
JP7316809B2 (en) * 2019-03-11 2023-07-28 キヤノン株式会社 Image processing device, image processing device control method, system, and program
CN110349117B (en) * 2019-06-28 2023-02-28 重庆工商大学 Infrared image and visible light image fusion method and device and storage medium
CN110706178B (en) * 2019-09-30 2023-01-06 杭州海康威视数字技术股份有限公司 Image fusion device, method, equipment and storage medium
CN111524175A (en) * 2020-04-16 2020-08-11 东莞市东全智能科技有限公司 Depth reconstruction and eye movement tracking method and system for asymmetric multiple cameras
CN111540003A (en) * 2020-04-27 2020-08-14 浙江光珀智能科技有限公司 Depth image generation method and device
CN111586314B (en) * 2020-05-25 2021-09-10 浙江大华技术股份有限公司 Image fusion method and device and computer storage medium
CN111383206B (en) * 2020-06-01 2020-09-29 浙江大华技术股份有限公司 Image processing method and device, electronic equipment and storage medium
IN202021032940A (en) * 2020-07-31 2020-08-28 .Us Priyadarsan

Also Published As

Publication number Publication date
CN114143420A (en) 2022-03-04
TWI767468B (en) 2022-06-11
TWI764484B (en) 2022-05-11
TW202211161A (en) 2022-03-16
TW202211674A (en) 2022-03-16
CN114143418A (en) 2022-03-04
TWI778476B (en) 2022-09-21
CN114143418B (en) 2023-12-01
CN114143443A (en) 2022-03-04
CN114143419B (en) 2023-12-26
TW202211673A (en) 2022-03-16
CN114143421A (en) 2022-03-04
TWI797528B (en) 2023-04-01
CN114143443B (en) 2024-04-05
CN114143421B (en) 2024-04-05
TW202211160A (en) 2022-03-16
TWI767484B (en) 2022-06-11
CN114143419A (en) 2022-03-04
CN114143420B (en) 2024-05-03

Similar Documents

Publication Publication Date Title
US7551797B2 (en) White balance adjustment
TWI498848B (en) Multi-exposure imaging system and white balance method
CN107948538B (en) Imaging method, imaging device, mobile terminal and storage medium
CN107846556B (en) Imaging method, imaging device, mobile terminal and storage medium
US11689822B2 (en) Dual sensor imaging system and privacy protection imaging method thereof
US11838648B2 (en) Image processing device, imaging apparatus, image processing method, and program for determining a condition for high dynamic range processing
JP5782311B2 (en) Imaging apparatus and control method thereof
TW202021339A (en) Infrared pre-flash for camera
US11496694B2 (en) Dual sensor imaging system and imaging method thereof
TWI767484B (en) Dual sensor imaging system and depth map calculation method thereof
US11496660B2 (en) Dual sensor imaging system and depth map calculation method thereof
JP2017152866A (en) Image processing system and image processing method
US11568526B2 (en) Dual sensor imaging system and imaging method thereof
US11405598B2 (en) Image processing apparatus, image processing method, and storage medium
US11418719B2 (en) Dual sensor imaging system and calibration method which includes a color sensor and an infrared ray sensor to perform image alignment and brightness matching
JP2002196389A (en) Photometric device and camera
WO2022061908A1 (en) Method for photographing panoramic image, photographing device, and mobile platform
TWI405458B (en) Digital camera sensitivity adjustment method
TW201643777A (en) Image capture method and image capture apparatus
JP2015118320A (en) Imaging apparatus, image processing method, and program
AU2005203381A1 (en) White balance adjustment
JPH03254581A (en) Exposure control mechanism for camera