TW202211161A - Dual sensor imaging system and privacy protection imaging method thereof - Google Patents

Dual sensor imaging system and privacy protection imaging method thereof Download PDF

Info

Publication number
TW202211161A
TW202211161A TW109146831A TW109146831A TW202211161A TW 202211161 A TW202211161 A TW 202211161A TW 109146831 A TW109146831 A TW 109146831A TW 109146831 A TW109146831 A TW 109146831A TW 202211161 A TW202211161 A TW 202211161A
Authority
TW
Taiwan
Prior art keywords
image
color
infrared
sensor
images
Prior art date
Application number
TW109146831A
Other languages
Chinese (zh)
Other versions
TWI797528B (en
Inventor
彭詩淵
鄭書峻
黃旭鍊
李運錦
賴國銘
Original Assignee
聚晶半導體股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 聚晶半導體股份有限公司 filed Critical 聚晶半導體股份有限公司
Priority to US17/214,937 priority Critical patent/US11689822B2/en
Publication of TW202211161A publication Critical patent/TW202211161A/en
Application granted granted Critical
Publication of TWI797528B publication Critical patent/TWI797528B/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/743—Bracketing, i.e. taking a series of images with varying exposure conditions

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Cameras In General (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Traffic Control Systems (AREA)
  • Radar Systems Or Details Thereof (AREA)
  • Measurement Of Optical Distance (AREA)
  • Burglar Alarm Systems (AREA)
  • Air Bags (AREA)

Abstract

A dual sensor imaging system and a privacy protection imaging method thereof are provided. The system is configured to control at least one color sensor and at least one IR sensor to respectively capture multiple color images and multiple IR images using multiple exposure conditions adapted for an imaging scene, adaptively select a combination of the color image and the IR image that can reveal details of the imaging scene, detect a feature area having features of a target of interest in the color image, and fuse the color image and IR image to generate a fused image having details of the imaging scene, crop an image of the feature area from the fused image and replace the cropped image with an image not belonging to the IR image, so as to generate a scene image.

Description

雙感測器攝像系統及其隱私保護攝像方法Dual-sensor camera system and privacy-preserving camera method therefor

本發明是有關於一種攝像系統及方法，且特別是有關於一種雙感測器攝像系統及其隱私保護攝像方法。The present invention relates to a camera system and method, and in particular, to a dual-sensor camera system and a privacy-preserving camera method.

相機的曝光條件（包括光圈、快門、感光度）會影響所拍攝影像的品質，因此許多相機在拍攝影像的過程中會自動調整曝光條件，以獲得清晰且明亮的影像。然而，在低光源或是背光等高反差的場景中，相機調整曝光條件的結果可能會產生雜訊過高或是部分區域過曝的結果，無法兼顧所有區域的影像品質。The camera's exposure conditions (including aperture, shutter, and sensitivity) affect the quality of the images captured, so many cameras automatically adjust exposure conditions during image capture to obtain clear and bright images. However, in scenes with high contrast such as low light source or backlight, the result of camera adjustment of exposure conditions may result in excessive noise or overexposure in some areas, and the image quality in all areas cannot be balanced.

對此，目前技術有採用一種新的影像感測器架構，其是利用紅外線（IR）感測器高光敏感度的特性，在影像感測器的色彩像素中穿插配置IR像素，以輔助亮度偵測。舉例來說，圖1是習知使用影像感測器擷取影像的示意圖。請參照圖1，習知的影像感測器10中除了配置有紅（R）、綠（G）、藍（B）等顏色像素外，還穿插配置有紅外線（I）像素。藉此，影像感測器10能夠將R、G、B顏色像素所擷取的色彩資訊12與I像素所擷取的亮度資訊14結合，而獲得色彩及亮度適中的影像16。In this regard, the current technology adopts a new image sensor architecture, which utilizes the characteristics of high light sensitivity of infrared (IR) sensors to intersperse and configure IR pixels among the color pixels of the image sensor to assist brightness detection. Measurement. For example, FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. Referring to FIG. 1 , in addition to red (R), green (G), blue (B) and other color pixels, the conventional image sensor 10 is also interspersed with infrared (I) pixels. Thereby, the image sensor 10 can combine the color information 12 captured by the R, G, and B color pixels with the luminance information 14 captured by the I pixel to obtain an image 16 with moderate color and brightness.

然而，在上述單一影像感測器的架構下，影像感測器中每個像素的曝光條件相同，因此只能選擇較適用於顏色像素或紅外線像素的曝光條件來擷取影像，結果仍無法有效地利用兩種像素的特性來改善所擷取影像的影像品質。However, in the above-mentioned single image sensor structure, the exposure conditions of each pixel in the image sensor are the same, so only the exposure conditions that are more suitable for color pixels or infrared pixels can be selected to capture images, and the result is still ineffective. The characteristics of the two pixels are used to improve the image quality of the captured image.

本發明提供一種雙感測器攝像系統及其隱私保護攝像方法，可在不侵犯攝像對象隱私的情況下，生成具備攝像場景細節的場景影像。The invention provides a dual-sensor camera system and a privacy protection camera method thereof, which can generate a scene image with details of the camera scene without infringing on the privacy of the camera object.

本發明的雙感測器攝像系統包括至少一個色彩感測器、至少一個紅外線感測器、儲存裝置以及耦接所述色彩感測器、紅外光感測器及儲存裝置的處理器。所述處理器經配置以載入並執行儲存在儲存裝置中的電腦程式以：控制色彩感測器及紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像；適應性選擇能顯露出攝像場景的細節的色彩影像及紅外線影像的組合；根據興趣對象的至少一個特徵，偵測所選擇的色彩影像中具有此些特徵的特徵區域；以及融合所選擇的色彩影像及紅外線影像以生成具備攝像場景細節的融合影像，裁切融合影像中的特徵區域的影像並以非屬於紅外線影像的影像取代，以生成場景影像。The dual-sensor camera system of the present invention includes at least one color sensor, at least one infrared sensor, a storage device, and a processor coupled to the color sensor, the infrared light sensor, and the storage device. The processor is configured to load and execute a computer program stored in the storage device to: control the color sensor and the infrared sensor to respectively capture a plurality of color images and multiple infrared images; adaptively selecting a combination of color images and infrared images that reveal details of the camera scene; based on at least one feature of an object of interest, detecting feature areas in the selected color image with such features; and fusing The selected color image and infrared image are used to generate a fusion image with details of the shooting scene, and the image of the characteristic area in the fusion image is cropped and replaced with an image that does not belong to the infrared image to generate a scene image.

本發明的雙感測器攝像系統的隱私保護攝像方法，適用於包括至少一個色彩感測器、至少一個紅外線感測器及處理器的雙感測器攝像系統。所述方法包括下列步驟：控制色彩感測器及紅外線感測器採用適用於攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像；適應性選擇能顯露出攝像場景的細節的色彩影像及紅外線影像的組合；根據興趣對象的至少一個特徵，偵測所選擇的色彩影像中具有所述特徵的特徵區域，以及融合所選擇的色彩影像及紅外線影像以生成具備攝像場景細節的融合影像，裁切融合影像中的特徵區域的影像並以非屬於紅外線影像的影像取代，以生成場景影像。The privacy-preserving camera method for a dual-sensor camera system of the present invention is suitable for a dual-sensor camera system comprising at least one color sensor, at least one infrared sensor and a processor. The method includes the following steps: controlling the color sensor and the infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for the shooting scene; The combination of the detailed color image and the infrared image; according to at least one feature of the object of interest, detecting a feature area in the selected color image with the feature, and fusing the selected color image and the infrared image to generate a camera scene with details The fused image of the fused image is cropped, and the image of the characteristic area in the fused image is cut and replaced with an image that does not belong to the infrared image to generate a scene image.

基於上述，本發明的雙感測器攝像系統及其隱私保護攝像方法利用獨立配置的色彩感測器及紅外線感測器採用適於當前攝像場景的不同曝光條件擷取多張影像，從中選擇出能夠顯露出攝像場景細節的色彩影像及紅外線影像的組合以進行融合，並將其中的敏感區域以非紅外線影像取代，例如:高動態範圍影像，從而在不侵犯攝像對象隱私的情況下，生成具備攝像場景細節的場景影像。Based on the above, the dual-sensor camera system and the privacy-preserving camera method of the present invention utilize independently configured color sensors and infrared sensors to capture multiple images under different exposure conditions suitable for the current camera scene, and select a The combination of color images and infrared images that can reveal the details of the camera scene for fusion, and replace the sensitive areas with non-infrared images, such as high dynamic range images, so as to generate images with A scene image that captures scene details.

本發明實施例揭露一種雙感測器攝像系統與隱私保護攝像方法，利用獨立配置的色彩及紅外線感測器分別擷取不同曝光條件下的多張影像，並選擇曝光條件適當的色彩及紅外線影像融合為結果影像，藉此補足色彩影像的紋理細節，提高所攝影像的影像品質。針對紅外線感測器所擷取的紅外線影像可能會有侵害拍攝對象隱私的疑慮，例如會顯露出穿著下的身體細節，本發明實施例的攝像方法可針對特定區域進行處理，從而在提高攝像品質的同時，避免造成上述侵害。Embodiments of the present invention disclose a dual-sensor camera system and a privacy-preserving camera method. Color and infrared sensors configured independently are used to capture multiple images under different exposure conditions, and color and infrared images with appropriate exposure conditions are selected. The resulting image is fused to complement the texture details of the color image and improve the image quality of the captured image. In view of the concern that the infrared image captured by the infrared sensor may infringe the privacy of the photographed subject, such as revealing the details of the body under the clothes, the imaging method of the embodiment of the present invention can be processed for a specific area, thereby improving the imaging quality. At the same time, avoid the above-mentioned damages.

圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。請參照圖2，本發明實施例的影像感測器20採用獨立配置色彩感測器22與紅外線（IR）感測器24的雙感測器架構，利用色彩感測器22與紅外線感測器24各自的特性，採用適於當前拍攝場景的多個曝光條件分別擷取多張影像，並從中選擇曝光條件適當的色彩影像22a與紅外線影像24a，透過影像融合的方式，使用紅外線影像24a來補足色彩影像22a中缺乏的紋理細節，從而獲得色彩及紋理細節均佳的場景影像26。FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. Referring to FIG. 2 , the image sensor 20 of the embodiment of the present invention adopts a dual-sensor structure in which a color sensor 22 and an infrared (IR) sensor 24 are independently configured, and the color sensor 22 and the infrared sensor are used. 24 have their respective characteristics, use multiple exposure conditions suitable for the current shooting scene to capture multiple images respectively, and select the color image 22a and infrared image 24a with appropriate exposure conditions from them, and use the infrared image 24a to complement the image fusion method. The lack of texture details in the color image 22a results in a scene image 26 with good color and texture details.

圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。請參照圖3，本實施例的雙感測器攝像系統30可配置於手機、平板電腦、筆記型電腦、導航裝置、行車紀錄器、數位相機、數位攝影機等電子裝置中，用以提供攝像功能。雙感測器攝像系統30包括至少一個色彩感測器32、至少一個紅外線感測器34、儲存裝置36及處理器38，其功能分述如下：FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 , the dual-sensor camera system 30 of this embodiment can be configured in electronic devices such as mobile phones, tablet computers, notebook computers, navigation devices, driving recorders, digital cameras, digital cameras, etc., to provide a camera function . The dual-sensor camera system 30 includes at least one color sensor 32, at least one infrared sensor 34, a storage device 36 and a processor 38, and its functions are described as follows:

色彩感測器32例如包括電荷耦合元件（Charge Coupled Device，CCD）、互補性氧化金屬半導體（Complementary Metal-Oxide Semiconductor，CMOS）元件或其他種類的感光元件，而可感測光線強度以產生攝像場景的影像。色彩感測器32例如是紅綠藍（RGB）影像感測器，其中包括紅（R）、綠（G）、藍（B）顏色像素，用以擷取攝像場景中的紅光、綠光、藍光等色彩資訊，並將這些色彩資訊合成以生成攝像場景的色彩影像。The color sensor 32 includes, for example, a Charge Coupled Device (CCD), a Complementary Metal-Oxide Semiconductor (CMOS) device, or other types of photosensitive devices, and can sense light intensity to generate a camera scene image. The color sensor 32 is, for example, a red-green-blue (RGB) image sensor, which includes red (R), green (G), and blue (B) color pixels for capturing red light and green light in the camera scene , blue light and other color information, and combine these color information to generate a color image of the camera scene.

紅外線感測器34例如包括CCD、CMOS元件或其他種類的感光元件，其經由調整感光元件的波長感測範圍，而能夠感測紅外光。紅外線感測器34例如是以上述感光元件作為像素來擷取攝像場景中的紅外光資訊，並將這些紅外光資訊合成以生成攝像場景的紅外線影像。The infrared sensor 34 includes, for example, a CCD, a CMOS element or other types of photosensitive elements, which can sense infrared light by adjusting the wavelength sensing range of the photosensitive element. The infrared sensor 34, for example, uses the above-mentioned photosensitive elements as pixels to capture infrared light information in the imaging scene, and synthesizes the infrared light information to generate an infrared image of the imaging scene.

儲存裝置36例如是任意型式的固定式或可移動式隨機存取記憶體（Random Access Memory，RAM）、唯讀記憶體（Read-Only Memory，ROM）、快閃記憶體（Flash memory）、硬碟或類似元件或上述元件的組合，而用以儲存可由處理器38執行的電腦程式。在一些實施例中，儲存裝置36例如還可儲存由色彩感測器32所擷取的色彩影像及紅外線感測器34所擷取的紅外線影像。The storage device 36 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard drive A disk or similar element, or a combination of the foregoing, for storing computer programs executable by the processor 38 . In some embodiments, the storage device 36 may also store, for example, the color image captured by the color sensor 32 and the infrared image captured by the infrared sensor 34 .

處理器38例如是中央處理單元（Central Processing Unit，CPU），或是其他可程式化之一般用途或特殊用途的微處理器（Microprocessor）、微控制器（Microcontroller）、數位訊號處理器（Digital Signal Processor，DSP）、可程式化控制器、特殊應用積體電路（Application Specific Integrated Circuits，ASIC）、可程式化邏輯裝置（Programmable Logic Device，PLD）或其他類似裝置或這些裝置的組合，本發明不在此限制。在本實施例中，處理器38可從儲存裝置36載入電腦程式，以執行本發明實施例的雙感測器攝像系統的隱私保護攝像方法。The processor 38 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessors), microcontrollers (Microcontrollers), and digital signal processors (Digital Signal Processors). Processor, DSP), programmable controller, application specific integrated circuit (Application Specific Integrated Circuits, ASIC), programmable logic device (Programmable Logic Device, PLD) or other similar devices or a combination of these devices, the present invention does not this limit. In this embodiment, the processor 38 can load a computer program from the storage device 36 to execute the privacy-preserving camera method of the dual-sensor camera system according to the embodiment of the present invention.

圖4是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。請同時參照圖3及圖4，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的隱私保護攝像方法的詳細步驟。FIG. 4 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 4 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following describes the privacy-preserving camera method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S402中，由處理器38控制色彩感測器32及紅外線感測器34採用適用於當前攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像。In step S402, the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for the current shooting scene.

在一些實施例中，處理器38例如是控制色彩感測器32及紅外線感測器34中的至少一者採用標準曝光條件來擷取攝像場景的至少一張標準影像，並使用這些標準影像來識別攝像場景。所述標準曝光條件例如包括採用既有測光技術所決定的光圈、快門、感光度等參數，而處理器38則根據在此曝光條件下所擷取之影像的色相（Hue）、明度（Value）、彩度（Chroma）、白平衡等影像參數的強弱或分佈來識別攝像場景，包括攝像場景的位置（室內或室外）、光源（高光源或低光源）、反差（高反差或低反差）、攝像物的種類（物品或人像）或狀態（動態或靜態）等。在其他實施例中，處理器38亦可採用定位方式來識別攝像場景或是直接接收使用者操作來設定攝像場景，在此不設限。In some embodiments, the processor 38 controls at least one of the color sensor 32 and the infrared sensor 34 to capture at least one standard image of the camera scene using standard exposure conditions, and uses these standard images to Identify the camera scene. The standard exposure conditions include, for example, parameters such as aperture, shutter, and sensitivity determined by using the existing light metering technology. , Chroma, white balance and other image parameters to identify the camera scene, including the location of the camera scene (indoor or outdoor), light source (high light source or low light source), contrast (high contrast or low contrast), The type (object or portrait) or state (dynamic or static) of the photographed object, etc. In other embodiments, the processor 38 may also use a positioning method to identify the camera scene or directly receive user operations to set the camera scene, which is not limited herein.

在一些實施例中，處理器38例如是以標準曝光條件中的曝光時間為基準，控制色彩感測器32及紅外線感測器34擷取曝光時間較短或較長的色彩影像，這些色彩影像彼此的曝光時間的差例如為介於-3至3的曝光值（Exposure Value，EV）中的任意值，在此不設限。舉例來說，若A影像比B影像亮一倍，則可將B影像的EV加1，以此類推，曝光值可以有小數（例如+0.3EV），在此不設限。In some embodiments, the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture color images with shorter or longer exposure times based on the exposure time in standard exposure conditions, for example. The difference between the exposure times of each other is, for example, any value in the exposure value (Exposure Value, EV) between -3 and 3, which is not limited here. For example, if the A image is twice as bright as the B image, the EV of the B image can be increased by 1, and so on, the exposure value can have a decimal (eg +0.3EV), which is not limited here.

在步驟S404中，由處理器38適應性選擇能顯露出攝像場景的細節的色彩影像及紅外線影像的組合。在一些實施例中，處理器38例如會控制色彩感測器32以適當的曝光時間擷取色彩影像，使得攝像場景的部分顏色細節可被保留，並確保之後融合的影像可顯露出攝像場景的顏色細節。所述適當的曝光時間例如是比會造成所擷取影像過曝的曝光時間還短一預設時間長度的曝光時間，所述預設時間長度例如為0.01至1秒中的任意值，在此不設限。In step S404, the processor 38 adaptively selects the combination of the color image and the infrared image that can reveal the details of the shooting scene. In some embodiments, the processor 38 controls, for example, the color sensor 32 to capture a color image with an appropriate exposure time, so that some color details of the camera scene can be preserved, and the fused image can reveal the details of the camera scene. Color details. The appropriate exposure time is, for example, an exposure time shorter than the exposure time that will cause the captured image to be overexposed by a predetermined time length, and the predetermined time length is, for example, any value in the range of 0.01 to 1 second, here No limit.

在一些實施例中，處理器38例如會先根據各張色彩影像的顏色細節，選擇其中一張色彩影像作為基準影像，接著辨識基準影像中缺乏紋理細節的至少一個缺陷區域，然後再根據各張紅外線影像中對應於這些缺陷區域的影像的紋理細節，選擇其中一張紅外線影像作為與基準影像融合的影像。In some embodiments, the processor 38 first selects one of the color images as the reference image according to the color details of each color image, and then identifies at least one defective area lacking texture details in the reference image, and then selects one of the color images as the reference image according to the color details of each The texture details of the images corresponding to these defect areas in the infrared images are selected, and one of the infrared images is selected as the image to be fused with the reference image.

詳言之，基於色彩感測器32每次只能採用單一曝光條件擷取色彩影像，在攝像場景為低光源或高反差的情況下，每一張色彩影像都可能會出現高雜訊、過曝或曝光不足的區域（即上述的缺陷區域）。此時，處理器38即可利用紅外線感測器34高光敏感度的特性，針對上述的缺陷區域，從先前擷取的多張紅外線影像中，選擇具備該缺陷區域的紋理細節的紅外線影像，而可用以補足色彩影像中缺陷區域的紋理細節。To be more specific, because the color sensor 32 can only use a single exposure condition to capture color images at a time, when the shooting scene is low light source or high contrast, each color image may have high noise and excessive noise. Areas that are exposed or underexposed (i.e. the defective areas described above). At this time, the processor 38 can use the characteristics of the high light sensitivity of the infrared sensor 34 to select the infrared image with the texture details of the defect area from the multiple infrared images captured previously for the above-mentioned defective area, and Can be used to complement texture details in defective areas in color images.

在步驟S406中，由處理器38根據興趣對象的至少一個特徵，偵測所選擇色彩影像中具有所述特徵的特徵區域。所述特徵例如是人類的身體特徵，例如臉部、軀幹、四肢等，或是人類穿著的特徵，例如面罩、衣服、褲子，在此不設限。In step S406, the processor 38 detects a feature area having the feature in the selected color image according to at least one feature of the object of interest. The features are, for example, human body features, such as face, torso, limbs, etc., or features worn by humans, such as face masks, clothes, and pants, which are not limited herein.

在一些實施例中，處理器38例如會利用機器學習模型來辨識色彩影像中的興趣對象以偵測特徵區域。其中，所述的機器學習模型例如是利用包括興趣對象的多張色彩影像以及對於各張色彩影像中的興趣對象的辨識結果所訓練。In some embodiments, the processor 38 uses, for example, a machine learning model to identify objects of interest in the color image to detect feature regions. The machine learning model is, for example, trained by using a plurality of color images including the object of interest and the recognition results of the object of interest in each color image.

詳言之，所述的機器學習模型例如是包括輸入層、至少一隱藏層及輸出層的卷積神經網路（Convolutional Neural Network，CNN）、深度神經網路（Deep Neural Network，DNN）、遞迴神經網路（Recurrent Neural Network，RNN）或其他具學習功能的模型，在此不設限。其中，處理器38例如是將包括興趣對象的多張色彩影像依序輸入輸入層，由各個隱藏層的多個神經元利用一激勵函數針對輸入層的輸出計算當次的輸出。所述的激勵函數例如是S（sigmoid）函數或是雙曲正切（tanh）函數，在此不設限，然後由輸出層利用如歸一化指數（softmax）函數的轉換函數將隱藏層的當次輸出轉換為興趣對象的預測結果。然後，處理器38會將預測結果與當次輸入的色彩影像對應的辨識結果比較，以根據比較結果更新隱藏層的各神經元的權重。其中，處理器38例如是以利用機器學習模型所輸出的預測結果與真實的辨識結果，來計算損失函數（loss function）並用以衡量機器學習模型的預測結果是否夠準確，據以更新隱藏層的各個神經元的權重。在其他實施例中，處理器38亦可利用梯度下降法（Gradient Descent，GD）或反向傳播法（Backpropagation，BP）來更新隱藏層的各個神經元的權重，在此不設限。最後，處理器38將重複上述步驟，藉此訓練機器學習模型來辨識興趣對象，並可取得興趣對象在色彩影像中所佔的區域以作為特徵區域。In detail, the machine learning model is, for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), an input layer, at least one hidden layer and an output layer. Recurrent Neural Network (RNN) or other models with learning function are not limited here. The processor 38, for example, sequentially inputs a plurality of color images including the object of interest into the input layer, and uses an excitation function to calculate the current output for the output of the input layer by a plurality of neurons in each hidden layer. The excitation function is, for example, an S (sigmoid) function or a hyperbolic tangent (tanh) function, which is not limited here, and then the output layer uses a transformation function such as a normalized exponential (softmax) function to convert the current value of the hidden layer. The secondary output is converted to the prediction result of the object of interest. Then, the processor 38 compares the prediction result with the recognition result corresponding to the current input color image, so as to update the weight of each neuron in the hidden layer according to the comparison result. The processor 38 uses, for example, the prediction result output by the machine learning model and the real identification result to calculate a loss function and use it to measure whether the prediction result of the machine learning model is accurate enough to update the hidden layer. The weight of each neuron. In other embodiments, the processor 38 may also use a gradient descent (GD) method or a backpropagation (Backpropagation, BP) method to update the weight of each neuron in the hidden layer, which is not limited herein. Finally, the processor 38 will repeat the above steps, thereby training the machine learning model to identify the object of interest, and can obtain the area occupied by the object of interest in the color image as a feature area.

在步驟S408中，由處理器38融合所選擇的色彩影像及紅外線影像，以生成具備攝像場景的細節的融合影像，並裁切此融合影像中的特徵區域影像並以非屬於紅外線影像的影像取代，以生成場景影像。所述非屬於紅外線影像的影像例如是上述的色彩影像或是由多張色彩影像經由高動態範圍（high dynamic range，HDR）處理所生成的影像，在此不設限。In step S408, the processor 38 fuses the selected color image and the infrared image to generate a fusion image with details of the camera scene, and crops the characteristic area image in the fusion image and replaces it with an image that does not belong to the infrared image , to generate a scene image. The image that does not belong to the infrared image is, for example, the above-mentioned color image or an image generated by a high dynamic range (high dynamic range, HDR) process from a plurality of color images, which is not limited herein.

在一些實施例中，處理器38例如是採用計算所選擇色彩影像及紅外線影像整張影像中對應像素之像素值的平均或加權平均的方式，或是採用其他影像融合方式，將所選擇的色彩影像及紅外線影像的整張影像直接融合。在一些實施例中，處理器38也可僅針對色彩影像中的缺陷區域，而使用紅外線影像中對應於該缺陷區域的影像來填補或取代色彩影像中缺陷區域的影像，在此不設限。In some embodiments, the processor 38, for example, calculates the average or weighted average of the pixel values of the corresponding pixels in the entire image of the selected color image and the infrared image, or uses other image fusion methods to combine the selected color image. The entire image of the image and the infrared image is directly fused. In some embodiments, the processor 38 may only target the defective area in the color image, and use the image corresponding to the defective area in the infrared image to fill or replace the image of the defective area in the color image, which is not limited herein.

在一些實施例中，處理器38例如是將所選擇的色彩影像及紅外線影像裁切掉特徵區域影像後再進行融合，之後將非屬於紅外線影像的影像貼上融合影像中的特徵區域，從而生成場景影像。藉此，可減少融合影像所需的計算量。In some embodiments, the processor 38, for example, cuts the selected color image and the infrared image out of the characteristic area image and then performs fusion, and then pastes the image that does not belong to the infrared image to the characteristic area in the fusion image, thereby generating scene image. In this way, the amount of computation required for fusing images can be reduced.

在一些實施例中，處理器38例如會控制色彩感測器32採用較所選擇的色彩影像的曝光時間長或短的多個曝光時間擷取多張色彩影像並執行高動態範圍處理，以生成具備特徵區域的細節的高動態範圍影像，並使用此高動態範圍影像來取代所裁切的融合影像中的特徵區域的影像。In some embodiments, the processor 38 controls, for example, the color sensor 32 to capture multiple color images with multiple exposure times that are longer or shorter than the exposure time of the selected color image and perform high dynamic range processing to generate A high dynamic range image with details of the feature area, and use the high dynamic range image to replace the image of the feature area in the cropped fused image.

詳細而言，處理器38例如會根據其所選擇的色彩影像的曝光時間，使用較此曝光時間為短的曝光時間以及較此曝光時間為長的曝光時間，控制色彩感測器32分別擷取曝光時間較短的色彩影像以及曝光時間較長的色彩影像，而結合使用原曝光時間擷取的色彩影像來實施HDR處理。即，從三張色彩影像中選擇具備較佳顏色及紋理細節的區域來補足其他色彩影像中欠缺細節的區域，從而獲得亮部及暗部細節均佳的高動態範圍影像。In detail, for example, the processor 38 controls the color sensor 32 to respectively capture the exposure time using a shorter exposure time and a longer exposure time according to the exposure time of the selected color image. A color image with a shorter exposure time and a color image with a longer exposure time are combined with the color image captured using the original exposure time for HDR processing. That is, areas with better color and texture details are selected from the three color images to make up for areas lacking details in other color images, so as to obtain a high dynamic range image with good details in both highlights and shadows.

在一些實施例中，處理器38可根據所選擇色彩影像的特徵區域的細節，選擇用以擷取多張色彩影像的曝光時間，使得所擷取的多張色彩影像經高動態範圍處理後，可生成具備此特徵區域細節的高動態範圍影像。舉例來說，若所選擇色彩影像的特徵區域因為過曝而缺乏顏色及紋理細節，則處理器38可選擇多個較短的曝光時間來擷取色彩影像並用以執行高動態範圍處理，藉此生成具備顏色及紋理細節的高動態範圍影像。類似地，若所選擇色彩影像的特徵區域因為曝光不足而缺乏顏色及紋理細節，則處理器38會選擇多個較長的曝光時間來擷取色彩影像並用以執行高動態範圍處理，藉此生成具備顏色及紋理細節的高動態範圍影像。In some embodiments, the processor 38 may select the exposure time for capturing the multiple color images according to the details of the characteristic regions of the selected color images, so that after the captured multiple color images are processed with high dynamic range, High dynamic range imagery with detail in this characteristic area can be produced. For example, if a characteristic area of the selected color image lacks color and texture details due to overexposure, the processor 38 may select multiple shorter exposure times to capture the color image and perform high dynamic range processing, thereby Generate high dynamic range images with color and texture detail. Similarly, if the characteristic area of the selected color image lacks color and texture details due to underexposure, the processor 38 will select a plurality of longer exposure times to capture the color image and perform high dynamic range processing, thereby generating High dynamic range images with color and texture detail.

在一些實施例中，處理器38例如會針對高動態範圍影像執行二維空間降噪（2D spatial denoise）等降噪（noise reduction，NR）處理，以減少高動態範圍影像中的雜訊，提高最終輸出影像的影像品質。In some embodiments, the processor 38, for example, performs noise reduction (NR) processing such as 2D spatial denoise (2D spatial denoise) on the high dynamic range image, so as to reduce the noise in the high dynamic range image and improve the The image quality of the final output image.

藉由上述方法，雙感測器攝像系統30不僅可生成可包括攝像場景的所有細節（顏色及紋理細節）的影像，且可將影像中特徵區域的影像以非屬於紅外線影像的影像（例如高動態範圍影像）取代，從而在不侵犯攝像對象隱私的情況下，提高所攝影像的影像品質。Through the above method, the dual-sensor camera system 30 can not only generate an image that can include all details (color and texture details) of the camera scene, but also can convert the image of the characteristic area in the image to the image that does not belong to the infrared image (eg, high-level image). Dynamic Range Image) to improve the image quality of the captured image without violating the subject's privacy.

圖5是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。請同時參照圖3及圖5，本實施例進一步說明上述針對整張影像進行融合的實施例的詳細實施方式。本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的隱私保護攝像方法的詳細步驟。FIG. 5 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 5 at the same time. This embodiment further describes the detailed implementation of the above-mentioned embodiment of performing fusion on the entire image. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following describes the detailed steps of the privacy-preserving camera method of this embodiment in combination with various elements of the dual-sensor camera system 30 .

在步驟S502中，由處理器38根據各張色彩影像的顏色細節，選擇其中一張色彩影像作為基準影像。在一實施例中，處理器38例如是選擇顏色細節最多的色彩影像作為基準影像。所述顏色細節的多寡例如可由色彩影像中過曝或曝光不足區域的大小來決定。詳言之，過曝區域像素的顏色趨近白色、曝光不足區域像素的顏色趨近黑色，因此這些區域的顏色細節會較少。因此，若色彩影像中包括較多的這類區域，代表其顏色細節較少，處理器38據此即可判斷出哪一張色彩影像的顏色細節最多，而用以作為基準影像。在其他實施例中，處理器38也可依據各張色彩影像的對比度、飽和度或其他影像參數來分辨其顏色細節的多寡，在此不設限。In step S502, the processor 38 selects one of the color images as the reference image according to the color details of each color image. In one embodiment, the processor 38 selects, for example, the color image with the most color details as the reference image. The amount of color detail can be determined, for example, by the size of the overexposed or underexposed areas in the color image. Specifically, the color of the pixels in the overexposed areas tends to be white, and the color of the pixels in the underexposed areas tends to be black, so there will be less color detail in these areas. Therefore, if the color image includes more such regions, it means that the color details are less, and the processor 38 can determine which color image has the most color details accordingly, and use it as the reference image. In other embodiments, the processor 38 can also distinguish the color details of each color image according to the contrast, saturation or other image parameters, which is not limited herein.

在步驟S504中，由處理器38辨識基準影像中缺乏紋理細節的至少一個缺陷區域。所述的缺陷區域例如是上述的過曝區域或曝光不足區域，或是在低光源下所擷取的具較高雜訊的區域，在此不設限。In step S504, the processor 38 identifies at least one defect region in the reference image that lacks texture details. The defect area is, for example, the above-mentioned overexposed area or underexposed area, or an area with higher noise captured under low light source, which is not limited herein.

在步驟S506中，由處理器38根據各張紅外線影像中對應於所述缺陷區域的影像的紋理細節，選擇其中一張紅外線影像。在一實施例中，處理器38例如是選擇對應於所述缺陷區域的影像的紋理細節最多的紅外線影像作為與基準影像融合的影像。其中，處理器38例如是依據各張紅外線影像的對比度或其他影像參數來分辨其紋理細節的多寡，在此不設限。In step S506, the processor 38 selects one of the infrared images according to the texture details of the images corresponding to the defective area in each of the infrared images. In one embodiment, the processor 38, for example, selects the infrared image corresponding to the image of the defect area with the most texture details as the image to be fused with the reference image. Wherein, the processor 38, for example, determines the amount of texture details of each infrared image according to the contrast ratio or other image parameters, which is not limited herein.

在步驟S508中，由處理器38對所選擇的色彩影像及紅外線影像執行特徵擷取，以擷取色彩影像及紅外線影像中的多個特徵，並根據所擷取特徵之間的對應關係將色彩影像及紅外線影像對齊。需說明的是，上述的特徵擷取及匹配的方式僅為舉例說明，在其他實施例中，處理器38亦可採用其他種類的影像對齊方式對色彩影像及紅外線影像進行對齊，在此不設限。In step S508, the processor 38 performs feature extraction on the selected color image and the infrared image, so as to extract a plurality of features in the color image and the infrared image, and classify the colors according to the corresponding relationship between the extracted features. Image and infrared image alignment. It should be noted that the above-mentioned methods of feature extraction and matching are only examples. In other embodiments, the processor 38 may also use other types of image alignment methods to align the color image and the infrared image, which is not set here. limit.

在步驟S510，由處理器38對經對齊的紅外線影像與基準影像進行影像融合，以生成補足所述缺陷區域的紋理細節的場景影像。In step S510, the processor 38 performs image fusion on the aligned infrared image and the reference image to generate a scene image that complements the texture details of the defect area.

在一些實施例中，處理器38例如是計算色彩影像及紅外線影像整張影像中對應像素之像素值的平均或加權平均的方式來對紅外線影像與基準影像進行影像融合。In some embodiments, the processor 38 performs image fusion on the infrared image and the reference image by, for example, calculating an average or weighted average of pixel values of corresponding pixels in the entire image of the color image and the infrared image.

在一些實施例中，處理器38例如是將基準影像的色彩空間由RGB色彩空間轉換至YUV色彩空間，並將轉換後基準影像的亮度分量以紅外線影像的亮度分量取代，然後將取代後的基準影像的色彩空間轉換回RGB色彩空間，以生成場景影像。在其他實施例中，處理器38亦可將基準影像的色彩空間轉換至YCbCr、CMYK或其他種類的色彩空間，並在取代亮度分量之後再轉換回原本的色彩空間，本實施例不限定色彩空間的轉換方式。In some embodiments, the processor 38, for example, converts the color space of the reference image from the RGB color space to the YUV color space, replaces the luminance component of the converted reference image with the luminance component of the infrared image, and then converts the replaced reference image The color space of the image is converted back to the RGB color space to generate the scene image. In other embodiments, the processor 38 can also convert the color space of the reference image to YCbCr, CMYK or other color spaces, and then convert back to the original color space after replacing the luminance component. This embodiment does not limit the color space conversion method.

詳言之，由於紅外線影像的亮度分量具有較佳的訊噪比（signal-to-noise ratio，SNR），且包括較多的攝像場景的紋理細節，因此以紅外線影像的亮度分量直接取代基準影像的亮度分量，可大幅增加基準影像中的紋理細節。In detail, since the luminance component of the infrared image has a better signal-to-noise ratio (SNR) and includes more texture details of the camera scene, the luminance component of the infrared image directly replaces the reference image. , which can greatly increase the texture detail in the reference image.

藉由上述方法，雙感測器攝像系統30即可利用紅外線影像來增加色彩影像的紋理細節，特別是針對紋理細節不足的區域，從而提高所攝影像的影像品質。With the above method, the dual-sensor camera system 30 can use the infrared image to increase the texture details of the color image, especially for areas with insufficient texture details, thereby improving the image quality of the captured image.

舉例來說，圖6是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的範例。請參照圖6，本實施例是通過上述圖5的隱私保護攝像方法，選擇出顏色細節最多的色彩影像62作為基準影像，並針對色彩影像62中缺乏紋理細節的缺陷區域（例如人臉區域62a），從採用不同曝光條件擷取的多張紅外線影像中選擇出該缺陷區域的紋理細節最多的紅外線影像64，用以與色彩影像62進行影像融合，從而獲得同時具備較多顏色細節及紋理細節的場景影像66。For example, FIG. 6 is an example of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. Referring to FIG. 6 , in this embodiment, the color image 62 with the most color details is selected as the reference image through the privacy-preserving camera method in the above-mentioned FIG. ), select the infrared image 64 with the most texture details in the defect area from the multiple infrared images captured under different exposure conditions, and use it for image fusion with the color image 62 to obtain more color details and texture details at the same time. scene image 66.

在一些實施例中，處理器38例如是將基準影像的色彩空間由RGB色彩空間轉換至YUV色彩空間，並將轉換後基準影像的缺陷區域的影像的亮度分量以紅外線影像的對應於所述缺陷區域的亮度分量取代，然後將取代後的基準影像的色彩空間轉換回RGB色彩空間，以生成場景影像。在其他實施例中，處理器38亦可將基準影像的色彩空間轉換至YCbCr、CMYK或其他種類的色彩空間，並在取代亮度分量之後再轉換回原本的色彩空間，本實施例不限定色彩空間的轉換方式。In some embodiments, the processor 38, for example, converts the color space of the reference image from the RGB color space to the YUV color space, and converts the luminance component of the image of the defective area of the converted reference image to the infrared image corresponding to the defect The luminance component of the area is replaced, and then the color space of the replaced reference image is converted back to the RGB color space to generate the scene image. In other embodiments, the processor 38 can also convert the color space of the reference image to YCbCr, CMYK or other color spaces, and then convert back to the original color space after replacing the luminance component. This embodiment does not limit the color space conversion method.

藉由上述方法，雙感測器攝像系統30即可利用紅外線影像來補足色彩影像中紋理細節不足的區域，從而提高所攝影像的影像品質。Through the above method, the dual-sensor camera system 30 can use the infrared image to supplement the areas with insufficient texture details in the color image, thereby improving the image quality of the captured image.

需說明的是，在一些實施例中，色彩影像中某些缺陷區域的紋理細節可能會因特定因素無法用紅外線影像來增強或補足，例如色彩感測器32與紅外線感測器34之間的視差（parallax）會造成紅外線感測器34被遮蔽。在此情況下，本發明實施例提供一種替代方式來增加缺陷區域的紋理細節，以最大程度地提高所攝影像的影像品質。It should be noted that, in some embodiments, the texture details of some defective areas in the color image may not be enhanced or complemented by the infrared image due to certain factors, such as the difference between the color sensor 32 and the infrared sensor 34 . Parallax can cause the infrared sensor 34 to be blocked. In this case, the embodiments of the present invention provide an alternative way to increase the texture details of the defect area, so as to maximize the image quality of the captured image.

圖7是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。請同時參照圖3及圖7，本實施例的方法適用於上述的雙感測器攝像系統30，以下即搭配雙感測器攝像系統30的各項元件說明本實施例的隱私保護攝像方法的詳細步驟。FIG. 7 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. Please refer to FIG. 3 and FIG. 7 at the same time. The method of this embodiment is applicable to the above-mentioned dual-sensor camera system 30 . The following describes the privacy-preserving camera method of this embodiment in combination with various elements of the dual-sensor camera system 30 . detailed steps.

在步驟S702中，由處理器38控制色彩感測器32及紅外線感測器34中的至少一者採用標準曝光條件來擷取攝像場景的至少一張標準影像，並使用這些標準影像來識別攝像場景。所述標準曝光條件的定義以及攝像場景的識別方式如前述實施例所述，在此不再贅述。In step S702, at least one of the color sensor 32 and the infrared sensor 34 is controlled by the processor 38 to capture at least one standard image of the camera scene using standard exposure conditions, and use these standard images to identify the camera Scenes. The definition of the standard exposure conditions and the way of identifying the imaging scene are as described in the foregoing embodiments, and will not be repeated here.

在步驟S704中，由處理器38控制色彩感測器32及紅外線感測器34採用適用於所識別之攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像。在步驟S706中，由處理器38根據各張色彩影像的顏色細節，選擇其中一張色彩影像作為基準影像。在步驟S708中，由處理器38根據興趣對象的至少一個特徵，偵測所選擇色彩影像中具有所述特徵的特徵區域。在步驟S710中，由處理器38控制色彩感測器32採用較所選擇的色彩影像的曝光時間長或短的多個曝光時間擷取多張色彩影像並執行高動態範圍處理，以生成具備特徵區域的細節的高動態範圍影像。在步驟S712中，由處理器38辨識基準影像中缺乏紋理細節的至少一個缺陷區域。上述步驟的實施方式分別與前述實施例的步驟S402~S408、S502~S504相同或相似，故其細節在此不再贅述。In step S704, the processor 38 controls the color sensor 32 and the infrared sensor 34 to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for the identified shooting scene. In step S706, the processor 38 selects one of the color images as the reference image according to the color details of each color image. In step S708, the processor 38 detects a feature area having the feature in the selected color image according to at least one feature of the object of interest. In step S710, the processor 38 controls the color sensor 32 to capture a plurality of color images using a plurality of exposure times that are longer or shorter than the exposure time of the selected color image and perform high dynamic range processing to generate a characteristic High dynamic range image of the detail of the area. In step S712, the processor 38 identifies at least one defect region in the reference image that lacks texture details. The implementation manners of the above steps are respectively the same as or similar to steps S402-S408 and S502-S504 in the foregoing embodiment, so the details are not repeated here.

與前述實施例不同的是，在步驟S714中，處理器38會判斷前述的多張紅外線影像中是否有紅外線影像包括基準影像中缺陷區域的紋理細節。其中，處理器38例如會檢視各張紅外線影像中對應於所述缺陷區域的區域是否有影像，以判斷紅外線感測器34是否被遮蔽，並判斷是否可用紅外線影像來填補基準影像中缺陷區域的紋理細節。Different from the foregoing embodiment, in step S714, the processor 38 determines whether any of the infrared images in the foregoing plurality of infrared images includes the texture details of the defective area in the reference image. The processor 38, for example, checks whether there is an image in the area corresponding to the defective area in each infrared image, to determine whether the infrared sensor 34 is blocked, and to determine whether the infrared image can be used to fill the defect area in the reference image. Texture details.

若有紅外線影像包括此缺陷區域的紋理細節，則在步驟S716中，處理器38會將基準影像中的所述缺陷區域的影像的亮度分量以紅外線影像中對應於所述缺陷區域的亮度分量取代，以生成補足所述缺陷區域的紋理細節的融合影像。If the infrared image includes the texture details of the defect area, in step S716, the processor 38 replaces the luminance component of the image of the defect area in the reference image with the luminance component corresponding to the defect area in the infrared image , to generate a fusion image that complements the texture details of the defective area.

若沒有紅外線影像包括此缺陷區域的紋理細節，則在步驟S914中，處理器38會將基準影像中的缺陷區域的影像以高動態範圍影像中對應於此缺陷區域的影像取代，以生成具備此缺陷區域的紋理細節的融合影像。If no infrared image includes the texture details of the defect area, in step S914, the processor 38 replaces the image of the defect area in the reference image with the image corresponding to the defect area in the high dynamic range image, so as to generate an image with this defect area. A fusion image of the texture details of the defect area.

在一些實施例中，處理器38可結合上述步驟S716及S718的處理方式，針對基準影像中的多個缺陷區域個別選用適當的處理方式，以最大程度地增加基準影像的細節，從而提高所攝影像的影像品質。In some embodiments, the processor 38 may combine the processing methods of the above-mentioned steps S716 and S718 to individually select appropriate processing methods for the plurality of defective areas in the reference image, so as to maximize the details of the reference image, thereby improving the captured image quality. image quality.

最後，在步驟S720中，由處理器38裁切融合影像中的特徵區域影像並貼上高動態範圍影像中的特徵區域影像，以生成場景影像。Finally, in step S720 , the processor 38 cuts the characteristic area image in the fusion image and pastes the characteristic area image in the high dynamic range image to generate a scene image.

藉由上述方法，雙感測器攝像系統30不僅可針對色彩影像中紋理細節不足的缺陷區域，利用紅外線影像或高動態範圍影像來補足紋理細節，且可進一步將融合影像中特徵區域的影像以高動態範圍影像取代，從而在不侵犯攝像對象隱私的情況下，提高所攝影像的影像品質。Through the above method, the dual-sensor camera system 30 can not only use infrared images or high dynamic range images to supplement the texture details for the defect areas with insufficient texture details in the color image, but also can further fuse the images of the feature areas in the images to make the texture details more accurate. High dynamic range images are replaced, thereby improving the image quality of the captured images without violating the privacy of the camera subjects.

綜上所述，本發明的雙感測器攝像系統及其隱私保護攝像方法藉獨立配置色彩感測器與紅外線感測器分別擷取多張影像，從中選擇曝光條件適當的影像進行融合，以使用紅外線影像填補或增加色彩影像中缺乏的紋理細節，且將融合影像中可能會侵害拍攝對象隱私的特徵區域以非屬於紅外線影像的影像取代，因此可在不侵犯攝像對象隱私的情況下，生成具備攝像場景細節的場景影像。To sum up, the dual-sensor camera system and the privacy-preserving camera method of the present invention capture multiple images by independently configuring the color sensor and the infrared sensor, and select images with appropriate exposure conditions for fusion, so as to Use infrared images to fill in or add texture details that are lacking in color images, and replace the feature areas in the fusion image that may infringe on the subject's privacy with images that are not infrared images, so it can be generated without infringing on the subject's privacy. A scene image with details of the camera scene.

10、20:影像感測器 12:色彩資訊 14:亮度資訊 16:影像 22:色彩感測器 22a、62:色彩影像 24:紅外線感測器 24a、64:紅外線影像 26、66:場景影像 30:雙感測器攝像系統 32:色彩感測器 34:紅外線感測器 36:儲存裝置 38:處理器 62a:人臉區域 R、G、B、I:像素 S402~S408、S502~S510、S702~S720:步驟10, 20: Image sensor 12: Color Information 14: Brightness information 16: Video 22: Color sensor 22a, 62: Color image 24: Infrared sensor 24a, 64: Infrared imagery 26, 66: Scene image 30: Dual-sensor camera system 32: Color Sensor 34: Infrared sensor 36: Storage Device 38: Processor 62a: face area R, G, B, I: pixels S402~S408, S502~S510, S702~S720: Steps

圖1是習知使用影像感測器擷取影像的示意圖。 圖2是依照本發明一實施例所繪示的使用影像感測器擷取影像的示意圖。 圖3是依照本發明一實施例所繪示的雙感測器攝像系統的方塊圖。 圖4是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。 圖5是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。 圖6是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的範例。 圖7是依照本發明一實施例所繪示的雙感測器攝像系統的隱私保護攝像方法的流程圖。FIG. 1 is a schematic diagram of conventionally using an image sensor to capture images. FIG. 2 is a schematic diagram of capturing an image using an image sensor according to an embodiment of the present invention. FIG. 3 is a block diagram of a dual-sensor camera system according to an embodiment of the present invention. FIG. 4 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. FIG. 5 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. FIG. 6 is an example of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention. FIG. 7 is a flowchart of a privacy-preserving camera method of a dual-sensor camera system according to an embodiment of the present invention.

S402~S408:步驟S402~S408: Steps

Claims (20)

一種雙感測器攝像系統，包括： 至少一色彩感測器； 至少一紅外線感測器； 儲存裝置，儲存電腦程式；以及 處理器，耦接所述至少一色彩感測器、所述至少一紅外光感測器及所述儲存裝置，經配置以載入並執行所述電腦程式以： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用適用於一攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像； 適應性選擇能顯露出所述攝像場景的細節的所述色彩影像及所述紅外線影像的組合； 根據興趣對象的至少一特徵，偵測所選擇的所述色彩影像中具有所述特徵的特徵區域；以及 融合所選擇的所述色彩影像及所述紅外線影像以生成具備所述攝像場景的所述細節的融合影像，裁切所述融合影像中的所述特徵區域的影像並以非屬於所述紅外線影像的影像取代，以生成場景影像。A dual-sensor camera system comprising: at least one color sensor; at least one infrared sensor; storage devices to store computer programs; and A processor, coupled to the at least one color sensor, the at least one infrared light sensor, and the storage device, is configured to load and execute the computer program to: controlling the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; adaptively selecting a combination of the color image and the infrared image that can reveal details of the camera scene; according to at least one feature of the object of interest, detecting a feature area having the feature in the selected color image; and Fusing the selected color image and the infrared image to generate a fusion image with the details of the shooting scene, cropping the image of the characteristic area in the fusion image and not belonging to the infrared image image to generate the scene image. 如請求項1所述的雙感測器攝像系統，其中所述處理器更包括： 控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間長或短的多個曝光時間擷取多張色彩影像並執行高動態範圍（high dynamic range，HDR）處理，以生成具備所述特徵區域的細節的高動態範圍影像，並用以取代所裁切的所述融合影像中的所述特徵區域的影像。The dual-sensor camera system of claim 1, wherein the processor further comprises: Controlling the at least one color sensor to capture a plurality of color images with a plurality of exposure times that are longer or shorter than the exposure time of the selected color image and perform high dynamic range (HDR) processing, so as to A high dynamic range image with details of the characteristic area is generated and used to replace the image of the characteristic area in the cropped fusion image. 如請求項2所述的雙感測器攝像系統，其中所述處理器包括： 根據所述色彩影像的所述特徵區域的所述細節選擇用以擷取所述多張色彩影像的所述曝光時間，使得所擷取的所述多張色彩影像經高動態範圍處理後，生成具備所述特徵區域的細節的所述高動態範圍影像。The dual-sensor camera system of claim 2, wherein the processor comprises: The exposure time for capturing the plurality of color images is selected according to the details of the characteristic regions of the color image, so that the captured plurality of color images are processed with high dynamic range to generate The high dynamic range image with details of the feature area. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器中的至少一者採用標準曝光條件擷取所述攝像場景的至少一標準影像，並使用所述至少一標準影像識別所述攝像場景。The dual-sensor camera system of claim 1, wherein the processor comprises: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture at least one standard image of the camera scene using standard exposure conditions, and using the at least one standard image to identify the camera scene. 如請求項1所述的雙感測器攝像系統，其中所述處理器包括： 根據各所述色彩影像的顏色細節，選擇所述色彩影像其中之一作為基準影像； 辨識所述基準影像中缺乏紋理細節的至少一缺陷區域；以及 根據各所述紅外線影像中對應於所述至少一缺陷區域的影像的紋理細節，選擇所述紅外線影像其中之一作為用以與所述基準影像融合的所述紅外線影像。The dual-sensor camera system of claim 1, wherein the processor comprises: selecting one of the color images as a reference image according to the color details of each of the color images; identifying at least one defect region in the reference image that lacks texture detail; and According to the texture details of the images corresponding to the at least one defect area in each of the infrared images, one of the infrared images is selected as the infrared image for fusion with the reference image. 如請求項5所述的雙感測器攝像系統，其中所述處理器包括： 選擇所述顏色細節最多的所述色彩影像作為所述基準影像；以及 選擇對應於所述至少一缺陷區域的影像的所述紋理細節最多的所述紅外線影像作為用以與所述基準影像融合的所述紅外線影像。The dual-sensor camera system of claim 5, wherein the processor comprises: selecting the color image with the most color detail as the reference image; and The infrared image corresponding to the image of the at least one defect area with the most detailed texture is selected as the infrared image for fusion with the reference image. 如請求項5所述的雙感測器攝像系統，其中所述處理器包括： 將所述基準影像中的所述至少一缺陷區域的影像的亮度分量以所述紅外線影像中對應於所述至少一缺陷區域的影像取代，以生成補足所述至少一缺陷區域的所述紋理細節的所述場景影像。The dual-sensor camera system of claim 5, wherein the processor comprises: replacing the luminance component of the image of the at least one defective area in the reference image with the image corresponding to the at least one defective area in the infrared image to generate the texture details that complement the at least one defective area of the scene image. 如請求項5所述的雙感測器攝像系統，其中所述處理器更包括： 判斷各所述紅外線影像是否包括所述至少一缺陷區域的所述紋理細節；以及 在所述紅外線影像均未包括所述紋理細節時，將所述基準影像中的所述至少一缺陷區域的影像以所述高動態範圍影像中對應於所述至少一缺陷區域的影像取代，以生成具備所述至少一缺陷區域的所述紋理細節的所述場景影像。The dual-sensor camera system of claim 5, wherein the processor further comprises: determining whether each of the infrared images includes the texture details of the at least one defective area; and When none of the infrared images includes the texture details, replace the image of the at least one defective area in the reference image with an image corresponding to the at least one defective area in the high dynamic range image, to generating the scene image with the texture details of the at least one defective area. 如請求項1所述的雙感測器攝像系統，其中所述處理器更包括： 利用一機器學習模型辨識所述色彩影像中的所述興趣對象以偵測所述特徵區域，其中所述機器學習模型是利用包括所述興趣對象的多張色彩影像以及對於各所述色彩影像中的所述興趣對象的辨識結果所訓練。The dual-sensor camera system of claim 1, wherein the processor further comprises: A machine learning model is used to identify the object of interest in the color image to detect the feature region, wherein the machine learning model uses a plurality of color images including the object of interest and for each of the color images trained on the recognition results of the object of interest. 如請求項9所述的雙感測器攝像系統，其中所述機器學習模型包括輸入層、至少一隱藏層及輸出層，所述處理器包括： 將所述色彩影像依序輸入所述輸入層，由各所述至少一隱藏層的多個神經元利用一激勵函數針對所述輸入層的輸出計算當次的輸出，並由所述輸出層將所述隱藏層當次的所述輸出轉換為所述興趣對象的預測結果； 將所述預測結果與當次輸入的所述色彩影像對應的辨識結果比較，以根據比較結果更新所述隱藏層的各所述神經元的權重；以及 重複上述步驟，訓練所述機器學習模型以辨識所述興趣對象。The dual-sensor camera system of claim 9, wherein the machine learning model includes an input layer, at least one hidden layer, and an output layer, and the processor includes: The color images are sequentially input to the input layer, and a plurality of neurons in each of the at least one hidden layer use an excitation function to calculate the current output for the output of the input layer, and the output layer will The current output of the hidden layer is converted into the prediction result of the object of interest; comparing the prediction result with the recognition result corresponding to the current input color image, so as to update the weight of each of the neurons in the hidden layer according to the comparison result; and Repeat the above steps to train the machine learning model to identify the object of interest. 一種雙感測器攝像系統的隱私保護攝像方法，所述雙感測器攝像系統包括至少一色彩感測器、至少一紅外線感測器及處理器，所述方法包括下列步驟： 控制所述至少一色彩感測器及所述至少一紅外線感測器採用適用於一攝像場景下的多個曝光條件分別擷取多張色彩影像及多張紅外線影像； 適應性選擇能顯露出所述攝像場景的細節的所述色彩影像及所述紅外線影像的組合； 根據興趣對象的至少一特徵，偵測所選擇的所述色彩影像中具有所述特徵的特徵區域；以及 融合所選擇的所述色彩影像及所述紅外線影像以生成具備所述攝像場景的所述細節的融合影像，裁切所述融合影像中的所述特徵區域的影像並以非屬於所述紅外線影像的影像取代，以生成場景影像。A privacy-preserving camera method for a dual-sensor camera system, the dual-sensor camera system includes at least one color sensor, at least one infrared sensor, and a processor, and the method includes the following steps: controlling the at least one color sensor and the at least one infrared sensor to capture a plurality of color images and a plurality of infrared images respectively using a plurality of exposure conditions suitable for a shooting scene; adaptively selecting a combination of the color image and the infrared image that can reveal details of the camera scene; according to at least one feature of the object of interest, detecting a feature area having the feature in the selected color image; and Fusing the selected color image and the infrared image to generate a fusion image with the details of the shooting scene, cropping the image of the characteristic area in the fusion image and not belonging to the infrared image image to generate the scene image. 如請求項11所述的方法，更包括： 控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間長或短的多個曝光時間擷取多張色彩影像並執行高動態範圍處理，以生成具備所述特徵區域的細節的高動態範圍影像，並使用所述高動態範圍影像取代所裁切的所述融合影像中的所述特徵區域的影像。The method according to claim 11, further comprising: Controlling the at least one color sensor to capture multiple color images with multiple exposure times that are longer or shorter than the exposure time of the selected color image and perform high dynamic range processing, so as to generate an image with the characteristic area. A detailed high dynamic range image is used, and the high dynamic range image is used to replace the image of the feature area in the cropped fusion image. 如請求項11所述的方法，其中識別所述雙感測器攝像系統的所述攝像場景的步驟包括： 控制所述至少一色彩感測器及所述至少一紅外線感測器中的至少一者採用標準曝光條件擷取所述攝像場景的至少一標準影像，並使用所述至少一標準影像識別所述攝像場景。The method of claim 11, wherein the step of identifying the camera scene of the dual-sensor camera system comprises: controlling at least one of the at least one color sensor and the at least one infrared sensor to capture at least one standard image of the camera scene using standard exposure conditions, and using the at least one standard image to identify the camera scene. 如請求項11所述的方法，其中適應性選擇能顯露出所述攝像場景的細節的所述色彩影像及所述紅外線影像的組合的步驟包括： 根據各所述色彩影像的顏色細節，選擇所述色彩影像其中之一作為基準影像； 辨識所述基準影像中缺乏紋理細節的至少一缺陷區域；以及 根據各所述紅外線影像中對應於所述至少一缺陷區域的影像的紋理細節，選擇所述紅外線影像其中之一作為用以與所述基準影像融合的所述紅外線影像。The method of claim 11, wherein the step of adaptively selecting a combination of the color image and the infrared image that reveals details of the camera scene comprises: selecting one of the color images as a reference image according to the color details of each of the color images; identifying at least one defect region in the reference image that lacks texture detail; and According to the texture details of the images corresponding to the at least one defect area in each of the infrared images, one of the infrared images is selected as the infrared image for fusion with the reference image. 如請求項14所述的方法，其中適應性選擇能顯露出所述攝像場景的細節的所述色彩影像及所述紅外線影像的組合的步驟包括： 選擇所述顏色細節最多的所述色彩影像作為所述基準影像；以及 選擇對應於所述至少一缺陷區域的影像的所述紋理細節最多的所述紅外線影像作為用以與所述基準影像融合的所述紅外線影像。The method of claim 14, wherein the step of adaptively selecting a combination of the color image and the infrared image that reveals details of the camera scene comprises: selecting the color image with the most color detail as the reference image; and The infrared image corresponding to the image of the at least one defect area with the most detailed texture is selected as the infrared image for fusion with the reference image. 如請求項14所述的方法，其中融合所選擇的所述色彩影像及所述紅外線影像，以生成具備所述攝像場景的所述細節的所述場景影像的步驟包括： 將所述基準影像中的所述至少一缺陷區域的影像的亮度分量以所述紅外線影像中對應於所述至少一缺陷區域的影像取代，以生成補足所述至少一缺陷區域的所述紋理細節的所述場景影像。The method of claim 14, wherein the step of fusing the selected color image and the infrared image to generate the scene image with the details of the camera scene comprises: replacing the luminance component of the image of the at least one defective area in the reference image with the image corresponding to the at least one defective area in the infrared image to generate the texture details that complement the at least one defective area of the scene image. 如請求項14所述的方法，其中在融合所選擇的所述色彩影像及所述紅外線影像，以生成具備所述攝像場景的所述細節的場景影像的步驟之前，所述方法更包括： 判斷各所述紅外線影像是否包括所述至少一缺陷區域的所述紋理細節；以及 在所述紅外線影像均未包括所述紋理細節時，將所述基準影像中的所述至少一缺陷區域的影像以所述高動態範圍影像中對應於所述至少一缺陷區域的影像取代，以生成具備所述至少一缺陷區域的所述紋理細節的所述場景影像。The method of claim 14, wherein before the step of fusing the selected color image and the infrared image to generate a scene image with the details of the camera scene, the method further comprises: determining whether each of the infrared images includes the texture details of the at least one defective area; and When none of the infrared images includes the texture details, replace the image of the at least one defective area in the reference image with an image corresponding to the at least one defective area in the high dynamic range image, to generating the scene image with the texture details of the at least one defective area. 如請求項11所述的方法，其中根據興趣對象的至少一特徵偵測所選擇的所述色彩影像及所述紅外線影像中具有所述特徵的特徵區域的步驟包括： 利用一機器學習模型辨識所述色彩影像中的所述興趣對象以偵測所述特徵區域，其中所述機器學習模型是利用包括所述興趣對象的多張色彩影像以及對於各所述色彩影像中的所述興趣對象的辨識結果所訓練。The method of claim 11, wherein the step of detecting the selected feature area with the feature in the color image and the infrared image according to at least one feature of the object of interest comprises: A machine learning model is used to identify the object of interest in the color image to detect the feature region, wherein the machine learning model uses a plurality of color images including the object of interest and for each of the color images trained on the recognition results of the object of interest. 如請求項18所述的方法，其中所述機器學習模型包括輸入層、至少一隱藏層及輸出層，在根據興趣對象的至少一特徵偵測所選擇的所述色彩影像及所述紅外線影像中具有所述特徵的特徵區域的步驟之前，更包括： 將所述色彩影像依序輸入所述輸入層，由各所述至少一隱藏層的多個神經元利用一激勵函數針對所述輸入層的輸出計算當次的輸出，由所述輸出層將所述隱藏層當次的所述輸出轉換為所述興趣對象的預測結果； 將所述預測結果與當次輸入的所述色彩影像對應的辨識結果比較，以根據比較結果更新所述隱藏層的各所述神經元的權重；以及 重複上述步驟，訓練所述機器學習模型以辨識所述興趣對象。The method of claim 18, wherein the machine learning model includes an input layer, at least one hidden layer, and an output layer, and in the color image and the infrared image selected by detecting at least one feature of an object of interest Before the step of having the feature area with said feature, it further includes: The color images are sequentially input to the input layer, and a plurality of neurons in each of the at least one hidden layer use an excitation function to calculate the current output for the output of the input layer, and the output layer will The current output of the hidden layer is converted into the prediction result of the object of interest; comparing the prediction result with the recognition result corresponding to the current input color image, so as to update the weight of each of the neurons in the hidden layer according to the comparison result; and Repeat the above steps to train the machine learning model to identify the object of interest. 如請求項11所述的方法，其中控制所述至少一色彩感測器採用較所選擇的所述色彩影像的曝光時間長或短的多個曝光時間擷取多張色彩影像並執行高動態範圍處理，以生成具備所述特徵區域的細節的高動態範圍影像的步驟包括： 根據所述色彩影像的所述特徵區域的所述細節選擇用以擷取所述多張色彩影像的所述曝光時間，使得所擷取的所述多張色彩影像經高動態範圍處理後，生成具備所述特徵區域的細節的所述高動態範圍影像。The method of claim 11, wherein the at least one color sensor is controlled to capture a plurality of color images with a plurality of exposure times that are longer or shorter than the exposure time of the selected color image and perform high dynamic range The step of processing to generate a high dynamic range image with details of the feature region includes: The exposure time for capturing the plurality of color images is selected according to the details of the characteristic regions of the color image, so that the captured plurality of color images are processed with high dynamic range to generate The high dynamic range image with details of the feature area.
TW109146831A 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof TWI797528B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/214,937 US11689822B2 (en) 2020-09-04 2021-03-29 Dual sensor imaging system and privacy protection imaging method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063074477P 2020-09-04 2020-09-04
US63/074,477 2020-09-04

Publications (2)

Publication Number Publication Date
TW202211161A true TW202211161A (en) 2022-03-16
TWI797528B TWI797528B (en) 2023-04-01

Family

ID=80438521

Family Applications (5)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109146831A TWI797528B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and privacy protection imaging method thereof
TW109146764A TWI764484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof
TW109146922A TWI767484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof

Family Applications Before (2)

Application Number Title Priority Date Filing Date
TW109145632A TWI767468B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof
TW109145614A TWI778476B (en) 2020-09-04 2020-12-23 Dual sensor imaging system and imaging method thereof

Family Applications After (2)

Application Number Title Priority Date Filing Date
TW109146764A TWI764484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and calibration method thereof
TW109146922A TWI767484B (en) 2020-09-04 2020-12-30 Dual sensor imaging system and depth map calculation method thereof

Country Status (2)

Country Link
CN (5) CN114143418B (en)
TW (5) TWI767468B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116091341B (en) * 2022-12-15 2024-04-02 南京信息工程大学 Exposure difference enhancement method and device for low-light image

Family Cites Families (70)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004246252A (en) * 2003-02-17 2004-09-02 Takenaka Komuten Co Ltd Apparatus and method for collecting image information
JP2005091434A (en) * 2003-09-12 2005-04-07 Noritsu Koki Co Ltd Position adjusting method and image reader with damage compensation function using the same
JP4244018B2 (en) * 2004-03-25 2009-03-25 ノーリツ鋼機株式会社 Defective pixel correction method, program, and defective pixel correction system for implementing the method
JP4341680B2 (en) * 2007-01-22 2009-10-07 セイコーエプソン株式会社 projector
US9307212B2 (en) * 2007-03-05 2016-04-05 Fotonation Limited Tone mapping for low-light video frame enhancement
EP3876510A1 (en) * 2008-05-20 2021-09-08 FotoNation Limited Capturing and processing of images using monolithic camera array with heterogeneous imagers
US8866920B2 (en) * 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
CN101404060B (en) * 2008-11-10 2010-06-30 北京航空航天大学 Human face recognition method based on visible light and near-infrared Gabor information amalgamation
US8749635B2 (en) * 2009-06-03 2014-06-10 Flir Systems, Inc. Infrared camera systems and methods for dual sensor applications
WO2010104490A1 (en) * 2009-03-12 2010-09-16 Hewlett-Packard Development Company, L.P. Depth-sensing camera system
JP5670456B2 (en) * 2009-08-25 2015-02-18 アイピーリンク・リミテッド Reduce noise in color images
US8478123B2 (en) * 2011-01-25 2013-07-02 Aptina Imaging Corporation Imaging devices having arrays of image sensors and lenses with multiple aperture sizes
JP2013115679A (en) * 2011-11-30 2013-06-10 Fujitsu General Ltd Imaging apparatus
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
TW201401186A (en) * 2012-06-25 2014-01-01 Psp Security Co Ltd System and method for identifying human face
US20150245062A1 (en) * 2012-09-25 2015-08-27 Nippon Telegraph And Telephone Corporation Picture encoding method, picture decoding method, picture encoding apparatus, picture decoding apparatus, picture encoding program, picture decoding program and recording medium
KR102070778B1 (en) * 2012-11-23 2020-03-02 엘지전자 주식회사 Rgb-ir sensor with pixels array and apparatus and method for obtaining 3d image using the same
EP2936799B1 (en) * 2012-12-21 2018-10-17 Flir Systems, Inc. Time spaced infrared image enhancement
TWM458748U (en) * 2012-12-26 2013-08-01 Chunghwa Telecom Co Ltd Image type depth information retrieval device
JP6055681B2 (en) * 2013-01-10 2016-12-27 株式会社 日立産業制御ソリューションズ Imaging device
CN104661008B (en) * 2013-11-18 2017-10-31 深圳中兴力维技术有限公司 The treating method and apparatus that color image quality is lifted under low light conditions
CN104021548A (en) * 2014-05-16 2014-09-03 中国科学院西安光学精密机械研究所 Method for acquiring 4D scene information
US9516295B2 (en) * 2014-06-30 2016-12-06 Aquifi, Inc. Systems and methods for multi-channel imaging based on multiple exposure settings
JP6450107B2 (en) * 2014-08-05 2019-01-09 キヤノン株式会社 Image processing apparatus, image processing method, program, and storage medium
JP6597636B2 (en) * 2014-12-10 2019-10-30 ソニー株式会社 Imaging apparatus, imaging method, program, and image processing apparatus
JP6185213B2 (en) * 2015-03-31 2017-08-23 富士フイルム株式会社 Imaging apparatus, image processing method of imaging apparatus, and program
WO2016192437A1 (en) * 2015-06-05 2016-12-08 深圳奥比中光科技有限公司 3d image capturing apparatus and capturing method, and 3d image system
JP2017011634A (en) * 2015-06-26 2017-01-12 キヤノン株式会社 Imaging device, control method for the same and program
CN105049829B (en) * 2015-07-10 2018-12-25 上海图漾信息科技有限公司 Optical filter, imaging sensor, imaging device and 3-D imaging system
CN105069768B (en) * 2015-08-05 2017-12-29 武汉高德红外股份有限公司 A kind of visible images and infrared image fusion processing system and fusion method
US10523855B2 (en) * 2015-09-24 2019-12-31 Intel Corporation Infrared and visible light dual sensor imaging system
TW201721269A (en) * 2015-12-11 2017-06-16 宏碁股份有限公司 Automatic exposure system and auto exposure method thereof
JP2017112401A (en) * 2015-12-14 2017-06-22 ソニー株式会社 Imaging device, apparatus and method for image processing, and program
CN206117865U (en) * 2016-01-16 2017-04-19 上海图漾信息科技有限公司 Range data monitoring device
JP2017163297A (en) * 2016-03-09 2017-09-14 キヤノン株式会社 Imaging apparatus
KR101747603B1 (en) * 2016-05-11 2017-06-16 재단법인 다차원 스마트 아이티 융합시스템 연구단 Color night vision system and operation method thereof
CN106815826A (en) * 2016-12-27 2017-06-09 上海交通大学 Night vision image Color Fusion based on scene Recognition
CN108280807A (en) * 2017-01-05 2018-07-13 浙江舜宇智能光学技术有限公司 Monocular depth image collecting device and system and its image processing method
US11145077B2 (en) * 2017-02-06 2021-10-12 Photonic Sensors & Algorithms, S.L. Device and method for obtaining depth information from a scene
CN108419062B (en) * 2017-02-10 2020-10-02 杭州海康威视数字技术股份有限公司 Image fusion apparatus and image fusion method
CN109474770B (en) * 2017-09-07 2021-09-14 华为技术有限公司 Imaging device and imaging method
CN109712102B (en) * 2017-10-25 2020-11-27 杭州海康威视数字技术股份有限公司 Image fusion method and device and image acquisition equipment
CN107846537B (en) * 2017-11-08 2019-11-26 维沃移动通信有限公司 A kind of CCD camera assembly, image acquiring method and mobile terminal
CN112788249B (en) * 2017-12-20 2022-12-06 杭州海康威视数字技术股份有限公司 Image fusion method and device, electronic equipment and computer readable storage medium
US10748247B2 (en) * 2017-12-26 2020-08-18 Facebook, Inc. Computing high-resolution depth images using machine learning techniques
US10757320B2 (en) * 2017-12-28 2020-08-25 Waymo Llc Multiple operating modes to expand dynamic range
TWI661726B (en) * 2018-01-09 2019-06-01 呂官諭 Image sensor with enhanced image recognition and application
CN110136183B (en) * 2018-02-09 2021-05-18 华为技术有限公司 Image processing method and device and camera device
CN108965654B (en) * 2018-02-11 2020-12-25 浙江宇视科技有限公司 Double-spectrum camera system based on single sensor and image processing method
CN110572583A (en) * 2018-05-18 2019-12-13 杭州海康威视数字技术股份有限公司 method for shooting image and camera
CN108961195B (en) * 2018-06-06 2021-03-23 Oppo广东移动通信有限公司 Image processing method and device, image acquisition device, readable storage medium and computer equipment
JP6574878B2 (en) * 2018-07-19 2019-09-11 キヤノン株式会社 Image processing apparatus, image processing method, imaging apparatus, program, and storage medium
JP7254461B2 (en) * 2018-08-01 2023-04-10 キヤノン株式会社 IMAGING DEVICE, CONTROL METHOD, RECORDING MEDIUM, AND INFORMATION PROCESSING DEVICE
CN109035193A (en) * 2018-08-29 2018-12-18 成都臻识科技发展有限公司 A kind of image processing method and imaging processing system based on binocular solid camera
PL3852350T3 (en) * 2018-09-14 2024-06-10 Zhejiang Uniview Technologies Co., Ltd. Automatic exposure method and apparatus for dual-light image, and dual-light image camera and machine storage medium
JP2020052001A (en) * 2018-09-28 2020-04-02 パナソニックＩｐマネジメント株式会社 Depth acquisition device, depth acquisition method, and program
US11176694B2 (en) * 2018-10-19 2021-11-16 Samsung Electronics Co., Ltd Method and apparatus for active depth sensing and calibration method thereof
CN109636732B (en) * 2018-10-24 2023-06-23 深圳先进技术研究院 Hole repairing method of depth image and image processing device
CN110248105B (en) * 2018-12-10 2020-12-08 浙江大华技术股份有限公司 Image processing method, camera and computer storage medium
US11120536B2 (en) * 2018-12-12 2021-09-14 Samsung Electronics Co., Ltd Apparatus and method for determining image sharpness
WO2020168465A1 (en) * 2019-02-19 2020-08-27 华为技术有限公司 Image processing device and method
US10972649B2 (en) * 2019-02-27 2021-04-06 X Development Llc Infrared and visible imaging system for device identification and tracking
JP7316809B2 (en) * 2019-03-11 2023-07-28 キヤノン株式会社 Image processing device, image processing device control method, system, and program
CN110349117B (en) * 2019-06-28 2023-02-28 重庆工商大学 Infrared image and visible light image fusion method and device and storage medium
CN110706178B (en) * 2019-09-30 2023-01-06 杭州海康威视数字技术股份有限公司 Image fusion device, method, equipment and storage medium
CN111524175A (en) * 2020-04-16 2020-08-11 东莞市东全智能科技有限公司 Depth reconstruction and eye movement tracking method and system for asymmetric multiple cameras
CN111540003A (en) * 2020-04-27 2020-08-14 浙江光珀智能科技有限公司 Depth image generation method and device
CN111586314B (en) * 2020-05-25 2021-09-10 浙江大华技术股份有限公司 Image fusion method and device and computer storage medium
CN111383206B (en) * 2020-06-01 2020-09-29 浙江大华技术股份有限公司 Image processing method and device, electronic equipment and storage medium
IN202021032940A (en) * 2020-07-31 2020-08-28 .Us Priyadarsan

Also Published As

Publication number Publication date
CN114143420A (en) 2022-03-04
TWI767468B (en) 2022-06-11
TWI764484B (en) 2022-05-11
TW202211674A (en) 2022-03-16
CN114143418A (en) 2022-03-04
TWI778476B (en) 2022-09-21
CN114143418B (en) 2023-12-01
CN114143443A (en) 2022-03-04
CN114143419B (en) 2023-12-26
TW202211673A (en) 2022-03-16
TW202211165A (en) 2022-03-16
CN114143421A (en) 2022-03-04
TWI797528B (en) 2023-04-01
CN114143443B (en) 2024-04-05
CN114143421B (en) 2024-04-05
TW202211160A (en) 2022-03-16
TWI767484B (en) 2022-06-11
CN114143419A (en) 2022-03-04
CN114143420B (en) 2024-05-03

Similar Documents

Publication Publication Date Title
US11699219B2 (en) System, method, and computer program for capturing an image with correct skin tone exposure
US11689822B2 (en) Dual sensor imaging system and privacy protection imaging method thereof
US20150170389A1 (en) Automatic selection of optimum algorithms for high dynamic range image processing based on scene classification
US20080144946A1 (en) Method of chromatic classification of pixels and method of adaptive enhancement of a color image
CN106464816A (en) Image processing apparatus and image processing method thereof
JP6685188B2 (en) Imaging device, image processing device, control method thereof, and program
CN106878605A (en) The method and electronic equipment of a kind of image generation based on electronic equipment
US11496694B2 (en) Dual sensor imaging system and imaging method thereof
TWI797528B (en) Dual sensor imaging system and privacy protection imaging method thereof
US11496660B2 (en) Dual sensor imaging system and depth map calculation method thereof
US11568526B2 (en) Dual sensor imaging system and imaging method thereof
US11869224B2 (en) Method and system for establishing light source information prediction model
Mangiat et al. Automatic scene relighting for video conferencing
US12020455B2 (en) Systems and methods for high dynamic range image reconstruction
WO2023070412A1 (en) Imaging method and imaging device for wading scene
Vallikumari et al. HDR scene detection and capturing strategy
US20220294968A1 (en) Systems and methods for high dynamic range image reconstruction
JP2003259231A (en) Automatic exposure controller and program thereof
CN114697483A (en) Device and method for shooting under screen based on compressed sensing white balance algorithm
CN114630095A (en) Automatic white balance method and device for target scene image and terminal
Kumari et al. Optimal exposure sets for high dynamic range scenes
TW201733349A (en) An image processing method and an electric device having a photograph function