WO2020168465A1

WO2020168465A1 - Image processing device and method

Info

Publication number: WO2020168465A1
Application number: PCT/CN2019/075472
Authority: WO
Inventors: 涂娇姣; 杨红明
Original assignee: 华为技术有限公司
Priority date: 2019-02-19
Filing date: 2019-02-19
Publication date: 2020-08-27
Also published as: CN113170048A

Abstract

Disclosed in the embodiments of the present application are an image processing device and method. The image processing device comprises: a first receiving interface, a second receiving interface and a processor. The image processing device can receive two image signals of a target scene. The first image signal is an infrared light image signal of high-resolution, comprising more luminance details. The second image signal is a visible light image signal of low-resolution, and a single pixel contains more chrominance information. The first image information including more luminance details and the second image information including more chrominance information are fused, to obtain a fused image, improving the detail presentation and luminance signal-to-noise ratio of the image, and retaining color information of the image.

Description

一种图像处理的装置和方法Image processing device and method

技术领域Technical field

本申请涉及图像处理领域，尤其涉及一种图像处理的装置和方法。This application relates to the field of image processing, and in particular to an image processing device and method.

背景技术Background technique

当图像传感器的感光能力有限时，在正常照度下图像传感器能够获得高质量的图像，但是在低照场景下，由于可见光的进光量严重不足，图像传感器获取的图像的亮度和色度的信噪比急剧下降，细节损失严重，颜色欠佳，导致图像质量严重恶化。在低照场景下获取的图像无论对于人眼主观观察还是机器识别场景，都无法满足要求。When the photosensitive ability of the image sensor is limited, the image sensor can obtain high-quality images under normal illuminance, but in low-light scenes, due to the serious shortage of visible light, the brightness and chromaticity of the image obtained by the image sensor are signal-to-noise The ratio drops sharply, the detail loss is serious, and the color is not good, resulting in serious deterioration of image quality. Images acquired in low-light scenes cannot meet the requirements for subjective observation by human eyes or machine recognition scenes.

为了提升低照场景下图像的质量，最常见的方案是采用图像处理技术对图像进行优化，例如降噪、增强、高动态范围处理等，然而，由于低照场景下图像传感器获取的图像本身包含的信息有限，图像处理技术并不能从本质上提升图像的质量。另外一种解决方案，在低照场景下，将图像传感器切换成红外补光模式，虽然可以大幅度提高亮度感光能力，但是图像的色彩信息全部丢失。In order to improve the quality of images in low-light scenes, the most common solution is to use image processing technology to optimize the images, such as noise reduction, enhancement, high dynamic range processing, etc. However, the images acquired by the image sensor in low-light scenes contain The information is limited, and image processing technology cannot essentially improve image quality. Another solution is to switch the image sensor to infrared fill light mode in low-light scenes. Although the brightness sensitivity can be greatly improved, the color information of the image is completely lost.

发明内容Summary of the invention

本申请实施例提供一种图像处理的装置和方法，提升了低照场景下获取的图像的质量。The embodiments of the present application provide an image processing apparatus and method, which improve the quality of images acquired in low-light scenes.

本申请第一方面提供了一种图像处理的装置，该装置包括：第一接收接口、第二接收接口和处理器；该第一接收接口，用于接收目标场景的第一图像信号，该第一图像信号包括第一亮度信息，该第一亮度信息是携带有红外光信息的亮度信息，该第一图像信号具有第一分辨率；该第二接收接口，用于接收该目标场景的第二图像信号，该第二图像信号包括第二亮度信息和第二色度信息，该第二亮度信息是排除了红外光信息的亮度信息，该第二图像信号具有第二分辨率，该第一分辨率高于该第二分辨率；该处理器，用于融合该第一图像信号和该第二图像信号，得到融合图像。The first aspect of the present application provides an image processing device, which includes: a first receiving interface, a second receiving interface, and a processor; the first receiving interface is used to receive a first image signal of a target scene, and the An image signal includes first brightness information, the first brightness information is brightness information carrying infrared light information, the first image signal has a first resolution; the second receiving interface is used to receive a second image of the target scene Image signal, the second image signal includes second luminance information and second chrominance information, the second luminance information is luminance information excluding infrared light information, the second image signal has a second resolution, the first resolution The rate is higher than the second resolution; the processor is used to fuse the first image signal and the second image signal to obtain a fused image.

本申请实施例提供的图像处理的装置，可以接收目标场景的两路图像信号，其中第一图像信号包括高分辨率的红外光图像信号，包含更多的亮度细节，第二图像信号为低分辨率的可见光图像信号，单个像素包含更多的色度信息，将包含更多的亮度细节的第一图像信息和包含更多色度信息的第二图像信息融合，得到融合图像，既提升了图像的细节表现和亮度信噪比，还很好保留了图像的色彩信息。融合图像的细节更丰富、信噪比更高、颜色表现更佳。The image processing device provided by the embodiment of the application can receive two image signals of the target scene, where the first image signal includes a high-resolution infrared light image signal containing more brightness details, and the second image signal is a low-resolution image signal. A single pixel contains more chromaticity information. The first image information containing more brightness details and the second image information containing more chromaticity information are fused to obtain a fused image, which improves the image The detailed performance and brightness signal-to-noise ratio, and also retain the color information of the image. The fusion image has richer details, higher signal-to-noise ratio, and better color performance.

在一种可能的实施方式中，该处理器具体用于：对该第二图像信号进行上采样处理，得到第三图像信号，该第三图像信号具有该第一分辨率；融合该第一图像信号和该第三图像信号，得到该融合图像。In a possible implementation, the processor is specifically configured to: perform up-sampling processing on the second image signal to obtain a third image signal, the third image signal having the first resolution; and to fuse the first image Signal and the third image signal to obtain the fused image.

在一种可能的实施方式中，该第三图像信号包括第三亮度信息，该融合图像包括融合亮度信息，该处理器具体用于：获取该第一亮度信息的高频细节信息，该第一亮度信息的高频细节信息包括该第一图像信号的亮度细节信息；对该第三亮度信息进行去噪处理，以滤除该第三亮度信息的高频噪声，得到第四亮度信息；叠加该第一亮度信息的高频细节信息和该第四亮度信息，得到该融合亮度信息。In a possible implementation manner, the third image signal includes third brightness information, the fused image includes fused brightness information, and the processor is specifically configured to: obtain high-frequency detail information of the first brightness information, and the first The high frequency detail information of the brightness information includes the brightness detail information of the first image signal; denoising processing is performed on the third brightness information to filter the high frequency noise of the third brightness information to obtain the fourth brightness information; The high-frequency detail information of the first brightness information and the fourth brightness information obtain the fused brightness information.

在一种可能的实施方式中，该处理器具体用于：对该第一亮度信息进行双边滤波，得到滤波后的第一亮度信息；该第一亮度信息减去该滤波后的第一亮度信息，得到该第一亮度信息的高频细节信号。In a possible implementation manner, the processor is specifically configured to: perform bilateral filtering on the first brightness information to obtain filtered first brightness information; and subtract the filtered first brightness information from the first brightness information , To obtain the high-frequency detail signal of the first brightness information.

在一种可能的实施方式中，该处理器具体用于：对该第三亮度信息进行双边滤波，以得到该第四亮度信息。In a possible implementation manner, the processor is specifically configured to: perform bilateral filtering on the third brightness information to obtain the fourth brightness information.

在一种可能的实施方式中，该第三图像信号还包括第三色度信息，该融合图像还包括融合色度信息，该处理器具体还用于：从该第三图像信号中提取该第三色度信息；对该第三色度信息进行颜色校正，得到该融合色度信息。In a possible implementation manner, the third image signal further includes third chrominance information, the fused image further includes fused chrominance information, and the processor is specifically further configured to: extract the first image signal from the third image signal. Three chromaticity information; perform color correction on the third chromaticity information to obtain the fused chromaticity information.

在一种可能的实施方式中，该处理器具体用于：根据该融合亮度信息与该第三亮度信息的差值对该第三色度信息进行补偿，以得到该融合色度信息。In a possible implementation manner, the processor is specifically configured to: compensate the third chrominance information according to the difference between the fused luminance information and the third luminance information to obtain the fused chrominance information.

在一种可能的实施方式中，该处理器具体还用于：组合该融合亮度信息和该融合色度信息，得到该融合图像。In a possible implementation manner, the processor is specifically further configured to: combine the fused brightness information and the fused chrominance information to obtain the fused image.

在一种可能的实施方式中，该装置还包括：第一图像传感器和第二图像传感器，该第一图像传感器具有该第一分辨率，该第二图像传感器具有该第二分辨率；该第一图像传感器，用于工作在红外补光模式，以生成该第一图像信号；该第二图像传感器，用于工作在红外光截止模式，以生成该第二图像信号。In a possible implementation manner, the device further includes: a first image sensor and a second image sensor, the first image sensor having the first resolution, the second image sensor having the second resolution; An image sensor is used to work in an infrared light supplement mode to generate the first image signal; the second image sensor is used to work in an infrared light cut-off mode to generate the second image signal.

本申请实施例提供的图像处理的装置，包括两个分辨率不同的图像传感器，在低照场景下，高分辨率的图像传感器切换为红外补光模式，图像传感器感光红外光生成的图像具有更多的细节信息，低分辨率的图像传感器获取可见光、截止红外光，避免红外光影响图像的颜色信息，由于低分辨率的图像传感器每个像素的尺寸更大，与等面积高分辨率的图像传感器相比，单个像素的感光能力更强、捕获的光子更多，可以获取更好的色度信息，也即，高分辨率图像传感器由于红外光补光获得了图像更多细节信息，低分辨率图像传感器由于单个像素尺寸面积更大获取图像更多颜色信息，将高分辨率和低分辨率图像传感器获取的两路信号进行融合，既提升了图像的细节表现和信噪比，还很好保留了图像的色彩信息。The image processing device provided by the embodiment of the application includes two image sensors with different resolutions. In a low-light scene, the high-resolution image sensor is switched to the infrared supplement light mode, and the image generated by the image sensor's photosensitive infrared light has more More detailed information, the low-resolution image sensor acquires visible light and cuts off infrared light to prevent infrared light from affecting the color information of the image. Because the size of each pixel of the low-resolution image sensor is larger, and the same area high-resolution image Compared with the sensor, a single pixel has stronger light sensitivity and captures more photons, which can obtain better chromaticity information. That is, high-resolution image sensors can obtain more detailed information of the image due to infrared light supplementation, and low resolution The high-speed image sensor can obtain more color information of the image due to the larger size and area of a single pixel, and fusion of the two signals obtained by the high-resolution and low-resolution image sensors, which not only improves the detail performance and signal-to-noise ratio of the image, but also very good The color information of the image is preserved.

在一种可能的实施方式中，该第一图像传感器和该第二图像传感器具有相同的靶面尺寸。In a possible implementation manner, the first image sensor and the second image sensor have the same target surface size.

本申请实施例提供的图像处理的装置，包括两个靶面尺寸相同、分辨率不同的图像传感器，在低照场景下，高分辨率的图像传感器切换为红外补光模式，图像传感器感光红外光生成的图像具有更多的细节信息，低分辨率的图像传感器获取可见光、截止红外光，避免红外光影响图像的颜色信息，由于低分辨率的图像传感器每个像素的尺寸更大，与等面积高分辨率的图像传感器相比，单个像素的感光能力更强、捕获的光子更多，可以获取更好的色度信息，也即，高分辨率图像传感器由于红外光补光获得了图像更多细节信息，低分辨率图像传感器由于单个像素尺寸面积更大获取图像更多颜色信息，将高分辨率和低分辨率图像传感器获取的两路信号进行融合，既提升了图像的细节表现和信噪比，还很好保留了图像的色彩信息。The image processing device provided by the embodiment of the application includes two image sensors with the same target surface size and different resolutions. In a low-light scene, the high-resolution image sensor is switched to the infrared supplement light mode, and the image sensor receives infrared light. The generated image has more detailed information. The low-resolution image sensor acquires visible light and cuts off the infrared light to prevent infrared light from affecting the color information of the image. Because the size of each pixel of the low-resolution image sensor is larger and equal to the area Compared with a high-resolution image sensor, a single pixel has stronger light sensitivity and captures more photons, and can obtain better chromaticity information. That is, a high-resolution image sensor can obtain more images due to infrared light. Detailed information, low-resolution image sensors can obtain more color information of the image due to the larger size and area of a single pixel. The two-channel signals obtained by the high-resolution and low-resolution image sensors are fused, which not only improves the detail performance and signal noise of the image Compared with, the color information of the image is well preserved.

在一种可能的实施方式中，该第一图像传感器为第一RGB Sensor，该第二图像传感器为第二RGB Sensor。In a possible implementation manner, the first image sensor is a first RGB Sensor, and the second image sensor is a second RGB Sensor.

本申请实施例提供的图像处理的装置，包括两个靶面尺寸相同、分辨率不同的RGB Sensor，在低照场景下，高分辨率的RGB Sensor切换为红外补光模式，RGB Sensor感光红外光生成的图像具有更多的细节信息，低分辨率的RGB Sensor获取可见光、截止红外光，避免红外光影响图像的颜色信息，由于低分辨率的RGB Sensor每个像素的尺寸更大，与等面积高分辨率的RGB Sensor相比，单个像素的感光能力更强、捕获的光子更多，可以获取更好的RGB颜色信息，也即，高分辨率RGB Sensor由于红外光补光获得了图像更多细节信息，低分辨率RGB Sensor由于单个像素尺寸面积更大获取图像更多颜色信息，将高分辨率和低分辨率RGB Sensor获取的两路信号进行融合，既提升了图像的细节表现和信噪比，还很好保留了图像的色彩信息。The image processing device provided by the embodiment of the present application includes two RGB Sensors with the same target surface size and different resolutions. In a low-light scene, the high-resolution RGB Sensor is switched to the infrared supplement light mode, and the RGB Sensor photosensitive infrared light The generated image has more detailed information. The low-resolution RGB Sensor acquires visible light and cuts off the infrared light to prevent infrared light from affecting the color information of the image. Because the low-resolution RGB Sensor has a larger size of each pixel and equal area Compared with the high-resolution RGB Sensor, a single pixel has stronger light sensitivity and captures more photons, which can obtain better RGB color information. That is, the high-resolution RGB Sensor can obtain more images due to the infrared light supplement. Detailed information, low-resolution RGB Sensor. Because a single pixel has a larger size and area to obtain more color information of the image, the two-channel signal obtained by the high-resolution and low-resolution RGB Sensor is fused, which improves the detail performance and signal noise of the image. Compared with, the color information of the image is well preserved.

在一种可能的实施方式中，该装置还包括：第一滤光模式切换器，用于将该第一图像传感器切换到该红外光补光模式。In a possible implementation manner, the device further includes: a first filter mode switch, configured to switch the first image sensor to the infrared light supplementary light mode.

在一种可能的实施方式中，该第一滤光模式切换器包括：全透光谱滤光片和红外截止滤光片；该第一滤光模式切换器，具体用于切换到该全透光谱滤光片，以使得该第一图像传感器感光红外光信号生成该红外光图像信号。In a possible implementation manner, the first filter mode switch includes: a full transmission spectrum filter and an infrared cut filter; the first filter mode switch is specifically configured to switch to the full transmission spectrum A filter, so that the first image sensor receives the infrared light signal to generate the infrared light image signal.

在一种可能的实施方式中，该装置还包括：红外灯，用于在该低照场景下提供该红外光信号。In a possible implementation manner, the device further includes: an infrared lamp for providing the infrared light signal in the low-light scene.

在一种可能的实施方式中，该红外光信号为该低照场景下红外灯提供的补光信号。In a possible implementation, the infrared light signal is a supplementary light signal provided by an infrared lamp in the low-light scene.

在一种可能的实施方式中，该装置还包括：第二滤光模式切换器，用于将该第二图像传感器切换到该红外光截止模式。In a possible implementation manner, the device further includes: a second filter mode switch, configured to switch the second image sensor to the infrared light cutoff mode.

在一种可能的实施方式中，该第二滤光模式切换器包括：该全透光谱滤光片和该红外截止滤光片；该第二滤光模式切换器，具体用于在该切换到该红外截止滤光片，以使得该第二图像传感器感光可见光信号生成该可见光图像信号。In a possible implementation manner, the second filter mode switch includes: the full transmission spectrum filter and the infrared cut filter; the second filter mode switch is specifically used to switch to The infrared cut-off filter enables the second image sensor to receive the visible light signal to generate the visible light image signal.

在一种可能的实施方式中，该装置还包括：镜头和分光器；该镜头，用于接收该目标场景的光信号；该分光器，用于将该目标场景的光信号分为第一光信号和第二光信号，该第一光信号为送给该第一图像传感器的光信号，该第二光信号为送给该第二图像传感器的光信号。In a possible implementation manner, the device further includes: a lens and a beam splitter; the lens is used to receive the optical signal of the target scene; the beam splitter is used to divide the optical signal of the target scene into the first light Signal and a second light signal, where the first light signal is a light signal sent to the first image sensor, and the second light signal is a light signal sent to the second image sensor.

在一种可能的实施方式中，该装置还包括：第一镜头，用于接收该目标场景的第一光信号，该第一光信号为送给该第一图像传感器的光信号；第二镜头，用于接收该目标场景的第二光信号，该第二光信号为送给该第二图像传感器的光信号。In a possible implementation manner, the device further includes: a first lens for receiving a first light signal of the target scene, where the first light signal is a light signal sent to the first image sensor; and a second lens , For receiving a second light signal of the target scene, where the second light signal is a light signal sent to the second image sensor.

在一种可能的实施方式中，该处理器还用于：对该第一图像信号和该第二图像信号进行对齐处理。In a possible implementation manner, the processor is further configured to: perform alignment processing on the first image signal and the second image signal.

在一种可能的实施方式中，该处理器还用于：对该第一图像信号或该第二图像信号进行去马赛克处理、颜色空间转换、降噪处理、对比度处理、图像增强或动态范围处理中的至少一项。In a possible implementation manner, the processor is further configured to: perform demosaicing, color space conversion, noise reduction, contrast processing, image enhancement, or dynamic range processing on the first image signal or the second image signal At least one of.

在一种可能的实施方式中，该处理器还用于：获取该第二图像的第二亮度信息，对该第二亮度信息进行上采样处理，得到该第三亮度信息，该第三亮度信息具有该第一分辨率。In a possible implementation manner, the processor is further configured to: obtain second brightness information of the second image, perform up-sampling processing on the second brightness information, to obtain the third brightness information, and the third brightness information With this first resolution.

在一种可能的实施方式中，该处理器还用于：获取该第二图像的第二色度信息，对该第二色度信息进行上采样处理，得到该第三色度信息，该第三色度信息具有该第一分辨率。In a possible implementation manner, the processor is further configured to: obtain second chrominance information of the second image, perform up-sampling processing on the second chrominance information, to obtain the third chrominance information, and The trichromatic information has this first resolution.

在一种可能的实施方式中，该第一分辨率和该第二分辨率的比例为：4:1。In a possible implementation manner, the ratio of the first resolution to the second resolution is 4:1.

在一种可能的实施方式中，该第一图像传感器工作在该红外光截止模式，以感光可见光信号，并生成可见光图像信号；关闭该第二图像传感器。In a possible implementation manner, the first image sensor works in the infrared light cutoff mode to sense visible light signals and generate visible light image signals; turn off the second image sensor.

在一种可能的实施方式中，该第一图像传感器工作在该红外光截止模式，生成第一可见光图像信号，该第二图像传感器工作在该红外光截止模式，生成第二可见光图像信号，融合该第一可见光图像信号和该第二可见光图像信号，得到融合可见光图像信号。In a possible implementation, the first image sensor works in the infrared light cut-off mode to generate a first visible light image signal, and the second image sensor works in the infrared light cut-off mode to generate a second visible light image signal, and fuse The first visible light image signal and the second visible light image signal obtain a fused visible light image signal.

本申请第二方面提供了一种图像处理的方法，该方法包括：接收目标场景的第一图像信号，该第一图像信号包括第一亮度信息，该第一亮度信息是携带有红外光信息的亮度信息，该第一图像信号具有第一分辨率；接收该目标场景的第二图像信号，该第二图像信号包括第二亮度信息和第二色度信息，该第二亮度信息是排除了红外光信息的亮度信息，该第二图像信号具有第二分辨率，该第一分辨率高于该第二分辨率；融合该第一图像信号和该第二图像信号，得到融合图像。A second aspect of the present application provides an image processing method. The method includes: receiving a first image signal of a target scene, the first image signal includes first brightness information, and the first brightness information carries infrared light information. Brightness information, the first image signal has a first resolution; receiving a second image signal of the target scene, the second image signal includes second brightness information and second chrominance information, the second brightness information excludes infrared The brightness information of the light information, the second image signal has a second resolution, and the first resolution is higher than the second resolution; the first image signal and the second image signal are fused to obtain a fused image.

本申请实施例提供的图像处理的方法，接收目标场景的两路图像信号，其中第一图像信号包括高分辨率的红外光图像信号，包含更多的亮度细节，第二图像信号为低分辨率的可见光图像信号，单个像素包含更多的色度信息，将包含更多的亮度细节的第一图像信息和包含更多色度信息的第二图像信息融合，得到融合图像，既提升了图像的细节表现和亮度信噪比，还很好保留了图像的色彩信息。融合图像的细节更丰富、信噪比更高、颜色表现更佳。The image processing method provided by the embodiment of the present application receives two image signals of a target scene, wherein the first image signal includes a high-resolution infrared light image signal, which contains more brightness details, and the second image signal is a low-resolution image signal. In the visible light image signal, a single pixel contains more chromaticity information. The first image information containing more brightness details and the second image information containing more chromaticity information are fused to obtain a fused image, which improves the image quality. The detail performance and brightness signal-to-noise ratio also preserve the color information of the image. The fusion image has richer details, higher signal-to-noise ratio, and better color performance.

在一种可能的实施方式中，该融合该第一图像信号和该第二图像信号，得到融合图像，具体包括：对该第二图像信号进行上采样处理，得到第三图像信号，该第三图像信号具有该第一分辨率；融合该第一图像信号和该第三图像信号，得到该融合图像。In a possible implementation manner, the fusing the first image signal and the second image signal to obtain a fused image specifically includes: performing up-sampling processing on the second image signal to obtain a third image signal. The image signal has the first resolution; the first image signal and the third image signal are fused to obtain the fused image.

由于第一图像信号和第二图像信号的分辨率不同，在进行融合之前，对低分辨率的第二图像信号进行上采样，使得处理后的第二图像信号与第一图像信号具有相同的分辨率。Because the resolutions of the first image signal and the second image signal are different, before the fusion, the low-resolution second image signal is up-sampled, so that the processed second image signal has the same resolution as the first image signal rate.

在一种可能的实施方式中，该第三图像信号包括第三亮度信息,该融合图像包括融合亮度信息，该融合该第一图像信号和该第三图像信号，得到该融合图像，具体包括：获取该第一亮度信息的高频细节信息，该第一亮度信息的高频细节信息包括该第一图像信号的亮度细节信息；对该第三亮度信息进行去噪处理，以滤除该第三亮度信息的高频噪声，得到第四亮度信息；叠加该第一亮度信息的高频细节信息和该第四亮度信息，得到该融合亮度信息。In a possible implementation manner, the third image signal includes third brightness information, the fusion image includes fusion brightness information, and the fusion of the first image signal and the third image signal to obtain the fusion image specifically includes: Acquire high-frequency detail information of the first brightness information, the high-frequency detail information of the first brightness information includes the brightness detail information of the first image signal; perform denoising processing on the third brightness information to filter out the third brightness information The high frequency noise of the brightness information obtains the fourth brightness information; the high frequency detail information of the first brightness information and the fourth brightness information are superimposed to obtain the fused brightness information.

第四亮度信息为可见光图像信号滤除高频噪声之后得到的亮度信息，包含了可见光图像信号的中低频信息和平均亮度信息；由于红外光图像信号的信噪比很高，噪声很少，第一亮度信息的高频细节信息主要为图像的亮度细节信息；由于第一亮度信息的高频细节信息主要包含红外光图像信号的高频亮度细节，第四亮度信息主要包含了可见光图像信号的中低频信息和平均亮度信息，结合两者得到融合亮度信息，一方面可以尽量保持融合后的亮度信息的平均亮度与可见光图像的平均亮度接近，另一方面也包含了来自红外光图像信号中的更多细节信息。The fourth brightness information is the brightness information obtained after the visible light image signal is filtered out of high-frequency noise, which contains the medium and low frequency information and the average brightness information of the visible light image signal; since the signal-to-noise ratio of the infrared light image signal is very high, the noise is very small. The high-frequency detail information of the first brightness information is mainly the brightness detail information of the image; because the high-frequency detail information of the first brightness information mainly contains the high-frequency brightness details of the infrared light image signal, the fourth brightness information mainly contains the middle of the visible light image signal. Low-frequency information and average brightness information are combined to obtain fused brightness information. On the one hand, the average brightness of the fused brightness information can be kept close to the average brightness of the visible light image. On the other hand, it also contains more information from the infrared light image signal. More detailed information.

在一种可能的实施方式中，该获取该第一亮度信息的高频细节信息，具体包括：对该第一亮度信息进行滤波处理，得到滤波后的第一亮度信息；该第一亮度信息减去该滤波后的第一亮度信息，得到该第一亮度信息的高频细节信息。In a possible implementation manner, the acquiring high-frequency detail information of the first brightness information specifically includes: filtering the first brightness information to obtain filtered first brightness information; and subtracting the first brightness information Remove the filtered first brightness information to obtain high-frequency detail information of the first brightness information.

在一种可能的实施方式中，对该第三亮度信息进行去噪处理，以滤除该第三亮度信息的高频噪声，得到第四亮度信息，具体包括：对该第三亮度信息进行双边滤波，以得到该第四亮度信息。In a possible implementation manner, performing denoising processing on the third brightness information to filter out high-frequency noise of the third brightness information to obtain the fourth brightness information specifically includes: performing bilaterally on the third brightness information Filtering to obtain the fourth brightness information.

在一种可能的实施方式中，该第三图像信号还包括第三色度信息，该融合图像还包括融合色度信息，该融合该第一图像信号和该第三图像信号，得到该融合图像，具体还包括：从该第三图像信号中提取该第三色度信息；对该第三色度信息进行颜色校正，得到该融合色度信息。In a possible implementation, the third image signal further includes third chrominance information, the fused image further includes fused chrominance information, and the first image signal and the third image signal are fused to obtain the fused image Specifically, it further includes: extracting the third chromaticity information from the third image signal; performing color correction on the third chromaticity information to obtain the fused chromaticity information.

在一种可能的实施方式中，该对该第三色度信息进行颜色校正，得到该融合色度信息，具体包括：根据该融合亮度信息与该第三亮度信息的差值对该第三色度信息进行补偿，以得到该融合色度信息。In a possible implementation manner, performing color correction on the third chrominance information to obtain the fused chrominance information specifically includes: according to the difference between the fused brightness information and the third brightness information, the third color The degree information is compensated to obtain the fusion chrominance information.

在一种可能的实施方式中，该融合该第一图像信号和该第三图像信号，得到该融合图像，具体还包括：组合该融合亮度信息和该融合色度信息，得到该融合图像。In a possible implementation manner, the fusing the first image signal and the third image signal to obtain the fused image specifically further includes: combining the fused brightness information and the fused chrominance information to obtain the fused image.

在一种可能的实施方式中，该方法还包括：对该第一图像信号或该第二图像信号进行去马赛克处理、颜色空间转换、降噪处理、对比度处理、图像增强或动态范围处理中的至少一项。In a possible implementation manner, the method further includes: performing demosaicing processing, color space conversion, noise reduction processing, contrast processing, image enhancement, or dynamic range processing on the first image signal or the second image signal. At least one.

在一种可能的实施方式中，该方法还包括：第一图像传感器感光红外光信号，生成该第一图像信号；第二图像传感器感光可见光信号，生成该第二图像信号，该第一图像传感器具有该第一分辨率，该第二图像传感器具有该第二分辨率。In a possible embodiment, the method further includes: the first image sensor receives infrared light signals to generate the first image signal; the second image sensor receives visible light signals to generate the second image signal, and the first image sensor Having the first resolution, the second image sensor has the second resolution.

在一种可能的实施方式中，该方法还包括：将该第一图像传感器切换到红外光补光模式，以感光该红外光信号；将该第二图像传感器切换到红外光截止模式，以感光该可见光信号。In a possible implementation manner, the method further includes: switching the first image sensor to an infrared light supplement light mode to receive the infrared light signal; and switching the second image sensor to an infrared light cutoff mode to receive light The visible light signal.

在一种可能的实施方式中，该将该第一图像传感器切换到红外光补光模式，以感光该红外光信号，具体包括：将第一滤光模式切换器切换到全透光谱滤光片，以使得该第一图像传感器切换到该红外光补光模式；该将该第二图像传感器切换到红外光截止模式，以感光该可见光信号，具体包括：将第二滤光模式切换器切换到红外截止滤光片，以使得该第一图像传感器切换到该红外光截止模式。In a possible implementation manner, switching the first image sensor to an infrared light supplementary light mode to receive the infrared light signal specifically includes: switching the first filter mode switch to a full transmission spectrum filter , So that the first image sensor is switched to the infrared light supplementary light mode; the second image sensor is switched to the infrared light cut-off mode to receive the visible light signal, specifically including: switching the second filter mode switch to An infrared cut filter, so that the first image sensor is switched to the infrared light cut mode.

在一种可能的实施方式中，该方法还包括：接收该目标场景的光信号；将该目标场景的光信号分为第一光信号和第二光信号，该第一光信号为送给该第一图像传感器的光信号，该第二光信号为送给该第二图像传感器的光信号。In a possible implementation manner, the method further includes: receiving an optical signal of the target scene; dividing the optical signal of the target scene into a first optical signal and a second optical signal, and the first optical signal is sent to the The optical signal of the first image sensor, and the second optical signal is the optical signal sent to the second image sensor.

在一种可能的实施方式中，该方法还包括：接收该目标场景的第一光信号，该第一光信号为送给该第一图像传感器的光信号；接收该目标场景的第二光信号，该第二光信号为送给该第二图像传感器的光信号。In a possible implementation manner, the method further includes: receiving a first light signal of the target scene, where the first light signal is a light signal sent to the first image sensor; and receiving a second light signal of the target scene , The second light signal is the light signal sent to the second image sensor.

在一种可能的实施方式中，在该融合该第一图像信号和该第二图像信号，得到融合图像之前，该方法还包括：对该第一图像信号和该第二图像信号进行对齐处理，得到处理后的第一图像信号和处理后的第二图像信号；该融合该第一图像信号和该第二图像信号，得到融合图像，具体包括：融合该处理后的第一图像信号和该处理后的第二图像信号，得到该融合图像。In a possible implementation manner, before fusing the first image signal and the second image signal to obtain a fused image, the method further includes: performing alignment processing on the first image signal and the second image signal, Obtain the processed first image signal and the processed second image signal; the fusion of the first image signal and the second image signal to obtain a fused image specifically includes: fusing the processed first image signal and the processing After the second image signal, the fused image is obtained.

在一种可能的实施方式中，在该将该第一RGB Sensor切换到红外光补光模式之前，该方法还包括：开启红外灯，以提供该红外光信号。In a possible implementation manner, before switching the first RGB Sensor to the infrared light supplementary light mode, the method further includes: turning on an infrared light to provide the infrared light signal.

本申请第三方面提供了一种图像处理的装置，该装置包括：第一接收接口、第二接收接口和融合模块；该第一接收接口，用于接收目标场景的第一图像信号，该第一图像信号包括第一亮度信息，该第一亮度信息是携带有红外光信息的亮度信息，该第一图像信号具有第一分辨率；该第二接收接口，用于接收该目标场景的第二图像信号，该第二图像信号包括第二亮度信息和第二色度信息，该第二亮度信息是排除了红外光信息的亮度信息，该第二图像信号具有第二分辨率，该第一分辨率高于该第二分辨率；该融合，用于融合该第一图像信号和该第二图像信号，得到融合图像。The third aspect of the present application provides an image processing device, which includes: a first receiving interface, a second receiving interface, and a fusion module; the first receiving interface is used to receive a first image signal of a target scene, and the An image signal includes first brightness information, the first brightness information is brightness information carrying infrared light information, the first image signal has a first resolution; the second receiving interface is used to receive a second image of the target scene Image signal, the second image signal includes second luminance information and second chrominance information, the second luminance information is luminance information excluding infrared light information, the second image signal has a second resolution, the first resolution The rate is higher than the second resolution; the fusion is used to fuse the first image signal and the second image signal to obtain a fused image.

在一种可能的实施方式中，该装置还包括：上采样模块，用于对该第二图像信号进行上采样处理，得到第三图像信号，该第三图像信号具有该第一分辨率；该融合模块，具体用于融合该第一图像信号和该第三图像信号，得到该融合图像。In a possible implementation manner, the device further includes: an up-sampling module, configured to perform up-sampling processing on the second image signal to obtain a third image signal, the third image signal having the first resolution; The fusion module is specifically used to fuse the first image signal and the third image signal to obtain the fused image.

在一种可能的实施方式中，该第三图像信号包括第三亮度信息，该融合图像包括融合亮度信息，该装置还包括：第一亮度处理模块和第二亮度处理模块，该第一亮度处理模块，用于获取该第一亮度信息的高频细节信息，该第一亮度信息的高频细节信息包括该第一图像信号的亮度细节信息；该第二亮度处理模块，用于对该第三亮度信息进行去噪处理，以滤除该第三亮度信息的高频噪声，得到第四亮度信息；该融合模块，具体用于叠加该第一亮度信息的高频细节信息和该第四亮度信息，得到该融合亮度信息。In a possible implementation manner, the third image signal includes third brightness information, the fused image includes fused brightness information, and the device further includes: a first brightness processing module and a second brightness processing module, the first brightness processing Module for acquiring the high frequency detail information of the first brightness information, the high frequency detail information of the first brightness information includes the brightness detail information of the first image signal; the second brightness processing module is used for the third The brightness information undergoes denoising processing to filter out the high-frequency noise of the third brightness information to obtain fourth brightness information; the fusion module is specifically used to superimpose the high-frequency detail information of the first brightness information and the fourth brightness information , Get the fusion brightness information.

在一种可能的实施方式中，该第一亮度处理模块，具体用于对该第一亮度信息进行双边滤波，得到滤波后的第一亮度信息；该第一亮度信息减去该滤波后的第一亮度信息，得到该第一亮度信息的高频细节信号。In a possible implementation manner, the first brightness processing module is specifically configured to perform bilateral filtering on the first brightness information to obtain filtered first brightness information; the first brightness information minus the filtered first brightness information One brightness information, the high-frequency detail signal of the first brightness information is obtained.

在一种可能的实施方式中，该第二亮度处理模块，具体用于对该第三亮度信息进行双边滤波，以得到该第四亮度信息。In a possible implementation manner, the second brightness processing module is specifically configured to perform bilateral filtering on the third brightness information to obtain the fourth brightness information.

在一种可能的实施方式中，该第三图像信号还包括第三色度信息，该融合图像还包括融合色度信息，该装置还包括色度处理模块：用于从该第三图像信号中提取该第三色度信息；对该第三色度信息进行颜色校正，得到该融合色度信息。In a possible implementation manner, the third image signal further includes third chrominance information, the fused image further includes fused chrominance information, and the device further includes a chrominance processing module: Extract the third chromaticity information; perform color correction on the third chromaticity information to obtain the fused chromaticity information.

在一种可能的实施方式中，该色度处理模块具体用于：根据该融合亮度信息与该第三亮度信息的差值对该第三色度信息进行补偿，以得到该融合色度信息。In a possible implementation manner, the chrominance processing module is specifically configured to: compensate the third chrominance information according to the difference between the fused luminance information and the third luminance information to obtain the fused chrominance information.

在一种可能的实施方式中，该融合模块具体还用于：组合该融合亮度信息和该融合色度信息，得到该融合图像。In a possible implementation, the fusion module is specifically further used to combine the fused brightness information and the fused chromaticity information to obtain the fused image.

在一种可能的实施方式中，该装置还包括：对齐模块，用于对该第一图像信号和该第二图像信号进行对齐处理。In a possible implementation manner, the device further includes: an alignment module, configured to perform alignment processing on the first image signal and the second image signal.

本申请第四方面提供了一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当其在计算机或处理器上运行时，使得该计算机或处理器执行如上述第二方面或者其任一种可能的实施方式中该的方法。The fourth aspect of the present application provides a computer-readable storage medium with instructions stored in the computer-readable storage medium, which when run on a computer or processor, cause the computer or processor to execute the second aspect or The method in any of its possible implementations.

本申请第五方面提供了一种包含指令的计算机程序产品，当其在计算机或处理器上运行时，使得该计算机或处理器执行如上述第二方面或者其任一种可能的实施方式中该的方法。The fifth aspect of the present application provides a computer program product containing instructions, which when it runs on a computer or processor, causes the computer or processor to execute the second aspect or any one of its possible implementations. Methods.

附图说明Description of the drawings

图1a为本申请实施例提供的一种示例性的RGB Sensor的示意图；Fig. 1a is a schematic diagram of an exemplary RGB Sensor provided by an embodiment of the application;

图1b为本申请实施例提供的另一种示例性的RGB Sensor的示意图；Fig. 1b is a schematic diagram of another exemplary RGB Sensor provided by an embodiment of the application;

图2a为本申请实施例提供的一种示例性的应用***架构示意图；FIG. 2a is a schematic diagram of an exemplary application system architecture provided by an embodiment of the application;

图2b为本申请实施例提供的另一种示例性的应用***架构示意图；2b is a schematic diagram of another exemplary application system architecture provided by an embodiment of the application;

图3为本申请实施例提供的另一种示例性的应用***架构示意图；3 is a schematic diagram of another exemplary application system architecture provided by an embodiment of the application;

图4为本申请实施例提供的一种示例性的图像处理装置的硬件架构示意图；4 is a schematic diagram of the hardware architecture of an exemplary image processing apparatus provided by an embodiment of the application;

图5为本申请实施例提供的一种示例性的图像处理装置示意图；FIG. 5 is a schematic diagram of an exemplary image processing device provided by an embodiment of the application;

图6为本申请实施例提供的另一种示例性的图像处理装置示意图；FIG. 6 is a schematic diagram of another exemplary image processing device provided by an embodiment of the application;

图7为本申请实施例提供的一种示例性的图像融合的方法流程图；FIG. 7 is a flowchart of an exemplary image fusion method provided by an embodiment of the application;

图8为本申请实施例提供的一种示例性的图像信号处理的方法流程示意图。FIG. 8 is a schematic flowchart of an exemplary image signal processing method provided by an embodiment of the application.

具体实施方式detailed description

本申请的说明书实施例和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元。方法、***、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。The terms "first", "second", etc. in the specification embodiments and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusion, for example, a series of steps or units are included. The method, system, product, or device is not necessarily limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。It should be understood that in this application, "at least one (item)" refers to one or more, and "multiple" refers to two or more. "And/or" is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, "A and/or B" can mean: only A, only B, and both A and B , Where A and B can be singular or plural. The character "/" generally indicates that the associated objects are in an "or" relationship. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or plural items (a). For example, at least one (a) of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, and c can be single or multiple.

人们对成像设备的图像质量的要求越来越高，例如在安防监控领域，安防监控设备获取的图像一方面要满足人眼主观感受的升级需求，另一方面，还需要满足各类机器识别的需求；再例如手机照相领域，无论是拍人像还是拍景色，对图像的清晰度和色彩表现都有越来越高的要求。成像设备获得的图像的质量与图像传感器的感光灵敏度以及拍摄场景的光线情况都有关系。People’s requirements for the image quality of imaging equipment are getting higher and higher. For example, in the field of security monitoring, the images acquired by security monitoring equipment must meet the upgrade requirements of the subjective perception of the human eye on the one hand, and also need to meet the requirements of various machine recognition. Demand; another example is the mobile phone camera field, whether it is to shoot portraits or scenery, there are increasingly higher requirements for image clarity and color performance. The quality of the image obtained by the imaging device is related to the light sensitivity of the image sensor and the light condition of the shooting scene.

下面先对本申请涉及的一些术语进行说明。The following first explains some terms involved in this application.

像素：图像传感器的技术指标之一，传感器上有许多感光单元，这些感光单元可以将光信号转换成电信号，每个感光单元对应一个像素，相同靶面尺寸内的像素数越多表示传感器的分辨率越高，分辨率越高代表传感器能够感测到的细节越多，获得的图像越清晰。图像传感器的靶面尺寸相同时，分辨率越高的传感器单个像素的尺寸越小，分辨率越低的传感器单个像素的尺寸越大。传感器的单个像素的尺寸越大时，单个像素的感光能力更强。图像信号的分辨率表示图像的像素数，例如一个图像的分辨率为1080*1080，则表示该图像有1080*1080个像素，对应一个1080*1080大小的数字矩阵，其中一个像素对应一个矩阵元素。靶面尺寸：图像传感器感光部分的大小，一般用英寸来表示。通常这个数据为图像传感器的对角线长度，靶面尺寸越大，意味着通光量越好，而靶面尺寸越小则比较容易获得更大的景深。Pixel: One of the technical indicators of the image sensor. There are many photosensitive units on the sensor. These photosensitive units can convert light signals into electrical signals. Each photosensitive unit corresponds to a pixel. The more pixels in the same target surface size, the more the sensor’s The higher the resolution, the higher the resolution means the more details the sensor can sense and the clearer the image obtained. When the size of the target surface of the image sensor is the same, the size of a single pixel of the sensor with a higher resolution is smaller, and the size of a single pixel of the sensor with a lower resolution is larger. The larger the size of a single pixel of the sensor, the stronger the sensitivity of a single pixel. The resolution of the image signal indicates the number of pixels of the image. For example, if the resolution of an image is 1080*1080, it means that the image has 1080*1080 pixels, corresponding to a 1080*1080 digital matrix, and one pixel corresponds to a matrix element . Target size: The size of the photosensitive part of the image sensor, generally expressed in inches. Usually this data is the diagonal length of the image sensor. The larger the size of the target surface, the better the light flux. The smaller the size of the target surface, the easier it is to obtain a larger depth of field.

感光度：传感器及相关的电子线路感应入射光信号的强弱，感光度越高，图像传感器的感光面对光的敏感度就越强。Sensitivity: The sensor and related electronic circuits sense the strength of the incident light signal. The higher the sensitivity, the stronger the sensitivity of the image sensor to light.

信噪比：有效信号对噪声信号的比值，示例性的，可以是信号电压对噪声电压的比值，信噪比的单位为dB，信噪比越大表示图像中的噪声越少，图像画面越干净。Signal-to-noise ratio: The ratio of effective signal to noise signal. Illustratively, it can be the ratio of signal voltage to noise voltage. The unit of signal-to-noise ratio is dB. The larger the signal-to-noise ratio, the less noise in the image and the larger the image. clean.

照度(Illuminance)：单位面积接收到的光通量，单位为勒克斯(Lux)。Illuminance: The luminous flux received per unit area, the unit is Lux.

YCC色彩空间：本申请中表示亮色分离的颜色空间，YCC三个分量分别代表亮度-色度-色度(Luma-Chroma-Chroma)；常见的YCC空间图像格式有YUV、YCbCr、ICtCp等。YCC color space: In this application, it represents the color space of bright color separation. The three components of YCC represent Luma-Chroma-Chroma (Luma-Chroma-Chroma); common YCC space image formats include YUV, YCbCr, ICtCp, etc.

红绿蓝传感器(Red Green Blue Sensor，RGB Sensor)是目前广泛应用的图像传感器之一，如图1a和图1b所示，为本申请实施例提供的两种示例性的不同分辨率的RGB Sensor的示意图。对于自然图像而言，每个像素包含R、G、B三个颜色分量，R代表红绿蓝三原色分量中的红色分量，G代表红绿蓝三原色分量中的蓝色分量，B代表红绿蓝三原色分量中的蓝色分量。图中每个格代表一个像素，RGB Sensor的每个像素仅仅感光R、G、B三分量中的一个颜色分量，因此RGB Sensor的感光能力有限。图1a和图1b中所示的两个RGB Sensor具有相同的靶面尺寸，图1a中的RGB Sensor具有16个像素，分辨率较高，图1b中的RGB Sensor具有4个像素，分辨率较低，图1a中的RGB Sensor和图1b中的RGB Sensor的分辨率比例为4:1。图1a中的图像传感器感光获得一个分辨率为4*4的图像，转换为数字图像信号时，该数字图像信号可以表示为一个4*4的矩阵，图1b中的图像传感器感光获得一个分辨率为2*2的图像，转换为数字图像信号时，该数字图像信号可以表示为一个2*2的矩阵。对图1b的传感器生成的数字图像信号进行上采样可以获得图1a的传感器生成的数字图像信号。The Red Green Blue Sensor (RGB Sensor) is one of the currently widely used image sensors. As shown in Fig. 1a and Fig. 1b, two exemplary RGB Sensors with different resolutions are provided in the embodiments of this application. Schematic diagram. For natural images, each pixel contains three color components R, G, and B. R represents the red component of the three primary color components of red, green and blue, G represents the blue component of the three primary color components of red, green and blue, and B represents red, green, and blue. The blue component of the three primary color components. Each grid in the figure represents a pixel, and each pixel of the RGB Sensor is only sensitive to one of the three components of R, G, and B. Therefore, the photosensitive ability of the RGB Sensor is limited. The two RGB Sensors shown in Figure 1a and Figure 1b have the same target surface size. The RGB Sensor in Figure 1a has 16 pixels and a higher resolution. The RGB Sensor in Figure 1b has 4 pixels and a higher resolution. Low, the resolution ratio of the RGB Sensor in Figure 1a to the RGB Sensor in Figure 1b is 4:1. The image sensor in Figure 1a obtains an image with a resolution of 4*4. When converted into a digital image signal, the digital image signal can be expressed as a 4*4 matrix. The image sensor in Figure 1b obtains a resolution from the image sensor. It is a 2*2 image, when converted into a digital image signal, the digital image signal can be expressed as a 2*2 matrix. Up-sampling the digital image signal generated by the sensor of FIG. 1b can obtain the digital image signal generated by the sensor of FIG. 1a.

以RGB Sensor为例，在低照场景下，由于光线较弱，RGB Sensor的感光能力急剧降低，导致RGB Sensor在低照场景下获取的图像在图像细节、颜色和信噪比等各方面均不尽人意。应当理解，低照场景为光线较弱的场景，示例性的，低照场景的照度阈值可以取1Lux，当环境的光线照度低于1Lux时，此时的环境为低照场景。在一种可选的情况下，低照场景的照度阈值也可以取其他值，本申请实施例提到的低照场景为环境的光线照度低于预设阈值时的场景，对该预设阈值的取值不做限定。为了提升低照场景下获取的图像的信噪比，提升图像的细节表现和色彩还原度，本申请实施例提出一种基于异构双RGB Sensor的图像处理方法和图像处理装置。Take the RGB Sensor as an example. In a low-light scene, due to the weak light, the sensitivity of the RGB Sensor is drastically reduced. As a result, the image obtained by the RGB Sensor in a low-light scene is different in image details, color, and signal-to-noise ratio. Satisfactory. It should be understood that a low-light scene is a low-light scene. Illustratively, the illuminance threshold of the low-light scene can be 1 Lux. When the light illuminance of the environment is lower than 1 Lux, the environment at this time is a low-light scene. In an optional case, the illuminance threshold of the low-light scene can also take other values. The low-light scene mentioned in the embodiment of this application is a scene when the light illuminance of the environment is lower than a preset threshold. The value of is not limited. In order to improve the signal-to-noise ratio of an image acquired in a low-light scene, and improve the detail performance and color reproduction of the image, an embodiment of the present application proposes an image processing method and an image processing device based on a heterogeneous dual RGB Sensor.

如图2a所示为本申请实施例提供的一种示例性的应用***架构示意图。Fig. 2a is a schematic diagram of an exemplary application system architecture provided by an embodiment of the application.

其中，201为外界的场景光信号，202为图像处理装置，图像处理装置202可以包括镜头2021、成像传感器2022和图像处理器2023，可选的，图像处理装置202还可以包括存储器(图中未示出)，用于存储图像处理装置生成的图像信号等；图像处理装置202还可以包括红外灯(图中未示出)，该红外灯集成在图像处理装置202内部，当外界场景光信号较弱时为成像传感器2022提供红外光补光。场景光信号201为图像处理装置202外界的景象的光信号，该外界景象的光信号通过图像处理装置202转换为电信号，电信号还可以被进一步转换为数字图像信号。具体的，场景光信号201通过镜头2021到达成像传感器2022，场景光信号201被成像传感器2022感光形成电信号，该电信号还可以通过模拟数字转换器(Analog to Digital Converter，ADC)进一步转换为数字图像信号，该数字图像信号经过图像处理器2023处理之后可以存储在图像处理装置202的存储器中、外部的存储器中或者送给显示装置203进行显示。在一种可选的情况中，图像处理装置202本身包括显示屏2024，如图2b所示，数字图像信号经过图像处理器2023处理之后发给显示屏2024进行显示，也可以发送给外部的显示装置进行显示。Among them, 201 is an external scene light signal, 202 is an image processing device, the image processing device 202 may include a lens 2021, an imaging sensor 2022, and an image processor 2023. Optionally, the image processing device 202 may also include a memory (not shown in the figure). (Shown), used to store image signals generated by the image processing device, etc.; the image processing device 202 may also include an infrared lamp (not shown in the figure), the infrared lamp is integrated inside the image processing device 202, when the external scene light signal When weak, the imaging sensor 2022 is provided with infrared light supplementary light. The scene light signal 201 is a light signal of a scene outside the image processing device 202, and the light signal of the outside scene is converted into an electric signal by the image processing device 202, and the electric signal can be further converted into a digital image signal. Specifically, the scene light signal 201 reaches the imaging sensor 2022 through the lens 2021, and the scene light signal 201 is light-sensitized by the imaging sensor 2022 to form an electrical signal. The electrical signal can also be further converted into a digital signal by an analog to digital converter (ADC). An image signal. After being processed by the image processor 2023, the digital image signal can be stored in the memory of the image processing device 202, in an external memory, or sent to the display device 203 for display. In an optional case, the image processing device 202 itself includes a display screen 2024, as shown in Figure 2b, the digital image signal is processed by the image processor 2023 and then sent to the display screen 2024 for display, or it can be sent to an external display. The device displays.

示例性的，图像处理装置202可以为小区的安防监控摄像头、智能交通电子眼设备、摄像机、照相机、移动手机、以及其他具有成像、照相或视频录制功能的终端设备。该成像传感器2022可以为RGB Sensor，该RGB Sensor可以为感光耦合元件(Charge-Coupled Device,CCD)或互补式金属氧化物半导体(Complementary Metal-Oxide Semiconductor，CMOS)传感器等。该图像处理器2023可以为专用的图像信号处理器(Image Signal Processor，ISP)，通用处理器中的具有图像处理功能的模块、图形处理单元(Graphics Processing Unit，GPU)或者具有图像或视频处理功能的集成电路等。显示屏2024可以是液晶显示器(Liquid Crystal Display，LCD)，发光二级管(Light Emitting Diode，LED)显示器，有机发光二极管(Organic Light-Emitting Diode，OLED)显示屏、阴极射线管(Cathode Ray Tube，CRT)显示屏等。应当理解，图像处理装置202既可以处理图像信号，也可以处理视频信号，视频信号可以体现为时间上连续的一帧一帧的图像信号。Exemplarily, the image processing device 202 may be a security surveillance camera, an intelligent transportation electronic eye device, a video camera, a camera, a mobile phone, and other terminal devices with imaging, photographing, or video recording functions in a community. The imaging sensor 2022 may be an RGB Sensor, and the RGB Sensor may be a Charge-Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) sensor. The image processor 2023 may be a dedicated image signal processor (Image Signal Processor, ISP), a module with image processing functions in a general-purpose processor, a graphics processing unit (Graphics Processing Unit, GPU), or an image or video processing function Integrated circuits, etc. The display screen 2024 can be a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, an Organic Light-Emitting Diode (OLED) display, a Cathode Ray Tube (Cathode Ray Tube) , CRT) display screen, etc. It should be understood that the image processing device 202 can process both image signals and video signals, and the video signals can be embodied as image signals that are continuous in time, frame by frame.

示例性的，该图像处理装置202为小区的安防监控摄像装置，场景光信号201为小区大门处的景象的光信号，安防监控摄像头对小区大门处的景象进行成像，形成小区大门处的图像信号。场景光信号为实际场景的光信号，该光信号通过传感器感光生成图像信号。此时，显示装置203可以是监控室中的监视器或监控屏幕。Exemplarily, the image processing device 202 is a security surveillance camera device of the community, the scene light signal 201 is the light signal of the scene at the gate of the cell, and the security surveillance camera images the scene at the gate of the cell to form an image signal at the gate of the cell . The scene light signal is the light signal of the actual scene, and the light signal generates an image signal through the sensor. At this time, the display device 203 may be a monitor or a monitoring screen in a monitoring room.

如图3所示，为本申请实施例提供的另一种示例性的应用***架构示意图。As shown in FIG. 3, another exemplary application system architecture schematic diagram provided by an embodiment of this application.

场景光信号301同场景光信号201，此处不再赘述。The scene light signal 301 is the same as the scene light signal 201, and will not be repeated here.

成像装置302包括镜头3021和成像传感器3022，成像装置302将感光生成的图像信号传送给图像处理装置303进行处理，并在图像处理装置303处理之后发送给显示装置304进行显示。示例性的，图像处理装置303可以为具有图像处理功能的芯片或处理器，该图像处理装置303包括接收接口3031和图像处理器3032，接收接口3031用于接收成像装置生成的图像信号，图像处理器3032请参考图像处理器2023部分的描述，图像处理器3032用于对图像信号进行处理，图像处理器3032可以进行的处理包括但不限于：对两路图像信号进行融合、对图像信号进行上采样或下采样、去马赛克处理、噪声去除、图像增强、动态范围处理、对比度处理、颜色校正等。应当理解，该图像处理器3032还可以处理视频信号。可选的，该图像处理装置303还包括发送接口(图中未示出)，通过发送接口将处理后的图像信号发送给显示装置304。显示装置304为具有显示功能的装置，例如可以是电视机、电脑、监视器、智能手机、移动终端等，可选的，显示装置304也可以为LCD显示屏，LED显示屏，OLED显示屏、CRT显示屏等。The imaging device 302 includes a lens 3021 and an imaging sensor 3022. The imaging device 302 transmits the image signal generated by light to the image processing device 303 for processing, and after processing by the image processing device 303, it is sent to the display device 304 for display. Exemplarily, the image processing device 303 may be a chip or processor with image processing functions. The image processing device 303 includes a receiving interface 3031 and an image processor 3032. The receiving interface 3031 is used to receive the image signal generated by the imaging device, and the image processing Please refer to the description of the image processor 2023. The image processor 3032 is used to process image signals. The processing that the image processor 3032 can perform includes but is not limited to: fusing two channels of image signals, and uploading image signals. Sampling or downsampling, demosaicing, noise removal, image enhancement, dynamic range processing, contrast processing, color correction, etc. It should be understood that the image processor 3032 can also process video signals. Optionally, the image processing device 303 further includes a sending interface (not shown in the figure), and the processed image signal is sent to the display device 304 through the sending interface. The display device 304 is a device with a display function, such as a TV, a computer, a monitor, a smart phone, a mobile terminal, etc., optionally, the display device 304 can also be an LCD display, an LED display, an OLED display, CRT display screen, etc.

在一种可选的情况中，成像装置302、图像处理装置303和显示装置304共同构成一个完整的移动终端，该移动终端同时具有图像获取、图像处理和图像显示功能，示例性的，该移动终端可以为手机、照相机、摄影机、监控摄像机等，该成像装置302为镜头和图像传感器组，该图像处理装置303为图像处理芯片或处理器组，该显示装置304为显示屏。In an optional situation, the imaging device 302, the image processing device 303, and the display device 304 together constitute a complete mobile terminal. The mobile terminal has image acquisition, image processing and image display functions at the same time. The terminal may be a mobile phone, a camera, a video camera, a surveillance camera, etc. The imaging device 302 is a lens and image sensor group, the image processing device 303 is an image processing chip or a processor group, and the display device 304 is a display screen.

如图4所示，为本申请实施例提供的一种示例性的图像处理装置的硬件架构示意图。图像处理装置400的硬件架构适用于图2a和图2b中的图像处理器2023，以及图3中的图像处理装置303。As shown in FIG. 4, it is a schematic diagram of the hardware architecture of an exemplary image processing apparatus provided by an embodiment of this application. The hardware architecture of the image processing device 400 is applicable to the image processor 2023 in FIGS. 2a and 2b and the image processing device 303 in FIG. 3.

示例性的，该图像处理装置400包括至少一个中央处理单元(Central Processing Unit，CPU)、至少一个存储器、GPU、解码器、专用的视频或图形处理器、接收接口和发送接口等。可选的，图像处理装置400还可以包括微处理器和微控制器(Microcontroller Unit，MCU)等。在一种可选的情况中，图像处理装置400的上述各个部分通过连接器相耦合，应当理解，本申请的各个实施例中，耦合是指通过特定方式的相互联系，包括直接相连或者通过其他设备间接相连，例如可以通过各类接口、传输线或总线等相连，这些接口通常是电性通信接口，但是也不排除可能是机械接口或其它形式的接口，本实施例对此不做限定。在一种可选的情况中，上述各部分集成在同一个芯片上；在另一种可选的情况中，CPU、GPU、解码器、接收接口以及发送接口集成在一个芯片上，该芯片内部的各部分通过总线访问外部的存储器。专用视频/图形处理器可以与CPU集成在同一个芯片上，也可以作为单独的处理器芯片存在，例如专用视频/图形处理器可以为专用ISP。在本申请实施例中涉及的芯片是以集成电路工艺制造在同一个半导体衬底上的***，也叫半导体芯片，其可以是利用集成电路工艺制作在所述衬底(通常是例如硅一类的半导体材料)上形成的集成电路的集合，其外层通常被半导体封装材料封装。所述集成电路可以包括各类功能器件，每一类功能器件包括逻辑门电路、金属氧化物半导体(Metal-Oxide-Semiconductor，MOS)晶体管、双极晶体管或二极管等晶体管，也可包括电容、电阻或电感等其他部件。每个功能器件可以独立工作或者在必要的驱动软件的作用下工作，可以实现通信、运算、或存储等各类功能。Exemplarily, the image processing device 400 includes at least one central processing unit (CPU), at least one memory, GPU, decoder, dedicated video or graphics processor, receiving interface, sending interface, etc. Optionally, the image processing device 400 may also include a microprocessor and a microcontroller (Microcontroller Unit, MCU), etc. In an optional situation, the above-mentioned parts of the image processing device 400 are coupled through a connector. It should be understood that, in the various embodiments of the present application, coupling refers to mutual connection in a specific manner, including direct connection or through other The devices are indirectly connected, such as through various interfaces, transmission lines or buses. These interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces, which are not limited in this embodiment. In an optional case, the above-mentioned parts are integrated on the same chip; in another optional case, the CPU, GPU, decoder, receiving interface, and transmitting interface are integrated on one chip, and the chip is The various parts of the bus access external memory. The dedicated video/graphics processor can be integrated with the CPU on the same chip, or it can exist as a separate processor chip. For example, the dedicated video/graphics processor can be a dedicated ISP. The chip involved in the embodiments of this application is a system manufactured on the same semiconductor substrate by an integrated circuit process, also called a semiconductor chip, which can be manufactured on the substrate by using an integrated circuit process (usually such as silicon). A collection of integrated circuits formed on a semiconductor material), the outer layer of which is usually encapsulated by a semiconductor packaging material. The integrated circuit may include various types of functional devices, and each type of functional device includes transistors such as logic gate circuits, Metal-Oxide-Semiconductor (MOS) transistors, bipolar transistors or diodes, and may also include capacitors and resistors. Or inductance and other components. Each functional device can work independently or under the action of necessary driver software, and can realize various functions such as communication, calculation, or storage.

可选的，CPU可以是一个单核(single-CPU)处理器或多核(multi-CPU)处理器；可选的，CPU可以是多个处理器构成的处理器组，多个处理器之间通过一个或多个总线彼此耦合。在一种可选的情况中，对于图像信号或视频信号的处理一部分由GPU完成，一部分由专用视频/图形处理器完成，还有可能由跑在通用CPU或GPU上的软件代码完成。Optionally, the CPU may be a single-CPU processor or a multi-CPU processor; optionally, the CPU may be a processor group composed of multiple processors, between multiple processors Coupled to each other through one or more buses. In an optional situation, part of the processing of the image signal or video signal is completed by the GPU, part is completed by a dedicated video/graphics processor, and may also be completed by software code running on a general-purpose CPU or GPU.

存储器，可用于存储计算机程序指令，包括操作***(Operation System，OS)、各种用户应用程序、以及用于执行本申请方案的程序代码在内的各类计算机程序代码；存储器还可以用于存储视频数据、图像信号数据等；CPU可以用于执行存储器中存储的计算机程序代码，以实现本申请实施例中的方法。可选的，存储器302可以是非掉电易失性存储器，例如是嵌入式多媒体卡(Embedded Multi Media Card，EMMC)、通用闪存存储(Universal Flash Storage，UFS)或只读存储器(Read-Only Memory，ROM)，或者是可存储静态信息和指令的其他类型的静态存储设备，还可以是掉电易失性存储器(volatile memory)，例如随机存取存储器(Random Access Memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的程序代码并能够由计算机存取的任何其他计算机可读存储介质，但不限于此。The memory can be used to store computer program instructions, including operating system (Operation System, OS), various user application programs, and various computer program codes used to execute the solution of the application; the memory can also be used to store Video data, image signal data, etc.; the CPU can be used to execute computer program codes stored in the memory to implement the methods in the embodiments of the present application. Optionally, the memory 302 may be a non-power-down volatile memory, such as Embedded MultiMedia Card (EMMC), Universal Flash Storage (UFS) or Read-Only Memory (Read-Only Memory, ROM), or other types of static storage devices that can store static information and instructions, or volatile memory (volatile memory), such as Random Access Memory (RAM), or can store information and Other types of dynamic storage devices for instructions can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory, CD-ROM or other optical discs Storage, optical disc storage (including compact discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store program codes in the form of instructions or data structures and can Any other computer-readable storage medium accessed by the computer, but not limited to this.

该接收接口可以为处理器芯片的数据输入的接口，在一种可选的情况下，该接收接口可以是高清晰度多媒体接口(High Definition Multimedia Interface，HDMI)。The receiving interface may be an interface for data input of the processor chip. In an optional case, the receiving interface may be a High Definition Multimedia Interface (HDMI).

如图5所示，为本申请实施例提供的一种示例性的图像处理装置示意图。As shown in FIG. 5, a schematic diagram of an exemplary image processing apparatus provided by an embodiment of this application.

该图像处理装置500包括：镜头501、分光器502、第一RGB Sensor 505、第二RGB Sensor 506以及处理器507，可选的，该图像处理装置500还可以包括第一滤光模式切换器503，第二滤光模式切换器504和红外灯508，第一RGB Sensor 505的分辨率较高，第二RGB Sensor 506的分辨率较低，第一RGB Sensor 505在低照场景下工作在红外光补光模式，第二RGB Sensor 506在低照场景下工作在红外光截止模式，应当理解，该第一RGB Sensor 505和第二RGB Sensor 506均可以工作在红外光补光模式和红外光截止模式。在一种可能的实施方式中，第一RGB Sensor 505和第二RGB Sensor 506具有相同的靶面尺寸，第一RGB Sensor 505的分辨率高于第二RGB Sensor506的分辨率，第一RGB Sensor 505单个像素的尺寸小于第二RGB Sensor 506单个像素的尺寸。The image processing device 500 includes: a lens 501, a beam splitter 502, a first RGB Sensor 505, a second RGB Sensor 506, and a processor 507. Optionally, the image processing device 500 may also include a first filter mode switch 503 , The second filter mode switch 504 and the infrared lamp 508, the resolution of the first RGB Sensor 505 is higher, the resolution of the second RGB Sensor 506 is lower, and the first RGB Sensor 505 works in infrared light in a low-light scene In the light supplement mode, the second RGB Sensor 506 works in the infrared light cut-off mode in low-light scenes. It should be understood that the first RGB Sensor 505 and the second RGB Sensor 506 can both work in the infrared light supplement light mode and the infrared light cut mode . In a possible implementation, the first RGB Sensor 505 and the second RGB Sensor 506 have the same target surface size, and the resolution of the first RGB Sensor 505 is higher than the resolution of the second RGB Sensor 506, and the first RGB Sensor 505 The size of a single pixel is smaller than the size of a single pixel of the second RGB Sensor 506.

镜头501，用于接收光信号，该光信号为图像处理装置外部的场景光信号。镜头501可以由多片镜片组成，也可以是由单个镜片组成，本申请实施例不做限定。The lens 501 is used to receive a light signal, which is a scene light signal outside the image processing device. The lens 501 may be composed of multiple lenses, or may be composed of a single lens, which is not limited in the embodiment of the present application.

分光器502，用于将镜头501接收到的光信号分为第一光信号和第二光信号，第一光信号为发送给第一RGB Sensor 505的光信号，第二光信号为发送给第二RGB Sensor 506的光信号。示例性的，该分光器502可以为分光棱镜，例如可以是分光镀膜棱镜，采用镀膜分光的方式将光信号分为第一光信号和第二光信号，该分光器502还可以是衍射分光棱镜、机械遮拦装置等其他类型的分光装置。The optical splitter 502 is used to divide the optical signal received by the lens 501 into a first optical signal and a second optical signal. The first optical signal is the optical signal sent to the first RGB Sensor 505, and the second optical signal is sent to the first optical signal. Two RGB Sensor 506 optical signal. Exemplarily, the beam splitter 502 may be a beam splitter prism, for example, a beam splitting coated prism, and the optical signal is divided into a first optical signal and a second optical signal by a coating beam splitting method. The beam splitter 502 may also be a diffraction beam splitting prism , Mechanical blocking device and other types of light splitting devices.

第一光信号通过第一滤光模式切换器503到达第一RGB Sensor 505，第二光信号通过第二滤光模式切换器504到达第二RGB Sensor 506。示例性的，第一滤光模式切换器503和第二滤光模式切换器504均包括全透光谱滤光片和红外截止滤光片，第一滤光模式切换器503和第二滤光模式切换器504还可以包括其他种类的滤光片，例如可以包括只能通过某个频段内的光谱的滤光片或者不允许某个频段内的光谱通过的滤光片，本申请实施例对此不做限定。第一滤光模式切换器503和第二滤光模式切换器504还可以包括动力***，该动力***用于实现全透光谱滤光片与红外截止滤光片之间的切换，该动力***可以是小型电磁、电机或其他动力源等。在一种可选的情况中，第一滤光模式切换器503和第二滤光模式切换器504可以为红外过滤器(Infrared-Cut， IR-Cut)，IR-Cut为一种双滤光片切换器，IR-Cut可以根据外部光信号的强弱进行切换，当光线较强时，IR-Cut切换为红外截止滤光片，以避免红外光导致的颜色偏差；当光线较弱时，例如当光照度低于1Lux时，IR-Cut切换为全透光谱滤光片，以提升低照场景下的光亮度。当第一滤光模式切换器503切换到全透光谱滤光片时，红外光和可见光均可以通过全透光谱滤光片到达第一RGB Sensor 505。当第二滤光模式切换器504切换到红外截止滤光片时，红外光无法通过红外截止滤光片，可见光信号通过红外截止滤光片到达第二RGB Sensor 506。The first optical signal passes through the first filter mode switch 503 to the first RGB Sensor 505, and the second optical signal passes through the second filter mode switch 504 to the second RGB Sensor 506. Exemplarily, the first filter mode switch 503 and the second filter mode switch 504 each include a full transmission spectrum filter and an infrared cut filter, the first filter mode switch 503 and the second filter mode The switch 504 may also include other types of filters, for example, it may include a filter that can only pass a spectrum in a certain frequency band or a filter that does not allow a spectrum in a certain frequency band to pass. Not limited. The first filter mode switch 503 and the second filter mode switch 504 may also include a power system, which is used to switch between the full transmission spectrum filter and the infrared cut filter, and the power system can It is a small electromagnetic, motor or other power source. In an optional situation, the first filter mode switch 503 and the second filter mode switch 504 may be infrared filters (Infrared-Cut, IR-Cut), and IR-Cut is a double filter. Chip switcher, IR-Cut can switch according to the strength of the external light signal. When the light is strong, IR-Cut is switched to an infrared cut filter to avoid color deviation caused by infrared light; when the light is weak, For example, when the illuminance is lower than 1Lux, IR-Cut is switched to a full transmission spectral filter to improve the brightness of the low-light scene. When the first filter mode switch 503 is switched to the full transmission spectrum filter, both infrared light and visible light can reach the first RGB Sensor 505 through the full transmission spectrum filter. When the second filter mode switch 504 switches to the infrared cut filter, the infrared light cannot pass through the infrared cut filter, and the visible light signal passes through the infrared cut filter to reach the second RGB Sensor 506.

当图像处理装置500工作在低照场景下时，开启红外灯508，用于提供红外光信号，第一滤光模式切换器503将第一RGB Sensor 505切换到红外光补光模式，第二滤光模式切换器504将第二RGB Sensor 506切换到红外光截止模式，具体的，第一滤光模式切换器503切换到全透光谱滤光片，以使得第一RGB Sensor 505感光第一过滤光信号生成第一图像信号，第一过滤光信号包括红外光信号和可见光信号，由于低照场景下可见光信号比较弱，而红外灯提供的红外光信号比较强，第一RGB Sensor 505在红外光补光模式下感光红外光信号生成的第一图像信号为红外光图像信号，第一图像信号包括第一亮度信息，可以近似认为红外光图像信号只包含亮度信息，不包含色度信息。第二滤光模式切换器504切换到红外截止滤光片，以使得第二RGB Sensor 506感光第二过滤光信号生成第二图像信号，第二过滤光信号包括可见光信号，不包括红外光信号，第二RGB Sensor 506在红外光截止模式下感光可见光信号生成第二图像信号，第二图像信号主要为可见光图像信号，第二图像信号包括第二亮度信息和第二色度信息。When the image processing device 500 is working in a low-light scene, the infrared light 508 is turned on to provide infrared light signals. The first filter mode switch 503 switches the first RGB Sensor 505 to the infrared light supplement mode, and the second filter The light mode switch 504 switches the second RGB Sensor 506 to the infrared light cut-off mode. Specifically, the first filter mode switch 503 switches to the full-transmission spectrum filter, so that the first RGB Sensor 505 is exposed to the first filtered light. The signal generates the first image signal. The first filtered light signal includes the infrared light signal and the visible light signal. Because the visible light signal is relatively weak in a low-light scene, and the infrared light signal provided by the infrared lamp is relatively strong, the first RGB Sensor 505 is used in the infrared light compensation. The first image signal generated by the photosensitive infrared light signal in the light mode is an infrared light image signal, and the first image signal includes first brightness information. It can be approximated that the infrared light image signal contains only brightness information and does not contain chromaticity information. The second filter mode switch 504 switches to the infrared cut filter, so that the second RGB Sensor 506 receives the second filtered light signal to generate a second image signal. The second filtered light signal includes a visible light signal but does not include an infrared light signal. The second RGB Sensor 506 generates a second image signal by sensitizing the visible light signal in the infrared light cutoff mode. The second image signal is mainly a visible light image signal, and the second image signal includes second luminance information and second chrominance information.

应当理解，红外光补光模式表示传感器可以接收红外光并感光红外光以生成红外光图像信号，红外光图像信号丢失了颜色信息，但是具有比较好的图像细节信息；红外光截止模式表示传感器不可以接收红外光，此时传感器感光的主要是可见光信号，生成可见光图像信号，可见光图像信号具有较好的颜色信息。滤光模式切换器是实现RGB Sensor在红外光补光模式和红外截止模式之间转换的一种实现方式，实际还可以包括其他切换方式，例如RGB Sensor本身有两种工作模式，通过硬件设置或软件设置可以使RGB Sensor在这两种工作模式之间切换，本申请实施例对此不做限定。It should be understood that the infrared light supplement light mode means that the sensor can receive infrared light and photosensitive infrared light to generate infrared light image signals. The infrared light image signal loses color information, but has better image detail information; infrared light cutoff mode means that the sensor does not It can receive infrared light. At this time, the sensor is mainly sensitive to visible light signals to generate visible light image signals. The visible light image signals have better color information. The filter mode switch is a way to realize the conversion between the infrared light fill mode and the infrared cut-off mode of the RGB Sensor. In fact, it can also include other switching methods. For example, the RGB Sensor itself has two working modes, which can be set by hardware or The software setting can make the RGB Sensor switch between these two working modes, which is not limited in the embodiment of the present application.

第一RGB Sensor 505，用于在低照场景下工作在红外光补光模式，并感光透过第一滤光模式切换器503的第一过滤光信号生成第一图像信号第一图像信号具有第一分辨率。The first RGB Sensor 505 is used to work in the infrared light supplement mode in a low-light scene, and to receive the first filtered light signal transmitted through the first filter mode switch 503 to generate a first image signal. The first image signal has a first image signal. One resolution.

第二RGB Sensor 506，用于在低照场景下工作在红外光截止模式，并感光透过第二滤光模式切换器504的第二过滤光信号生成第二图像信号。第二图像信号具有第二分辨率。第一分辨率大于第二分辨率，示例性的，第一分辨率与第二分辨率的比例可以是4:1。第一分辨率与第二分辨率也可以是16:1，或者其他比例。示例性的，第一图像信号和第二图像信号都是原始图像文件raw格式的图像。The second RGB Sensor 506 is configured to work in the infrared light cut-off mode in a low-light scene, and to receive the second filtered light signal transmitted through the second filter mode switch 504 to generate a second image signal. The second image signal has a second resolution. The first resolution is greater than the second resolution. For example, the ratio of the first resolution to the second resolution may be 4:1. The first resolution and the second resolution can also be 16:1 or other ratios. Exemplarily, the first image signal and the second image signal are both images in the raw format of the original image file.

处理器507，用于分别接收第一RGB Sensor 505生成的第一图像信号和第二RGB Sensor 506生成的第二图像信号，第一图像信号为高分辨率红外图像信号，亮度信噪比相对更高，图像细节信息更丰富；第二图像信号主要为可见光图像信号，可见光图像信号包括亮度信息和色度信息，由于第二RGB Sensor 506的分辨率较低，单个像素的尺寸更大，感光能力更强，在低照场景下尽可能获得更好的色度信息，较好保留图像的色彩信息。处理器507对红外图像信号进行第一图像处理，该第一图像处理可以包括去马赛克处理、颜色空间转换、降噪处理、动态范围处理、对比度处理、图像增强等，由于红外图像信号损失了颜色信息，可以近似认为红外图像信号只包含亮度信息，不包含色度信息，通过第一图像处理，提升红外图像信号的亮度信息的信噪比和图像细节；示例性的，处理器507对第一图像信号进行去马赛克处理，将第一图像信号从raw格式转换为RGB格式的图像，或者进一步的，通过颜色空间转换将RGB格式的图像转换成YUV格式或者其他目标格式的图像。The processor 507 is configured to respectively receive the first image signal generated by the first RGB Sensor 505 and the second image signal generated by the second RGB Sensor 506. The first image signal is a high-resolution infrared image signal with a relatively higher luminance signal-to-noise ratio High, the image detail information is richer; the second image signal is mainly the visible light image signal, the visible light image signal includes brightness information and chromaticity information, because the second RGB Sensor 506 has a lower resolution, the size of a single pixel is larger, and the photosensitive ability Stronger, get better chroma information as much as possible in low-light scenes, and better retain the color information of the image. The processor 507 performs first image processing on the infrared image signal. The first image processing may include demosaicing, color space conversion, noise reduction, dynamic range processing, contrast processing, image enhancement, etc. The infrared image signal loses color. Information, it can be approximated that the infrared image signal contains only brightness information, not chrominance information. Through the first image processing, the signal-to-noise ratio and image details of the brightness information of the infrared image signal are improved; for example, the processor 507 The image signal is subjected to demosaic processing to convert the first image signal from a raw format to an image in an RGB format, or further, an image in an RGB format is converted into an image in a YUV format or other target formats through color space conversion.

处理器507对可见光图像信号进行第二图像处理，该第二图像处理可以包括去马赛克处理、降噪处理、动态范围处理、对比度处理、图像增强等，通过第二图像处理，提升可见光图像信号的信噪比和色彩表现。示例性的，处理器507对第二图像信号进行去马赛克处理，将第二图像信号从raw格式转换为RGB格式的图像，或者进一步的，通过颜色空间转换将RGB格式的图像转换成YUV格式或者其他目标格式的图像。进一步的，处理器507还可以对第二图像信号进行上采样处理，得到第三图像信号，该第三图像信号与第一图像信号具有相同的分辨率，或者也可以说，使得上采样处理后的第二图像信号与第一图像信号具有相同的分辨率，该第三图像信号为上采样处理后的第二图像信号。应当理解，本申请实施例对第二图像处理和上采样处理的顺序不做限定，对第一图像处理和第二图像处理中的多种处理的顺序也不做限定。The processor 507 performs second image processing on the visible light image signal. The second image processing may include demosaicing, noise reduction, dynamic range processing, contrast processing, image enhancement, etc., through the second image processing, the visible light image signal is improved S/N ratio and color performance. Exemplarily, the processor 507 performs demosaic processing on the second image signal, converts the second image signal from a raw format to an image in an RGB format, or further, converts an image in an RGB format into a YUV format through color space conversion, or Images in other target formats. Further, the processor 507 may also perform up-sampling processing on the second image signal to obtain a third image signal, which has the same resolution as the first image signal, or it can also be said that after the up-sampling process The second image signal has the same resolution as the first image signal, and the third image signal is the second image signal after upsampling processing. It should be understood that the embodiment of the present application does not limit the sequence of the second image processing and the up-sampling processing, nor does it limit the sequence of multiple processing in the first image processing and the second image processing.

处理器507，还可以用于分别对亮度信息和色度信息进行处理。示例性的，处理器507分别获取可见光图像信号的色度信息和亮度信息，并获取红外光图像信号的亮度信息，处理器507包括两条处理通路，其中一条处理通路用来将红外光图像信号的亮度信息和可见光图像信号的亮度信息进行融合，得到融合图像的亮度信息，另一条处理通路用来处理见光图像信号的色度信息，例如对可见光图像信号的色度信息进行颜色校正等。进一步的，将融合图像的亮度信息与融合图像的色度信息合到一起，得到融合图像。The processor 507 may also be used to process brightness information and chrominance information respectively. Exemplarily, the processor 507 separately obtains the chromaticity information and brightness information of the visible light image signal, and obtains the brightness information of the infrared light image signal. The processor 507 includes two processing paths, one of which is used to convert the infrared light image signal The brightness information and the brightness information of the visible light image signal are fused to obtain the brightness information of the fused image. Another processing path is used to process the chromaticity information of the visible light image signal, such as color correction of the chromaticity information of the visible light image signal. Further, the brightness information of the fusion image and the chrominance information of the fusion image are combined to obtain the fusion image.

可选的，处理器507还可以对融合得到的图像进行第三图像处理，以进一步提升融合图像的信噪比、细节表现和色彩表现，示例性的，第三图像处理可以包括降噪处理、动态范围处理、对比度处理、图像增强、色彩校正等。Optionally, the processor 507 may also perform third image processing on the fused image to further improve the signal-to-noise ratio, detail performance, and color performance of the fused image. For example, the third image processing may include noise reduction processing, Dynamic range processing, contrast processing, image enhancement, color correction, etc.

在低照场景下，高分辨率的第一RGB Sensor工作在红外光补光模式，提升第一RGB Sensor的亮度感光性能，获取的第一图像信号具有更多的亮度细节、且亮度信息的信噪比更高，低分辨率的第二RGB Sensor工作在红外光截止模式，虽然分辨率下降，但是人眼对颜色的分辨率并不敏感，而且由于低分辨率的RGB Sensor单个像素的尺寸更大，单个像素的感光性能提高，获取的第二图像信号的颜色信息更加还原且颜色信号的信噪比更高，通过低分辨率的RGB Sensor获取颜色信号既不影响人眼对颜色的分辨率，还提升了获取的图像的颜色表现。第一图像信号提供高分辨率的亮度信息，第二图像信号提供更还原信噪比更高的色度信息，将第一图像信号和第二图像信号进行融合，得到最终输出的融合图像，该融合图像细节更丰富、颜色表现更佳、且信噪比更高，提升了低照场景下获取的图像的质量。In low-light scenes, the high-resolution first RGB Sensor works in the infrared light supplement mode to improve the brightness and sensitivity performance of the first RGB Sensor. The acquired first image signal has more brightness details and brightness information. The noise ratio is higher. The low-resolution second RGB Sensor works in infrared light cutoff mode. Although the resolution is reduced, the human eye is not sensitive to the color resolution, and because the low-resolution RGB Sensor has a larger single pixel size Larger, the photosensitive performance of a single pixel is improved, the color information of the second image signal obtained is more restored and the signal-to-noise ratio of the color signal is higher. The color signal obtained through the low-resolution RGB Sensor will not affect the color resolution of the human eye , It also improves the color performance of the acquired image. The first image signal provides high-resolution brightness information, and the second image signal provides chrominance information with a more restored signal-to-noise ratio. The first image signal and the second image signal are fused to obtain the final output fused image. The fusion image has richer details, better color performance, and higher signal-to-noise ratio, which improves the quality of images acquired in low-light scenes.

如图6所示，为本申请实施例提供的另一种示例性的图像处理装置示意图。As shown in FIG. 6, a schematic diagram of another exemplary image processing apparatus provided by an embodiment of this application.

该图像处理装置600包括：第一镜头601、第二镜头602、第一RGB Sensor 605、第二RGB Sensor 606以及处理器607，可选的，该图像处理装置600还可以包括第一滤光模式切换器603，第二滤光模式切换器604和红外灯608，第一RGB Sensor 605的分辨率较高，第二RGB Sensor 606的分辨率较低，第一RGB Sensor 605在低照场景下工作在红外光补光模式，第二RGB Sensor 606在低照场景下工作在红外光截止模式，应当理解，该第一RGB Sensor 605和第二RGB Sensor 606均可以工作在红外光补光模式和红外光截止模式。The image processing device 600 includes: a first lens 601, a second lens 602, a first RGB Sensor 605, a second RGB Sensor 606, and a processor 607. Optionally, the image processing device 600 may also include a first filter mode Switcher 603, second filter mode switcher 604 and infrared lamp 608, the resolution of the first RGB Sensor 605 is higher, the resolution of the second RGB Sensor 606 is lower, and the first RGB Sensor 605 works in low-light scenes In the infrared light supplementary mode, the second RGB Sensor 606 works in the infrared cut-off mode in low-light scenes. It should be understood that the first RGB Sensor 605 and the second RGB Sensor 606 can work in the infrared light supplementary mode and infrared Light cut-off mode.

第一镜头601，用于接收第一场景光信号；The first lens 601 is used to receive a first scene light signal;

第二镜头602，用于接收第二场景光信号；The second lens 602 is used to receive a second scene light signal;

应当理解，第一场景光信号和第二场景光信号为两个镜头接收的同一个场景的两束光信号。It should be understood that the first scene light signal and the second scene light signal are two light signals of the same scene received by the two lenses.

第一场景光信号通过第一滤光模式切换器603到达第一RGB Sensor 605，第二场景光信号通过第二滤光模式切换器604到达第二RGB Sensor 606。第一滤光模式切换器603和第二滤光模式切换器604请参考图5对应部分的描述，此处不再赘述。The light signal of the first scene reaches the first RGB Sensor 605 through the first filter mode switch 603, and the light signal of the second scene reaches the second RGB Sensor 606 through the second filter mode switch 604. For the first filter mode switch 603 and the second filter mode switch 604, please refer to the description of the corresponding part of FIG. 5, and details are not described herein again.

当图像处理装置600工作在低照场景下时，开启红外灯608，用于提供红外光信号，第一滤光模式切换器603将第一RGB Sensor 605切换到红外光补光模式，第二滤光模式切换器604将第二RGB Sensor 606切换到红外光截止模式，应当理解，红外光补光模式表示传感器可以接收红外光并感光红外光以生成红外光图像信号，红外光图像信号丢失了颜色信息，但是具有比较好的图像细节信息；红外光截止模式表示传感器不可以接收红外光，此时传感器感光的主要是可见光信号，生成可见光图像信号，可见光图像信号具有较好的颜色信息。滤光模式切换器如何实现RGB Sensor的工作模式的切换请参考图5相关实施例部分的描述，此处不再赘述。When the image processing device 600 is working in a low-light scene, the infrared lamp 608 is turned on to provide infrared light signals, and the first filter mode switch 603 switches the first RGB Sensor 605 to the infrared light supplement mode, and the second filter The light mode switch 604 switches the second RGB Sensor 606 to the infrared light cut-off mode. It should be understood that the infrared light supplementary light mode means that the sensor can receive infrared light and photosensitive infrared light to generate an infrared light image signal, and the infrared light image signal loses its color. Infrared light cut-off mode means that the sensor cannot receive infrared light. At this time, the sensor is mainly sensitive to visible light signals and generates visible light image signals. The visible light image signals have better color information. For how the filter mode switcher implements the switching of the working mode of the RGB Sensor, please refer to the description of the related embodiment in FIG. 5, which will not be repeated here.

第一RGB Sensor 605、第二RGB Sensor 606以及处理器607的功能可以参照第一RGB Sensor 505、第二RGB Sensor 506以及处理器507部分的描述，此处不再赘述。The functions of the first RGB Sensor 605, the second RGB Sensor 606, and the processor 607 can refer to the description of the first RGB Sensor 505, the second RGB Sensor 506, and the processor 507, which will not be repeated here.

下面给出一种另一种示例性的实施方式。Another exemplary embodiment is given below.

第一RGB Sensor 605，用于获取待成像场景的红外光图像信号，该红外光图像信号包括亮度信息；The first RGB Sensor 605 is used to obtain an infrared light image signal of the scene to be imaged, and the infrared light image signal includes brightness information;

第二RGB Sensor 606，用于获取该待成像场景的可见光图像信号，该可见光图像信号包括色度信息和亮度信息；红外光图像信号的分辨率高于可见光图像信号的分辨率，或者说红外光图像信号的像素数比可见光图像信号的像素数多。The second RGB Sensor 606 is used to obtain the visible light image signal of the scene to be imaged. The visible light image signal includes chromaticity information and brightness information; the resolution of the infrared light image signal is higher than that of the visible light image signal, or infrared light The number of pixels of the image signal is larger than the number of pixels of the visible light image signal.

处理器607，用于分别获取红外光图像信号的亮度信息和可见光图像信号的亮度信息，并融合该红外光图像信号的亮度信息和该可见光图像信号的亮度信息，得到融合图像的亮度信息；The processor 607 is configured to obtain the brightness information of the infrared light image signal and the brightness information of the visible light image signal, and fuse the brightness information of the infrared light image signal and the brightness information of the visible light image signal to obtain the brightness information of the fused image;

处理器607，还用于获取可见光图像信号的色度信息，并对该色度信息进行处理，得到融合图像的色度信息，对色度信息的处理包括但不限于颜色校正、颜色增强、颜色去噪等。The processor 607 is also used to obtain the chromaticity information of the visible light image signal, and process the chromaticity information to obtain the chromaticity information of the fused image. The processing of the chromaticity information includes but is not limited to color correction, color enhancement, and color Denoising etc.

处理器607，还用于将融合图像的亮度信息与融合图像的色度信息合到一起，得到融合图像。The processor 607 is also configured to combine the brightness information of the fused image with the chroma information of the fused image to obtain the fused image.

应当理解，在一种可选的情况中，由于可见光图像信号与红外光图像信号的分辨率不同，在对两者进行融合时，需要将可见光图像信号与红外光图像信号调整为相同分辨率。例如，可以对低分辨率的可见光图像信号进行上采样，使得可见光图像信号的分辨率等于红外光图像信号的分辨率。可以先对可见光图像信号进行上采样处理，再分离可见光图像信号的亮度信息与色度信息；或者也可以先分离可见光图像信号的亮度信息和色度信息之后，再分别对亮度信息和色度信息进行上采样，例如在融合可见光图像信号的亮度信息与红外光图像信号的亮度信息时，对可见光图像信号的亮度信息进行上采样处理。It should be understood that in an optional case, since the resolution of the visible light image signal and the infrared light image signal are different, when the two are fused, the visible light image signal and the infrared light image signal need to be adjusted to the same resolution. For example, the low-resolution visible light image signal can be up-sampled so that the resolution of the visible light image signal is equal to the resolution of the infrared light image signal. The visible light image signal can be up-sampled first, and then the luminance information and chrominance information of the visible light image signal can be separated; or the luminance information and chrominance information of the visible light image signal can be separated first, and then the luminance information and chrominance information can be separately analyzed. Up-sampling is performed, for example, when the brightness information of the visible light image signal is fused with the brightness information of the infrared light image signal, the up-sampling process is performed on the brightness information of the visible light image signal.

在一种可能的实施方式中，在获取可见光图像信号的亮度信息之前，处理器607还用于对可见光图像信号进行上采样处理，得到第二可见光图像信号，使得第二可见光图像信号的分辨率等于红外光图像信号的分辨率，处理器607，具体用于获取第二可见光图像信号的亮度信息，并融合第二可见光图像信号的亮度信息与红外光图像信号的亮度信息。In a possible implementation, before acquiring the brightness information of the visible light image signal, the processor 607 is further configured to perform up-sampling processing on the visible light image signal to obtain the second visible light image signal, so that the resolution of the second visible light image signal is Equal to the resolution of the infrared light image signal, the processor 607 is specifically configured to obtain the brightness information of the second visible light image signal, and fuse the brightness information of the second visible light image signal and the brightness information of the infrared light image signal.

在一种可能的实施方式中，在融合红外光图像的亮度信息和可见光图像的亮度信息时，处理器607还用于对可见光图像信号的亮度信息进行上采样处理，使得上采样处理后的可见光图像信号的亮度信息的分辨率等于红外光图像信号的亮度信息的分辨率。处理器607，还用于对可见光图像信号的色度信息进行上采样处理。图6所示的图像处理装置600具有双镜头、双Sensor结构，每个镜头对应一个RGB Sensor。应当理解，由于两路图像信号来自两个镜头，虽然两路图像信号是对同一个场景的两束光信号进行成像得到的，但是由于两个镜头之间有相对位置，导致两路图像的场景并不是完全一致的，存在一定的视差。在一种可选的方案中，在处理器607对第一图像信号和第二图像信号进行融合之前，需要对第一图像信号和第二图像信号进行对齐处理。In a possible implementation, when fusing the brightness information of the infrared light image and the brightness information of the visible light image, the processor 607 is further configured to perform up-sampling processing on the brightness information of the visible light image signal, so that the up-sampled visible light The resolution of the brightness information of the image signal is equal to the resolution of the brightness information of the infrared light image signal. The processor 607 is further configured to perform up-sampling processing on the chrominance information of the visible light image signal. The image processing device 600 shown in FIG. 6 has a dual-lens and dual-Sensor structure, and each lens corresponds to one RGB Sensor. It should be understood that since the two image signals come from two lenses, although the two image signals are obtained by imaging two light signals of the same scene, there is a relative position between the two lenses, resulting in a scene of two images It is not completely consistent, and there is a certain parallax. In an optional solution, before the processor 607 fuses the first image signal and the second image signal, the first image signal and the second image signal need to be aligned.

在低照场景下，高分辨率的第一RGB Sensor工作在红外光补光模式，提升第一RGB Sensor的亮度感光性能，获取的第一图像信号具有更多的亮度细节、且亮度信息的信噪比更高，低分辨率的第二RGB Sensor工作在红外光截止模式，虽然分辨率下降，但是人眼对颜色的分辨率并不敏感，而且由于低分辨率的RGB Sensor单个像素的尺寸更大，单个像素的感光性能提高，获取的第二图像信号的颜色信息更加还原且颜色信号的信噪比更高，通过低分辨率的RGB Sensor获取颜色信号既不影响人眼对颜色的分辨率，还提升了获取的图像的颜色表现。将第一图像信号和第二图像信号进行融合，得到最终输出的融合图像，该融合图像细节更丰富、颜色表现更佳、且信噪比更高，提升了低照场景下获取的图像的质量。In low-light scenes, the high-resolution first RGB Sensor works in the infrared light supplement mode to improve the brightness and sensitivity performance of the first RGB Sensor. The acquired first image signal has more brightness details and brightness information. The noise ratio is higher. The low-resolution second RGB Sensor works in infrared light cutoff mode. Although the resolution is reduced, the human eye is not sensitive to the color resolution, and because the low-resolution RGB Sensor has a larger single pixel size Larger, the photosensitive performance of a single pixel is improved, the color information of the second image signal obtained is more restored and the signal-to-noise ratio of the color signal is higher. The color signal obtained through the low-resolution RGB Sensor will not affect the color resolution of the human eye , It also improves the color performance of the acquired image. The first image signal and the second image signal are fused to obtain the final output fused image, which has richer details, better color performance, and higher signal-to-noise ratio, which improves the quality of images acquired in low-light scenes .

对于图5和图6所示的图像处理装置，在正常照度下，可以仅使用高分辨率的第一RGB Sensor获取第一图像信号即可满足要求。示例性的，将第一RGB Sensor切换到红外光截止模式，以捕获可见光图像信号，第二RGB Sensor正常感光，处理器仅接收第一图像信号，丢弃第二RGB Sensor获取的图像信号，无需对两路图像信号进行融合处理；在一种可选的方案中，也可以关闭第二RGB Sensor，例如切断对第二RGB Sensor的供电，或者使第二RGB Sensor进入低功耗从而无法感光生成图像信号，由于可以根据场景照度的具体情况自适应开启或关闭第二RGB Sensor或者将第二RGB Sensor调整为低功耗，在确保图像质量的前提下尽可能节省图像处理装置的功耗。在一种可选的方案中，在正常照度下，也可以将两个RGB Sensor均切换到红外光截止模式，获取两路可见光图像信号，并对两路可见光图像信号进行融合处理，以提升最终获取的彩色图像的信噪比和颜色表现。For the image processing devices shown in FIG. 5 and FIG. 6, under normal illuminance, only the first high-resolution RGB Sensor can be used to obtain the first image signal to meet the requirements. Exemplarily, the first RGB Sensor is switched to the infrared light cutoff mode to capture visible light image signals, and the second RGB Sensor is normally light-sensitive, and the processor only receives the first image signal, and discards the image signal obtained by the second RGB Sensor, without any The two image signals are fused; in an optional solution, the second RGB Sensor can also be turned off, such as cutting off the power supply to the second RGB Sensor, or making the second RGB Sensor enter low power consumption so that it cannot generate images Signal, since the second RGB Sensor can be turned on or off adaptively or adjusted to low power consumption according to the specific situation of the scene illumination, the power consumption of the image processing device can be saved as much as possible on the premise of ensuring the image quality. In an optional solution, under normal illumination, both RGB Sensors can be switched to infrared light cutoff mode to obtain two visible light image signals, and perform fusion processing on the two visible light image signals to improve the final The signal-to-noise ratio and color performance of the acquired color image.

基于相同构思，本申请还提供了一种图像信号融合的方法。如图7所示，为本申请实施例提供的一种示例性的图像信号融合的方法流程示意图。该方法先对可见光图像信号进行上采样处理之后再分离可见光图像信号的色度信息和亮度信息，该方法包括：Based on the same concept, this application also provides a method for image signal fusion. As shown in FIG. 7, it is a schematic flowchart of an exemplary image signal fusion method provided by an embodiment of this application. The method first performs up-sampling processing on the visible light image signal and then separates the chrominance information and brightness information of the visible light image signal. The method includes:

获取可见光图像信号，该可见光图像信号包括亮度信息和色度信息，该可见光图像信号具有第一分辨率；Acquiring a visible light image signal, the visible light image signal including luminance information and chromaticity information, and the visible light image signal having a first resolution;

获取红外光图像信号，该红外光图像信号包括亮度信息，该红外光图像信号具有第二分辨率，第一分辨率低于第二分辨率。An infrared light image signal is acquired, the infrared light image signal includes brightness information, the infrared light image signal has a second resolution, and the first resolution is lower than the second resolution.

对获取的可见光图像信号进行上采样处理，使得处理后的可见光图像信号的分辨率与红外光图像信号的分辨率一致。Up-sampling processing is performed on the acquired visible light image signal, so that the resolution of the processed visible light image signal is consistent with the resolution of the infrared light image signal.

从处理后的可见光图像信号中提取第一亮度信息，该第一亮度信息为排除了红外光信息的亮度信息；对第一亮度信息进行滤波去噪处理，得到第二亮度信息，所述第二亮度信息为第一亮度信息滤除高频噪声之后得到的亮度信息，示例性的，可以采用双边滤波器或导向滤波器(Guided Filter)对第一亮度信息进行滤波处理，去除了高频噪声，得到的第二亮度信息包含了可见光图像信号的中低频信息和平均亮度信息。The first brightness information is extracted from the processed visible light image signal, the first brightness information is brightness information excluding infrared light information; the first brightness information is filtered and denoised, and the second brightness information is obtained. The brightness information is the brightness information obtained after the first brightness information is filtered out of high-frequency noise. Illustratively, a bilateral filter or a guided filter (Guided Filter) may be used to filter the first brightness information to remove high-frequency noise. The obtained second brightness information contains the low-mid frequency information and average brightness information of the visible light image signal.

从红外光图像信号中提取第三亮度信息，该第三亮度信息为携带有红外光信息的亮度信息，应当理解，红外光图像信号仅包括亮度信息，不包括色度信息，因此获取的红外光图像信号可直接作为第三亮度信息；The third brightness information is extracted from the infrared light image signal. The third brightness information is the brightness information that carries the infrared light information. It should be understood that the infrared light image signal only includes brightness information and does not include chromaticity information. Therefore, the acquired infrared light The image signal can be directly used as the third brightness information;

获取第三亮度信息的高频细节信息，该高频细节信息主要为红外光图像信号的高频亮度细节信息，本领域人员可知，图像的细节信息和噪声转换到频域对应高频信息，由于红外光图像信号的信噪比很高，噪声很少，高频部分主要为图像的亮度细节信息。Obtain the high-frequency detail information of the third brightness information. The high-frequency detail information is mainly the high-frequency brightness detail information of the infrared light image signal. Those skilled in the art know that the image detail information and noise are converted to the frequency domain corresponding to the high-frequency information. The signal-to-noise ratio of the infrared light image signal is very high, the noise is very little, and the high frequency part is mainly the brightness detail information of the image.

具体的，获取第三亮度信息的高频细节信息，包括：Specifically, acquiring the high-frequency detail information of the third brightness information includes:

对第三亮度信息进行滤波处理，得到第四亮度信息，示例性的，可以采用双边滤波器或导向滤波器等对第三亮度信息进行滤波处理，去除了高频信息，得到的第四亮度信息包含了红外光图像信号的中低频信息和平均亮度信息。The third brightness information is filtered to obtain the fourth brightness information. Illustratively, a bilateral filter or a guided filter can be used to filter the third brightness information to remove high frequency information to obtain the fourth brightness information. Contains the low frequency information and average brightness information of the infrared image signal.

第三亮度信息减去第四亮度信息，得到第五亮度信息，由于红外光图像信号的信噪比较高，经过双边滤波器滤波去除的高频信息主要是红外光图像信号的高频细节信息，因此该第五亮度信息主要包含了红外光图像信号的高频亮度细节。The fourth brightness information is subtracted from the third brightness information to obtain the fifth brightness information. Since the signal-to-noise ratio of the infrared light image signal is high, the high frequency information removed by the bilateral filter is mainly the high frequency detail information of the infrared light image signal Therefore, the fifth brightness information mainly contains the high-frequency brightness details of the infrared light image signal.

叠加第二亮度信息和第五亮度信息，得到融合亮度信息。由于第二亮度信息主要包含了可见光图像信号的中低频信息和平均亮度信息，第五亮度信息主要包含红外光图像信号的高频亮度细节，结合两者得到融合亮度信息，一方面可以尽量保持融合后的亮度信息的平均亮度与可见光图像的平均亮度接近，另一方面也包含了来自红外光图像信号中的更多细节信息。The second brightness information and the fifth brightness information are superimposed to obtain the fused brightness information. Since the second brightness information mainly contains the middle and low frequency information and average brightness information of the visible light image signal, the fifth brightness information mainly contains the high frequency brightness details of the infrared light image signal. Combine the two to obtain the fused brightness information. On the one hand, the fusion can be kept as much as possible. The average brightness of the subsequent brightness information is close to the average brightness of the visible light image, on the other hand, it also contains more detailed information from the infrared light image signal.

从处理后的可见光图像信号中提取获取第一色度信息，该第一色度信息为可见光图像信号的色度信息；对第一色度信息进行颜色校正，得到第二色度信息；示例性的，可以根据融合后的亮度与可见光图像的亮度之间的差异，对色度信息做相同的补偿。可选的，对第一色度信息的处理还可以包括颜色增强、颜色去噪等。Extract and obtain first chromaticity information from the processed visible light image signal, where the first chromaticity information is the chromaticity information of the visible light image signal; perform color correction on the first chromaticity information to obtain the second chromaticity information; exemplary Yes, the chrominance information can be compensated the same according to the difference between the fused brightness and the brightness of the visible light image. Optionally, the processing of the first chrominance information may also include color enhancement, color denoising, and the like.

组合该融合亮度信息与该第二色度信息，得到完整的融合图像。该融合图像兼具良好的颜色表现和更多的细节信息。Combining the fused brightness information and the second chrominance information to obtain a complete fused image. The fusion image has both good color performance and more detailed information.

在一种可能的实施方式中，在对可见光图像信号进行上采样处理之前，还可以对可见光图像信号进行去马赛克、颜色空间转换、图像增强、去噪处理等；对应的，在对第三亮度信息进行滤波处理之前，也可以对红外光图像信号进行去马赛克、颜色空间转换、图像增强、去噪处理等。In a possible implementation manner, before upsampling the visible light image signal, demosaicing, color space conversion, image enhancement, denoising processing, etc. can also be performed on the visible light image signal; correspondingly, the third brightness Before the information is filtered, the infrared image signal can also be demosaiced, color space conversion, image enhancement, and denoising processing.

在一种可能的实施方式中，在得到完整的融合图像之后，还可以对融合图像进一步进行图像处理，该图像处理包括但不限于：对比度调整、图像增强、颜色空间转换等。例如，如果图像融合过程是在RGB空间进行的，而目标输出图像为YCC色彩空间的图像格式，则可以经过颜色空间转换将融合图像从RGB格式转换为YCC色彩空间的图像格式。该方法将上采样处理后的可见光图像信号分离为亮度信息和色度信息，并对亮度信息和色度信息进行分别处理，最后再合并处理后的亮度信息与色度信息，使得得到的融合图像的亮度和色度表现都得到了提升，提高了低照场景下获得的图像质量。In a possible implementation manner, after the complete fusion image is obtained, further image processing may be performed on the fusion image. The image processing includes but is not limited to: contrast adjustment, image enhancement, color space conversion, and the like. For example, if the image fusion process is performed in the RGB space and the target output image is an image format of the YCC color space, the fusion image can be converted from the RGB format to the image format of the YCC color space through color space conversion. This method separates the up-sampled visible light image signal into luminance information and chrominance information, and processes the luminance information and chrominance information separately, and finally combines the processed luminance information and chrominance information to make the resulting fusion image The brightness and chroma performance of the camera has been improved, and the image quality obtained in low-light scenes has been improved.

如图8所示，为本申请实施例提供的一种示例性的图像信号处理的方法流程示意图。该方法包括：As shown in FIG. 8, it is a schematic flowchart of an exemplary image signal processing method provided by an embodiment of this application. The method includes:

801、接收目标场景的第一图像信号，该第一图像信号包括第一亮度信息，该第一亮度信息是携带有红外光信息的亮度信息，该第一图像信号具有第一分辨率；801. Receive a first image signal of a target scene, where the first image signal includes first brightness information, the first brightness information is brightness information carrying infrared light information, and the first image signal has a first resolution.

802、接收该目标场景的第二图像信号，该第二图像信号包括第二亮度信息和第二色度信息，该第二亮度信息是排除了红外光信息的亮度信息，该第二图像信号具有第二分辨率，该第一分辨率高于该第二分辨率；802. Receive a second image signal of the target scene, where the second image signal includes second brightness information and second chrominance information, the second brightness information is brightness information excluding infrared light information, and the second image signal has A second resolution, where the first resolution is higher than the second resolution;

803、融合该第一图像信号和该第二图像信号，得到融合图像803. Fuse the first image signal and the second image signal to obtain a fused image

该第一图像信号为高分辨率的红外光图像信号，包含更多的亮度细节，亮度信噪比更高；第二图像信号为低分辨率的可见光图像信号，单个像素感光能力更强，包含了更多色度信息。将该第一图像信号和第二图像信号融合得到的融合图像细节更丰富、信噪比更高、颜色表现更佳。The first image signal is a high-resolution infrared light image signal, which contains more brightness details and has a higher brightness signal-to-noise ratio; the second image signal is a low-resolution visible light image signal, which has a stronger sensitivity to a single pixel, including More chromaticity information. The fused image obtained by fusing the first image signal and the second image signal has richer details, higher signal-to-noise ratio, and better color performance.

本申请实施例包括亮度和色度两个处理通道，在进行图像融合时，对亮度信号和色度信号分别进行处理。在亮度处理通道中，对红外光图像信号和可见光图像信号的亮度进行融合。第四亮度信息为可见光图像信号滤除高频噪声之后得到的亮度信息，包含了可见光图像信号的中低频信息和平均亮度信息；由于红外光图像信号的信噪比很高，噪声很少，第一亮度信息的高频细节信息主要为图像的亮度细节信息；由于第一亮度信息的高频细节信息主要包含红外光图像信号的高频亮度细节，第四亮度信息主要包含了可见光图像信号的中低频信息和平均亮度信息，结合两者得到融合亮度信息，一方面可以尽量保持融合后的亮度信息的平均亮度与可见光图像的平均亮度接近，另一方面也包含了来自红外光图像信号中的更多细节信息。The embodiment of the application includes two processing channels of luminance and chrominance. When performing image fusion, the luminance signal and the chrominance signal are processed separately. In the brightness processing channel, the brightness of the infrared light image signal and the visible light image signal are fused. The fourth brightness information is the brightness information obtained after the visible light image signal is filtered out of high-frequency noise, which contains the medium and low frequency information and the average brightness information of the visible light image signal; since the signal-to-noise ratio of the infrared light image signal is very high, the noise is very small. The high-frequency detail information of the first brightness information is mainly the brightness detail information of the image; because the high-frequency detail information of the first brightness information mainly contains the high-frequency brightness details of the infrared light image signal, the fourth brightness information mainly contains the middle of the visible light image signal. Low-frequency information and average brightness information are combined to obtain fused brightness information. On the one hand, the average brightness of the fused brightness information can be kept close to the average brightness of the visible light image. On the other hand, it also contains more information from the infrared light image signal. More detailed information.

本申请实施例还提供一种计算机可读存储介质，该计算机可读存储介质中存储有指令，当其在计算机或处理器上运行时，使得计算机或处理器执行本申请实施例提供的任一个方法。The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores instructions that, when run on a computer or processor, cause the computer or the processor to execute any one provided in the embodiments of the present application. method.

本申请实施例还提供一种包含指令的计算机程序产品，当其在计算机或处理器上运行时，使得计算机或处理器执行本申请实施例提供的任一个方法。The embodiments of the present application also provide a computer program product containing instructions, which when run on a computer or a processor, cause the computer or the processor to execute any of the methods provided in the embodiments of the present application.

以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

一种图像信号处理的装置，其特征在于，所述装置包括：第一接收接口、第二接收接口和处理器；An image signal processing device, characterized in that the device includes: a first receiving interface, a second receiving interface, and a processor;

所述第一接收接口，用于接收目标场景的第一图像信号，所述第一图像信号包括第一亮度信息，所述第一亮度信息是携带有红外光信息的亮度信息，所述第一图像信号具有第一分辨率；The first receiving interface is configured to receive a first image signal of a target scene, the first image signal includes first brightness information, the first brightness information is brightness information carrying infrared light information, and the first The image signal has a first resolution;

所述第二接收接口，用于接收所述目标场景的第二图像信号，所述第二图像信号包括第二亮度信息和第二色度信息，所述第二亮度信息是排除了红外光信息的亮度信息，所述第二图像信号具有第二分辨率，所述第一分辨率高于所述第二分辨率；The second receiving interface is configured to receive a second image signal of the target scene, the second image signal includes second brightness information and second chroma information, and the second brightness information excludes infrared light information Brightness information, the second image signal has a second resolution, and the first resolution is higher than the second resolution;

所述处理器，用于融合所述第一图像信号和所述第二图像信号，得到融合图像。The processor is configured to fuse the first image signal and the second image signal to obtain a fused image.
根据权利要求1所述的装置，其特征在于，所述处理器具体用于：The device according to claim 1, wherein the processor is specifically configured to:

对所述第二图像信号进行上采样处理，得到第三图像信号，所述第三图像信号具有所述第一分辨率；Performing up-sampling processing on the second image signal to obtain a third image signal, the third image signal having the first resolution;

融合所述第一图像信号和所述第三图像信号，得到所述融合图像。Fusing the first image signal and the third image signal to obtain the fused image.
根据权利要求2所述的装置，其特征在于，所述第三图像信号包括第三亮度信息，所述融合图像包括融合亮度信息，所述处理器具体用于：The device according to claim 2, wherein the third image signal includes third brightness information, the fused image includes fused brightness information, and the processor is specifically configured to:

获取所述第一亮度信息的高频细节信息，所述第一亮度信息的高频细节信息包括所述第一图像信号的亮度细节信息；Acquiring high frequency detail information of the first brightness information, where the high frequency detail information of the first brightness information includes the brightness detail information of the first image signal;

对所述第三亮度信息进行去噪处理，以滤除所述第三亮度信息的高频噪声，得到第四亮度信息；Performing denoising processing on the third brightness information to filter out high frequency noise of the third brightness information to obtain fourth brightness information;

叠加所述第一亮度信息的高频细节信息和所述第四亮度信息，得到所述融合亮度信息。The high-frequency detail information of the first brightness information and the fourth brightness information are superimposed to obtain the fused brightness information.
根据权利要求3所述的装置，其特征在于，所述第三图像信号还包括第三色度信息，所述融合图像还包括融合色度信息，所述处理器具体还用于：The device according to claim 3, wherein the third image signal further includes third chrominance information, the fused image further includes fused chrominance information, and the processor is further configured to:

从所述第三图像信号中提取所述第三色度信息；Extracting the third chrominance information from the third image signal;

对所述第三色度信息进行颜色校正，得到所述融合色度信息。Perform color correction on the third chromaticity information to obtain the fused chromaticity information.
根据权利要求4所述的装置，其特征在于，所述处理器具体用于：The device according to claim 4, wherein the processor is specifically configured to:

根据所述融合亮度信息与所述第三亮度信息的差值对所述第三色度信息进行补偿，以得到所述融合色度信息。Compensating the third chrominance information according to the difference between the fused luminance information and the third luminance information to obtain the fused chrominance information.
根据权利要求4或5所述的装置，其特征在于，所述处理器具体还用于：The device according to claim 4 or 5, wherein the processor is further configured to:

组合所述融合亮度信息和所述融合色度信息，得到所述融合图像。Combining the fused brightness information and the fused chrominance information to obtain the fused image.
根据权利要求1至6任一项所述的装置，其特征在于，所述装置还包括：第一图像传感器和第二图像传感器，所述第一图像传感器具有所述第一分辨率，所述第二图像传感器具有所述第二分辨率；The device according to any one of claims 1 to 6, wherein the device further comprises: a first image sensor and a second image sensor, the first image sensor having the first resolution, the The second image sensor has the second resolution;

所述第一图像传感器，用于工作在红外补光模式，以生成所述第一图像信号；The first image sensor is configured to work in an infrared supplementary light mode to generate the first image signal;

所述第二图像传感器，用于工作在红外光截止模式，以生成所述第二图像信号。The second image sensor is configured to work in an infrared light cut-off mode to generate the second image signal.
根据权利要求7所述的装置，其特征在于，所述第一图像传感器为第一RGB Sensor，所述第二图像传感器为第二RGB Sensor。7. The device of claim 7, wherein the first image sensor is a first RGB Sensor, and the second image sensor is a second RGB Sensor.
根据权利要求7或8所述的装置，其特征在于，所述装置还包括：The device according to claim 7 or 8, wherein the device further comprises:

第一滤光模式切换器，用于将所述第一图像传感器切换到所述红外光补光模式。The first filter mode switch is used to switch the first image sensor to the infrared light supplementary light mode.
根据权利要求9所述的装置，其特征在于，所述第一滤光模式切换器包括：全透光谱滤光片和红外截止滤光片；9. The device of claim 9, wherein the first filter mode switch comprises: a full transmission spectrum filter and an infrared cut filter;

所述第一滤光模式切换器，具体用于切换到所述全透光谱滤光片，以使得所述第一图像传感器感光红外光信号生成所述红外光图像信号。The first filter mode switch is specifically configured to switch to the full transmission spectrum filter, so that the first image sensor receives infrared light signals to generate the infrared light image signals.
根据权利要求7或8所述的装置，其特征在于，所述装置还包括：The device according to claim 7 or 8, wherein the device further comprises:

第二滤光模式切换器，用于将所述第二图像传感器切换到所述红外光截止模式。The second filter mode switch is used to switch the second image sensor to the infrared light cutoff mode.
根据权利要求11所述的装置，其特征在于，所述第二滤光模式切换器包括：所述全透光谱滤光片和所述红外截止滤光片；11. The device according to claim 11, wherein the second filter mode switch comprises: the full transmission spectrum filter and the infrared cut filter;

所述第二滤光模式切换器，具体用于在所述切换到所述红外截止滤光片，以使得所述第二图像传感器感光可见光信号生成所述可见光图像信号。The second filter mode switcher is specifically configured to switch to the infrared cut filter, so that the second image sensor receives visible light signals to generate the visible light image signals.
根据权利要求7至12任一项所述的装置，其特征在于，所述装置还包括：镜头和分光器；The device according to any one of claims 7 to 12, wherein the device further comprises: a lens and a beam splitter;

所述镜头，用于接收所述目标场景的光信号；The lens is used to receive the light signal of the target scene;

所述分光器，用于将所述目标场景的光信号分为第一光信号和第二光信号，所述第一光信号为送给所述第一图像传感器的光信号，所述第二光信号为送给所述第二图像传感器的光信号。The optical splitter is configured to divide the optical signal of the target scene into a first optical signal and a second optical signal, the first optical signal is the optical signal sent to the first image sensor, and the second The light signal is the light signal sent to the second image sensor.
根据权利要求7至12任一项所述的装置，其特征在于，所述装置还包括：The device according to any one of claims 7 to 12, wherein the device further comprises:

第一镜头，用于接收所述目标场景的第一光信号，所述第一光信号为送给所述第一图像传感器的光信号；A first lens for receiving a first light signal of the target scene, where the first light signal is a light signal sent to the first image sensor;

第二镜头，用于接收所述目标场景的第二光信号，所述第二光信号为送给所述第二图像传感器的光信号。The second lens is used to receive a second light signal of the target scene, where the second light signal is a light signal sent to the second image sensor.
根据权利要求14所述的装置，其特征在于，所述处理器还用于：The device according to claim 14, wherein the processor is further configured to:

对所述第一图像信号和所述第二图像信号进行对齐处理。Perform alignment processing on the first image signal and the second image signal.
一种图像信号处理的方法，其特征在于，所述方法包括：An image signal processing method, characterized in that the method includes:

接收目标场景的第一图像信号，所述第一图像信号包括第一亮度信息，所述第一亮度信息是携带有红外光信息的亮度信息，所述第一图像信号具有第一分辨率；Receiving a first image signal of a target scene, where the first image signal includes first brightness information, the first brightness information is brightness information carrying infrared light information, and the first image signal has a first resolution;

接收所述目标场景的第二图像信号，所述第二图像信号包括第二亮度信息和第二色度信息，所述第二亮度信息是排除了红外光信息的亮度信息，所述第二图像信号具有第二分辨率，所述第一分辨率高于所述第二分辨率；A second image signal of the target scene is received, the second image signal includes second brightness information and second chrominance information, the second brightness information is brightness information excluding infrared light information, and the second image The signal has a second resolution, the first resolution being higher than the second resolution;

融合所述第一图像信号和所述第二图像信号，得到融合图像。Fusing the first image signal and the second image signal to obtain a fused image.
根据权利要求16所述的方法，其特征在于，所述融合所述第一图像信号和所述第二图像信号，得到融合图像，具体包括：The method according to claim 16, wherein the fusing the first image signal and the second image signal to obtain a fused image specifically comprises:

对所述第二图像信号进行上采样处理，得到第三图像信号，所述第三图像信号具有所述第一分辨率；Performing up-sampling processing on the second image signal to obtain a third image signal, the third image signal having the first resolution;

融合所述第一图像信号和所述第三图像信号，得到所述融合图像。Fusing the first image signal and the third image signal to obtain the fused image.
根据权利要求17所述的方法，其特征在于，所述第三图像信号包括第三亮度信息,所述融合图像包括融合亮度信息，所述融合所述第一图像信号和所述第三图像信号，得到所述融合图像，具体包括：The method according to claim 17, wherein the third image signal includes third brightness information, the fused image includes fused brightness information, and the first image signal and the third image signal are fused , To obtain the fused image, which specifically includes:

获取所述第一亮度信息的高频细节信息，所述第一亮度信息的高频细节信息包括所述第一图像信号的亮度细节信息；Acquiring high frequency detail information of the first brightness information, where the high frequency detail information of the first brightness information includes the brightness detail information of the first image signal;

对所述第三亮度信息进行去噪处理，以滤除所述第三亮度信息的高频噪声，得到第四亮度信息；叠加所述第一亮度信息的高频细节信息和所述第四亮度信息，得到所述融合亮度信息。Perform denoising processing on the third brightness information to filter out high-frequency noise of the third brightness information to obtain fourth brightness information; superimpose the high-frequency detail information of the first brightness information and the fourth brightness Information to obtain the fused brightness information.
根据权利要求18所述的方法，其特征在于，所述获取所述第一亮度信息的高频细节信息，具体包括：The method according to claim 18, wherein the obtaining high frequency detail information of the first brightness information specifically comprises:

对所述第一亮度信息进行滤波处理，得到滤波后的第一亮度信息；Filtering the first brightness information to obtain filtered first brightness information;

所述第一亮度信息减去所述滤波后的第一亮度信息，得到所述第一亮度信息的高频细节信息。Subtracting the filtered first brightness information from the first brightness information to obtain high-frequency detail information of the first brightness information.
根据权利要求18或19所述的方法，其特征在于，所述第三图像信号还包括第三色度信息，所述融合图像还包括融合色度信息，所述融合所述第一图像信号和所述第三图像信号，得到所述融合图像，具体还包括：The method according to claim 18 or 19, wherein the third image signal further includes third chrominance information, the fused image further includes fused chrominance information, and the first image signal is combined with The third image signal to obtain the fused image specifically further includes:

从所述第三图像信号中提取所述第三色度信息；Extracting the third chrominance information from the third image signal;

对所述第三色度信息进行颜色校正，得到所述融合色度信息。Perform color correction on the third chromaticity information to obtain the fused chromaticity information.
根据权利要求20所述的方法，其特征在于，所述对所述第三色度信息进行颜色校正，得到所述融合色度信息，具体包括：22. The method of claim 20, wherein said performing color correction on said third chromaticity information to obtain said fused chromaticity information specifically comprises:

根据所述融合亮度信息与所述第三亮度信息的差值对所述第三色度信息进行补偿，以得到所述融合色度信息。Compensating the third chrominance information according to the difference between the fused luminance information and the third luminance information to obtain the fused chrominance information.
根据权利要求20或21所述的方法，其特征在于，所述融合所述第一图像信号和所述第三图像信号，得到所述融合图像，具体还包括：The method according to claim 20 or 21, wherein the fusing the first image signal and the third image signal to obtain the fused image specifically further comprises:

组合所述融合亮度信息和所述融合色度信息，得到所述融合图像。Combining the fused brightness information and the fused chrominance information to obtain the fused image.
根据权利要求16至22任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 16 to 22, wherein the method further comprises:

第一图像传感器感光红外光信号，生成所述第一图像信号；The first image sensor receives infrared light signals to generate the first image signal;

第二图像传感器感光可见光信号，生成所述第二图像信号，所述第一图像传感器具有所述第一分辨率，所述第二图像传感器具有所述第二分辨率。The second image sensor receives visible light signals to generate the second image signal, the first image sensor has the first resolution, and the second image sensor has the second resolution.
根据权利要求23所述的方法，其特征在于，所述第一图像传感器为第一RGB Sensor，所述第二图像传感器为第二RGB Sensor。22. The method of claim 23, wherein the first image sensor is a first RGB Sensor, and the second image sensor is a second RGB Sensor.
根据权利要求23或24所述的方法，其特征在于，所述方法还包括：The method according to claim 23 or 24, wherein the method further comprises:

将所述第一图像传感器切换到红外光补光模式，以感光所述红外光信号；Switching the first image sensor to an infrared light supplementary light mode to receive the infrared light signal;

将所述第二图像传感器切换到红外光截止模式，以感光所述可见光信号。The second image sensor is switched to an infrared light cut-off mode to receive the visible light signal.
根据权利要求25所述的方法，其特征在于，所述将所述第一图像传感器切换到红外光补光模式，以感光所述红外光信号，具体包括：The method according to claim 25, wherein the switching the first image sensor to an infrared light supplementary light mode to receive the infrared light signal specifically comprises:

将第一滤光模式切换器切换到全透光谱滤光片，以使得所述第一图像传感器切换到所述红外光补光模式；Switching the first filter mode switch to a full transmission spectral filter, so that the first image sensor is switched to the infrared light supplementary light mode;

所述将所述第二图像传感器切换到红外光截止模式，以感光所述可见光信号，具体包括：The switching the second image sensor to the infrared light cut-off mode to receive the visible light signal specifically includes:

将第二滤光模式切换器切换到红外截止滤光片，以使得所述第一图像传感器切换到所述红外光截止模式。The second filter mode switch is switched to an infrared cut filter, so that the first image sensor is switched to the infrared cut mode.
根据权利要求23至26任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 23 to 26, wherein the method further comprises:

接收所述目标场景的光信号；Receiving the optical signal of the target scene;

将所述目标场景的光信号分为第一光信号和第二光信号，所述第一光信号为送给所述第一图像传感器的光信号，所述第二光信号为送给所述第二图像传感器的光信号。The optical signal of the target scene is divided into a first optical signal and a second optical signal, the first optical signal is the optical signal sent to the first image sensor, and the second optical signal is the optical signal sent to the Optical signal of the second image sensor.
根据权利要求23至26任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 23 to 26, wherein the method further comprises:

接收所述目标场景的第一光信号，所述第一光信号为送给所述第一图像传感器的光信号；Receiving a first light signal of the target scene, where the first light signal is a light signal sent to the first image sensor;

接收所述目标场景的第二光信号，所述第二光信号为送给所述第二图像传感器的光信号。Receiving a second light signal of the target scene, where the second light signal is a light signal sent to the second image sensor.
根据权利要求28所述的方法，其特征在于，在所述融合所述第一图像信号和所述第二图像信号，得到融合图像之前，所述方法还包括：The method according to claim 28, characterized in that, before said fusing the first image signal and the second image signal to obtain a fused image, the method further comprises:

对所述第一图像信号和所述第二图像信号进行对齐处理，得到处理后的第一图像信号和处理后的第二图像信号；Performing alignment processing on the first image signal and the second image signal to obtain a processed first image signal and a processed second image signal;

所述融合所述第一图像信号和所述第二图像信号，得到融合图像，具体包括：The fusing the first image signal and the second image signal to obtain a fused image specifically includes:

融合所述处理后的第一图像信号和所述处理后的第二图像信号，得到所述融合图像。Fusion of the processed first image signal and the processed second image signal to obtain the fused image.
一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当所述指令在计算机或处理器上运行时，使得所述计算机或处理器执行如权利要求16-29任一项所述的方法。A computer-readable storage medium, the computer-readable storage medium stores instructions, when the instructions run on a computer or a processor, the computer or the processor executes any one of claims 16-29 The method described.
一种包含指令的计算机程序产品，当其在计算机或处理器上运行时，使得所述计算机或处理器执行如权利要求16-29任一项所述的方法。A computer program product containing instructions that, when run on a computer or processor, causes the computer or processor to execute the method according to any one of claims 16-29.