WO2021196554A1

WO2021196554A1 - Image sensor, processing system and method, electronic device, and storage medium

Info

Publication number: WO2021196554A1
Application number: PCT/CN2020/119966
Authority: WO
Inventors: 杨鑫
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-03-30
Filing date: 2020-10-09
Publication date: 2021-10-07
Also published as: CN111432099B; CN111432099A

Abstract

An image sensor (10), a high dynamic range image processing system (100) and method, an electronic device (1000), and a readable storage medium (400). The high dynamic range image processing system (100) comprises the image sensor (10), a color high dynamic fusion unit (30), and an image processor (20). Pixels in a same sub-unit of a pixel array (11) in the image sensor (10) are exposed at different time to obtain multiple frames of original images. The color high dynamic fusion unit (30) and the image processor (20) are used for performing high dynamic range processing, image processing, and fusion algorithm processing on the multiple frames of original images to obtain a target image.

Description

图像传感器、处理***及方法、电子设备和存储介质Image sensor, processing system and method, electronic device and storage medium

优先权信息Priority information

本申请请求2020年3月30日向中国国家知识产权局提交的、专利申请号为202010233813.0的专利申请的优先权和权益，并且通过参照将其全文并入此处。This application requests the priority and rights of the patent application with the patent application number 202010233813.0 filed with the State Intellectual Property Office of China on March 30, 2020, and the full text is incorporated herein by reference.

技术领域Technical field

本申请涉及图像处理技术领域，特别涉及一种图像传感器、高动态范围图像处理***及方法、电子设备和计算机可读存储介质。This application relates to the field of image processing technology, and in particular to an image sensor, a high dynamic range image processing system and method, an electronic device, and a computer-readable storage medium.

背景技术Background technique

普通相机因受到动态范围的限制，不能记录极端亮或者暗的细节。尤其在拍摄场景的光比较大的时候，很容易出现过曝或者欠曝的情况。具有高动态范围(High-Dynamic Range,HDR)功能的相机，在大光比情况下拍摄图像，无论是高光、暗位都能够具有比普通相机更佳的表现。Ordinary cameras cannot record extremely bright or dark details due to the limitation of dynamic range. Especially when the light in the shooting scene is relatively large, it is prone to overexposure or underexposure. A camera with a high dynamic range (High-Dynamic Range, HDR) function can capture images in a large light ratio, and it can perform better than ordinary cameras in both high light and dark positions.

发明内容Summary of the invention

本申请实施方式提供一种图像传感器、高动态范围图像处理***及方法、电子设备和计算机可读存储介质。The embodiments of the present application provide an image sensor, a high dynamic range image processing system and method, an electronic device, and a computer-readable storage medium.

本申请实施方式提供的图像传感器包括像素阵列。所述像素阵列包括多个全色感光像素和多个彩色感光像素。所述彩色感光像素具有比所述全色感光像素更窄的光谱响应。所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元。每个所述子单元包括多个单颜色感光像素及多个全色感光像素。当所述图像传感器中的像素阵列曝光时，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光。其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像。以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像。以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像。The image sensor provided by the embodiment of the present application includes a pixel array. The pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. The color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. When the pixel array in the image sensor is exposed, for multiple photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, and at least one single-color photosensitive pixel is less than Exposure at a second exposure time of the first exposure time, at least one of the panchromatic photosensitive pixels is exposed at a third exposure time that is less than or equal to the first exposure time, and at least one of the panchromatic photosensitive pixels is less than the Exposure at the fourth exposure time of the first exposure time. Wherein, the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image. The single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image. The panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain a panchromatic original image .

本申请实施方式提供的高动态范围图像处理***包括图像传感器、彩色高动态融合单元和图像处理器。所述图像传感器包括像素阵列。所述像素阵列包括多个全色感光像素和多个彩色感光像素。所述彩色感光像素具有比所述全色感光像素更窄的光谱响应。所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元。每个所述子单元包括多个单颜色感光像素及多个全色感光像素。当所述图像传感器中的像素阵列曝光时，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光。其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像。以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像。以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像。所述彩色高动态融合单元和所述图像处理器用于对所述第一彩色原始图像、所述第二彩色原始图像和所述全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。The high dynamic range image processing system provided by the embodiment of the present application includes an image sensor, a color high dynamic fusion unit, and an image processor. The image sensor includes a pixel array. The pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. The color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. When the pixel array in the image sensor is exposed, for multiple photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, and at least one single-color photosensitive pixel is less than Exposure at a second exposure time of the first exposure time, at least one of the panchromatic photosensitive pixels is exposed at a third exposure time that is less than or equal to the first exposure time, and at least one of the panchromatic photosensitive pixels is less than the Exposure at the fourth exposure time of the first exposure time. Wherein, the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image. The single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image. The panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain a panchromatic original image . The color high dynamic fusion unit and the image processor are used to perform high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the panchromatic original image. Target image.

本申请实施方式提供的高动态范围图像处理方法用于高动态范围图像处理***。所述高动态范围图像处理***包括图像传感器。所述图像传感器包括像素阵列。所述像素阵列包括多个全色感光像素和多个彩色感光像素。所述彩色感光像素具有比所述全色感光像素更窄的光谱响应。所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元。每个所述子单元包括多个单颜色感光像素及多个全色感光像素。所述高动态范围图像处理方法包括：控制所述像素阵列曝光，其中，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光，其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像，以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像；和对所述第一彩色原始图像、所述第二彩色原始图像和所述全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。The high dynamic range image processing method provided by the embodiment of the present application is used in a high dynamic range image processing system. The high dynamic range image processing system includes an image sensor. The image sensor includes a pixel array. The pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. The color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel. The pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The high dynamic range image processing method includes: controlling the exposure of the pixel array, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed at a first exposure time, and at least one The single-color photosensitive pixel is exposed at a second exposure time less than the first exposure time, at least one of the full-color photosensitive pixel is exposed at a third exposure time less than or equal to the first exposure time, and at least one of the full-color photosensitive pixels is exposed at a third exposure time less than or equal to the first exposure time. The color photosensitive pixels are exposed at a fourth exposure time that is less than the first exposure time, wherein the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image, and The single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image, the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the first full-color information The panchromatic photosensitive pixels exposed for four exposure times generate second panchromatic information to obtain a panchromatic original image; and perform processing on the first color original image, the second color original image, and the panchromatic original image High dynamic range processing, image processing and fusion algorithm processing get the target image.

本申请实施方式提供的电子设备包括镜头、壳体及上述任意一项实施例所述的高动态范围图像处理***。The electronic equipment provided by the embodiments of the present application includes a lens, a housing, and the high dynamic range image processing system described in any one of the above embodiments.

本申请实施方式提供的包含计算机程序的非易失性计算机可读存储介质中，所述计算机程序被处理器执行时，使得所述处理器执行上述任意一项实施例所述的高动态范围图像处理方法。In the non-volatile computer-readable storage medium containing a computer program provided by the embodiment of the present application, when the computer program is executed by a processor, the processor executes the high dynamic range image according to any one of the above embodiments. Approach.

本申请实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the present application.

附图说明Description of the drawings

本申请的上述和/或附加的方面和优点从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

图1是本申请实施方式的高动态图像处理***的示意图；Fig. 1 is a schematic diagram of a high dynamic image processing system according to an embodiment of the present application;

图2是本申请实施方式的像素阵列的示意图；Fig. 2 is a schematic diagram of a pixel array according to an embodiment of the present application;

图3是本申请实施方式的感光像素的截面示意图；3 is a schematic cross-sectional view of a photosensitive pixel according to an embodiment of the present application;

图4是本申请实施方式的感光像素的像素电路图；4 is a pixel circuit diagram of a photosensitive pixel according to an embodiment of the present application;

图5至图10是本申请实施方式的像素阵列中最小重复单元的排布示意图；5 to 10 are schematic diagrams of the arrangement of the smallest repeating unit in the pixel array of the embodiment of the present application;

图11和图12是本申请某些实施方式的图像传感器输出的原始图像的示意图；11 and 12 are schematic diagrams of original images output by image sensors according to some embodiments of the present application;

图13是本申请实施方式的高动态图像处理***的示意图；FIG. 13 is a schematic diagram of a high dynamic image processing system according to an embodiment of the present application;

图14是本申请实施方式的像素相加处理的示意图；FIG. 14 is a schematic diagram of pixel addition processing in an embodiment of the present application;

图15是本申请实施方式的像素求平均处理的示意图；15 is a schematic diagram of pixel averaging processing in an embodiment of the present application;

图16和图17是本申请实施方式的高动态图像处理***的示意图；16 and 17 are schematic diagrams of a high dynamic image processing system according to an embodiment of the present application;

图18是本申请实施方式的黑电平矫正处理的示意图；FIG. 18 is a schematic diagram of black level correction processing according to an embodiment of the present application;

图19是本申请实施方式的镜头阴影矫正处理的示意图；FIG. 19 is a schematic diagram of lens shading correction processing according to an embodiment of the present application;

图20和图21是本申请实施方式的坏点补偿处理处理的示意图；FIG. 20 and FIG. 21 are schematic diagrams of dead pixel compensation processing in an embodiment of the present application;

图22至图25是本申请实施方式的去马赛克处理的示意图；Figures 22 to 25 are schematic diagrams of demosaicing processing in an embodiment of the present application;

图26是本申请实施方式的色调映射处理的Vout和Vin之间的映射关系示意图；FIG. 26 is a schematic diagram of the mapping relationship between Vout and Vin in the tone mapping process of the embodiment of the present application;

图27是本申请实施方式的亮度对齐处理的示意图；FIG. 27 is a schematic diagram of brightness alignment processing according to an embodiment of the present application;

图28是本申请实施方式的电子设备的结构示意图；FIG. 28 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

图29是本申请某些实施方式的图像获取方法的流程示意图；FIG. 29 is a schematic flowchart of an image acquisition method according to some embodiments of the present application;

图30是本申请某些实施方式的非易失性计算机可读存储介质与处理器的交互示意图。FIG. 30 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor in some embodiments of the present application.

具体实施方式Detailed ways

下面详细描述本申请的实施方式，实施方式的示例在附图中示出，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本申请，而不能理解为对本申请的限制。The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the drawings, and the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The following embodiments described with reference to the drawings are exemplary, and are only used to explain the present application, and cannot be understood as a limitation to the present application.

下文的公开提供了许多不同的实施方式或例子用来实现本申请的实施方式的不同结构。为了简化本申请的实施方式的公开，下文中对特定例子的部件和设置进行描述。当然，它们仅仅为示例，并且目的不在于限制本申请。The following disclosure provides many different embodiments or examples for realizing the different structures of the embodiments of the present application. In order to simplify the disclosure of the embodiments of the present application, the components and settings of specific examples are described below. Of course, they are only examples, and are not intended to limit the application.

请参阅图1和图2，本申请实施方式的图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素。彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元，每个最小重复单元包含多个子单元。每个子单元包括多个单颜色感光像素及多个全色感光像素。当所述图像传感器10中的像素阵列11曝光时，对于同一子单元中的多个感光像素，至少一个单颜色感光像素以第一曝光时间曝光，至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光，至少一个全色感光像素以小于或等于第一曝光时间的第三曝光时间曝光，至少一个全色感光像素以小于第一曝光时间的第四曝光时间曝光。其中，以第一曝光时间曝光的单颜色感光像素生成第一彩色信息得到第一彩色原始图像。以第二曝光时间曝光的单颜色感光像素生成第二彩色信息得到第二彩色原始图像。以第三曝光时间曝光的全色感光像素生成第一全色信息、以第四曝光时间曝光的全色感光像素生成第二全色信息，以得到全色原始图像。请参阅图1和图2，本申请实施方式的高动态范围图像处理***100包括图像传感器10、彩色高动态融合单元30和图像处理器20。图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素。彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元，每个最小重复单元包含多个子单元。每个子单元包括多个单颜色感光像素及多个全色感光像素。当所述图像传感器10中的像素阵列11曝光时，对于同一子单元中的多个感光像素，至少一个单颜色感光像素以第一曝光时间曝光，至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光，至少一个全色感光像素以小于或等于第一曝光时间的第三曝光时间曝光，至少一个全色感光像素以小于第一曝光时间的第四曝光时间曝光。其中，以第一曝光时间曝光的单颜色感光像素生成第一彩色信息得到第一彩色原始图像。以第二曝光时间曝光的单颜色感光像素生成第二彩色信息得到第二彩色原始图像。以第三曝光时间曝光的全色感光像素生成第一全色信息、以第四曝光时间曝光的全色感光像素生成第二全色信息，以得到全色原始图像。彩色高动态融合单元30和图像处理器20用于对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。Please refer to FIG. 1 and FIG. 2, the image sensor 10 of the embodiment of the present application includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. When the pixel array 11 in the image sensor 10 is exposed, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed for the first exposure time, and at least one single-color photosensitive pixel is exposed for less than the first exposure time At least one full-color photosensitive pixel is exposed at a third exposure time less than or equal to the first exposure time, and at least one full-color photosensitive pixel is exposed at a fourth exposure time less than the first exposure time. Wherein, the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image. The single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image. The panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain a panchromatic original image. 1 and FIG. 2, the high dynamic range image processing system 100 according to the embodiment of the present application includes an image sensor 10, a color high dynamic fusion unit 30 and an image processor 20. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. When the pixel array 11 in the image sensor 10 is exposed, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed for the first exposure time, and at least one single-color photosensitive pixel is exposed for less than the first exposure time At least one full-color photosensitive pixel is exposed at a third exposure time less than or equal to the first exposure time, and at least one full-color photosensitive pixel is exposed at a fourth exposure time less than the first exposure time. Wherein, the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image. The single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image. The panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain a panchromatic original image. The color high dynamic fusion unit 30 and the image processor 20 are used to perform high dynamic range processing, image processing and fusion algorithm processing on the first color original image, the second color original image and the panchromatic original image to obtain the target image.

请参阅图1，在某些实施方式中，高动态范围图像处理***100还包括全色高动态融合单元40。全色原始图像包括第一全色原始图像和第二全色原始图像；以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。彩色高动态融合单元30、全色高动态融合单元40和图像处理器20用于对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。Referring to FIG. 1, in some embodiments, the high dynamic range image processing system 100 further includes a panchromatic high dynamic fusion unit 40. The panchromatic original image includes a first panchromatic original image and a second panchromatic original image; the panchromatic photosensitive pixels exposed at the third exposure time generate the first panchromatic information to obtain the first panchromatic original image, which is exposed at the fourth exposure time The panchromatic photosensitive pixel generates the second panchromatic information to obtain the second panchromatic original image. The color high-dynamic fusion unit 30, the panchromatic high-dynamic fusion unit 40, and the image processor 20 are used to perform high dynamics on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image. Range processing, image processing and fusion algorithm processing get the target image.

请参阅图1，在某些实施方式中，图像处理器20包括彩色处理模块2021、全色处理模块2022和融合模块204。图像处理包括第一图像处理和第二图像处理；彩色高动态融合单元30用于将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像；全色高动态融合单元40用于将第一全色原始图像和第二全色原始图像融合得到高动态全色图像；彩色处理模块2021用于对高动态彩色图像进行第一图像处理，得到彩色中间图像；全色处理模块2022用于对高动态全色图像进行第二图像处理，得到全色中间图像；融合模块204用于对彩色中间图像和全色中间图像进行融合算法处理得到目标图像。Referring to FIG. 1, in some embodiments, the image processor 20 includes a color processing module 2021, a panchromatic processing module 2022, and a fusion module 204. Image processing includes first image processing and second image processing; the color high dynamic fusion unit 30 is used to fuse the first color original image and the second color original image to obtain a high dynamic color image; the full color high dynamic fusion unit 40 is used to combine The first full-color original image and the second full-color original image are fused to obtain a high-dynamic full-color image; the color processing module 2021 is used to perform first image processing on the high-dynamic color image to obtain a color intermediate image; the full-color processing module 2022 is used to The second image processing is performed on the high dynamic panchromatic image to obtain a panchromatic intermediate image; the fusion module 204 is configured to perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.

请参阅图1，在某些实施方式中，图像处理器20包括彩色处理模块2021、全色处理模块2022和融合模块204。图像处理包括第一图像处理和第二图像处理，第一图像处理包括第一图像子处理和第二图像子处理。彩色处理模块2021用于对第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；全色处理模块2022用于对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像。彩色高动态融合单元30用于将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像。全色高动态融合单元40用于将第一全色中间图像和第二全色中间图像融合得到高动态全色图像。彩色处理模块2021还用于对高动态彩色图像进行第二图像子处理，得到彩色中间图像；融合模块204用于对彩色中间图像和高动态全色图像进行融合算法处理得到目标图像。Referring to FIG. 1, in some embodiments, the image processor 20 includes a color processing module 2021, a panchromatic processing module 2022, and a fusion module 204. The image processing includes a first image processing and a second image processing, and the first image processing includes a first image sub-processing and a second image sub-processing. The color processing module 2021 is configured to perform a first image sub-processing on the first color original image to obtain a first color intermediate image, and perform the first image sub-processing on the second color original image to obtain a second color intermediate image; a full-color processing module 2022 is used to perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image. The color high dynamic fusion unit 30 is used for fusing the first color intermediate image and the second color intermediate image to obtain a high dynamic color image. The panchromatic high dynamic fusion unit 40 is used for fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image. The color processing module 2021 is also used to perform second image sub-processing on the high dynamic color image to obtain a color intermediate image; the fusion module 204 is used to perform a fusion algorithm processing on the color intermediate image and the high dynamic panchromatic image to obtain the target image.

请参阅图1，在某些实施方式中，图像处理器20包括彩色处理模块2021、全色处理模块2022和融合模块204，图像处理包括第一图像处理和第二图像处理。彩色处理模块2021用于对第一彩色原始图像进行第一图像处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像处理，得到第二彩色中间图像。全色处理模块2022用于对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像。彩色高动态融合单元30用于将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像；全色高动态融合单元40用于将第一全色中间图像和第二全色中间图像融合得到高动态全色图像；融合模块40用于对高动态彩色图像和高动态全色图像进行融合算法处理得到目标图像。1, in some embodiments, the image processor 20 includes a color processing module 2021, a panchromatic processing module 2022, and a fusion module 204, and the image processing includes first image processing and second image processing. The color processing module 2021 is configured to perform first image processing on the first color original image to obtain a first color intermediate image, and perform first image processing on the second color original image to obtain a second color intermediate image. The panchromatic processing module 2022 is configured to perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image. The color high dynamic fusion unit 30 is used for fusing the first color intermediate image and the second color intermediate image to obtain a high dynamic color image; the panchromatic high dynamic fusion unit 40 is used for combining the first full color intermediate image and the second full color intermediate image The high-dynamic panchromatic image is obtained by fusion; the fusion module 40 is used to perform fusion algorithm processing on the high-dynamic color image and the high-dynamic panchromatic image to obtain the target image.

在某些实施方式中，第一图像处理包括黑电平矫正处理、镜头阴影矫正处理、去马赛克处理、坏点补偿处理、色彩矫正处理、全局色调映射处理、色彩转换处理中的一个或多个。第二图像处理包括黑电平矫正处理、镜头阴影矫正处理、坏点补偿处理、全局色调映射处理中的一个或多个。In some embodiments, the first image processing includes one or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing. . The second image processing includes one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

请参阅图1，在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。彩色高动态融合单元30用于将第一彩色原始图像和第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的第一彩色原始图像，再融合亮度对齐后的第一彩色原始图像及第二彩色原始图像以得到高动态彩色图像。全色高动态融合单元30用于将第一全色原始图像和第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的第一全色原始图像，再融合亮度对齐后的第一全色原始图像及第二全色原始图像以得到高动态全色图像。Referring to FIG. 1, in some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. The color high dynamic fusion unit 30 is used to perform brightness alignment processing on the first color original image and the second color original image to obtain a first color original image after brightness alignment, and then merge the first color original image and the second color original image after brightness alignment. Two-color original image to obtain a highly dynamic color image. The panchromatic high dynamic fusion unit 30 is configured to perform brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the first panchromatic original image after brightness alignment, and then merge the first panchromatic original image after brightness alignment. Color original image and the second panchromatic original image to obtain a high dynamic panchromatic image.

请参阅图1，在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。彩色高动态融合单元30用于将第一彩色中间图像和第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的第一彩色中间图像，再融合亮度对齐后的第一彩色中间图像及第二彩色中间图像以得到高动态彩色图像。全色高动态融合单元40用于将第一全色中间图像和第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的第一全色中间图像，再融合亮度对齐后的第一全色中间图像及第二全色中间图像以得到高动态全色图像。Referring to FIG. 1, in some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. The color high dynamic fusion unit 30 is used to perform brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain a first color intermediate image after brightness alignment, and then merge the first color intermediate image and the second color intermediate image after brightness alignment. Two color intermediate images to obtain high dynamic color images. The panchromatic high dynamic fusion unit 40 is used to perform brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image after brightness alignment, and then merge the first panchromatic intermediate image after brightness alignment. The color intermediate image and the second panchromatic intermediate image are used to obtain a high dynamic panchromatic image.

请参阅图13，在某些实施方式中，第三曝光时间等于第四曝光时间，第三曝光时间大于第二曝光时间，并且小于第一曝光时间。高动态范围图像处理***100还包括全色信息融合单元50全色信息融合单元50用于将每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息和以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像；或者全色原始图像包括第一全色原始图像和第二全色原始图像，以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像，全色信息融合单元50用于将第一全色原始图像的每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息与第二全色原始图像的对应子单元中以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像。Referring to FIG. 13, in some embodiments, the third exposure time is equal to the fourth exposure time, and the third exposure time is greater than the second exposure time and less than the first exposure time. The high dynamic range image processing system 100 also includes a panchromatic information fusion unit 50. The panchromatic information fusion unit 50 is used to combine the first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time in each subunit and the fourth The second panchromatic information generated by the panchromatic photosensitive pixels exposed during the exposure time is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image; or the panchromatic original image includes the first panchromatic original image and the second panchromatic The original image, the panchromatic photosensitive pixels exposed at the third exposure time generate the first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate the second panchromatic information to obtain the second panchromatic Original image, the panchromatic information fusion unit 50 is used to correspond the first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time in each subunit of the first panchromatic original image and the second panchromatic original image The second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time in the subunit is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image.

请参阅图1，在某些实施方式中，图像处理器20还包括：接收单元201和内存单元203。接收单元201用于接收第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像和高动态全色图像中的一个或多个。内存单元203用于暂存第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像、高动态全色图像和全色中间图像中的一个或多个。Referring to FIG. 1, in some embodiments, the image processor 20 further includes: a receiving unit 201 and a memory unit 203. The receiving unit 201 is configured to receive one of a first color original image, a second color original image, a full color original image, a first full color original image, a second full color original image, a high dynamic color image, and a high dynamic full color image Or more. The memory unit 203 is used to temporarily store the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, the high dynamic color image, the high dynamic full color image and the full color image. One or more of the color intermediate images.

在某些实施方式中，彩色高动态融合单元30和全色高动态融合单元40集成在图像传感器10中；或彩色高动态融合单元30和全色高动态融合单元40集成在图像处理器20中。In some embodiments, the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image sensor 10; or the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image processor 20 .

请参阅图21，本申请还提供一种高动态范围图像处理方法。本申请实施方式的高动态范围图像处理方法用于高动态范围图像处理***100。高动态范围图像处理***100包括图像传感器10。图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素。彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元，每个最小重复单元包含多个子单元。每个子单元包括多个单颜色感光像素及多个全色感光像素。高动态范围图像处理方法包括：Please refer to FIG. 21. This application also provides a high dynamic range image processing method. The high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100. The high dynamic range image processing system 100 includes an image sensor 10. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. High dynamic range image processing methods include:

01：控制像素阵列11曝光。其中，对于同一子单元中的多个感光像素，至少一个单颜色感光像素以第一曝光时间曝光，至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光，至少一个全色感光像素以小于或等于第一曝光时间的第三曝光时间曝光，至少一个全色感光像素以小于第一曝光时间的第四曝光时间曝光。其中，以第一曝光时间曝光的单颜色感光像素生成第一彩色信息得到第一彩色原始图像，以第二曝光时间曝光的单颜色感光像素生成第二彩色信息得到第二彩色原始图像，以第三曝光时间曝光的全色感光像素生成第一全色信息、以第四曝光时间曝光的全色感光像素生成第二全色信息，以得到全色原始图像；和01: Control the exposure of the pixel array 11. Among them, for multiple photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, at least one single-color photosensitive pixel is exposed with a second exposure time less than the first exposure time, and at least one full-color photosensitive pixel is exposed The pixels are exposed at a third exposure time less than or equal to the first exposure time, and at least one full-color photosensitive pixel is exposed at a fourth exposure time less than the first exposure time. The single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image, and the single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color original image. Panchromatic photosensitive pixels exposed for three exposure times generate first panchromatic information, and panchromatic photosensitive pixels exposed for a fourth exposure time generate second panchromatic information to obtain a panchromatic original image; and

02：对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。02: Perform high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the full-color original image to obtain the target image.

本申请实施方式的高动态范围图像处理方法通过控制像素阵列11中每个子单元内的多个感光像素以不同的曝光时间曝光，并且根据不同曝光时间的感光像素将生成多张图像，以便后续处理单元对此多张图像进行高动态范围处理，从而得到具有高动态范围的目标图像，使得在无需提高图像传感器10的感光像素硬件参数的情况下，就能实现高动态范围功能，使得目标图像的亮处、暗位都能够具有更佳的表现，有利于提高成像性能的同时，有助于降低成本。The high dynamic range image processing method of the embodiment of the present application controls multiple photosensitive pixels in each subunit of the pixel array 11 to be exposed with different exposure times, and generates multiple images according to the photosensitive pixels with different exposure times for subsequent processing The unit performs high dynamic range processing on the multiple images to obtain a target image with high dynamic range, so that the high dynamic range function can be realized without increasing the hardware parameters of the photosensitive pixel of the image sensor 10, so that the target image is Both bright and dark positions can have better performance, which is conducive to improving imaging performance and helping to reduce costs.

在某些实施方式中，全色原始图像包括第一全色原始图像和第二全色原始图像。其中，步骤02：以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。In some embodiments, the full-color original image includes a first full-color original image and a second full-color original image. Wherein, step 02: the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain the second panchromatic image. Full-color original image. Perform high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the panchromatic original image to obtain the target image including the first color original image, the second color original image, and the first panchromatic The original image and the second panchromatic original image undergo high dynamic range processing, image processing and fusion algorithm processing to obtain the target image.

在某些实施方式中，图像处理包括第一图像处理和第二图像处理。步骤对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像；将第一全色原始图像和第二全色原始图像融合得到高动态全色图像；对高动态彩色图像进行第一图像处理，得到彩色中间图像；对高动态全色图像进行第二图像处理，得到全色中间图像；和对彩色中间图像和全色中间图像进行融合算法处理得到目标图像。In some embodiments, the image processing includes a first image processing and a second image processing. The steps of performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image include: the first color original image Fuse with the second color original image to obtain a high dynamic color image; fuse the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image; perform the first image processing on the high dynamic color image to obtain a color intermediate image ; Perform the second image processing on the high dynamic panchromatic image to obtain the panchromatic intermediate image; and perform the fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.

在某些实施方式中，图像处理包括第一图像处理和第二图像处理。第一图像处理包括第一图像子处理和第二图像子处理。步骤对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：对第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像；将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像；将第一全色中间图像和第二全色中间图像融合得到高动态全色图像；对高动态彩色图像进行第二图像子处理，得到彩色中间图像；和对彩色中间图像和高动态全色图像进行融合算法处理得到目标图像。In some embodiments, the image processing includes a first image processing and a second image processing. The first image processing includes a first image sub-processing and a second image sub-processing. The steps of performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image include: the first color original image Perform the first image sub-processing to obtain the first color intermediate image, perform the first image sub-processing on the second color original image to obtain the second color intermediate image; perform the second image processing on the first full-color original image to obtain the first Panchromatic intermediate image, perform second image processing on the second panchromatic original image to obtain the second panchromatic intermediate image; fuse the first color intermediate image and the second color intermediate image to obtain a high dynamic color image; combine the first panchromatic image The intermediate image and the second panchromatic intermediate image are fused to obtain a high dynamic panchromatic image; the high dynamic color image is subjected to the second image sub-processing to obtain the color intermediate image; and the color intermediate image and the high dynamic panchromatic image are processed by the fusion algorithm Target image.

在某些实施方式中，图像处理包括第一图像处理和第二图像处理。步骤对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：对第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像；将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像；将第一全色中间图像和第二全色中间图像融合得到高动态全色图像；和对高动态彩色图像和高动态全色图像进行融合算法处理得到目标图像。In some embodiments, the image processing includes a first image processing and a second image processing. The steps of performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image include: the first color original image Perform the first image sub-processing to obtain the first color intermediate image, perform the first image sub-processing on the second color original image to obtain the second color intermediate image; perform the second image processing on the first full-color original image to obtain the first Panchromatic intermediate image, perform second image processing on the second panchromatic original image to obtain the second panchromatic intermediate image; fuse the first color intermediate image and the second color intermediate image to obtain a high dynamic color image; combine the first panchromatic image The intermediate image and the second panchromatic intermediate image are fused to obtain a high dynamic panchromatic image; and the high dynamic color image and the high dynamic panchromatic image are processed by a fusion algorithm to obtain the target image.

在某些实施方式中，第一图像处理包括：黑电平矫正处理、镜头阴影矫正处理、去马赛克处理、坏点补偿处理、色彩矫正处理、全局色调映射处理、色彩转换处理中的一个或多个。第二图像处理包括：黑电平矫正处理、镜头阴影矫正处理、坏点补偿处理、全局色调映射处理中的一个或多个。In some embodiments, the first image processing includes one or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing. indivual. The second image processing includes one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。步骤将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像包括：将第一彩色原始图像和第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的第一彩色原始图像，再融合亮度对齐后的第一彩色原始图像及第二彩色原始图像以得到高动态彩色图像。步骤将第一全色原始图像和第二全色原始图像融合得到高动态全色图像包括：将第一全色原始图像和第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的第一全色原始图像，再融合亮度对齐后的第一全色原始图像及第二全色原始图像以得到高动态全色图像。In some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. The step of fusing the first color original image and the second color original image to obtain a high dynamic color image includes: performing brightness alignment processing on the first color original image and the second color original image to obtain the first color original image after brightness alignment, Then, the first color original image and the second color original image with the brightness aligned are merged to obtain a high dynamic color image. The step of fusing the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image includes: performing brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the brightness-aligned second A full-color original image, and then merge the first full-color original image and the second full-color original image after brightness alignment to obtain a high-dynamic full-color image.

在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。步骤将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像包括：将第一彩色中间图像和第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的第一彩色中间图像，再融合亮度对齐后的第一彩色中间图像及第二彩色中间图像以得到高动态彩色图像。步骤将第一全色中间图像和第二全色中间图像融合得到高动态全色图像包括：将第一全色中间图像和第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的第一全色中间图像，再融合亮度对齐后的第一全色中间图像及第二全色中间图像以得到高动态全色图像。In some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. The step of fusing the first color intermediate image and the second color intermediate image to obtain a high dynamic color image includes: performing brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain a first color intermediate image with aligned brightness; Then, the first color intermediate image and the second color intermediate image with the aligned brightness are merged to obtain a high dynamic color image. The step of fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image includes: performing brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image after brightness alignment. A panchromatic intermediate image is combined with the first panchromatic intermediate image and the second panchromatic intermediate image after brightness alignment to obtain a high dynamic panchromatic image.

在某些实施方式中，第三曝光时间等于第四曝光时间，第三曝光时间大于第二曝光时间，并且小于第一曝光时间。高动态范围图像处理方法还包括：将每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息和以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像；或者全色原始图像包括第一全色原始图像和第二全色原始图像。图像传感器10以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。高动态范围图像处理方法还包括：将第一全色原始图像的每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息与第二全色原始图像的对应子单元中以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像。In some embodiments, the third exposure time is equal to the fourth exposure time, and the third exposure time is greater than the second exposure time and less than the first exposure time. The high dynamic range image processing method further includes: first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time in each subunit and second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time The information undergoes pixel addition processing or pixel averaging processing to obtain a full-color original image; or the full-color original image includes a first full-color original image and a second full-color original image. The image sensor 10 generates first panchromatic information to obtain the first panchromatic original image by the panchromatic photosensitive pixels exposed at the third exposure time, and generates the second panchromatic information to obtain the second panchromatic image from the panchromatic photosensitive pixels exposed at the fourth exposure time. The original image. The high dynamic range image processing method further includes: first panchromatic information generated by the panchromatic photosensitive pixels exposed at a third exposure time in each subunit of the first panchromatic original image and corresponding subunits of the second panchromatic original image The second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image.

在某些实施方式中，高动态范围图像处理方法还包括：接收第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像和高动态全色图像中的一个或多个；和暂存第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像、高动态全色图像、彩色中间图像和全色中间图像中的一个或多个。In some embodiments, the high dynamic range image processing method further includes: receiving a first color original image, a second color original image, a full color original image, a first full color original image, a second full color original image, and a high dynamic range image. One or more of a color image and a high dynamic full color image; and temporarily store the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, and the high One or more of dynamic color image, high dynamic full color image, color intermediate image, and full color intermediate image.

请参阅图28，本申请还提供一种电子设备1000。本申请实施方式的电子设备1000包括镜头300、壳体200及上述任意一项实施方式的高动态范围图像处理***100。镜头300、高动态范围图像处理***100与壳体200结合。镜头300与高动态范围图像处理***100的图像传感器10配合成像。Please refer to FIG. 28. The present application also provides an electronic device 1000. The electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 of any one of the above embodiments. The lens 300 and the high dynamic range image processing system 100 are combined with the housing 200. The lens 300 cooperates with the image sensor 10 of the high dynamic range image processing system 100 for imaging.

请参阅29，本申请还提供一种包含计算机程序的非易失性计算机可读存储介质400。该计算机程序被处理器60执行时，使得处理器60执行上述任意一项实施方式所述的高动态范围图像处理方法。Please refer to 29. This application also provides a non-volatile computer-readable storage medium 400 containing a computer program. When the computer program is executed by the processor 60, the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.

本申请实施方式的图像传感器10和高动态范围图像处理***100通过控制像素阵列11中每个子单元内的多个感光像素以不同的曝光时间曝光，并且根据不同曝光时间的感光像素将生成多张图像，以便后续处理单元对此多张图像进行高动态范围处理，从而得到具有高动态范围的目标图像，使得在无需提高图像传感器10的感光像素硬件参数的情况下，就能实现高动态范围功能，使得目标图像的亮处、暗位都能够具有更佳的表现，有利于提高成像性能的同时，有助于降低成本。The image sensor 10 and the high dynamic range image processing system 100 of the embodiment of the present application control the multiple photosensitive pixels in each subunit of the pixel array 11 to be exposed at different exposure times, and generate multiple photos according to the photosensitive pixels with different exposure times. Image, so that the subsequent processing unit performs high dynamic range processing on the multiple images, thereby obtaining a target image with high dynamic range, so that the high dynamic range function can be realized without increasing the hardware parameters of the photosensitive pixels of the image sensor 10 , So that the bright and dark parts of the target image can have better performance, which is beneficial to improve imaging performance and reduce costs.

图2是本申请实施方式中的图像传感器10的示意图。图像传感器10包括像素阵列11、垂直驱动单元12、控制单元13、列处理单元14和水平驱动单元15。FIG. 2 is a schematic diagram of the image sensor 10 in the embodiment of the present application. The image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.

例如，图像传感器10可以采用互补金属氧化物半导体(CMOS，Complementary Metal Oxide Semiconductor)感光元件或者电荷耦合元件(CCD，Charge-coupled Device)感光元件。For example, the image sensor 10 may adopt a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.

例如，像素阵列11包括以阵列形式二维排列(即二维矩阵形式排布)的多个感光像素110(图3所示)，每个感光像素110包括光电转换元件1111(图4所示)。每个感光像素110根据入射在其上的光的强度将光转换为电荷。For example, the pixel array 11 includes a plurality of photosensitive pixels 110 (shown in FIG. 3) arranged two-dimensionally in an array (ie, arranged in a two-dimensional matrix), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 (shown in FIG. 4) . Each photosensitive pixel 110 converts light into electric charge according to the intensity of light incident thereon.

例如，垂直驱动单元12包括移位寄存器和地址译码器。垂直驱动单元12包括读出扫描和复位扫描功能。读出扫描是指顺序地逐行扫描单位感光像素110，从这些单位感光像素110逐行地读取信号。例如，被选择并被扫描的感光像素行中的每一感光像素110输出的信号被传输到列处理单元14。复位扫描用于复位电荷，光电转换元件的光电荷被丢弃，从而可以开始新的光电荷的积累。For example, the vertical driving unit 12 includes a shift register and an address decoder. The vertical drive unit 12 includes readout scanning and reset scanning functions. The readout scan refers to sequentially scanning the unit photosensitive pixels 110 line by line, and reading signals from these unit photosensitive pixels 110 line by line. For example, the signal output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row is transmitted to the column processing unit 14. The reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element is discarded, so that the accumulation of new photocharge can be started.

例如，由列处理单元14执行的信号处理是相关双采样(CDS)处理。在CDS处理中，取出从所选感光像素行中的每一感光像素110输出的复位电平和信号电平，并且计算电平差。因而，获得了一行中的感光像素110的信号。列处理单元14可以具有用于将模拟像素信号转换为数字格式的模数(A/D)转换功能。For example, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS process, the reset level and the signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated. Thus, the signals of the photosensitive pixels 110 in a row are obtained. The column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into a digital format.

例如，水平驱动单元15包括移位寄存器和地址译码器。水平驱动单元15顺序逐列扫描像素阵列11。通过水平驱动单元15执行的选择扫描操作，每一感光像素列被列处理单元14顺序地处理，并且被顺序输出。For example, the horizontal driving unit 15 includes a shift register and an address decoder. The horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14, and is sequentially output.

例如，控制单元13根据操作模式配置时序信号，利用多种时序信号来控制垂直驱动单元12、列处理单元14和水平驱动单元15协同工作。For example, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14 and the horizontal driving unit 15 to work together.

图3是本申请实施方式中一种感光像素110的示意图。感光像素110包括像素电路111、滤光片112、及微透镜113。沿感光像素110的收光方向，微透镜113、滤光片112、及像素电路111依次设置。微透镜113用于汇聚光线，滤光片112用于供某一波段的光线通过并过滤掉其余波段的光线。像素电路111用于将接收到的光线转换为电信号，并将生成的电信号提供给图2所示的列处理单元14。FIG. 3 is a schematic diagram of a photosensitive pixel 110 in an embodiment of the present application. The photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a micro lens 113. Along the light-receiving direction of the photosensitive pixel 110, the microlens 113, the filter 112, and the pixel circuit 111 are arranged in sequence. The microlens 113 is used for condensing light, and the filter 112 is used for passing light of a certain waveband and filtering out the light of other wavebands. The pixel circuit 111 is used to convert the received light into electrical signals, and provide the generated electrical signals to the column processing unit 14 shown in FIG. 2.

图4是本申请实施方式中一种感光像素110的像素电路111的示意图。图4中像素电路111可应用在图2所示的像素阵列11内的每个感光像素110(图3所示)中。下面结合图2至图4对像素电路111的工作原理进行说明。FIG. 4 is a schematic diagram of a pixel circuit 111 of a photosensitive pixel 110 in an embodiment of the present application. The pixel circuit 111 in FIG. 4 can be applied to each photosensitive pixel 110 (shown in FIG. 3) in the pixel array 11 shown in FIG. The working principle of the pixel circuit 111 will be described below with reference to FIGS. 2 to 4.

如图4所示，像素电路111包括光电转换元件1111(例如，光电二极管)、曝光控制电路(例如，转移晶体管1112)、复位电路(例如，复位晶体管1113)、放大电路(例如，放大晶体管1114)和选择电路(例如，选择晶体管1115)。在本申请的实施例中，转移晶体管1112、复位晶体管1113、放大晶体管1114和选择晶体管1115例如是MOS管，但不限于此。As shown in FIG. 4, the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), and an amplification circuit (for example, an amplification transistor 1114). ) And a selection circuit (for example, a selection transistor 1115). In the embodiment of the present application, the transfer transistor 1112, the reset transistor 1113, the amplifying transistor 1114, and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.

例如，光电转换元件1111包括光电二极管，光电二极管的阳极例如连接到地。光电二极管将所接收的光转换为电荷。光电二极管的阴极经由曝光控制电路(例如，转移晶体管1112)连接到浮动扩散单元FD。浮动扩散单元FD与放大晶体管1114的栅极、复位晶体管1113的源极连接。For example, the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to the ground, for example. The photodiode converts the received light into electric charge. The cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (for example, a transfer transistor 1112). The floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.

例如，曝光控制电路为转移晶体管1112，曝光控制电路的控制端TG为转移晶体管1112的栅极。当有效电平(例如，VPIX电平)的脉冲通过曝光控制线(例如图18所示的TX)传输到转移晶体管1112的栅极时，转移晶体管1112导通。转移晶体管1112将光电二极管光电转换的电荷传输到浮动扩散单元FD。For example, the exposure control circuit is a transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112. When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line (for example, TX shown in FIG. 18), the transfer transistor 1112 is turned on. The transfer transistor 1112 transfers the charge photoelectrically converted by the photodiode to the floating diffusion unit FD.

例如，复位晶体管1113的漏极连接到像素电源VPIX。复位晶体管113的源极连接到浮动扩散单元FD。在电荷被从光电二极管转移到浮动扩散单元FD之前，有效复位电平的脉冲经由复位线(例如图18所示的RX)传输到复位晶体管113的栅极，复位晶体管113导通。复位晶体管113将浮动扩散单元FD复位到像素电源VPIX。For example, the drain of the reset transistor 1113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charge is transferred from the photodiode to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via a reset line (for example, RX shown in FIG. 18), and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

例如，放大晶体管1114的栅极连接到浮动扩散单元FD。放大晶体管1114的漏极连接到像素电源VPIX。在浮动扩散单元FD被复位晶体管1113复位之后，放大晶体管1114经由选择晶体管1115通过输出端OUT输出复位电平。在光电二极管的电荷被转移晶体管1112转移之后，放大晶体管1114经由选择晶体管1115通过输出端OUT输出信号电平。For example, the gate of the amplifying transistor 1114 is connected to the floating diffusion unit FD. The drain of the amplifying transistor 1114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 1113, the amplifying transistor 1114 outputs the reset level through the output terminal OUT via the selection transistor 1115. After the charge of the photodiode is transferred by the transfer transistor 1112, the amplifying transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.

例如，选择晶体管1115的漏极连接到放大晶体管1114的源极。选择晶体管1115的源极通过输出端OUT连接到图2中的列处理单元14。当有效电平的脉冲通过选择线被传输到选择晶体管1115的栅极时，选择晶体管1115导通。放大晶体管1114输出的信号通过选择晶体管1115传输到列处理单元14。For example, the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114. The source of the selection transistor 1115 is connected to the column processing unit 14 in FIG. 2 through the output terminal OUT. When the pulse of the active level is transmitted to the gate of the selection transistor 1115 through the selection line, the selection transistor 1115 is turned on. The signal output by the amplifying transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.

需要说明的是，本申请实施例中像素电路111的像素结构并不限于图4所示的结构。例如，像素电路111也可以具有三晶体管像素结构，其中放大晶体管1114和选择晶体管1115的功能由一个晶体管完成。例如，曝光控制电路也不局限于单个转移晶体管1112的方式，其它具有控制端控制导通功能的电子器件或结构均可以作为本申请实施例中的曝光控制电路，本申请实施方式中的单个转移晶体管1112的实施方式简单、成本低、易于控制。It should be noted that the pixel structure of the pixel circuit 111 in the embodiment of the present application is not limited to the structure shown in FIG. 4. For example, the pixel circuit 111 may also have a three-transistor pixel structure, in which the functions of the amplifying transistor 1114 and the selecting transistor 1115 are performed by one transistor. For example, the exposure control circuit is not limited to the way of a single transfer transistor 1112, and other electronic devices or structures with the function of controlling the conduction of the control terminal can be used as the exposure control circuit in the embodiment of the present application. The implementation of the transistor 1112 is simple, low in cost, and easy to control.

图5至图10是本申请某些实施方式的像素阵列11(图2所示)中的感光像素110(图3所示)的排布示意图。感光像素110包括两类，一类为全色感光像素W，另一类为彩色感光像素。图5至图10仅示出了一个最小重复单元中的多个感光像素110的排布。对图5至图10所示的最小重复单元在行和列上多次复制，即可形成像素阵列11。每个最小重复单元均由多个全色感光像素W和多个彩色感光像素组成。每个最小重复单元包括多个子单元。每个子单元内包括多个单颜色感光像素和多个全色感光像素W。其中，图5至图8所示的最小重复单元中，每个子单元中的全色感光像素W和彩色感光像素交替设置。图9和图10所示的最小重复单元中，每个子单元中，同一行的多个感光像素110为同一类别的感光像素110；或者，同一列的多个感光像素110为同一类别的感光像素110。5 to 10 are schematic diagrams of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the pixel array 11 (shown in FIG. 2) according to some embodiments of the present application. The photosensitive pixels 110 include two types, one is a full-color photosensitive pixel W, and the other is a color photosensitive pixel. 5 to 10 only show the arrangement of a plurality of photosensitive pixels 110 in a minimum repeating unit. The smallest repeating unit shown in FIGS. 5 to 10 is copied multiple times in rows and columns to form the pixel array 11. Each minimum repeating unit is composed of multiple full-color photosensitive pixels W and multiple color photosensitive pixels. Each minimum repeating unit includes multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels W. Among them, in the smallest repeating unit shown in FIGS. 5 to 8, the full-color photosensitive pixel W and the color photosensitive pixel in each sub-unit are alternately arranged. In the smallest repeating unit shown in FIGS. 9 and 10, in each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category; or, multiple photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category 110.

具体地，例如，图5为本申请一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为4行4列16个感光像素110，子单元为2行2列4个感光像素110。排布方式为：Specifically, for example, FIG. 5 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit of an embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W表示全色感光像素；A表示多个彩色感光像素中的第一颜色感光像素；B表示多个彩色感光像素中的第二颜色感光像素；C表示多个彩色感光像素中的第三颜色感光像素。W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

例如，如图5所示，对于每个子单元，全色感光像素W和单颜色感光像素交替设置。For example, as shown in FIG. 5, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

例如，如图5所示，子单元的类别包括三类。其中，第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A；第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B；第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元，分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中，一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1(例如图5中左上角和右下角连接的方向)，两个第二类子单元UB设置在第二对角线方向D2(例如图5中右上角和左下角连接的方向)。第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。For example, as shown in Figure 5, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in the first diagonal direction D1 (for example, the direction connecting the upper left corner and the lower right corner in FIG. 5), and two second type subunits UB are arranged In the second diagonal direction D2 (for example, the direction where the upper right corner and the lower left corner are connected in FIG. 5). The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

需要说明的是，在其他实施方式中，第一对角线方向D1也可以是右上角和左下角连接的方向，第二对角线方向D2也可以是左上角和右下角连接的方向。另外，这里的“方向”并非单一指向，可以理解为指示排布的“直线”的概念，可以有直线两端的双向指向。下文图6至图10中对第一对角线方向D1及第二对角线方向D2的解释与此处相同。It should be noted that in other embodiments, the first diagonal direction D1 may also be a direction connecting the upper right corner and the lower left corner, and the second diagonal direction D2 may also be a direction connecting the upper left corner and the lower right corner. In addition, the "direction" here is not a single direction, but can be understood as the concept of a "straight line" indicating the arrangement, and there may be two-way directions at both ends of the straight line. The explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIGS. 6 to 10 is the same as here.

再例如，图6为本申请另一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为6行6列36个感光像素110，子单元为3行3列9个感光像素110。排布方式为：For another example, FIG. 6 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in a minimum repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-units are 9 photosensitive pixels 110 in 3 rows and 3 columns. The arrangement method is:

例如，如图6所示，对于每个子单元，全色感光像素W和单颜色感光像素交替设置。For example, as shown in FIG. 6, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

例如，如图6所示，子单元的类别包括三类。其中，第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A；第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B；第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元，分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中，一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1，两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。For example, as shown in Figure 6, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1, and two second type subunits UB are arranged in a second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

再例如，图7为本申请又一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为8行8列64个感光像素110，子单元为4行4列16个感光像素110。排布方式为：For another example, FIG. 7 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the minimum repeating unit is 8 rows and 8 columns and 64 photosensitive pixels 110, and the sub-units are 4 rows and 4 columns and 16 photosensitive pixels 110. The arrangement method is:

例如，如图7所示，对于每个子单元，全色感光像素W和单颜色感光像素交替设置。For example, as shown in FIG. 7, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

例如，如图7所示，子单元的类别包括三类。其中，第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A；第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B；第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元，分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中，一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1，两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。For example, as shown in Figure 7, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1, and two second type subunits UB are arranged in a second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

具体地，例如，图8为本申请再一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为4行4列16个感光像素110，子单元为2行2列4个感光像素110。排布方式为：Specifically, for example, FIG. 8 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

图8所示的最小重复单元中感光像素110的排布与图5所示的最小重复单元中感光像素110的排布大致相同，其不同之处在于，图8中位于左下角的第二类子单元UB中的全色感光像素W与单颜色感光像素的交替顺序与图5中位于左下角的第二类子单元UB中的全色感光像素W与单颜色感光像素的交替顺序不一致，并且，图8中的第三类子单元UC中的全色感光像素W与单颜色感光像素的交替顺序与图5中位于右下角的第三类子单元UC中的全色感光像素W与单颜色感光像素的交替顺序也不一致。具体地，图5中位于左下角的第二类子单元UB中，第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第二颜色感光像素B)，第二行的感光像素110的交替顺序为单颜色感光像素(即第二颜色感光像素B)、全色感光像素W；而图8中位于左下角的第二类子单元UB中，第一行的感光像素110的交替顺序为单颜色感光像素(即第二颜色感光像素B)、全色感光像素W，第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第二颜色感光像素B)。图5中位于右下角的第三类子单元UC中，第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C)，第二行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W；而图8中位于右下角的第三类子单元UC中，第一行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W，第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C)。The arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. 8 is roughly the same as the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. The alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the subunit UB is inconsistent with the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the second type of subunit UB in the lower left corner of FIG. 5, and , The alternating sequence of the full-color photosensitive pixel W and the single-color photosensitive pixel in the third type subunit UC in FIG. 8 is the same as the full-color photosensitive pixel W and the single-color photosensitive pixel W in the third type subunit UC in the lower right corner of FIG. 5 The alternating sequence of photosensitive pixels is also inconsistent. Specifically, in the second type subunit UB in the lower left corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (ie, second-color photosensitive pixels B), and The alternating sequence of the two rows of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B) and full-color photosensitive pixels W; and in the second-type subunit UB in the lower left corner of FIG. 8, the first row The alternating sequence of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (ie The second color photosensitive pixel B). In the third type of subunit UC in the lower right corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels C), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, a third-color photosensitive pixel C) and a full-color photosensitive pixel W; and in the third type subunit UC in the lower right corner of FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence of the single-color photosensitive pixel (ie the third color photosensitive pixel C), the full-color photosensitive pixel W, the alternating sequence of the photosensitive pixel 110 in the second row is the full-color photosensitive pixel W, the single-color photosensitive pixel (ie the third color Photosensitive pixel C).

如图8所示，图8中的第一类子单元UA中的全色感光像素W与单颜色感光像素的交替顺序与第三类子单元UC中的全色感光像W素与单颜色感光像素的交替顺序不一致。具体地，图8所示的第一类子单元CA中，第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第一颜色感光像素A)，第二行的感光像素110的交替顺序为单颜色感光像素(即第一颜色感光像素A)、全色感光像素W；而图8所示的第三类子单元CC中，第一行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W，第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C)。也即是说，同一最小重复单元中，不同子单元内的全色感光像素W与彩色感光像素的交替顺序可以是一致的(如图5所示)，也可以是不一致的(如图8所示)。As shown in FIG. 8, the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the first type subunit UA in FIG. The alternating sequence of pixels is not consistent. Specifically, in the first type of sub-unit CA shown in FIG. 8, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, first-color photosensitive pixels A), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, the first color photosensitive pixel A), a full-color photosensitive pixel W; and in the third type of subunit CC shown in FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence is single-color photosensitive pixels (that is, third-color photosensitive pixels C), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels). Pixel C). That is to say, in the same minimum repeating unit, the alternating sequence of full-color photosensitive pixels W and color photosensitive pixels in different subunits can be the same (as shown in Figure 5) or inconsistent (as shown in Figure 8). Show).

再例如，图9为本申请还一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为4行4列16个感光像素110，子单元为2行2列4个感光像素110。排布方式为：For another example, FIG. 9 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

例如，如图9所示，对于每个子单元，同一行的多个感光像素110为同一类别的感光像素110。其中，同一类别的感光像素110包括：(1)均为全色感光像素W；(2)均为第一颜色感光像素A；(3)均为第二颜色感光像素B；(4)均为第三颜色感光像素C。For example, as shown in FIG. 9, for each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

例如，如图9所示，子单元的类别包括三类。其中，第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A；第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B；第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元，分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中，一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1，两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。For example, as shown in Figure 9, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1, and two second type subunits UB are arranged in a second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

再例如，图10为本申请还一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中，最小重复单元为4行4列16个感光像素110，子单元为2行2列4个感光像素110。排布方式为：For another example, FIG. 10 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

例如，如图10所示，对于每个子单元，同一列的多个感光像素110为同一类别的感光像素110。其中，同一类别的感光像素110包括：(1)均为全色感光像素W；(2)均为第一颜色感光像素A；(3)均为第二颜色感光像素B；(4)均为第三颜色感光像素C。For example, as shown in FIG. 10, for each sub-unit, a plurality of photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

例如，如图10所示，子单元的类别包括三类。其中，第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A；第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B；第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元，分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中，一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1，两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如，第一对角线和第二对角线垂直。For example, as shown in Figure 10, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1, and two second type subunits UB are arranged in a second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

例如，在其他实施方式中，同一最小重复单元中，也可以是部分子单元内的同一行的多个感光像素110为同一类别的感光像素110，其余部分子单元内的同一列的多个感光像素110为同一类别的感光像素110。For example, in other embodiments, in the same minimum repeating unit, multiple photosensitive pixels 110 in the same row in some sub-units may be photosensitive pixels 110 of the same category, and multiple photosensitive pixels 110 in the same column in the remaining sub-units The pixels 110 are photosensitive pixels 110 of the same type.

例如，如图5至图10所示的最小重复单元中，第一颜色感光像素A可以为红色感光像素R；第二颜色感光像素B可以为绿色感光像素G；第三颜色感光像素C可以为蓝色感光像素Bu。For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a green photosensitive pixel G; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

例如，如图5至图10所示的最小重复单元中，第一颜色感光像素A可以为红色感光像素R；第二颜色感光像素B可以为黄色感光像素Y；第三颜色感光像素C可以为蓝色感光像素Bu。For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a yellow photosensitive pixel Y; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

例如，如图5至图10所示的最小重复单元中，第一颜色感光像素A可以为品红色感光像素M；第二颜色感光像素B可以为青色感光像素Cy；第三颜色感光像素C可以为黄色感光像素Y。For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a magenta photosensitive pixel M; the second color photosensitive pixel B may be a cyan photosensitive pixel Cy; and the third color photosensitive pixel C may It is the yellow photosensitive pixel Y.

需要说明的是，在一些实施例中，全色感光像素W的响应波段可为可见光波段(例如，400nm-760nm)。例如，全色感光像素W上设置有红外滤光片，以实现红外光的滤除。在另一些实施例中，全色感光像素W的响应波段为可见光波段和近红外波段(例如，400nm-1000nm)，与图像传感器10(图1所示)中的光电转换元件1111(图4所示)的响应波段相匹配。例如，全色感光像素W可以不设置滤光片或者设置可供所有波段的光线通过的滤光片，全色感光像素W的响应波段由光电转换元件1111的响应波段确定，即两者相匹配。本申请的实施例包括但不局限于上述波段范围。It should be noted that, in some embodiments, the response band of the full-color photosensitive pixel W may be the visible light band (for example, 400 nm-760 nm). For example, the full-color photosensitive pixel W is provided with an infrared filter to filter out infrared light. In other embodiments, the response wavelength bands of the full-color photosensitive pixel W are visible light and near-infrared wavelengths (for example, 400nm-1000nm), and the photoelectric conversion element 1111 (shown in FIG. 4) in the image sensor 10 (shown in FIG. 1) (Shown) to match the response band. For example, the full-color photosensitive pixel W may not be provided with a filter or a filter that can pass light of all wavelength bands. The response band of the full-color photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two match. . The embodiments of the present application include, but are not limited to, the above-mentioned waveband range.

请结合图1至图3、图5及图11，在某些实施方式中，控制单元13用以控制像素阵列11曝光。其中，对于同一子单元中的多个感光像素110，至少一个单颜色感光像素以第一曝光时间曝光，至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光，至少一个全色感光像素W以小于或等于第一曝光时间的第三曝光时间曝光，至少一个全色感光像素W以小于第一曝光时间的第四曝光时间曝光。像素阵列11中以第一曝光时间曝光的多个单颜色感光像素可以生成第一彩色信息，以第二曝光时间曝光的多个单颜色感光像素可以生成第二彩色信息，以第三曝光时间曝光的多个全色感光像素W可以生成第一全色信息，以第四曝光时间曝光的多个全色感光像素W可以生成第二全色信息。第一彩色信息可以形成第一彩色原始图像。第二彩色信息可以形成第二彩色原始图像。第一全色信息和第二全色信息可以生成全色原始图像。高动态范围图像处理***100中的彩色高动态融合单元30和图像处理器20可以对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。Please refer to FIGS. 1 to 3, FIG. 5 and FIG. 11. In some embodiments, the control unit 13 is used to control the exposure of the pixel array 11. Among them, for a plurality of photosensitive pixels 110 in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, at least one single-color photosensitive pixel is exposed with a second exposure time less than the first exposure time, and at least one full-color photosensitive pixel is exposed The photosensitive pixel W is exposed at a third exposure time less than or equal to the first exposure time, and at least one full-color photosensitive pixel W is exposed at a fourth exposure time less than the first exposure time. The plurality of single-color photosensitive pixels exposed at the first exposure time in the pixel array 11 may generate first color information, and the plurality of single-color photosensitive pixels exposed at the second exposure time may generate second color information, which are exposed at the third exposure time. The plurality of panchromatic photosensitive pixels W may generate the first panchromatic information, and the plurality of panchromatic photosensitive pixels W exposed at the fourth exposure time may generate the second panchromatic information. The first color information may form a first color original image. The second color information can form a second color original image. The first panchromatic information and the second panchromatic information can generate a panchromatic original image. The color high dynamic fusion unit 30 and the image processor 20 in the high dynamic range image processing system 100 can perform high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the panchromatic original image. Get the target image.

需要说明的是，在某些实施例中，像素阵列11的曝光过程可以是：(1)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110、以第三曝光时间曝光的感光像素110及以第四曝光时间曝光的感光像素110依次序曝光(其中四者的曝光顺序不作限制)，且四者的曝光进行时间均不重叠；(2)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110、以第三曝光时间曝光的感光像素110及以第四曝光时间曝光的感光像素110依次序曝光(其中四者的曝光顺序不作限制)，且四者的曝光进行时间存在部分重叠；(3)所有以较短的曝光时间曝光的感光像素110的曝光进行时间均位于以最长的曝光时间曝光的感光像素110的曝光进行时间内，例如，以第二曝光时间曝光的所有单颜色感光像素的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内，以第三曝光时间曝光的所有全色感光像素W的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内，以第四曝光时间曝光的所有全色感光像素W的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内。本申请实施方式的高动态范围处理***100可采用第(3)种曝光方式，使用该种曝光方式可以缩短像素阵列11所需要的整体曝光时间，有利于提升图像的帧率。It should be noted that, in some embodiments, the exposure process of the pixel array 11 may be: (1) the photosensitive pixels 110 exposed at the first exposure time, the photosensitive pixels 110 exposed at the second exposure time, and the third exposure time. The time-exposed photosensitive pixel 110 and the photosensitive pixel 110 exposed at the fourth exposure time are sequentially exposed (the exposure sequence of the four is not limited), and the exposure time of the four does not overlap; (2) the first exposure time The exposed photosensitive pixels 110, the photosensitive pixels 110 exposed at the second exposure time, the photosensitive pixels 110 exposed at the third exposure time, and the photosensitive pixels 110 exposed at the fourth exposure time are sequentially exposed (the exposure order of the four is not limited ), and there is a partial overlap in the exposure time of the four; (3) The exposure time of all the photosensitive pixels 110 exposed with a shorter exposure time is within the exposure time of the photosensitive pixels 110 exposed with the longest exposure time For example, the exposure time of all the single-color photosensitive pixels exposed at the second exposure time is within the exposure time of all the single-color photosensitive pixels exposed at the first exposure time, and all the full-color photosensitive pixels exposed at the third exposure time The exposure time of the pixel W is within the exposure time of all single-color photosensitive pixels exposed at the first exposure time, and the exposure time of all the full-color photosensitive pixels W exposed at the fourth exposure time is within the first exposure time The exposure time of all single-color photosensitive pixels exposed. The high dynamic range processing system 100 of the embodiment of the present application can adopt the (3) exposure method, which can shorten the overall exposure time required by the pixel array 11, which is beneficial to increase the frame rate of the image.

例如，如图1至图3、及图11所示，像素阵列11中的所有全色感光像素W的曝光时间相同(即第三曝光时间等于第四曝光时间)。具体地，对于每个子单元中的多个(图11所示为4个)感光像素110，一个单颜色感光像素以第一曝光时间(例如图11所示的长曝光时间L)曝光，一个单颜色感光像素以第二曝光时间(例如图11所示的短曝光时间S)曝光，两个全色感光像素W分别以第三曝光时间和第四曝光时间(例如第三曝光时间和第四曝光时间均为图11所示的中曝光时间M)曝光。像素阵列11曝光结束后，图像传感器10可以输出三张原始图像，分别为：(1)第一彩色原始图像，由以长曝光时间L曝光的多个单颜色感光像素生成的第一彩色信息组成；(2)第二彩色原始图像，由以短曝光时间S曝光的多个单颜色感光像素生成的第二彩色信息组成；(3)全色原始图像，由以中曝光时间M曝光的多个全色感光像素W生成的第一全色信息和第二全色信息组成。全色原始图像可以包括第一全色原始图像和第二全色原始图像。在某些实施例中，图像传感器10可以输出四张原始图像，分别为：(1)第一彩色原始图像，由以长曝光时间L曝光的多个单颜色感光像素生成的第一彩色信息组成；(2)第二彩色原始图像，由以短曝光时间S曝光的多个单颜色感光像素生成的第二彩色信息组成；(3)第一全色原始图像，由以第三曝光时间M曝光的多个全色感光像素生成的第一全色信息生成；(4)第二全色原始图像，由以第四曝光时间M曝光的多个全色感光像素生成的第二全色信息生成。图11所示为图像传感器10输出三张原始图像的情况。For example, as shown in FIGS. 1 to 3 and FIG. 11, the exposure time of all panchromatic photosensitive pixels W in the pixel array 11 is the same (that is, the third exposure time is equal to the fourth exposure time). Specifically, for a plurality of (four as shown in FIG. 11) photosensitive pixels 110 in each sub-unit, one single-color photosensitive pixel is exposed for the first exposure time (for example, the long exposure time L shown in FIG. 11), and one single-color photosensitive pixel is exposed The color photosensitive pixels are exposed for the second exposure time (for example, the short exposure time S shown in FIG. 11), and the two full-color photosensitive pixels W are respectively exposed for the third exposure time and the fourth exposure time (for example, the third exposure time and the fourth exposure time). The time is the middle exposure time M) exposure shown in FIG. 11. After the pixel array 11 is exposed, the image sensor 10 can output three original images, which are: (1) The first color original image, which is composed of first color information generated by multiple single-color photosensitive pixels exposed with a long exposure time L (2) The second color original image is composed of second color information generated by a plurality of single-color photosensitive pixels exposed with a short exposure time S; (3) A full-color original image is composed of a plurality of exposure time M exposed The first panchromatic information generated by the panchromatic photosensitive pixel W and the second panchromatic information are composed. The full-color original image may include a first full-color original image and a second full-color original image. In some embodiments, the image sensor 10 can output four original images, which are: (1) The first color original image, which is composed of first color information generated by a plurality of single-color photosensitive pixels exposed with a long exposure time L (2) The second color original image is composed of second color information generated by a plurality of single-color photosensitive pixels exposed with a short exposure time S; (3) The first full-color original image is exposed by a third exposure time M The first panchromatic information generated by the plurality of panchromatic photosensitive pixels is generated; (4) the second panchromatic original image is generated by the second panchromatic information generated by the plurality of panchromatic photosensitive pixels exposed at the fourth exposure time M. FIG. 11 shows a situation where the image sensor 10 outputs three original images.

又例如，如图1至图3、及图12所示，同一所述子单元中的部分全色感光像素W以第四曝光时间曝光，其余全色感光像素W以第三曝光时间曝光，其中第三曝光时间和第四曝光时间不同。全色原始图像可以包括第一全色原始图像和第二全色原始图像。具体地，对于每个子单元中的(图12所示为4个)感光像素110，一个单颜色感光像素以第一曝光时间(例如图12所示的长曝光时间L)曝光，一个单颜色感光像素以第二曝光时间(例如图12所示的短曝光时间S)曝光，一个全色感光像素W以第三曝光时间(例如图12所示的长曝光时间L曝光)，一个全色感光像素W以第四曝光时间(例如图12所示的短曝光时间S)曝光。像素阵列11曝光结束后，图像传感器10可以输出四张原始图像，分别为：(1)第一彩色原始图像，由以长曝光时间L曝光的多个单颜色感光像素生成的第一彩色信息生成；(2)第二彩色原始图像，由以短曝光时间S曝光的多个单颜色感光像素生成的第二彩色信息生成；(3)第一全色原始图像，由以长曝光时间L曝光的多个全色感光像素生成的第一全色信息生成；(4)第二全色原始图像，由以短曝光时间S曝光的多个全色感光像素生成的第二全色信息生成。在某些实施例中，图像传感器10可以输出三张原始图像，分别为：(1)第一彩色原始图像，由以长曝光时间L曝光的多个单颜色感光像素生成的第一彩色信息生成；(2)第二彩色原始图像，由以短曝光时间S曝光的多个单颜色感光像素生成的第二彩色信息生成；(3)全色原始图像，由第一全色信息和第二全色信息组成，第一全色信息由以长曝光时间L曝光的多个全色感光像素W生成，第二全色信息以短曝光时间S曝光的多个全色感光像素W生成。图12所示为图像传感器10输出四张原始图像的情况。For another example, as shown in FIG. 1 to FIG. 3 and FIG. 12, part of the full-color photosensitive pixels W in the same subunit is exposed at the fourth exposure time, and the remaining full-color photosensitive pixels W are exposed at the third exposure time. The third exposure time is different from the fourth exposure time. The full-color original image may include a first full-color original image and a second full-color original image. Specifically, for the photosensitive pixels 110 (four as shown in FIG. 12) in each subunit, one single-color photosensitive pixel is exposed for the first exposure time (for example, the long exposure time L shown in FIG. 12), and one single-color photosensitive pixel is exposed Pixels are exposed for the second exposure time (for example, the short exposure time S shown in FIG. 12), one full-color photosensitive pixel W is exposed for the third exposure time (for example, the long exposure time L shown in FIG. 12), and one full-color photosensitive pixel W is exposed for the fourth exposure time (for example, the short exposure time S shown in FIG. 12). After the pixel array 11 is exposed, the image sensor 10 can output four original images, which are: (1) The first color original image, which is generated by the first color information generated by a plurality of single-color photosensitive pixels exposed with a long exposure time L (2) The second color original image is generated by the second color information generated by a plurality of single-color photosensitive pixels exposed with a short exposure time S; (3) The first full-color original image is generated by a long exposure time L The first panchromatic information generated by the plurality of panchromatic photosensitive pixels is generated; (4) the second panchromatic original image is generated by the second panchromatic information generated by the plurality of panchromatic photosensitive pixels exposed with a short exposure time S. In some embodiments, the image sensor 10 may output three original images, which are: (1) The first color original image, which is generated by the first color information generated by a plurality of single-color photosensitive pixels exposed with a long exposure time L (2) The second color original image is generated by the second color information generated by multiple single-color photosensitive pixels exposed with a short exposure time S; (3) The full-color original image is generated by the first full-color information and the second full-color information The color information is composed of the first panchromatic information generated by a plurality of panchromatic photosensitive pixels W exposed with a long exposure time L, and the second panchromatic information is generated by a plurality of panchromatic photosensitive pixels W exposed with a short exposure time S. FIG. 12 shows a situation where the image sensor 10 outputs four original images.

请参阅图13，在某些实施方式中，高动态范围图像处理***100还可以包括全色信息融合单元50。高动态范围图像处理***100中，第三曝光时间可以等于第四曝光时间，第三曝光时间可以大于第二曝光时间，并且小于第一曝光时间。例如，第三曝光时间可以等于第四曝光时间，均为中时间M曝光；第二曝光时间可以为短时间S曝光；第一曝光时间可以为长时间L曝光。在某些实施例中，全色信息融合单元50用于将每个子单元中以第三曝光时间曝光的全色感光像素W生成的第一全色信息和以第四曝光时间曝光的全色感光像素W生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像。在另一些实施例中，全色原始图像包括第一全色原始图像和第二全色原始图像。以第三曝光时间曝光的全色感光像素W生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素W生成第二全色信息得到第二全色原始图像。全色信息融合单元50用于将第一全色原始图像的每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息与第二全色原始图像的对应子单元中以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像。本申请实施方式的全色信息融合单元50对图像进行像素相加处理或像素求平均处理，将两个相同或相应坐标的像素通过像素相加处理或像素求平均处理合成为一个像素值，有利于减轻原始图像数据中的像素坏点或者环境光干扰对图像的成像质量的影响，有助于提高动态范围图像处理***100的成像质量。Referring to FIG. 13, in some embodiments, the high dynamic range image processing system 100 may further include a panchromatic information fusion unit 50. In the high dynamic range image processing system 100, the third exposure time may be equal to the fourth exposure time, and the third exposure time may be greater than the second exposure time and less than the first exposure time. For example, the third exposure time may be equal to the fourth exposure time, both of which are medium time M exposure; the second exposure time may be short time S exposure; and the first exposure time may be long time L exposure. In some embodiments, the panchromatic information fusion unit 50 is used to combine the first panchromatic information generated by the panchromatic photosensitive pixel W exposed at the third exposure time and the panchromatic photosensitive pixel W exposed at the fourth exposure time in each subunit. The second panchromatic information generated by the pixel W is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image. In other embodiments, the full-color original image includes a first full-color original image and a second full-color original image. The panchromatic photosensitive pixel W exposed at the third exposure time generates the first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixel W exposed at the fourth exposure time generates the second panchromatic information to obtain the second panchromatic original image. image. The panchromatic information fusion unit 50 is used to combine the first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time in each subunit of the first panchromatic original image and the corresponding subunits of the second panchromatic original image. The second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image. The full-color information fusion unit 50 of the embodiment of the present application performs pixel addition processing or pixel averaging processing on an image, and combines two pixels with the same or corresponding coordinates into one pixel value through pixel addition processing or pixel averaging processing. It is helpful to reduce the impact of defective pixels in the original image data or ambient light interference on the imaging quality of the image, and helps to improve the imaging quality of the dynamic range image processing system 100.

本申请实施方式中的像素相加处理的过程可以是，第一全色信息中的每个像素值和第二全色信息中对应位置的每个像素值进行像素相加，而得到一个新像素值，将得到的新像素值置于全色原始图像的对应位置的像素格中，从而得到全色原始图像。具体地，请参阅图14，此处以全色信息融合单元50对第一全色原始图像和第二全色原始图像进行像素相加处理得到全色原始图像，第三曝光时间为长时间L曝光，第四曝光时间为短时间S曝光为例进行说明。第一全色原始图像中的第一全色信息中的左上角像素值L1和对应位置的第二全色原始图像中的第二全色信息中的左上角像素值S1进行相加，将所得的像素值(L1+S1)作为对应位置的新的像素值，将所有的像素格都进行上述步骤的像素相加处理之后，得到如图14所示的全色原始图像。此时，全色原始图像的像素个数与第一全色原始图像和第二全色原始图像的像素个数相同。The process of pixel addition processing in the embodiments of the present application may be: each pixel value in the first panchromatic information and each pixel value in the corresponding position in the second panchromatic information are pixel-added to obtain a new pixel Value, place the obtained new pixel value in the pixel grid at the corresponding position of the panchromatic original image, thereby obtaining the panchromatic original image. Specifically, please refer to FIG. 14, where the panchromatic information fusion unit 50 performs pixel addition processing on the first panchromatic original image and the second panchromatic original image to obtain the panchromatic original image, and the third exposure time is long L exposure. , The fourth exposure time is a short time S exposure as an example for description. The upper left pixel value L1 in the first panchromatic information in the first panchromatic original image and the upper left pixel value S1 in the second panchromatic information in the second panchromatic original image at the corresponding position are added together, and the result is The pixel value (L1+S1) of is used as the new pixel value of the corresponding position. After all pixel grids are subjected to the pixel addition processing of the above steps, the full-color original image as shown in FIG. 14 is obtained. At this time, the number of pixels of the panchromatic original image is the same as the number of pixels of the first panchromatic original image and the second panchromatic original image.

本申请实施方式中的像素求平均处理的过程可以是，第一全色信息中的每个像素值和第二全色信息中对应位置的每个像素值进行对应的像素相加，将所得结果除以2，而得到一个新像素值，将得到的新像素值置于全色原始图像的对应位置的像素格中，从而得到全色原始图像。请参阅图15，此处以全色信息融合单元50对第一全色原始图像和第二全色原始图像进行像素求平均处理得到全色原始图像，第三曝光时间与第四曝光时间相等，均为中等时间M曝光为例进行说明。第一全色原始图像中的第一全色信息中的左上角像素值M1和对应位置的第二全色原始图像中的第二全色信息中的左上角像素值M1’进行相加，将所得的像素值(M1+M1’)/2作为对应位置的新的像素值，并将新的像素值置于全色原始图像的对应位置的像素格中。将第一全色原始图像和第二全色原始图像所有的像素格都进行上述步骤的像素相加之后，得到如图15所示的全色原始图像。The process of pixel averaging processing in the embodiment of the present application may be: each pixel value in the first panchromatic information and each pixel value in the corresponding position in the second panchromatic information are added to the corresponding pixel, and the result is Divide by 2 to obtain a new pixel value, and place the obtained new pixel value in the pixel grid at the corresponding position of the panchromatic original image, thereby obtaining the panchromatic original image. Please refer to FIG. 15, where the panchromatic information fusion unit 50 performs pixel averaging processing on the first panchromatic original image and the second panchromatic original image to obtain the panchromatic original image. The third exposure time is equal to the fourth exposure time, both Take the medium time M exposure as an example. The upper left pixel value M1 in the first panchromatic information in the first panchromatic original image and the upper left pixel value M1' in the second panchromatic information in the second panchromatic original image at the corresponding position are added together, and The obtained pixel value (M1+M1')/2 is used as the new pixel value of the corresponding position, and the new pixel value is placed in the pixel grid of the corresponding position of the panchromatic original image. After adding all the pixel grids of the first full-color original image and the second full-color original image to the pixels in the above steps, the full-color original image as shown in FIG. 15 is obtained.

请参阅图1，在另一些实施方式中，高动态范围图像处理***100还可以包括全色高动态融合单元40(此时，高动态范围图像处理***100无需包括全色信息融合单元50)。全色原始图像包括第一全色原始图像和第二全色原始图像。以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。彩色高动态融合单元30、全色高动态融合单元40和图像处理器20用于对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。1, in other embodiments, the high dynamic range image processing system 100 may further include a panchromatic high dynamic fusion unit 40 (at this time, the high dynamic range image processing system 100 does not need to include the panchromatic information fusion unit 50). The full-color original image includes a first full-color original image and a second full-color original image. The panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information to obtain a first panchromatic original image, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain a second panchromatic original image. The color high-dynamic fusion unit 30, the panchromatic high-dynamic fusion unit 40, and the image processor 20 are used to perform high dynamics on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image. Range processing, image processing and fusion algorithm processing get the target image.

请参阅图1，在某些实施方式中，图像处理器20包括图像前端处理单元202和融合模块204。图像前端处理单元202包括彩色处理模块2021和全色处理模块2022。图像处理包括第一图像处理和第二图像处理。彩色高动态融合单元30可以将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像。全色高动态融合单元40可以将第一全色原始图像和第二全色原始图像融合得到高动态全色图像。彩色处理模块2021可以对高动态彩色图像进行第一图像处理，得到彩色中间图像。全色处理模块2022可以对高动态全色图像进行第二图像处理，得到全色中间图像。融合模块204可以对彩色中间图像和全色中间图像进行融合算法处理得到目标图像。Referring to FIG. 1, in some embodiments, the image processor 20 includes an image front-end processing unit 202 and a fusion module 204. The image front-end processing unit 202 includes a color processing module 2021 and a full-color processing module 2022. Image processing includes first image processing and second image processing. The color high dynamic fusion unit 30 can fuse the first color original image and the second color original image to obtain a high dynamic color image. The panchromatic high dynamic fusion unit 40 can merge the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image. The color processing module 2021 may perform first image processing on the high dynamic color image to obtain a color intermediate image. The panchromatic processing module 2022 can perform second image processing on the high dynamic panchromatic image to obtain a panchromatic intermediate image. The fusion module 204 may perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.

请参阅图16，在另一些实施方式中，图像处理器20包括图像前端处理单元202和融合模块204。图像前端处理单元202包括彩色处理模块2021和全色处理模块2022。彩色处理模块2021包括第一子处理单元20211和第二子处理单元20212。图像处理包括第一图像处理和第二图像处理，第一图像处理包括第一图像子处理和第二图像子处理。彩色处理模块2021(中的第一子处理单元20211)可以对第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像。全色处理模块2022可以对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像。彩色高动态融合单元30可以将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像。全色高动态融合单元40可以将第一全色中间图像和第二全色中间图像融合得到高动态全色图像。彩色处理模块2021(中的第二子处理单元20212)可以对高动态彩色图像进行第二图像子处理，得到彩色中间图像。融合模块204可以对彩色中间图像和高动态全色图像进行融合算法处理得到目标图像。彩色高动态融合单元30和全色高动态融合单元40可以集成在图像前端处理单元202中，也可以集成在图像处理器20中。Referring to FIG. 16, in other embodiments, the image processor 20 includes an image front-end processing unit 202 and a fusion module 204. The image front-end processing unit 202 includes a color processing module 2021 and a full-color processing module 2022. The color processing module 2021 includes a first sub-processing unit 20211 and a second sub-processing unit 20112. The image processing includes a first image processing and a second image processing, and the first image processing includes a first image sub-processing and a second image sub-processing. The color processing module 2021 (the first sub-processing unit 20211) may perform first image sub-processing on the first color original image to obtain a first color intermediate image, and perform the first image sub-processing on the second color original image to obtain the first image sub-processing. Two color intermediate images. The panchromatic processing module 2022 may perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image. The color high dynamic fusion unit 30 can merge the first color intermediate image and the second color intermediate image to obtain a high dynamic color image. The panchromatic high dynamic fusion unit 40 can merge the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image. The color processing module 2021 (the second sub-processing unit 20212 in) may perform second image sub-processing on the high dynamic color image to obtain a color intermediate image. The fusion module 204 can perform fusion algorithm processing on the color intermediate image and the high dynamic panchromatic image to obtain the target image. The color high-dynamic fusion unit 30 and the full-color high-dynamic fusion unit 40 may be integrated in the image front-end processing unit 202, or may be integrated in the image processor 20.

请参阅图17，在又一些实施方式中，图像处理器20包括图像前端处理单元202和融合模块204。图像前端处理单元202包括彩色处理模块2021和全色处理模块2022。图像处理包括第一图像处理和第二图像处理。彩色处理模块2021可以对第一彩色原始图像进行第一图像处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像处理，得到第二彩色中间图像。全色处理模块2022可以对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像。彩色高动态融合单元30可以将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像。全色高动态融合单元40可以将第一全色中间图像和第二全色中间图像融合得到高动态全色图像。融合模块204可以对高动态彩色图像和高动态全色图像进行融合算法处理得到目标图像。Please refer to FIG. 17. In still other embodiments, the image processor 20 includes an image front-end processing unit 202 and a fusion module 204. The image front-end processing unit 202 includes a color processing module 2021 and a full-color processing module 2022. Image processing includes first image processing and second image processing. The color processing module 2021 may perform first image processing on the first color original image to obtain a first color intermediate image, and perform the first image processing on the second color original image to obtain a second color intermediate image. The panchromatic processing module 2022 may perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image. The color high dynamic fusion unit 30 can merge the first color intermediate image and the second color intermediate image to obtain a high dynamic color image. The panchromatic high dynamic fusion unit 40 can merge the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image. The fusion module 204 can perform fusion algorithm processing on the high dynamic color image and the high dynamic panchromatic image to obtain the target image.

彩色处理模块2021中，第一图像处理包括：黑电平矫正处理、镜头阴影矫正处理、去马赛克处理、坏点补偿处理、色彩矫正处理、全局色调映射处理、色彩转换处理中的一个或多个；全色处理模块2022中，第二图像处理包括：黑电平矫正处理、镜头阴影矫正处理、坏点补偿处理、全局色调映射处理中的一个或多个。In the color processing module 2021, the first image processing includes: one or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing In the panchromatic processing module 2022, the second image processing includes one or more of black level correction processing, lens shading correction processing, dead pixel compensation processing, and global tone mapping processing.

由于图像传感器10采集的信息经过一系列转换生成原始图像数据。以8bit数据为例，单个像素的有效值是0～255，但是实际图像传感器10中的模数转换芯片的精度可能无法将电压值很小的一部分转换出来，便容易造成生成图像的暗部细节的损失。黑电平矫正处理的过程可以是，彩色处理模块2021或全色处理模块2022在图像传感器10输出的原始图像数据的基础上，将每个像素值减去一个固定值。各颜色通道(例如红色通道、绿色通道、蓝色通道和全色通道，在某些实施例中，红色通道指的是图像传感器10输出的图像中由红色感光像素生成的红色信息，绿色通道指的是图像传感器10输出的图像中由绿色感光像素生成的绿色信息，红色通道指的是图像传感器10输出的图像中由蓝色感光像素生成的蓝色信息，全色通道指的是图像传感器10输出的图像中由全色感光像素生成的全色信息)对应的固定值可以是一样，也可以是不一样。例如，请参阅图17，图像传感器10输出第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像，图像处理器20接收第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像后，彩色处理模块2021对第一彩色原始图像和第二彩色原始图像进行第一图像处理中的黑电平矫正处理；全色处理模块2022对第一全色原始图像和第二全色原始图像进行第二图像处理中的黑电平矫正处理。以彩色处理模块2021对第一彩色原始图像进行黑电平矫正处理为例，第一彩色原始图像中具有红色通道、绿色通道和蓝色通道。请参阅图18，彩色处理模块2021对第一彩色原始图像进行黑电平矫正处理，第一彩色原始图像中所有的像素值均减去固定值5，从而得到经过黑电平矫正处理的第一彩色原始图像。同时图像传感器10在AD的输入之前加上一个固定的偏移量5(或者其他数值)，使输出的像素值在5(或者其他数值)～255之间，配合黑电平矫正处理，能使得本申请实施方式的图像传感器10和高动态范围图像处理***100得到的图像的暗部的细节完全保留的同时，不增大或减小图像的像素值，有利于提高成像质量。The information collected by the image sensor 10 undergoes a series of conversions to generate original image data. Taking 8bit data as an example, the effective value of a single pixel is 0-255, but the accuracy of the analog-to-digital conversion chip in the actual image sensor 10 may not be able to convert a small part of the voltage value, which will easily cause the dark details of the generated image loss. The process of black level correction processing may be that the color processing module 2021 or the panchromatic processing module 2022 subtracts a fixed value from each pixel value on the basis of the original image data output by the image sensor 10. Each color channel (such as a red channel, a green channel, a blue channel, and a panchromatic channel. In some embodiments, the red channel refers to the red information generated by the red photosensitive pixels in the image output by the image sensor 10, and the green channel refers to Is the green information generated by the green photosensitive pixels in the image output by the image sensor 10. The red channel refers to the blue information generated by the blue photosensitive pixels in the image output by the image sensor 10, and the panchromatic channel refers to the image sensor 10. The fixed value corresponding to the full-color information generated by the full-color photosensitive pixel in the output image may be the same or different. For example, referring to FIG. 17, the image sensor 10 outputs a first color original image, a second color original image, a first full color original image, and a second full color original image, and the image processor 20 receives the first color original image, the second color original image, and the second full color original image. After the color original image, the first full color original image, and the second full color original image, the color processing module 2021 performs the black level correction processing in the first image processing on the first color original image and the second color original image; full color The processing module 2022 performs black level correction processing in the second image processing on the first full-color original image and the second full-color original image. Taking the color processing module 2021 performing black level correction processing on the first color original image as an example, the first color original image has a red channel, a green channel, and a blue channel. Referring to FIG. 18, the color processing module 2021 performs black level correction processing on the first color original image, and all pixel values in the first color original image are subtracted from a fixed value of 5, thereby obtaining the first black level correction processing. Color original image. At the same time, the image sensor 10 adds a fixed offset of 5 (or other values) before the input of AD, so that the output pixel value is between 5 (or other values) to 255. With the black level correction processing, it can make While the details of the dark parts of the image obtained by the image sensor 10 and the high dynamic range image processing system 100 of the embodiment of the present application are completely preserved, the pixel value of the image is not increased or decreased, which is beneficial to improving the imaging quality.

镜头阴影是由于镜头对于光学折射不均匀导致的镜头周围出现阴影的情况，即影像区的中心和四周的接收到的光强程度不一致的现象。镜头阴影矫正处理的过程可以是，彩色处理模块2021或全色处理模块2022在图像传感器10输出的原始图像数据或经过黑电平矫正处理的图像数据的基础上，将被处理图像进行网格划分，再通过各网格区域邻近的或者自身及邻近周的补偿系效,采用双线性插值方法对图像进行镜头阴影矫正。下文以对第一彩色原始图像进行镜头阴影矫正处理为例进行说明，如图19所示，彩色处理模块2021将第一彩色原始图像(即被处理图像)进行划分，均等地分为十六个网格，十六个网格中每个网格具有一预设好的补偿系数。然后，彩色处理模块2021根据各网格区域邻近的或者自身及其邻近的补偿系效通过双线性插值方法对图像进行阴影矫正。R2为图示的经过镜头阴影矫正处理的第一彩色中间图像中虚线框内的像素值，R1为图示的第一彩色原始图像中的虚线框内的像素值。R2＝R1*k1，k1由R1像素邻近的的网格的补偿系数1.10、1.04、1.05和1.09进行双线性插值获得。设图像的坐标记为(x,y)，x从左第一个像素开始往右计数，y从上第一个像素开始往下计数，x和y均为自然数，如图像边上的标识所示。例如，R1的坐标为(3,3)，则R1在各网格补偿系数图中的坐标应为(0.75,0.75)。f(x,y)表示各网格补偿系数图中坐标为(x，y)的补偿值。则f(0.75,j0.75)为R1在各网格补偿系数图中对应的补偿系数值，则有f(0.75,0.75)＝(0.25)*(0.25)*f(0,0)+0.25*0.75*f(0,1)+0.75*0.25*f(1,0)+0.75*0.75f(1,1)＝0.0625*1.11+0.1875*1.10+0.1875*1.09+0.5625*1.03。各网格的补偿系数在彩色处理模块2021或全色处理模块2022进行镜头阴影矫正处理之前已经预先设置。各网格的补偿系数可由如下方法确定：(1)将镜头300置于光线强度和色温恒定且均一的密闭装置内，并使镜头300在该密闭装置内正对亮度分布均匀的纯灰色的目标对象拍摄得到灰度图像；(2)将灰度图像进行网格划分(例如划分为16个网格),得到划分为不同网格区域的灰度图像；(3)计算灰度图像的不同网格区域的补偿系数。确定了镜头300的补偿系数之后，本申请的高动态范围图像处理***100将该补偿系数预先设置在彩色处理模块2021或全色处理模块2022中，当高动态范围图像处理***100中的彩色处理模块2021或全色处理模块2022对图像进行镜头阴影矫正处理时，该补偿系数被获取，彩色处理模块2021或全色处理模块2022再根据各网格区域的补偿系效，采用双线性插值方法对图像进行镜头阴影矫正处理。Lens shadow is the phenomenon that the lens has a shadow around the lens caused by the uneven optical refraction of the lens, that is, the intensity of the received light in the center and the surrounding area of the image area is inconsistent. The process of lens shading correction processing may be that the color processing module 2021 or panchromatic processing module 2022 performs grid division on the processed image based on the original image data output by the image sensor 10 or the image data subjected to the black level correction processing. , And then use the bilinear interpolation method to perform lens shading correction on the image through the compensation effect of each grid area adjacent or itself and adjacent circumferences. The following takes the lens shading correction processing on the first color original image as an example for description. As shown in FIG. 19, the color processing module 2021 divides the first color original image (that is, the processed image) into sixteen equally Grid, each of the sixteen grids has a preset compensation coefficient. Then, the color processing module 2021 performs shading correction on the image through the bilinear interpolation method according to the compensation effect of each grid area adjacent or itself and its adjacent compensation system. R2 is the pixel value in the dashed frame in the first color intermediate image that has undergone lens shading correction processing, and R1 is the pixel value in the dashed frame in the first color original image shown in the figure. R2=R1*k1, k1 is obtained by bilinear interpolation of the compensation coefficients 1.10, 1.04, 1.05, and 1.09 of the grid adjacent to the R1 pixel. Suppose the coordinates of the image are (x, y), x is counted from the first pixel from the left to the right, y is counted from the first pixel on the top, and both x and y are natural numbers, as indicated by the logo on the edge of the image Show. For example, if the coordinates of R1 are (3,3), the coordinates of R1 in each grid compensation coefficient map should be (0.75,0.75). f(x, y) represents the compensation value of the coordinate (x, y) in each grid compensation coefficient graph. Then f(0.75,j0.75) is the compensation coefficient value corresponding to R1 in each grid compensation coefficient graph, then f(0.75,0.75)=(0.25)*(0.25)*f(0,0)+0.25 *0.75*f(0,1)+0.75*0.25*f(1,0)+0.75*0.75f(1,1)=0.0625*1.11+0.1875*1.10+0.1875*1.09+0.5625*1.03. The compensation coefficient of each grid has been preset before the color processing module 2021 or the panchromatic processing module 2022 performs lens shading correction processing. The compensation coefficient of each grid can be determined by the following methods: (1) Place the lens 300 in a closed device with constant and uniform light intensity and color temperature, and make the lens 300 face a pure gray target with uniform brightness distribution in the closed device The object is shot to obtain a grayscale image; (2) The grayscale image is gridded (for example, divided into 16 grids) to obtain the grayscale image divided into different grid areas; (3) The different grids of the grayscale image are calculated The compensation coefficient of the grid area. After determining the compensation coefficient of the lens 300, the high dynamic range image processing system 100 of the present application presets the compensation coefficient in the color processing module 2021 or the panchromatic processing module 2022. When the color processing in the high dynamic range image processing system 100 When the module 2021 or panchromatic processing module 2022 performs lens shading correction processing on the image, the compensation coefficient is obtained, and the color processing module 2021 or panchromatic processing module 2022 then adopts the bilinear interpolation method according to the compensation effect of each grid area Perform lens shading correction processing on the image.

图像传感器10的像素阵列11上的感光像素110存在工艺上的缺陷，或光信号进行转化为电信号的过程中出现错误，从而造成图像上像素信息错误，导致图像中的像素值不准确，这些有缺陷的像素表现在输出的图像上即为图像坏点。图像坏点可能存在，因此需要对图像进行坏点补偿处理。坏点补偿处理可以包括如下步骤：(1)以待检测像素点为中心像素点建立相同颜色的感光像素的像素点的3×3像素矩阵；(2)以所述中心像素点的周围像素点为参考点，判断所述中心像素点的色值与所述周围像素点的差值是否均大于第一阈值，如果是，则该中心像素点为坏点，如果否，则该中心像素点为正常点；(3)对判定为坏点的中心像素点进行双线性插值得到校正后的像素值。请参阅图20，以彩色处理模块2021对经过镜头阴影矫正处理的第一彩色原始图像进行坏点补偿处理进行说明。图20中的第一张图中的R1为待检测像素点，彩色处理模块2021以R1为中心像素点建立与R1的感光像素相同颜色的像素点的3×3像素矩阵，得到图20中的第二张图。并以中心像素点R1的所述周围像素点为参考点，判断中心像素点R1的色值与所述周围像素点的差值是否均大于第一阈值Q(Q在彩色处理模块2021中预设)。如果是，则该中心像素点R1为坏点，如果否，则该中心像素点R1为正常点。如果R1是坏点，则对R1进行双线性插值得到校正后的像素值R1’(图中展示的为R1是坏点的情况)得到图20中的第三张图。请参阅图21，以全色处理模块2022对经过镜头阴影矫正处理的第一全色原始图像进行坏点补偿处理进行说明。图21中的第一张图中的W1为待检测像素点，全色处理模块2022以W1为中心像素点建立与W1的感光像素相同颜色的像素点的3×3像素矩阵，得到图21中的第二张图。并以中心像素点W1的周围像素点为参考点，判断中心像素点W1的色值与所述周围像素点的差值是否均大于第一阈值K(K在全色处理模块2022中预设)。如果是，则该中心像素点W1为坏点，如果否，则该中心像素点W1为正常点。如果W1是坏点，则对W1进行双线性插值得到校正后的像素值W1’(图中展示的为W1是坏点的情况)得到图21中的第三张图。本申请实施方式的彩色处理模块2021和全色处理模块2022可以对图像进行坏点补偿处理，有利于高动态范围图像处理***100消除高动态范围图像处理***100的成像过程中，由于感光像素存在工艺上的缺陷，或光信号进行转化为电信号的过程中出现错误而产生的图像坏点，进而提高高动态范围图像处理***100形成的目标图像的像素值的准确性，从而使得本申请实施方式具有更好的成像效果。The photosensitive pixels 110 on the pixel array 11 of the image sensor 10 have process defects, or errors occur in the process of converting optical signals into electrical signals, resulting in incorrect pixel information on the image, resulting in inaccurate pixel values in the image. Defective pixels appearing on the output image are image dead pixels. Image dead pixels may exist, so the image needs to be compensated for dead pixels. The dead pixel compensation process may include the following steps: (1) create a 3×3 pixel matrix of the same color photosensitive pixels with the pixel to be detected as the center pixel; (2) use the surrounding pixels of the central pixel As a reference point, it is judged whether the difference between the color value of the central pixel and the surrounding pixels is greater than the first threshold. If yes, the central pixel is a dead pixel; if not, the central pixel is Normal point; (3) Perform bilinear interpolation on the central pixel point judged as a bad point to obtain the corrected pixel value. Referring to FIG. 20, the color processing module 2021 is used to perform dead pixel compensation processing on the first color original image that has undergone lens shading correction processing. R1 in the first picture in FIG. 20 is the pixel to be detected, and the color processing module 2021 uses R1 as the center pixel to establish a 3×3 pixel matrix of pixels of the same color as the photosensitive pixel of R1 to obtain the second pixel in FIG. 20 Zhang pictures. And taking the surrounding pixels of the central pixel R1 as a reference point, it is determined whether the difference between the color value of the central pixel R1 and the surrounding pixel is greater than the first threshold Q (Q is preset in the color processing module 2021 ). If it is, the central pixel R1 is a bad pixel, and if not, the central pixel R1 is a normal pixel. If R1 is a dead pixel, bilinear interpolation is performed on R1 to obtain the corrected pixel value R1' (the case where R1 is a dead pixel is shown in the figure) to obtain the third image in FIG. 20. Please refer to FIG. 21 to describe the dead pixel compensation processing performed by the panchromatic processing module 2022 on the first panchromatic original image that has undergone lens shading correction processing. W1 in the first picture in FIG. 21 is the pixel to be detected. The full-color processing module 2022 takes W1 as the center pixel to establish a 3×3 pixel matrix of pixels of the same color as the photosensitive pixel of W1 to obtain the first pixel in FIG. 21 Two pictures. And taking the surrounding pixels of the central pixel W1 as a reference point, it is determined whether the difference between the color value of the central pixel W1 and the surrounding pixels is greater than the first threshold K (K is preset in the panchromatic processing module 2022) . If it is, the central pixel W1 is a bad pixel, and if not, the central pixel W1 is a normal pixel. If W1 is a dead pixel, perform bilinear interpolation on W1 to obtain the corrected pixel value W1' (shown in the figure is the case where W1 is a dead pixel) to obtain the third image in FIG. 21. The color processing module 2021 and the full color processing module 2022 of the embodiment of the present application can perform dead pixel compensation processing on the image, which is beneficial to the high dynamic range image processing system 100 to eliminate the presence of photosensitive pixels in the imaging process of the high dynamic range image processing system 100 Process defects, or image defects caused by errors in the process of converting optical signals into electrical signals, thereby improving the accuracy of the pixel values of the target image formed by the high dynamic range image processing system 100, thereby enabling the implementation of this application The method has a better imaging effect.

由于本申请实施方式所得的第一彩色原始图像和第二彩色原始图像(或第一彩色中间图像、第二彩色中间图像、高动态彩色图像)的每个像素格中均为单颜色像素，没有其他颜色的光学信息，因此需要对第一彩色原始图像和第二彩色原始图像(或第一彩色中间图像、第二彩色中间图像、高动态彩色图像)进行去马赛克处理。以彩色处理模块2021对第一彩色原始图像(例如包括红色通道、绿色通道和蓝色通道)进行去马赛克处理为例进行说明，去马赛克处理的步骤包括如下步骤：(1)将第一彩色原始图像分解成第一红色原始图像、第一绿色原始图像和第一蓝色原始图像，如图22所示，所得的第一红色原始图像、第一绿色原始图像和第一蓝色原始图像中部分像素格没有像素值。(2)采用双线性插值方法分别对第一红色原始图像、第一绿色原始图像和第一蓝色原始图像进行插值处理。如图23所示，彩色处理模块2021采用双线性插值方法对第一蓝色原始图像进行插值处理。图23的待插值像素B1根据B1周围的四个像素B2、B3、B4和B5进行双线性插值，得到B1的插值像素B1’。图23的第一张图中的所有空白处的待插值像素均遍历地采用该双线性插值的方式补全像素值，得到插值后的第一蓝色原始图像。如图24所示，彩色处理模块2021采用双线性插值方法对第一绿色原始图像进行插值处理。图24的待插值像素G1根据G1周围的四个像素G2、G3、G4和G5进行双线性插值，得到G1的插值像素G1’。图24的第一张图中的所有空白处的待插值像素均遍历地采用该双线性插值的方式补全像素值，得到插值后的第一绿色原始图像。与之类似地，彩色处理模块2021可以采用双线性插值方法对第一红色原始图像进行插值处理，得到插值后的第一红色原始图像。(3)将插值后的第一红色原始图像、插值后的第一绿色原始图像和插值后的第一蓝色原始图像重新合成为具有3个颜色通道的一张图像，如图25所示。色彩处理模块2021对彩色图像进行去马赛克处理，有利于本申请实施方式将具有单颜色通道的像素值的彩色图像补全为具有多个颜色通道的彩色图像，从而在单颜色的感光像素的硬件基础上保持图像色彩的完整呈现。Since each pixel grid of the first color original image and the second color original image (or the first color intermediate image, the second color intermediate image, and the high dynamic color image) obtained by the implementation of this application are single-color pixels, there is no Optical information of other colors, therefore, it is necessary to perform demosaicing processing on the first color original image and the second color original image (or the first color intermediate image, the second color intermediate image, and the high dynamic color image). Taking the color processing module 2021 to perform demosaic processing on the first color original image (for example, including the red channel, the green channel, and the blue channel) as an example, the demosaic processing steps include the following steps: (1) The first color original image The image is decomposed into a first red original image, a first green original image, and a first blue original image, as shown in FIG. 22, the first red original image, the first green original image, and the first blue original image are obtained. The pixel grid has no pixel value. (2) The first red original image, the first green original image, and the first blue original image are respectively interpolated by using a bilinear interpolation method. As shown in FIG. 23, the color processing module 2021 uses a bilinear interpolation method to perform interpolation processing on the first blue original image. The pixel B1 to be interpolated in FIG. 23 performs bilinear interpolation according to the four pixels B2, B3, B4, and B5 around B1 to obtain the interpolated pixel B1' of B1. All the pixels to be interpolated in the blanks in the first image of FIG. 23 are traversed by using the bilinear interpolation method to complete the pixel values, and the interpolated first blue original image is obtained. As shown in FIG. 24, the color processing module 2021 uses a bilinear interpolation method to perform interpolation processing on the first green original image. The pixel to be interpolated G1 in FIG. 24 performs bilinear interpolation according to the four pixels G2, G3, G4, and G5 around G1 to obtain the interpolated pixel G1' of G1. The pixels to be interpolated in all blank spaces in the first image of FIG. 24 are traversed to use the bilinear interpolation method to complete the pixel values to obtain the interpolated first green original image. Similarly, the color processing module 2021 may use a bilinear interpolation method to perform interpolation processing on the first red original image to obtain the interpolated first red original image. (3) Re-synthesize the interpolated first red original image, the interpolated first green original image, and the interpolated first blue original image into an image with 3 color channels, as shown in FIG. 25. The color processing module 2021 performs demosaic processing on the color image, which is beneficial for the implementation of the present application to complete the color image with the pixel value of a single color channel into a color image with multiple color channels, so that the hardware of the single-color photosensitive pixel On the basis of maintaining the complete presentation of the image color.

色彩矫正处理具体可以为利用一个色彩校正矩阵对图像(该图像可以为经过去马赛克处理的第一彩色原始图像、第二彩色原始图像、第一彩色中间图像或第二彩色中间图像)的各像素的各颜色通道值进行一次校正，从而实现了对图像色彩的矫正。如下所示：The color correction processing can specifically be the use of a color correction matrix to adjust the pixels of the image (the image can be the first color original image, the second color original image, the first color intermediate image, or the second color intermediate image that has undergone a demosaicing process). The value of each color channel is corrected once to realize the correction of the image color. As follows:

其中，色彩矫正矩阵(Color Correction Matrix,CCM)在色彩处理模块中预设。例如，色彩矫正矩阵具体可以为：Among them, the color correction matrix (CCM) is preset in the color processing module. For example, the color correction matrix can be specifically:

色彩处理模块通过对图像中的所有像素遍历地通过以上色彩矫正矩阵进行色彩矫正处理，可以得到经过色彩矫正处理的图像。本申请实施方式中色彩矫正处理有利于消除图像或视频帧中因为有色光源等造成的颜色严重偏差、图像中人或物体颜色失真的问题，使得本申请实施方式的高动态范围图像处理***100能够恢复图像原始色彩，提高了图像的视觉效果。The color processing module traverses all pixels in the image and performs color correction processing through the above color correction matrix to obtain an image that has undergone color correction processing. The color correction processing in the embodiment of the present application is beneficial to eliminate the serious color deviation caused by colored light sources in the image or video frame, and the color distortion of people or objects in the image, so that the high dynamic range image processing system 100 of the embodiment of the present application can Restore the original color of the image and improve the visual effect of the image.

色调映射处理可以包括如下步骤：(1)把图像(该图像可以为经过色彩矫正处理的第一彩色原始图像、第二彩色原始图像、第一彩色中间图像或第二彩色中间图像)的灰度值归一化到区间[0,1]内，记归一化后的灰度值为Vin；(2)设Vout＝Y(Vin)，Vout和Vin之间的映射关系可以为如图26所示；(3)把Vout乘上255(当设定输出图像的灰度值为256阶时，乘上255，在其他设定时，可以为其他数值)后再四舍五入取整数，得到了色调映射处理后的图像。对于高动态范围的图像而言，其灰度值的二进制位数往往高于8位(普通的灰度图像灰度值的二进制位数一般是8位)，而许多显示器的灰度只有8位，因此对高动态范围的图像的颜色进行变换，有利于高动态范围的图像具有更高的兼容性，能在常规的显示器上显示。另外，由于高动态范围图像一般灰度值分布得很不均匀，只有少数的像素点较亮，大部分像素都分布在灰度值较低的区间，本申请实施方式的高动态范围图像处理***100对图像的色调映射处理并非线性的映射，而是在灰度值较低的区间的映射关系的斜率大于在灰度值较高的区间的映射关系的斜率，如图26所示，有利灰度值较低的区间内不同灰度值的像素点的区分度，而大部分像素都分布在灰度值较低的区间，因而使得本申请实施方式的高动态图像处理***100具有更好的成像效果。The tone mapping process may include the following steps: (1) The grayscale of an image (the image may be a first color original image, a second color original image, a first color intermediate image, or a second color intermediate image that has undergone color correction processing) The value is normalized to the interval [0,1], and the normalized gray value is Vin; (2) Set Vout=Y(Vin), the mapping relationship between Vout and Vin can be as shown in Figure 26 Show; (3) Multiply Vout by 255 (when the grayscale value of the output image is set to 256 levels, multiply by 255, in other settings, it can be other values) and then round to an integer to get the tone mapping The processed image. For images with high dynamic range, the binary digits of the gray value are often higher than 8 bits (the binary digits of the gray value of ordinary gray-scale images are generally 8 bits), and the gray scale of many displays is only 8 bits Therefore, the color of the high dynamic range image is changed, which is beneficial for the high dynamic range image to have higher compatibility and can be displayed on a conventional monitor. In addition, because the high dynamic range image generally has a very uneven distribution of gray values, only a few pixels are brighter, and most of the pixels are distributed in the interval with a lower gray value. The high dynamic range image processing system of the embodiment of the present application The tone mapping processing of 100 pairs of images is non-linear, but the slope of the mapping relationship in the interval with lower gray value is greater than the slope of the mapping relationship in the interval with higher gray value. As shown in Figure 26, it is beneficial to gray The degree of discrimination of pixels with different gray values in the interval with a lower degree value, and most pixels are distributed in the interval with a lower gray value, so that the high dynamic image processing system 100 of the embodiment of the present application has better Imaging effect.

为了图像具有更广泛的应用场景或者具有更高效率的传输格式，本申请实施方式的高动态范围图像处理***100可以对图像(可以为经过色调映射处理的第一彩色原始图像、第二彩色原始图像、第一彩色中间图像或第二彩色中间图像)进行色彩转换处理，将图像由一个色彩空间(例如RGB色彩空间)转换成另一个色彩空间(例如YUV色彩空间)从而具有更广泛的应用场景或者具有更高效率的传输格式。在具体的实施例中，色彩转换处理的步骤可以为对图像中的所有像素值的R、G和B通道像素值进行如下公式转换得到Y、U和V通道像素值：(1)Y＝0.30R+0.59G+0.11B；(2)U＝0.493(B－Y)；(3)V＝0.877(R－Y)；从而将该图像由RGB色彩空间转换为YUV色彩空间。由于YUV色彩空间中的亮度信号Y和色度信号U和V是分离的，并且人眼对亮度的敏感超过色度，色彩转换处理将图像由RGB色彩空间转换为YUV色彩空间有利于本申请实施方式的高动态范围图像处理***100后续的其他图像处理对图像进行色度信息的压缩，在不影响图像观看效果的同时，能减小图像的信息量，从而提高图像的传输效率。In order for the image to have a wider range of application scenarios or a more efficient transmission format, the high dynamic range image processing system 100 of the embodiment of the present application can perform image processing (which may be the first color original image and the second color original image that have undergone tone mapping processing). The image, the first color intermediate image or the second color intermediate image) performs color conversion processing to convert the image from one color space (such as RGB color space) to another color space (such as YUV color space), which has a wider range of application scenarios Or have a more efficient transmission format. In a specific embodiment, the step of color conversion processing may be to convert the R, G, and B channel pixel values of all the pixel values in the image to obtain the Y, U, and V channel pixel values: (1) Y=0.30 R+0.59G+0.11B; (2) U=0.493 (B-Y); (3) V=0.877 (R-Y); thereby converting the image from the RGB color space to the YUV color space. Since the luminance signal Y and the chrominance signal U and V in the YUV color space are separated, and the human eye is more sensitive to luminance than chrominance, the color conversion process to convert the image from the RGB color space to the YUV color space is beneficial to the implementation of this application The subsequent image processing of the high dynamic range image processing system 100 compresses the chrominance information of the image, which can reduce the amount of information of the image while not affecting the viewing effect of the image, thereby improving the transmission efficiency of the image.

在某些实施方式中，第三曝光时间等于第二曝光时间，第四曝光时间等于第一曝光时间。彩色高动态融合单元30可以将第一彩色原始图像和第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的第一彩色原始图像，再融合亮度对齐后的第一彩色原始图像及第二彩色原始图像以得到高动态彩色图像。全色高动态融合单元40可以将第一全色原始图像和第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的第一全色原始图像，再融合亮度对齐后的第一全色原始图像及第二全色原始图像以得到高动态全色图像。In some embodiments, the third exposure time is equal to the second exposure time, and the fourth exposure time is equal to the first exposure time. The color high dynamic fusion unit 30 may perform brightness alignment processing on the first color original image and the second color original image to obtain a first color original image after brightness alignment, and then merge the first color original image and the second color original image after brightness alignment. Color original images to obtain highly dynamic color images. The panchromatic high dynamic fusion unit 40 may perform brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the first panchromatic original image after brightness alignment, and then merge the first panchromatic brightness aligned. The original image and the second panchromatic original image to obtain a high dynamic panchromatic image.

具体地，彩色高动态融合单元30或全色高动态融合单元40对图像进行的高动态范围处理可以包括亮度对齐处理。彩色高动态融合单元30或全色高动态融合单元40对图像(该图像可以为第一彩色原始图像和所述第二彩色原始图像、第一彩色中间图像和所述第二彩色中间图像、第一全色原始图像和所述第二全色原始图像或者第一全色中间图像和所述第二全色中间图像中的其中一组，下文中以第一彩色中间图像和所述第二彩色中间图像为例进行说明)进行亮度对齐处理包括如下步骤：(1)识别第一彩色中间图像中像素值大于第一预设阈值的过曝图像像素；(2)对于每一个过曝图像像素，以该过曝图像像素为中心扩展预定区域；(3)在预定区域内寻找像素值小于第一预设阈值的中间图像像素；(4)利用中间图像像素及第二彩色中间图像对过曝图像像素的像素值进行修正；(5)利用过曝图像像素的修正后的像素值更新第一彩色中间图像以得到亮度对齐后的第一彩色中间图像。具体地，请结合图27，假设图像像素P12(图27中第一彩色中间图像内标记有虚线圆圈的图像像素)的像素值V1大于第一预设阈值V0，即图像像素P12为过曝图像像素P12，则彩色高动态融合单元30或全色高动态融合单元40以过曝图像像素P12为中心扩展一个预定区域，例如，图27所示的3*3区域。当然，在其他实施例中，也可以是4*4区域、5*5区域、10*10区域等，在此不作限制。随后，彩色高动态融合单元30或全色高动态融合单元40在 3*3的预定区域内寻找像素值小于第一预设阈值V0的中间图像像素，例如图27中的图像像素P21(图27中第一彩色中间图像内标记有点画线圆圈的图像像素)的像素值V2小于第一预设阈值V0，则图像像素P21即为中间图像像素P21。随后，彩色高动态融合单元30或全色高动态融合单元40在第二彩色中间图像中寻找与过曝图像像素P12及中间图像像素P21分别对应的图像像素，即图像像素P1’2’(图27中第二彩色中间图像内标记有虚线圆圈的图像像素)和图像像素P2’1’(图27中第二彩色中间图像内标记有点画线圆圈的图像像素)，其中，图像像素P1’2’与过曝图像像素P12对应，图像像素P2’1’与中间图像像素P21对应，图像像素P1’2’的像素值为V3，图像像素P2’1’的像素值为V4。随后，处理器根据V1’/V3＝V2/V4来计算出V1’，并利用V1’的值来替换掉V1的值。由此，即可计算出过曝图像像素P12的实际像素值。彩色高动态融合单元30或全色高动态融合单元40对第一彩色中间图像中的每一个过曝图像像素均执行这一亮度对齐的处理过程，即可得到亮度对齐后的第一彩色中间图像。由于亮度对齐后的第一彩色中间图像中的过曝图像像素的像素值经过了修正，亮度对齐后的第一彩色中间图像中的每个图像像素的像素值均较为准确。Specifically, the high dynamic range processing performed on the image by the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 may include brightness alignment processing. The color high dynamic fusion unit 30 or the pan-color high dynamic fusion unit 40 pair images (the image may be the first color original image and the second color original image, the first color intermediate image and the second color intermediate image, the first A full-color original image and the second full-color original image or a group of the first full-color intermediate image and the second full-color intermediate image, hereinafter referred to as the first color intermediate image and the second color The intermediate image is taken as an example for description) The brightness alignment processing includes the following steps: (1) Identify the overexposed image pixels in the first color intermediate image whose pixel value is greater than the first preset threshold; (2) For each overexposed image pixel, Extend the predetermined area with the overexposed image pixel as the center; (3) Find the intermediate image pixels whose pixel value is less than the first preset threshold in the predetermined area; (4) Use the intermediate image pixels and the second color intermediate image to compare the overexposed image The pixel value of the pixel is corrected; (5) The first color intermediate image is updated by using the corrected pixel value of the overexposed image pixel to obtain the first color intermediate image with aligned brightness. Specifically, please refer to Figure 27, assuming that the pixel value V1 of the image pixel P12 (the image pixel marked with a dashed circle in the first color intermediate image in Figure 27) is greater than the first preset threshold V0, that is, the image pixel P12 is an overexposed image For the pixel P12, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 extends a predetermined area with the overexposed image pixel P12 as the center, for example, the 3*3 area shown in FIG. 27. Of course, in other embodiments, it may also be a 4*4 area, a 5*5 area, a 10*10 area, etc., which is not limited here. Subsequently, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 searches for intermediate image pixels with a pixel value less than the first preset threshold V0 in a predetermined area of 3*3, such as the image pixel P21 in FIG. 27 (FIG. 27 If the pixel value V2 of the image pixel marked with a dotted circle in the first color intermediate image is less than the first preset threshold V0, the image pixel P21 is the intermediate image pixel P21. Subsequently, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 searches the second color intermediate image for image pixels corresponding to the overexposed image pixel P12 and the intermediate image pixel P21 respectively, that is, the image pixel P1'2' (Figure 27 in the second color intermediate image marked with a dotted circle in the image pixels) and image pixel P2'1' (in Figure 27 in the second color intermediate image marked with a dotted circle in the image pixels), where the image pixel P1'2 'Corresponds to the overexposed image pixel P12, the image pixel P2 '1' corresponds to the intermediate image pixel P21, the pixel value of the image pixel P1 '2' is V3, and the pixel value of the image pixel P2 '1' is V4. Subsequently, the processor calculates V1' according to V1'/V3=V2/V4, and replaces the value of V1 with the value of V1'. Thus, the actual pixel value of the overexposed image pixel P12 can be calculated. The color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 performs this brightness alignment process on each of the overexposed image pixels in the first color intermediate image to obtain the first color intermediate image after brightness alignment . Since the pixel value of the overexposed image pixel in the first color intermediate image after brightness alignment is corrected, the pixel value of each image pixel in the first color intermediate image after brightness alignment is relatively accurate.

高动态范围处理过程中，在获取到亮度对齐后的第一彩色中间图像及第二彩色中间图像后，彩色高动态融合单元30或全色高动态融合单元40可以对亮度对齐后的第一彩色中间图像及第二彩色中间图像进行融合以得到高动态彩色图像。具体地，彩色高动态融合单元30或全色高动态融合单元40首先对亮度对齐后的第一彩色中间图像进行运动检测，以识别亮度对齐后的第一彩色中间图像中是否存在运动模糊区域。若亮度对齐后的第一彩色中间图像中不存在运动模糊区域，则直接融合亮度对齐后的第一彩色中间图像及第二彩色中间图像以得到高动态彩色图像。若亮度对齐后的第一彩色中间图像中存在运动模糊区域，则将第一彩色中间图像中的运动模糊区域剔除，只融合第二彩色中间图像的所有区域以及亮度对齐后的第一彩色中间图像中除运动模糊区域以外的区域以得到高动态彩色图像。其中，高动态彩色图像的分辨率小于像素阵列11的分辨率。具体地，在融合亮度对齐后的第一彩色中间图像及第二彩色中间图像时，若亮度对齐后的第一彩色中间图像中不存在运动模糊区域，则此时两张中间图像的融合遵循以下原则：(1)亮度对齐后的第一彩色中间图像中，过曝区域的图像像素的像素值直接替换为第二彩色中间图像中对应于该过曝区域的图像像素的像素值；(2)亮度对齐后的第一彩色中间图像中，欠曝区域的图像像素的像素值为：长曝光像素值除以长短像素值比例；(3)亮度对齐后的第一彩色中间图像中，未欠曝也未过曝区域的图像像素的像素值为：长曝光像素值除以长短像素值比例。若亮度对齐后的第一彩色中间图像中存在运动模糊区域，则此时两张中间图像的融合除了遵循上述三个原则外，还需要遵循第(4)个原则：亮度对齐后的第一彩色中间图像中，运动模糊区域的图像像素的像素值直接替换为第二彩色中间图像中对应于该运动模糊区域的图像像素的像素值。需要说明的是，对于欠曝区域以及未欠曝也未过曝区域而言，这些区域内的图像像素的像素值为长曝光像素值除以长短像素值比例，即VL/(VL/VS)＝VS’，其中，VL表示长曝光像素值，VS表示段曝光像素值，VS’表示计算出来的欠曝区域以及未欠曝也未过曝区域中图像像素的像素值。VS’的信噪比会大于VS的信噪比。本申请实施方式的高动态范围图像处理***100通过彩色高动态融合单元30或全色高动态融合单元40对图像进行高动态范围处理，先对图像进行亮度对齐处理，再对亮度对齐后的图像与另一图像进行融合进行融合，得到高动态彩色图像或高动态全色图像，使得高动态范围图像处理***100形成的目标图像具有更大的动态范围，进而具有更好的成像效果。In the high dynamic range processing process, after the first color intermediate image and the second color intermediate image after brightness alignment are obtained, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 can adjust the brightness aligned first color The intermediate image and the second color intermediate image are fused to obtain a highly dynamic color image. Specifically, the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40 first performs motion detection on the first color intermediate image after brightness alignment, to identify whether there is a motion blur area in the first color intermediate image after brightness alignment. If there is no motion blur area in the first color intermediate image after brightness alignment, the first color intermediate image and the second color intermediate image after brightness alignment are directly merged to obtain a high dynamic color image. If there is a motion blur area in the first color intermediate image after brightness alignment, remove the motion blur area in the first color intermediate image, and only fuse all areas of the second color intermediate image and the first color intermediate image after brightness alignment In order to obtain high dynamic color image in the area other than the motion blur area. Among them, the resolution of the high dynamic color image is smaller than the resolution of the pixel array 11. Specifically, when fusing the first color intermediate image and the second color intermediate image after brightness alignment, if there is no motion blur area in the first color intermediate image after brightness alignment, the fusion of the two intermediate images at this time follows the following Principle: (1) In the first color intermediate image after brightness alignment, the pixel value of the image pixel in the overexposed area is directly replaced with the pixel value of the image pixel in the second color intermediate image corresponding to the overexposed area; (2) In the first color intermediate image after brightness alignment, the pixel value of the image pixels in the underexposure area is: the long-exposure pixel value divided by the ratio of long and short pixel values; (3) In the first color intermediate image after brightness alignment, there is no under-exposure The pixel value of the image pixel in the area that is not overexposed is: the long-exposure pixel value divided by the ratio of the long-short pixel value. If there is a motion blur area in the first color intermediate image after brightness alignment, the fusion of the two intermediate images at this time must follow the above three principles, and also need to follow the (4) principle: the first color after brightness alignment In the intermediate image, the pixel value of the image pixel in the motion blur area is directly replaced with the pixel value of the image pixel in the second color intermediate image corresponding to the motion blur area. It should be noted that for under-exposed areas and areas that are neither under-exposed nor over-exposed, the pixel value of the image pixels in these areas is the long-exposure pixel value divided by the ratio of the long-short pixel value, that is, VL/(VL/VS) =VS', where VL represents the long exposure pixel value, VS represents the segment exposure pixel value, and VS' represents the calculated pixel value of the image pixel in the under-exposed area and the neither under-exposed nor over-exposed area. The signal-to-noise ratio of VS’ will be greater than the signal-to-noise ratio of VS. The high dynamic range image processing system 100 of the embodiment of the present application performs high dynamic range processing on the image through the color high dynamic fusion unit 30 or the panchromatic high dynamic fusion unit 40, first performs brightness alignment processing on the image, and then performs brightness alignment on the image after the brightness alignment. Fusion with another image is performed to obtain a high dynamic color image or a high dynamic full color image, so that the target image formed by the high dynamic range image processing system 100 has a larger dynamic range, and thus has a better imaging effect.

在另一些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。彩色高动态融合单元30用于：将第一彩色中间图像和第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的第一彩色中间图像，再融合亮度对齐后的第一彩色中间图像及第二彩色中间图像以得到高动态彩色图像。全色高动态融合单元40用于：将第一全色中间图像和第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的第一全色中间图像，再融合亮度对齐后的第一全色中间图像及第二全色中间图像以得到高动态全色图像。其中彩色高动态融合单元30或全色高动态融合单元40对图像进行亮度对齐处理的具体过程和上文一致，此处不重复。In other embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. The color high dynamic fusion unit 30 is used to: perform brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain a first color intermediate image after brightness alignment, and then merge the first color intermediate image and the brightness aligned first color intermediate image. The second color intermediate image to obtain a highly dynamic color image. The panchromatic high dynamic fusion unit 40 is used to: perform brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image after brightness alignment, and then merge the first panchromatic intermediate image with brightness alignment. The panchromatic intermediate image and the second panchromatic intermediate image are used to obtain a high dynamic panchromatic image. The specific process of the color high-dynamic fusion unit 30 or the full-color high-dynamic fusion unit 40 for the brightness alignment processing of the image is the same as the above, and will not be repeated here.

融合模块204对彩色图像和全色图像(彩色图像可以为上文所述的彩色中间图像或高动态彩色图像，全色图像可以为上文所述的全色中间图像和高动态全色图像)进行融合算法处理。若以融合模块204对彩色中间图像和全色中间图像进行融合算法处理为例进行说明，彩色中间图像具有R(即红色)、G(即绿色)和B(即蓝色)三个颜色通道的彩色信息，全色中间图像具有全色信息，全色信息又可称为亮度信息，融合算法处理的具体过程可以为如下：(1)根据彩色中间图像，计算各个像素对应的辅助值Y，其中，Y＝(R*w1+B*w2+G*w3)/(w1+w2+w3)，R为像素对应的R通道的值，G为像素对应的G通道的值，B为像素对应的B通道的值，w1、w2和w3为权重值；(2)计算彩色中间图像中各个通道值与辅助值Y的比例，得到各个像素对应的参考通道值K1、K2和K3，其中，K1＝R/Y，K2＝G/Y，K3＝B/Y；(3)对所述参考通道值K1、K2和K3进行色彩降噪处理；(4)将对应像素上的全色信息Y’与色彩降噪后的参考通道值K1-K3进行融合，生成融合后的RGB三通道值R’、G’和B’，得到目标图像；其中，R’＝K1*Y’；G’＝K2*Y’；B’＝K3*Y’。本申请实施方式的融合模块204对彩色图像和全色图像进行融合算法处理，使得最终形成的目标图像的来源同时具有彩色信息和亮度信息，由于人眼对亮度的敏感超过色度，因而对于人眼视觉特性而言，本申请实施方式的高动态范围图像处理***100具有更好的成像效果，最终得到的目标图像也更贴近人眼视觉。The fusion module 204 pairs color images and panchromatic images (the color image can be the color intermediate image or the high dynamic color image described above, and the panchromatic image can be the panchromatic intermediate image and the high dynamic panchromatic image described above) Perform fusion algorithm processing. If the fusion module 204 performs fusion algorithm processing on the color intermediate image and the panchromatic intermediate image as an example, the color intermediate image has three color channels of R (that is, red), G (that is, green), and B (that is, blue). Color information, panchromatic intermediate image has panchromatic information, panchromatic information can also be called brightness information, the specific process of fusion algorithm processing can be as follows: (1) According to the color intermediate image, calculate the auxiliary value Y corresponding to each pixel, where , Y=(R*w1+B*w2+G*w3)/(w1+w2+w3), R is the value of the R channel corresponding to the pixel, G is the value of the G channel corresponding to the pixel, and B is the value corresponding to the pixel The value of the B channel, w1, w2, and w3 are the weight values; (2) Calculate the ratio of each channel value in the color intermediate image to the auxiliary value Y to obtain the reference channel values K1, K2, and K3 corresponding to each pixel, where K1= R/Y, K2=G/Y, K3=B/Y; (3) Perform color noise reduction processing on the reference channel values K1, K2, and K3; (4) Combine the full color information Y'on the corresponding pixel with The reference channel values K1-K3 after color noise reduction are fused to generate the fused RGB three-channel values R', G'and B'to obtain the target image; where R'=K1*Y'; G'=K2* Y'; B'=K3*Y'. The fusion module 204 of the embodiment of the present application performs fusion algorithm processing on the color image and the panchromatic image, so that the source of the final target image has both color information and brightness information. Since the human eye is more sensitive to brightness than chroma, it is In terms of eye vision characteristics, the high dynamic range image processing system 100 of the embodiment of the present application has a better imaging effect, and the final target image obtained is closer to human vision.

请参阅图1，在某些实施方式中，彩色高动态融合单元30和全色高动态融合单元40集成在图像传感器10中；请参阅图16，在另外的实施方式中，彩色高动态融合单元30和全色高动态融合单元40集成在图像处理器20中。彩色高动态融合单元30和全色高动态融合单元40集成在图像传感器10或图像处理器20中使得本申请实施方式的高动态范围图像处理***100在无需提高图像传感器的硬件性能的情况下实现高动态范围处理，同时彩色高动态融合单元30和全色高动态融合单元40将高动态范围处理的功能独立地封装起来，有利于降低产品设计过程中的设计难度，提高设计更改的便捷性。Please refer to FIG. 1, in some embodiments, the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image sensor 10; please refer to FIG. 16, in another embodiment, the color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image processor 20. The color high dynamic fusion unit 30 and the panchromatic high dynamic fusion unit 40 are integrated in the image sensor 10 or the image processor 20 so that the high dynamic range image processing system 100 of the embodiment of the present application is implemented without improving the hardware performance of the image sensor High dynamic range processing. At the same time, the color high dynamic range fusion unit 30 and the full-color high dynamic range fusion unit 40 independently encapsulate the high dynamic range processing function, which is beneficial to reduce the design difficulty in the product design process and improve the convenience of design changes.

图像处理器20还可以包括接收单元201和内存单元203。接收单元201用于接收第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像和高动态全色图像中的一个或多个；内存单元203用于暂存第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像、高动态全色图像和全色中间图像中的一个或多个。图像处理器20设置接收单元201和内存单元203将图像的接收、处理和存储分开，有利于高动态范围图像处理***100的各模块具有更加独立的封装，从而使得高动态范围图像处理***100具有更高的执行效率和更好的防干扰效果，此外，还有利于降低高动态范围图像处理***100的重新设计过程的设计难度，从而降低成本。The image processor 20 may further include a receiving unit 201 and a memory unit 203. The receiving unit 201 is configured to receive one of a first color original image, a second color original image, a full color original image, a first full color original image, a second full color original image, a high dynamic color image, and a high dynamic full color image Or more; the memory unit 203 is used to temporarily store the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, the high dynamic color image, and the high dynamic full color image. One or more of a color image and a full-color intermediate image. The image processor 20 is provided with a receiving unit 201 and a memory unit 203 to separate the receiving, processing and storage of images, which is conducive to more independent packaging of the modules of the high dynamic range image processing system 100, so that the high dynamic range image processing system 100 has Higher execution efficiency and better anti-interference effect, in addition, are also beneficial to reduce the design difficulty of the redesign process of the high dynamic range image processing system 100, thereby reducing costs.

电子设备1000可以是手机、平板电脑、笔记本电脑、智能穿戴设备(例如智能手表、智能手环、智能眼镜、智能头盔)、无人机、头显设备等，在此不作限制。The electronic device 1000 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which are not limited here.

本申请实施方式的电子设备1000通过控制像素阵列11中每个子单元内的多个感光像素以不同的曝光时间曝光，并且根据不同曝光时间的感光像素将生成多张图像，以便后续处理单元对此多张图像进行高动态范围处理，从而得到具有高动态范围的目标图像，使得在无需提高图像传感器10的感光像素硬件参数的情况下，就能实现高动态范围功能，使得目标图像的亮处、暗位都能够具有更佳的表现，有利于提高成像性能的同时，有助于降低成本。The electronic device 1000 in the embodiment of the present application controls the multiple photosensitive pixels in each subunit of the pixel array 11 to be exposed at different exposure times, and generates multiple images according to the photosensitive pixels with different exposure times, so that subsequent processing units can respond to them. Multiple images are processed with high dynamic range to obtain a target image with high dynamic range, so that the high dynamic range function can be realized without the need to increase the hardware parameters of the photosensitive pixel of the image sensor 10, so that the bright part of the target image, The dark position can have better performance, which is conducive to improving imaging performance and helping to reduce costs.

在某些实施方式中，全色原始图像包括第一全色原始图像和第二全色原始图像。其中，以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括(即02)：In some embodiments, the full-color original image includes a first full-color original image and a second full-color original image. Among them, the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixels exposed with the fourth exposure time generate second panchromatic information to obtain the second panchromatic original image. image. Performing high dynamic range processing, image processing and fusion algorithm processing on the first color original image, the second color original image, and the panchromatic original image to obtain the target image includes (ie 02):

对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。High dynamic range processing, image processing and fusion algorithm processing are performed on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image.

在某些实施方式中，图像处理包括第一图像处理和第二图像处理。对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：In some embodiments, the image processing includes a first image processing and a second image processing. Performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image includes:

将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像；Fusing the first color original image and the second color original image to obtain a highly dynamic color image;

将第一全色原始图像和第二全色原始图像融合得到高动态全色图像；Fusing the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image;

对高动态彩色图像进行第一图像处理，得到彩色中间图像；Perform the first image processing on the high dynamic color image to obtain a color intermediate image;

对高动态全色图像进行第二图像处理，得到全色中间图像；和Performing second image processing on the high dynamic panchromatic image to obtain a panchromatic intermediate image; and

对彩色中间图像和全色中间图像进行融合算法处理得到目标图像。The fusion algorithm is performed on the color intermediate image and the panchromatic intermediate image to obtain the target image.

在某些实施方式中，图像处理包括第一图像处理和第二图像处理。第一图像处理包括第一图像子处理和第二图像子处理。对第一彩色原始图像、第二彩色原始图像、第一全色原始图像和第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：In some embodiments, the image processing includes a first image processing and a second image processing. The first image processing includes a first image sub-processing and a second image sub-processing. Performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain the target image includes:

对第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；Performing first image sub-processing on the first color original image to obtain a first color intermediate image, and performing the first image sub-processing on the second color original image to obtain a second color intermediate image;

对第一全色原始图像进行第二图像处理，得到第一全色中间图像，对第二全色原始图像进行第二图像处理，得到第二全色中间图像；Performing second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and performing second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image;

将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像；Fusing the first color intermediate image and the second color intermediate image to obtain a highly dynamic color image;

将第一全色中间图像和第二全色中间图像融合得到高动态全色图像；Fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image;

对高动态彩色图像进行第二图像子处理，得到彩色中间图像；和Performing the second image sub-processing on the high dynamic color image to obtain a color intermediate image; and

对彩色中间图像和高动态全色图像进行融合算法处理得到目标图像。The fusion algorithm is performed on the color intermediate image and the high dynamic panchromatic image to obtain the target image.

将第一全色中间图像和第二全色中间图像融合得到高动态全色图像；和Fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image; and

对高动态彩色图像和高动态全色图像进行融合算法处理得到目标图像。The high dynamic color image and the high dynamic panchromatic image are processed by the fusion algorithm to obtain the target image.

在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。将第一彩色原始图像和第二彩色原始图像融合得到高动态彩色图像包括：In some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. Fusion of the first color original image and the second color original image to obtain a high dynamic color image includes:

将第一彩色原始图像和第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的第一彩色原始图像，再融合亮度对齐后的第一彩色原始图像及第二彩色原始图像以得到高动态彩色图像。Perform brightness alignment processing on the first color original image and the second color original image to obtain the first color original image after brightness alignment, and then merge the first color original image and the second color original image after brightness alignment to obtain high dynamics Color image.

将第一全色原始图像和第二全色原始图像融合得到高动态全色图像包括：Fusion of the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image includes:

将第一全色原始图像和第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的第一全色原始图像，再融合亮度对齐后的第一全色原始图像及第二全色原始图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the first panchromatic original image after brightness alignment, and then merge the first panchromatic original image and the second panchromatic original image after brightness alignment Image to get a high dynamic panchromatic image.

在某些实施方式中，第三曝光时间等于第一曝光时间，第四曝光时间等于第二曝光时间。将第一彩色中间图像和第二彩色中间图像融合得到高动态彩色图像包括：In some embodiments, the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time. Fusion of the first color intermediate image and the second color intermediate image to obtain a high dynamic color image includes:

将第一彩色中间图像和第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的第一彩色中间图像，再融合亮度对齐后的第一彩色中间图像及第二彩色中间图像以得到高动态彩色图像。Perform brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain the first color intermediate image after brightness alignment, and then merge the first color intermediate image and the second color intermediate image after brightness alignment to obtain high dynamics Color image.

将第一全色中间图像和第二全色中间图像融合得到高动态全色图像包括：Fusion of the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image includes:

将第一全色中间图像和第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的第一全色中间图像，再融合亮度对齐后的第一全色中间图像及第二全色中间图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image after brightness alignment, and then merge the first panchromatic intermediate image and the second panchromatic intermediate image after brightness alignment Image to get a high dynamic panchromatic image.

在某些实施方式中，第三曝光时间等于第四曝光时间，第三曝光时间大于第二曝光时间，并且小于第一曝光时间。高动态范围图像处理方法还包括：In some embodiments, the third exposure time is equal to the fourth exposure time, and the third exposure time is greater than the second exposure time and less than the first exposure time. High dynamic range image processing methods also include:

将每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息和以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像；或者The first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time and the second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time in each subunit are subjected to pixel addition processing or pixel calculation. Averaging processing to obtain a full-color original image; or

全色原始图像包括第一全色原始图像和第二全色原始图像。图像传感器10以第三曝光时间曝光的全色感光像素生成第一全色信息得到第一全色原始图像，以第四曝光时间曝光的全色感光像素生成第二全色信息得到第二全色原始图像。高动态范围图像处理方法还包括：The full-color original image includes a first full-color original image and a second full-color original image. The image sensor 10 generates first panchromatic information to obtain the first panchromatic original image by the panchromatic photosensitive pixels exposed at the third exposure time, and generates the second panchromatic information to obtain the second panchromatic image from the panchromatic photosensitive pixels exposed at the fourth exposure time. The original image. High dynamic range image processing methods also include:

将第一全色原始图像的每个子单元中以第三曝光时间曝光的全色感光像素生成的第一全色信息与第二全色原始图像的对应子单元中以第四曝光时间曝光的全色感光像素生成的第二全色信息进行像素相加处理或像素求平均处理，以得到全色原始图像。The first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time in each subunit of the first panchromatic original image and the corresponding subunits of the second panchromatic original image exposed at the fourth exposure time The second panchromatic information generated by the color photosensitive pixels is subjected to pixel addition processing or pixel averaging processing to obtain a panchromatic original image.

在某些实施方式中，高动态范围图像处理方法还包括：In some embodiments, the high dynamic range image processing method further includes:

接收第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像和高动态全色图像中的一个或多个；和Receiving one or more of a first color original image, a second color original image, a full color original image, a first full color original image, a second full color original image, a high dynamic color image, and a high dynamic full color image; and

暂存第一彩色原始图像、第二彩色原始图像、全色原始图像、第一全色原始图像、第二全色原始图像、高动态彩色图像、高动态全色图像、彩色中间图像和全色中间图像中的一个或多个。Temporarily store the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, the high dynamic color image, the high dynamic full color image, the color intermediate image, and the full color One or more of the intermediate images.

上述任意一项实施方式的高动态范围图像处理方法的具体实施过程与前述高动态范围图像处理***100获得目标图像的具体实施过程相同，在此不再展开说明。The specific implementation process of the high dynamic range image processing method of any one of the above embodiments is the same as the specific implementation process of the aforementioned high dynamic range image processing system 100 to obtain a target image, and will not be further described here.

综上，本申请实施方式的图像传感器10、高动态范围图像处理***100及方法、电子设备1000和计算机可读存储介质400通过控制像素阵列11中每个子单元内的多个感光像素以不同的曝光时间曝光，并且根据不同曝光时间的感光像素将生成多张图像，以便后续处理单元对此多张图像进行高动态范围处理，从而得到具有高动态范围的目标图像，使得在无需提高图像传感器10的感光像素硬件参数的情况下，就能实现高动态范围功能，使得目标图像的亮处、暗位都能够具有更佳的表现，有利于提高成像性能的同时，有助于降低成本。In summary, the image sensor 10, the high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiment of the present application control the multiple photosensitive pixels in each subunit of the pixel array 11 to be different Exposure time exposure, and according to different exposure time photosensitive pixels will generate multiple images, so that the subsequent processing unit will perform high dynamic range processing on these multiple images, thereby obtaining a target image with high dynamic range, so that there is no need to improve the image sensor 10 Under the condition of the hardware parameters of the photosensitive pixel, the high dynamic range function can be realized, so that the bright and dark parts of the target image can have better performance, which is beneficial to improve imaging performance and reduce costs.

此外，相关技术中，图像处理器仅能对传统的由彩色感光像素组成的像素阵列形成的图像进行处理，不适用同时具有彩色感光像素和全色感光像素的像素阵列产生的图像。本申请实施方式的图像传感器10、高动态范围图像处理***100及方法、电子设备1000和计算机可读存储介质400适用于具有彩色感光像素和全色感光像素的像素阵列产生的图像。在同样的光环境和其他辅助硬件下，全色感光像素能接收比彩色感光像素更多的光线，从而能提高最终形成的图像的亮度，并且人眼对亮度的敏感超过色度，从而使得本申请实施方式的图像传感器10、高动态范围图像处理***100及方法、电子设备1000和计算机可读存储介质400具有更好的成像效果。In addition, in the related art, the image processor can only process the image formed by the traditional pixel array composed of color photosensitive pixels, and is not applicable to the image produced by the pixel array having both color photosensitive pixels and full-color photosensitive pixels. The image sensor 10, the high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiment of the present application are applicable to images generated by a pixel array having color photosensitive pixels and full-color photosensitive pixels. Under the same light environment and other auxiliary hardware, full-color photosensitive pixels can receive more light than color photosensitive pixels, which can improve the brightness of the final image, and the human eye is more sensitive to brightness than chromaticity, which makes the The image sensor 10, the high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the application embodiment have better imaging effects.

相关技术中采用例如提高快门速度或选取感光响应曲线呈对数形式的感光像素的方式，对高动态相机的图像传感器的硬件参数提出较高要求。本申请实施方式的图像传感器10、高动态范围图像处理***100及方法、电子设备1000和计算机可读存储介质400在无需提高图像传感器的硬件参数要求的情况下，通过在图像传感器中设置彩色高动态融合单元30、全色高动态融合单元40和融合模块204，就能实现高动态范围处理功能，进而得到成像效果更好的图像。In the related art, methods such as increasing the shutter speed or selecting photosensitive pixels whose photosensitive response curve is in a logarithmic form put forward higher requirements on the hardware parameters of the image sensor of the high-dynamic camera. The image sensor 10, the high dynamic range image processing system 100 and method, the electronic device 1000, and the computer-readable storage medium 400 of the embodiment of the present application do not need to increase the hardware parameter requirements of the image sensor. The dynamic fusion unit 30, the panchromatic high dynamic fusion unit 40, and the fusion module 204 can realize the high dynamic range processing function, thereby obtaining an image with better imaging effect.

在本申请的实施方式的描述中，需要说明的是，除非另有明确的规定和限定，术语“安装”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接或可以相互通讯；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请的实施方式中的具体含义。In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly defined and limited, the term “installation” should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral Connection; it can be mechanical connection, it can be electrical connection or it can communicate with each other; it can be directly connected, it can also be indirectly connected through an intermediate medium, it can be the internal communication of two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above-mentioned terms in the embodiments of the present application can be understood according to specific circumstances.

流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。Any process or method description described in the flowchart or described in other ways herein can be understood as a module, segment, or part of code that includes one or more executable instructions for implementing specific logical functions or steps of the process , And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of the present application belong.

在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行***、装置或设备(如基于计算机的***、包括处理模块的***或其他可以从指令执行***、装置或设备取指令并执行指令的***)使用，或结合这些指令执行***、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行***、装置或设备或结合这些指令执行***、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(控制方法)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。The logic and/or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for realizing logic functions, and can be embodied in any computer-readable medium, For use by instruction execution systems, devices, or equipment (such as computer-based systems, systems including processing modules, or other systems that can fetch instructions from instruction execution systems, devices, or equipment and execute instructions), or combine these instruction execution systems, devices Or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, device, or device or in combination with these instruction execution systems, devices, or devices. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (control method), portable computer disk cases (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable and editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because it can be used, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable media if necessary. The program is processed in a manner to obtain the program electronically, and then stored in the computer memory.

应当理解，本申请的实施方式的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行***执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。It should be understood that each part of the embodiments of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate array (PGA), field programmable gate array (FPGA), etc.

本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。A person of ordinary skill in the art can understand that all or part of the steps carried in the method of the foregoing embodiments can be implemented by a program instructing relevant hardware to complete. The program can be stored in a computer-readable storage medium. When executed, it includes one of the steps of the method embodiment or a combination thereof.

此外，在本申请的各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

上述提到的存储介质可以是只读存储器，磁盘或光盘等。The aforementioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

在本说明书的描述中，参考术语“某些实施方式”等的描述意指结合所述实施方式或示例描述的具体特征、结构或者特点包含于本申请的至少一个实施方式中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式。而且，描述的具体特征、结构或者特点可以在任何的一个或多个实施方式中以合适的方式结合。In the description of this specification, the description with reference to the term "certain embodiments" and the like means that a specific feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present application. In this specification, the schematic representation of the above-mentioned terms does not necessarily refer to the same embodiment. Moreover, the described specific features, structures or characteristics can be combined in any one or more embodiments in an appropriate manner.

尽管上面已经示出和描述了本申请实施方式，可以理解，上述实施方式是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型，本申请的范围由权利要求及其等同物限定。Although the embodiments of the present application have been shown and described above, it can be understood that the foregoing embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can perform the above-mentioned embodiments within the scope of the present application. For changes, modifications, substitutions and variations, the scope of this application is defined by the claims and their equivalents.

Claims

一种图像传感器，其特征在于，包括像素阵列，所述像素阵列包括多个全色感光像素和多个彩色感光像素，所述彩色感光像素具有比所述全色感光像素更窄的光谱响应，所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色感光像素及多个全色感光像素，当所述图像传感器中的像素阵列曝光时，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光；其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像，以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像。An image sensor, characterized by comprising a pixel array, the pixel array comprising a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels having a narrower spectral response than the panchromatic photosensitive pixels, The pixel array includes a minimum repeating unit, each of the minimum repeating units includes a plurality of sub-units, each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels, when the pixels in the image sensor During the array exposure, for a plurality of photosensitive pixels in the same subunit, at least one of the single-color photosensitive pixels is exposed at a first exposure time, and at least one of the single-color photosensitive pixels is exposed at a first exposure time that is less than the first exposure time. Exposure at two exposure times, at least one of the panchromatic photosensitive pixels is exposed at a third exposure time less than or equal to the first exposure time, and at least one of the panchromatic photosensitive pixels is exposed at a fourth exposure time less than the first exposure time Time exposure; wherein the single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image, and the single-color photosensitive pixels exposed at the second exposure time generate a second The color information is used to obtain a second color original image, the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate a second panchromatic information. Color information to get the full-color original image.
一种高动态范围图像处理***，其特征在于，包括图像传感器、彩色高动态融合单元和图像处理器；A high dynamic range image processing system, which is characterized by comprising an image sensor, a color high dynamic fusion unit and an image processor;

所述图像传感器包括像素阵列，所述像素阵列包括多个全色感光像素和多个彩色感光像素，所述彩色感光像素具有比所述全色感光像素更窄的光谱响应，所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色感光像素及多个全色感光像素，当所述图像传感器中的像素阵列曝光时，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光；其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像，以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像；The image sensor includes a pixel array, the pixel array includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels, and the pixel array includes The smallest repeating unit, each of the smallest repeating units includes a plurality of sub-units, each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels, when the pixel array in the image sensor is exposed to In the multiple photosensitive pixels in the same subunit, at least one of the single-color photosensitive pixels is exposed with a first exposure time, and at least one of the single-color photosensitive pixels is exposed with a second exposure time that is less than the first exposure time, At least one of the panchromatic photosensitive pixels is exposed at a third exposure time less than or equal to the first exposure time, and at least one of the panchromatic photosensitive pixels is exposed at a fourth exposure time less than the first exposure time; wherein, The single-color photosensitive pixels exposed at the first exposure time generate first color information to obtain a first color original image, and the single-color photosensitive pixels exposed at the second exposure time generate second color information to obtain a second color information For a color original image, the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate second panchromatic information to obtain Full-color original image;

所述彩色高动态融合单元和所述图像处理器用于对所述第一彩色原始图像、所述第二彩色原始图像和所述全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。The color high dynamic fusion unit and the image processor are used to perform high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the panchromatic original image. Target image.
根据权利要求2所述的高动态范围图像处理***，其特征在于，所述高动态范围图像处理***还包括全色高动态融合单元，所述全色原始图像包括第一全色原始图像和第二全色原始图像；The high dynamic range image processing system according to claim 2, wherein the high dynamic range image processing system further comprises a panchromatic high dynamic fusion unit, and the panchromatic original image includes a first panchromatic original image and a first panchromatic original image. Two full-color original images;

以所述第三曝光时间曝光的所述全色感光像素生成所述第一全色信息得到所述第一全色原始图像，以所述第四曝光时间曝光的所述全色感光像素生成所述第二全色信息得到所述第二全色原始图像；The panchromatic photosensitive pixels exposed at the third exposure time generate the first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate the first panchromatic information. Obtaining the second panchromatic original image by using the second panchromatic information;

所述彩色高动态融合单元、所述全色高动态融合单元和所述图像处理器用于对所述第一彩色原始图像、所述第二彩色原始图像、所述第一全色原始图像和所述第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。The color high-dynamic fusion unit, the full-color high-dynamic fusion unit, and the image processor are used to compare the first color original image, the second color original image, the first full-color original image and the image processor. The second panchromatic original image is subjected to high dynamic range processing, image processing and fusion algorithm processing to obtain a target image.
根据权利要求3所述的高动态范围图像处理***，其特征在于，所述图像处理器包括彩色处理模块、全色处理模块和融合模块，所述图像处理包括第一图像处理和第二图像处理；The high dynamic range image processing system according to claim 3, wherein the image processor includes a color processing module, a panchromatic processing module, and a fusion module, and the image processing includes a first image processing and a second image processing. ；

所述彩色高动态融合单元用于将所述第一彩色原始图像和所述第二彩色原始图像融合得到高动态彩色图像；The color high dynamic fusion unit is used to fuse the first color original image and the second color original image to obtain a high dynamic color image;

所述全色高动态融合单元用于将所述第一全色原始图像和所述第二全色原始图像融合得到高动态全色图像；The panchromatic high dynamic fusion unit is used for fusing the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image;

所述彩色处理模块用于对所述高动态彩色图像进行第一图像处理，得到彩色中间图像；The color processing module is configured to perform first image processing on the high dynamic color image to obtain a color intermediate image;

所述全色处理模块用于对所述高动态全色图像进行第二图像处理，得到全色中间图像；The panchromatic processing module is configured to perform second image processing on the high dynamic panchromatic image to obtain a panchromatic intermediate image;

所述融合模块用于对所述彩色中间图像和所述全色中间图像进行融合算法处理得到所述目标图像。The fusion module is used to perform fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.
根据权利要求3所述的高动态范围图像处理***，其特征在于，所述图像处理器包括彩色处理模块、全色处理模块和融合模块，所述图像处理包括第一图像处理和第二图像处理，所述第一图像处理包括第一图像子处理和第二图像子处理；The high dynamic range image processing system according to claim 3, wherein the image processor includes a color processing module, a panchromatic processing module, and a fusion module, and the image processing includes a first image processing and a second image processing. , The first image processing includes a first image sub-processing and a second image sub-processing;

所述彩色处理模块用于对所述第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对所述第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；The color processing module is configured to perform first image sub-processing on the first color original image to obtain a first color intermediate image, and perform first image sub-processing on the second color original image to obtain a second color intermediate image ；

所述全色处理模块用于对所述第一全色原始图像进行第二图像处理，得到第一全色中间图像，对所述第二全色原始图像进行第二图像处理，得到第二全色中间图像；The panchromatic processing module is configured to perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic image. Color intermediate image

所述彩色高动态融合单元用于将所述第一彩色中间图像和所述第二彩色中间图像融合得到高动态彩色图像；The color high dynamic fusion unit is used for fusing the first color intermediate image and the second color intermediate image to obtain a high dynamic color image;

所述全色高动态融合单元用于将所述第一全色中间图像和所述第二全色中间图像融合得到高动态全色图像；The panchromatic high dynamic fusion unit is used for fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image;

所述彩色处理模块还用于对所述高动态彩色图像进行第二图像子处理，得到彩色中间图像；The color processing module is further configured to perform a second image sub-processing on the high dynamic color image to obtain a color intermediate image;

所述融合模块用于对所述彩色中间图像和所述高动态全色图像进行融合算法处理得到所述目标图像。The fusion module is used to perform fusion algorithm processing on the color intermediate image and the high dynamic panchromatic image to obtain the target image.
根据权利要求3所述的高动态范围图像处理***，其特征在于，所述图像处理器包括彩色处理模块、全色处理模块和融合模块，所述图像处理包括第一图像处理和第二图像处理；The high dynamic range image processing system according to claim 3, wherein the image processor includes a color processing module, a panchromatic processing module, and a fusion module, and the image processing includes a first image processing and a second image processing. ；

所述彩色处理模块用于对所述第一彩色原始图像进行第一图像处理，得到第一彩色中间图像，对所述第二彩色原始图像进行第一图像处理，得到第二彩色中间图像；The color processing module is configured to perform first image processing on the first color original image to obtain a first color intermediate image, and perform first image processing on the second color original image to obtain a second color intermediate image;

所述全色处理模块用于对所述第一全色原始图像进行第二图像处理，得到第一全色中间图像，对所述第二全色原始图像进行第二图像处理，得到第二全色中间图像；The panchromatic processing module is configured to perform second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and perform second image processing on the second panchromatic original image to obtain a second panchromatic image. Color intermediate image

所述彩色高动态融合单元用于将所述第一彩色中间图像和所述第二彩色中间图像融合得到高动态彩色图像；The color high dynamic fusion unit is used for fusing the first color intermediate image and the second color intermediate image to obtain a high dynamic color image;

所述全色高动态融合单元用于将所述第一全色中间图像和所述第二全色中间图像融合得到高动态全色图像；The panchromatic high dynamic fusion unit is used for fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image;

所述融合模块用于对所述高动态彩色图像和所述高动态全色图像进行融合算法处理得到所述目标图像。The fusion module is used to perform fusion algorithm processing on the high dynamic color image and the high dynamic panchromatic image to obtain the target image.
根据权利要求4至6任意一项所述的高动态范围图像处理***，其特征在于，所述第一图像处理包括：The high dynamic range image processing system according to any one of claims 4 to 6, wherein the first image processing comprises:

黑电平矫正处理、镜头阴影矫正处理、去马赛克处理、坏点补偿处理、色彩矫正处理、全局色调映射处理、色彩转换处理中的一个或多个；One or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing;

所述第二图像处理包括：The second image processing includes:

所述黑电平矫正处理、所述镜头阴影矫正处理、所述坏点补偿处理、所述全局色调映射处理中的一个或多个。One or more of the black level correction processing, the lens shading correction processing, the dead pixel compensation processing, and the global tone mapping processing.
根据权利要求4所述的高动态范围图像处理***，其特征在于，所述第三曝光时间等于所述第一曝光时间，所述第四曝光时间等于所述第二曝光时间，所述彩色高动态融合单元用于：The high dynamic range image processing system of claim 4, wherein the third exposure time is equal to the first exposure time, the fourth exposure time is equal to the second exposure time, and the color is high The dynamic fusion unit is used for:

将所述第一彩色原始图像和所述第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的所述第一彩色原始图像，再融合亮度对齐后的所述第一彩色原始图像及所述第二彩色原始图像以得到高动态彩色图像；Perform brightness alignment processing on the first color original image and the second color original image to obtain the first color original image after brightness alignment, and then merge the first color original image after brightness alignment and all the images. Said second color original image to obtain a high dynamic color image;

所述全色高动态融合单元用于：The panchromatic high dynamic fusion unit is used for:

将所述第一全色原始图像和所述第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的所述第一全色原始图像，再融合亮度对齐后的所述第一全色原始图像及所述第二全色原始图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the first panchromatic original image with aligned brightness, and then merge the first panchromatic with the aligned brightness The original image and the second panchromatic original image to obtain a high dynamic panchromatic image.
根据权利要求5或6所述的高动态范围图像处理***，其特征在于，所述第三曝光时间等于所述第一曝光时间，所述第四曝光时间等于所述第二曝光时间，所述彩色高动态融合单元用于：The high dynamic range image processing system according to claim 5 or 6, wherein the third exposure time is equal to the first exposure time, the fourth exposure time is equal to the second exposure time, and the The color high dynamic fusion unit is used for:

将所述第一彩色中间图像和所述第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的所述第一彩色中间图像，再融合亮度对齐后的所述第一彩色中间图像及所述第二彩色中间图像以得到高动态彩色图像；Perform brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain the first color intermediate image after brightness alignment, and then merge the first color intermediate image after brightness alignment and all the images. Said second color intermediate image to obtain a high dynamic color image;

所述全色高动态融合单元用于：The panchromatic high dynamic fusion unit is used for:

将所述第一全色中间图像和所述第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的所述第一全色中间图像，再融合亮度对齐后的所述第一全色中间图像及所述第二全色中间图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image with aligned brightness, and then merge the first panchromatic with aligned brightness The intermediate image and the second panchromatic intermediate image are used to obtain a high dynamic panchromatic image.
根据权利要求2所述的高动态范围图像处理***，其特征在于，所述第三曝光时间等于所述第四曝光时间，所述第三曝光时间大于所述第二曝光时间，并且小于所述第一曝光时间，所述高动态范围图像处理***还包括全色信息融合单元；The high dynamic range image processing system according to claim 2, wherein the third exposure time is equal to the fourth exposure time, and the third exposure time is greater than the second exposure time and less than the For the first exposure time, the high dynamic range image processing system further includes a panchromatic information fusion unit;

所述全色信息融合单元用于：The panchromatic information fusion unit is used for:

将每个所述子单元中以所述第三曝光时间曝光的所述全色感光像素生成的所述第一全色信息和以所述第四曝光时间曝光的所述全色感光像素生成的所述第二全色信息进行像素相加处理或像素求平均处理，以得到所述全色原始图像；或者The first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time and the panchromatic photosensitive pixels exposed at the fourth exposure time in each of the subunits are generated Performing pixel addition processing or pixel averaging processing on the second panchromatic information to obtain the panchromatic original image; or

所述全色原始图像包括第一全色原始图像和第二全色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成所述第一全色信息得到所述第一全色原始图像，以所述第四曝光时间曝光的所述全色感光像素生成所述第二全色信息得到所述第二全色原始图像，所述全色信息融合单元用于：The panchromatic original image includes a first panchromatic original image and a second panchromatic original image. The panchromatic photosensitive pixels exposed at the third exposure time generate the first panchromatic information to obtain the first panchromatic image. Color original image, the panchromatic photosensitive pixels exposed at the fourth exposure time generate the second panchromatic information to obtain the second panchromatic original image, and the panchromatic information fusion unit is configured to:

将所述第一全色原始图像的每个所述子单元中以所述第三曝光时间曝光的所述全色感光像素生成的所述第一全色信息与所述第二全色原始图像的对应所述子单元中以所述第四曝光时间曝光的所述全色感光像素生成的所述第二全色信息进行像素相加处理或像素求平均处理，以得到所述全色原始图像。The first panchromatic information and the second panchromatic original image generated by the panchromatic photosensitive pixels exposed at the third exposure time in each of the subunits of the first panchromatic original image Perform pixel addition processing or pixel averaging processing on the second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time in the subunit to obtain the panchromatic original image .
根据权利要求4至6任意一项所述的高动态范围图像处理***，其特征在于，所述图像处理器还包括：The high dynamic range image processing system according to any one of claims 4 to 6, wherein the image processor further comprises:

接收单元，所述接收单元用于接收所述第一彩色原始图像、所述第二彩色原始图像、所述全色原始图像、所述第一全色原始图像、所述第二全色原始图像、所述高动态彩色图像和所述高动态全色图像中的一个或多个；和A receiving unit, the receiving unit is configured to receive the first color original image, the second color original image, the full color original image, the first full color original image, and the second full color original image , One or more of the high dynamic color image and the high dynamic full color image; and

内存单元，所述内存单元用于暂存所述第一彩色原始图像、所述第二彩色原始图像、所述全色原始图像、所述第一全色原始图像、所述第二全色原始图像、所述高动态彩色图像、所述高动态全色图像和所述全色中间图像中的一个或多个。A memory unit for temporarily storing the first color original image, the second color original image, the panchromatic original image, the first panchromatic original image, and the second panchromatic original image One or more of the image, the high dynamic color image, the high dynamic panchromatic image, and the panchromatic intermediate image.
根据权利要求3所述的高动态范围图像处理***，其特征在于，所述彩色高动态融合单元和所述全色高动态融合单元集成在所述图像传感器中；或所述彩色高动态融合单元和所述全色高动态融合单元集成在所述图像处理器中。The high dynamic range image processing system according to claim 3, wherein the color high dynamic fusion unit and the panchromatic high dynamic fusion unit are integrated in the image sensor; or the color high dynamic fusion unit And the panchromatic high dynamic fusion unit are integrated in the image processor.
一种高动态范围图像处理方法，用于高动态范围图像处理***，其特征在于，所述高动态范围图像处理***包括图像传感器，所述图像传感器包括像素阵列，所述像素阵列包括多个全色感光像素和多个彩色感光像素，所述彩色感光像素具有比所述全色感光像素更窄的光谱响应，所述像素阵列包括最小重复单元，每个所述最小重复单元包含多个子单元，每个所述子单元包括多个单颜色感光像素及多个全色感光像素；所述高动态范围图像处理方法包括：A high dynamic range image processing method for a high dynamic range image processing system, wherein the high dynamic range image processing system includes an image sensor, the image sensor includes a pixel array, and the pixel array includes a plurality of pixels. A color photosensitive pixel and a plurality of color photosensitive pixels, the color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel, the pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of subunits, Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels; the high dynamic range image processing method includes:

控制所述像素阵列曝光，其中，对于同一所述子单元中的多个感光像素，至少一个所述单颜色感光像素以第一曝光时间曝光，至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光，至少一个所述全色感光像素以小于或等于所述第一曝光时间的第三曝光时间曝光，至少一个所述全色感光像素以小于所述第一曝光时间的第四曝光时间曝光，其中，以所述第一曝光时间曝光的所述单颜色感光像素生成第一彩色信息得到第一彩色原始图像，以所述第二曝光时间曝光的所述单颜色感光像素生成第二彩色信息得到第二彩色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成第一全色信息、以所述第四曝光时间曝光的所述全色感光像素生成第二全色信息，以得到全色原始图像；和The pixel array exposure is controlled, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed for a first exposure time, and at least one single-color photosensitive pixel is less than the first exposure time. Exposure at a second exposure time of one exposure time, at least one of the panchromatic photosensitive pixels is exposed at a third exposure time less than or equal to the first exposure time, and at least one of the panchromatic photosensitive pixels is exposed at a third exposure time less than or equal to the first exposure time Exposure at the fourth exposure time of time, wherein the single color photosensitive pixel exposed at the first exposure time generates first color information to obtain a first color original image, and the single color exposed at the second exposure time The photosensitive pixels generate second color information to obtain a second color original image, the panchromatic photosensitive pixels exposed at the third exposure time generate first panchromatic information, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate first panchromatic information. The pixels generate second panchromatic information to obtain a panchromatic original image; and

对所述第一彩色原始图像、所述第二彩色原始图像和所述全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。High dynamic range processing, image processing, and fusion algorithm processing are performed on the first color original image, the second color original image, and the panchromatic original image to obtain a target image.
根据权利要求13所述的高动态范围图像处理方法，其特征在于，所述全色原始图像包括第一全色原始图像和第二全色原始图像；以所述第三曝光时间曝光的所述全色感光像素生成所述第一全色信息得到所述第一全色原始图像，以所述第四曝光时间曝光的所述全色感光像素生成所述第二全色信息得到所述第二全色原始图像；所述对所述第一彩色原始图像、所述第二彩色原始图像和所述全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：The high dynamic range image processing method according to claim 13, wherein the panchromatic original image comprises a first panchromatic original image and a second panchromatic original image; the panchromatic original image exposed at the third exposure time Panchromatic photosensitive pixels generate the first panchromatic information to obtain the first panchromatic original image, and the panchromatic photosensitive pixels exposed at the fourth exposure time generate the second panchromatic information to obtain the second panchromatic image. Full-color original image; said performing high dynamic range processing, image processing, and fusion algorithm processing on the first color original image, the second color original image, and the full-color original image to obtain a target image includes:

对所述第一彩色原始图像、所述第二彩色原始图像、所述第一全色原始图像和所述第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像。High dynamic range processing, image processing, and fusion algorithm processing are performed on the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image to obtain a target image.
根据权利要求14所述的高动态范围图像处理方法，其特征在于，所述图像处理包括第一图像处理和第二图像处理，所述对所述第一彩色原始图像、所述第二彩色原始图像、所述第一全色原始图像和所述第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：The high dynamic range image processing method according to claim 14, wherein the image processing includes a first image processing and a second image processing, and the processing of the first color original image and the second color original image The image, the first full-color original image, and the second full-color original image to obtain a target image through high dynamic range processing, image processing, and fusion algorithm processing include:

将所述第一彩色原始图像和所述第二彩色原始图像融合得到高动态彩色图像；Fusing the first color original image and the second color original image to obtain a highly dynamic color image;

将所述第一全色原始图像和所述第二全色原始图像融合得到高动态全色图像；Fusing the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image;

对所述高动态彩色图像进行第一图像处理，得到彩色中间图像；Performing first image processing on the high dynamic color image to obtain a color intermediate image;

对所述高动态全色图像进行第二图像处理，得到全色中间图像；和Performing second image processing on the high dynamic panchromatic image to obtain a panchromatic intermediate image; and

对所述彩色中间图像和所述全色中间图像进行融合算法处理得到所述目标图像。Performing fusion algorithm processing on the color intermediate image and the panchromatic intermediate image to obtain the target image.
根据权利要求14所述的高动态范围图像处理方法，其特征在于，所述图像处理包括第一图像处理和第二图像处理，所述第一图像处理包括第一图像子处理和第二图像子处理；所述对所述第一彩色原始图像、所述第二彩色原始图像、所述第一全色原始图像和所述第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：The high dynamic range image processing method according to claim 14, wherein the image processing includes a first image processing and a second image processing, and the first image processing includes a first image sub-processing and a second image sub-processing. Processing; the first color original image, the second color original image, the first full-color original image, and the second full-color original image for high dynamic range processing, image processing and fusion algorithm processing Obtaining the target image includes:

对所述第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对所述第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；Performing a first image sub-processing on the first color original image to obtain a first color intermediate image, and performing a first image sub-processing on the second color original image to obtain a second color intermediate image;

对所述第一全色原始图像进行第二图像处理，得到第一全色中间图像，对所述第二全色原始图像进行第二图像处理，得到第二全色中间图像；Performing second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and performing second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image;

将所述第一彩色中间图像和所述第二彩色中间图像融合得到高动态彩色图像；Fusing the first color intermediate image and the second color intermediate image to obtain a highly dynamic color image;

将所述第一全色中间图像和所述第二全色中间图像融合得到高动态全色图像；Fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image;

对所述高动态彩色图像进行第二图像子处理，得到彩色中间图像；和Performing a second image sub-processing on the high dynamic color image to obtain a color intermediate image; and

对所述彩色中间图像和所述高动态全色图像进行融合算法处理得到所述目标图像。Performing fusion algorithm processing on the color intermediate image and the high dynamic panchromatic image to obtain the target image.
根据权利要求14所述的高动态范围图像处理方法，其特征在于，所述图像处理包括第一图像处理和第二图像处理；所述对所述第一彩色原始图像、所述第二彩色原始图像、所述第一全色原始图像和所述第二全色原始图像进行高动态范围处理、图像处理和融合算法处理得到目标图像包括：The high dynamic range image processing method according to claim 14, wherein the image processing includes first image processing and second image processing; The image, the first full-color original image, and the second full-color original image to obtain a target image through high dynamic range processing, image processing, and fusion algorithm processing include:

对所述第一彩色原始图像进行第一图像子处理，得到第一彩色中间图像，对所述第二彩色原始图像进行第一图像子处理，得到第二彩色中间图像；Performing a first image sub-processing on the first color original image to obtain a first color intermediate image, and performing a first image sub-processing on the second color original image to obtain a second color intermediate image;

对所述第一全色原始图像进行第二图像处理，得到第一全色中间图像，对所述第二全色原始图像进行第二图像处理，得到第二全色中间图像；Performing second image processing on the first panchromatic original image to obtain a first panchromatic intermediate image, and performing second image processing on the second panchromatic original image to obtain a second panchromatic intermediate image;

将所述第一彩色中间图像和所述第二彩色中间图像融合得到高动态彩色图像；Fusing the first color intermediate image and the second color intermediate image to obtain a highly dynamic color image;

将所述第一全色中间图像和所述第二全色中间图像融合得到高动态全色图像；和Fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image; and

对所述高动态彩色图像和所述高动态全色图像进行融合算法处理得到所述目标图像。Performing fusion algorithm processing on the high dynamic color image and the high dynamic panchromatic image to obtain the target image.
根据权利要求15至17任意一项所述的高动态范围图像处理方法，其特征在于，所述第一图像处理包括：The high dynamic range image processing method according to any one of claims 15 to 17, wherein the first image processing comprises:

黑电平矫正处理、镜头阴影矫正处理、去马赛克处理、坏点补偿处理、色彩矫正处理、全局色调映射处理、色彩转换处理中的一个或多个；One or more of black level correction processing, lens shading correction processing, demosaicing processing, dead pixel compensation processing, color correction processing, global tone mapping processing, and color conversion processing;

所述第二图像处理包括：The second image processing includes:

所述黑电平矫正处理、所述镜头阴影矫正处理、所述坏点补偿处理、所述全局色调映射处理中的一个或多个。One or more of the black level correction processing, the lens shading correction processing, the dead pixel compensation processing, and the global tone mapping processing.
根据权利要求15所述的高动态范围图像处理方法，其特征在于，所述第三曝光时间等于所述第一曝光时间，所述第四曝光时间等于所述第二曝光时间；所述将所述第一彩色原始图像和所述第二彩色原始图像融合得到高动态彩色图像包括：The high dynamic range image processing method according to claim 15, wherein the third exposure time is equal to the first exposure time, the fourth exposure time is equal to the second exposure time; The fusion of the first color original image and the second color original image to obtain a high dynamic color image includes:

将所述第一彩色原始图像和所述第二彩色原始图像进行亮度对齐处理，以得到亮度对齐后的所述第一彩色原始图像，再融合亮度对齐后的所述第一彩色原始图像及所述第二彩色原始图像以得到高动态彩色图像；Perform brightness alignment processing on the first color original image and the second color original image to obtain the first color original image after brightness alignment, and then merge the first color original image after brightness alignment and all the images. Said second color original image to obtain a high dynamic color image;

所述将所述第一全色原始图像和所述第二全色原始图像融合得到高动态全色图像包括：The fusing the first panchromatic original image and the second panchromatic original image to obtain a high dynamic panchromatic image includes:

将所述第一全色原始图像和所述第二全色原始图像进行亮度对齐处理，以得到亮度对齐后的所述第一全色原始图像，再融合亮度对齐后的所述第一全色原始图像及所述第二全色原始图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic original image and the second panchromatic original image to obtain the first panchromatic original image with aligned brightness, and then merge the first panchromatic with the aligned brightness The original image and the second panchromatic original image to obtain a high dynamic panchromatic image.
根据权利要求16或17所述的高动态范围图像处理方法，其特征在于，所述第三曝光时间等于所述第一曝光时间，所述第四曝光时间等于所述第二曝光时间；所述将所述第一彩色中间图像和所述第二彩色中间图像融合得到高动态彩色图像包括：The high dynamic range image processing method according to claim 16 or 17, wherein the third exposure time is equal to the first exposure time, and the fourth exposure time is equal to the second exposure time; The fusion of the first color intermediate image and the second color intermediate image to obtain a high dynamic color image includes:

将所述第一彩色中间图像和所述第二彩色中间图像进行亮度对齐处理，以得到亮度对齐后的所述第一彩色中间图像，再融合亮度对齐后的所述第一彩色中间图像及所述第二彩色中间图像以得到高动态彩色图像；Perform brightness alignment processing on the first color intermediate image and the second color intermediate image to obtain the first color intermediate image after brightness alignment, and then merge the first color intermediate image after brightness alignment and all the images. Said second color intermediate image to obtain a high dynamic color image;

所述将所述第一全色中间图像和所述第二全色中间图像融合得到高动态全色图像包括：The fusing the first panchromatic intermediate image and the second panchromatic intermediate image to obtain a high dynamic panchromatic image includes:

将所述第一全色中间图像和所述第二全色中间图像进行亮度对齐处理，以得到亮度对齐后的所述第一全色中间图像，再融合亮度对齐后的所述第一全色中间图像及所述第二全色中间图像以得到高动态全色图像。Perform brightness alignment processing on the first panchromatic intermediate image and the second panchromatic intermediate image to obtain the first panchromatic intermediate image with aligned brightness, and then merge the first panchromatic with aligned brightness The intermediate image and the second panchromatic intermediate image are used to obtain a high dynamic panchromatic image.
根据权利要求13所述的高动态范围图像处理方法，其特征在于，所述第三曝光时间等于所述第四曝光时间，所述第三曝光时间大于所述第二曝光时间，并且小于所述第一曝光时间，所述高动态范围图像处理方法还包括：The high dynamic range image processing method according to claim 13, wherein the third exposure time is equal to the fourth exposure time, and the third exposure time is greater than the second exposure time and less than the The first exposure time, the high dynamic range image processing method further includes:

将每个所述子单元中以所述第三曝光时间曝光的所述全色感光像素生成的所述第一全色信息和以所述第四曝光时间曝光的所述全色感光像素生成的所述第二全色信息进行像素相加处理或像素求平均处理，以得到所述全色原始图像；或者The first panchromatic information generated by the panchromatic photosensitive pixels exposed at the third exposure time and the panchromatic photosensitive pixels exposed at the fourth exposure time in each of the subunits are generated Performing pixel addition processing or pixel averaging processing on the second panchromatic information to obtain the panchromatic original image; or

所述全色原始图像包括第一全色原始图像和第二全色原始图像，以所述第三曝光时间曝光的所述全色感光像素生成所述第一全色信息得到所述第一全色原始图像，以所述第四曝光时间曝光的所述全色感光像素生成所述第二全色信息得到所述第二全色原始图像，所述高动态范围图像处理方法还包括：The panchromatic original image includes a first panchromatic original image and a second panchromatic original image. The panchromatic photosensitive pixels exposed at the third exposure time generate the first panchromatic information to obtain the first panchromatic image. Color original image, the panchromatic photosensitive pixels exposed at the fourth exposure time generate the second panchromatic information to obtain the second panchromatic original image, and the high dynamic range image processing method further includes:

将所述第一全色原始图像的每个所述子单元中以所述第三曝光时间曝光的所述全色感光像素生成的所述第一全色信息与所述第二全色原始图像的对应所述子单元中以所述第四曝光时间曝光的所述全色感光像素生成的所述第二全色信息进行像素相加处理或像素求平均处理，以得到所述全色原始图像。The first panchromatic information and the second panchromatic original image generated by the panchromatic photosensitive pixels exposed at the third exposure time in each of the subunits of the first panchromatic original image Perform pixel addition processing or pixel averaging processing on the second panchromatic information generated by the panchromatic photosensitive pixels exposed at the fourth exposure time in the subunit to obtain the panchromatic original image .
根据权利要求15至17任意一项所述的高动态范围图像处理方法，其特征在于，所述高动态范围图像处理方法还包括：The high dynamic range image processing method according to any one of claims 15 to 17, wherein the high dynamic range image processing method further comprises:

接收所述第一彩色原始图像、所述第二彩色原始图像、所述全色原始图像、所述第一全色原始图像、所述第二全色原始图像、所述高动态彩色图像和所述高动态全色图像中的一个或多个；和Receiving the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, the high dynamic color image, and the One or more of the high dynamic panchromatic images; and

暂存所述第一彩色原始图像、所述第二彩色原始图像、所述全色原始图像、所述第一全色原始图像、所述第二全色原始图像、所述高动态彩色图像、所述高动态全色图像、所述彩色中间图像和所述全色中间图像中的一个或多个。Temporarily store the first color original image, the second color original image, the full color original image, the first full color original image, the second full color original image, the high dynamic color image, One or more of the high dynamic panchromatic image, the color intermediate image, and the panchromatic intermediate image.
一种电子设备，其特征在于，包括：An electronic device, characterized in that it comprises:

镜头；Lens

壳体；及Shell; and

权利要求2至12任意一项所述的高动态范围图像处理***，所述镜头、所述高动态范围图像处理***与所述壳体结合，所述镜头与所述高动态范围图像处理***的图像传感器配合成像。The high dynamic range image processing system according to any one of claims 2 to 12, the lens, the high dynamic range image processing system and the housing are combined, and the lens and the high dynamic range image processing system The image sensor cooperates with imaging.
一种包含计算机程序的非易失性计算机可读存储介质，其特征在于，所述计算机程序被处理器执行时，使得所述处理器执行权利要求13至22任意一项所述的高动态范围图像处理方法。A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the high dynamic range of any one of claims 13 to 22 Image processing method.