WO2021196553A1 - 高动态范围图像处理***及方法、电子设备和可读存储介质 - Google Patents
高动态范围图像处理***及方法、电子设备和可读存储介质 Download PDFInfo
- Publication number
- WO2021196553A1 WO2021196553A1 PCT/CN2020/119959 CN2020119959W WO2021196553A1 WO 2021196553 A1 WO2021196553 A1 WO 2021196553A1 CN 2020119959 W CN2020119959 W CN 2020119959W WO 2021196553 A1 WO2021196553 A1 WO 2021196553A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- high dynamic
- color
- original image
- dynamic range
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/88—Camera processing pipelines; Components thereof for processing colour signals for colour balance, e.g. white-balance circuits or colour temperature control
Definitions
- This application relates to the field of image processing technology, and in particular to a high dynamic range image processing system, a high dynamic range image processing method, electronic equipment, and a non-volatile computer-readable storage medium.
- a camera may be provided in an electronic device such as a mobile phone to realize a photographing function.
- An image sensor for receiving light can be set in the camera.
- the image sensor may be provided with a filter array.
- the embodiments of the present application provide a high dynamic range image processing system, a high dynamic range image processing method, electronic equipment, and a non-volatile computer-readable storage medium.
- the embodiment of the present application provides a high dynamic range image processing system.
- the high dynamic range image processing system includes an image sensor, an image fusion module and a high dynamic range image processing module.
- the image sensor includes a pixel array, the pixel array includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels, and the pixel array includes The smallest repeating unit, each of the smallest repeating units includes a plurality of sub-units, each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels, and the pixel array in the image sensor is exposed to light.
- the single-color photosensitive pixels is exposed for a first exposure time, and at least one of the single-color photosensitive pixels is exposed for a second exposure that is less than the first exposure time Time exposure, at least one of the full-color photosensitive pixels is exposed at a third exposure time that is less than the first exposure time.
- the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains a first color original image
- the second color generated by the single-color photosensitive pixel exposed at the second exposure time The information obtains a second color original image
- the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image.
- the image fusion module and the high dynamic range image processing module are used to perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image To get the first high dynamic range image.
- the first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array.
- the first high dynamic range image is processed by an image processor to obtain a second high dynamic range image.
- the embodiments of the present application provide a high dynamic range image processing method.
- the high dynamic range image processing method is used in a high dynamic range image processing system.
- the high dynamic range image processing system includes an image sensor, the image sensor includes a pixel array, the pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels, and the color photosensitive pixels have a higher sensitivity than the full-color photosensitive pixels.
- the pixel has a narrower spectral response.
- the pixel array includes a minimum repeating unit, each of the minimum repeating units includes a plurality of sub-units, and each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels.
- the high dynamic range image processing method includes: controlling the exposure of the pixel array, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed at a first exposure time, and at least one The single-color photosensitive pixel is exposed at a second exposure time that is less than the first exposure time, and at least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time.
- the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains a first color original image
- the second color generated by the single-color photosensitive pixel exposed at the second exposure time The information obtains a second color original image
- the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image.
- performing fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain a first high dynamic range image.
- the first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array.
- the first high dynamic range image is processed by an image processor to obtain a second high dynamic range image.
- the embodiment of the present application provides an electronic device.
- the electronic device includes a lens, a housing, and the above-mentioned high dynamic range image processing system.
- the lens and the high dynamic range image processing system are combined with the housing, and the lens cooperates with the image sensor of the high dynamic range image processing system for imaging.
- the embodiments of the present application provide a non-volatile computer-readable storage medium containing a computer program.
- the processor is caused to execute the above-mentioned high dynamic range image processing method.
- FIG. 1 is a schematic diagram of a high dynamic range image processing system according to an embodiment of the present application
- FIG. 2 is a schematic diagram of a pixel array according to an embodiment of the present application.
- FIG. 3 is a schematic cross-sectional view of a photosensitive pixel according to an embodiment of the present application.
- FIG. 4 is a pixel circuit diagram of a photosensitive pixel according to an embodiment of the present application.
- FIG. 5 is a schematic diagram of the arrangement of the smallest repeating unit in a pixel array according to an embodiment of the present application
- FIG. 6 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application.
- FIG. 7 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application.
- FIG. 8 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application.
- FIG. 9 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application.
- FIG. 10 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application.
- FIG. 11 is a schematic diagram of an original image output by an image sensor according to an embodiment of the present application.
- FIG. 12 is a schematic diagram of an image fusion processing principle according to an embodiment of the present application.
- FIG. 13 is a schematic diagram of another image fusion processing principle according to an embodiment of the present application.
- FIG. 14 is a schematic diagram of brightness alignment processing according to an embodiment of the present application.
- FIG. 15 is a schematic diagram of a high dynamic range processing principle according to an embodiment of the present application.
- 16 is a schematic diagram of another high dynamic range image processing system according to an embodiment of the present application.
- FIG. 17 is a schematic diagram of lens shading correction processing according to an embodiment of the present application.
- FIG. 18 is a schematic diagram of another high dynamic range image processing system according to an embodiment of the present application.
- FIG. 19 is a schematic diagram of another high dynamic range processing principle according to an embodiment of the present application.
- FIG. 20 is a schematic diagram of an original image output by another image sensor according to an embodiment of the present application.
- FIG. 21 is a schematic diagram of another high dynamic range processing principle according to an embodiment of the present application.
- FIG. 22 is a schematic diagram of another image fusion processing principle according to an embodiment of the present application.
- FIG. 23 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
- FIG. 24 is a schematic flowchart of a method for acquiring a high dynamic range image according to an embodiment of the present application.
- FIG. 25 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor according to an embodiment of the present application.
- the high dynamic range image processing system 100 includes an image sensor 10, an image fusion module 20 and a high dynamic range image processing module 30.
- the image sensor 10 includes a pixel array 11.
- the pixel array 11 includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, and the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels.
- the pixel array 11 includes a minimum repeating unit, each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels.
- the pixel array 11 in the image sensor 10 is exposed to light, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed at a first exposure time, and at least one single-color photosensitive pixel is exposed at a first exposure time that is less than the first exposure time. Exposure at a second exposure time, at least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time.
- the first color information generated by the single-color photosensitive pixels exposed at the first exposure time obtains the first color original image
- the second color information generated by the single-color photosensitive pixels exposed at the second exposure time obtains the second color original image
- the full-color photosensitive pixels exposed at the third exposure time generate a first full-color original image.
- the image fusion module 20 and the high dynamic range image processing module 30 are used to perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image .
- the first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array.
- the first high dynamic range image is processed by the image processor 40 to obtain a second high dynamic range image.
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time, and the fourth exposure time is less than or equal to the first exposure time.
- the exposure time is greater than the third exposure time.
- the image fusion module 20 is used for fusing the first color original image and the second panchromatic original image into a first intermediate image, and fusing the second color original image and the first panchromatic original image into a second intermediate image, with a fourth exposure
- the second panchromatic information generated by the time-exposed single-color photosensitive pixels obtains the second panchromatic original image.
- the high dynamic range image processing module 30 is used to fuse the first intermediate image and the second intermediate image into a first high dynamic range image.
- the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31 and a brightness mapping unit 33.
- the high dynamic range image processing unit 31 is used for fusing the first intermediate image and the second intermediate image into a third high dynamic range image.
- the brightness mapping unit 33 is configured to perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image.
- the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistical unit 35.
- the high dynamic range image processing unit 31 is used for fusing the first intermediate image and the second intermediate image into a third high dynamic range image;
- the lens shading correction unit 37 is used for correcting the third high dynamic range image to obtain a high dynamic range corrected image;
- the statistical unit 35 is used to process the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- the high dynamic range image processing module 30 includes a statistical unit 35, the statistical unit 35 is used to process the first intermediate image and the second intermediate image to obtain statistical data, the statistical data is provided to the image processing
- the device 40 performs automatic exposure processing and/or automatic white balance processing.
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time, and the fourth exposure time is less than or equal to the first exposure time.
- the exposure time is greater than the third exposure time.
- the high dynamic range image processing module 30 is used for fusing the first color original image and the second color original image into a first high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into a first high Dynamic panchromatic original image.
- the image fusion module 20 is used for fusing the first high dynamic color original image and the first high dynamic full color original image into a first high dynamic range image.
- the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31 and a brightness mapping unit 33.
- the high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into the second high Dynamic full-color original image;
- the brightness mapping unit 33 is used to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image, and perform brightness mapping on the second high dynamic full color original image to obtain the first high Dynamic panchromatic original image.
- the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistics unit 35.
- the high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into the second high Dynamic panchromatic original image;
- the lens shading correction unit 37 is used to correct the second highly dynamic color original image to obtain a highly dynamic color corrected image, and correct the second highly dynamic panchromatic original image to obtain a highly dynamic panchromatic corrected image;
- the statistical unit 35 It is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- all panchromatic photosensitive pixels in the same subunit are exposed at the third exposure time; the high dynamic range image processing module 30 is used to combine the first color original image with the second color original image Fusion into a first high dynamic color original image; the image fusion module 20 is used for fusing the first high dynamic color original image and the first panchromatic original image into a first high dynamic range image.
- the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31 and a brightness mapping unit 33.
- the high dynamic range image processing unit 31 is used to fuse the first color original image and the second color original image into a second high dynamic color original image;
- the brightness mapping unit 33 is used to perform brightness mapping on the second high dynamic color original image to obtain The first high-dynamic color original image.
- the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistics unit 35.
- the high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image.
- the lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image.
- the statistical unit 35 is used to process the high dynamic color correction image and the first full-color original image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- both the image fusion module 20 and the high dynamic range image processing module 30 are integrated in the image sensor.
- the high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100.
- the high dynamic range image processing system 100 includes an image sensor 10.
- the image sensor 10 includes a pixel array 11.
- the pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels.
- the pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels.
- High dynamic range image processing methods include:
- the first high dynamic range image includes multiple color images Pixels, a plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by the image processor to obtain the second high dynamic range image.
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image; for the first color original image, the second color original image and the first panchromatic original
- Performing fusion algorithm processing and high dynamic range processing on the image to obtain the first high dynamic range image includes: fusing the first color original image and the second panchromatic original image into a first intermediate image, and combining the second color original image with the first full color original image.
- the color original image is fused into a second intermediate image; and the first intermediate image and the second intermediate image are fused into a first high dynamic range image.
- fusing the first intermediate image and the second intermediate image into a first high dynamic range image includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; and The three high dynamic range images are subjected to brightness mapping to obtain the first high dynamic range image.
- the high dynamic range image processing method further includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; obtaining a high dynamic range corrected image from the third high dynamic range image; And processing the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
- the high dynamic range image processing method further includes: processing the first intermediate image and the second intermediate image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing .
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image.
- Performing fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image includes: combining the first color original image and the second color original image Fusion into a first high-dynamic color original image, fusing the first full-color original image and the second full-color original image into a first high-dynamic full-color original image; and combining the first high-dynamic color original image with the first high-dynamic full-color original image The color original image is fused into the first high dynamic range image.
- the first color original image and the second color original image are fused into a first high dynamic color original image
- the first panchromatic original image and the second panchromatic original image are fused into a first high dynamic color original image.
- the color original image includes: fusing the first color original image and the second color original image into a second high-dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into a second high-dynamic panchromatic original image Image; and brightness mapping is performed on the second high dynamic color original image to obtain the first high dynamic color original image, and brightness mapping is performed on the second high dynamic panchromatic original image to obtain the first high dynamic full color original image.
- the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image, and combining the first panchromatic original image with the second panchromatic original image.
- the color correction image and the high dynamic full color correction image are used to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- all panchromatic photosensitive pixels in the same subunit are exposed at the third exposure time; the first color original image, the second color original image, and the first panchromatic original image are processed by a fusion algorithm and are highly dynamic
- the range processing to obtain the first high dynamic range image includes: fusing the first color original image and the second color original image into a first high dynamic color original image; and combining the first high dynamic color original image and the first panchromatic original image Fusion is the first high dynamic range image.
- fusing the first color original image and the second color original image into a first high dynamic color original image includes: fusing the first color original image and the second color original image into a second high dynamic color original image And performing brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image.
- the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image; correcting the second high dynamic color original image to obtain a high dynamic color Correcting the image; and processing the high dynamic color correction image and the first full-color original image to obtain statistical data, which is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- the present application also provides an electronic device 1000.
- the electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 described in any one of the above embodiments.
- This application also provides a non-volatile computer-readable storage medium 400 containing a computer program.
- the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.
- the high dynamic range image processing system 100 of the embodiment of the present application performs fusion algorithm processing and high dynamic range processing on the panchromatic original image and color original image output by the image sensor 10 through the image fusion module 20 and the high dynamic range image processing module 30.
- the first high dynamic range image is input into the image processor to complete subsequent processing, thereby solving the problem that the image processor 40 cannot directly arrange the image pixels in a non-Bayer array. The problem of processing cloth images.
- FIG. 2 is a schematic diagram of the image sensor 10 in the embodiment of the present application.
- the image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.
- the image sensor 10 may adopt a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.
- CMOS complementary metal oxide semiconductor
- CCD Charge-coupled Device
- the pixel array 11 includes a plurality of photosensitive pixels 110 (shown in FIG. 3) arranged two-dimensionally in an array (ie, arranged in a two-dimensional matrix), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 (shown in FIG. 4) .
- Each photosensitive pixel 110 converts light into electric charge according to the intensity of light incident thereon.
- the vertical driving unit 12 includes a shift register and an address decoder.
- the vertical drive unit 12 includes readout scanning and reset scanning functions.
- the readout scan refers to sequentially scanning the unit photosensitive pixels 110 line by line, and reading signals from these unit photosensitive pixels 110 line by line.
- the signal output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row is transmitted to the column processing unit 14.
- the reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element is discarded, so that the accumulation of new photocharge can be started.
- the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing.
- CDS correlated double sampling
- the reset level and the signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated.
- the signals of the photosensitive pixels 110 in a row are obtained.
- the column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into a digital format.
- A/D analog-to-digital
- the horizontal driving unit 15 includes a shift register and an address decoder.
- the horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14, and is sequentially output.
- control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14 and the horizontal driving unit 15 to work together.
- FIG. 3 is a schematic diagram of a photosensitive pixel 110 in an embodiment of the present application.
- the photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a micro lens 113. Along the light-receiving direction of the photosensitive pixel 110, the microlens 113, the filter 112, and the pixel circuit 111 are arranged in sequence.
- the microlens 113 is used for condensing light
- the filter 112 is used for passing light of a certain waveband and filtering out the light of other wavebands.
- the pixel circuit 111 is used to convert the received light into electrical signals, and provide the generated electrical signals to the column processing unit 14 shown in FIG. 2.
- FIG. 4 is a schematic diagram of a pixel circuit 111 of a photosensitive pixel 110 in an embodiment of the present application.
- the pixel circuit 111 in FIG. 4 can be applied to each photosensitive pixel 110 (shown in FIG. 3) in the pixel array 11 shown in FIG.
- the working principle of the pixel circuit 111 will be described below with reference to FIGS. 2 to 4.
- the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), and an amplification circuit (for example, an amplification transistor 1114). ) And a selection circuit (for example, a selection transistor 1115).
- the transfer transistor 1112, the reset transistor 1113, the amplifying transistor 1114, and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.
- the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to the ground, for example.
- the photodiode converts the received light into electric charge.
- the cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (for example, a transfer transistor 1112).
- the floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.
- the exposure control circuit is a transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112.
- the transfer transistor 1112 When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line, the transfer transistor 1112 is turned on.
- the transfer transistor 1112 transfers the charge photoelectrically converted by the photodiode to the floating diffusion unit FD.
- the drain of the reset transistor 1113 is connected to the pixel power supply VPIX.
- the source of the reset transistor 113 is connected to the floating diffusion unit FD.
- a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on.
- the reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.
- the gate of the amplifying transistor 1114 is connected to the floating diffusion unit FD.
- the drain of the amplifying transistor 1114 is connected to the pixel power supply VPIX.
- the amplifying transistor 1114 After the floating diffusion unit FD is reset by the reset transistor 1113, the amplifying transistor 1114 outputs the reset level through the output terminal OUT via the selection transistor 1115. After the charge of the photodiode is transferred by the transfer transistor 1112, the amplifying transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.
- the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114.
- the source of the selection transistor 1115 is connected to the column processing unit 14 in FIG. 2 through the output terminal OUT.
- the selection transistor 1115 is turned on.
- the signal output by the amplifying transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.
- the pixel structure of the pixel circuit 111 in the embodiment of the present application is not limited to the structure shown in FIG. 4.
- the pixel circuit 111 may also have a three-transistor pixel structure, in which the functions of the amplifying transistor 1114 and the selecting transistor 1115 are performed by one transistor.
- the exposure control circuit is not limited to the way of a single transfer transistor 1112, and other electronic devices or structures with the function of controlling the conduction of the control terminal can be used as the exposure control circuit in the embodiment of the present application.
- the implementation of the transistor 1112 is simple, low in cost, and easy to control.
- 5 to 10 are schematic diagrams of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the pixel array 11 (shown in FIG. 2) according to some embodiments of the present application.
- the photosensitive pixels 110 include two types, one is a full-color photosensitive pixel W, and the other is a color photosensitive pixel.
- 5 to 10 only show the arrangement of a plurality of photosensitive pixels 110 in a minimum repeating unit. The smallest repeating unit shown in FIGS. 5 to 10 is copied multiple times in rows and columns to form the pixel array 11. Each minimum repeating unit is composed of multiple full-color photosensitive pixels W and multiple color photosensitive pixels. Each minimum repeating unit includes multiple subunits.
- Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels W.
- the full-color photosensitive pixel W and the color photosensitive pixel in each sub-unit are alternately arranged.
- multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category; or, multiple photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category 110.
- FIG. 5 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit of an embodiment of the application.
- the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110
- the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.
- the categories of subunits include three categories.
- the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A
- the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B
- the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C.
- Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC.
- a first type subunit UA and a third type subunit UC are arranged in the first diagonal direction D1 (for example, the direction connecting the upper left corner and the lower right corner in FIG. 5), and two second type subunits UB are arranged In the second diagonal direction D2 (for example, the direction where the upper right corner and the lower left corner are connected in FIG. 5).
- the first diagonal direction D1 is different from the second diagonal direction D2.
- the first diagonal line and the second diagonal line are perpendicular.
- first diagonal direction D1 may also be a direction connecting the upper right corner and the lower left corner
- second diagonal direction D2 may also be a direction connecting the upper left corner and the lower right corner
- direction here is not a single direction, but can be understood as the concept of a "straight line” indicating the arrangement, and there may be two-way directions at both ends of the straight line.
- the explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIGS. 6 to 10 is the same as here.
- FIG. 6 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in a minimum repeating unit according to another embodiment of the application.
- the smallest repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-units are 9 photosensitive pixels 110 in 3 rows and 3 columns.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.
- the categories of subunits include three categories.
- the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A
- the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B
- the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C.
- Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC.
- one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1
- two second type subunits UB are arranged in the second diagonal direction D2.
- the first diagonal direction D1 is different from the second diagonal direction D2.
- the first diagonal line and the second diagonal line are perpendicular.
- FIG. 7 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application.
- the minimum repeating unit is 8 rows and 8 columns and 64 photosensitive pixels 110
- the sub-units are 4 rows and 4 columns and 16 photosensitive pixels 110.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.
- the categories of subunits include three categories.
- the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A
- the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B
- the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C.
- Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC.
- one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1
- two second type subunits UB are arranged in the second diagonal direction D2.
- the first diagonal direction D1 is different from the second diagonal direction D2.
- the first diagonal line and the second diagonal line are perpendicular.
- FIG. 8 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application.
- the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. 8 is roughly the same as the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG.
- the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the subunit UB is inconsistent with the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the second type of subunit UB in the lower left corner of FIG. 5, and ,
- the alternating sequence of the full-color photosensitive pixel W and the single-color photosensitive pixel in the third type subunit UC in FIG. 8 is the same as the full-color photosensitive pixel W and the single-color photosensitive pixel W in the third type subunit UC in the lower right corner of FIG.
- the alternating sequence of photosensitive pixels is also inconsistent. Specifically, in the second type subunit UB in the lower left corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (ie, second-color photosensitive pixels B), and The alternating sequence of the two rows of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B) and full-color photosensitive pixels W; and in the second-type subunit UB in the lower left corner of FIG.
- the first row The alternating sequence of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (ie The second color photosensitive pixel B).
- the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels C), and the second row
- the alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, a third-color photosensitive pixel C) and a full-color photosensitive pixel W; and in the third type subunit UC in the lower right corner of FIG.
- the photosensitive pixels 110 in the first row The alternating sequence of the single-color photosensitive pixel (ie the third color photosensitive pixel C), the full-color photosensitive pixel W, the alternating sequence of the photosensitive pixel 110 in the second row is the full-color photosensitive pixel W, the single-color photosensitive pixel (ie the third color Photosensitive pixel C).
- the alternating sequence of pixels is not consistent.
- the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, first-color photosensitive pixels A), and the second row
- the alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, the first color photosensitive pixel A), a full-color photosensitive pixel W; and in the third type of subunit CC shown in FIG.
- the photosensitive pixels 110 in the first row The alternating sequence is single-color photosensitive pixels (that is, third-color photosensitive pixels C), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels). Pixel C). That is to say, in the same minimum repeating unit, the alternating sequence of full-color photosensitive pixels W and color photosensitive pixels in different subunits can be the same (as shown in Figure 5) or inconsistent (as shown in Figure 8). Show).
- FIG. 9 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application.
- the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category.
- the photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.
- the categories of subunits include three categories.
- the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A
- the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B
- the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C.
- Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC.
- one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1
- two second type subunits UB are arranged in the second diagonal direction D2.
- the first diagonal direction D1 is different from the second diagonal direction D2.
- the first diagonal line and the second diagonal line are perpendicular.
- FIG. 10 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application.
- the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110.
- the arrangement method is:
- W represents the full-color photosensitive pixel
- A represents the first color photosensitive pixel among the multiple color photosensitive pixels
- B represents the second color photosensitive pixel among the multiple color photosensitive pixels
- C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.
- a plurality of photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category.
- the photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.
- the categories of subunits include three categories.
- the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A
- the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B
- the third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C.
- Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC.
- a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1
- two second type subunits UB are arranged in a second diagonal direction D2.
- the first diagonal direction D1 is different from the second diagonal direction D2.
- the first diagonal line and the second diagonal line are perpendicular.
- multiple photosensitive pixels 110 in the same row in some sub-units may be photosensitive pixels 110 of the same category, and multiple photosensitive pixels 110 in the same column in the remaining sub-units
- the pixels 110 are photosensitive pixels 110 of the same type.
- the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a green photosensitive pixel G; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.
- the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a yellow photosensitive pixel Y; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.
- the first color photosensitive pixel A may be a magenta photosensitive pixel M; the second color photosensitive pixel B may be a cyan photosensitive pixel Cy; and the third color photosensitive pixel C may It is the yellow photosensitive pixel Y.
- the response band of the full-color photosensitive pixel W may be the visible light band (for example, 400 nm-760 nm).
- the full-color photosensitive pixel W is provided with an infrared filter to filter out infrared light.
- the response wavelength bands of the full-color photosensitive pixel W are visible light and near-infrared wavelengths (for example, 400nm-1000nm), and the photoelectric conversion element 1111 (shown in FIG. 4) in the image sensor 10 (shown in FIG. 1) (Shown) to match the response band.
- the full-color photosensitive pixel W may not be provided with a filter or a filter that can pass light of all wavelength bands.
- the response band of the full-color photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two match. .
- the embodiments of the present application include, but are not limited to, the above-mentioned waveband range.
- the control unit 13 controls the pixel array 11 to expose.
- at least one single-color photosensitive pixel is exposed with a first exposure time
- at least one single-color photosensitive pixel is exposed with a second exposure time less than the first exposure time
- at least one full-color photosensitive pixel is exposed
- the photosensitive pixel W is exposed at a third exposure time that is less than or equal to the first exposure time.
- the plurality of single-color photosensitive pixels exposed at the first exposure time in the pixel array 11 may generate first color information
- the plurality of single-color photosensitive pixels exposed at the second exposure time may generate second color information, which are exposed at the third exposure time.
- a plurality of panchromatic photosensitive pixels W can generate panchromatic information.
- the first color information may form a first color original image.
- the second color information can form a second color original image.
- Panchromatic information can generate a panchromatic original image.
- part of the panchromatic photosensitive pixels W in the same subunit is exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels W are exposed at the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time.
- a single-color photosensitive pixel takes the first exposure time (for example, the long exposure time L shown in FIG. 11) Exposure, a single-color photosensitive pixel is exposed for the second exposure time (for example, the short exposure time S shown in FIG. 11), and a full-color photosensitive pixel W is exposed for the third exposure time (for example, the short exposure time S shown in FIG. 11) , One full-color photosensitive pixel W is exposed for the fourth exposure time (for example, the long exposure time L shown in FIG. 11).
- the exposure process of the pixel array 11 may be: (1) the photosensitive pixels 110 exposed at the first exposure time, the photosensitive pixels 110 exposed at the second exposure time, and the third exposure time.
- the time-exposed photosensitive pixel 110 and the photosensitive pixel 110 exposed at the fourth exposure time are sequentially exposed (the exposure sequence of the four is not limited), and the exposure time of the four does not overlap; (2) the first exposure time
- the exposed photosensitive pixels 110, the photosensitive pixels 110 exposed at the second exposure time, the photosensitive pixels 110 exposed at the third exposure time, and the photosensitive pixels 110 exposed at the fourth exposure time are sequentially exposed (the exposure order of the four is not limited ), and there is a partial overlap in the exposure time of the four;
- the exposure time of all the photosensitive pixels 110 exposed with a shorter exposure time is within the exposure time of the photosensitive pixels 110 exposed with the longest exposure time
- the exposure time of all the single-color photosensitive pixels exposed at the second exposure time is within the exposure time of all the single-color photosensitive pixels exposed at the
- the image sensor 10 can output four original images, which are: (1) The first color original image, which is generated by multiple single-color photosensitive pixels exposed with a long exposure time L (first exposure time) (2)
- the second color original image is composed of the second color information generated by multiple single-color photosensitive pixels exposed with a short exposure time S (second exposure time); (3)
- the first full The color original image is composed of the first panchromatic information generated by multiple panchromatic photosensitive pixels W (third exposure time) exposed with a short exposure time S;
- the second panchromatic original image is composed of a long exposure time L (Fourth Exposure Time) It is composed of second panchromatic information generated by a plurality of panchromatic photosensitive pixels W exposed.
- the image fusion module 20 performs fusion processing on the first color original image and the second panchromatic original image to obtain a first intermediate image, and performs fusion processing on the second color original image and the first panchromatic original image to obtain the first intermediate image. Two intermediate images.
- the image fusion module 20 first separates the color and brightness of the first color original image to obtain a color-brightness separated image.
- LIT stands for brightness and CLR stands for color.
- CLR stands for color.
- the image fusion module 20 can combine RGB The first color original image in the space is converted into a color-brightness separated image in YCrCb space.
- Y in YCrCb is the brightness LIT in the color-brightness separated image
- Cr and Cb in YCrCb are the color CLR in the color-brightness separated image.
- the image fusion module 20 can also convert the first color original image of RGB into a color-light-separated image in Lab space.
- L in Lab is the brightness LIT in the color-light-separated image
- a and a in Lab b is the color CLR in the color-light-separated image.
- the LIT+CLR in the color-light separation image shown in FIG. 12 does not mean that the pixel value of each pixel is formed by adding L and CLR, but only that the pixel value of each pixel is composed of LIT and CLR.
- the image fusion module 20 fuses the brightness of the color-brightness separated image and the brightness of the second full-color original image.
- the pixel value of each panchromatic pixel W in the second panchromatic original image is the brightness value of each panchromatic pixel
- the image fusion module 20 can separate the LIT of each pixel in the color-brightness image with the panchromatic pixel.
- the W of the panchromatic pixel at the corresponding position in the intermediate image is added to obtain the pixel value after brightness correction.
- the image fusion module 20 forms a brightness-corrected color-brightness separated image according to a plurality of brightness-corrected pixel values, and then uses color space conversion to convert the brightness-corrected color-brightness separated image into a first intermediate image.
- the image fusion module 20 performs fusion processing on the second color original image and the first panchromatic original image to obtain a second intermediate image.
- the acquisition process of the second intermediate image is the same as the acquisition process of the first intermediate image, and will not be repeated here.
- the image fusion module 20 can also use other methods to perform fusion processing, which is not limited here.
- the fusion processing of the color original image and the full-color original image by the image fusion module 20 can increase the brightness of the intermediate image obtained after fusion.
- the first color original image is composed of first color information generated by multiple single-color photosensitive pixels exposed with a long exposure time L
- the second full-color original image is also exposed with a long exposure time L
- the second panchromatic information generated by a plurality of panchromatic photosensitive pixels W is composed, so the exposure time corresponding to all the image pixels in the first intermediate image obtained by the fusion process of the first color original image and the second panchromatic original image is Long exposure time L.
- the second color original image is composed of second color information generated by a plurality of single-color photosensitive pixels exposed with a short exposure time S
- the first full-color original image is also composed of a plurality of single-color photosensitive pixels exposed with a short exposure time S
- the first panchromatic information generated by the panchromatic photosensitive pixel W is composed, so the exposure time corresponding to all image pixels in the second intermediate image obtained by the fusion process of the second color original image and the first panchromatic original image is short exposure Time S.
- the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31 and a brightness mapping unit 33.
- the high dynamic range image processing unit 31 is used to fuse the first intermediate image and the second intermediate image into a third high dynamic range image;
- the brightness mapping unit 33 is used to perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image Range image.
- the process of the high dynamic range image processing unit 31 fusing the first intermediate image and the second intermediate image may include brightness alignment processing.
- the high dynamic range image processing unit 31 performing brightness alignment processing on the first intermediate image and the second intermediate image includes the following steps: (1) identifying overexposed image pixels in the first intermediate image with pixel values greater than a first preset threshold; (2) ) For each overexposed image pixel, expand the predetermined area with the overexposed image pixel as the center; (3) Find the intermediate image pixel with the pixel value less than the first preset threshold in the predetermined area; (4) Use the intermediate image pixel and The second intermediate image corrects the pixel values of the pixels of the overexposed image; (5) the first intermediate image is updated with the corrected pixel values of the pixels of the overexposed image to obtain the first intermediate image with the brightness aligned.
- the high dynamic range image processing unit 31 expands a predetermined area with the overexposed image pixel P12 as the center, for example, the 3*3 area shown in FIG. 14.
- it may also be a 4*4 area, a 5*5 area, a 10*10 area, etc., which is not limited here.
- the high dynamic range image processing unit 31 searches for an intermediate image pixel with a pixel value less than the first preset threshold V0 in a predetermined area of 3*3, such as image pixel P21 in FIG. 14 (marked in the first intermediate image in FIG. 14). If the pixel value V2 of the image pixel with a dotted circle is less than the first preset threshold V0, the image pixel P21 is the intermediate image pixel P21. Subsequently, the high dynamic range image processing unit 31 searches the second intermediate image for image pixels corresponding to the overexposed image pixel P12 and the intermediate image pixel P21 respectively, that is, the image pixel P1'2' (marked in the second intermediate image in FIG. 14).
- the high dynamic range image processing unit 31 performs this brightness alignment process on each overexposed image pixel in the first intermediate image to obtain the first intermediate image after brightness alignment. Since the pixel value of the overexposed image pixel in the first intermediate image after brightness alignment is corrected, the pixel value of each image pixel in the first intermediate image after brightness alignment is relatively accurate.
- the high dynamic range image processing unit 31 may fuse the first intermediate image and the second intermediate image with the brightness aligned to obtain The third high dynamic color image.
- the high dynamic range image processing unit 31 first performs motion detection on the first intermediate image after brightness alignment to identify whether there is a motion blur area in the first intermediate image after brightness alignment. If there is no motion blur area in the first intermediate image after brightness alignment, the first intermediate image and the second intermediate image after brightness alignment are directly merged to obtain the first high dynamic range image.
- the motion blur area in the first intermediate image after brightness alignment is eliminated, and only all areas of the second intermediate image are merged and the first intermediate image after brightness alignment is removed. The area outside the area to obtain the first high dynamic range image.
- the fusion of the two intermediate images at this time follows the following principles: ( 1) In the first intermediate image after brightness alignment, the pixel value of the image pixel in the overexposed area is directly replaced with the pixel value of the image pixel corresponding to the overexposed area in the second intermediate image; (2) the first intermediate image after brightness alignment In an intermediate image, the pixel value of the image pixel in the under-exposed area is: the long-exposure pixel value divided by the ratio of the long-short pixel value; (3) the first intermediate image after brightness alignment, the image in the area that is neither under-exposed nor over-exposed The pixel value of a pixel: the long-exposure pixel value divided by the ratio of the long-short pixel value.
- the fusion of the two intermediate images at this time must follow the above three principles, and also need to follow the (4) principle: the first intermediate image after brightness alignment , The pixel value of the image pixel in the motion blur area is directly replaced with the pixel value of the image pixel in the second intermediate image corresponding to the motion blur area.
- the signal-to-noise ratio of VS’ will be greater than the signal-to-noise ratio of VS.
- the dynamic range of the obtained image can be increased, and the imaging effect of the image can be improved.
- the high dynamic range image processing unit 31 may also use other methods to fuse the first intermediate image and the second intermediate image with the brightness aligned to obtain the third high dynamic color image.
- the high dynamic range image processing unit 31 may also perform motion blur detection on the first intermediate image and the second intermediate image after the brightness is aligned, and perform motion blur detection on the detected first intermediate image and the second intermediate image.
- the motion blur is eliminated to obtain the first intermediate image after the motion blur is eliminated and the second intermediate image after the motion blur is eliminated.
- the high dynamic range image processing unit 31 obtains the first intermediate image after removing the motion blur and the second intermediate image after removing the motion blur
- the first intermediate image after removing the motion blur and the second intermediate image after removing the motion blur are processed again.
- the intermediate images are fused to obtain the third high dynamic range image with high dynamic range, which is not limited here.
- the high dynamic range image processing unit 31 transmits the third high dynamic range image to the brightness mapping unit 33.
- the brightness mapping unit 33 subjects the third high dynamic range image to brightness mapping processing to obtain the first high dynamic range image.
- the bit width of the data of each image pixel in the first high dynamic range image is smaller than the bit width of the data of each image pixel in the third high dynamic range image.
- a third high dynamic range image with a bit width of 16 bits can be obtained.
- the brightness mapping unit 33 may perform brightness mapping processing on the third high dynamic range image with a bit width of 16 bits to obtain the first high dynamic range image with a bit width of 10 bits.
- the third high dynamic range image with a bit width of 16 bits may also be subjected to brightness mapping processing to obtain the first high dynamic range image with a bit width of 12 bits, which is not limited here. In this way, the data volume of the high dynamic range image is reduced through the brightness mapping process, thereby avoiding the problem that the image processor 40 cannot process the high dynamic range image with excessive data volume, and is beneficial to improve the image processor 40 processing high dynamic range image. speed.
- the high dynamic range image processing unit 31 can transmit the first high dynamic range image to the image processor 40 for subsequent processing such as black level, demosaicing, color conversion, lens shading correction, dead pixel compensation, global tone mapping, etc., to obtain the first high dynamic range image.
- the multiple color image pixels in the first high dynamic range image are all arranged in a Bayer array, and the pixel value of each image pixel only contains the information of one color channel, while the pixels of each image pixel in the second high dynamic range image The values all contain the information of each color channel.
- the high dynamic range image processing module 30 further includes a statistical unit 35, which is used to process the first intermediate image and the second intermediate image to obtain statistical data.
- the statistical unit 35 provides the statistical data to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- the image processor 40 can perform at least one of automatic exposure processing and automatic white balance processing according to the statistical data.
- the image processor 40 performs automatic exposure processing based on statistical data; or, the image processor 40 performs automatic white balance processing based on statistical data; or, the image processor 40 performs automatic exposure processing and automatic white balance processing based on statistical data.
- the image processor 40 can perform automatic exposure and automatic white balance processing according to the statistical data, which is beneficial to improve the quality of the image finally output by the image processor 40.
- the high dynamic range image processing module 30 further includes a lens shading correction unit 37, which is used to correct the third high dynamic range image to obtain a high dynamic range corrected image. Specifically, after the high dynamic range image processing unit 31 fuses the first intermediate image and the second intermediate image into a third high dynamic range image, the lens shading correction unit 37 performs lens shading correction processing on the third high dynamic range image to obtain a high dynamic range image. Dynamic range correction image. The specific process of lens shading correction processing is shown in Figure 17. The lens shading correction unit 37 divides the third high dynamic range image into sixteen grids equally, and each of the sixteen grids has one The preset compensation coefficient.
- the lens shading correction unit 37 performs shading correction on the image by the bilinear interpolation method according to the compensation effect of each grid area adjacent or itself and its vicinity.
- R2 is the pixel value in the dotted frame in the third high dynamic range image that has undergone lens shading correction processing
- R1 is the pixel value in the dotted frame in the first color original image shown in the figure.
- R2 R1*k1
- k1 is obtained by bilinear interpolation of the compensation coefficients 1.10, 1.04, 1.05, and 1.09 of the grid adjacent to the R1 pixel.
- the coordinates of the image are (x, y), x is counted from the first pixel from the left to the right, y is counted from the first pixel on the top, and both x and y are natural numbers, as indicated by the logo on the edge of the image Show.
- the coordinates of R1 are (3,3), the coordinates of R1 in each grid compensation coefficient map should be (0.75,0.75).
- f(x, y) represents the compensation value of the coordinate (x, y) in each grid compensation coefficient graph.
- the compensation coefficient of each grid is set in advance before the lens shading correction unit 37 performs lens shading correction processing.
- the lens shading correction unit 37 transmits the high dynamic range correction image to the statistics unit 35.
- the statistical unit 35 is configured to process the high dynamic range correction image to obtain statistical data, and provide the statistical data to the image processor 40 for automatic exposure processing and/or automatic white balance processing, that is, the statistical data is provided to the image processor 40 To perform at least one of automatic exposure processing and automatic white balance processing.
- the image processor 40 performs automatic exposure based on the statistical data
- the quality of the image obtained by processing and/or automatic white balance processing is higher. It should be noted that both the image fusion module 20 and the high dynamic range processing module 30 are integrated in the image sensor 10.
- the high dynamic range image processing system 100 shown in FIG. 16 first fuses the color original image and the panchromatic original image through the image fusion module 20 to obtain the first intermediate image and the second intermediate image, and then passes the high dynamic range image
- the processing module 30 performs high dynamic range processing on the first intermediate image and the second intermediate image to obtain the first high dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.
- the image sensor 10 obtains the first color original image, the second color original image, the first full color original image, and the second full color original image
- the four images The image is transmitted to the high dynamic range image processing module 30.
- the high dynamic range image processing module 30 fuses the first color original image and the second color original image into a first high dynamic color original image, and combines the first full color original image with the second full color original image.
- the color original image is fused into the first high dynamic panchromatic original image.
- the high dynamic range image processing module 30 transmits the first high dynamic full color original image and the first high dynamic color original image to the image fusion module 20 for fusion processing to finally obtain the first high dynamic range image.
- the image sensor 10 obtains the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image
- the high dynamic range image processing unit 31 in the high dynamic range image processing module 30 fuses the first color original image and the second color original image into a second high dynamic color original image Image, fusing the first panchromatic original image and the second panchromatic original image into a second high dynamic panchromatic original image.
- the specific fusion process is the same as the specific process of fusing the first intermediate image and the second intermediate image into the third high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.
- the brightness mapping unit 33 is configured to perform brightness mapping on the second high dynamic color original image to obtain a first high dynamic color original image with a small amount of data, and perform brightness mapping on the second high dynamic full color original image to obtain a small amount of data
- the first high dynamic panchromatic original image is the same as the specific process of mapping the brightness of the third high dynamic range image to the first high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.
- the lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image, and correct the second high dynamic panchromatic original image to obtain a high dynamic full color corrected image.
- the specific correction process is the same as the process of performing lens shading correction on the third high dynamic range image in the embodiment shown in FIG. 16 and FIG. 17, and will not be repeated here.
- the statistical unit 35 is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and transmit the statistical data to the image processor 40, so that the image processor 40 can perform automatic exposure and automatic exposure based on the statistical information. At least one of the white balance processing.
- the statistical unit 35 can also directly process the first color original image, the second color original image, the first full-color original image, and the second full-color original image to obtain statistical data, and transmit the statistical data to the image processor 40 , So that the image processor 40 can perform at least one of automatic exposure and automatic white balance processing according to the statistical information.
- the high dynamic range image processing module 30 After the high dynamic range image processing module 30 obtains the first high dynamic color original image and the first high dynamic full color original image, the two images are transmitted to the image fusion module 20 for fusion processing to obtain the first high dynamic range image.
- the specific process of the image fusion module 20 fusing the first high dynamic color original image and the first high dynamic panchromatic original image into the first high dynamic range image is the same as in the embodiment shown in FIG. 12, combining the first color original image and the second high dynamic range image.
- the specific fusion process of fusion of the panchromatic original image into the first intermediate image is the same, and will not be repeated here.
- the high dynamic range image processing system 100 shown in FIG. 18 first uses the high dynamic range image processing module 30 to fuse the color original image and the full color original image to obtain the first high dynamic color original image and the first high dynamic range image processing module 30. Then, the first high-dynamic color original image and the first high-dynamic panchromatic original image are merged by the image fusion module 20 to obtain the first high-dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.
- all panchromatic photosensitive pixels W in the pixel array 11 are exposed at the third exposure time, and the third exposure time may be greater than the second exposure time, so that all panchromatic photosensitive pixels W All are exposed with the medium exposure time M; or, the third exposure time is equal to the first exposure time, so that all the panchromatic photosensitive pixels W are exposed with the long exposure time L.
- the third exposure time can also be equal to or less than the second exposure time, Exposure of the full-color photosensitive pixel W with a short exposure time is not limited here.
- the third exposure time is greater than the second exposure time, that is, all the full-color photosensitive pixels W are exposed with the medium exposure time M as an example.
- a single-color photosensitive pixel has a first exposure time (for example, the long exposure time shown in FIG. 20).
- One single-color photosensitive pixel is exposed for the second exposure time (for example, the short exposure time S shown in FIG. 20), and the two full-color photosensitive pixels W are both exposed for the third exposure time (for example, the medium exposure time shown in FIG. 20).
- the exposure process of the pixel array 11 may be: (1) the photosensitive pixel 110 exposed at the first exposure time, the photosensitive pixel 110 exposed at the second exposure time, and the third exposure
- the time-exposed photosensitive pixels 110 are sequentially exposed (the exposure sequence of the three is not limited), and the exposure time of the three does not overlap; (2) the photosensitive pixels 110 exposed at the first exposure time are exposed at the second exposure time
- the exposed photosensitive pixels 110 and the photosensitive pixels 110 exposed at the third exposure time are sequentially exposed (the exposure sequence of the three is not limited), and the exposure time of the three overlaps partially; (3) All exposures are shorter
- the exposure time of the time-exposed photosensitive pixels 110 are all within the exposure time of the photosensitive pixels 110 that are exposed with the longest exposure time.
- the exposure time of all the single-color photosensitive pixels exposed at the second exposure time are all within the time
- the exposure time of all the single-color photosensitive pixels exposed at the first exposure time is within the exposure time of all the full-color photosensitive pixels W exposed at the third exposure time are within the exposure time of all the single-color photosensitive pixels exposed at the first exposure time Within time.
- the pixel array 11 adopts the (3) exposure method, which can shorten the overall exposure time required by the pixel array 11, which is beneficial to increase the frame rate of the image.
- the image sensor 10 can output three original images, which are: (1) The first color original image, which is generated by multiple single-color photosensitive pixels exposed with a long exposure time L (first exposure time) (2)
- the second color original image is composed of the second color information generated by multiple single-color photosensitive pixels exposed with a short exposure time S (second exposure time);
- the first full The color original image is composed of first panchromatic information generated by a plurality of panchromatic photosensitive pixels W exposed at a medium exposure time M (third exposure time).
- the image sensor 10 first transmits the first color original image and the second color original image to the high dynamic range image processing module 30 for high dynamic range processing to obtain the first high dynamic color original image, and then transfers the first color original image to the high dynamic range image processing module 30.
- a high dynamic color original image and a first panchromatic original image are transmitted to the image fusion module 20 for fusion algorithm processing to obtain the first high dynamic range image.
- the image sensor 10 transmits the first color original image, the second color original image, and the first panchromatic to the high dynamic range image processing module 30.
- the high dynamic range image in the high dynamic range image processing module 30 The processing unit 31 fuses the first color original image and the second color original image into a second high-dynamic color original image.
- the specific fusion process is the same as that of the first intermediate image and the second intermediate image in the embodiment shown in FIG. 15
- the specific process of the three high dynamic range images is the same, so I won’t repeat them here.
- the brightness mapping unit 33 is configured to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image with a small amount of data.
- the specific process is the same as the specific process of mapping the brightness of the third high dynamic range image to the first high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.
- the lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image.
- the specific correction process is the same as the process of performing lens shading correction on the third high dynamic range image in the embodiment shown in FIGS. 16 and 17. I will not repeat them here.
- the statistical unit 35 is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and transmit the statistical data to the image processor 40, so that the image processor 40 can perform automatic exposure and automatic exposure based on the statistical information. At least one of the white balance processing.
- the statistical unit 35 can also directly process the first color original image, the second color original image, and the first full-color original image to obtain statistical data, and transmit the statistical data to the image processor 40 so that the image processor 40 can At least one of automatic exposure and automatic white balance processing is performed based on the statistical information.
- the high dynamic range image processing module 30 transmits the first high dynamic color original image and the first panchromatic original image to the image fusion module 20 for fusion processing to obtain the first high dynamic range image.
- the first panchromatic original image obtained by the image sensor 10 includes a plurality of panchromatic image pixels W and a plurality of empty image pixels N (NULL), wherein the empty image pixels are neither complete Color image pixels are also not color image pixels.
- the position of the empty image pixel N in the first full-color original image can be regarded as there is no image pixel in that position, or the pixel value of the empty image pixel can be regarded as zero.
- the sub-unit includes two full-color image pixels W and two color image pixels (color image pixel A, color image pixel B, Or color image pixel C).
- the first full-color original image also has a sub-unit corresponding to each sub-unit in the pixel array 11, and the sub-unit of the first full-color original image includes two full-color image pixels W and two empty image pixels N, The positions of the two empty image pixels N correspond to the positions of the two color image pixels in the subunit of the pixel array 11.
- the image fusion module 20 may further process the first full-color original image to obtain a full-color intermediate image.
- each sub-unit includes a plurality of empty image pixels N and a plurality of panchromatic image pixels.
- some sub-units include two empty image pixels N and two panchromatic image pixels W.
- the image fusion module 20 may use the pixel values of all panchromatic image pixels in the subunit including the empty image pixel N and the panchromatic image pixel W as the panchromatic large pixel W in the subunit to obtain a panchromatic intermediate image.
- the resolution of the panchromatic intermediate image at this time is the same as the resolution of the first high dynamic color original image, so as to facilitate the fusion of the panchromatic intermediate image and the first high dynamic color original image.
- the specific fusion process of the panchromatic intermediate image and the first high dynamic color original image is the same as the specific fusion process of fusing the first color original image and the second panchromatic original image into the first intermediate image in the embodiment shown in FIG. 12. I will not repeat them here.
- the high dynamic range image processing system 100 shown in FIG. 18 first fused the color original image and the panchromatic original image through the high dynamic range image processing module 30 to obtain the first high dynamic color original image, and then passed the image fusion module 20
- the first high dynamic color original image and the first panchromatic original image are merged to obtain the first high dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.
- the present application also provides an electronic device 1000.
- the electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 described in any one of the above embodiments.
- the lens 300 and the high dynamic range image processing system 100 are combined with the housing 200.
- the lens 300 cooperates with the image sensor 10 of the high dynamic range image processing system 100 for imaging.
- the electronic device 1000 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which are not limited here.
- a smart wearable device such as a smart watch, a smart bracelet, a smart glasses, a smart helmet
- a drone a head-mounted display device, etc., which are not limited here.
- the electronic device 1000 of the embodiment of the present application performs fusion algorithm processing and high dynamic range processing on the full-color original image and the color original image output by the image sensor 10 through the image fusion module 20 and the high dynamic range image processing module 30 to obtain image pixels.
- the first high dynamic range image arranged in a Bayer array, and then the first high dynamic range image is input to the image processor for subsequent processing, thereby solving the problem that the image processor 40 cannot directly perform processing on the image with image pixels arranged in a non-Bayer array. Handling issues.
- the high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100.
- the high dynamic range image processing system 100 includes an image sensor 10.
- the image sensor 10 includes a pixel array 11.
- the pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels.
- the pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels.
- High dynamic range image processing methods include:
- the first high dynamic range image includes multiple color images Pixels, a plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by the image processor to obtain the second high dynamic range image.
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image; for the first color original image, the second color original image and the first panchromatic original
- Performing fusion algorithm processing and high dynamic range processing on the image to obtain the first high dynamic range image includes: fusing the first color original image and the second panchromatic original image into a first intermediate image, and combining the second color original image with the first full color original image.
- the color original image is fused into a second intermediate image; and the first intermediate image and the second intermediate image are fused into a first high dynamic range image.
- fusing the first intermediate image and the second intermediate image into a first high dynamic range image includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; and The three high dynamic range images are subjected to brightness mapping to obtain the first high dynamic range image.
- the high dynamic range image processing method further includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; obtaining a high dynamic range corrected image from the third high dynamic range image; And processing the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
- the high dynamic range image processing method further includes: processing the first intermediate image and the second intermediate image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing .
- part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time.
- the fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image.
- Performing fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image includes: combining the first color original image and the second color original image Fusion into a first high-dynamic color original image, fusing the first full-color original image and the second full-color original image into a first high-dynamic full-color original image; and combining the first high-dynamic color original image with the first high-dynamic full-color original image The color original image is fused into the first high dynamic range image.
- the first color original image and the second color original image are fused into a first high dynamic color original image
- the first panchromatic original image and the second panchromatic original image are fused into a first high dynamic color original image.
- the color original image includes: fusing the first color original image and the second color original image into a second high-dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into a second high-dynamic panchromatic original image Image; and brightness mapping is performed on the second high dynamic color original image to obtain the first high dynamic color original image, and brightness mapping is performed on the second high dynamic panchromatic original image to obtain the first high dynamic full color original image.
- the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image, and combining the first panchromatic original image with the second panchromatic original image.
- the color correction image and the high dynamic full color correction image are used to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- all panchromatic photosensitive pixels in the same subunit are exposed at the third exposure time; the first color original image, the second color original image, and the first panchromatic original image are processed by a fusion algorithm and are highly dynamic
- the range processing to obtain the first high dynamic range image includes: fusing the first color original image and the second color original image into a first high dynamic color original image; and combining the first high dynamic color original image and the first panchromatic original image Fusion is the first high dynamic range image.
- fusing the first color original image and the second color original image into a first high dynamic color original image includes: fusing the first color original image and the second color original image into a second high dynamic color original image And performing brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image.
- the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image; correcting the second high dynamic color original image to obtain a high dynamic color Correcting the image; and processing the high dynamic color correction image and the first full-color original image to obtain statistical data, which is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.
- the specific implementation process of the high dynamic range image processing method of any one of the foregoing embodiments is the same as the specific implementation process of the aforementioned high dynamic range image processing system 100 to obtain a high dynamic range image, and will not be further described here.
- the present application also provides a non-volatile computer-readable storage medium 400 containing a computer program.
- the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.
- the pixel array 11 is exposed, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, and at least one single-color photosensitive pixel is exposed with a second exposure time that is less than the first exposure time, At least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time; wherein, the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains the first color original image, and the first color original image is obtained at the second exposure time.
- the second color information generated by the exposed single-color photosensitive pixels obtains a second color original image, and the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image; and
- the first high dynamic range image including a plurality of color image pixels, A plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by the image processor to obtain a second high dynamic range image.
- the processor 60 When the computer program is executed by the processor 60, the processor 60 is caused to perform the following steps:
- the first high dynamic color original image and the first panchromatic original image are fused into a first high dynamic range image.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Color Television Image Signal Generators (AREA)
- Image Processing (AREA)
Abstract
一种高动态范围图像处理***(100)、高动态范围图像处理方法、电子设备(1000)和计算机可读存储介质。高动态范围图像处理***(100)包括图像传感器(10)、图像融合模块(20)和高动态范围图像处理模块(30)。图像传感器(10)中的像素阵列(11)曝光。图像融合模块(20)和高动态范围图像处理模块(30)用于对第一彩色原始图像、第二彩色原始图像和全色原始图像进行高动态范围处理及融合算法处理以得到第一高动态范围图像。
Description
优先权信息
本申请请求2020年4月3日向中国国家知识产权局提交的、专利申请号为202010259292.6的专利申请的优先权和权益,并且通过参照将其全文并入此处。
本申请涉及图像处理技术领域,特别涉及一种高动态范围图像处理***、高动态范围图像处理方法、电子设备及非易失性计算机可读存储介质。
手机等电子设备中可以设置有摄像头以实现拍照功能。摄像头内可以设置用于接收光线的图像传感器。图像传感器中可以设置有滤光片阵列。
发明内容
本申请实施方式提供了一种高动态范围图像处理***、高动态范围图像处理方法、电子设备及非易失性计算机可读存储介质。
本申请实施方式提供一种高动态范围图像处理***。高动态范围图像处理***包括图像传感器、图像融合模块及高动态范围图像处理模块。所述图像传感器包括像素阵列,所述像素阵列包括多个全色感光像素和多个彩色感光像素,所述彩色感光像素具有比所述全色感光像素更窄的光谱响应,所述像素阵列包括最小重复单元,每个所述最小重复单元包含多个子单元,每个所述子单元包括多个单颜色感光像素及多个全色感光像素,所述图像传感器中的像素阵列曝光。其中,对于同一所述子单元中的多个感光像素,至少一个所述单颜色感光像素以第一曝光时间曝光,至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光,至少一个所述全色感光像素以小于所述第一曝光时间的第三曝光时间曝光。其中,以所述第一曝光时间曝光的所述单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以所述第二曝光时间曝光的所述单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以所述第三曝光时间曝光的所述全色感光像素生成第一全色原始图像。所述图像融合模块及所述高动态范围图像处理模块用于对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像。所述第一高动态范围图像包含多个彩色图像像素,多个所述彩色图像像素呈拜耳阵列排布。所述第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
本申请实施方式提供一种高动态范围图像处理方法。所述高动态范围图像处理方法用于高动态范围图像处理***。所述高动态范围图像处理***包括图像传感器,所述图像传感器包括像素阵列,所述像素阵列包括多个全色感光像素和多个彩色感光像素,所述彩色感光像素具有比所述全色感光像素更窄的光谱响应,所述像素阵列包括最小重复单元,每个所述最小重复单元包含多个子单元,每个所述子单元包括多个单颜色感光像素及多个全色感光像素。所述高动态范围图像处理方法包括:控制所述像素阵列曝光,其中,对于同一所述子单元中的多个感光像素,至少一个所述单颜色感光像素以第一曝光时间曝光,至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光,至少一个所述全色感光像素以小于所述第一曝光时间的第三曝光时间曝光。其中,以所述第一曝光时间曝光的所述单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以所述第二曝光时间曝光的所述单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以所述第三曝光时间曝光的所述全色感光像素生成第一全色原始图像。及对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像。所述第一高动态范围图像包含多个彩色图像像素,多个所述彩色图像像素呈拜耳阵列排布。所述第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
本申请实施方式提供一种电子设备。所述电子设备包括镜头、壳体及上述的高动态范围图像处理***。所述镜头、所述高动态范围图像处理***与所述壳体结合,所述镜头与所述高动态范围图像处理***的图像传感器配合成像。
本申请实施方式提供一种包含计算机程序的非易失性计算机可读存储介质。所述计算机程序被处理器执行时,使得所述处理器执行上述的高动态范围图像处理方法。
本申请实施方式的附加方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本申请的实践了解到。
本申请的上述和/或附加的方面和优点可以从结合下面附图对实施方式的描述中将变得明显和容易理解,其中:
图1是本申请实施方式的一种高动态范围图像处理***的示意图;
图2是本申请实施方式的一种像素阵列的示意图;
图3是本申请实施方式的一种感光像素的截面示意图;
图4是本申请实施方式的一种感光像素的像素电路图;
图5是本申请实施方式的一种像素阵列中最小重复单元的排布示意图;
图6是本申请实施方式的又一种像素阵列中最小重复单元的排布示意图;
图7是本申请实施方式的又一种像素阵列中最小重复单元的排布示意图;
图8是本申请实施方式的又一种像素阵列中最小重复单元的排布示意图;
图9是本申请实施方式的又一种像素阵列中最小重复单元的排布示意图;
图10是本申请实施方式的又一种像素阵列中最小重复单元的排布示意图;
图11是本申请实施方式的一种图像传感器输出的原始图像的示意图;
图12是本申请实施方式的一种图像融合处理原理的示意图;
图13是本申请实施方式的又一种图像融合处理原理的示意图;
图14是本申请实施方式的一种亮度对齐处理的示意图;
图15是本申请实施方式的一种高动态范围处理原理的示意图;
图16是本申请实施方式的又一种高动态范围图像处理***的示意图;
图17是本申请实施方式的一种镜头阴影矫正处理的示意图;
图18是本申请实施方式的又一种高动态范围图像处理***的示意图;
图19是本申请实施方式的又一种高动态范围处理原理的示意图;
图20是本申请实施方式的又一种图像传感器输出的原始图像的示意图;
图21是本申请实施方式的又一种高动态范围处理原理的示意图;
图22是本申请实施方式的又一种图像融合处理原理的示意图;
图23本申请实施方式的一种电子设备的结构示意图;
图24是本申请实施方式的一种高动态范围图像获取方法的流程示意图;
图25是本申请实施方式的一种非易失性计算机可读存储介质与处理器的交互示意图。
下面详细描述本申请的实施方式,所述实施方式的示例在附图中示出,其中,相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的,仅用于解释本申请的实施方式,而不能理解为对本申请的实施方式的限制。
请参阅图1,本申请实施方式提供一种高动态范围图像处理***100。高动态范围图像处理***100包括图像传感器10、图像融合模块20及高动态范围图像处理模块30。图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素,彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元,每个最小重复单元包含多个子单元,每个子单元包括多个单颜色感光像素及多个全色感光像素。图像传感器10中的像素阵列11曝光,其中,对于同一子单元中的多个感光像素,至少一个单颜色感光像素以第一曝光时间曝光,至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光,至少一个全色感光像素以小于第一曝光时间的第三曝光时间曝光。其中,以第一曝光时间曝光的单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以第二曝光时间曝光的单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以第三曝光时间曝光的全色感光像素生成第一全色原始图像。图像融合模块20及高动态范围图像处理模块30用于对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像。第一高动态范围图像包含多个彩色图像像素,多个彩色图像像素呈拜耳阵列排布。第一高动态范围图像由图像处理器40处理以得到第二高动态范围图像。
请参阅图1,在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间。图像融合模块20用于将第一彩色原始图像与第二全色原始图像融合为第一中间图像,将第二彩色原始图像与第一全色原始图像融合为第二中间图像,以第四曝光时间曝光的单颜色感光像素生成的第二全色信息得到第二全色原始图像。高动态范围图像处理模块30用于将第一中间图像与第二中间图像融合为第一 高动态范围图像。
请参阅图16,在某些实施方式中,高动态范围图像处理模块30包括高动态范围图像处理单元31及亮度映射单元33。高动态范围图像处理单元31用于将第一中间图像及第二中间图像融合为第三高动态范围图像。亮度映射单元33用于对第三高动态范围图像进行亮度映射以得到第一高动态范围图像。
请参阅图16,在某些实施方式中,高动态范围图像处理模块30包括高动态范围图像处理单元31、镜头阴影校正单元37及统计单元35。高动态范围图像处理单元31用于将第一中间图像及第二中间图像融合为第三高动态范围图像;镜头阴影校正单元37用于校正第三高动态范围图像以得到高动态范围校正图像;统计单元35用于处理高动态范围校正图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
请参阅图16,在某些实施方式中,高动态范围图像处理模块30包括统计单元35,统计单元35用于处理第一中间图像及第二中间图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
请参阅图18,在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间。高动态范围图像处理模块30用于将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像。图像融合模块20用于将第一高动态彩色原始图像与第一高动态全色原始图像融合为第一高动态范围图像。
请参阅图18,在某些实施方式中,高动态范围图像处理模块30还包括高动态范围图像处理单元31及亮度映射单元33。高动态范围图像处理单元31用于将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;亮度映射单元33用于对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像,对第二高动态全色原始图像进行亮度映射以得到第一高动态全色原始图像。
请参阅图18,在某些实施方式中,高动态范围图像处理模块30还包括高动态范围图像处理单元31、镜头阴影校正单元37及统计单元35。高动态范围图像处理单元31用于将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;镜头阴影校正单元37用于校正第二高动态彩色原始图像以得到高动态彩色校正图像,校正第二高动态全色原始图像以得到高动态全色校正图像;统计单元35用于处理高动态彩色校正图像及高动态全色校正图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
请参阅图18,在某些实施方式中,同一子单元中的全部全色感光像素以第三曝光时间曝光;高动态范围图像处理模块30用于将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像;图像融合模块20用于将第一高动态彩色原始图像与第一全色原始图像融合为第一高动态范围图像。
请参阅图18,在某些实施方式中,高动态范围图像处理模块30还包括高动态范围图像处理单元31及亮度映射单元33。高动态范围图像处理单元31用于将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像;亮度映射单元33用于对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像。
请参阅图18,在某些实施方式中,高动态范围图像处理模块30还包括高动态范围图像处理单元31、镜头阴影校正单元37及统计单元35。高动态范围图像处理单元31用于将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像。镜头阴影校正单元37用于校正第二高动态彩色原始图像以得到高动态彩色校正图像。统计单元35用于处理高动态彩色校正图像及第一全色原始图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,图像融合模块20及高动态范围图像处理模块30均集成在所述图像传感器中。
请参阅图24,本申请还提供一种高动态范围图像处理方法。本申请实施方式的高动态范围图像处理方法用于高动态范围图像处理***100。高动态范围图像处理***100包括图像传感器10。图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素。彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元,每个最小重复单元包含多个子单元。每个子单元包括多个单颜色感光像素及多个全色感光像素。高动态范围图像处理方法包括:
01:像素阵列11曝光,其中,对于同一子单元中的多个感光像素,至少一个单颜色感光像素以第一曝光时间曝光,至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光,至少一个全色感光像素以小于第一曝光时间的第三曝光时间曝光;其中,以第一曝光时间曝光的单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以第二曝光时间曝光的单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以第三曝光时间曝光的全色感光像素生成第一全色原始图像;及
02:对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,第一高动态范围图像包含多个彩色图像像素,多个彩色图像像素呈拜耳阵列排布,第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间,并且以第四曝光时间曝光的单颜色感光像素生成的第二全色信息得到第二全色原始图像;对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:对第一彩色原始图像与第二全色原始图像融合为第一中间图像,将第二彩色原始图像与第一全色原始图像融合为第二中间图像;及将第一中间图像与第二中间图像融合为第一高动态范围图像。
在某些实施方式中,将第一中间图像与第二中间图像融合为第一高动态范围图像包括:将第一中间图像及第二中间图像融合为第三高动态范围图像;及对所第三高动态范围图像进行亮度映射以得到第一高动态范围图像。
在某些实施方式中,高动态范围图像处理方法还包括:将第一中间图像及第二中间图像融合为第三高动态范围图像;对第三高动态范围图像以得到高动态范围校正图像;及处理高动态范围校正图像以获得统计数据,统计数据提供给图像处理器以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,高动态范围图像处理方法还包括:处理第一中间图像及第二中间图像以获得统计数据,统计数据提供给图像处理器以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间,并且以第四曝光时间曝光的单颜色感光像素生成的第二全色信息得到第二全色原始图像。对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像;及将第一高动态彩色原始图像与第一高动态全色原始图像融合为第一高动态范围图像。
在某些实施方式中,将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;及对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像,对第二高动态全色原始图像进行亮度映射以得到第一高动态全色原始图像。
在某些实施方式中,高动态范围图像处理方法包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;校正第二高动态彩色原始图像以得到高动态彩色校正图像,校正第二高动态全色原始图像以得到高动态全色校正图像;及处理高动态彩色校正图像及高动态全色校正图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,同一子单元中的全部全色感光像素以第三曝光时间曝光;对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像;及将第一高动态彩色原始图像与第一全色原始图像融合为第一高动态范围图像。
在某些实施方式中,第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像;及对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像。
在某些实施方式中,高动态范围图像处理方法包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像;校正第二高动态彩色原始图像以得到高动态彩色校正图像;及处理高动态彩色校正图像及第一全色原始图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
请参阅图23,本申请还提供一种电子设备1000。本申请实施方式的电子设备1000包括镜头300、壳体200及上述任意一个实施方式所述的高动态范围图像处理***100。
请参阅25,本申请还提供一种包含计算机程序的非易失性计算机可读存储介质400。该计算机程序被处理器60执行时,使得处理器60执行上述任意一个实施方式所述的高动态范围图像处理方法。
本申请实施方式的高动态范围图像处理***100通过图像融合模块20及高动态范围图像处理模块30对图像传感器10输出的全色原始图像和彩色原始图像事先进行融合算法处理及高动态范围处理,以得到图像像素呈拜耳阵列排布的第一高动态范围图像,再把第一高动态范围图像输入图像处理器中完成后续处理,从而解决图像处理器40不能直接对图像像素呈非拜耳阵列排布的图像进行处理的问题。
下面结合附图对本申请作进一步说明。
图2是本申请实施方式中的图像传感器10的示意图。图像传感器10包括像素阵列11、垂直驱动单元12、控制单元13、列处理单元14和水平驱动单元15。
例如,图像传感器10可以采用互补金属氧化物半导体(CMOS,Complementary Metal Oxide Semiconductor)感光元件或者电荷耦合元件(CCD,Charge-coupled Device)感光元件。
例如,像素阵列11包括以阵列形式二维排列(即二维矩阵形式排布)的多个感光像素110(图3所示),每个感光像素110包括光电转换元件1111(图4所示)。每个感光像素110根据入射在其上的光的强度将光转换为电荷。
例如,垂直驱动单元12包括移位寄存器和地址译码器。垂直驱动单元12包括读出扫描和复位扫描功能。读出扫描是指顺序地逐行扫描单位感光像素110,从这些单位感光像素110逐行地读取信号。例如,被选择并被扫描的感光像素行中的每一感光像素110输出的信号被传输到列处理单元14。复位扫描用于复位电荷,光电转换元件的光电荷被丢弃,从而可以开始新的光电荷的积累。
例如,由列处理单元14执行的信号处理是相关双采样(CDS)处理。在CDS处理中,取出从所选感光像素行中的每一感光像素110输出的复位电平和信号电平,并且计算电平差。因而,获得了一行中的感光像素110的信号。列处理单元14可以具有用于将模拟像素信号转换为数字格式的模数(A/D)转换功能。
例如,水平驱动单元15包括移位寄存器和地址译码器。水平驱动单元15顺序逐列扫描像素阵列11。通过水平驱动单元15执行的选择扫描操作,每一感光像素列被列处理单元14顺序地处理,并且被顺序输出。
例如,控制单元13根据操作模式配置时序信号,利用多种时序信号来控制垂直驱动单元12、列处理单元14和水平驱动单元15协同工作。
图3是本申请实施方式中一种感光像素110的示意图。感光像素110包括像素电路111、滤光片112、及微透镜113。沿感光像素110的收光方向,微透镜113、滤光片112、及像素电路111依次设置。微透镜113用于汇聚光线,滤光片112用于供某一波段的光线通过并过滤掉其余波段的光线。像素电路111用于将接收到的光线转换为电信号,并将生成的电信号提供给图2所示的列处理单元14。
图4是本申请实施方式中一种感光像素110的像素电路111的示意图。图4中像素电路111可应用在图2所示的像素阵列11内的每个感光像素110(图3所示)中。下面结合图2至图4对像素电路111的工作原理进行说明。
如图4所示,像素电路111包括光电转换元件1111(例如,光电二极管)、曝光控制电路(例如,转移晶体管1112)、复位电路(例如,复位晶体管1113)、放大电路(例如,放大晶体管1114)和选择电路(例如,选择晶体管1115)。在本申请的实施例中,转移晶体管1112、复位晶体管1113、放大晶体管1114和选择晶体管1115例如是MOS管,但不限于此。
例如,光电转换元件1111包括光电二极管,光电二极管的阳极例如连接到地。光电二极管将所接收的光转换为电荷。光电二极管的阴极经由曝光控制电路(例如,转移晶体管1112)连接到浮动扩散单元FD。浮动扩散单元FD与放大晶体管1114的栅极、复位晶体管1113的源极连接。
例如,曝光控制电路为转移晶体管1112,曝光控制电路的控制端TG为转移晶体管1112的栅极。当有效电平(例如,VPIX电平)的脉冲通过曝光控制线传输到转移晶体管1112的栅极时,转移晶体管1112导通。转移晶体管1112将光电二极管光电转换的电荷传输到浮动扩散单元FD。
例如,复位晶体管1113的漏极连接到像素电源VPIX。复位晶体管113的源极连接到浮动扩散单元FD。在电荷被从光电二极管转移到浮动扩散单元FD之前,有效复位电平的脉冲经由复位线传输到复位晶体管113的栅极,复位晶体管113导通。复位晶体管113将浮动扩散单元FD复位到像素电源VPIX。
例如,放大晶体管1114的栅极连接到浮动扩散单元FD。放大晶体管1114的漏极连接到像素电源VPIX。在浮动扩散单元FD被复位晶体管1113复位之后,放大晶体管1114经由选择晶体管 1115通过输出端OUT输出复位电平。在光电二极管的电荷被转移晶体管1112转移之后,放大晶体管1114经由选择晶体管1115通过输出端OUT输出信号电平。
例如,选择晶体管1115的漏极连接到放大晶体管1114的源极。选择晶体管1115的源极通过输出端OUT连接到图2中的列处理单元14。当有效电平的脉冲通过选择线被传输到选择晶体管1115的栅极时,选择晶体管1115导通。放大晶体管1114输出的信号通过选择晶体管1115传输到列处理单元14。
需要说明的是,本申请实施例中像素电路111的像素结构并不限于图4所示的结构。例如,像素电路111也可以具有三晶体管像素结构,其中放大晶体管1114和选择晶体管1115的功能由一个晶体管完成。例如,曝光控制电路也不局限于单个转移晶体管1112的方式,其它具有控制端控制导通功能的电子器件或结构均可以作为本申请实施例中的曝光控制电路,本申请实施方式中的单个转移晶体管1112的实施方式简单、成本低、易于控制。
图5至图10是本申请某些实施方式的像素阵列11(图2所示)中的感光像素110(图3所示)的排布示意图。感光像素110包括两类,一类为全色感光像素W,另一类为彩色感光像素。图5至图10仅示出了一个最小重复单元中的多个感光像素110的排布。对图5至图10所示的最小重复单元在行和列上多次复制,即可形成像素阵列11。每个最小重复单元均由多个全色感光像素W和多个彩色感光像素组成。每个最小重复单元包括多个子单元。每个子单元内包括多个单颜色感光像素和多个全色感光像素W。其中,图5至图8所示的最小重复单元中,每个子单元中的全色感光像素W和彩色感光像素交替设置。图9和图10所示的最小重复单元中,每个子单元中,同一行的多个感光像素110为同一类别的感光像素110;或者,同一列的多个感光像素110为同一类别的感光像素110。
具体地,例如,图5为本申请一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为4行4列16个感光像素110,子单元为2行2列4个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
例如,如图5所示,对于每个子单元,全色感光像素W和单颜色感光像素交替设置。
例如,如图5所示,子单元的类别包括三类。其中,第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A;第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B;第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元,分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中,一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1(例如图5中左上角和右下角连接的方向),两个第二类子单元UB设置在第二对角线方向D2(例如图5中右上角和左下角连接的方向)。第一对角线方向D1与第二对角线方向D2不同。例如,第一对角线和第二对角线垂直。
需要说明的是,在其他实施方式中,第一对角线方向D1也可以是右上角和左下角连接的方向,第二对角线方向D2也可以是左上角和右下角连接的方向。另外,这里的“方向”并非单一指向,可以理解为指示排布的“直线”的概念,可以有直线两端的双向指向。下文图6至图10中对第一对角线方向D1及第二对角线方向D2的解释与此处相同。
再例如,图6为本申请另一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为6行6列36个感光像素110,子单元为3行3列9个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
例如,如图6所示,对于每个子单元,全色感光像素W和单颜色感光像素交替设置。
例如,如图6所示,子单元的类别包括三类。其中,第一类子单元UA包括多个全色感光像素W和 多个第一颜色感光像素A;第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B;第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元,分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中,一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1,两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如,第一对角线和第二对角线垂直。
再例如,图7为本申请又一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为8行8列64个感光像素110,子单元为4行4列16个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
例如,如图7所示,对于每个子单元,全色感光像素W和单颜色感光像素交替设置。
例如,如图7所示,子单元的类别包括三类。其中,第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A;第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B;第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元,分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中,一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1,两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如,第一对角线和第二对角线垂直。
具体地,例如,图8为本申请再一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为4行4列16个感光像素110,子单元为2行2列4个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
图8所示的最小重复单元中感光像素110的排布与图5所示的最小重复单元中感光像素110的排布大致相同,其不同之处在于,图8中位于左下角的第二类子单元UB中的全色感光像素W与单颜色感光像素的交替顺序与图5中位于左下角的第二类子单元UB中的全色感光像素W与单颜色感光像素的交替顺序不一致,并且,图8中的第三类子单元UC中的全色感光像素W与单颜色感光像素的交替顺序与图5中位于右下角的第三类子单元UC中的全色感光像素W与单颜色感光像素的交替顺序也不一致。具体地,图5中位于左下角的第二类子单元UB中,第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第二颜色感光像素B),第二行的感光像素110的交替顺序为单颜色感光像素(即第二颜色感光像素B)、全色感光像素W;而图8中位于左下角的第二类子单元UB中,第一行的感光像素110的交替顺序为单颜色感光像素(即第二颜色感光像素B)、全色感光像素W,第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第二颜色感光像素B)。图5中位于右下角的第三类子单元UC中,第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C),第二行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W;而图8中位于右下角的第三类子单元UC中,第一行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W,第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C)。
如图8所示,图8中的第一类子单元UA中的全色感光像素W与单颜色感光像素的交替顺序与第三类子单元UC中的全色感光像W素与单颜色感光像素的交替顺序不一致。具体地,图8所示的第一类子单元CA中,第一行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第一颜色感光像素A),第二行的感光像素110的交替顺序为单颜色感光像素(即第一颜色感光像素A)、全色感 光像素W;而图8所示的第三类子单元CC中,第一行的感光像素110的交替顺序为单颜色感光像素(即第三颜色感光像素C)、全色感光像素W,第二行的感光像素110的交替顺序为全色感光像素W、单颜色感光像素(即第三颜色感光像素C)。也即是说,同一最小重复单元中,不同子单元内的全色感光像素W与彩色感光像素的交替顺序可以是一致的(如图5所示),也可以是不一致的(如图8所示)。
再例如,图9为本申请还一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为4行4列16个感光像素110,子单元为2行2列4个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
例如,如图9所示,对于每个子单元,同一行的多个感光像素110为同一类别的感光像素110。其中,同一类别的感光像素110包括:(1)均为全色感光像素W;(2)均为第一颜色感光像素A;(3)均为第二颜色感光像素B;(4)均为第三颜色感光像素C。
例如,如图9所示,子单元的类别包括三类。其中,第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A;第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B;第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元,分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中,一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1,两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如,第一对角线和第二对角线垂直。
再例如,图10为本申请还一个实施例的最小重复单元中感光像素110(图3所示)的排布示意图。其中,最小重复单元为4行4列16个感光像素110,子单元为2行2列4个感光像素110。排布方式为:
W表示全色感光像素;A表示多个彩色感光像素中的第一颜色感光像素;B表示多个彩色感光像素中的第二颜色感光像素;C表示多个彩色感光像素中的第三颜色感光像素。
例如,如图10所示,对于每个子单元,同一列的多个感光像素110为同一类别的感光像素110。其中,同一类别的感光像素110包括:(1)均为全色感光像素W;(2)均为第一颜色感光像素A;(3)均为第二颜色感光像素B;(4)均为第三颜色感光像素C。
例如,如图10所示,子单元的类别包括三类。其中,第一类子单元UA包括多个全色感光像素W和多个第一颜色感光像素A;第二类子单元UB包括多个全色感光像素W和多个第二颜色感光像素B;第三类子单元UC包括多个全色感光像素W和多个第三颜色感光像素C。每个最小重复单元包括四个子单元,分别为一个第一类子单元UA、两个第二类子单元UB及一个第三类子单元UC。其中,一个第一类子单元UA与一个第三类子单元UC设置在第一对角线方向D1,两个第二类子单元UB设置在第二对角线方向D2。第一对角线方向D1与第二对角线方向D2不同。例如,第一对角线和第二对角线垂直。
例如,在其他实施方式中,同一最小重复单元中,也可以是部分子单元内的同一行的多个感光像素110为同一类别的感光像素110,其余部分子单元内的同一列的多个感光像素110为同一类别的感光像素110。
例如,如图5至图10所示的最小重复单元中,第一颜色感光像素A可以为红色感光像素R;第二颜色感光像素B可以为绿色感光像素G;第三颜色感光像素C可以为蓝色感光像素Bu。
例如,如图5至图10所示的最小重复单元中,第一颜色感光像素A可以为红色感光像素R;第二颜色感光像素B可以为黄色感光像素Y;第三颜色感光像素C可以为蓝色感光像素Bu。
例如,如图5至图10所示的最小重复单元中,第一颜色感光像素A可以为品红色感光像素M;第二颜色感光像素B可以为青色感光像素Cy;第三颜色感光像素C可以为黄色感光像素Y。
需要说明的是,在一些实施例中,全色感光像素W的响应波段可为可见光波段(例如,400nm-760nm)。例如,全色感光像素W上设置有红外滤光片,以实现红外光的滤除。在另一些实施例中,全色感光像素W的响应波段为可见光波段和近红外波段(例如,400nm-1000nm),与图像传感器10(图1所示)中的光电转换元件1111(图4所示)的响应波段相匹配。例如,全色感光像素W可以不设置滤光片或者设置可供所有波段的光线通过的滤光片,全色感光像素W的响应波段由光电转换元件1111的响应波段确定,即两者相匹配。本申请的实施例包括但不局限于上述波段范围。
请结合图1至图3、图5及图11,在某些实施方式中,控制单元13控制像素阵列11曝光。其中,对于同一子单元中的多个感光像素110,至少一个单颜色感光像素以第一曝光时间曝光,至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光,至少一个全色感光像素W以小于或等于第一曝光时间的第三曝光时间曝光。像素阵列11中以第一曝光时间曝光的多个单颜色感光像素可以生成第一彩色信息,以第二曝光时间曝光的多个单颜色感光像素可以生成第二彩色信息,以第三曝光时间曝光的多个全色感光像素W可以生成全色信息。第一彩色信息可以形成第一彩色原始图像。第二彩色信息可以形成第二彩色原始图像。全色信息可以生成全色原始图像。
在一些实施例中,同一子单元中的部分全色感光像素W以第四曝光时间曝光,其余全色感光像素W以第三曝光时间曝光。其中,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间。
其中,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间。具体地,对于每个子单元中的(图11所示为4个)感光像素110(图3所示),一个单颜色感光像素以第一曝光时间(例如图11所示的长曝光时间L)曝光,一个单颜色感光像素以第二曝光时间(例如图11所示的短曝光时间S)曝光,一个全色感光像素W以第三曝光时间(例如图11所示的短曝光时间S)曝光,一个全色感光像素W以第四曝光时间(例如图11所示的长曝光时间L曝光)。
需要说明的是,在某些实施例中,像素阵列11的曝光过程可以是:(1)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110、以第三曝光时间曝光的感光像素110及以第四曝光时间曝光的感光像素110依次序曝光(其中四者的曝光顺序不作限制),且四者的曝光进行时间均不重叠;(2)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110、以第三曝光时间曝光的感光像素110及以第四曝光时间曝光的感光像素110依次序曝光(其中四者的曝光顺序不作限制),且四者的曝光进行时间存在部分重叠;(3)所有以较短的曝光时间曝光的感光像素110的曝光进行时间均位于以最长的曝光时间曝光的感光像素110的曝光进行时间内,例如,以第二曝光时间曝光的所有单颜色感光像素的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内,以第三曝光时间曝光的所有全色感光像素W的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内,以第四曝光时间曝光的所有全色感光像素W的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内。在本申请的具体实施例中,图像传感器10采用第(3)种曝光方式,使用该种曝光方式可以缩短像素阵列11所需要的整体曝光时间,有利于提升图像的帧率。
像素阵列11曝光结束后,图像传感器10可以输出四张原始图像,分别为:(1)第一彩色原始图像,由以长曝光时间L(第一曝光时间)曝光的多个单颜色感光像素生成的第一彩色信息组成;(2)第二彩色原始图像,由以短曝光时间S(第二曝光时间)曝光的多个单颜色感光像素生成的第二彩色信息组成;(3)第一全色原始图像,由以短曝光时间S曝光的多个全色感光像素W(第三曝光时间)生成的第一全色信息组成;(4)第二全色原始图像,由以长曝光时间L(第四曝光时间)曝光的多个全色感光像素W生成的第二全色信息组成。
请参阅图11和图16,图像传感器10获得第一彩色原始图像、第二彩色原始图像、第一全色原始图像、及第二全色原始图像之后,会将这四张图像传输至图像融合模块20,图像融合模块20对第一彩色原始图像及第二全色原始图像进行融合处理以获得第一中间图像,并且对第二彩色原始图像及第一全色原始图像进行融合处理以获得第二中间图像。
以第一彩色原始图像为例,如图12及图16所示,图像融合模块20首先分离第一彩色原始图像的色彩和亮度以获取色亮分离图像,图12中的色亮分离图像中的LIT表示亮度,CLR表示色彩。具体地,假设单颜色感光像素A为红色感光像素R,单颜色感光像素B为绿色感光像素G,单颜色感光像素C为蓝色感光像素Bu,则:(1)图像融合模块20可以将RGB空间的第一彩色原始图像转换为YCrCb空间的色亮分离图像,此时YCrCb中的Y即为色亮分离图像中的亮度LIT,YCrCb中的Cr和Cb即为色亮分离图像中的色彩CLR;(2)图像融合模块20也可以将RGB的第一彩色原始图像转换为Lab空间的色亮分离图像,此时Lab中的L即为色亮分离图像中的亮度LIT,Lab中的a和b即为色亮分离图像中的色彩CLR。需要说明的是,图12所示色亮分离图像中LIT+CLR并不表示每个像素的像素值由L和CLR相加而成,仅表示每个像素的像素值是由LIT和CLR组成。
随后,图像融合模块20融合色亮分离图像的亮度以及第二全色原始图像的亮度。示例地,第二全色原始图像中每个全色像素W的像素值即为每个全色像素的亮度值,图像融合模块20可以将将色亮分离图像中每个像素的LIT与全色中间图像中对应位置的全色像素的W相加,即可得到亮度修正后的像素值。图像融合模块20根据多个亮度修正后的像素值形成一张亮度修正后的色亮分离图像,再利用色彩空间转换将亮度修正后的色亮分离图像转换为第一中间图像。
同样地,请参阅图13和图16,图像融合模块20对第二彩色原始图像与第一全色原始图像进行融合 处理以获得第二中间图像。第二中间图像的获取过程与第一中间图像的获取过程相同,在此不作赘述。当然,图像融合模块20也可以利用其它方式进行融合处理,在此不作限制。通过图像融合模块20对彩色原始图像和全色原始图像的融合处理,可以使得提升融合后得到的中间图像的亮度。
需要说明的是,由于第一彩色原始图像是由以长曝光时间L曝光的多个单颜色感光像素生成的第一彩色信息组成,并且第二全色原始图像也是由以长曝光时间L曝光的多个全色感光像素W生成的第二全色信息组成,所以由第一彩色原始图像与第二全色原始图像经过融合处理获得的第一中间图像中的所有图像像素对应的曝光时间均为长曝光时间L。同样地,由于第二彩色原始图像是由以短曝光时间S曝光的多个单颜色感光像素生成的第二彩色信息组成,并且第一全色原始图像也是由以短曝光时间S曝光的多个全色感光像素W生成的第一全色信息组成,所以由第二彩色原始图像与第一全色原始图像经过融合处理获得的第二中间图像中的所有图像像素对应的曝光时间均为短曝光时间S。
图像融合模块20获得第一中间图像及第二中间图像后,会将这两张图像传输至高动态范围图像处理模块30进行高动态处理以获得第一高动态范围图像。示例地,请参阅图16,高动态范围图像处理模块30包括高动态范围图像处理单元31及亮度映射单元33。高动态范围图像处理单元31用于将第一中间图像及第二中间图像融合为第三高动态范围图像;亮度映射单元33用于对第三高动态范围图像进行亮度映射以得到第一高动态范围图像。
具体地,请参阅图16,高动态范围图像处理单元31对第一中间图像及第二中图像进行融合的过程可以包括亮度对齐处理。高动态范围图像处理单元31对第一中间图像及第二中间图像进行亮度对齐处理包括如下步骤:(1)识别第一中间图像中像素值大于第一预设阈值的过曝图像像素;(2)对于每一个过曝图像像素,以该过曝图像像素为中心扩展预定区域;(3)在预定区域内寻找像素值小于第一预设阈值的中间图像像素;(4)利用中间图像像素及第二中间图像对过曝图像像素的像素值进行修正;(5)利用过曝图像像素的修正后的像素值更新第一中间图像以得到亮度对齐后的第一中间图像。具体地,请结合图14,假设图像像素P12(图14中第一中间图像内标记有虚线圆圈的图像像素)的像素值V1大于第一预设阈值V0,即图像像素P12为过曝图像像素P12,则高动态范围图像处理单元31以过曝图像像素P12为中心扩展一个预定区域,例如,图14所示的3*3区域。当然,在其他实施例中,也可以是4*4区域、5*5区域、10*10区域等,在此不作限制。随后,高动态范围图像处理单元31在3*3的预定区域内寻找像素值小于第一预设阈值V0的中间图像像素,例如图14中的图像像素P21(图14中第一中间图像内标记有点画线圆圈的图像像素)的像素值V2小于第一预设阈值V0,则图像像素P21即为中间图像像素P21。随后,高动态范围图像处理单元31在第二中间图像中寻找与过曝图像像素P12及中间图像像素P21分别对应的图像像素,即图像像素P1’2’(图14中第二中间图像内标记有虚线圆圈的图像像素)和图像像素P2’1’(图14中第二中间图像内标记有点画线圆圈的图像像素),其中,图像像素P1’2’与过曝图像像素P12对应,图像像素P2’1’与中间图像像素P21对应,图像像素P1’2’的像素值为V3,图像像素P2’1’的像素值为V4。随后,根据V1’/V3=V2/V4来计算出V1’,并利用V1’的值来替换掉V1的值。由此,即可计算出过曝图像像素P12的实际像素值。高动态范围图像处理单元31对第一中间图像中的每一个过曝图像像素均执行这一亮度对齐的处理过程,即可得到亮度对齐后的第一中间图像。由于亮度对齐后的第一中间图像中的过曝图像像素的像素值经过了修正,亮度对齐后的第一中间图像中的每个图像像素的像素值均较为准确。
高动态范围处理过程中,在获取到亮度对齐的第一中间图像及第二中间图像后,高动态范围图像处理单元31可以对亮度对齐后的第一中间图像及第二中间图像进行融合以得到第三高动态彩色图像。请参阅图15,具体地,高动态范围图像处理单元31首先对亮度对齐后的第一中间图像进行运动检测,以识别亮度对齐后的第一中间图像中是否存在运动模糊区域。若亮度对齐后的第一中间图像中不存在运动模糊区域,则直接融合亮度对齐后的第一中间图像及第二中间图像以得到第一高动态范围图像。若亮度对齐后的第一中间图像中存在运动模糊区域,则将第一中间图像中的运动模糊区域剔除,只融合第二中间图像的所有区域以及亮度对齐后的第一中间图像中除运动模糊区域以外的区域以得到第一高动态范围图像。具体地,在融合亮度对齐后的第一中间图像及第二中间图像时,若亮度对齐后的第一中间图像中不存在运动模糊区域,则此时两张中间图像的融合遵循以下原则:(1)亮度对齐后的第一中间图像中,过曝区域的图像像素的像素值直接替换为第二中间图像中对应于该过曝区域的图像像素的像素值;(2)亮度对齐后的第一中间图像中,欠曝区域的图像像素的像素值为:长曝光像素值除以长短像素值比例;(3)亮度对齐后的第一中间图像中,未欠曝也未过曝区域的图像像素的像素值为:长曝光像素值除以长短像素值比例。若亮度对齐后的第一中间图像中存在运动模糊区域,则此时两张中间图像的融合除了遵循上述三个原则外,还需要遵循第(4)个原则:亮度对齐后的第一中间图像中,运动模糊区域的图像像素的 像素值直接替换为第二中间图像中对应于该运动模糊区域的图像像素的像素值。需要说明的是,对于欠曝区域以及未欠曝也未过曝区域而言,这些区域内的图像像素的像素值为长曝光像素值除以长短像素值比例,即VL/(VL/VS)=VS’,其中,VL表示长曝光像素值,VS表示短曝光像素值,VS’表示计算出来的欠曝区域以及未欠曝也未过曝区域中图像像素的像素值。VS’的信噪比会大于VS的信噪比。
通过高动态范围图像处理单元31对中间图像进行高动态范围处理,可以提升获得的图像的动态范围,提升该图像的成像效果。
当然,高动态范围图像处理单元31也可以采用其他的方法将亮度对齐后的第一中间图像及第二中间图像进行融合以得到第三高动态彩色图像。例如,高动态范围图像处理单元31还可以对亮度对齐后的第一中间图像及第二中间图像进行运动模糊检测,并对检测到的第一中间图像及第二中间图像上存在的运动模糊区域进行运动模糊的消除,以获得消除运动模糊后的第一中间图像及消除运动模糊的第二中间图像。高动态范围图像处理单元31在获取到消除运动模糊后的第一中间图像及消除运动模糊后的第二中间图像后,再对消除运动模糊后的第一中间图像及消除运动模糊后的第二中间图像进行融合,以获得具有高动态范围的第三高动态范围图像,在此不作限制。
高动态范围图像处理单元31在获得第三高动态范围图像后,将第三高动态范围图像传输至亮度映射单元33。亮度映射单元33将第三高动态范围图像进行亮度映射处理以获得第一高动态范围图像。其中,第一高动态范围图像内每个图像像素的数据的位宽要小于第三高动态范围图像内每个图像像素的数据的位宽。
示例地,数据的位宽为10bit的第一中间图像和第二中间图像经过高动态范围图像处理单元31的高动态范围处理后,可获得位宽为16bit的第三高动态范围图像。亮度映射单元33可对位宽为16bit的第三高动态范围图像进行亮度映射处理,以获得位宽为10bit的第一高动态范围图像。当然,在一些实施例中,也可以将位宽为16bit的第三高动态范围图像进行亮度映射处理,以获得位宽为12bit的第一高动态范围图像,在此不作限制。如此,通过亮度映射处理来减小高动态范围图像的数据量,从而避免图像处理器40无法处理数据量过大的高动态范围图像的问题,有利于提升图像处理器40处理高动态范围图像的速度。
高动态范围图像处理单元31可以将第一高动态范围图像传输至图像处理器40进行黑电平、去马赛克、色彩转换、镜头阴影校正、坏点补偿、全局色调映射等后续处理,以获得第二高动态范围图像。第一高动态范围图像中的多个彩色图像像素均呈拜耳阵列排布,每个图像像素的像素值仅包含一个颜色通道的信息,而第二高动态范围图像中的每一个图像像素的像素值均包含各个颜色通道的信息。
请参阅图16,高动态范围图像处理模块30还包括统计单元35,统计单元35用于处理第一中间图像及第二中间图像以获得统计数据。统计单元35获取到统计数据后,将该统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。也即是说,该图像处理器40接收到统计数据后,可以根据统计数据进行自动曝光处理和自动白平衡处理中至少一种。例如,图像处理器40根据统计数据进行自动曝光处理;或者,图像处理器40根据统计数据进行自动白平衡处理;或者,图像处理器40根据统计数据进行自动曝光处理及自动白平衡处理。如此,图像处理器40可以根据统计数据进行自动曝光及自动白平衡处理,有利于提升图像处理器40最终输出的图像的质量。
请参阅图16,高动态范围图像处理模块30还包括镜头阴影校正单元37,镜头阴影校正单元37用于校正第三高动态范围图像以得到高动态范围校正图像。具体地,高动态范围图像处理单元31将第一中间图像及第二中间图像融合为第三高动态范围图像后,镜头阴影校正单元37对第三高动态范围图像进行镜头阴影校正处理以获得高动态范围校正图像。镜头阴影校正处理的具体过程如图17所示,镜头阴影校正单元37将第三高动态范围图像进行划分,均等地分为十六个网格,十六个网格中每个网格具有一预设好的补偿系数。然后,镜头阴影校正单元37根据各网格区域邻近的或者自身及其邻近的补偿系效通过双线性插值方法对图像进行阴影矫正。R2为图示的经过镜头阴影矫正处理的第三高动态范围图像中虚线框内的像素值,R1为图示的第一彩色原始图像中的虚线框内的像素值。R2=R1*k1,k1由R1像素邻近的的网格的补偿系数1.10、1.04、1.05和1.09进行双线性插值获得。设图像的坐标记为(x,y),x从左第一个像素开始往右计数,y从上第一个像素开始往下计数,x和y均为自然数,如图像边上的标识所示。例如,R1的坐标为(3,3),则R1在各网格补偿系数图中的坐标应为(0.75,0.75)。f(x,y)表示各网格补偿系数图中坐标为(x,y)的补偿值。则f(0.75,j0.75)为R1在各网格补偿系数图中对应的补偿系数值,则有f(0.75,0.75)=(0.25)*(0.25)*f(0,0)+0.25*0.75*f(0,1)+0.75*0.25*f(1,0)+0.75*0.75f(1,1)=0.0625*1.11+0.1875*1.10+0.1875*1.09+0.5625*1.03。各网格的补偿系数在镜头阴影校正单元37进行镜头阴影 矫正处理之前已经预先设置。
镜头阴影校正单元37在获得高动态范围校正图像后,将该高动态范围校正图像传输至统计单元35。统计单元35用于处理高动态范围校正图像以获得统计数据,并将该统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理,即该统计数据提供给图像处理器40以进行自动曝光处理和自动白平衡处理中至少一种。
由于先对第三高动态范围图像进行镜头阴影校正,再处理阴影校正后的高动态范围校正图像以得到统计数据,避免了镜头阴影的影响,从而使图像处理器40根据统计数据进行的自动曝光处理和/或自动白平衡处理获得的图像的品质更高。需要说明的是,图像融合模块20及高动态范围处理模块30均集成在图像传感器10中。
综上,图16所示的高动态范围图像处理***100先通过图像融合模块20对彩色原始图像和全色原始图像进行融合以得到第一中间图像和第二中间图像,再通过高动态范围图像处理模块30对第一中间图像和第二中间图像进行高动态范围处理以得到第一高动态范围图像。由于第一高动态范围图像中的多个彩色图像像素呈拜耳阵列排布,因此,第一高动态范围图像可以直接被图像处理器40处理。
在另一些实施例中,请结合图18,当图像传感器10获得第一彩色原始图像、第二彩色原始图像、第一全色原始图像、及第二全色原始图像之后,会将这四张图像传输至高动态范围图像处理模块30,高动态范围图像处理模块30将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像。随后,高动态范围图像处理模块30再将第一高动态全色原始图像及第一高动态彩色原始图像传输至图像融合模块20进行融合处理最终获得第一高动态范围图像。
具体地,请参阅图1、图11、图18及图19,图像传感器10获得第一彩色原始图像、第二彩色原始图像、第一全色原始图像、及第二全色原始图像之后,会将这四张图像传输至高动态范围图像处理模块30,高动态范围图像处理模块30中的高动态范围图像处理单元31将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像。具体融合过程与如图16所示实施例中将第一中间图像及第二中间图像融合为第三高动态范围图像具体过程相同,在此不作赘述。
亮度映射单元33用于对第二高动态彩色原始图像进行亮度映射以得到数据量较小的第一高动态彩色原始图像,对第二高动态全色原始图像进行亮度映射以得到数据量较小的第一高动态全色原始图像。具体的亮度映射过程与图16所示实施例中将第三高动态范围图像亮度映射成第一高动态范围图像的具体过程相同,在此不作赘述。
镜头阴影校正单元37用于校正第二高动态彩色原始图像以得到高动态彩色校正图像,校正第二高动态全色原始图像以得到高动态全色校正图像。具体校正过程与图16及图17所示实施例中对第三高动态范围图像进行镜头阴影校正的过程相同,在此不作赘述。
统计单元35用于处理高动态彩色校正图像及高动态全色校正图像以获得统计数据,并将该统计数据传输至图像处理器40,使图像处理器40可根据该统计信息进行自动曝光及自动白平衡处理中的至少一种处理。当然,统计单元35也可以直接处理第一彩色原始图像、第二彩色原始图像、第一全色原始图像及第二全色原始图像以获得统计数据,并将该统计数据传输至图像处理器40,使图像处理器40可根据该统计信息进行自动曝光及自动白平衡处理中的至少一种处理。
高动态范围图像处理模块30获得第一高动态彩色原始图像与第一高动态全色原始图像后,将这两张图像传输至图像融合模块20进行融合处理以获得第一高动态范围图像。图像融合模块20将第一高动态彩色原始图像与第一高动态全色原始图像融合为第一高动态范围图像的具体过程与图12所示实施例中,将第一彩色原始图像及第二全色原始图像融合为第一中间图像的具体融合过程相同,在此不作赘述。
综上,图18所示的高动态范围图像处理***100先通过高动态范围图像处理模块30对彩色原始图像和全色原始图像进行融合以得到第一高动态彩色原始图像及第一高动态全色原始图像,再通过图像融合模块20对第一高动态彩色原始图像及第一高动态全色原始图像进行融合以得到第一高动态范围图像。由于第一高动态范围图像中的多个彩色图像像素呈拜耳阵列排布,因此,第一高动态范围图像可以直接被图像处理器40处理。
在又一些实施例中,如图20所示,像素阵列11中的所有全色感光像素W均以第三曝光时间曝光,第三曝光时间可以大于第二曝光时间,使所有全色感光像素W均以中曝光时间M曝光;或者,第三曝光时间等于第一曝光时间,使所有全色感光像素W均以长曝光时间L曝光,当然第三曝光时间也可以等于或小于第二曝光时间,使全色感光像素W以短曝光时间曝光,在此不作限制。以下以第三曝光时间大于第二曝光时间,即所有全色感光像素W均以中曝光时间M曝光为例进行说明。具体地,对于每 个子单元中的多个(图20所示为4个)感光像素110(图3所示),一个单颜色感光像素以第一曝光时间(例如图20所示的长曝光时间L)曝光,一个单颜色感光像素以第二曝光时间(例如图20所示的短曝光时间S)曝光,两个全色感光像素W均以第三曝光时间(例如图20所示的中曝光时间M)曝光。
需要说明的是,在某些实施例中,像素阵列11的曝光过程可以是:(1)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110及以第三曝光时间曝光的感光像素110依次序曝光(其中三者的曝光顺序不作限制),且三者的曝光进行时间均不重叠;(2)以第一曝光时间曝光的感光像素110、以第二曝光时间曝光的感光像素110及以第三曝光时间曝光的感光像素110依次序曝光(其中三者的曝光顺序不作限制),且三者的曝光进行时间存在部分重叠;(3)所有以较短的曝光时间曝光的感光像素110的曝光进行时间均位于以最长的曝光时间曝光的感光像素110的曝光进行时间内,例如,以第二曝光时间曝光的所有单颜色感光像素的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内,以第三曝光时间曝光的所有全色感光像素W的曝光进行时间均位于以第一曝光时间曝光的所有单颜色感光像素的曝光进行时间内。在本申请的具体实施例中,像素阵列11采用第(3)种曝光方式,使用该种曝光方式可以缩短像素阵列11所需要的整体曝光时间,有利于提升图像的帧率。
像素阵列11曝光结束后,图像传感器10可以输出三张原始图像,分别为:(1)第一彩色原始图像,由以长曝光时间L(第一曝光时间)曝光的多个单颜色感光像素生成的第一彩色信息组成;(2)第二彩色原始图像,由以短曝光时间S(第二曝光时间)曝光的多个单颜色感光像素生成的第二彩色信息组成;(3)第一全色原始图像,由以中曝光时间M(第三曝光时间)曝光的多个全色感光像素W生成的第一全色信息组成。
请参阅图18及图20,图像传感器10首先将第一彩色原始图像及第二彩色原始图像传输至高动态范围图像处理模块30进行高动态范围处理以获得第一高动态彩色原始图像,再将第一高动态彩色原始图像及第一全色原始图像传输至图像融合模块20进行融合算法处理以得到第一高动态范围图像。
具体地,请参阅图21,图像传感器10将第一彩色原始图像、第二彩色原始图像及第一全色传输至高动态范围图像处理模块30,高动态范围图像处理模块30中的高动态范围图像处理单元31将第一彩色原始图像及第二彩色原始图像融合为第二高动态彩色原始图像,具体融合过程与如图15所示实施例中将第一中间图像及第二中间图像融合为第三高动态范围图像具体过程相同,在此不作赘述。
亮度映射单元33用于对第二高动态彩色原始图像进行亮度映射以得到数据量较小的第一高动态彩色原始图像。具体过程与图16所示实施例中将第三高动态范围图像亮度映射成第一高动态范围图像的具体过程相同,在此不作赘述。
镜头阴影校正单元37用于校正第二高动态彩色原始图像以得到高动态彩色校正图像。具体校正过程与图16及17所示实施例中对第三高动态范围图像进行镜头阴影校正的过程相同。在此不作赘述。
统计单元35用于处理高动态彩色校正图像及高动态全色校正图像以获得统计数据,并将该统计数据传输至图像处理器40,使图像处理器40可根据该统计信息进行自动曝光及自动白平衡处理中的至少一种处理。当然,统计单元35也可以直接处理第一彩色原始图像、第二彩色原始图像及第一全色原始图像以获得统计数据,并将该统计数据传输至图像处理器40,使图像处理器40可根据该统计信息进行自动曝光及自动白平衡处理中的至少一种处理。
高动态范围图像处理模块30获得第一高动态彩色原始图像后,将第一高动态彩色原始图像及第一全色原始图像传输至图像融合模块20进行融合处理以获得第一高动态范围图像。具体地,请参阅图20及图22,图像传感器10获得的第一全色原始图像包括多个全色图像像素W及多个空图像像素N(NULL),其中,空图像像素既不为全色图像像素,也不为彩色图像像素,第一全色原始图像中空图像像素N所处位置可视为该位置没有图像像素,或者可以将空图像像素的像素值视为零。比较像素阵列11与全色原始图像可知,对于像素阵列11中的每一个子单元,该子单元包括两个全色图像像素W和两个彩色图像像素(彩色图像像素A、彩色图像像素B、或彩色图像像素C)。第一全色原始图像中也具有与像素阵列11中的每一个子单元对应的一个子单元,第一全色原始图像的子单元包括两个全色图像像素W和两个空图像像素N,两个空图像像素N所处位置对应像素阵列11的子单元中的两个彩色图像像素所处的位置。
图像融合模块20可以对第一全色原始图像作进一步处理得到全色中间图像。示例地,每个子单元都包括多个空图像像素N和多个全色图像像素。具体地,某些子单元包括两个空图像像素N和两个全色图像像素W。图像融合模块20可以将包括空图像像素N和全色图像像素W的子单元中的所有全色图像像素的像素值作为与该子单元中全色大像素W,以获得全色中间图像。此时的全色中间图像的分辨率与第一高动态彩色原始图像的分辨率相同,以方便全色中间图像与第一高动态彩色原始图像融合。全色中间图像与第一高动态彩色原始图像的具体融合过程与图12所示实施例中,将第一彩色原始图像及 第二全色原始图像融合为第一中间图像的具体融合过程相同,在此不作赘述。
综上,图18所示的高动态范围图像处理***100先通过高动态范围图像处理模块30对彩色原始图像和全色原始图像进行融合以得到第一高动态彩色原始图像,再通过图像融合模块20对第一高动态彩色原始图像及第一全色原始图像进行融合以得到第一高动态范围图像。由于第一高动态范围图像中的多个彩色图像像素呈拜耳阵列排布,因此,第一高动态范围图像可以直接被图像处理器40处理。
请参阅图1及图23,本申请还提供一种电子设备1000。本申请实施方式的电子设备1000包括镜头300、壳体200及上述任意一个实施方式所述的高动态范围图像处理***100。镜头300、高动态范围图像处理***100与壳体200结合。镜头300与高动态范围图像处理***100的图像传感器10配合成像。
电子设备1000可以是手机、平板电脑、笔记本电脑、智能穿戴设备(例如智能手表、智能手环、智能眼镜、智能头盔)、无人机、头显设备等,在此不作限制。
本申请实施方式的电子设备1000通过图像融合模块20及高动态范围图像处理模块30对图像传感器10输出的全色原始图像和彩色原始图像事先进行融合算法处理及高动态范围处理,以得到图像像素呈拜耳阵列排布的第一高动态范围图像,再把第一高动态范围图像输入图像处理器中完成后续处理,从而解决图像处理器40不能直接对图像像素呈非拜耳阵列排布的图像进行处理的问题。
请参阅图24,本申请还提供一种高动态范围图像处理方法。本申请实施方式的高动态范围图像处理方法用于高动态范围图像处理***100。高动态范围图像处理***100包括图像传感器10。图像传感器10包括像素阵列11。像素阵列11包括多个全色感光像素和多个彩色感光像素。彩色感光像素具有比全色感光像素更窄的光谱响应。像素阵列11包括最小重复单元,每个最小重复单元包含多个子单元。每个子单元包括多个单颜色感光像素及多个全色感光像素。高动态范围图像处理方法包括:
01:像素阵列11曝光,其中,对于同一子单元中的多个感光像素,至少一个单颜色感光像素以第一曝光时间曝光,至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光,至少一个全色感光像素以小于第一曝光时间的第三曝光时间曝光;其中,以第一曝光时间曝光的单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以第二曝光时间曝光的单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以第三曝光时间曝光的全色感光像素生成第一全色原始图像;及
02:对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,第一高动态范围图像包含多个彩色图像像素,多个彩色图像像素呈拜耳阵列排布,第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间,并且以第四曝光时间曝光的单颜色感光像素生成的第二全色信息得到第二全色原始图像;对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:对第一彩色原始图像与第二全色原始图像融合为第一中间图像,将第二彩色原始图像与第一全色原始图像融合为第二中间图像;及将第一中间图像与第二中间图像融合为第一高动态范围图像。
在某些实施方式中,将第一中间图像与第二中间图像融合为第一高动态范围图像包括:将第一中间图像及第二中间图像融合为第三高动态范围图像;及对所第三高动态范围图像进行亮度映射以得到第一高动态范围图像。
在某些实施方式中,高动态范围图像处理方法还包括:将第一中间图像及第二中间图像融合为第三高动态范围图像;对第三高动态范围图像以得到高动态范围校正图像;及处理高动态范围校正图像以获得统计数据,统计数据提供给图像处理器以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,高动态范围图像处理方法还包括:处理第一中间图像及第二中间图像以获得统计数据,统计数据提供给图像处理器以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,同一子单元中的部分全色感光像素以第四曝光时间曝光,其余全色感光像素以第三曝光时间曝光,第四曝光时间小于或等于第一曝光时间,且大于第三曝光时间,并且以第四曝光时间曝光的单颜色感光像素生成的第二全色信息得到第二全色原始图像。对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像;及将第一高动态彩色原始图像与第一高动态全色原始图像融合为第一高动态范围图像。
在某些实施方式中,将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第一高动态全色原始图像包括:将第一彩色原始图像与第 二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;及对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像,对第二高动态全色原始图像进行亮度映射以得到第一高动态全色原始图像。
在某些实施方式中,高动态范围图像处理方法包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像,将第一全色原始图像与第二全色原始图像融合为第二高动态全色原始图像;校正第二高动态彩色原始图像以得到高动态彩色校正图像,校正第二高动态全色原始图像以得到高动态全色校正图像;及处理高动态彩色校正图像及高动态全色校正图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
在某些实施方式中,同一子单元中的全部全色感光像素以第三曝光时间曝光;对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像包括:将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像;及将第一高动态彩色原始图像与第一全色原始图像融合为第一高动态范围图像。
在某些实施方式中,第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像;及对第二高动态彩色原始图像进行亮度映射以得到第一高动态彩色原始图像。
在某些实施方式中,高动态范围图像处理方法包括:将第一彩色原始图像与第二彩色原始图像融合为第二高动态彩色原始图像;校正第二高动态彩色原始图像以得到高动态彩色校正图像;及处理高动态彩色校正图像及第一全色原始图像以获得统计数据,统计数据提供给图像处理器40以进行自动曝光处理和/或自动白平衡处理。
上述任意一项实施方式的高动态范围图像处理方法的具体实施过程与前述高动态范围图像处理***100获得高动态范围图像的具体实施过程相同,在此不再展开说明。
请参阅图25,本申请还提供一种包含计算机程序的非易失性计算机可读存储介质400。该计算机程序被处理器60执行时,使得处理器60执行上述任意一个实施方式所述的高动态范围图像处理方法。
例如,请参阅图1、图3、图11及图25,计算机程序被处理器60执行时,使得处理器60执行以下步骤:
像素阵列11曝光,其中,对于同一子单元中的多个感光像素,至少一个单颜色感光像素以第一曝光时间曝光,至少一个单颜色感光像素以小于第一曝光时间的第二曝光时间曝光,至少一个全色感光像素以小于第一曝光时间的第三曝光时间曝光;其中,以第一曝光时间曝光的单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以第二曝光时间曝光的单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以第三曝光时间曝光的全色感光像素生成第一全色原始图像;及
对第一彩色原始图像、第二彩色原始图像及第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,第一高动态范围图像包含多个彩色图像像素,多个彩色图像像素呈拜耳阵列排布,第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
再例如,请参阅图25,计算机程序被处理器60执行时,使得处理器60执行以下步骤:
将第一彩色原始图像与第二彩色原始图像融合为第一高动态彩色原始图像;及
将第一高动态彩色原始图像与第一全色原始图像融合为第一高动态范围图像。
在本说明书的描述中,参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施方式或示例中。在本说明书中,对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。此外,在不相互矛盾的情况下,本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本申请的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本申请的实施例所属技术领域的技术人员所理解。
尽管上面已经示出和描述了本申请的实施方式,可以理解的是,上述实施方式是示例性的,不能理解为对本申请的限制,本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。
Claims (25)
- 一种高动态范围图像处理***,其特征在于,包括图像传感器、图像融合模块及高动态范围图像处理模块;所述图像传感器包括像素阵列,所述像素阵列包括多个全色感光像素和多个彩色感光像素,所述彩色感光像素具有比所述全色感光像素更窄的光谱响应,所述像素阵列包括最小重复单元,每个所述最小重复单元包含多个子单元,每个所述子单元包括多个单颜色感光像素及多个全色感光像素,所述图像传感器中的像素阵列曝光,其中,对于同一所述子单元中的多个感光像素,至少一个所述单颜色感光像素以第一曝光时间曝光,至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光,至少一个所述全色感光像素以小于所述第一曝光时间的第三曝光时间曝光;其中,以所述第一曝光时间曝光的所述单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以所述第二曝光时间曝光的所述单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以所述第三曝光时间曝光的所述全色感光像素生成第一全色原始图像;所述图像融合模块及所述高动态范围图像处理模块用于对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,所述第一高动态范围图像包含多个彩色图像像素,多个所述彩色图像像素呈拜耳阵列排布,所述第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
- 根据权利要求1所述的高动态范围图像处理***,其特征在于,同一所述子单元中的部分所述全色感光像素以第四曝光时间曝光,其余所述全色感光像素以所述第三曝光时间曝光,所述第四曝光时间小于或等于所述第一曝光时间,且大于所述第三曝光时间;所述图像融合模块用于将所述第一彩色原始图像与第二全色原始图像融合为第一中间图像,将所述第二彩色原始图像与所述第一全色原始图像融合为第二中间图像,以所述第四曝光时间曝光的所述单颜色感光像素生成的第二全色信息得到所述第二全色原始图像;所述高动态范围图像处理模块用于将所述第一中间图像与所述第二中间图像融合为所述第一高动态范围图像。
- 根据权利要求2所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块包括高动态范围图像处理单元及亮度映射单元;所述高动态范围图像处理单元用于将所述第一中间图像及所述第二中间图像融合为第三高动态范围图像;所述亮度映射单元用于对所述第三高动态范围图像进行亮度映射以得到所述第一高动态范围图像。
- 根据权利要求2所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块包括高动态范围图像处理单元、镜头阴影校正单元及统计单元;所述高动态范围图像处理单元用于将所述第一中间图像及所述第二中间图像融合为第三高动态范围图像;所述镜头阴影校正单元用于校正所述第三高动态范围图像以得到高动态范围校正图像;所述统计单元用于处理所述高动态范围校正图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求2所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块包括统计单元,所述统计单元用于处理所述第一中间图像及所述第二中间图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求1所述的高动态范围图像处理***,其特征在于,同一所述子单元中的部分所述全色感光像素以第四曝光时间曝光,其余所述全色感光像素以所述第三曝光时间曝光,所述第四曝光时间小于或等于所述第一曝光时间,且大于所述第三曝光时间;所述高动态范围图像处理模块用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第一高动态全色原始图像;所述图像融合模块用于将所述第一高动态彩色原始图像与所述第一高动态全色原始图像融合为所述第一高动态范围图像。
- 根据权利要求6所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块还包括高动态范围图像处理单元及亮度映射单元;所述高动态范围图像处理单元用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第二高动态全色原始图像;所述亮度映射单元用于对所述第二高动态彩色原始图像进行亮度映射以得到所述第一高动态彩色原始图像,对所述第二高动态全色原始图像进行亮度映射以得到所述第一高动态全色原始图像。
- 根据权利要求6所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块还包括高动态范围图像处理单元、镜头阴影校正单元及统计单元;所述高动态范围图像处理单元用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第二高动态全色原始图像;所述镜头阴影校正单元用于校正所述第二高动态彩色原始图像以得到高动态彩色校正图像,校正所述第二高动态全色原始图像以得到高动态全色校正图像;所述统计单元用于处理所述高动态彩色校正图像及所述高动态全色校正图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求1所述的高动态范围图像处理***,其特征在于,同一所述子单元中的全部所述全色感光像素以第三曝光时间曝光;所述高动态范围图像处理模块用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像;所述图像融合模块用于将所述第一高动态彩色原始图像与所述第一全色原始图像融合为所述第一高动态范围图像。
- 根据权利要求9所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块还包括高动态范围图像处理单元及亮度映射单元;所述高动态范围图像处理单元用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像;所述亮度映射单元用于对所述第二高动态彩色原始图像进行亮度映射以得到所述第一高动态彩色原始图像。
- 根据权利要求9所述的高动态范围图像处理***,其特征在于,所述高动态范围图像处理模块还包括高动态范围图像处理单元、镜头阴影校正单元及统计单元;所述高动态范围图像处理单元用于将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像;所述镜头阴影校正单元用于校正所述第二高动态彩色原始图像以得到高动态彩色校正图像;所述统计单元用于处理所述高动态彩色校正图像及所述第一全色原始图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求1所述的高动态范围图像处理***,其特征在于,所述图像融合模块及所述高动态范围图像处理模块均集成在所述图像传感器中。
- 一种高动态范围图像处理方法,用于高动态范围图像处理***,其特征在于,所述高动态范围图像处理***包括图像传感器,所述图像传感器包括像素阵列,所述像素阵列包括多个全色感光像素和多个彩色感光像素,所述彩色感光像素具有比所述全色感光像素更窄的光谱响应,所述像素阵列包括最小重复单元,每个所述最小重复单元包含多个子单元,每个所述子单元包括多个单颜色感光像素及多个全色感光像素;所述高动态范围图像处理方法包括:所述像素阵列曝光,其中,对于同一所述子单元中的多个感光像素,至少一个所述单颜色感光像素以第一曝光时间曝光,至少一个所述单颜色感光像素以小于所述第一曝光时间的第二曝光时间曝光,至少一个所述全色感光像素以小于所述第一曝光时间的第三曝光时间曝光;其中,以所述第一曝光时间曝光的所述单颜色感光像素生成的第一彩色信息得到第一彩色原始图像,以所述第二曝光时间曝光的所述单颜色感光像素生成的第二彩色信息得到第二彩色原始图像,以所述第三曝光时间曝光的所述全色感光像素生成第一全色原始图像;及对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,所述第一高动态范围图像包含多个彩色图像像素,多个所述彩色图像像素呈拜耳阵列排布,所述第一高动态范围图像由图像处理器处理以得到第二高动态范围图像。
- 根据权利要求13所述的高动态范围图像处理方法,其特征在于,同一所述子单元中的部分所述全色感光像素以第四曝光时间曝光,其余所述全色感光像素以所述第三曝光时间曝光,所述第四曝光时间小于或等于所述第一曝光时间,且大于所述第三曝光时间,并且以所述第四曝光时间曝光的所述单颜色感光像素生成的第二全色信息得到所述第二全色原始图像;所述对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,包括:对所述第一彩色原始图像与第二全色原始图像融合为第一中间图像,将所述第二彩色原始图像与所述第一全色原始图像融合为第二中间图像;及将所述第一中间图像与所述第二中间图像融合为所述第一高动态范围图像。
- 根据权利要求14所述的高动态范围图像处理方法,其特征在于,所述将所述第一中间图像与所述第二中间图像融合为所述第一高动态范围图像包括:将所述第一中间图像及所述第二中间图像融合为第三高动态范围图像;及对所述第三高动态范围图像进行亮度映射以得到所述第一高动态范围图像。
- 根据权利要求14所述的高动态范围图像处理方法,其特征在于,所述高动态范围图像处理方法,还包括:将所述第一中间图像及所述第二中间图像融合为第三高动态范围图像;对所述第三高动态范围图像以得到高动态范围校正图像;及处理所述高动态范围校正图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求14所述的高动态范围图像处理方法,其特征在于,所述高动态范围图像处理方法,还包括:处理所述第一中间图像及所述第二中间图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求13所述的高动态范围图像处理方法,其特征在于,同一所述子单元中的部分所述全色感光像素以第四曝光时间曝光,其余所述全色感光像素以所述第三曝光时间曝光,所述第四曝光时间小于或等于所述第一曝光时间,且大于所述第三曝光时间,并且以所述第四曝光时间曝光的所述单颜色感光像素生成的第二全色信息得到所述第二全色原始图像;所述对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第一高动态全色原始图像;及将所述第一高动态彩色原始图像与所述第一高动态全色原始图像融合为所述第一高动态范围图像。
- 根据权利要求18所述的高动态范围图像处理方法,其特征在于,所述将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第一高动态全色原始图像,包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第二高动态全色原始图像;及对所述第二高动态彩色原始图像进行亮度映射以得到所述第一高动态彩色原始图像,对所述第二高动态全色原始图像进行亮度映射以得到所述第一高动态全色原始图像。
- 根据权利要求18所述的高动态范围图像处理方法,其特征在于,所述将高动态范围图像处理方法,还包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像,将所述第一全色原始图像与所述第二全色原始图像融合为第二高动态全色原始图像;校正所述第二高动态彩色原始图像以得到高动态彩色校正图像,校正所述第二高动态全色原始图像以得到高动态全色校正图像;及处理所述高动态彩色校正图像及所述高动态全色校正图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 根据权利要求13所述的高动态范围图像处理方法,其特征在于,同一所述子单元中的全部所述全色感光像素以第三曝光时间曝光;所述对所述第一彩色原始图像、所述第二彩色原始图像及所述第一全色原始图像进行融合算法处理及高动态范围处理以得到第一高动态范围图像,包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像;及将所述第一高动态彩色原始图像与所述第一全色原始图像融合为所述第一高动态范围图像。
- 根据权利要求21所述的高动态范围图像处理方法,其特征在于,所述将所述第一彩色原始图像与所述第二彩色原始图像融合为第一高动态彩色原始图像,包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像;及对所述第二高动态彩色原始图像进行亮度映射以得到所述第一高动态彩色原始图像。
- 根据权利要求21所述的高动态范围图像处理方法,其特征在于,所述高动态范围图像处理方法,还包括:将所述第一彩色原始图像与所述第二彩色原始图像融合为第二高动态彩色原始图像;校正所述第二高动态彩色原始图像以得到高动态彩色校正图像;及处理所述高动态彩色校正图像及所述第一全色原始图像以获得统计数据,所述统计数据提供给所述图像处理器以进行自动曝光处理和/或自动白平衡处理。
- 一种电子设备,其特征在于,包括:镜头;壳体;及权利要求1至12任意一项所述的高动态范围图像处理***,所述镜头、所述高动态范围图像处理***与所述壳体结合,所述镜头与所述高动态范围图像处理***的图像传感器配合成像。
- 一种包含计算机程序的非易失性计算机可读存储介质,其特征在于,所述计算机程序被处理器执行时,使得所述处理器执行权利要求13至23任意一项所述的高动态范围图像处理方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010259292.6 | 2020-04-03 | ||
CN202010259292.6A CN111479071B (zh) | 2020-04-03 | 2020-04-03 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021196553A1 true WO2021196553A1 (zh) | 2021-10-07 |
Family
ID=71749629
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/119959 WO2021196553A1 (zh) | 2020-04-03 | 2020-10-09 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111479071B (zh) |
WO (1) | WO2021196553A1 (zh) |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111479071B (zh) * | 2020-04-03 | 2021-05-07 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
CN111970459B (zh) * | 2020-08-12 | 2022-02-18 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
CN111970460B (zh) * | 2020-08-17 | 2022-05-20 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
CN111970461B (zh) * | 2020-08-17 | 2022-03-22 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
WO2022109802A1 (en) * | 2020-11-24 | 2022-06-02 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Color imaging system |
KR20220084578A (ko) * | 2020-12-14 | 2022-06-21 | 에스케이하이닉스 주식회사 | 이미지 센싱 장치 |
CN114697537B (zh) * | 2020-12-31 | 2024-05-10 | 浙江清华柔性电子技术研究院 | 图像获取方法、图像传感器及计算机可读存储介质 |
CN112887571B (zh) * | 2021-01-27 | 2022-06-10 | 维沃移动通信有限公司 | 图像传感器、摄像模组和电子设备 |
CN113676635B (zh) * | 2021-08-16 | 2023-05-05 | Oppo广东移动通信有限公司 | 高动态范围图像的生成方法、装置、电子设备和存储介质 |
CN115883974B (zh) * | 2023-03-08 | 2023-05-30 | 淄博凝眸智能科技有限公司 | 基于分块曝光的hdr图像生成方法、***及可读介质 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105578065A (zh) * | 2015-12-18 | 2016-05-11 | 广东欧珀移动通信有限公司 | 高动态范围图像的生成方法、拍照装置和终端 |
CN107635118A (zh) * | 2014-09-19 | 2018-01-26 | 豪威科技股份有限公司 | 彩色滤光器阵列、图像传感器以及用于减少光谱串扰的方法 |
US20180309949A1 (en) * | 2017-04-21 | 2018-10-25 | Dartmouth College | Quanta Image Sensor with Polarization-Sensitive Jots |
CN110740272A (zh) * | 2019-10-31 | 2020-01-31 | Oppo广东移动通信有限公司 | 图像采集方法、摄像头组件及移动终端 |
CN111479071A (zh) * | 2020-04-03 | 2020-07-31 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7688368B2 (en) * | 2006-01-27 | 2010-03-30 | Eastman Kodak Company | Image sensor with improved light sensitivity |
US8045024B2 (en) * | 2009-04-15 | 2011-10-25 | Omnivision Technologies, Inc. | Producing full-color image with reduced motion blur |
US8237831B2 (en) * | 2009-05-28 | 2012-08-07 | Omnivision Technologies, Inc. | Four-channel color filter array interpolation |
US8203615B2 (en) * | 2009-10-16 | 2012-06-19 | Eastman Kodak Company | Image deblurring using panchromatic pixels |
-
2020
- 2020-04-03 CN CN202010259292.6A patent/CN111479071B/zh active Active
- 2020-10-09 WO PCT/CN2020/119959 patent/WO2021196553A1/zh active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107635118A (zh) * | 2014-09-19 | 2018-01-26 | 豪威科技股份有限公司 | 彩色滤光器阵列、图像传感器以及用于减少光谱串扰的方法 |
CN105578065A (zh) * | 2015-12-18 | 2016-05-11 | 广东欧珀移动通信有限公司 | 高动态范围图像的生成方法、拍照装置和终端 |
US20180309949A1 (en) * | 2017-04-21 | 2018-10-25 | Dartmouth College | Quanta Image Sensor with Polarization-Sensitive Jots |
CN110740272A (zh) * | 2019-10-31 | 2020-01-31 | Oppo广东移动通信有限公司 | 图像采集方法、摄像头组件及移动终端 |
CN111479071A (zh) * | 2020-04-03 | 2020-07-31 | Oppo广东移动通信有限公司 | 高动态范围图像处理***及方法、电子设备和可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN111479071B (zh) | 2021-05-07 |
CN111479071A (zh) | 2020-07-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021196553A1 (zh) | 高动态范围图像处理***及方法、电子设备和可读存储介质 | |
WO2021179806A1 (zh) | 图像获取方法、成像装置、电子设备及可读存储介质 | |
WO2021196554A1 (zh) | 图像传感器、处理***及方法、电子设备和存储介质 | |
WO2021208593A1 (zh) | 高动态范围图像处理***及方法、电子设备和存储介质 | |
WO2021212763A1 (zh) | 高动态范围图像处理***及方法、电子设备和可读存储介质 | |
WO2022088311A1 (zh) | 图像处理方法、摄像头组件及移动终端 | |
WO2022007469A1 (zh) | 图像获取方法、摄像头组件及移动终端 | |
WO2021223364A1 (zh) | 高动态范围图像处理***及方法、电子设备和可读存储介质 | |
WO2021179805A1 (zh) | 图像传感器、摄像头组件、移动终端及图像获取方法 | |
WO2021103818A1 (zh) | 图像传感器、控制方法、摄像头组件及移动终端 | |
WO2022036817A1 (zh) | 图像处理方法、图像处理***、电子设备及可读存储介质 | |
US11902674B2 (en) | Image acquisition method, camera assembly, and mobile terminal | |
WO2021159944A1 (zh) | 图像传感器、摄像头组件及移动终端 | |
CN112738493B (zh) | 图像处理方法、图像处理装置、电子设备及可读存储介质 | |
CN111970459B (zh) | 高动态范围图像处理***及方法、电子设备和可读存储介质 | |
CN112822475B (zh) | 图像处理方法、图像处理装置、终端及可读存储介质 | |
WO2022088310A1 (zh) | 图像处理方法、摄像头组件及移动终端 | |
CN111031297B (zh) | 图像传感器、控制方法、摄像头组件和移动终端 | |
WO2021046691A1 (zh) | 图像采集方法、摄像头组件及移动终端 | |
WO2022141743A1 (zh) | 图像处理方法、图像处理***、电子设备及可读存储介质 | |
US20220279108A1 (en) | Image sensor and mobile terminal | |
CN112235485B (zh) | 图像传感器、图像处理方法、成像装置、终端及可读存储介质 | |
CN112738494B (zh) | 图像处理方法、图像处理***、终端设备及可读存储介质 | |
CN114424517B (zh) | 图像传感器、控制方法、摄像头组件及移动终端 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20929197 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20929197 Country of ref document: EP Kind code of ref document: A1 |