WO2022185345A2 - Optimal color filter array and a demosaicing method thereof - Google Patents
Optimal color filter array and a demosaicing method thereof Download PDFInfo
- Publication number
- WO2022185345A2 WO2022185345A2 PCT/IN2022/050667 IN2022050667W WO2022185345A2 WO 2022185345 A2 WO2022185345 A2 WO 2022185345A2 IN 2022050667 W IN2022050667 W IN 2022050667W WO 2022185345 A2 WO2022185345 A2 WO 2022185345A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- color
- image
- cfa
- color filter
- image sensor
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 239000003086 colorant Substances 0.000 claims description 9
- 230000003595 spectral effect Effects 0.000 claims description 9
- 230000001131 transforming effect Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 230000006978 adaptation Effects 0.000 claims 1
- 241001428800 Cell fusing agent virus Species 0.000 abstract description 4
- 208000036971 interstitial lung disease 2 Diseases 0.000 abstract description 4
- 230000003287 optical effect Effects 0.000 abstract description 2
- 206010034960 Photophobia Diseases 0.000 description 5
- 208000013469 light sensitivity Diseases 0.000 description 5
- 238000002834 transmittance Methods 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 206010034972 Photosensitivity reaction Diseases 0.000 description 1
- 238000000205 computational method Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000036211 photosensitivity Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
Definitions
- Optimal color filter array and a demosaicing method thereof [0001] A method for capturing raw image data for a full-color image by a digital image sensor is provided having a color filter array (CFA), the color filter elements of which may transmit a combination of plural colors, unlike conventional CFAs which allow only one color per color filter element.
- CFA color filter array
- An image demosaicing method for demosaicing the color information is described in detail.
- the field of present invention is color filter array (CFA).
- CFA color filter array
- the present invention provides a method to capture accurate color information with improved light sensitivity, and a demosaicing method to generate full color image from the captured color information which exhibit better image quality and lower computational complexity over conventional methods and systems.
- image sensors comprise an array of photosensitive pixels that are often arranged in a regular pattern of minimum repeat units.
- a color filter array pattern is typically fabricated on the pixel array to capture color information, in which case different filter elements are associated with different color-channels.
- the color filters inevitably reduces the amount of light reaching each pixel, thereby reducing the photosensitivity of each pixel.
- Image demosaicing (also color reconstruction) is an image processing stage to convert the raw image data received from image sensor into into full color RGB data by interpolation.
- the semiconductor imaging sensor available in digital color cameras is a grayscale sensor that is not capable of detecting different colors.
- a color filter array is overlaid the semiconductor imaging sensor.
- the CFA is a grid of color filters overlaying the imaging sensor so that each sensor element receives light corresponding to a single color component. In this way a mosaic image is captured where image elements have intensity value for one of three or more color channels (Red, Green and Blue for example).
- Demosaicing is a process whereby intensity values for the other two channels are estimated for each of the image elements.
- the present disclosure relates to color filter arrays. More specifically, and without limitations, this disclosure relates to a method for capturing color information via a digital pixel image sensor equipped with a CFA and demosaicing of the raw image sensor data to obtain a full-color image.
- the method includes a CFA, color filter elements of which may transmit a combination of plural different colors, unlike most conventional CFAs that allow image elements to store intensity values in one of three or more color channels (Red, Green and Blue for example).
- the raw image data captured by the image sensor may be deemed a superposition of arrays of pixel values of plural color components instead of discrete arrays of pixel values, each indicating a raw intensity of one of the component colors.
- the disclosure also describes, inter alia, a demosaicing method, adapted to reconstruct a full color image from the color information obtained via image sensor overlaid with the new CFA.
- the raw image data obtained by described digital image sensor is represented as systems of linear equations inN variables, wherein theN is number of color components in color model being used.
- the demosaicing method for each particular pixel represented in the CFA data, calculates a new value for the particular pixel by passing pixel values of all pixels in the minimum repeat unit through an algorithm.
- an improved version of the Bayer CFA is provided.
- the new CFA has the special feature that each color filter element in the CFA pattern transmits a combination of the three RGB colors. This is unlike Bayer CFA which transmit a single component of RGB colors for each color filter element.
- the light intensity transmitted through the new CFA over underlying pixel group is significantly increased, thereby improving light sensitivity of image sensor.
- a special demosaicing method is provided for transforming the input pixel values to the corresponding output RGB color values.
- this example can be further generalized to all the other known CFA patterns and may be used in further research studies.
- the demosaicing process reconstructs a full-color image from incomplete color samples output from an image sensor overlaid with a CFA by estimating the missing color values for each pixel in a raw capture (see, for example, Guenter US20040001641A1). This is required due to the fact that most raw captures record a single color value for each image pixel.
- demosaicing does not specifically address fundamental undersampling in typical camera designs.
- An object of this invention is to capture color information at a lower expense of intensity of incoming light than conventional CFAs.
- Method presented herein allows for determination of spectral transmittance of each color filter element in the CFA pattern corresponding to maximum intensity of light transmitted through the CFA.
- the supplemental incoming light enhances light sensitivity.
- the centerpiece of this invention is that the increase in incident light intensity as well as the following demosaicing operation requires no additional compromise on image quality, and that the manufacturing can be carried out at no extra cost and complexity.
- a sensor pixel is representable by the matrix equation
- matrices denote intensity of different color components c k of the color model being used in the incident light, color filter transmittance corresponding to each color component c k , the pixel intensity value, respectively; and is the quantization function for transforming the input original image data to quantized image data. Since each photosite produce an electrical charge directly proportional to the light intensity it receives with no wavelength specificity, arc independent variables.
- J axa is also the variable matrix representing color information about the pixel group.
- C 1 xN can be determined by solving Equation (1) using Cramer’s rule (see Azamat, Akhtyamov, et al.) under condition that total number of unique F i,j ⁇ H axb (representing independent linear equations) are equal to or greater than N (total number of unknown variables ) ⁇
- a full-color image can be obtained by applying any known demosaicing method to matrix P axb , which is defined by the conditional statement
- Optimal H aXb corresponding to a desired spectral responses can be formulated through computational methods by substituting the spectral responses into C 1 xN , and then solving Equation (1) for H aXb to maximize ⁇ i,j P i,j .
- R,G,B are desired spectral responses of Bayer filter as shown in FIG. 1, wherein
- F 1,1 the actual spectral transmission characteristic of F 1,1 , defined as , is obtained by applying min-max normalization method to perform [0, 1] norma lization on , as shown in FIG. 2. Therefore, F 1,2 can be calculated by using the following formula:
- the light intensity transmitted through the CFA pattern on underlying pixel group is which is equivalent to about 148% as much light sensitivity as available via the Bayer filter.
- the final RGB color values in C 1 x3 are based on weighted average of RGB pixel values of all pixels of the minimum repeat unit, and thus color accuracy and color consistency in final full-color image is better than when measured by Bayer image sensor, wherein each color component value is measured at a single camera-sensor pixel location, and then interpolated for neighboring pixels.
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
A method for capturing digital color image data is provided wherein one or more color filter element(s) in color filter array (CFA) pattern is/are optical mix of two or more color filters, each adapted to transmit a single color component.The obtained raw image sensor data comprises image elements with intensity values in plural color channels. This is unlike conventional CFAs, wherein each image element in raw image sensor data has intensity value in one of the color channels (Red, Green or Blue for example). The final sensor image data may be deemed a tradeoff between color image and grayscale image. A CFA image demosaicing method is described to convert raw image sensor data where image elements have grayscale intensity value into a full-color image where image elements have actual intensity values in all color channels.
Description
Description
Title of Invention : Optimal color filter array and a demosaicing method thereof [0001] A method for capturing raw image data for a full-color image by a digital image sensor is provided having a color filter array (CFA), the color filter elements of which may transmit a combination of plural colors, unlike conventional CFAs which allow only one color per color filter element. An image demosaicing method for demosaicing the color information is described in detail.
Technical Field
[0002] The field of present invention is color filter array (CFA). In particular the present invention provides a method to capture accurate color information with improved light sensitivity, and a demosaicing method to generate full color image from the captured color information which exhibit better image quality and lower computational complexity over conventional methods and systems.
Background Art
[0003] Typically, image sensors comprise an array of photosensitive pixels that are often arranged in a regular pattern of minimum repeat units. A color filter array pattern is typically fabricated on the pixel array to capture color information, in which case different filter elements are associated with different color-channels. The color filters inevitably reduces the amount of light reaching each pixel, thereby reducing the photosensitivity of each pixel.
[0004] Image demosaicing (also color reconstruction) is an image processing stage to convert the raw image data received from image sensor into into full color RGB data by interpolation. However, the semiconductor imaging sensor available in digital color cameras is a grayscale sensor that is not capable of detecting different colors. To enable color images to be captured, a color filter array is overlaid the semiconductor imaging sensor. The CFA is a grid of color filters overlaying the imaging sensor so that each sensor element receives light corresponding to a single color component. In this way a mosaic image is captured where image elements have intensity value for one of three or more color channels (Red, Green and Blue for example). Demosaicing is a process whereby intensity values for the other two channels are estimated for each of the image elements.
Summary of Invention
[0005] The present disclosure relates to color filter arrays. More specifically, and without limitations, this disclosure relates to a method for capturing color information via a digital pixel image sensor equipped
with a CFA and demosaicing of the raw image sensor data to obtain a full-color image. The method includes a CFA, color filter elements of which may transmit a combination of plural different colors, unlike most conventional CFAs that allow image elements to store intensity values in one of three or more color channels (Red, Green and Blue for example). The raw image data captured by the image sensor may be deemed a superposition of arrays of pixel values of plural color components instead of discrete arrays of pixel values, each indicating a raw intensity of one of the component colors.
[0006] The disclosure also describes, inter alia, a demosaicing method, adapted to reconstruct a full color image from the color information obtained via image sensor overlaid with the new CFA. The raw image data obtained by described digital image sensor is represented as systems of linear equations inN variables, wherein theN is number of color components in color model being used. The demosaicing method, for each particular pixel represented in the CFA data, calculates a new value for the particular pixel by passing pixel values of all pixels in the minimum repeat unit through an algorithm.
[0007] In one example, an improved version of the Bayer CFA is provided. The new CFA has the special feature that each color filter element in the CFA pattern transmits a combination of the three RGB colors. This is unlike Bayer CFA which transmit a single component of RGB colors for each color filter element. Thus, the light intensity transmitted through the new CFA over underlying pixel group is significantly increased, thereby improving light sensitivity of image sensor.
A special demosaicing method is provided for transforming the input pixel values to the corresponding output RGB color values.
[0008] By using the same method as described above, this example can be further generalized to all the other known CFA patterns and may be used in further research studies.
Technical Problem
[0009] Virtually all exisiting digital cameras can only capture one of three primary colors in each photosite on digital image sensor, and so they discard roughly two-thirds of the incoming light. Clearly, light sensitivity of the camera can be improved by reducing the portion of unused light.
[0010] The demosaicing process reconstructs a full-color image from incomplete color samples output from an image sensor overlaid with a CFA by estimating the missing color values for each pixel in a raw capture (see, for example, Guenter US20040001641A1). This is required due to the fact that most raw captures record a single color value for each image pixel. However, demosaicing does not specifically
address fundamental undersampling in typical camera designs.
[0011] Thus, there’s a need for CFA that aids the demosaicing process to achieve minimum false color and zipper effect artifacts with low computational complexity by virtue of storing plural color component values for each pixel in raw image.
Solution to Problem
[0012] An object of this invention is to capture color information at a lower expense of intensity of incoming light than conventional CFAs. Method presented herein allows for determination of spectral transmittance of each color filter element in the CFA pattern corresponding to maximum intensity of light transmitted through the CFA. The supplemental incoming light enhances light sensitivity.
[0013] It is a further object of the invention to present a method of composing a CFA pattern, which superposes different color information so that the following demosaicing processing exhibits superior contrast and color accuracy in the final full-color image. This is achieved by enabling all or fewer color sensor pixels to capture plural different colors simultaneously, thus effectively reducing total number of missing pixel color component values.
[0014] The centerpiece of this invention is that the increase in incident light intensity as well as the following demosaicing operation requires no additional compromise on image quality, and that the manufacturing can be carried out at no extra cost and complexity.
Description of Embodiments
[0015] In general, a sensor pixel is representable by the matrix equation
, wherein matrices
denote intensity of different color components ck of the color model being used in the incident light, color filter transmittance corresponding to each color component ck, the pixel intensity value, respectively; and
is the quantization function for transforming the input original image data to quantized image data. Since each photosite produce an electrical charge directly proportional to the light intensity it receives with no wavelength specificity, arc independent variables.
[0016] Inductively, a α x b minimum repeat unit under assumption that incident light intensity C1 x3 on each color filter element in the corresponding CFA pattern is uniform may be expressed by the matrix equation
(1)
such that Fi,j ∈ {FNx1 } and Kaxb[i,j] = Pi,j such that Pi,j ∈ {P1 x 1 } . and wherein matrices Haxb and Kaxb denote spectral transmittance of color filters in the corresponding CFA pattern and grayscale intensity values of the pixel group underlying HaXb, respectively.
Note that Jaxa is also the variable matrix representing color information about the pixel group.
C1 xN can be determined by solving Equation (1) using Cramer’s rule (see Azamat, Akhtyamov, et al.) under condition that total number of unique Fi,j ∈ Haxb (representing independent linear equations) are equal to or greater than N (total number of unknown variables )·
A full-color image can be obtained by applying any known demosaicing method to matrix Paxb, which is defined by the conditional statement
Optimal HaXb corresponding to a desired spectral responses can be formulated through computational methods by substituting the spectral responses into C1 xN , and then solving Equation (1) for HaXb to maximize ∑i,j Pi,j.
[0017] For simplicity and clarity, consider an example in which a 2 x 2 minimum repeat unit comprising a RGB color filter array (CFA) is given by
(2)
, wherein
[0018] In (Bayer US3971065A), the Bayer CFA pattern is configured as
, or equivalently s1, 2 = VR,S1,1 = S2,2 = VG,S2,1 = VB.
Therefore, amount of light transmitted through the Bayer filter on underlying pixel group is
[0019] Consider next the alternative example of Bayer pattern H2X2 wherein optical filters have ratios of spectral transmittance as follows:
Let as shown in FIG. 2, wherein R,G,B are desired spectral responses of Bayer
filter as shown in FIG. 1, wherein
Since the maximum value of color filter transmittance is equal to or lesser than 1, the actual spectral transmission characteristic of F1,1 , defined as , is obtained by applying min-max normalization method to perform [0, 1] norma
lization on , as shown in FIG. 2. Therefore, F1,2 can be calculated by using the following formula:
The operations discussed above may be performed iteratively for each Fi,j ∈ H2X2 to obtain
(3)
In this case, the light intensity transmitted through the CFA pattern on underlying pixel group is
which is equivalent to about 148% as much light sensitivity as available via the Bayer filter.
[0020] Let be designated solution to the raw Bayer-type image pattern
corresponding to H2X2 as described in equation (3), which is obtained by solving equation (2).
Since
, one may apply to raw image pattern the transformation
to make the RGB raw data in C1 x3 compatible with any known color demosaicing algorithm used to demosaic the Bayer image data (as discussed in Gunturk, Bahadir K., et al.).
Note further that the final RGB color values in C1 x3 are based on weighted average of RGB pixel values of all pixels of the minimum repeat unit, and thus color accuracy and color consistency in final full-color image is better than when measured by Bayer image sensor, wherein each color component value is measured at a single camera-sensor pixel location, and then interpolated for neighboring pixels.
Patent Citations
Bayer, Bryce E. "Color imaging array." U.S. Patent No. 3,971,065. 20 Jul. 1976.
Guenter, Brian Kevin. "Demosaicing graphical content." U.S. Patent No. 7,346,225. 18 Mar. 2008.
Non-Patent Citations
Azamat, Akhtyamov, et al. "Cramer’s rule for nonsingular mx n matrices." Teaching of Mathematics 20.1
(2017): 13-19.
Gunturk, Bahadir K., et al. "Demosaicking: color filter array interpolation." IEEE Signal processing magazine 22.1 (2005): 44-54.
Claims
1. A method of determining the optimal spectral transmission characteristic of color filter elements in a color filter array (CFA) overlaying photodetectors of an image sensor array for capturing raw image data.
An image sensor array characterized by the above comprises a plurality of minimum repeat units, each minimum repeat unit comprising pixel(s) having spectral response corresponding to plurality of component colors.
2. An adaptation of CFA image demosaicing method, for transforming the received raw image sensor data to a mosaiced image, separated into color planes so that it can be further demosaiced by known techniques.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| IN202211041012 | 2022-07-18 | ||
| IN202211041012 | 2022-07-18 |
Publications (3)
| Publication Number | Publication Date |
|---|---|
| WO2022185345A2 true WO2022185345A2 (en) | 2022-09-09 |
| WO2022185345A3 WO2022185345A3 (en) | 2022-11-10 |
| WO2022185345A4 WO2022185345A4 (en) | 2022-12-01 |
Family
ID=83155629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IN2022/050667 WO2022185345A2 (en) | 2022-07-18 | 2022-07-25 | Optimal color filter array and a demosaicing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2022185345A2 (en) |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9071721B1 (en) * | 2012-12-21 | 2015-06-30 | Google Inc. | Camera architecture having a repositionable color filter array |
-
2022
- 2022-07-25 WO PCT/IN2022/050667 patent/WO2022185345A2/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022185345A3 (en) | 2022-11-10 |
| WO2022185345A4 (en) | 2022-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10136107B2 (en) | Imaging systems with visible light sensitive pixels and infrared light sensitive pixels | |
| CN108650497B (en) | Imaging system with transparent filter pixels | |
| US8723995B2 (en) | Extended dynamic range in color imagers | |
| US20070159542A1 (en) | Color filter array with neutral elements and color image formation | |
| US11258992B2 (en) | Image processing method and filter array including wideband filter elements and narrowband filter elements | |
| US8294797B2 (en) | Apparatus and method of generating a high dynamic range image | |
| WO2021212763A1 (en) | High-dynamic-range image processing system and method, electronic device and readable storage medium | |
| US8248496B2 (en) | Image processing apparatus, image processing method, and image sensor | |
| US7072509B2 (en) | Electronic image color plane reconstruction | |
| EP2039149A2 (en) | Solid-state image sensor | |
| US20140078346A1 (en) | Imaging device and image generation method | |
| KR20070011429A (en) | The reproduction of alternative forms of light from an object using a digital imaging system | |
| CN209345244U (en) | Imaging system and camera model | |
| US20140307135A1 (en) | Color imaging element | |
| US8929682B2 (en) | Calibrating image sensors | |
| US8976275B2 (en) | Color imaging element | |
| CA2443658C (en) | Method and apparatus for sensing and interpolating color image data | |
| US8692910B2 (en) | Image processing device, image signal correction method, correction matrix calculation method, and imaging device | |
| WO2007082289A2 (en) | Color filter array with neutral elements and color image formation | |
| WO2022185345A2 (en) | Optimal color filter array and a demosaicing method thereof | |
| JP5036524B2 (en) | Image processing apparatus, image processing method, program, and imaging apparatus | |
| JP2006211631A (en) | Solid-state imaging device and imaging apparatus | |
| CN102300099A (en) | Image processing apparatus, image processing method, and image processing program | |
| US20130038772A1 (en) | Image processing apparatus and image processing method | |
| US20230254595A1 (en) | Optimal color filter array and a demosaicing method thereof |