WO2022094970A1 - Electrical device, method of generating image data, and non-transitory computer readable medium - Google Patents

Electrical device, method of generating image data, and non-transitory computer readable medium Download PDF

Info

Publication number
WO2022094970A1
WO2022094970A1 PCT/CN2020/127263 CN2020127263W WO2022094970A1 WO 2022094970 A1 WO2022094970 A1 WO 2022094970A1 CN 2020127263 W CN2020127263 W CN 2020127263W WO 2022094970 A1 WO2022094970 A1 WO 2022094970A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
green
binning
binning image
element block
Prior art date
Application number
PCT/CN2020/127263
Other languages
French (fr)
Inventor
Hajime Numata
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp., Ltd. filed Critical Guangdong Oppo Mobile Telecommunications Corp., Ltd.
Priority to CN202080105817.9A priority Critical patent/CN116324866A/en
Priority to PCT/CN2020/127263 priority patent/WO2022094970A1/en
Publication of WO2022094970A1 publication Critical patent/WO2022094970A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H04N25/46—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H04N23/843—Demosaicing, e.g. interpolating colour pixel values

Definitions

  • the present disclosure relates to a method of generating an image data, an electrical device implementing such method and a non-transitory computer readable medium including program instructions stored thereon for performing such method.
  • Electrical devices such as smartphones and tablet terminals are widely used in our daily life.
  • many of the electrical devices are equipped with a camera assembly to capture an image.
  • Some of the electrical devices are portable and are thus easy to carry. Therefore, a user of the electrical device can easily take a picture of an object by using the camera assembly of the electrical device anytime, anywhere.
  • One of the widely known formats is a Bayer format which includes a sparse image data.
  • FIG. 9 is a diagram showing an example of a prior art for performing Bayer Binning of four pixels.
  • FIG. 10 is a diagram showing an example in which Bayer Binning of four pixels is executed to generate binning image data in the prior art shown in FIG. 9.
  • the sensor analog binning of the Pixel Array is performed with same on all pixel in the sensor (the pixel position) due to the resource restriction.
  • the noise quality using 4 pixels binning is better (FIG. 10) .
  • the G (green) information is not obtained on the R (Red) and B (Blue) pixel position.
  • FIG. 11 is a diagram showing another example of the prior art of performing binning of two diagonally arranged pixels.
  • FIG. 12 is a diagram showing an example in which binning of two diagonally arranged pixels is executed to generate binning data in the prior art shown in FIG. 11.
  • the analog binning is performed only 2 pixels in 4 pixels binning block to avoid color mixture at the R-G block of pixel position and the B-G block of pixel position.
  • these G pixels improve resolution on G channel. Therefore, the resolution of the CFA binning is better than the binned Bayer CFA.
  • the noise quality of the binning 2 pixels is worse than the noise quality of the binning 4 pixels due to amount of the accumulated electron.
  • FIG. 13 is a diagram showing an example of a conventional technique for converting GYCG image data acquired by using a GYCG sensor into RGB image data.
  • the such sensor obtains the YGC image.
  • the exact intensity of each color depends on spectrum of light and object reflection.
  • the cyan intensity is not equal to the result of adding the green intensity and the blue intensity. Therefore, it is not able to clearly separate a complementary color intensity into 2 primary color component.
  • the present invention proposes to perform 4 pixels binning in the new color filter array to reduce the effect of color mixing and improve the noise quality.
  • the present disclosure aims to solve at least one of the technical problems mentioned above. Accordingly, the present disclosure needs to provide a method of generating a target image data and an electrical device implementing such method.
  • an electrical device may include:
  • a camera assembly including an image sensor, configured to capture an image of an object and to generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements; and
  • the main processor acquires binning image data of the element blocks, the binning image data being generated by the camera assembly by combining charges of the four physical pixel elements of the element blocks,
  • the main processor calculates estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block,
  • the main processor calculates estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block, and
  • the main processor calculates estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
  • the main processor acquires the binning image data of the green element block, the estimated blue binning image data, and the estimated red binning image data, as image data conforming to the Bayer format.
  • the green element blocks, the blue element blocks, and the red element blocks have rectangular shape
  • the four green physical pixel elements are arranged corresponding to four corners, respectively,
  • the two blue physical pixel elements are located on a first diagonal line
  • the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively
  • the two red physical pixel elements are located on a first diagonal line
  • the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively.
  • the main processor calculates the estimated blue binning image data, by subtracting the estimated green binning image data from the binning image data of the blue element block, and
  • the main processor calculates the estimated red binning image data, by subtracting the estimated green binning image data from the binning image data of the red element block.
  • the main processor calculates the estimated green binning image data corresponding to the two green physical pixel elements, based on average value calculated from the binning image data of the plurality of green element blocks.
  • the main processor calculates the estimated green binning image data for calculating the estimated blue binning image data, based on the average value of the binning image data of the two or four green element blocks adjacent to the blue element block, and
  • the main processor calculates the estimated green binning image data for calculating the estimated red binning image data, based on the average value of the binning image data of the two or four green element blocks adjacent to the red element block.
  • the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the blue element block in the first direction and the second direction orthogonal to the first direction,
  • the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the blue element block in the first direction
  • the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the blue element block in the second direction
  • the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the first direction,
  • the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the second direction,
  • the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value
  • the main processor sets half of the total average value as the estimated green binning image data
  • the main processor sets half of the first average value as the estimated green binning image data
  • the main processor sets half of the second average value as the estimated green binning image data.
  • the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the red element block in the first direction and the second direction orthogonal to the first direction,
  • the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the red element block in the first direction
  • the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the red element block in the second direction
  • the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the first direction
  • the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the second direction,
  • the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value
  • the main processor sets half of the total average value as the estimated green binning image data
  • the main processor sets half of the first average value as the estimated green binning image data
  • the main processor sets half of the second average value as the estimated green binning image data.
  • a method of generating an image data includes:
  • the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
  • a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following:
  • the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
  • FIG. 1 illustrates a plan view of a first side of an electrical device according to an embodiment of the present disclosure
  • FIG. 2 illustrates a plan view of a second side of the electrical device according to the embodiment of the present disclosure
  • FIG. 3 illustrates a block diagram of the electrical device according to the embodiment of the present disclosure
  • FIG. 4 is a diagram showing an example of the configuration of an image sensor of a camera assembly that performs Bayer Binning of four pixels according to the embodiment of the present disclosure
  • FIG. 5 is a diagram showing an example in which Bayer Binning is executed for four pixels to generate binning image data according to the embodiment of the present disclosure
  • FIG. 6 is a diagram showing a specific example of the flow of according to the embodiment of the present disclosure for generating the estimated blue binning image data shown in FIG. 5;
  • FIG. 7 is a diagram showing a specific example of the flow of the present disclosure for generating estimated red binning image data
  • FIG. 8 is a diagram for explaining the effect of according to the embodiment of the present disclosure.
  • FIG. 9 is a diagram showing an example of a prior art for performing Bayer Binning of four pixels.
  • FIG. 10 is a diagram showing an example in which Bayer Binning of four pixels is executed to generate binning image data in the prior art shown in FIG. 9;
  • FIG. 11 is a diagram showing another example of the prior art of performing binning of two diagonally arranged pixels
  • FIG. 12 is a diagram showing an example in which binning of two diagonally arranged pixels is executed to generate binning data in the prior art shown in FIG. 11;
  • FIG. 13 is a diagram showing an example of a conventional technique for converting GYCG image data acquired by using a GYCG sensor into RGB image data.
  • FIG. 1 illustrates a plan view of a first side of an electrical device 10 according to an embodiment of the present disclosure
  • FIG. 2 illustrates a plan view of a second side of the electrical device 10 according to the embodiment of the present disclosure.
  • the first side may be referred to as a back side of the electrical device 10 whereas the second side may be referred to as a front side of the electrical device 10.
  • the electrical device 10 may include a display 20 and a camera assembly 30.
  • the camera assembly 30 includes a first main camera 32, a second main camera 34 and a sub camera 36.
  • the first main camera 32 and the second main camera 34 can capture an image in a first side of the electrical device 10 and the sub camera 36 can capture an image in the second side of the electrical device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called out-cameras whereas the sub camera 36 is a so-called in-camera.
  • the electrical device 10 can be a mobile phone, a tablet computer, a personal digital assistant, and so on.
  • the electrical device 10 may have less than three cameras or more than three cameras.
  • the electrical device 10 may have two, four, five, and so on, cameras.
  • FIG. 3 illustrates a block diagram of the electrical device 10 according to the present embodiment.
  • the electrical device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46 and a communication circuit 48.
  • the display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power supply circuit 46 and the communication circuit 48 are connected each other via a bus 50.
  • the main processor 40 executes one or more programs stored in the memory 44.
  • the main processor 40 implements various applications and data processing of the electrical device 10 by executing the programs.
  • the main processor 40 may be one or more computer processors.
  • the main processor 40 is not limited to one CPU core, but it may have a plurality of CPU cores.
  • the main processor 40 may be a main CPU of the electrical device 10, an image processing unit (IPU) or a DSP provided with the camera assembly 30.
  • the image signal processor 42 controls the camera assembly 30 and processes various kinds of image data captured by the camera assembly 30 to generate a target image data.
  • the image signal processor 42 can execute a de-mosaic process, a noise reduction process, an auto exposure process, an auto focus process, an auto white balance process, a high dynamic range process and so on, to the image data captured by the camera assembly 30.
  • the main processor 40 and the image signal processor 42 collaborate with each other to generate a target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture the image of the object by the camera assembly 30 and execute various kinds of image processes to the captured image data.
  • the memory 44 stores a program to be executed by the main processor 40 and various kinds of data. For example, data of the captured image are stored in the memory 44.
  • the memory 44 may include a high-speed RAM memory, and/or a non-volatile memory such as a flash memory and a magnetic disk memory. That is, the memory 44 may include a non-transitory computer readable medium, in which the program is stored.
  • the power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.
  • BMU battery management unit
  • the communication circuit 48 is configured to receive and transmit data to communicate with base stations of the telecommunication network system, the Internet or other devices via wireless communication.
  • the wireless communication may adopt any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication) , CDMA (Code Division Multiple Access) , LTE (Long Term Evolution) , LTE-Advanced, 5th generation (5G) .
  • the communication circuit 48 may include an antenna and a RF (radio frequency) circuit.
  • FIG. 4 is a diagram showing an example of the configuration of an image sensor of a camera assembly that performs Bayer Binning of four pixels according to the embodiment of the present disclosure.
  • the camera assembly 30 includes an image sensor that captures an image of an object and generates color image data.
  • the image sensor has the green element blocks GK, the blue element blocks BK, and the red element blocks RK arranged in an array of Bayer format at each pixel position in order to generate color image data.
  • the green element blocks GK, the blue element blocks BK, and the red element blocks RK contain four physical pixel elements, respectively.
  • the green element blocks GK, the blue element blocks BK, and the red element blocks RK have a rectangular shape. That is, the pixel array of the present embodiment employs 2X2 binning technology.
  • the green element block GK includes four green physical pixel elements G.
  • the four green physical pixel elements G are arranged corresponding to the four corners, respectively.
  • the signal value (the binning image data) 4GBIN of the green is generated by combing four electric charges in the 4 green physical pixel elements G.
  • the blue element block BK includes two blue physical pixel elements B and two green physical pixel elements G.
  • the two blue physical pixel elements B are located on the first diagonal line
  • the two green physical pixel elements G are located on the second diagonal line, corresponding to each of the four corners, respectively.
  • the signal value (the binning image data) 4CBIN (BG) of the cyan generated by combing four electric charges in the two blue physical pixel elements B and the two green physical pixel elements G.
  • the red element block RK includes two red physical pixel elements R and two green physical pixel elements G.
  • the two red physical pixel elements R are located on the first diagonal line
  • the two green physical pixel elements G are located on the second diagonal line, corresponding to each of the four corners, respectively.
  • the signal value (the binning image data) 4YBIN (RG) of the yellow generated by combing four electric charges in the two red physical pixel elements R and the two green physical pixel elements G.
  • FIG. 5 is a diagram showing an example in which Bayer Binning is executed for four pixels to generate binning image data according to the embodiment of the present disclosure.
  • FIG. 5 shows an example of calculating the estimated blue binning image data EB, a case of calculating the estimated red binning image data ER is also shown in the same manner.
  • the image generation process is executed by, for example, the main processor 40 in order to generate the image data.
  • program instructions of the image generation process are stored in the non-transitory computer readable medium of the memory 44.
  • the main processor 40 implements the image generation process.
  • the main processor 40 acquires the binning image data 4GBIN, 4CBIN, and 4YBIN of the element blocks GK, BK, and RK.
  • the binning image data 4GBIN, 4CBIN, and 4YBIN are generated by the camera assembly 30 by combining the charges of the four physical pixel elements of the element blocks GK, BK, and RK.
  • the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G based on the binning image data 4GBIN of the green element block GK.
  • the main processor 40 calculates the estimated blue binning image data EB, by subtracting the estimated green binning image data G 2pix from the binning image data 4CBIN (BG) of the blue element block BK.
  • the main processor 40 calculates the estimated red binning image data ER, by subtracting the estimated green binning image data G 2pix from the binning image data 4YBIN (RG) of the red element block RK.
  • FIG. 6 is a diagram showing a specific example of the flow of according to the embodiment of the present disclosure for generating the estimated blue binning image data shown in FIG. 5.
  • the main processor 40 calculates the total average value G ave4 of the binning image data Ga, Gb, Gc, and Gd (4GBIN) of the four green element blocks GK adjacent to the blue element block BK in the first direction and the second direction orthogonal to the first direction.
  • G ave4 (G a +G b +G c +G d ) /4
  • the main processor 40 calculates the first average value G aveV of the binning image data G a and G c of the two green element blocks GK adjacent to the blue element block BK in the first direction.
  • the main processor 40 calculates the second average value G aveH of the binning image data G b and G d of the two green element blocks GK adjacent to the blue element block BK in the second direction.
  • the main processor 40 calculates the first absolute value G difV of the difference between the binning image data G a and G c of the two green element blocks GK adjacent to the blue element block BK in the first direction.
  • the main processor 40 calculates the second absolute value G difH of the difference between the binning image data G b and G d of the two green element blocks GK adjacent to the blue element block BK in the second direction.
  • the main processor 40 calculates the reference value G dir based on the ratio of the first absolute value G difV and the second absolute value G difH .
  • the main processor 40 sets the total average value G ave4 as an interpolate green value G 4pix .
  • the main processor 40 sets the first average value G difV as an interpolate green value G 4pix .
  • the main processor 40 sets the second average value G difH as an interpolate green value G 4pix .
  • the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G, based on the average value calculated from the binning image data of the plurality of green element blocks GK.
  • the main processor 40 calculates the estimated green binning image data G 2pix for calculating the estimated blue binning image data EB, based on the average value G 4pix of the binning image data of the two or four green element blocks GK adjacent to the blue element block BK.
  • FIG. 7 is a diagram showing a specific example of the flow of the present disclosure for generating estimated red binning image data.
  • the flow for generating the estimated red binning image data shown in FIG. 7 is the same as the flow shown in FIG. 6 described above.
  • the main processor 40 calculates the total average value G ave4 of the binning image data Ga, Gb, Gc, and Gd (4GBIN) of the four green element blocks GK adjacent to the red element block RK in the first direction and the second direction orthogonal to the first direction.
  • G ave4 (G a +G b +G c +G d ) /4
  • the main processor 40 calculates the first average value G aveV of the binning image data G a and G c of the two green element blocks GK adjacent to the red element block RK in the first direction.
  • the main processor 40 calculates the second average value G aveH of the binning image data G b and G d of the two green element blocks GK adjacent to the red element block RK in the second direction.
  • the main processor 40 calculates the first absolute value G difV of the difference between the binning image data G a and G c of the two green element blocks GK adjacent to the red element block RK in the first direction.
  • the main processor 40 calculates the second absolute value G difH of the difference between the binning image data G b and G d of the two green element blocks GK adjacent to the red element block RK in the second direction.
  • the main processor 40 calculates the reference value G dir based on the ratio of the first absolute value G difV and the second absolute value G difH .
  • the main processor 40 sets the total average value G ave4 as an interpolate green value G 4pix .
  • the main processor 40 sets the first average value G difV as an interpolate green value G 4pix .
  • the main processor 40 sets the second average value G difH as an interpolate green value G 4ix .
  • step S12 of FIG. 7 the main processor 40 uses half of the interpolate green value G 4pix as the estimated green binning image data G2pix. That is, 2 pixels binning G (G 2p i x ) is estimated from 4 pixels binning G (G 4p i x ) .
  • the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G, based on the average value calculated from the binning image data of the plurality of green element blocks GK.
  • the main processor 40 calculates the estimated green binning image data G 2pix for calculating the estimated red binning image data ER, based on the average value G 4pix of the binning image data of the two or four green element blocks GK adjacent to the red element block RK.
  • the main processor 40 calculates the estimated red binning image data ER, by subtracting the estimated green binning image data G 2pix adjacent to the red element block RK from the binning image data 4YBIN (RG) of the red element block RK.
  • the main processor 40 of the electrical device 10 acquires the binning image data 4GBIN of the green element block GK, the estimated blue binning image data EB, and the estimated red binning image data ER, as image data conforming to the Bayer format.
  • the main processor 40 outputs the acquired image data conforming to the Bayer format to, for example, the image signal processor 42.
  • the image signal processor 42 executes a predetermined image processing on the input image data.
  • FIG. 8 is a diagram for explaining the effect of according to the embodiment of the present disclosure.
  • a camera assembly 30 including an image sensor, configured to capture an image of an object and to generate color image data
  • the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data
  • the green element block, the blue element block, and the red element block each including four physical pixel elements
  • the green element block comprises four green physical pixel elements
  • the blue element block comprises two blue physical pixel elements and two green physical pixel elements
  • the red element block comprises two red physical pixel elements and two green physical pixel elements
  • a main processor 40 that performs image process.
  • the main processor 40 acquires binning image data of the element blocks, the binning image data being generated by the camera assembly 30 by combining charges of the four physical pixel elements of the element blocks, the main processor 40 calculates estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block, the main processor 40 calculates estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block, and the main processor 40 calculates estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
  • the present invention proposes to perform 4 pixels binning in the new color filter array to reduce the effect of color mixing and improve the noise quality 2 pixels binning.
  • first and second are used herein for purposes of description and are not intended to indicate or imply relative importance or significance or to imply the number of indicated technical features.
  • the feature defined with “first” and “second” may comprise one or more of this feature.
  • a plurality of means two or more than two, unless specified otherwise.
  • the terms “mounted” , “connected” , “coupled” and the like are used broadly, and may be, for example, fixed connections, detachable connections, or integral connections; may also be mechanical or electrical connections; may also be direct connections or indirect connections via intervening structures; may also be inner communications of two elements, which can be understood by those skilled in the art according to specific situations.
  • a structure in which a first feature is "on" or “below” a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are contacted via an additional feature formed therebetween.
  • a first feature "on” , “above” or “on top of” a second feature may include an embodiment in which the first feature is right or obliquely “on” , “above” or “on top of” the second feature, or just means that the first feature is at a height higher than that of the second feature; while a first feature “below” , “under” or “on bottom of” a second feature may include an embodiment in which the first feature is right or obliquely “below” , "under” or “on bottom of” the second feature, or just means that the first feature is at a height lower than that of the second feature.
  • Any process or method described in a flow chart or described herein in other ways may be understood to include one or more modules, segments or portions of codes of executable instructions for achieving specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, in which it should be understood by those skilled in the art that functions may be implemented in a sequence other than the sequences shown or discussed, including in a substantially identical sequence or in an opposite sequence.
  • the logic and/or step described in other manners herein or shown in the flow chart, for example, a particular sequence table of executable instructions for realizing the logical function may be specifically achieved in any computer readable medium to be used by the instruction execution system, device or equipment (such as the system based on computers, the system comprising processors or other systems capable of obtaining the instruction from the instruction execution system, device and equipment and executing the instruction) , or to be used in combination with the instruction execution system, device and equipment.
  • the computer readable medium may be any device adaptive for including, storing, communicating, propagating or transferring programs to be used by or in combination with the instruction execution system, device or equipment.
  • the computer readable medium comprise but are not limited to: an electronic connection (an electronic device) with one or more wires, a portable computer enclosure (a magnetic device) , a random access memory (RAM) , a read only memory (ROM) , an erasable programmable read-only memory (EPROM or a flash memory) , an optical fiber device and a portable compact disk read-only memory (CDROM) .
  • the computer readable medium may even be a paper or other appropriate medium capable of printing programs thereon, this is because, for example, the paper or other appropriate medium may be optically scanned and then edited, decrypted or processed with other appropriate methods when necessary to obtain the programs in an electric manner, and then the programs may be stored in the computer memories.
  • each part of the present disclosure may be realized by the hardware, software, firmware or their combination.
  • a plurality of steps or methods may be realized by the software or firmware stored in the memory and executed by the appropriate instruction execution system.
  • the steps or methods may be realized by one or a combination of the following techniques known in the art: a discrete logic circuit having a logic gate circuit for realizing a logic function of a data signal, an application-specific integrated circuit having an appropriate combination logic gate circuit, a programmable gate array (PGA) , a field programmable gate array (FPGA) , etc.
  • each function cell of the embodiments of the present disclosure may be integrated in a processing module, or these cells may be separate physical existence, or two or more cells are integrated in a processing module.
  • the integrated module may be realized in a form of hardware or in a form of software function modules. When the integrated module is realized in a form of software function module and is sold or used as a standalone product, the integrated module may be stored in a computer readable storage medium.
  • the storage medium mentioned above may be read-only memories, magnetic disks, CD, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

An electrical device includes a camera assembly, including an image sensor, configured to capture an image of an object and to generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements; and a main processor that performs image process.

Description

ELECTRICAL DEVICE, METHOD OF GENERATING IMAGE DATA, AND NON-TRANSITORY COMPUTER READABLE MEDIUM TECHNICAL FIELD
The present disclosure relates to a method of generating an image data, an electrical device implementing such method and a non-transitory computer readable medium including program instructions stored thereon for performing such method.
BACKGROUND
Electrical devices such as smartphones and tablet terminals are widely used in our daily life. Nowadays, many of the electrical devices are equipped with a camera assembly to capture an image. Some of the electrical devices are portable and are thus easy to carry. Therefore, a user of the electrical device can easily take a picture of an object by using the camera assembly of the electrical device anytime, anywhere.
There are many formats to capture the image of the object and generate the target image data thereof. One of the widely known formats is a Bayer format which includes a sparse image data.
Here, FIG. 9 is a diagram showing an example of a prior art for performing Bayer Binning of four pixels. FIG. 10 is a diagram showing an example in which Bayer Binning of four pixels is executed to generate binning image data in the prior art shown in FIG. 9.
For example, as shown in FIG. 9, the sensor analog binning of the Pixel Array is performed with same on all pixel in the sensor (the pixel position) due to the resource restriction. In the Pixel Array, the noise quality using 4 pixels binning is better (FIG. 10) . However, the G (green) information is not obtained on the R (Red) and B (Blue) pixel position.
On the other hand, FIG. 11 is a diagram showing another example of the prior art of performing binning of two diagonally arranged pixels. FIG. 12 is a diagram showing an example in which binning of two diagonally arranged pixels is executed to generate binning data in the prior art shown in FIG. 11.
As shown in FIG. 11, in the color filter sensor having the CFA (Color Filter Array) that G (green) is sprinkled at the R-G block and the B-G block, the analog binning is performed only 2 pixels in 4 pixels binning block to avoid color mixture at the R-G block of pixel position and the B-G block of pixel position. In the CFA binning, these G pixels improve resolution on G channel. Therefore, the resolution of the CFA binning is better than the binned Bayer CFA. However, in the CFA binning, as shown FIG. 12, the noise quality of the binning 2 pixels is worse than the noise quality of the binning 4 pixels due to amount of the accumulated electron.
Next, FIG. 13 is a diagram showing an example of a conventional technique for converting GYCG image data acquired by using a GYCG sensor into RGB image data.
As shown in FIG. 13, there is a sensor with different color array like GYCG (Y=Yellow, G=Green, C=Cyan) . The such sensor obtains the YGC image. The standard image format requires the RGB (R=Red, G=Green, B=Blue) image, so it is needed to convert the sensor color image with Color Correction Matrix, as follow.
Figure PCTCN2020127263-appb-000001
However, the exact intensity of each color depends on spectrum of light and object reflection. For example, the cyan intensity is not equal to the result of adding the green intensity and the blue intensity. Therefore, it is not able to clearly separate a complementary color intensity into 2 primary color component.
Therefore, the present invention proposes to perform 4 pixels binning in the new color filter  array to reduce the effect of color mixing and improve the noise quality.
SUMMARY
The present disclosure aims to solve at least one of the technical problems mentioned above. Accordingly, the present disclosure needs to provide a method of generating a target image data and an electrical device implementing such method.
In accordance with the present disclosure, an electrical device may include:
a camera assembly, including an image sensor, configured to capture an image of an object and to generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements; and
a main processor that performs image process,
wherein the main processor acquires binning image data of the element blocks, the binning image data being generated by the camera assembly by combining charges of the four physical pixel elements of the element blocks,
the main processor calculates estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block,
the main processor calculates estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block, and
the main processor calculates estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
In some embodiments, wherein the main processor acquires the binning image data of the green element block, the estimated blue binning image data, and the estimated red binning image data, as image data conforming to the Bayer format.
In some embodiments, wherein
the green element blocks, the blue element blocks, and the red element blocks have rectangular shape,
in the green element block, the four green physical pixel elements are arranged corresponding to four corners, respectively,
in the blue element block, the two blue physical pixel elements are located on a first diagonal line, and the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively, and
in the red element block, the two red physical pixel elements are located on a first diagonal line, and the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively.
In some embodiments, wherein
the main processor calculates the estimated blue binning image data, by subtracting the estimated green binning image data from the binning image data of the blue element block, and
the main processor calculates the estimated red binning image data, by subtracting the estimated green binning image data from the binning image data of the red element block.
In some embodiments, wherein
the main processor calculates the estimated green binning image data corresponding to the two green physical pixel elements, based on average value calculated from the binning image data of the plurality of green element blocks.
In some embodiments, wherein
the main processor calculates the estimated green binning image data for calculating the estimated blue binning image data, based on the average value of the binning image data of the  two or four green element blocks adjacent to the blue element block, and
the main processor calculates the estimated green binning image data for calculating the estimated red binning image data, based on the average value of the binning image data of the two or four green element blocks adjacent to the red element block.
In some embodiments, wherein
the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the blue element block in the first direction and the second direction orthogonal to the first direction,
the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the blue element block in the first direction,
the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the blue element block in the second direction,
the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the first direction,
the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the second direction,
the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value, and
wherein,
when the reference value is less than a preset threshold value, the main processor sets half of the total average value as the estimated green binning image data,
when the reference value is equal to or higher than the threshold value and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green binning image data, and
when the reference value is equal to or greater than the threshold value and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green binning image data.
In some embodiments, wherein
the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the red element block in the first direction and the second direction orthogonal to the first direction,
the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the red element block in the first direction,
the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the red element block in the second direction,
the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the first direction,
the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the second direction,
the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value, and
wherein,
when the reference value is less than a preset threshold value, the main processor sets half of the total average value as the estimated green binning image data,
when the reference value is equal to or higher than the threshold value and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green binning image data, and
when the reference value is equal to or greater than the threshold value and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green binning image data.
In accordance with the present disclosure, a method of generating an image data, includes:
acquiring binning image data of element blocks, the binning image data being generated by combining charges of the four physical pixel elements of the element blocks, wherein the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
calculating estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block;
calculating estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block; and
calculating estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
In accordance with the present disclosure, a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following:
acquiring binning image data of element blocks, the binning image data being generated by combining charges of the four physical pixel elements of the element blocks, wherein the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
calculating estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block;
calculating estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block; and
calculating estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
BRIEF DESCRIPTION OF THE DRAWINGS
These and/or other aspects and advantages of embodiments of the present disclosure will become apparent and more readily appreciated from the following descriptions made with reference to the drawings, in which:
FIG. 1 illustrates a plan view of a first side of an electrical device according to an embodiment of the present disclosure;
FIG. 2 illustrates a plan view of a second side of the electrical device according to the embodiment of the present disclosure;
FIG. 3 illustrates a block diagram of the electrical device according to the embodiment of the present disclosure;
FIG. 4 is a diagram showing an example of the configuration of an image sensor of a camera assembly that performs Bayer Binning of four pixels according to the embodiment of the present disclosure;
FIG. 5 is a diagram showing an example in which Bayer Binning is executed for four pixels  to generate binning image data according to the embodiment of the present disclosure;
FIG. 6 is a diagram showing a specific example of the flow of according to the embodiment of the present disclosure for generating the estimated blue binning image data shown in FIG. 5;
FIG. 7 is a diagram showing a specific example of the flow of the present disclosure for generating estimated red binning image data;
FIG. 8 is a diagram for explaining the effect of according to the embodiment of the present disclosure;
FIG. 9 is a diagram showing an example of a prior art for performing Bayer Binning of four pixels;
FIG. 10 is a diagram showing an example in which Bayer Binning of four pixels is executed to generate binning image data in the prior art shown in FIG. 9;
FIG. 11 is a diagram showing another example of the prior art of performing binning of two diagonally arranged pixels;
FIG. 12 is a diagram showing an example in which binning of two diagonally arranged pixels is executed to generate binning data in the prior art shown in FIG. 11; and
FIG. 13 is a diagram showing an example of a conventional technique for converting GYCG image data acquired by using a GYCG sensor into RGB image data.
DETAILED DESCRIPTION
Embodiments of the present disclosure will be described in detail and examples of the embodiments will be illustrated in the accompanying drawings. The same or similar elements and the elements having same or similar functions are denoted by like reference numerals throughout the descriptions. The embodiments described herein with reference to the drawings are explanatory, which aim to illustrate the present disclosure, but shall not be construed to limit the present disclosure.
FIG. 1 illustrates a plan view of a first side of an electrical device 10 according to an embodiment of the present disclosure and FIG. 2 illustrates a plan view of a second side of the electrical device 10 according to the embodiment of the present disclosure. The first side may be referred to as a back side of the electrical device 10 whereas the second side may be referred to as a front side of the electrical device 10.
As shown in FIG. 1 and FIG. 2, the electrical device 10 may include a display 20 and a camera assembly 30. In the present embodiment, the camera assembly 30 includes a first main camera 32, a second main camera 34 and a sub camera 36. The first main camera 32 and the second main camera 34 can capture an image in a first side of the electrical device 10 and the sub camera 36 can capture an image in the second side of the electrical device 10. Therefore, the first main camera 32 and the second main camera 34 are so-called out-cameras whereas the sub camera 36 is a so-called in-camera. As an example, the electrical device 10 can be a mobile phone, a tablet computer, a personal digital assistant, and so on.
Although the electrical device 10 according to the present embodiment has three cameras, the electrical device 10 may have less than three cameras or more than three cameras. For example, the electrical device 10 may have two, four, five, and so on, cameras.
FIG. 3 illustrates a block diagram of the electrical device 10 according to the present embodiment. As shown in FIG. 3, in addition to the display 20 and the camera assembly 30, the electrical device 10 may include a main processor 40, an image signal processor 42, a memory 44, a power supply circuit 46 and a communication circuit 48. The display 20, the camera assembly 30, the main processor 40, the image signal processor 42, the memory 44, the power supply circuit 46 and the communication circuit 48 are connected each other via a bus 50.
The main processor 40 executes one or more programs stored in the memory 44. The main processor 40 implements various applications and data processing of the electrical device 10 by executing the programs. The main processor 40 may be one or more computer processors. The main processor 40 is not limited to one CPU core, but it may have a plurality of CPU cores. The  main processor 40 may be a main CPU of the electrical device 10, an image processing unit (IPU) or a DSP provided with the camera assembly 30.
The image signal processor 42 controls the camera assembly 30 and processes various kinds of image data captured by the camera assembly 30 to generate a target image data. For example, the image signal processor 42 can execute a de-mosaic process, a noise reduction process, an auto exposure process, an auto focus process, an auto white balance process, a high dynamic range process and so on, to the image data captured by the camera assembly 30.
In the present embodiment, the main processor 40 and the image signal processor 42 collaborate with each other to generate a target image data of the object captured by the camera assembly 30. That is, the main processor 40 and the image signal processor 42 are configured to capture the image of the object by the camera assembly 30 and execute various kinds of image processes to the captured image data.
The memory 44 stores a program to be executed by the main processor 40 and various kinds of data. For example, data of the captured image are stored in the memory 44.
The memory 44 may include a high-speed RAM memory, and/or a non-volatile memory such as a flash memory and a magnetic disk memory. That is, the memory 44 may include a non-transitory computer readable medium, in which the program is stored.
The power supply circuit 46 may have a battery such as a lithium-ion rechargeable battery and a battery management unit (BMU) for managing the battery.
The communication circuit 48 is configured to receive and transmit data to communicate with base stations of the telecommunication network system, the Internet or other devices via wireless communication. The wireless communication may adopt any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication) , CDMA (Code Division Multiple Access) , LTE (Long Term Evolution) , LTE-Advanced, 5th generation (5G) . The communication circuit 48 may include an antenna and a RF (radio frequency) circuit.
Here, FIG. 4 is a diagram showing an example of the configuration of an image sensor of a camera assembly that performs Bayer Binning of four pixels according to the embodiment of the present disclosure.
As shown in FIG. 4, the camera assembly 30 includes an image sensor that captures an image of an object and generates color image data.
Then, for example, as shown in FIG. 4, the image sensor has the green element blocks GK, the blue element blocks BK, and the red element blocks RK arranged in an array of Bayer format at each pixel position in order to generate color image data. The green element blocks GK, the blue element blocks BK, and the red element blocks RK contain four physical pixel elements, respectively. The green element blocks GK, the blue element blocks BK, and the red element blocks RK have a rectangular shape. That is, the pixel array of the present embodiment employs 2X2 binning technology.
As shown in FIG. 4, the green element block GK includes four green physical pixel elements G. In the green element block GK, the four green physical pixel elements G are arranged corresponding to the four corners, respectively.
For example, the signal value (the binning image data) 4GBIN of the green is generated by combing four electric charges in the 4 green physical pixel elements G.
In particular, in this embodiment, as shown in FIG. 4, the blue element block BK includes two blue physical pixel elements B and two green physical pixel elements G. In the blue element block BK, the two blue physical pixel elements B are located on the first diagonal line, and the two green physical pixel elements G are located on the second diagonal line, corresponding to each of the four corners, respectively.
For example, the signal value (the binning image data) 4CBIN (BG) of the cyan generated by combing four electric charges in the two blue physical pixel elements B and the two green physical pixel elements G.
In particular, in this embodiment, as shown in FIG. 4, the red element block RK includes two red physical pixel elements R and two green physical pixel elements G. In the red element block RK, the two red physical pixel elements R are located on the first diagonal line, and the two green physical pixel elements G are located on the second diagonal line, corresponding to each of the four corners, respectively.
For example, the signal value (the binning image data) 4YBIN (RG) of the yellow generated by combing four electric charges in the two red physical pixel elements R and the two green physical pixel elements G.
Next, an example of the operation including the image processing in which the electrical device 10 having the above configuration acquires the image data conforming to Bayer format will be described. FIG. 5 is a diagram showing an example in which Bayer Binning is executed for four pixels to generate binning image data according to the embodiment of the present disclosure. Although FIG. 5 shows an example of calculating the estimated blue binning image data EB, a case of calculating the estimated red binning image data ER is also shown in the same manner.
In the present embodiment, the image generation process is executed by, for example, the main processor 40 in order to generate the image data.
In addition, in the present embodiment, program instructions of the image generation process are stored in the non-transitory computer readable medium of the memory 44. When the program instructions are read out from the memory 44 and executed in the main processor 40, the main processor 40 implements the image generation process.
First, the main processor 40 acquires the binning image data 4GBIN, 4CBIN, and 4YBIN of the element blocks GK, BK, and RK. The binning image data 4GBIN, 4CBIN, and 4YBIN are generated by the camera assembly 30 by combining the charges of the four physical pixel elements of the element blocks GK, BK, and RK.
Then, the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G based on the binning image data 4GBIN of the green element block GK.
Next, the main processor 40 calculates the estimated blue binning image data EB, by subtracting the estimated green binning image data G 2pix from the binning image data 4CBIN (BG) of the blue element block BK.
Similarly, the main processor 40 calculates the estimated red binning image data ER, by subtracting the estimated green binning image data G 2pix from the binning image data 4YBIN (RG) of the red element block RK.
Here, a specific example of the flow of the embodiment of the present invention for generating the estimated blue binning image data shown in FIG. 5 will be described. FIG. 6 is a diagram showing a specific example of the flow of according to the embodiment of the present disclosure for generating the estimated blue binning image data shown in FIG. 5.
For example, in step S1 of FIG. 6, as shown in the following formula, the main processor 40 calculates the total average value G ave4 of the binning image data Ga, Gb, Gc, and Gd (4GBIN) of the four green element blocks GK adjacent to the blue element block BK in the first direction and the second direction orthogonal to the first direction.
G ave4= (G a+G b+G c+G d) /4
In addition, as shown in the formula below, the main processor 40 calculates the first average value G aveV of the binning image data G a and G c of the two green element blocks GK adjacent to the blue element block BK in the first direction.
G aveV= (G a+G c) /2
In addition, as shown in the formula below, the main processor 40 calculates the second average value G aveH of the binning image data G b and G d of the two green element blocks GK adjacent to the blue element block BK in the second direction.
G aveH= (G b+G d) /2
In addition, as shown in the formula below, the main processor 40 calculates the first absolute value G difV of the difference between the binning image data G a and G c of the two green element blocks GK adjacent to the blue element block BK in the first direction.
G difV=|G a-G c|
In addition, as shown in the formula below, the main processor 40 calculates the second absolute value G difH of the difference between the binning image data G b and G d of the two green element blocks GK adjacent to the blue element block BK in the second direction.
G difH=|G b-G d|
In addition, as shown in the formula below, the main processor 40 calculates the reference value G dir based on the ratio of the first absolute value G difV and the second absolute value G difH.
G dir=|1-G difV/G difH|
Here, as shown in the following formula, when the reference value G dir is less than a preset threshold value T dir, the main processor 40 sets the total average value G ave4 as an interpolate green value G 4pix.
On the other hand, when the reference value G dir is equal to or higher than the threshold value T dir and the first absolute value G aveV is less than the second absolute value G aveH, the main processor 40 sets the first average value G difV as an interpolate green value G 4pix.
On the other hand, when the reference value G dir is equal to or greater than the threshold value and the first absolute value G aveV is equal to or greater than the second absolute value, the main processor 40 sets the second average value G difH as an interpolate green value G 4pix.
Figure PCTCN2020127263-appb-000002
Next, in step S2 of FIG. 6, as shown in the following formula, the main processor 40 uses half of the interpolate green value G 4pix as the estimated green binning image data G2pix. That is, 2 pixels binning G (G  2p  i  x ) is estimated from 4 pixels binning G (G  4p  i  x ) .
G 2pix=G 4pix/2
In this way, the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G, based on the average value calculated from the binning image data of the plurality of green element blocks GK.
In particular, the main processor 40 calculates the estimated green binning image data G 2pix for calculating the estimated blue binning image data EB, based on the average value G 4pix of the binning image data of the two or four green element blocks GK adjacent to the blue element block BK.
Next, in step S3 of FIG. 6, as shown in the following formula, the main processor 40 calculates the estimated blue binning image data EB, by subtracting the estimated green binning image data G 2pix adjacent to the blue element block BK from the binning image data 4CBIN (BG) of the blue element block BK.
EB=BG-G 2pix
On the other hand, FIG. 7 is a diagram showing a specific example of the flow of the present disclosure for generating estimated red binning image data. The flow for generating the estimated red binning image data shown in FIG. 7 is the same as the flow shown in FIG. 6 described above.
For example, in step S11 of FIG. 7, as shown in the following formula, the main processor 40 calculates the total average value G ave4 of the binning image data Ga, Gb, Gc, and Gd (4GBIN) of the four green element blocks GK adjacent to the red element block RK in the first direction and the second direction orthogonal to the first direction.
G ave4= (G a+G b+G c+G d) /4
In addition, as shown in the formula below, the main processor 40 calculates the first average value G aveV of the binning image data G a and G c of the two green element blocks GK adjacent to the red element block RK in the first direction.
G aveV= (G a+G c) /2
In addition, as shown in the formula below, the main processor 40 calculates the second average value G aveH of the binning image data G b and G d of the two green element blocks GK adjacent to the red element block RK in the second direction.
G aveH= (G b+G d) /2
In addition, as shown in the formula below, the main processor 40 calculates the first absolute value G difV of the difference between the binning image data G a and G c of the two green element blocks GK adjacent to the red element block RK in the first direction.
G difV=|G a-G c|
In addition, as shown in the formula below, the main processor 40 calculates the second absolute value G difH of the difference between the binning image data G b and G d of the two green element blocks GK adjacent to the red element block RK in the second direction.
G difH=|G b-G d|
In addition, as shown in the formula below, the main processor 40 calculates the reference value G dir based on the ratio of the first absolute value G difV and the second absolute value G difH.
G dir=|1-G difV/G difH|
Here, as shown in the following formula, when the reference value G dir is less than a preset threshold value T dir, the main processor 40 sets the total average value G ave4 as an interpolate green value G 4pix.
On the other hand, when the reference value G dir is equal to or higher than the threshold value T dir and the first absolute value G aveV is less than the second absolute value G aveH, the main processor 40 sets the first average value G difV as an interpolate green value G 4pix.
On the other hand, when the reference value G dir is equal to or greater than the threshold value and the first absolute value G aveV is equal to or greater than the second absolute value, the main processor 40 sets the second average value G difH as an interpolate green value G 4ix.
Figure PCTCN2020127263-appb-000003
Next, in step S12 of FIG. 7, as shown in the following formula, the main processor 40 uses half of the interpolate green value G 4pix as the estimated green binning image data G2pix. That is, 2 pixels binning G (G  2p  i  x ) is estimated from 4 pixels binning G (G  4p  i  x ) .
G 2pix=G 4pix/2
In this way, the main processor 40 calculates the estimated green binning image data G 2pix corresponding to the two green physical pixel elements G, based on the average value calculated from the binning image data of the plurality of green element blocks GK.
In particular, the main processor 40 calculates the estimated green binning image data G 2pix for calculating the estimated red binning image data ER, based on the average value G 4pix of the binning image data of the two or four green element blocks GK adjacent to the red element block RK.
Next, in step S13 of FIG. 7, as shown in the following formula, the main processor 40 calculates the estimated red binning image data ER, by subtracting the estimated green binning image data G 2pix adjacent to the red element block RK from the binning image data 4YBIN (RG) of the red element block RK.
ER=RG-G 2pix
According to the above flow, the main processor 40 of the electrical device 10 acquires the binning image data 4GBIN of the green element block GK, the estimated blue binning image data EB, and the estimated red binning image data ER, as image data conforming to the Bayer format.
Then, the main processor 40 outputs the acquired image data conforming to the Bayer format to, for example, the image signal processor 42. Then, the image signal processor 42 executes a predetermined image processing on the input image data.
Here, FIG. 8 is a diagram for explaining the effect of according to the embodiment of the present disclosure.
As described above, in accordance with the electrical device 10 according to the present embodiment, comprises a camera assembly 30, including an image sensor, configured to capture an image of an object and to generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements; and a main processor 40 that performs image process. The main processor 40 acquires binning image data of the element blocks, the binning image data being generated by the camera assembly 30 by combining charges of the four physical pixel elements of the element blocks, the main processor 40 calculates estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block, the main processor 40 calculates estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block, and the main processor 40 calculates estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
Therefore, as shown FIG. 8, the present invention proposes to perform 4 pixels binning in the new color filter array to reduce the effect of color mixing and improve the noise quality 2 pixels binning.
In the description of embodiments of the present disclosure, it is to be understood that terms such as "central" , "longitudinal" , "transverse" , "length" , "width" , "thickness" , "upper" , "lower" , "front" , "rear" , "back" , "left" , "right" , "vertical" , "horizontal" , "top" , "bottom" , "inner" , "outer" , "clockwise" and "counterclockwise" should be construed to refer to the orientation or the position as described or as shown in the drawings under discussion. These relative terms are only used to simplify description of the present disclosure, and do not indicate or imply that the device or element referred to must have a particular orientation, or constructed or operated in a particular orientation. Thus, these terms cannot be constructed to limit the present disclosure.
In addition, terms such as "first" and "second" are used herein for purposes of description and are not intended to indicate or imply relative importance or significance or to imply the number of indicated technical features. Thus, the feature defined with "first" and "second" may comprise one or more of this feature. In the description of the present disclosure, "a plurality of" means two or more than two, unless specified otherwise.
In the description of embodiments of the present disclosure, unless specified or limited otherwise, the terms "mounted" , "connected" , "coupled" and the like are used broadly, and may be, for example, fixed connections, detachable connections, or integral connections; may also be mechanical or electrical connections; may also be direct connections or indirect connections via intervening structures; may also be inner communications of two elements, which can be understood by those skilled in the art according to specific situations.
In the embodiments of the present disclosure, unless specified or limited otherwise, a  structure in which a first feature is "on" or "below" a second feature may include an embodiment in which the first feature is in direct contact with the second feature, and may also include an embodiment in which the first feature and the second feature are not in direct contact with each other, but are contacted via an additional feature formed therebetween. Furthermore, a first feature "on" , "above" or "on top of" a second feature may include an embodiment in which the first feature is right or obliquely "on" , "above" or "on top of" the second feature, or just means that the first feature is at a height higher than that of the second feature; while a first feature "below" , "under" or "on bottom of" a second feature may include an embodiment in which the first feature is right or obliquely "below" , "under" or "on bottom of" the second feature, or just means that the first feature is at a height lower than that of the second feature.
Various embodiments and examples are provided in the above description to implement different structures of the present disclosure. In order to simplify the present disclosure, certain elements and settings are described in the above. However, these elements and settings are only by way of example and are not intended to limit the present disclosure. In addition, reference numbers and/or reference letters may be repeated in different examples in the present disclosure. This repetition is for the purpose of simplification and clarity and does not refer to relations between different embodiments and/or settings. Furthermore, examples of different processes and materials are provided in the present disclosure. However, it would be appreciated by those skilled in the art that other processes and/or materials may be also applied.
Reference throughout this specification to "an embodiment" , "some embodiments" , "an exemplary embodiment" , "an example" , "a specific example" or "some examples" means that a particular feature, structure, material, or characteristics described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Thus, the appearances of the above phrases throughout this specification are not necessarily referring to the same embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.
Any process or method described in a flow chart or described herein in other ways may be understood to include one or more modules, segments or portions of codes of executable instructions for achieving specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, in which it should be understood by those skilled in the art that functions may be implemented in a sequence other than the sequences shown or discussed, including in a substantially identical sequence or in an opposite sequence.
The logic and/or step described in other manners herein or shown in the flow chart, for example, a particular sequence table of executable instructions for realizing the logical function, may be specifically achieved in any computer readable medium to be used by the instruction execution system, device or equipment (such as the system based on computers, the system comprising processors or other systems capable of obtaining the instruction from the instruction execution system, device and equipment and executing the instruction) , or to be used in combination with the instruction execution system, device and equipment. As to the specification, "the computer readable medium" may be any device adaptive for including, storing, communicating, propagating or transferring programs to be used by or in combination with the instruction execution system, device or equipment. More specific examples of the computer readable medium comprise but are not limited to: an electronic connection (an electronic device) with one or more wires, a portable computer enclosure (a magnetic device) , a random access memory (RAM) , a read only memory (ROM) , an erasable programmable read-only memory (EPROM or a flash memory) , an optical fiber device and a portable compact disk read-only memory (CDROM) . In addition, the computer readable medium may even be a paper or other appropriate medium capable of printing programs thereon, this is because, for example, the paper or other appropriate medium may be optically scanned and then edited, decrypted or  processed with other appropriate methods when necessary to obtain the programs in an electric manner, and then the programs may be stored in the computer memories.
It should be understood that each part of the present disclosure may be realized by the hardware, software, firmware or their combination. In the above embodiments, a plurality of steps or methods may be realized by the software or firmware stored in the memory and executed by the appropriate instruction execution system. For example, if it is realized by the hardware, likewise in another embodiment, the steps or methods may be realized by one or a combination of the following techniques known in the art: a discrete logic circuit having a logic gate circuit for realizing a logic function of a data signal, an application-specific integrated circuit having an appropriate combination logic gate circuit, a programmable gate array (PGA) , a field programmable gate array (FPGA) , etc.
Those skilled in the art shall understand that all or parts of the steps in the above exemplifying method of the present disclosure may be achieved by commanding the related hardware with programs. The programs may be stored in a computer readable storage medium, and the programs comprise one or a combination of the steps in the method embodiments of the present disclosure when run on a computer.
In addition, each function cell of the embodiments of the present disclosure may be integrated in a processing module, or these cells may be separate physical existence, or two or more cells are integrated in a processing module. The integrated module may be realized in a form of hardware or in a form of software function modules. When the integrated module is realized in a form of software function module and is sold or used as a standalone product, the integrated module may be stored in a computer readable storage medium.
The storage medium mentioned above may be read-only memories, magnetic disks, CD, etc.
Although embodiments of the present disclosure have been shown and described, it would be appreciated by those skilled in the art that the embodiments are explanatory and cannot be construed to limit the present disclosure, and changes, modifications, alternatives and variations can be made in the embodiments without departing from the scope of the present disclosure.

Claims (10)

  1. An electrical device, comprising:
    a camera assembly, including an image sensor, configured to capture an image of an object and to generate color image data, wherein the image sensor has green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements; and
    a main processor that performs image process,
    wherein the main processor acquires binning image data of the element blocks, the binning image data being generated by the camera assembly by combining charges of the four physical pixel elements of the element blocks,
    the main processor calculates estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block,
    the main processor calculates estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block, and
    the main processor calculates estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
  2. The electrical device according to claim 1, wherein the main processor acquires the binning image data of the green element block, the estimated blue binning image data, and the estimated red binning image data, as image data conforming to the Bayer format.
  3. The electrical device according to claim 1, wherein
    the green element blocks, the blue element blocks, and the red element blocks have rectangular shape,
    in the green element block, the four green physical pixel elements are arranged corresponding to four corners, respectively,
    in the blue element block, the two blue physical pixel elements are located on a first diagonal line, and the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively, and
    in the red element block, the two red physical pixel elements are located on a first diagonal line, and the two green physical pixel elements are located on a second diagonal line, corresponding to each of four corners, respectively.
  4. The electrical device according to claim 1, wherein
    the main processor calculates the estimated blue binning image data, by subtracting the estimated green binning image data from the binning image data of the blue element block, and
    the main processor calculates the estimated red binning image data, by subtracting the estimated green binning image data from the binning image data of the red element block.
  5. The electrical device according to claim 4, wherein
    the main processor calculates the estimated green binning image data corresponding to the two green physical pixel elements, based on average value calculated from the binning image data of the plurality of green element blocks.
  6. The electrical device according to claim 5, wherein
    the main processor calculates the estimated green binning image data for calculating the estimated blue binning image data, based on the average value of the binning image data of the two or four green element blocks adjacent to the blue element block, and
    the main processor calculates the estimated green binning image data for calculating the estimated red binning image data, based on the average value of the binning image data of the two or four green element blocks adjacent to the red element block.
  7. The electrical device according to claim 6, wherein
    the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the blue element block in the first direction and the second direction orthogonal to the first direction,
    the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the blue element block in the first direction,
    the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the blue element block in the second direction,
    the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the first direction,
    the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the blue element block in the second direction,
    the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value, and
    wherein,
    when the reference value is less than a preset threshold value, the main processor sets half of the total average value as the estimated green binning image data,
    when the reference value is equal to or higher than the threshold value and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green binning image data, and
    when the reference value is equal to or greater than the threshold value and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green binning image data.
  8. The electrical device according to claim 6, wherein
    the main processor calculates total average value of the binning image data of the four green element blocks adjacent to the red element block in the first direction and the second direction orthogonal to the first direction,
    the main processor calculates a first average value of the binning image data of the two green element blocks adjacent to the red element block in the first direction,
    the main processor calculates a second average value of the binning image data of the two green element blocks adjacent to the red element block in the second direction,
    the main processor calculates a first absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the first direction,
    the main processor calculates a second absolute value of the difference between the binning image data of the two green element blocks adjacent to the red element block in the second direction,
    the main processor calculates a reference value based on a ratio of the first absolute value and the second absolute value, and
    wherein,
    when the reference value is less than a preset threshold value, the main processor sets half of the total average value as the estimated green binning image data,
    when the reference value is equal to or higher than the threshold value and the first absolute value is less than the second absolute value, the main processor sets half of the first average value as the estimated green binning image data, and
    when the reference value is equal to or greater than the threshold value and the first absolute value is equal to or greater than the second absolute value, the main processor sets half of the second average value as the estimated green binning image data.
  9. A method of generating an image data, comprising:
    acquiring binning image data of element blocks, the binning image data being generated by combining charges of the four physical pixel elements of the element blocks, wherein the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
    calculating estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block;
    calculating estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block; and
    calculating estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
  10. A non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following:
    acquiring binning image data of element blocks, the binning image data being generated by combining charges of the four physical pixel elements of the element blocks, wherein the element blocks includes green element blocks, blue element blocks, and red element blocks arranged in an array of Bayer format at each pixel position in order to generate color image data, the green element block, the blue element block, and the red element block each including four physical pixel elements, and wherein the green element block comprises four green physical pixel elements, the blue element block comprises two blue physical pixel elements and two green physical pixel elements, and the red element block comprises two red physical pixel elements and two green physical pixel elements;
    calculating estimated green binning image data corresponding to the two green physical pixel elements based on the binning image data of the green element block;
    calculating estimated blue binning image data, by subtracting estimated green binning image data from the binning image data of the blue element block; and
    calculating estimated red binning image data, by subtracting estimated green binning image data from the binning image data of the red element block.
PCT/CN2020/127263 2020-11-06 2020-11-06 Electrical device, method of generating image data, and non-transitory computer readable medium WO2022094970A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080105817.9A CN116324866A (en) 2020-11-06 2020-11-06 Electronic device, method of generating image data, and non-transitory computer readable medium
PCT/CN2020/127263 WO2022094970A1 (en) 2020-11-06 2020-11-06 Electrical device, method of generating image data, and non-transitory computer readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/127263 WO2022094970A1 (en) 2020-11-06 2020-11-06 Electrical device, method of generating image data, and non-transitory computer readable medium

Publications (1)

Publication Number Publication Date
WO2022094970A1 true WO2022094970A1 (en) 2022-05-12

Family

ID=81458516

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/127263 WO2022094970A1 (en) 2020-11-06 2020-11-06 Electrical device, method of generating image data, and non-transitory computer readable medium

Country Status (2)

Country Link
CN (1) CN116324866A (en)
WO (1) WO2022094970A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150189198A1 (en) * 2014-01-02 2015-07-02 Byung-Chul Park Method of binning pixels in an image sensor and an image sensor for performing the same
CN108377369A (en) * 2018-02-05 2018-08-07 中国科学院长春光学精密机械与物理研究所 The Binning methods of Bayer format coloured image
CN110517189A (en) * 2019-08-02 2019-11-29 Oppo广东移动通信有限公司 Image processing method and device and storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150189198A1 (en) * 2014-01-02 2015-07-02 Byung-Chul Park Method of binning pixels in an image sensor and an image sensor for performing the same
CN108377369A (en) * 2018-02-05 2018-08-07 中国科学院长春光学精密机械与物理研究所 The Binning methods of Bayer format coloured image
CN110517189A (en) * 2019-08-02 2019-11-29 Oppo广东移动通信有限公司 Image processing method and device and storage medium

Also Published As

Publication number Publication date
CN116324866A (en) 2023-06-23

Similar Documents

Publication Publication Date Title
US10531019B2 (en) Image processing method and apparatus, and electronic device
US10348962B2 (en) Image processing method and apparatus, and electronic device
US10339632B2 (en) Image processing method and apparatus, and electronic device
US10559070B2 (en) Image processing method and apparatus, and electronic device
US10264178B2 (en) Control method and apparatus, and electronic device
US10438321B2 (en) Image processing method and apparatus, and electronic device
EP3327665B1 (en) Image processing method and apparatus, and electronic device
US10249021B2 (en) Image processing method and apparatus, and electronic device
US10382709B2 (en) Image processing method and apparatus, and electronic device
US10165205B2 (en) Image processing method and apparatus, and electronic device
US10262395B2 (en) Image processing method and apparatus, and electronic device
US9143747B2 (en) Color imaging element and imaging device
US9185375B2 (en) Color imaging element and imaging device
CN102752607A (en) Method and device for achieving trailing special effect during image moving
WO2022094970A1 (en) Electrical device, method of generating image data, and non-transitory computer readable medium
WO2021243709A1 (en) Method of generating target image data, electrical device and non-transitory computer readable medium
WO2022094937A1 (en) Electrical device, method of generating image data, and non-transitory computer readable medium
WO2022016385A1 (en) Method of generating corrected pixel data, electrical device and non-transitory computer readable medium
WO2022047671A1 (en) Method of removing noise in image and electrical device
KR101137611B1 (en) System and method for effectively performing an image data transformation procedure
WO2021138867A1 (en) Method for electronic device with a plurality of cameras and electronic device
WO2022047614A1 (en) Method of generating target image data, electrical device and non-transitory computer readable medium
WO2021253166A1 (en) Method of generating target image data and electrical device
WO2022174460A1 (en) Sensor, electrical device, and non-transitory computer readable medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20960431

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20960431

Country of ref document: EP

Kind code of ref document: A1