CN114363486A

CN114363486A - Image sensor, camera module, electronic equipment, image generation method and device

Info

Publication number: CN114363486A
Application number: CN202111524625.4A
Authority: CN
Inventors: 李小涛
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-12-14
Filing date: 2021-12-14
Publication date: 2022-04-15
Also published as: WO2023109264A1

Abstract

The application relates to an image sensor, a camera module, an electronic device, an image generation method, an apparatus, a computer-readable storage medium and a computer program product. The image sensor comprises a filter array and a pixel array, wherein the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; each subunit comprises a panchromatic optical filter and a color optical filter, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the full-color filter is greater than that transmitted by the color filter; each filter comprises N rows and N columns of sub-filters with the same color as the filter, wherein N is a positive integer. The image sensor can improve the definition of images.

Description

Image sensor, camera module, electronic equipment, image generation method and device

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an image sensor, a camera module, an electronic device, an image generating method, an image generating apparatus, a computer-readable storage medium, and a computer program product.

Background

With the development of computer technology, most of electronic devices such as mobile phones and the like are equipped with cameras so as to realize a photographing function through the cameras. An image sensor is arranged in the camera, and a color image is acquired through the image sensor. In order to realize the acquisition of color images, a filter array arranged in a Bayer (Bayer) array is generally disposed in an image sensor, so that a plurality of pixels in the image sensor can receive light passing through corresponding filters, thereby generating pixel signals having different color channels, and further generating an image.

However, the image produced by conventional image sensors is of low sharpness.

Disclosure of Invention

The embodiment of the application provides an image sensor, a camera module, an electronic device, an image generation method, an image generation device, a computer readable storage medium and a computer program product, which can improve the definition of imaging.

An image sensor comprising a filter array and a pixel array, the filter array comprising a minimal repeating unit comprising a plurality of filter sets, each filter set comprising a first subunit and a second subunit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal.

A camera module comprises a lens and the image sensor; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

An electronic device, comprising:

the camera module; and

the casing, the module setting of making a video recording is in on the casing.

The image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of optical filter sets, and each optical filter set comprises a first subunit and a second subunit; the first subunit and the second subunit both comprise panchromatic filters and color filters, the color filters in the first subunit are arranged on diagonal lines in the first subunit, the color filters in the second subunit are arranged on anti-diagonal lines in the second subunit, the arrangement of the color filters in the diagonal direction and the anti-diagonal direction is more balanced, and the color channels have stronger resolving power during imaging.

The light quantity transmitted by the panchromatic filter is larger than that transmitted by the color filter, and more light quantity can be obtained through the panchromatic filter during shooting, so that shooting parameters do not need to be adjusted, and the imaging definition under dark light is improved under the condition that the shooting stability is not influenced. When imaging under the dim light, stability and definition can be considered, and the stability and the definition of imaging under the dim light are both higher.

An image generation method is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal;

the method comprises the following steps:

in a full-resolution mode, reading out full-resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters, and reading out full-resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters;

generating a full-resolution target image based on each of the full-resolution panchromatic pixel values and each of the full-resolution color pixel values.

An image generation device is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repetition unit, the minimum repetition unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal

The device comprises:

the reading module is used for reading out full-resolution panchromatic pixel values of panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters and reading out full-resolution color pixel values of color pixels corresponding to each color sub-filter in the color filters in a full-resolution mode;

an image generation module to generate a full-resolution target image based on each of the full-resolution panchromatic pixel values and each of the full-resolution color pixel values.

An electronic device comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of the image generation method as described above.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method as described above.

A computer program product comprising a computer program which, when executed by a processor, carries out the steps of the method as described above.

In the full-resolution mode, the full-resolution panchromatic pixel value is read out from the panchromatic pixel corresponding to each panchromatic sub-filter in the panchromatic filter, and the full-resolution color pixel value is read out from the color pixel corresponding to each color sub-filter in the color filter; and the light inlet quantity of the panchromatic filter is larger than that of the color filter, so that the panchromatic channel information can be fused into the image, and the whole light inlet quantity is improved, and therefore, the full-resolution target image with more information and clearer detail analysis can be generated based on each full-resolution panchromatic pixel value and each full-resolution color pixel value.

In the filter array, the minimum repeating unit comprises a plurality of filter sets, each filter set comprising a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic filters and color filters, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on anti-diagonal lines in the second subunit, so that the arrangement of the color filters in the diagonal direction and the anti-diagonal direction is more balanced, and the color channels have stronger resolving power when a full-resolution target image is generated.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of an electronic device in one embodiment;

FIG. 2 is an exploded view of an image sensor in one embodiment;

FIG. 3 is a schematic diagram of the connection of a pixel array and readout circuitry in one embodiment;

FIG. 4 is a schematic diagram illustrating an arrangement of minimum repeating units in a filter array in which N is 1 according to an embodiment;

FIG. 5 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 1;

FIG. 6 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 1;

FIG. 7 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 1;

FIG. 8 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to an embodiment where N is 2;

FIG. 9 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 2;

FIG. 10 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 2;

FIG. 11 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 2;

FIG. 12 is a diagram illustrating an arrangement of minimum repeating units in a filter array according to another embodiment where N is 3;

FIG. 13 is a schematic flow chart diagram illustrating an image generation method according to an embodiment;

FIG. 14 is a schematic flow chart diagram illustrating an image generation method according to an embodiment;

FIG. 15 is a schematic illustration of a first target image in one embodiment;

FIG. 16 is a schematic flow chart illustrating the generation of a second target image in one embodiment;

FIG. 17 is a schematic of a first color image and a first panchromatic image in one embodiment;

FIG. 18 is a diagram illustrating a second target image in one embodiment;

FIG. 19 is a schematic view of a second target image in another embodiment;

FIG. 20 is a schematic illustration of a second target image in another embodiment;

FIG. 21 is a schematic view of a second target image in another embodiment;

FIG. 22 is a schematic flow chart illustrating the generation of a third target image in one embodiment;

FIG. 23 is a schematic of a second panchromatic image, a bi-color second color image and a single color second color image in one embodiment;

FIG. 24 is a schematic illustration of a third target image in one embodiment;

FIG. 25 is a schematic view of a third target image in another embodiment;

FIG. 26 is a schematic flow chart illustrating the generation of a fourth target image in one embodiment;

FIG. 27 is a diagram illustrating a fourth target image, according to an embodiment;

FIG. 28 is a block diagram showing the configuration of an image generating apparatus according to an embodiment;

fig. 29 is a schematic diagram of the internal structure of the electronic device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image, without departing from the scope of the present application. Both the first target image and the second target image are target images, but they are not the same target image.

In one embodiment, the electronic device 100 includes a mobile phone, a tablet computer, a notebook computer, a teller machine, a gate, a smart watch, a head-up display device, etc., and it is understood that the electronic device 100 may also be any other device with image processing function. The electronic device 100 includes a camera module 20, a processor 30, and a housing 40. The camera module 20 and the processor 30 are both disposed in the casing 40, and the casing 40 can also be used to mount functional modules of the electronic device 100, such as a power supply device and a communication device, so that the casing 40 provides protection against dust, falling, water, and the like for the functional modules.

The camera module 20 may be a front camera module, a rear camera module, a side camera module, a screen camera module, etc., without limitation. The camera module 20 includes a lens and an image sensor 21, when the camera module 20 takes an image, light passes through the lens and reaches the image sensor 21, and the image sensor 21 is used for converting an optical signal irradiated onto the image sensor 21 into an electrical signal.

In one embodiment, as shown in FIG. 2, the image sensor 21 includes a microlens array 22, a filter array 23, and a pixel array 24.

The micro lens array 22 includes a plurality of micro lenses 221, the sub-filters in the filter array 23, and the pixels in the pixel array 24 are arranged in a one-to-one correspondence, the micro lenses 221 are configured to collect incident light, the collected light passes through the corresponding sub-filters, then is projected onto the pixels, and is received by the corresponding pixels, and the pixels convert the received light into electrical signals.

In one embodiment, the filter array 23 includes a plurality of minimal repeating units 230. Minimal repeating unit 230 includes a plurality of filter sets, each filter set including a first subunit 233 and a second subunit 234; the first sub-unit 233 and the second sub-unit 234 each include a panchromatic filter 235 and a color filter 236, the color filters 236 in the first sub-unit 233 are arranged on diagonal lines in the first sub-unit 233, and the color filters 236 in the second sub-unit 234 are arranged on opposite diagonal lines in the second sub-unit 234; the amount of light entering the panchromatic filter 235 is greater than the amount of light entering the color filter 236; each panchromatic filter 235 includes N rows and N columns of panchromatic sub-filters, each color filter 236 includes N rows and N columns of color sub-filters, the N rows and N columns of color sub-filters are the same color as the color filter, and N is a positive integer. In fig. 2, N is 2.

The diagonal line may be a connecting line between the upper left corner and the lower right corner, or a connecting line between the upper right corner and the lower left corner. The anti-diagonal line can be a connecting line of the upper left corner and the lower right corner, and can also be a connecting line of the upper right corner and the lower left corner. The diagonal and anti-diagonal are perpendicular to each other. That is, if the diagonal line is the connecting line of the upper left corner and the lower right corner, the anti-diagonal line is the connecting line of the upper right corner and the lower left corner; if the diagonal line is the line connecting the upper right corner and the lower left corner, the anti-diagonal line is the line connecting the upper left corner and the lower right corner.

The image sensor comprises a filter array 23 and a pixel array 24, wherein the filter array 23 comprises a minimum repeating unit 230, the minimum repeating unit 230 comprises a plurality of filter sets, and each filter set comprises a first subunit 233 and a second subunit 234; the first subunit 233 and the second subunit 234 both include a panchromatic filter 235 and a color filter 236, the color filter 236 in the first subunit 233 is arranged on a diagonal line in the first subunit 233, and the color filter in the second subunit 234 is arranged on an opposite diagonal line in the second subunit 234, so that the arrangement of the color filters 236 in the diagonal direction and the opposite diagonal direction is more balanced, and the color channel has stronger resolving power during imaging.

The amount of light transmitted through the panchromatic filter 235 is greater than that transmitted through the color filter 236, and more light can be obtained through the panchromatic filter 236 during shooting, so that shooting parameters do not need to be adjusted, and the imaging definition under dark light is improved under the condition that the shooting stability is not affected. When imaging under the dim light, stability and definition can be considered, and the stability and the definition of imaging under the dim light are both higher.

In one embodiment, each row and each column of the minimum repeating unit 230 includes the color filter 236 of each color, that is, the color filters 236 of the respective colors are arranged in a dispersed manner, so that the color resolution and the brightness change resolution are improved, and the color filters 236 of the respective colors are arranged in a mixed manner, so that the risk of false color is reduced. It is understood that the panchromatic filter 235 and the color filter 236 are alternately arranged in each row and each column, that is, the panchromatic filter 235 accounts for 50% of the minimum repeating unit, the first filter set or the second filter set, and the light entering amount of each local area in the image can be increased.

In one embodiment, the panchromatic filter 235 and the color filter 236 are alternately arranged on each row or each column, so that the color resolution of each row or each column of the image can be improved, and the image can be more colorful.

In one embodiment, minimum repeating unit 230 includes 2 first filter sets 231 and 2 second filter sets 232, 2 first filter sets 231 arranged on a diagonal of minimum repeating unit 230, and 2 second filter sets 232 arranged on an opposite diagonal of minimum repeating unit 230. The 2 first filter sets 231 and the 2 second filter sets 232 are arranged in a matrix.

Wherein, 2 first filter sets are completely the same, and 2 second filter sets are completely the same.

In one embodiment, each filter set includes only panchromatic filters 235 and 2-color filters 236. The color filters 236 include a first color filter, a second color filter, and a third color filter. The first color filter, the second color filter and the third color filter are filters of three different colors. The colors of the first color filter, the second color filter and the third color filter can be set according to the requirement. For example, the first color filter may be a red filter, the second color filter may be a green filter, and the third color filter may be a blue filter.

For example, the wavelength band of the light transmitted by the color filter 236 may correspond to the wavelength band of red light, the wavelength band of green light, or the wavelength band of blue light, and the wavelength band of the light transmitted by the panchromatic filter 235 is the wavelength band of all visible light, that is, the color filter 236 only allows light of a specific color to pass through, while the panchromatic filter 235 can pass light of all colors. Of course, the wavelength band of the transmitted light of the color filter 236 may also correspond to the wavelength band of other color lights, such as magenta light, purple light, cyan light, yellow light, etc., and is not limited herein.

Further, the color filters 236 in the first filter set 231 include a first color filter and a second color filter, and the color filters 236 in the second filter set 232 include a second color filter and a third color filter.

In one embodiment, each filter set includes 2 first subunits 233 and 2 second subunits 234; 2 first subunits 233 are arranged in a first row direction of the filter set, and 2 second subunits 234 are arranged in a second row direction of the filter set, the first row direction and the second row direction being arranged adjacently; alternatively, 2 first subunits 233 are arranged in a first column direction of the filter set, and 2 second subunits 234 are arranged in a second column direction of the filter set, the first column direction and the second column direction being adjacently arranged.

As shown in fig. 2, the first filter set 231 includes 2 first subunits 233 and 2 second subunits 234, the 2 first subunits 233 are arranged in the first row direction of the first filter set 231, and the 2 second subunits 234 are arranged in the second row direction of the filter set. In other embodiments, 2 first subunits 233 are arranged in a first column direction of first filter set 231 and 2 second subunits 234 are arranged in a second column direction of the filter set.

Further, the first sub-unit 233 and the second sub-unit 234, which contain filters of the same color in the same filter set, are arranged on the diagonal of the same filter set.

The first sub-unit 233 and the second sub-unit 234, which include filters of the same color, in the same filter set are arranged on the diagonal line of the same filter set, and the color filters 236 in the first sub-unit are arranged on the diagonal line in the first sub-unit, and the color filters 236 in the second sub-unit are arranged on the anti-diagonal line in the second sub-unit, so that the color filters 236 in the first sub-unit 233 arranged on the diagonal line of the same filter set are arranged on the diagonal line in the first sub-unit, and the color filters 236 in the second sub-unit 234 arranged on the diagonal line of the same filter set are arranged on the anti-diagonal line in the second sub-unit, which can improve the distribution balance of the color filters 236 on the diagonal line and the anti-diagonal line.

It is to be understood that the first sub-unit 233 and the second sub-unit 234 each include a panchromatic filter 235 and a color filter 236, and thus the first sub-unit 233 and the second sub-unit 234 including the same color filter, that is, the first sub-unit 233 and the second sub-unit 234 including the same color filter 236. As shown in fig. 2, the first filter set 231 includes 2 first subunits 233 and 2 second subunits 234, the 2 first subunits 233 are arranged in the row direction of the first filter set 231, and the 2 second subunits 234 are arranged in the row direction of the first filter set 231, and the first subunits 233 and the second subunits 234 including filters of the same color are arranged on the diagonal line of the first filter set 231.

Further, the color filters 236 include a first color filter, a second color filter, and a third color filter; one first subunit 233 of the 2 first subunits 233 in the first filter set 231 includes a first color filter, and the other first subunit 233 includes a second color filter; one of the second subunits 234 of the 2 second subunits 234 in the first filter set 231 includes a first color filter, and the other second subunit 234 includes a second color filter.

Similarly, the color filters 236 include a first color filter, a second color filter, and a third color filter; one first subunit 233 of the 2 first subunits 233 in the second filter set 232 includes a second color filter, and the other first subunit 233 includes a third color filter; one second subunit 234 of the 2 second subunits 234 in the second filter set 232 includes a second color filter, and the other second subunit 234 includes a third color filter.

The pixel array 24 includes a plurality of pixels, and the pixels of the pixel array 24 are disposed corresponding to the sub-filters of the filter array 23. The pixel array 24 is configured to receive light rays passing through the filter array 23 to generate electrical signals.

Where the pixel array 24 is configured to receive light rays passing through the filter array 23 to generate an electrical signal, it means that the pixel array 24 is used to perform photoelectric conversion on light rays of a given set of scenes of a subject passing through the filter array 23 to generate an electrical signal. The rays of a scene of a given set of subjects are used to generate image data. For example, the subject is a building, and the scene of a given set of subjects refers to the scene in which the building is located, and other objects may be included in the scene.

In one embodiment, pixel array 24 includes a plurality of minimal repeating units 240, minimal repeating units 240 further including a plurality of panchromatic pixels 241 and a plurality of different color pixels 242, each row and each column including color pixels of each color; each panchromatic pixel 242 corresponds to one of the sub-filters in panchromatic filter 235, and panchromatic pixel 242 receives light passing through the corresponding sub-filter to generate an electrical signal. Each color pixel 242 corresponds to one sub-filter of the color filter 236, and the color pixel 242 receives light passing through the corresponding sub-filter to generate an electrical signal.

As shown in fig. 3, the readout circuit 25 is electrically connected to the pixel array 24, and controls exposure of the pixel array 24 and reading and output of pixel values of the pixels. The readout circuit 25 includes a vertical driving unit 251, a control unit 252, a column processing unit 253, and a horizontal driving unit 254. The vertical driving unit 251 includes a shift register and an address decoder. The vertical driving unit 251 includes a readout scanning and reset scanning functions. The control unit 252 configures timing signals according to an operation mode, and controls the vertical driving unit 251, the column processing unit 253, and the horizontal driving unit 254 to cooperatively operate using various timing signals. The column processing unit 253 may have an analog-to-digital (a/D) conversion function for converting analog pixel signals into a digital format. The horizontal driving unit 254 includes a shift register and an address decoder. The horizontal driving unit 254 sequentially scans the pixel array 24 column by column.

When N is 1, each panchromatic filter 235 includes 1 row and 1 column of panchromatic sub-filters, and each color filter 236 includes 1 row and 1 column of color sub-filters, that is, each panchromatic sub-filter is a panchromatic filter 235, and each color sub-filter is a color filter 236.

In one embodiment, as shown in fig. 4, N is 1, the minimum repeating unit includes 8 rows, 8 columns and 64 filters, and the arrangement is as follows:

where w denotes a panchromatic filter 235, and a, b, and c each denote a color filter 236.

In one embodiment, as shown in fig. 5, N is 1, the minimum repeating unit includes 8 rows, 8 columns and 64 filters, and the arrangement is as follows:

In one embodiment, as shown in fig. 6, N is 1, the minimum repeating unit includes 8 rows, 8 columns and 64 filters, and the arrangement is as follows:

In one embodiment, as shown in fig. 7, N is 1, the minimum repeating unit includes 8 rows, 8 columns and 64 filters, and the arrangement is as follows:

W may be a white filter, a is a red filter, b is a green filter, and c is a blue filter, or for example, a is a magenta filter, b is a cyan filter, and c is a yellow filter, and the like, which is not limited herein.

It is understood that the electronic device adjusts the minimal repeating unit to obtain a new minimal repeating unit. For example, the electronic device may rotate the minimal repeating unit of fig. 4 by 90 degrees counterclockwise, which may result in the minimal repeating unit of fig. 7; exchanging the positions of the color filter a and the color filter c in the minimum repeating unit of fig. 4, the minimum repeating unit of fig. 5 can be obtained; after the color filter a and the color filter c in the minimum repeating unit of fig. 4 are exchanged, they are further rotated 90 degrees counterclockwise, and the minimum repeating unit of fig. 6 can be obtained.

In the minimum repeating unit of fig. 4 to 7, the b color filters have the minimum area of 4 by 4 as the arrangement period, and the sampling rates of the b color filters in the diagonal and anti-diagonal directions are uniform, the arrangement is more uniform, and thus the b channel has a stronger resolving power in the horizontal direction, the vertical direction, and the oblique direction. Similarly, the sampling rates of the color filter a and the color filter c in the diagonal and anti-diagonal directions of the local area are consistent, the arrangement is more balanced, and therefore, the channel a and the channel c have stronger resolving power. The color filters a, b, c and w are arranged more dispersedly, so that the filter array has full arrangement property in the horizontal and vertical directions, and the resolution in the diagonal and diagonal directions is taken into account.

In one embodiment, as shown in fig. 8, N is 2, the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, and the arrangement is as follows:

where w denotes a panchromatic sub-filter, and a, b, and c each denote a color sub-filter.

In one embodiment, as shown in fig. 9, N is 2, the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, and the arrangement is as follows:

In one embodiment, as shown in fig. 10, N is 2, the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, and the arrangement is as follows:

In one embodiment, as shown in fig. 11, N is 2, the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, and the arrangement is as follows:

W may be a white sub-filter, a is a red sub-filter, b is a green sub-filter, and c is a blue sub-filter, or for example, a is a magenta sub-filter, b is a cyan sub-filter, and c is a yellow sub-filter, and the like, which is not limited herein.

It is understood that the electronic device adjusts the minimal repeating unit to obtain a new minimal repeating unit. For example, the electronic device may rotate the minimal repeating unit of fig. 8 by 90 degrees counterclockwise, which may result in the minimal repeating unit of fig. 11; exchanging the positions of the color filter a and the color filter c in the minimum repeating unit of fig. 8, the minimum repeating unit of fig. 9 can be obtained; after the color filter a and the color filter c in the minimum repeating unit of fig. 8 are exchanged, they are further rotated 90 degrees counterclockwise, and the minimum repeating unit of fig. 10 can be obtained.

In the minimum repeating unit of fig. 8 to 11, the b color sub-filters have the minimum area of 8 by 8 as the arrangement period, and the sampling rates of the b color sub-filters in the diagonal and anti-diagonal directions are the same, the arrangement is more uniform, and thus the b channel has a stronger resolving power in the horizontal direction, the vertical direction and the oblique direction. Similarly, the sampling rates of the color sub-filters a and the color sub-filters c in the diagonal and anti-diagonal directions of the local area are consistent, the arrangement is more balanced, and therefore the channels a and c have stronger resolving power. The color sub-filters a, the color sub-filters b, the color sub-filters c and the panchromatic sub-filters w are arranged more dispersedly, so that the filter array has the property of full arrangement in the horizontal and vertical directions, and the resolution in the diagonal and oblique diagonal directions is taken into consideration.

It should be noted that N may also be 3, 4, or 5 or other positive integers, and the arrangement manner is similar to that where N is 1 or 2, which is not described herein again. Fig. 12 is a minimum repeating unit in one embodiment where N is 3.

In one embodiment, a camera module is further provided, and the camera module comprises a lens and the image sensor; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

In an embodiment, an electronic device is further provided, which includes the camera module; and the camera module is arranged on the shell.

In one implementation, an image generation method is provided and applied to an image sensor, the image sensor including a filter array and a pixel array, the filter array including a minimum repeating unit, the minimum repeating unit including a plurality of filter sets, each filter set including a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the full-color filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and the N columns of color sub-filters are the same as those of the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is disposed corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

As shown in fig. 13, the image generation method includes:

in step 1302, in the full-resolution mode, a full-resolution panchromatic pixel value is read out from a panchromatic pixel corresponding to each panchromatic sub-filter in the panchromatic filter, and a full-resolution color pixel value is read out from a color pixel corresponding to each color sub-filter in the color filter.

The full resolution mode is a mode in which each sub-filter is read out as one pixel.

The color filter has a narrower spectral response than that of the panchromatic filter, so that the incident light quantity transmitted by the panchromatic filter is greater than that transmitted by the color filter, namely the wave band width of the light transmitted by the color filter is smaller than that transmitted by the panchromatic filter, the panchromatic filter transmits more light, a corresponding panchromatic pixel obtained through the panchromatic filter has a higher signal-to-noise ratio, and the panchromatic pixel contains more information and can analyze more texture details. The signal-to-noise ratio refers to a ratio between a normal signal and a noise signal. The higher the signal-to-noise ratio of a pixel, the higher the proportion of normal signals that the pixel contains, and the more information that is resolved from the pixel.

The color pixels 242 may be G (Green), R (Red, Blue), B (Blue), and the like, but are not limited thereto.

And under the condition that a shooting instruction is received, detecting whether a resolution mode required to be used is selected by a user, and under the condition that the resolution mode required to be used is not selected by the user, or under the condition that the resolution mode required to be used is not selected by the user, the preview shooting is not used, and the current environment is not in a night scene mode, responding to the shooting instruction by using the full resolution mode.

In the full resolution mode, light transmitted by a panchromatic sub-filter in the panchromatic filter is projected onto a corresponding panchromatic pixel 241, and the panchromatic pixel 241 receives the light transmitted by the panchromatic sub-filter to generate an electrical signal. The light transmitted by the color sub-filters in the color filters is projected onto the corresponding color pixels 242, and the color pixels 242 transmit the light of the corresponding color sub-filters to generate electrical signals.

Each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, and each panchromatic filter corresponds to N rows and N columns of panchromatic pixels 241. Each color filter includes N rows and N columns of color sub-filters of the same color, and each color filter corresponds to N rows and N columns of color pixels 242. N is a positive integer greater than or equal to 2.

In other embodiments, N may also be 1, i.e., 1 panchromatic pixel 241 per panchromatic filter and 1 color pixel 242 per color filter.

At step 1304, a full-resolution target image is generated based on each full-resolution panchromatic pixel value and each full-resolution color pixel value.

The electronic device may read pixel values from each full-resolution panchromatic pixel value and each full-resolution color pixel value to generate a full-resolution target image according to a preset pixel reading mode. The preset pixel reading mode is a preset pixel reading mode.

In the image generation method, in the full-resolution mode, a full-resolution full-color pixel value is read out from a full-color pixel corresponding to each full-color sub-filter in the full-color filter, and a full-resolution color pixel value is read out from a color pixel corresponding to each color sub-filter in the color filter; and the light inlet quantity of the panchromatic filter is larger than that of the color filter, so that the panchromatic channel information can be fused into the image, and the whole light inlet quantity is improved, and therefore, the full-resolution target image with more information and clearer detail analysis can be generated based on each full-resolution panchromatic pixel value and each full-resolution color pixel value.

In one embodiment, as shown in fig. 14, the method further includes:

step 1402, in the first resolution mode, merging and reading out panchromatic pixels corresponding to each panchromatic sub-filter in each panchromatic filter to obtain a first panchromatic pixel value, and merging and reading out color pixels corresponding to each color sub-filter in each color filter to obtain a first color pixel value; the resolution corresponding to the first resolution mode is less than the resolution corresponding to the full resolution mode.

The first resolution mode refers to a one-level pixel merging and reading mode with balanced resolution, power consumption, signal-to-noise ratio and frame rate. The first resolution mode may specifically be a default mode for image, video capturing.

And under the condition that a shooting instruction is received, detecting whether a resolution mode required to be used is selected by a user, and responding to the shooting instruction by using the first resolution mode under the condition that the first resolution mode is detected to be selected by the user or under the condition that the resolution mode required to be used is not selected by the user, preview shooting is not used and a current environment is not a night scene mode.

In the first resolution mode, light transmitted by a panchromatic sub-filter in the panchromatic filters is projected onto a corresponding panchromatic pixel 241, and the panchromatic pixel 241 receives the light transmitted by the panchromatic sub-filter to generate an electrical signal. The light transmitted by the color sub-filters in the color filters is projected onto the corresponding color pixels 242, and the color pixels 242 transmit the light of the corresponding color sub-filters to generate electrical signals.

The merged readout refers to summing pixel values of a plurality of pixels or calculating an average value of the pixel values of the plurality of pixels.

In one embodiment, for each panchromatic filter, the panchromatic pixels 241 corresponding to the respective panchromatic sub-filters are averaged and the average is read out as the first panchromatic pixel value. In another embodiment, for each panchromatic filter, the panchromatic pixels 241 corresponding to the respective panchromatic sub-filters are added, and the sum obtained by the addition is read out as the first panchromatic pixel value. In other embodiments, the electronic device may also use other methods to combine the panchromatic pixels 241 corresponding to the panchromatic sub-filters to read out the first panchromatic pixel values, which is not limited herein.

In one embodiment, the color pixels 242 corresponding to the respective color sub-filters are averaged for each color filter, and the average value is read as the first color pixel value. In another embodiment, for each color filter, the color pixels corresponding to the respective color sub-filters are added, and the sum obtained by the addition is read out as the first color pixel value. In other embodiments, the electronic device may also use other methods to combine the color pixels 242 corresponding to the color sub-filters to read out the first color pixel values, which is not limited herein.

Note that the manner of reading out the first panchromatic pixel values in combination may be the same or different for each panchromatic filter. The manner of reading out the first color pixel values in combination may be the same for each color filter or may be different. The manner of reading out the first panchromatic pixel value and the first color pixel value in combination may be the same or different for the panchromatic filter and the color filter.

At step 1404, a first target image is generated based on the respective first panchromatic pixel values and the respective first color pixel values.

The electronics can read pixel values from the respective first panchromatic pixel values and the respective first color pixel values in accordance with a preset pixel reading scheme to generate a first target image. The preset pixel reading mode is a preset pixel reading mode. Taking the minimum repeating unit of the filter array 23 as the arrangement of fig. 8 as an example, the generated first target image is as shown in fig. 15. In the first target image of fig. 15, w denotes a first panchromatic pixel value, and a, b, and c denote first color pixel values of three different colors.

In the embodiment, in the first resolution mode, panchromatic pixels 241 corresponding to all panchromatic sub-filters in each panchromatic filter are combined and read out to obtain a first panchromatic pixel value, and color pixels 242 corresponding to all color sub-filters in each color filter are combined and read out to obtain a first color pixel value, and the amount of light transmitted by the panchromatic filters is larger than the amount of light transmitted by the color filters, so that panchromatic channel information can be fused into an image, the overall light input amount can be improved, and a first target image with more information and clearer detail analysis can be generated based on all the first panchromatic pixel values and all the first color pixel values.

In one embodiment, as shown in fig. 16, after the first target image is generated, the method further includes:

a step 1602, in the second resolution mode, reading out the second panchromatic pixel values in a combination manner corresponding to the plurality of first panchromatic pixel values in each subunit in the first target image, and generating a first panchromatic image based on the respective second panchromatic pixel values; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode, and the subunits comprise a first subunit and a second subunit.

The second resolution mode is a mode used in a scene where the resolution requirement is lower than that of the first resolution mode, and is a two-level pixel merge read mode with low resolution, low power consumption, high signal-to-noise ratio, and high frame rate. The resolution and power consumption corresponding to the second resolution mode are smaller than those corresponding to the first resolution mode. The signal-to-noise ratio and the frame rate corresponding to the second resolution mode are larger than those corresponding to the first resolution mode.

The second resolution mode may specifically be a preview mode in image capturing, a preview mode in video capturing, or a night view mode in image capturing and video capturing in a night view, which has low resolution requirements, but is not limited thereto. Preview modes of video shooting such as 1080p video preview, application video preview, and the like.

In the case where a shooting instruction is received, it is determined whether the shooting instruction is preview shooting. If the shooting instruction is preview shooting, the second resolution mode is triggered. Or the electronic device detects whether the current environment is a night scene, and triggers the second resolution mode under the condition that the current environment is the night scene. Alternatively, when the user selects the second resolution mode, the readout mode corresponding to the second resolution mode is triggered.

Specifically, the electronic device merges and reads out second panchromatic pixel values from a plurality of first panchromatic pixel values corresponding to each of the sub-units in the first target image, and reads the pixel values from the respective second panchromatic pixel values in accordance with a preset pixel reading manner to generate a first panchromatic image.

It can be understood that each pixel value in the first target image is obtained by combining the pixels corresponding to the sub-filters in each filter in the filter array in the first resolution mode, and each pixel value in the first target image corresponds to each filter in the filter array and also corresponds to a plurality of sub-filters in each filter.

In the second resolution mode, the electronics determine a plurality of pixel values for each subunit in the first target image, obtain a plurality of first panchromatic pixels from the plurality of pixel values and read out second panchromatic pixel values, and obtain a plurality of first color pixel values of the same color from the plurality of pixel values and read out second color pixel values.

It is understood that the combination of the readouts may include one of averaging, summing, or weighted averaging, and is not limited herein.

Step 1604 combines and reads out second color pixel values of the first target image corresponding to a plurality of same colors in each sub-cell, and generates a first color image based on the respective second color pixel values.

Specifically, in the second resolution mode, the electronic device reads out the second color pixel values in combination of the first color pixel values corresponding to the plurality of same colors in each sub-unit in the first target image, and reads the pixel values from the respective second color pixel values according to a preset pixel reading manner to generate the first color image.

Taking fig. 15 as an example of the first target image, the first color image is generated as indicated by 1702 in fig. 17, and the first full-color image is generated as indicated by 1704. In fig. 17, w denotes a second full-color pixel value, and a, b, and c denote second color pixel values of three different colors.

Step 1606 generates a second target image based on the first panchromatic image and the first color image.

When the first panchromatic image and the first color image need to be packed and transmitted, the electronic device can generate a second target image based on the first panchromatic image and the first color image and transmit the second target image.

Specifically, the electronic device arranges the second panchromatic pixel values of each line in the first panchromatic image and the second color pixel values of each line in the first color image to generate a second target image; or arranging the second panchromatic pixel value of each column in the first panchromatic image and the second color pixel value of each column in the first color image to generate a second target image.

Fig. 18 and 19 are schematic diagrams of a second target image obtained by arranging the second panchromatic pixel values in each row of the first panchromatic image and the second color pixel values in each row of the first color image. Fig. 20 and 21 are second target images obtained by arranging the second panchromatic pixel values of each column in the first panchromatic image and the second color pixel values of each column in the first color image.

In another embodiment, the electronic device may further combine the pixel values at the same position in the first panchromatic image and the first color image to obtain combined pixel values at corresponding positions, and construct the second target image based on the combined pixel values. The combination may adopt one of averaging, weighted averaging, or summing.

In other embodiments, the electronic device may also generate the second target image in other manners, which is not limited herein.

In the present embodiment, in the second resolution mode, the plurality of first panchromatic pixel values corresponding to each sub-unit in the first target image are read out in combination with the second panchromatic pixel values, and the plurality of first color pixel values corresponding to the same color in each sub-unit in the first target image are read out in combination with the second color pixel values, so that the different color pixels 242 can be arranged in a mixed manner, and the second color pixels, such as RGB pixels, in the generated second target image are distributed more uniformly and have higher image quality. In addition, the resolution and the image size of the obtained second target image are further reduced, the panchromatic pixel 241 has a higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effects of lower power consumption and better signal-to-noise ratio of the two-level pixel combination output are achieved. In addition, under the second resolution mode, full-color pixels are arranged in a full-array mode, interpolation is not needed, and the overall resolution is improved. Meanwhile, in the full-size case, color pixels of respective colors such as the first color pixel and the third color pixel are more dispersedly balanced in the diagonal direction or the anti-diagonal direction.

In another embodiment, the method further comprises: under the second resolution mode, merging and reading out fifth panchromatic pixel values of panchromatic pixels corresponding to the panchromatic sub-filters of the plurality of panchromatic filters in each first sub-unit or the second sub-unit, and generating a third panchromatic image based on the fifth panchromatic pixel values; combining and reading out fifth color pixel values of color pixels corresponding to the color sub-filters of the multiple color filters with the same color in each first sub-unit or second sub-unit, and generating a third color image based on the fifth color pixel values; a fifth target image is generated based on the third panchromatic image and the third color image.

The manner of the combined reading may be one of averaging, weighted averaging, or adding.

When the third full-color image and the third color image need to be packed and transmitted, the electronic device may generate a fifth target image based on the third full-color image and the third color image, and then transmit the fifth target image.

Wherein generating a fifth target image based on the third panchromatic image and the third color image comprises: arranging the fifth panchromatic pixel value of each line in the third panchromatic image and the fifth color pixel value of each line in the third color image to generate a fifth target image; or arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third panchromatic image to generate a fifth target image.

In another embodiment, the electronic device may further combine pixel values of the same position in the third panchromatic image and the third color image to obtain combined pixel values of corresponding positions, and construct the fifth target image based on the combined pixel values. The merging and reading can adopt one of averaging, weighted averaging or adding and summing.

In other embodiments, the electronic device may also generate the fifth target image in other manners, which is not limited herein.

In this embodiment, in the second resolution mode, the third panchromatic image and the third color image can be generated more quickly by reading out the fifth panchromatic pixel value by combining the panchromatic pixels corresponding to the respective panchromatic sub-filters of the plurality of panchromatic filters in each of the first sub-unit or the second sub-unit and reading out the fifth color pixel value by combining the color pixels corresponding to the respective color sub-filters of the plurality of same-color filters in each of the first sub-unit or the second sub-unit, thereby generating the fifth target image more quickly.

Moreover, the above embodiment can mix and arrange different color pixels, so that the fourth color pixel values, such as RGB pixels, in the generated fifth target image are distributed more uniformly and the image quality is higher. In addition, the resolution and the image size of the obtained fifth target image are further reduced, the panchromatic pixel 241 has a higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effects of lower power consumption and better signal-to-noise ratio of the two-level pixel combination output are achieved.

In one embodiment, as shown in fig. 22, the method further includes:

step 2202, in a third resolution mode, reading out third panchromatic pixel values in a combination manner corresponding to a plurality of second panchromatic pixel values in the same filter set in the first panchromatic image, and generating a second panchromatic image based on each third panchromatic pixel value; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode.

The third resolution mode is a mode used in a scene with a lower resolution requirement than the second resolution mode, and is a three-level pixel merging and reading mode with low resolution, low power consumption, high signal-to-noise ratio and high frame rate. The resolution and power consumption corresponding to the third resolution mode are smaller than those corresponding to the second resolution mode. The signal-to-noise ratio and the frame rate corresponding to the third resolution mode are greater than those corresponding to the second resolution mode.

The third resolution mode may specifically be a preview mode in image capturing, a preview mode in video capturing, or a night view mode in image capturing and video capturing in a night view, which has low resolution requirements, but is not limited thereto. Preview modes of video capture such as 720p video preview, application video preview, etc.

The electronic device determines each filter set from the filter array, acquires a plurality of second panchromatic pixel values in the first panchromatic image obtained by each filter set, and combines the plurality of second panchromatic pixel values to read out a third panchromatic pixel value. The electronics read pixel values from each of the third panchromatic pixel values in accordance with a preset pixel reading scheme to produce a second panchromatic image.

Step 2204, merging the second color pixel values of the same color corresponding to the same filter set in the first color image, reading out the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color, and generating a second color image of two colors and a second color image of a single color based on the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color; the bi-color second color image includes third color pixel values of the first color and third color pixel values of the third color, and the mono-color second color image includes third color pixel values of the second color.

The third color pixel value of the first color is a pixel value read out by a pixel corresponding to the first color filter, the third color pixel value of the second color is a pixel value read out by a pixel corresponding to the second color filter, and the third color pixel value of the third color is a pixel value read out by a pixel corresponding to the third color filter.

The electronic device determines each filter group from the filter array, acquires a plurality of second color pixel values of the same color in the first color image obtained by each filter group, the second color pixel values of the same color include a second color pixel value of the first color, a second color pixel value of the second color and a second color pixel value of the third color, combines the second color pixel values of the plurality of first colors to read out a third panchromatic pixel value of the first color, combines the second color pixel values of the plurality of second colors to read out a third panchromatic pixel value of the second color, and combines the second color pixel values of the plurality of third colors to read out a third panchromatic pixel value of the third color.

The bi-colored second color image includes third color pixel values of the first color and third color pixel values of the third color. The monochromatic second color image includes third color pixel values of the second color. Wherein the third color pixel values of the first color are arranged on diagonal lines of the bi-color second color image, and the third color pixel values of the third color are arranged on anti-diagonal lines of the bi-color second color image.

Taking the first color image and the first panchromatic image of fig. 17 as an example, in the third resolution mode, the electronics merge a plurality of second panchromatic pixel values in the first panchromatic image 1704 corresponding to the same filter set to read out third panchromatic pixel values, and generate a second panchromatic image 2302 in fig. 23 based on the respective third panchromatic pixel values; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode; a third color pixel value of the first color, a third color pixel value of the second color, and a third color pixel value of the third color are read out by combining a plurality of second color pixel values of the same color in the same filter group in the first color image 1702, and a second color image 2304 of two colors and a second color image 2306 of a single color in fig. 23 are generated based on the third color pixel value of the first color, the third color pixel value of the second color, and the third color pixel value of the third color. In fig. 23, w denotes a third panchromatic pixel value, and a, b, and c denote third color pixel values of three different colors.

Step 2206, a third target image is generated based on the second panchromatic image, the bi-color second color image and the single-color second color image.

When the second panchromatic image, the two-color second color image and the single-color second color image need to be packed and transmitted, the electronic device may generate a third target image based on the second panchromatic image, the two-color second color image and the single-color second color image, and then transmit the third target image.

Specifically, the electronic device arranges each line of third panchromatic pixel values in the second panchromatic image, each line of third color pixel values in the bi-color second color image and each line of third color pixel values in the single-color second color image to generate a second target image; or arranging each column of third panchromatic pixel values in the second panchromatic image, each column of third panchromatic pixel values in the two-color second color image and each column of third color pixel values in the single-color second color image to generate a second target image.

Taking the second panchromatic image 2302, the bi-color second color image 2304 and the mono-color second color image 2306 of fig. 23 as an example, fig. 24 is a third target image generated by arranging the third panchromatic pixel values of each line in the second panchromatic image, the third color pixel values of each line in the bi-color second color image and the third color pixel values of each line in the mono-color second color image in one embodiment; fig. 25 is a third target image generated by arranging third panchromatic pixel values in each column of the second panchromatic image, third color pixel values in each column of the bi-color second color image, and third color pixel values in each column of the mono-color second color image in another embodiment.

It should be noted that the third panchromatic pixel value in the second panchromatic image, the third color pixel value in the two-color second color image, and the third color pixel value in the single-color second color image in the same coordinate are not limited in order when they are arranged.

In other embodiments, the electronic device may also generate the third target image in other manners, which is not limited herein.

In the present embodiment, in the third resolution mode, the third panchromatic pixel values are read out by combining a plurality of second panchromatic pixel values corresponding to the same filter set in the first panchromatic image, the third color pixel values of the first color, the third color pixel values of the second color and the third color pixel values of the third color are read out by combining a plurality of second panchromatic pixel values corresponding to the same color in the same filter set in the first color image, and the different color pixels can be arranged in a mixed manner, so that the third color pixels, such as RGB pixels, in the generated third target image are distributed more uniformly and the image quality is higher. And the resolution and the image size of the obtained third target image are further reduced, the panchromatic pixel has higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effects of lower power consumption and better signal-to-noise ratio of three-level pixel combination output are achieved. In the third resolution mode, the third target image includes full-color pixels, so that the overall resolution can be improved. Meanwhile, in the third resolution mode, the pixels of the same color do not need to be merged in a spanning period, interpolation is not needed, and the overall resolution is improved. In the full-size case, the color pixels of the respective colors, such as the pixel values of the first color and the pixel values of the third color, are more dispersedly equalized in the diagonal line or the anti-diagonal line. Wherein, the full arrangement means that each coordinate has the pixel, and interpolation estimation is not needed.

In another embodiment, the method further comprises: in the third resolution mode, merging and reading out sixth panchromatic pixel values of panchromatic pixels corresponding to the panchromatic sub-filters of the panchromatic filters in each filter set, and generating a fourth panchromatic image based on the sixth panchromatic pixel values; combining and reading a sixth color pixel value of the first color, a sixth color pixel value of the second color and a sixth color pixel value of the third color from color pixels corresponding to color sub-filters of a plurality of color filters of the same color in each filter set, and generating a fourth color image of two colors and a fourth color image of a single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color and the sixth color pixel value of the third color; the dual color fourth color image includes sixth color pixel values of the first color and sixth color pixel values of the third color, and the single color fourth color image includes sixth color pixel values of the second color; a sixth target image is generated based on the fourth full-color image, the bi-color fourth color image, and the single-color fourth color image.

When the fourth full-color image, the dual-color fourth color image, and the single-color fourth color image need to be packed and transmitted, the electronic device may generate a sixth target image based on the fourth full-color image, the dual-color fourth color image, and the single-color fourth color image, and then transmit the sixth target image.

Wherein generating a sixth target image based on the fourth panchromatic image, the bi-color fourth color image and the single-color fourth color image comprises: arranging sixth panchromatic pixel values of each line in the fourth panchromatic image, sixth color pixel values of each line in the fourth color image of the double color and sixth color pixel values of each line in the fourth color image of the single color to generate a sixth target image; or arranging the sixth panchromatic pixel value of each column in the fourth panchromatic image, the sixth color pixel value of each column in the fourth color image of the double color and the sixth color pixel value of each column in the fourth color image of the single color to generate a sixth target image.

In other embodiments, the electronic device may also generate the sixth target image in other manners, which is not limited herein.

In the embodiment, in the third resolution mode, the panchromatic pixels corresponding to the panchromatic sub-filters of the panchromatic filters in each filter set are combined to read out the fifth panchromatic pixel value, so that the fourth panchromatic image can be generated more quickly; and combining the color pixels corresponding to the color sub-filters of the plurality of color filters of the same color in each filter set to read out a fifth color pixel value of the first color, a fifth color pixel value of the second color and a fifth color pixel value of the third color, so that a fourth color image of two colors and a fourth color image of a single color can be generated more quickly. Moreover, the above embodiment can mix and arrange different color pixels, so that the sixth color pixel values, such as RGB pixels, in the generated sixth target image are distributed more uniformly and the image quality is higher. In addition, the resolution and the image size of the obtained sixth target image are further reduced, the panchromatic pixel 241 has a higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effects of lower power consumption and better signal-to-noise ratio of three-level pixel combination output are achieved.

In one embodiment, as shown in fig. 26, the method further includes:

step 2602, in the fourth resolution mode, reading out a fourth panchromatic pixel value by combining each third panchromatic pixel value in the second panchromatic image; the resolution corresponding to the fourth resolution mode is smaller than the resolution corresponding to the third resolution mode.

The fourth resolution mode is a mode used in a scene with a lower resolution requirement than the third resolution mode, and is a four-level pixel merging and reading mode with low resolution, low power consumption, high signal-to-noise ratio and high frame rate. The resolution and power consumption corresponding to the fourth resolution mode are smaller than those corresponding to the third resolution mode. The signal-to-noise ratio and the frame rate corresponding to the fourth resolution mode are greater than those corresponding to the third resolution mode.

The fourth resolution mode may specifically be a preview mode during image capturing, a preview mode during video capturing, or a scene with low resolution requirements, such as a night scene mode during image capturing and video capturing, but is not limited thereto. Preview modes of video capture such as 480p video preview, application video preview, etc.

The electronics determine each minimal repeating unit from the filter array, obtain a plurality of third panchromatic pixel values in the resulting second panchromatic image for each minimal repeating unit, and read out the fourth panchromatic pixel values in combination with the plurality of third panchromatic pixel values.

Step 2604 combines a plurality of third color pixel values of the first color in the bi-color second color image to read a fourth color pixel value of the first color, combines a plurality of third color pixel values of the third color in the bi-color second color image to read a fourth color pixel value of the third color, and combines a plurality of third color pixel values of the second color in the mono-color second color image to read a fourth color pixel value of the second color.

The electronic device determines each minimal repeating unit from the filter array, acquires a plurality of third color pixel values of the same color in the second color image obtained by each minimal repeating unit, wherein the third color pixel values of the same color comprise a third color pixel value of the first color, a third color pixel value of the second color and a third color pixel value of the third color, combines the third color pixel values of the plurality of first colors to read out a fourth panchromatic pixel value of the first color, combines the third color pixel values of the plurality of second colors to read out a fourth panchromatic pixel value of the second color, and combines the third color pixel values of the plurality of third colors to read out a fourth panchromatic pixel value of the third color.

Step 2606 generates a fourth target image based on the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color.

Specifically, the electronic device alternates the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color corresponding to the same minimal repeating unit to generate a fourth target image. It should be noted that the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color and the fourth color pixel value of the third color corresponding to the same minimal repeating unit are not limited in order in arrangement.

Taking the second panchromatic image 2302, the bi-color second color image 2304 and the mono-color second color image 2306 of fig. 23 as an example, fig. 27 is a schematic diagram of a fourth object image in one embodiment.

In the present embodiment, in the fourth resolution mode, the fourth panchromatic pixel value is read out by combining the respective third panchromatic pixel values in the second panchromatic image, the fourth color pixel value of the first color is read out by combining the third color pixel values of the plurality of first colors in the two-color second color image, the fourth color pixel value of the third color is read out by combining the third color pixel values of the plurality of third colors in the two-color second color image, and the fourth color pixel value of the second color is read out by combining the third color pixel values of the plurality of second colors in the single-color second color image, so that the resolution and the image size of the obtained fourth target image are further reduced, and the panchromatic pixel has a higher signal-to-noise ratio and the frame rate of the image is high, thereby achieving the image processing effects of lower power consumption and better signal-to-noise ratio of the four-level pixel combined output. In addition, in the fourth resolution mode, the fourth target image can be matched with the high-pixel image sensor, and both high resolution under high pixels and high signal-to-noise ratio under low pixels are considered. Meanwhile, in the fourth resolution mode, the pixels of the same color do not need to be merged across the period, interpolation is not needed, and the overall resolution is improved.

In another embodiment, the method further comprises: under a fourth resolution mode, merging and reading out a seventh panchromatic pixel value of panchromatic pixels corresponding to all the panchromatic sub-filters of the plurality of panchromatic filters in the minimum repetition unit, and merging and reading out a seventh color pixel value of color pixels corresponding to all the color sub-filters of the plurality of color filters of the same color in the minimum repetition unit; a seventh target image is generated based on the respective seventh panchromatic pixel values and the respective seventh color pixel values.

It should be noted that the seventh panchromatic pixel value and each seventh color pixel value corresponding to the same minimal repeating unit are not limited in order of arrangement. The color filter comprises a first color filter, a second color filter and a third color filter. The electronic device combines color pixels corresponding to sub-filters of the plurality of first color filters in the minimum repetition unit to read a seventh color pixel value of the first color, combines color pixels corresponding to sub-filters of the plurality of second color filters in the minimum repetition unit to read a seventh color pixel value of the second color, and combines color pixels corresponding to sub-filters of the plurality of third color filters in the minimum repetition unit to read a seventh color pixel value of the third color.

In this embodiment, in the fourth resolution mode, the panchromatic pixels corresponding to the panchromatic sub-filters of the plurality of panchromatic filters in the minimum repetition are combined to read out the seventh panchromatic pixel value, the color pixels corresponding to the color sub-filters of the plurality of color filters of the same color in the minimum repetition unit are combined to read out the seventh color pixel value, and the resolution and the image size of the seventh target image obtained based on the seventh panchromatic pixel value and the seventh color pixel value are further reduced. In addition, in the fourth resolution mode, the fourth target image can be matched with the high-pixel image sensor, and both high resolution under high pixels and high signal-to-noise ratio under low pixels are considered. Meanwhile, in the fourth resolution mode, the pixels of the same color do not need to be merged across the period, interpolation is not needed, and the overall resolution is improved.

In one embodiment, another image generation method is provided and applied to an image sensor, the image sensor including a filter array and a pixel array, the filter array including a minimum repeating unit, the minimum repeating unit including a plurality of filter sets, each filter set including a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the full-color filter is greater than that transmitted by the color filter; each pixel in the pixel array is arranged corresponding to the optical filter of the optical filter array, and the pixel array is configured to receive the light rays passing through the optical filter array to generate an electric signal; the image generation method comprises the following steps: in the full-resolution mode, reading out full-resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic filter, and reading out full-resolution color pixel values from color pixels corresponding to each color filter; a full-resolution target image is generated based on the respective full-resolution panchromatic pixel values and the respective full-resolution color pixel values.

The principle of generating the full-resolution target image in this embodiment is similar to that of generating the full-resolution target image in the embodiment of fig. 12, and is not described herein again.

It should be noted that if N is 1, that is, the optical filter does not include a sub-optical filter, the optical filter array further has a first resolution mode, a second resolution mode and a third resolution mode, and the principle of the first resolution mode corresponding to N being 1 is similar to the principle of the second resolution mode corresponding to N being greater than or equal to 2, the principle of the second resolution mode corresponding to N being 1 is similar to the principle of the third resolution mode corresponding to N being greater than or equal to 2, and the principle of the third resolution mode corresponding to N being 1 is similar to the principle of the fourth resolution mode corresponding to N being greater than or equal to 2, which is not described herein again.

It should be understood that, although the respective steps in the flowcharts of fig. 13, 14, 16, 22 and 26 are sequentially shown as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 13, 14, 16, 22, and 26 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be alternated or performed with other steps or at least some of the sub-steps or stages of other steps.

Fig. 28 is a block diagram showing the configuration of an image generating apparatus according to an embodiment. As shown in fig. 28, there is provided an image generating apparatus applied to an image sensor, the image sensor including a filter array and a pixel array, the filter array including a minimum repeating unit, the minimum repeating unit including a plurality of filter sets, each filter set including a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the full-color filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and the N columns of color sub-filters are the same as those of the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal;

the image generation apparatus includes: a readout module 2802 and an image generation module 2804, wherein:

a reading module 2802 configured to, in a full resolution mode, read out full-resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters, and read out full-resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters.

An image generation module 2804 generates a full-resolution target image based on the respective full-resolution panchromatic pixel values and the respective full-resolution color pixel values.

The image generating apparatus described above reads out, in the full-resolution mode, a full-resolution panchromatic pixel value from a panchromatic pixel corresponding to each panchromatic sub-filter in the panchromatic filter, and a full-resolution color pixel value from a color pixel corresponding to each color sub-filter in the color filter; and the light inlet quantity of the panchromatic filter is larger than that of the color filter, so that the panchromatic channel information can be fused into the image, and the whole light inlet quantity is improved, and therefore, the full-resolution target image with more information and clearer detail analysis can be generated based on each full-resolution panchromatic pixel value and each full-resolution color pixel value.

In one embodiment, the readout module 2802 is further configured to, in the first resolution mode, read out the first panchromatic pixel values from the panchromatic pixels corresponding to the respective panchromatic sub-filters in each panchromatic filter in a merged manner, and read out the first color pixel values from the color pixels corresponding to the respective color sub-filters in each color filter in a merged manner; the resolution corresponding to the first resolution mode is smaller than the resolution corresponding to the full resolution mode; the image generation module 2804 is also configured to generate a first target image based on the respective first panchromatic pixel values and the respective first color pixel values.

In one embodiment, the readout module 2802 is further configured to read out the second panchromatic pixel values in the first target image in combination with the plurality of first panchromatic pixel values corresponding to each of the sub-units in the second resolution mode, the sub-units including a first sub-unit and a second sub-unit; the image generation module 2804 is further configured to generate a first panchromatic image based on the respective second panchromatic pixel values; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode; the readout module 2802 is further configured to read out second color pixel values from the first target image by combining the first color pixel values corresponding to a plurality of same colors in each sub-cell, and the image generation module 2804 is further configured to generate a first color image based on the respective second color pixel values; the image generation module 2804 is further configured to generate a second target image based on the first panchromatic image and the first color image.

In one embodiment, the image generation module 2804 is further configured to arrange the second panchromatic pixel values of each row in the first panchromatic image and the second color pixel values of each row in the first color image to generate a second target image; or arranging the second panchromatic pixel value of each column in the first panchromatic image and the second color pixel value of each column in the first color image to generate a second target image.

In one embodiment, the readout module 2802 is further configured to read out third panchromatic pixel values from the first panchromatic image corresponding to a plurality of second panchromatic pixel values in the same filter set in a third resolution mode, and the image generation module 2804 is further configured to generate a second panchromatic image based on the respective third panchromatic pixel values; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode; the reading module 2802 is further configured to read out a third color pixel value of the first color, a third color pixel value of the second color, and a third color pixel value of the third color by combining a plurality of second color pixel values of the same color in the same filter set in the first color image, and the image generating module 2804 is further configured to generate a second color image of a dual color and a second color image of a single color based on the third color pixel value of the first color, the third color pixel value of the second color, and the third color pixel value of the third color; the bi-color second color image comprises third color pixel values of the first color and third color pixel values of the third color, and the mono-color second color image comprises third color pixel values of the second color; the image generation module 2804 is further configured to generate a third target image based on the second panchromatic image, the two-color second color image, and the single-color second color image.

In one embodiment, the image generating module 2804 is further configured to arrange each row of third panchromatic pixel values in the second panchromatic image, each row of third color pixel values in the bi-color second color image, and each row of third color pixel values in the single-color second color image to generate a second target image; or arranging each column of third panchromatic pixel values in the second panchromatic image, each column of third panchromatic pixel values in the two-color second color image and each column of third color pixel values in the single-color second color image to generate a second target image.

In one embodiment, the readout module 2802 is further configured to read out the fourth panchromatic pixel values in the second panchromatic image in combination with the respective third panchromatic pixel values in the fourth resolution mode; the resolution corresponding to the fourth resolution mode is smaller than the resolution corresponding to the third resolution mode; the readout module 2802 is further configured to read out a fourth color pixel value of the first color by combining a plurality of third color pixel values of the first color in the two-color second color image, read out a fourth color pixel value of the third color by combining a plurality of third color pixel values of the third color in the two-color second color image, and read out a fourth color pixel value of the second color by combining a plurality of third color pixel values of the second color in the one-color second color image; the image generation module 2804 is further configured to generate a fourth target image based on the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color.

In one embodiment, the readout module 2802 is further configured to read out a fifth panchromatic pixel value from a combination of panchromatic pixels corresponding to respective panchromatic sub-filters of the plurality of panchromatic filters in each of the first sub-unit and the second sub-unit in the second resolution mode, and the image generation module 2804 is further configured to generate a third panchromatic image based on the respective fifth panchromatic pixel values; the reading module 2802 is further configured to read out a fifth color pixel value by combining color pixels corresponding to respective color sub-filters of a plurality of color filters of the same color in each of the first sub-unit and the second sub-unit, and the image generating module 2804 is further configured to generate a third color image based on the respective fifth color pixel values; the image generation module 2804 is further configured to generate a fifth target image based on the third panchromatic image and the third color image.

In one embodiment, the image generation module 2804 is further configured to arrange the fifth panchromatic pixel values of each row in the third panchromatic image and the fifth color pixel values of each row in the third color image to generate a fifth target image; or arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third panchromatic image to generate a fifth target image.

In one embodiment, the readout module 2802 is further configured to read out a sixth panchromatic pixel value by combining panchromatic pixels corresponding to respective panchromatic sub-filters of the plurality of panchromatic filters in each filter set in the third resolution mode, and the image generation module 2804 is further configured to generate a fourth panchromatic image based on the respective sixth panchromatic pixel values; the reading module 2802 is further configured to read out a sixth color pixel value of the first color, a sixth color pixel value of the second color, and a sixth color pixel value of the third color by combining the color pixels corresponding to the color sub-filters of the plurality of color filters of the same color in each filter group, and the image generating module 2804 is further configured to generate a fourth color image of a dual color and a fourth color image of a single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color, and the sixth color pixel value of the third color; the dual color fourth color image includes sixth color pixel values of the first color and sixth color pixel values of the third color, and the single color fourth color image includes sixth color pixel values of the second color; the image generation module 2804 is further configured to generate a sixth target image based on the fourth panchromatic image, the bi-color fourth color image, and the single-color fourth color image.

In one embodiment, the image generation module 2804 is further configured to arrange the sixth panchromatic pixel values of each line in the fourth panchromatic image, the sixth color pixel values of each line in the bi-color fourth color image, and the sixth color pixel values of each line in the single-color fourth color image to generate a sixth target image; or arranging the sixth panchromatic pixel value of each column in the fourth panchromatic image, the sixth color pixel value of each column in the fourth color image of the double color and the sixth color pixel value of each column in the fourth color image of the single color to generate a sixth target image.

In an embodiment, the reading module 2802 is further configured to, in the fourth resolution mode, combine and read a seventh panchromatic pixel value from a panchromatic pixel corresponding to each of the plurality of panchromatic filters in the minimal repeating unit, and combine and read a seventh color pixel value from a color pixel corresponding to each of the plurality of color filters of the same color in the minimal repeating unit; the image generation module 2804 is further configured to generate a seventh target image based on each seventh panchromatic pixel value and each seventh color pixel value.

The division of the modules in the image generating apparatus is merely for illustration, and in other embodiments, the image generating apparatus may be divided into different modules as needed to complete all or part of the functions of the image generating apparatus.

For specific limitations of the image generation apparatus, reference may be made to the above limitations of the image generation method, which are not described herein again. The respective modules in the image generating apparatus described above may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

Fig. 29 is a schematic diagram of the internal structure of the electronic device in one embodiment. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, and a wearable device. The electronic device includes a processor and a memory connected by a system bus. The processor may include one or more processing units, among others. The processor may be a CPU (Central Processing Unit), a DSP (Digital Signal processor), or the like. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor for implementing an image generation method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium.

The implementation of each module in the image generation apparatus provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. Program modules constituted by such computer programs may be stored on the memory of the electronic device. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the image generation method.

Embodiments of the present application also provide a computer program product containing instructions which, when run on a computer, cause the computer to perform an image generation method.

Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. The nonvolatile Memory may include a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a flash Memory. Volatile Memory can include RAM (Random Access Memory), which acts as external cache Memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDR SDRAM (Double Data Rate Synchronous Random Access Memory), ESDRAM (Enhanced Synchronous Dynamic Random Access Memory), SLDRAM (Synchronous Link Dynamic Random Access Memory), RDRAM (Random Dynamic Random Access Memory), and DRmb DRAM (Dynamic Random Access Memory).

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. An image sensor comprising a filter array and a pixel array, the filter array comprising a minimal repeating unit comprising a plurality of filter sets, each filter set comprising a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal.

2. The image sensor of claim 1, wherein each row and each column comprises a color filter of each color.

3. The image sensor of claim 1, wherein the minimal repeating unit comprises 2 first filter sets and 2 second filter sets, the 2 first filter sets being arranged on a diagonal of the minimal repeating unit, the 2 second filter sets being arranged on an opposite diagonal of the minimal repeating unit.

4. The image sensor of claim 3, wherein the 2 first filter sets are identical and the 2 second filter sets are identical.

5. The image sensor of claim 3, wherein each filter set includes only panchromatic filters and 2 color filters.

6. The image sensor of claim 5, wherein the color filters in the first filter set comprise a first color filter and a second color filter, and wherein the color filters in the second filter set comprise a second color filter and a third color filter.

7. The image sensor of claim 3, wherein each filter set comprises 2 first subunits and 2 second subunits; the 2 first subunits are arranged in a first row direction of the filter set, and the 2 second subunits are arranged in a second row direction of the filter set, wherein the first row direction and the second row direction are arranged adjacently; or, the 2 first subunits are arranged in a first column direction of the filter set, and the 2 second subunits are arranged in a second column direction of the filter set, where the first column direction and the second column direction are arranged adjacently.

8. The image sensor of claim 7, wherein the color filter comprises a first color filter, a second color filter, and a third color filter; one of the 2 first subunits in the first filter set comprises a first color filter, and the other first subunit comprises a second color filter; one of the 2 second subunits in the first filter set comprises a first color filter, and the other second subunit comprises a second color filter.

9. The image sensor of claim 7, wherein the color filters comprise a first color filter, a second color filter, and a third color filter; one of the 2 first subunits in the second filter set comprises a second color filter, and the other first subunit comprises a third color filter; one of the 2 second subunits in the second filter set includes a second color filter, and the other second subunit includes a third color filter.

10. The image sensor of claim 7, wherein the first sub-unit and the second sub-unit of the same color filter in the same filter set are arranged on a diagonal line or an opposite diagonal line of the same filter set.

11. The image sensor of claim 1, wherein N is 1, the minimal repeating unit comprises 64 filters in 8 rows and 8 columns, and the arrangement is as follows:

or

Or

Or

Where w denotes a full color filter, and a, b, and c each denote a color filter.

12. The image sensor of claim 1, wherein N is 2, the minimal repeating unit comprises 16 rows and 16 columns of 256 sub-filters, and the arrangement is:

or

Or

Or

13. A camera module, characterized in that the camera module comprises a lens and an image sensor according to any one of claims 1-12; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

14. An electronic device, comprising:

the camera module of claim 13; and

the casing, the module setting of making a video recording is in on the casing.

15. An image generation method is applied to an image sensor and is characterized in that the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repetition unit, the minimum repetition unit comprises a plurality of filter sets, and each filter set comprises a first sub-unit and a second sub-unit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal;

the method comprises the following steps:

16. The method of claim 15, further comprising:

under a first resolution mode, merging and reading out first panchromatic pixel values of panchromatic pixels corresponding to all panchromatic sub-filters in each panchromatic filter, and merging and reading out first color pixel values of color pixels corresponding to all color sub-filters in each color filter; the resolution corresponding to the first resolution mode is smaller than the resolution corresponding to the full resolution mode;

a first target image is generated based on each of the first panchromatic pixel values and each of the first color pixel values.

17. The method of claim 16, wherein after generating the first target image, further comprising:

in a second resolution mode, merging a plurality of first panchromatic pixel values corresponding to each subunit in the first target image to read out second panchromatic pixel values, and generating a first panchromatic image based on the respective second panchromatic pixel values; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode, and the subunits comprise the first subunits and the second subunits;

merging and reading out second color pixel values of a plurality of same colors in each subunit in the first target image, and generating a first color image based on the respective second color pixel values;

generating a second target image based on the first panchromatic image and the first color image.

18. The method of claim 17, wherein generating a second target image based on the first panchromatic image and the first color image comprises:

arranging each row of second panchromatic pixel values in the first panchromatic image and each row of second color pixel values in the first color image to generate a second target image; or

And arranging each column of second panchromatic pixel values in the first panchromatic image and each column of second color pixel values in the first color image to generate a second target image.

19. The method of claim 17, further comprising:

in a third resolution mode, merging a plurality of second panchromatic pixel values corresponding to the same filter set in the first panchromatic image to read out third panchromatic pixel values, and generating a second panchromatic image based on each third panchromatic pixel value; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode;

combining a plurality of second color pixel values of the same color in the same filter set in the first color image, reading out a third color pixel value of the first color, a third color pixel value of the second color and a third color pixel value of the third color, and generating a second color image of two colors and a second color image of a single color based on the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color; the bi-color second color image comprises third color pixel values of a first color and third color pixel values of a third color, and the mono-color second color image comprises third color pixel values of a second color;

generating a third target image based on the second panchromatic image, the bi-color second color image and the single-color second color image.

20. The method of claim 19, wherein generating a third target image based on the second panchromatic image, a bi-color second color image, and a single-color second color image comprises:

arranging third panchromatic pixel values of each line in the second panchromatic image, third color pixel values of each line in the bi-color second color image and third color pixel values of each line in the single-color second color image to generate a second target image; or

And arranging each column of third panchromatic pixel values in the second panchromatic image, each column of third panchromatic pixel values in the two-color second color image and each column of third color pixel values in the single-color second color image to generate a second target image.

21. The method of claim 19, further comprising:

in a fourth resolution mode, reading out a fourth panchromatic pixel value in combination with each third panchromatic pixel value in the second panchromatic image; the resolution corresponding to the fourth resolution mode is smaller than the resolution corresponding to the third resolution mode;

combining a plurality of third color pixel values of the first color in the two-color second color image to read out a fourth color pixel value of the first color, combining a plurality of third color pixel values of the third color in the two-color second color image to read out a fourth color pixel value of the third color, and combining a plurality of third color pixel values of the second color in the single-color second color image to read out a fourth color pixel value of the second color;

generating a fourth target image based on the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color.

22. The method of claim 15, further comprising:

under a second resolution mode, merging and reading out fifth panchromatic pixel values of panchromatic pixels corresponding to the panchromatic sub-filters of the plurality of panchromatic filters in each first sub-unit or the second sub-unit, and generating a third panchromatic image based on the fifth panchromatic pixel values;

combining and reading out fifth color pixel values of color pixels corresponding to each color sub-filter of a plurality of color filters with the same color in each first sub-unit or second sub-unit, and generating a third color image based on each fifth color pixel value;

generating a fifth target image based on the third panchromatic image and the third color image.

23. The method of claim 22, wherein generating a fifth target image based on the third panchromatic image and the third color image comprises:

arranging each row of fifth panchromatic pixel values in the third panchromatic image and each row of fifth color pixel values in the third panchromatic image to generate a fifth target image; or

And arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third panchromatic image to generate a fifth target image.

24. The method of claim 15, further comprising:

in a third resolution mode, merging and reading out sixth panchromatic pixel values of panchromatic pixels corresponding to the panchromatic sub-filters of the panchromatic filters in each filter set, and generating a fourth panchromatic image based on the sixth panchromatic pixel values;

combining and reading a sixth color pixel value of the first color, a sixth color pixel value of the second color and a sixth color pixel value of the third color from color pixels corresponding to color sub-filters of a plurality of color filters of the same color in each filter set, and generating a fourth color image of two colors and a fourth color image of a single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color and the sixth color pixel value of the third color; the dual-color fourth color image includes sixth color pixel values of the first color and sixth color pixel values of the third color, and the single-color fourth color image includes sixth color pixel values of the second color;

generating a sixth target image based on the fourth panchromatic image, the bi-color fourth color image and the single-color fourth color image.

25. The method of claim 24, wherein generating a sixth target image based on the fourth panchromatic image, the bi-color fourth color image and the single-color fourth color image comprises:

arranging sixth panchromatic pixel values of each line in the fourth panchromatic image, sixth color pixel values of each line in the bi-color fourth color image and sixth color pixel values of each line in the single-color fourth color image to generate a sixth target image; or

And arranging each column of sixth panchromatic pixel values in the fourth panchromatic image, each column of sixth color pixel values in the fourth color image of the double color and each column of sixth color pixel values in the fourth color image of the single color to generate a sixth target image.

26. The method of claim 15, further comprising:

under a fourth resolution mode, merging and reading out a seventh panchromatic pixel value of a panchromatic pixel corresponding to each panchromatic sub-filter of the plurality of panchromatic filters in the minimum repetition unit, and merging and reading out a seventh color pixel value of a color pixel corresponding to each color sub-filter of the plurality of color filters of the same color in the minimum repetition unit;

generating a seventh target image based on each of the seventh panchromatic pixel values and each of the seventh color pixel values.

27. An image generation device is applied to an image sensor and is characterized in that the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repetition unit, the minimum repetition unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit both comprise panchromatic optical filters and color optical filters, the color optical filters in the first subunit are arranged on diagonal lines in the first subunit, and the color optical filters in the second subunit are arranged on anti-diagonal lines in the second subunit; the light entering amount transmitted by the panchromatic filter is greater than that transmitted by the color filter; each panchromatic filter comprises N rows and N columns of panchromatic sub-filters, each color filter comprises N rows and N columns of color sub-filters, the colors of the N rows and N columns of color sub-filters are the same as the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate an electric signal;

the device comprises:

28. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, causes the processor to perform the steps of the image generation method of any of claims 15 to 26.

29. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 16 to 26.

30. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method according to any one of claims 15 to 26 when executed by a processor.