CN109064414B

CN109064414B - Image denoising method and device

Info

Publication number: CN109064414B
Application number: CN201810737255.4A
Authority: CN
Inventors: 李俊
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2018-07-06
Filing date: 2018-07-06
Publication date: 2020-11-10
Anticipated expiration: 2038-07-06
Also published as: CN109064414A

Abstract

The invention provides an image denoising method and device, and relates to the technical field of image processing. The method comprises the following steps: respectively carrying out down-sampling on the original image according to a first preset mode and a second preset mode to obtain a first down-sampled image and a second down-sampled image; denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image; and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image. According to the embodiment of the invention, the original image is downsampled in two different preset modes, then is denoised respectively to obtain a first denoised image and a second denoised image, and then is upsampled based on the two denoised images to obtain a denoised image corresponding to the original image, so that the denoising effect of the original image is improved.

Description

Image denoising method and device

Technical Field

The embodiment of the invention relates to the technical field of image processing, in particular to an image denoising method and device.

Background

At present, with the continuous improvement of the demand of people on digital photographing effect, the denoising of the original image data is more and more important.

In the prior art, most of the traditional image denoising algorithms perform single-method down-sampling and denoising on an original image.

The inventor finds in the research process that the prior art proposal has the following disadvantages: for original image data, a single method is adopted for down sampling and denoising, and the denoising effect is poor.

Disclosure of Invention

The embodiment of the invention provides an image denoising method and device, and aims to solve the problem of poor denoising effect in the prior art.

In a first aspect, an embodiment of the present invention provides an image denoising method, including:

respectively carrying out down-sampling on the original image according to a first preset mode and a second preset mode to obtain a first down-sampled image and a second down-sampled image;

denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image;

and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image.

In a second aspect, an embodiment of the present invention further provides an image denoising device, including:

the down-sampling module is used for respectively performing down-sampling on the original image according to a first preset mode and a second preset mode to obtain a first down-sampled image and a second down-sampled image;

a de-noising image obtaining module, configured to de-noise the first down-sampling image and the second down-sampling image respectively to obtain a first de-noising image and a second de-noising image;

and the de-noising image generating module is used for performing up-sampling on the basis of the first de-noising image and the second de-noising image to obtain a de-noising image corresponding to the original image.

In a third aspect of the first sub-downsampling module and the second sub-downsampling module, the embodiment of the present invention further provides a mobile terminal, where the mobile terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the image denoising method according to the present invention are implemented.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the image denoising method according to the present invention are implemented.

In the embodiment of the invention, the original image is respectively downsampled according to a first preset mode and a second preset mode to obtain a first downsampled image and a second downsampled image; denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image; and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image. In the embodiment of the invention, the original image is downsampled in two different preset modes, then is denoised respectively to obtain a first denoised image and a second denoised image, and then is upsampled based on the two denoised images to obtain a denoised image corresponding to the original image, so that the denoising effect of the original image is improved.

Drawings

Fig. 1 is a flowchart illustrating an image denoising method according to a first embodiment of the present invention;

FIG. 2 is a flowchart illustrating an image denoising method according to a second embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a distribution of pixels of a raw image in Bayer format according to a second embodiment of the present invention;

fig. 4 is a schematic diagram illustrating a first downsampled image provided in the second embodiment of the present invention for the bayer pattern raw image shown in fig. 3;

fig. 5 is a schematic diagram illustrating each set of first bayer pattern data, which are independent of each other, in a bayer pattern raw image provided in the second embodiment of the present invention;

fig. 6 illustrates a second downsampled image schematic diagram for the bayer pattern raw image illustrated in fig. 3 according to a second embodiment of the present invention;

FIG. 7 is a flowchart illustrating the steps provided in the second embodiment of the present invention to obtain a second downsampled image;

FIG. 8 is a diagram illustrating a second denoised image provided in the second embodiment of the invention;

FIG. 9 is a schematic diagram of a first denoised image provided in the second embodiment of the invention;

fig. 10 shows a schematic diagram of intermediate up-sampled data provided in the second embodiment of the present invention;

fig. 11 is a schematic diagram illustrating data after normalization processing is performed on a fifth pixel in the intermediate upsampled data according to the second embodiment of the present invention;

fig. 12 shows a schematic diagram of fourth bayer pattern data obtained with respect to fig. 9 and 11, provided in the second embodiment of the present invention;

fig. 13 shows a schematic diagram of a denoised image corresponding to the original image shown in fig. 3 obtained by rounding the color values of the pixels in fig. 12 according to the second embodiment of the present invention;

FIG. 14 is a block diagram illustrating an image denoising apparatus according to a third embodiment of the present invention;

FIG. 15 is a block diagram illustrating another image denoising apparatus according to a third embodiment of the present invention;

fig. 16 is a schematic hardware structure diagram of a mobile terminal in an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example one

Referring to fig. 1, a flowchart of an image denoising method provided in a first embodiment of the present invention is shown, which specifically includes the following steps:

step 101, according to a first preset mode and a second preset mode, down-sampling an original image respectively to obtain a first down-sampled image and a second down-sampled image.

In this embodiment of the present invention, the original image may be image data that has just been acquired by the digital photographing apparatus, which is not particularly limited in this embodiment of the present invention.

In the embodiment of the present invention, a first preset manner of downsampling an original image may be: in the present invention, the method may further include a nearest neighbor sampling method, a quadratic interpolation method, a bicubic convolution method, and the like, and in the sampling process, an appropriate sampling manner may be set according to the image attribute information and the like of the original image.

In the embodiment of the present invention, the second preset manner of downsampling the original image may be: in the present invention, the method may further include a nearest neighbor sampling method, a quadratic interpolation method, a bicubic convolution method, and the like, and in the sampling process, an appropriate sampling manner may be set according to the image attribute information and the like of the original image.

It should be noted that, in the process of down-sampling the same original image, different first preset modes and second preset modes are adopted, so that different factors of the original image can be concerned, and then a subsequently obtained de-noised image corresponding to the original image also concerns different factors of the original image.

102, denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image.

In the embodiment of the invention, the first downsampled image and the second downsampled image are respectively denoised to obtain a first denoised image and a second denoised image. In the embodiment of the present invention, the denoising methods for the first downsampled image and the second downsampled image may be the same or different. This is not particularly limited in the embodiments of the present invention.

For example, for the first downsampled image, denoising may be performed by using at least one of denoising methods such as non-local mean denoising, discrete cosine transform based denoising, mean denoising, and the like. For the second down-sampled image, at least one of the denoising methods such as non-local mean value denoising, discrete cosine transform based denoising, mean value denoising, and the like can also be adopted for denoising.

And denoising the first downsampled image to obtain a first denoised image. And denoising the second down-sampling image to obtain a second denoised image.

In the embodiment of the present invention, the first downsampled image of the original image pays attention to the first characteristic of the original image, and retains the first characteristic information of the original image, so that the first denoised image obtained based on the first downsampled image also pays attention to the first characteristic information of the original image; the second downsampled image of the original image pays attention to the second characteristic of the original image, and retains the second characteristic information of the original image, so that the second denoised image obtained based on the second downsampled image also pays attention to the second characteristic information of the original image. Compared with the prior art, only a single factor is considered, and the denoising effect of the original image is improved.

103, performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image.

In the embodiment of the invention, the up-sampling is carried out based on the first denoised image and the second denoised image, so as to obtain the denoised image of the original image.

In specific application, the second denoised image can be inserted by adopting an interpolation algorithm on the basis of the first denoised image, so that the denoised image of the original image is obtained. The denoised image of the original image may comprise the same number of pixels as the original image. In the embodiment of the present invention, this is not particularly limited.

In specific application, the first denoised image can be inserted by adopting an interpolation algorithm on the basis of the second denoised image, so that the denoised image of the original image is obtained. The denoised image of the original image may comprise the same number of pixels as the original image. In the embodiment of the present invention, this is not particularly limited.

The interpolation algorithm may be an edge image-based difference value, a region-based image difference value, and the like, which are not particularly limited in the embodiment of the present invention.

In the embodiment of the present invention, the first denoised image of the original image pays attention to the first characteristic of the original image, and retains the first characteristic information of the original image; the second denoised image of the original image pays attention to the second characteristic of the original image, and the second characteristic information of the original image is reserved, so that the denoised image corresponding to the original image is obtained by performing upsampling based on the first denoised image and the second denoised image, and the first characteristic information and the second characteristic information of the original image are also paid attention to. Compared with the prior art, only a single factor is considered, and the denoising effect of the original image is improved.

In the embodiment of the invention, the original image is respectively downsampled according to a first preset mode and a second preset mode to obtain a first downsampled image and a second downsampled image; denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image; and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image. In the embodiment of the invention, the original image is downsampled in two different preset modes, then is denoised respectively to obtain a first denoised image and a second denoised image, and then is upsampled based on the two denoised images to obtain the denoised image corresponding to the original image.

Example two

Referring to fig. 2, a flowchart of an image denoising method provided in the second embodiment of the present invention is shown, where the original image may include: raw image in bayer format. The method may specifically comprise the steps of:

step 201, in the bayer original image, downsampling each first bayer pattern data into a second pixel to obtain a first downsampled image; the first bayer pattern data includes four first pixels in a 2 × 2 matrix.

In the embodiment of the present invention, referring to fig. 3, fig. 3 shows a bayer pattern raw image pixel distribution diagram. The bayer-format raw image may include: first bayer pattern data. In fig. 3, a first bayer pattern data may be outlined as an ellipse 11. A first bayer pattern data may be data acquired by a photosensitive element on the acquisition terminal of the raw image acquisition. In the embodiment of the present invention, this is not particularly limited.

In the embodiment of the present invention, in downsampling the bayer pattern raw image into the first downsampling, each of the first bayer pattern data may be independent of each other. For example, as shown in fig. 3, after four first pixels framed by the ellipse 11 are down-sampled to one second pixel, the first bayer pattern data having the pixel coordinates (1,3), (1,4), (2,3), and (2,4) in fig. 3 may be selected and then down-sampled to obtain the next second pixel. That is, in the raw image of the bayer format, each pixel may be selected only once. That is, there are no overlapping pixels between the respective first bayer pattern data.

In the bayer-format raw image, one pixel coordinate point may include only one color value, for example, as shown in fig. 3, a pixel coordinate point with a pixel coordinate of (1,1) includes a color value that is a G color value.

In an embodiment of the present invention, a first bayer pattern data may include four first pixels in a 2 x 2 matrix. Specifically, referring to a first bayer pattern data outlined by the ellipse 11 in fig. 3, the first bayer pattern data may include four first pixels in a 2 × 2 matrix, and specifically, the pixel coordinates may be: four first pixels of (1,1), (1,2), (2,1), (2,2), and one first bayer pattern data may include two first pixels corresponding to G color values, one first pixel corresponding to R color values, and one first pixel corresponding to B color values. In a first bayer pattern data, the first pixels corresponding to two G color values are not adjacent, and the first pixels corresponding to R color values and the first pixels corresponding to B color values are not adjacent.

For example, referring to a first bayer pattern data shown by an ellipse 11 in fig. 3, a first pixel corresponding to a G color value at the upper left corner and a first pixel corresponding to a G color value at the lower right corner are not adjacent to each other. The first pixel corresponding to the G color value in the upper left corner may be located in the same row, adjacent column as the first pixel corresponding to the R color value. The first pixel corresponding to the G color value at the lower right corner may be located in the same column as the first pixel corresponding to the R color value, in an adjacent row, and the first pixel corresponding to the B color value and the first pixel corresponding to the G color value at the upper left corner are located in the same column and in the same row as the first pixel corresponding to the G color value at the lower right corner.

In the embodiment of the present invention, in the bayer pattern raw image, each of the first bayer pattern data that are independent of each other may be downsampled into one second pixel to obtain the first downsampled data.

In a specific application, the pixel values of the four first pixels included in each first bayer pattern data may be averaged, or a weighted average value may be calculated to obtain one second pixel, which is not specifically limited in this embodiment of the present invention.

For example, the pixel values of four first pixels included in each first bayer pattern data may be arithmetically averaged to obtain one second pixel, that is, the pixel value of the second pixel may be the arithmetic average of the pixel values of the four first pixels. The pixel values of the four first pixels included in each first bayer pattern data may also be subjected to geometric averaging to obtain a second pixel, that is, the pixel value of the second pixel may be the geometric average of the pixel values of the four first pixels. The pixel values of the four first pixels included in each first bayer pattern data may also be subjected to harmonic averaging to obtain a second pixel, that is, the pixel value of the second pixel may be the harmonic average of the pixel values of the four first pixels. The pixel values of the four first pixels included in each first bayer pattern data may be averaged according to a certain weight to obtain a second pixel, that is, the pixel value of the second pixel may be a weighted average of the pixel values of the four first pixels. In the embodiment of the present invention, this is not particularly limited.

In the embodiment of the present invention, in the bayer pattern raw image, each first bayer pattern data is downsampled to one second pixel, and a first downsampled image is obtained.

In the embodiment of the present invention, the second pixel is an average value or a weighted average value of pixel values of four color channels included in each first bayer pattern data, and the like, so that the two pixels associate the four color channels of each first bayer pattern data in the bayer pattern raw image, and mainly pay attention to luminance information of the bayer pattern raw image. Therefore, the bayer pattern raw image is obtained as a result of the association of the four color channels of the first downsampled image, and mainly represents the luminance information of the bayer pattern raw image.

For example, referring to fig. 4, fig. 4 shows a first downsampled image schematic for the bayer pattern raw image shown in fig. 3. In fig. 4, the second pixel 12 at the pixel coordinate (1,1) may be the down-sampled second pixel of the four first pixels included in the first bayer pattern data 11 in fig. 3. By analogy, in fig. 4, the second pixel at the pixel coordinate (1,2) may be the second pixel after down-sampling the first bayer format data composed of four first pixels in fig. 3, i.e., the pixel coordinate (1,3), the pixel coordinate (1,4), the pixel coordinate (2,3), and the pixel coordinate (2, 4). In fig. 4, the second pixel at the pixel coordinate (2,1) may be the second pixel after down-sampling of the first bayer pattern data composed of four first pixels in fig. 3, i.e., the pixel coordinate (3,1), the pixel coordinate (3,2), the pixel coordinate (4,1), and the pixel coordinate (4, 2). In the embodiment of the present invention, this is not particularly limited.

In fig. 4, the pixel value of the second pixel 12 at the pixel coordinate (1,1) may be a G color value at the pixel coordinate (1,1), an R color value at the pixel coordinate (1,2), a B color value at the pixel coordinate (2,1), an arithmetic average value of G color values at the pixel coordinate (2,2), or the like included in the first bayer pattern data indicated by 11 in the bayer pattern raw image shown in fig. 3. In the embodiment of the present invention, this is not particularly limited.

For example, in fig. 4, the pixel value of the second pixel 12 at the pixel coordinate (1,1) may be (G color value of pixel coordinate (1,1) + R color value of pixel coordinate (1,2) + B color value of pixel coordinate (2,1) + G color value of pixel coordinate (2, 2)/4.

In a specific application, the down-sampling of the bayer pattern raw image is equivalent to replacing four first pixels included in each first bayer pattern data, which are independent of each other, in the bayer pattern raw image by the second pixel, so that the number of pixels included in the first down-sampled image is one fourth of the number of pixels included in the bayer pattern raw image. Through the downsampling, the subsequent operation amount is greatly reduced, and the denoising speed is accelerated.

Meanwhile, in the embodiment of the present invention, in the bayer pattern raw image, each of the first bayer pattern data that are independent of each other is downsampled into one second pixel, so as to obtain a first downsampled image of the bayer pattern raw image. Equivalently, the color values of four pixels, namely four channels are converted into one pixel value, the relevance of the four channels is concerned, and then partial brightness information of the Bayer format original image is obtained. Namely, the first downsampled image of the bayer pattern raw image, pays attention to the correlation of four channels, and partial luminance information of the bayer pattern raw image is retained.

Step 202, in the bayer original image, taking every four first bayer pattern data as a group, and downsampling each group of first bayer pattern data into a second bayer pattern data to obtain a second downsampled image; wherein, each group of first Bayer format data is a matrix of 2 x 2.

In the embodiment of the present invention, every four first bayer pattern data are set as a group, and each group of first bayer pattern data is downsampled into one second bayer pattern data to obtain second downsampled data. The four first bayer pattern data are in the form of a 2 × 2 matrix.

In the embodiment of the present invention, each set of the first bayer pattern data is independent of each other, that is, each two sets of the first bayer pattern data are located at different positions in the bayer pattern raw image, or there is no overlap between each two sets of the first bayer pattern data.

In a specific application, referring to fig. 5, fig. 5 shows a schematic diagram of each set of first bayer pattern data, which are independent of each other, in a bayer pattern raw image. The large oval box 15 in fig. 5 is a set of first bayer pattern data that are independent of each other. The mutually independent set of first bayer pattern data is in the form of a 2 x 2 matrix. The mutually independent set of first bayer pattern data may be composed of first bayer pattern data 1 framed by a small ellipse 11, first bayer pattern data 2 framed by a small ellipse 13, first bayer pattern data 3 framed by a small ellipse 14, and first bayer pattern data 4 framed by an ellipse adjacent to both the small ellipse 14 and the small ellipse 13. In fig. 5, the first bayer pattern data 1 framed by the small ellipse 11, the first bayer pattern data 2 framed by the small ellipse 13, the first bayer pattern data 3 framed by the small ellipse 14, and the first bayer pattern data 4 framed by ellipses adjacent to both the small ellipse 14 and the small ellipse 13 have no overlapping pixels with each other.

In the embodiment of the present invention, each two sets of the first bayer pattern data are independent from each other, or each two sets of the first bayer pattern data are located at different positions in the bayer pattern raw image, or each two sets of the first bayer pattern data do not overlap with each other, which means that each two sets of the first bayer pattern data do not overlap with each other. For example, referring to FIG. 5, the large ellipse 15 in FIG. 5 frames a set of first Bayer format data, and the large ellipse 30 frames a set of first Bayer format data, with no overlapping pixels.

Referring to fig. 6, fig. 6 shows a second downsampled image schematic for the bayer pattern raw image shown in fig. 3.

In fig. 6, the oval area 16 may be a second bayer pattern data obtained by down-sampling a set of first bayer pattern data indicated by 15 in the large oval area in fig. 5.

In fig. 6, one second bayer pattern data outlined by the elliptical region 16 may include four third pixels. In fig. 6, the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) may be four third pixels included in one second bayer pattern data obtained by down-sampling 15 mutually independent four first bayer pattern data, which are outlined in a large ellipse in fig. 5. The four third pixels are also in the form of a 2 x 2 matrix. The four third pixels belong to four color channels, respectively. Specifically, in fig. 6, in four third pixels included in one second bayer pattern data framed by the elliptical region 16, the pixel coordinate (1,1) belongs to the first G color channel, the pixel coordinate (1,2) belongs to the R color channel, the pixel coordinate (2,1) belongs to the B color channel, and the pixel coordinate (2,2) may belong to the second G color channel.

In fig. 6, the pixel coordinates (1,3), the pixel coordinates (1,4), the pixel coordinates (2,3), and the pixel coordinates (2,4) may be one second bayer pattern data obtained by down-sampling 30 sets of first bayer pattern data outlined by a large ellipse in fig. 5.

In the embodiment of the invention, in the bayer pattern raw image, every four first bayer pattern data are taken as a group, each group of first bayer pattern data is downsampled into one second bayer pattern data, and a second downsampled image is obtained, wherein each group of first bayer pattern data is a 2 x 2 matrix. Therefore, in the embodiment of the present invention, the second downsampled image also belongs to the bayer format.

In the embodiment of the present invention, in the bayer pattern raw image, every four first bayer pattern data are used as a group, and each group of first bayer pattern data is downsampled into one second bayer pattern data to obtain a second downsampled image. Through the downsampling, the subsequent operation amount is greatly reduced, and the denoising speed is accelerated.

In the embodiment of the invention, in the bayer pattern raw image, every four first bayer pattern data are used as a group, each group of first bayer pattern data is downsampled into a second bayer pattern data, so as to obtain a second downsampled image, which is equivalent to converting the four first bayer pattern data into the second bayer pattern data, and the second bayer pattern data also retains four channels, so that partial color information of the bayer pattern raw image is retained. Therefore, in the second downsampled image, four channels are relatively independent, and partial color information of the bayer-format raw image is retained.

In an embodiment of the present invention, optionally, referring to fig. 7, fig. 7 shows a flowchart of the steps of obtaining the second down-sampled image. Specifically, in the bayer-formatted original image, taking every four first bayer-formatted data as a group, and down-sampling each group of the first bayer-formatted data into a second bayer-formatted data, the step of obtaining a second down-sampled image may include:

a substep S21 of dividing every fourth first bayer pattern data into a set in the original image; wherein, each group of first Bayer format data is a matrix of 2 x 2.

In the embodiment of the present invention, in the raw image, every four first bayer pattern data are divided into one group, and each group of the first bayer pattern data is a matrix of 2 × 2.

For example, referring to fig. 5, 15 of the large oval boxes in fig. 5 is to divide four first bayer pattern data into a group. The set of first bayer pattern data is in the form of a 2 x 2 matrix. The set of first bayer pattern data may be composed of first bayer pattern data 1 framed by a small ellipse 11, first bayer pattern data 2 framed by a small ellipse 13, first bayer pattern data 3 framed by a small ellipse 14, and first bayer pattern data 4 framed by an ellipse adjacent to both the small ellipse 14 and the small ellipse 13.

In fig. 5, a set of first bayer pattern data 1, first bayer pattern data 2, first bayer pattern data 3, and first bayer pattern data 4 is in a 2 × 2 matrix form.

In sub-step S22, the first pixel included in each set of the first bayer pattern data is down-sampled into four third pixels according to the color channel to which each pixel belongs.

In the bayer-format raw image, every sixteen first pixels included in each set of first bayer-format data are obtained, and four first pixels included in each set of first bayer-format data belong to four color channels, respectively.

For example, referring to fig. 5, the large oval box 15 in fig. 5 is a set of first bayer pattern data. The set of bayer pattern data may include four first bayer pattern data that may be composed of first bayer pattern data 1 framed by a small ellipse 11, first bayer pattern data 2 framed by a small ellipse 13, first bayer pattern data 3 framed by a small ellipse 14, and first bayer pattern data 4 framed by an ellipse adjacent to both the small ellipse 14 and the small ellipse 13. In fig. 5, the first bayer pattern data 1, the first bayer pattern data 2, and the first bayer pattern data 3 are independent of each other, and the first bayer pattern data 4 is in the form of a 2 × 2 matrix.

In the embodiment of the present invention, the four first bayer pattern data are independent of each other, which means that there are no overlapping pixels of the four first bayer pattern data. For example, as shown in fig. 5, the first bayer pattern data 1 framed by the small ellipse 11, the first bayer pattern data 2 framed by the small ellipse 13, the first bayer pattern data 3 framed by the small ellipse 14, and the first bayer pattern data 4 framed by ellipses adjacent to both the small ellipse 14 and the small ellipse 13 have no overlapping pixels with each other.

In the embodiment of the present invention, one first bayer pattern data includes four first pixels. The four first pixels belong to four color channels, respectively. For example, in fig. 5, the first bayer pattern data 1 framed by the small ellipse 11, that is, the four first pixels are included, which are: a first pixel at pixel coordinate (1,1), a first pixel at pixel coordinate (1,2), a first pixel at pixel coordinate (2,1), a first pixel at pixel coordinate (2, 2). The first pixel of pixel coordinate (1,1) belongs to the first G color channel, the first pixel of pixel coordinate (1,2) belongs to the R color channel, the first pixel of pixel coordinate (2,1) belongs to the B color channel, and the first pixel of pixel coordinate (2,2) belongs to the second G color channel.

In the embodiment of the present invention, sixteen first pixels included in each set of the first bayer pattern data are acquired. For example, referring to fig. 5, a set of first bayer pattern data 15, which is a large oval frame, includes a first pixel of pixel coordinates (1,1), a first pixel of pixel coordinates (1,2), a first pixel of pixel coordinates (1,3), a first pixel of pixel coordinates (1,4), a first pixel of pixel coordinates (2,1), a first pixel of pixel coordinates (2,2), a first pixel of pixel coordinates (2,3), a first pixel of pixel coordinates (2,4), a first pixel of pixel coordinates (3,1), a first pixel of pixel coordinates (3,2), a first pixel of pixel coordinates (3,3), a first pixel of pixel coordinates (3,4), a first pixel of pixel coordinates (4,1), a first pixel of pixel coordinates (4,2), a first pixel of pixel coordinates (4,3), The first pixel of pixel coordinates (4,4), for a total of 16 first pixels.

In fig. 5, a first pixel at pixel coordinates (1,3), a first pixel at pixel coordinates (1,4), a first pixel at pixel coordinates (2,3), and a first pixel at pixel coordinates (2, 4). The first bayer pattern data 2 framed by the small ellipse 13, the first pixel of the pixel coordinate (1,3) belongs to the first G color channel, the first pixel of the pixel coordinate (1,4) belongs to the R color channel, the first pixel of the pixel coordinate (2,3) belongs to the B color channel, and the first pixel of the pixel coordinate (2,4) belongs to the second G color channel.

In fig. 5, a first pixel at pixel coordinate (3,1), a first pixel at pixel coordinate (3,2), a first pixel at pixel coordinate (4,1), and a first pixel at pixel coordinate (4, 2). The first bayer pattern data 3 framed by the small ellipse 14, the first pixel of the pixel coordinate (3,1) belongs to the first G color channel, the first pixel of the pixel coordinate (3,2) belongs to the R color channel, the first pixel of the pixel coordinate (4,1) belongs to the B color channel, and the first pixel of the pixel coordinate (4,2) belongs to the second G color channel.

In fig. 5, a first pixel at pixel coordinates (3,3), a first pixel at pixel coordinates (3,4), a first pixel at pixel coordinates (4,3), a first pixel at pixel coordinates (4, 4). Belonging to the first bayer pattern data 4, the first pixel of the pixel coordinates (3,3) belongs to the first G color channel, the first pixel of the pixel coordinates (3,4) belongs to the R color channel, the first pixel of the pixel coordinates (4,3) belongs to the B color channel, and the first pixel of the pixel coordinates (4,4) belongs to the second G color channel.

In the embodiment of the present invention, in the bayer pattern raw image, sixteen first pixels included in each set of first bayer pattern data are down-sampled into four third pixels according to a channel to which each first pixel belongs, and a second down-sampled image is obtained.

In a specific application, for example, referring to fig. 6, in fig. 6, the third image of the pixel coordinate (1,1) may be the first G color channel, and may be an average value of four first G color channels in every four first bayer pattern data included in the set of first bayer pattern data. For example, in fig. 5, the first pixel of pixel coordinate (1,1) belongs to the first G color channel, the first pixel of pixel coordinate (1,3) belongs to the first G color channel, the first pixel of pixel coordinate (3,1) belongs to the first G color channel, and the first pixel of pixel coordinate (3,3) belongs to the first G color channel, then, in fig. 6, the third pixel of pixel coordinate (1,1) may be, in fig. 5, the first pixel of pixel coordinate (1,1), the first pixel of pixel coordinate (1,3), the first pixel of pixel coordinate (3,1), the first pixel of pixel coordinate (3,3), and the average of these four pixel values belonging to the first G color channel.

In the embodiment of the present invention, the average value may be an arithmetic average value, a geometric average value, a harmonic average value, a weighted average value, and the like, and this is not particularly limited in the embodiment of the present invention. When the average value here is an arithmetic average value, in fig. 6, the pixel value of the third pixel at the pixel coordinate (1,1) is equal to (in fig. 5, the pixel value of the first pixel at the pixel coordinate (1,1) + the pixel value of the first pixel at the pixel coordinate (1,3) + the pixel value of the first pixel at the pixel coordinate (3, 1))/4. In fig. 6, the third pixel of pixel coordinate (1,1) also belongs to the first G color channel.

Similarly, in fig. 6, the pixel value of the third pixel at the pixel coordinate (1,2) is equal to (in fig. 5, the pixel value of the first pixel at the pixel coordinate (1,2) + the pixel value of the first pixel at the pixel coordinate (1,4) + the pixel value of the first pixel at the pixel coordinate (3,2) + the pixel value of the first pixel at the pixel coordinate (3, 4))/4. In fig. 5, the first pixel of pixel coordinates (1,2), the first pixel of pixel coordinates (1,4), the first pixel of pixel coordinates (3,2), the first pixel of pixel coordinates (3,4) all belong to the R color channel,

in fig. 6, the third pixel of the pixel coordinates (1,2) belongs to the R color channel.

Similarly, in fig. 6, the pixel value of the third pixel at the pixel coordinate (2,1) is equal to (in fig. 5, the pixel value of the first pixel at the pixel coordinate (2,1) + the pixel value of the first pixel at the pixel coordinate (2,3) + the pixel value of the first pixel at the pixel coordinate (4,1) + the pixel value of the first pixel at the pixel coordinate (4, 3))/4. In fig. 5, the first pixel of the pixel coordinate (2,1), the first pixel of the pixel coordinate (2,3), the first pixel of the pixel coordinate (4,1), and the first pixel of the pixel coordinate (4,3) all belong to the B color channel, and in fig. 6, the third pixel of the pixel coordinate (2,1) belongs to the B color channel.

Similarly, in fig. 6, the pixel value of the third pixel at the pixel coordinate (2,2) is equal to (in fig. 5, the pixel value of the first pixel at the pixel coordinate (2,2) + the pixel value of the first pixel at the pixel coordinate (2,4) + the pixel value of the first pixel at the pixel coordinate (4,2) + the pixel value of the first pixel at the pixel coordinate (4, 4))/4. In fig. 5, the first pixel of pixel coordinates (2,2), the first pixel of pixel coordinates (2,4), the first pixel of pixel coordinates (4,2), and the first pixel of pixel coordinates (4,4) all belong to the second G color channel, and in fig. 6, the third pixel of pixel coordinates (2,2) belongs to the second G color channel.

Based on the same method, 16 first pixels included in each set of first bayer pattern data are down-sampled into one second bayer pattern data according to the channel to which each first pixel belongs.

In the embodiment of the present invention, the four third pixels belong to four different color channels, respectively. For example, in fig. 6, the third pixel of pixel coordinate (1,1) also belongs to the first G color channel, the third pixel of pixel coordinate (1,2) belongs to the R color channel, the third pixel of pixel coordinate (2,1) belongs to the B color channel, and the third pixel of pixel coordinate (2,2) belongs to the second G color channel.

Sub-step S23, composing the second bayer pattern data with the four third pixels to obtain the second downsampled image.

In the embodiment of the present invention, the four third pixels constitute a second bayer pattern data. In a specific application, for example, referring to fig. 6, in fig. 6, a third pixel of pixel coordinates (1,1), a third pixel of pixel coordinates (1,2), a third pixel of pixel coordinates (2,1), and a third pixel of pixel coordinates (2,2) form a second bayer pattern data. In fig. 6, the oval area 16 may be a group of first bayer pattern data indicated by 15 in fig. 5, which is outlined by a large oval, and the second bayer pattern data obtained by down-sampling.

In the embodiment of the present invention, the four third pixels are grouped into the second bayer pattern data to obtain the second downsampled image.

In the embodiment of the present invention, the four third pixels belong to four different color channels, respectively. For example, in fig. 6, the third pixel of pixel coordinate (1,1) also belongs to the first G color channel, the third pixel of pixel coordinate (1,2) belongs to the R color channel, the third pixel of pixel coordinate (2,1) belongs to the B color channel, the third pixel of pixel coordinate (2,2) belongs to the second G color channel, and the second downsampled image also includes four color channels. Referring to fig. 6, fig. 6 shows a second downsampled image schematic for the bayer pattern raw image shown in fig. 3.

In a specific application, the down-sampling of the raw data is equivalent to replacing every four first bayer pattern data in the raw data by the one second bayer pattern data, so that the second down-sampled image may include one fourth of the number of pixels of the bayer pattern raw image. Through the downsampling, the subsequent operation amount is greatly reduced, and the denoising speed is accelerated.

In the embodiment of the invention, in the bayer pattern raw image, every four first bayer pattern data are used as a group of first bayer pattern data, sixteen first pixels included in each group of first bayer pattern data are downsampled into second bayer pattern data according to a channel to which each first pixel belongs, so as to obtain a second downsampled image, which is equivalent to converting the four first bayer pattern data into the second bayer pattern data, the second bayer pattern data also retains four channels, and the four channels are sampled relatively independently, so that partial color information of the bayer pattern raw image is retained. Therefore, in the second downsampled image, four channels are relatively independent, and partial color information of the bayer-format raw image is retained.

In an embodiment of the present invention, the first downsampled image of the bayer pattern raw image and the second downsampled image of the bayer pattern raw image include the same number of pixels, which is one fourth of the number of pixels included in the bayer pattern raw image. In the subsequent denoising process through the first downsampling image and the second downsampling image, compared with the Bayer format original image, the denoising speed is high due to the fact that the number of pixels is reduced.

In the embodiment of the present invention, the first downsampling image of the bayer pattern raw image focuses on the correlation of four channels, and retains partial luminance information of the bayer pattern raw image; and the four channels of the second downsampling image of the Bayer format original image are relatively independent, and partial color information of the Bayer format original image is reserved.

Step 203, denoising the first downsampled image to obtain a first denoised image.

In an embodiment of the invention, the first downsampled image is dessicated to obtain a first dessicated image. In a specific application, the first downsampled image can be denoised by adopting at least one of non-local mean denoising, discrete cosine transform based denoising, mean denoising and the like. In the embodiment of the present invention, this is not particularly limited.

In the embodiment of the present invention, optionally, a plurality of tests may be performed on the acquisition terminal of the bayer pattern raw image, and the corresponding relationship between different pixel brightness and noise under each illumination intensity is mainly obtained for the camera of the acquisition terminal.

Specifically, the acquisition terminal itself may be fixed, the acquisition object may also be kept unchanged, the illumination intensity at which the acquisition terminal is located is changed, specifically, the illumination intensity received by the camera of the acquisition terminal is changed, and under each illumination intensity, the corresponding relationship between different pixel brightness and noise is obtained. Under one illumination intensity, the method can correspond to the corresponding relation between the brightness and the noise of different pixels. In the embodiment of the present invention, this is not particularly limited.

In the embodiment of the present invention, in the process of denoising the first downsampled image, the correspondence between the illumination intensity, the pixel brightness and the noise corresponding to the acquisition terminal of the bayer pattern raw image may be used as a reference. In the embodiment of the present invention, this is not particularly limited.

In a specific application, the illumination intensity of the bayer pattern raw image may be obtained, the pixel brightness of the first downsampled image may be determined, and the noise of the first downsampled image may be determined according to the correspondence relationship between the illumination intensity, the pixel brightness, and the noise. The illumination intensity of the bayer pattern raw image may be an attribute of the bayer pattern raw image itself, and the bayer pattern raw image is determined, and the corresponding illumination intensity is determined. The illumination intensity of the bayer pattern raw image may be the illumination intensity received by the acquisition terminal when the bayer pattern raw image is acquired. In the embodiment of the present invention, this is not particularly limited.

In a specific application, the light intensity of the bayer pattern raw image may be obtained from the attribute information of the bayer pattern raw image and the like. For example, the light intensity of the bayer pattern raw image may be obtained from a property file such as a header file of the bayer pattern raw image. In the embodiment of the present invention, this is not particularly limited.

And 204, denoising the second downsampled image based on the first denoised image to obtain a second denoised image.

In the embodiment of the invention, the second downsampled image is dehumidified based on the first denoised image, and a second dehumidified image is obtained.

In a specific application, at least one of non-local mean value denoising, discrete cosine transform based denoising, mean value denoising and other denoising can be adopted to denoise the second down-sampling image. In the embodiment of the present invention, this is not particularly limited.

For example, discrete cosine transform based denoising may be employed to denoise the second downsampled image. In the second downsampling image denoising process, denoising may be performed based on the correspondence between the illumination intensity, the pixel brightness, and the noise. In the embodiment of the present invention, this is not particularly limited.

In this embodiment of the present invention, optionally, when denoising the second downsampled image by using a non-local mean value, the denoising the second downsampled image based on the first denoised image to obtain a second denoised image includes: determining a first inter-block similarity of the first denoised image; determining a second inter-block similarity of a second down-sampling image based on the first inter-block similarity according to the corresponding relation between the first denoised image and the second down-sampling image; and denoising the second down-sampling image by adopting a non-local mean value based on the second inter-block similarity to obtain a second denoised image.

Specifically, if the non-local mean denoising is adopted in the process of denoising the second down-sampling image, the second down-sampling image may be denoised based on the first denoised image. Specifically, when denoising is performed on a second downsampled image, non-local mean denoising is adopted, a first inter-block similarity of a first denoised image can be determined, then a second inter-block similarity of the second downsampled image is determined based on the first inter-block similarity according to a corresponding relation between the first denoised image and the second downsampled image, and the second downsampled image is denoised based on the second inter-block similarity to obtain a second denoised image.

Specifically, the second inter-block similarity of the second downsampled image is determined based on the first inter-block similarity according to the corresponding relationship between the first denoised image and the second downsampled image, which may be based on the corresponding relationship between the first downsampled image corresponding to the first denoised image and the second downsampled image, and the first inter-block similarity is converted into the second inter-block similarity of the second downsampled image.

In the embodiment of the invention, after the second inter-block similarity of the second down-sampling image is determined, the second denoised image corresponding to the second down-sampling image is determined according to the self information of the second down-sampling image. On the basis of the first denoised image, in the process of denoising the second downsampled image, the first denoised image is a denoised image of the Bayer format original image, which is equivalent to the denoised image based on the Bayer format original image, the second downsampled image is denoised, the original image is denoised twice to a certain extent, and the method can be understood as denoising the Bayer format original image twice, so that the denoising effect is better.

In the embodiment of the present invention, the first denoised image and the second denoised image include one fourth of the number of pixels included in the bayer pattern raw image. In the embodiment of the present invention, the first downsampled image of the bayer pattern original image pays attention to the correlation of four channels, and retains partial luminance information of the bayer pattern original image, so that the first denoised image obtained based on the first downsampled image also pays attention to the correlation of four channels, and retains partial luminance information of the bayer pattern original image; four channels of the second downsampled image of the bayer pattern original image are relatively independent, and partial color information of the bayer pattern original image is reserved, so that the same four channels of the second denoised image obtained based on the second downsampled image are relatively independent, and partial color information of the bayer pattern original image is reserved.

Step 205, replacing four fourth pixels in the first denoised image with third bayer pattern data in the second denoised image, respectively, to obtain intermediate up-sampled data; the positions of the four fourth pixels in the first denoised image correspond to the positions of the fifth pixels in the third bayer pattern data in the second denoised image.

In the embodiment of the invention, the fourth pixels in the first denoised image are respectively replaced by the third Bayer format data in the second denoised image, and intermediate up-sampling data is obtained; the positions of the four fourth pixels in the first denoised image correspond to the positions of the fifth pixels in the third bayer pattern data in the second denoised image.

In an embodiment of the present invention, the second denoised image may include third bayer pattern data. The third bayer pattern data also includes four fifth pixels. For example, referring to fig. 8, fig. 8 shows a schematic diagram of a second denoised image.

In fig. 8, the oval region 17 indicates the third bayer pattern data. The third bayer pattern data includes four fifth pixels, i.e., the third bayer pattern data indicated by the elliptical region 17, i.e., the fifth pixel having the pixel coordinate (1,1), the fifth pixel having the pixel coordinate (1,2), the fifth pixel having the pixel coordinate (2,1), and the fifth pixel having the pixel coordinate (2, 2).

In the embodiment of the present invention, the color value corresponding to each channel of the third bayer pattern data may be obtained. For example, referring to fig. 8, in the third bayer pattern data indicated by the oval region 17, the color value of the first G channel corresponding to the pixel coordinate (1,1) is 60, the color value of the R channel corresponding to the pixel coordinate (1,2) is 70, the color value of the B channel corresponding to the pixel coordinate (2,1) is 80, and the color value of the second G channel corresponding to the pixel coordinate (2,2) is 80.

In an embodiment of the present invention, the first denoised image comprises four fourth pixels. For example, referring to fig. 9, fig. 9 shows a schematic diagram of a first denoised image. In fig. 9, the pixel coordinate (1,1) indicated by the circular area 18 can be a fourth pixel in the first denoised image. In fig. 9, the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), the pixel coordinates (2,2), and the four fourth pixels indicated by the circular area 19 are in a 2 × 2 matrix form.

In the implementation of the present invention, the pixel value corresponding to the fourth pixel may be obtained, for example, as shown in fig. 9, the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) indicated by the circular area 19 may be 120,130,140, and 150, respectively.

In the embodiment of the present invention, a third bayer pattern data of the second denoised image is used to replace four fourth pixels in the first denoised image, respectively, so as to obtain the intermediate up-sampled data. That is, replacing the four fourth pixels indicated by the circular region 19 in fig. 9 with the third bayer pattern data indicated by the elliptical region 17 in fig. 8 described above, i.e., the four fourth pixels indicated by the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2), corresponds to replacing each of the four fourth pixels in fig. 9 with 4 pixels of one third bayer pattern data.

For example, referring to FIG. 10, FIG. 10 shows a schematic diagram of intermediate up-sampled data. The area outlined by the oval area 20 in fig. 10 is the corresponding middle up-sampled data after the third bayer pattern data indicated by the oval area 17 in fig. 8 replaces the four fourth pixels indicated by the circular area 18 in fig. 9.

In fig. 10, the color value of the first G channel corresponding to the pixel coordinate (1,1) is 60, the color value of the R channel corresponding to the pixel coordinate (1,2) is 70, the color value of the B channel corresponding to the pixel coordinate (2,1) is 80, and the color value of the second G channel corresponding to the pixel coordinate (2,2) is 80; the color value of the first G channel corresponding to the pixel coordinate (1,3) is 60, the color value of the R channel corresponding to the pixel coordinate (1,4) is 70, the color value of the B channel corresponding to the pixel coordinate (2,3) is 80, and the color value of the second G channel corresponding to the pixel coordinate (2,4) is 80; the color value of the first G channel corresponding to the pixel coordinate (3,1) is 60, the color value of the R channel corresponding to the pixel coordinate (3,2) is 70, the color value of the B channel corresponding to the pixel coordinate (4,1) is 80, and the color value of the second G channel corresponding to the pixel coordinate (4,2) is 80; the color value of the first G channel corresponding to the pixel coordinate (3,3) is 60, the color value of the R channel corresponding to the pixel coordinate (3,4) is 70, the color value of the B channel corresponding to the pixel coordinate (4,3) is 80, and the color value of the second G channel corresponding to the pixel coordinate (4,4) is 80.

In an embodiment of the present invention, the position of the fourth pixel in the first dried image corresponds to the position of the third bayer pattern data, and the position of the fourth pixel in the second dried image corresponds to the position of the 16 corresponding first pixels in the original image before the drying and down-sampling of the fourth pixel and the position of the 16 corresponding first pixels in the original image before the drying and down-sampling of the third bayer pattern data correspond to the same position of the 16 corresponding first pixels in the four corresponding first bayer pattern data in the original image.

For example, in fig. 9, the pixel coordinates (1,1), pixel coordinates (1,2), pixel coordinates (2,1), and pixel coordinates (2,2) of the circular region 19 are marked, and before the de-drying and sampling, the 16 first pixels in the raw bayer pattern image are: in fig. 3, a first pixel of pixel coordinates (1,1), a first pixel of pixel coordinates (1,2), a first pixel of pixel coordinates (1,3), a first pixel of pixel coordinates (1,4), a first pixel of pixel coordinates (2,1), a first pixel of pixel coordinates (2,2), a first pixel of pixel coordinates (2,3), a first pixel of pixel coordinates (2,4), a first pixel of pixel coordinates (3,1), a first pixel of pixel coordinates (3,2), a first pixel of pixel coordinates (3,3), a first pixel of pixel coordinates (3,4), a first pixel of pixel coordinates (4,1), a first pixel of pixel coordinates (4,2), a first pixel of pixel coordinates (4,3), a first pixel of pixel coordinates (4, 4). In fig. 8, the third bayer pattern data is indicated by the oval region 17. Before the drying and sampling, in the raw bayer pattern image, as shown in fig. 5, 16 first pixels included in the first bayer pattern data 1, the first bayer pattern data 2, the first bayer pattern data 3, the first bayer pattern data 4, and the four first bayer pattern data are respectively the same.

In an embodiment of the present invention, the intermediate up-sampled data includes four times the number of pixels of the first denoised image and four times the number of pixels of the second denoised image. Meanwhile, the second denoised image is formed by third Bayer format data and comprises independence and color information of four channels of the Bayer format original image, the first denoised image takes relevance of the four channels of the Bayer format original image and Bayer format original image brightness information into consideration, and the intermediate up-sampling data not only comprises the independence and the color information of the four channels of the Bayer format original image, but also comprises the relevance of the four channels and the Bayer format original image brightness information.

In the embodiment of the present invention, since the number of pixels included in the first denoised image is one fourth of the number of pixels included in the bayer pattern raw image, the upsampling is equivalent to multiplying the number of pixels included in the first denoised image by four, and the number of pixels included in the intermediate sample data is the same as the number of pixels included in the bayer pattern raw image. Meanwhile, the second denoised image is formed by third Bayer format data and comprises independence and color information of four channels of the Bayer format original image, the first denoised image takes relevance of the four channels of the Bayer format original image and Bayer format original image brightness information into consideration, and the intermediate up-sampling data not only comprises the independence and the color information of the four channels of the Bayer format original image, but also comprises the relevance of the four channels and the Bayer format original image brightness information.

In step 206, normalization processing is performed on the four fifth pixels of each third bayer pattern data in the intermediate up-sampled data, so as to obtain four sixth pixels.

In this embodiment of the present invention, normalization processing may be performed on four fifth pixels included in each third bayer pattern data in the third bayer pattern data, so as to obtain four sixth pixels.

In the embodiment of the present invention, the data is up-sampled in the middle, and every two third bayer pattern data have no overlapping portion.

In this embodiment of the present invention, as shown in fig. 10, in the area framed by the oval area 20 in fig. 10, the color value of the first G channel corresponding to the pixel coordinate (1,1) is 60, the color value of the R channel corresponding to the pixel coordinate (1,2) is 70, the color value of the B channel corresponding to the pixel coordinate (2,1) is 80, and the color value of the second G channel corresponding to the pixel coordinate (2,2) is 80, that is, the third bayer pattern data in the intermediate up-sampled data is referred to.

In a specific application, normalization processing may be performed on the four fifth pixels included in the third bayer pattern data to obtain four sixth pixels.

In the embodiment of the present invention, for example, if the third bayer pattern data is composed of the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) in the region framed by the oval region 20 in fig. 10, the third bayer pattern data includes four fifth pixels, and the four fifth pixels are in a 2 × 2 matrix form. For example, in the region framed by the elliptical region 20 in fig. 10, the color value of the first G channel corresponding to the pixel coordinate (1,1) is 60, the color value of the R channel corresponding to the pixel coordinate (1,2) is 70, the color value of the B channel corresponding to the pixel coordinate (2,1) is 80, the color value of the second G channel corresponding to the pixel coordinate (2,2) is 80, and the four fifth pixels included in the third bayer pattern data are normalized, which can be obtained as shown in fig. 11, where fig. 11 shows a data schematic diagram after normalization processing of the fifth pixel in the intermediate up-sampled data.

In an embodiment of the present invention, the normalization processing may be performed on the four fifth pixels included in the third bayer pattern data by taking an average value of the four fifth pixels included in the third bayer pattern data, and dividing a color value of each channel of the third bayer pattern data by an average value of the four fifth pixels in the third bayer pattern data. For example, referring to fig. 11, the data obtained by performing normalization processing on the fifth pixel in the intermediate up-sampled data, which is framed by the elliptical region 21, is a processing result of the third bayer pattern data, which corresponds to the method, in which the color value of the first G channel corresponding to the pixel coordinate (1,1) is 60, the color value of the R channel corresponding to the pixel coordinate (1,2) is 70, the color value of the B channel corresponding to the pixel coordinate (2,1) is 80, and the color value of the second G channel corresponding to the pixel coordinate (2,2) is 80.

In fig. 11, the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) are four sixth pixels obtained by normalizing, in fig. 10, the color value of the first G channel corresponding to the pixel coordinates (1,1) is 60, the color value of the R channel corresponding to the pixel coordinates (1,2) is 70, the color value of the B channel corresponding to the pixel coordinates (2,1) is 80, and the color value of the second G channel corresponding to the pixel coordinates (2,2) is 80.

And step 207, multiplying the four sixth pixels by fourth pixels at corresponding positions respectively to obtain fourth bayer pattern data, so as to obtain a denoised image corresponding to the original image.

In the embodiment of the present invention, four sixth pixels are respectively multiplied by the fourth pixel at the corresponding position to obtain a fourth bayer pattern data, so as to obtain a denoised image corresponding to the original image.

In a specific application, referring to fig. 11, the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) marked by the elliptical region 21 may be four sixth pixels obtained after the first normalization processing is performed on the four fifth pixels of the third bayer pattern data.

Multiplying the four sixth pixels by the fourth pixels at the corresponding positions, that is, multiplying the G color value corresponding to the pixel coordinate (1,1) in fig. 11 by the fourth pixel value corresponding to the pixel coordinate (1,1) in fig. 9, multiplying the R color value corresponding to the pixel coordinate (1,2) in fig. 11 by the fourth pixel value corresponding to the pixel coordinate (1,2) in fig. 9, multiplying the B color value corresponding to the pixel coordinate (2,1) in fig. 11 by the fourth pixel value corresponding to the pixel coordinate (2,1) in fig. 9, multiplying the G color value corresponding to the pixel coordinate (2,2) in fig. 11 by the fourth pixel value corresponding to the pixel coordinate (2,2) in fig. 9, to obtain the G color value corresponding to the pixel coordinate (1,1), the R color value corresponding to the pixel coordinate (1,2), and the pixel coordinate (2) in fig. 12, 1) the corresponding B color value, the G color value corresponding to pixel coordinate (2, 2).

In fig. 12, pixel coordinates (1,1), pixel coordinates (1,2), pixel coordinates (2,1), and pixel coordinates (2,2) constitute a fourth bayer pattern data. Referring to fig. 12, fig. 12 shows a fourth bayer pattern data pattern obtained for fig. 9 and 11, according to the example shown in fig. 9 and 11.

In the embodiment of the present invention, if the four sixth pixels are multiplied by the fourth pixels at corresponding positions respectively, the color value of each pixel in the fourth bayer pattern data obtained has a fraction or decimal, such as a fraction or a decimal

In fig. 12, pixel coordinates (1,1), pixel coordinates (1,2), pixel coordinates (2,1), and pixel coordinates (2,2) indicate that, in the fourth bayer pattern data, a color value of a pixel that is a decimal or a fractional number may be rounded. And in the fourth bayer pattern data, rounding all color values of which the color values of the pixels are decimal or fractional, so as to obtain a denoised image of the original image.

For example, referring to fig. 13, fig. 13 shows a schematic diagram of a denoised image corresponding to the original image shown in fig. 3 obtained by rounding the pixel color values in fig. 12. In fig. 13, the pixel coordinates (1,1), the pixel coordinates (1,2), and the pixel coordinates (2,1) framed by the ellipse 32 are the rounding result for the pixel color values of the pixel coordinates (1,1), the pixel coordinates (1,2), the pixel coordinates (2,1), and the pixel coordinates (2,2) in fig. 12.

In fig. 13, the area framed by the elliptical area 33 is a denoised image corresponding to a third bayer pattern data framed by 17 in fig. 8 and four fourth pixels framed by 19 in fig. 9, which are obtained by upsampling four bayer pattern raw images framed by 15 in fig. 5. In the embodiment of the invention, the de-noising image of the Bayer format original image is obtained by analogy.

In this embodiment of the present invention, optionally, after obtaining the intermediate up-sampled data, four fifth pixels in the third bayer pattern data in the intermediate up-sampled data may be directly multiplied by the fourth pixels at corresponding positions, without performing normalization on the four fifth pixels in the third bayer pattern data, so as to obtain a bayer pattern data. If the fourth pixel in the third bayer pattern data is multiplied by the fourth pixel in the corresponding position, the product may be 255 if the product is greater than 255. Similarly, if the product is a fraction or a decimal, the product is also rounded to obtain a denoised image corresponding to the original image, which is not specifically limited in the embodiment of the present invention.

In the embodiment of the invention, the relevance of four channels of the bayer pattern raw image and the luminance information of the bayer pattern raw image are considered for the first denoised image, the independence and the color information of four channels of the bayer pattern raw image are considered for the second denoised image, and then the relevance of the four channels, the luminance information of the bayer pattern raw image, the independence of the four channels and the color information of the bayer pattern raw image are fully considered in the process of obtaining the denoised image of the bayer pattern raw image by upsampling the first denoised image of the first downsampled image and the second denoised image of the second downsampled image, so that the denoising effect of the bayer pattern raw image is improved.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the embodiments of the application.

EXAMPLE III

Referring to fig. 14, which is a block diagram of a structure of an image denoising device 300 according to a third embodiment of the present invention, the image denoising device 300 may specifically include:

a down-sampling module 301, configured to perform down-sampling on an original image according to a first preset manner and a second preset manner, respectively, to obtain a first down-sampled image and a second down-sampled image;

a denoised image obtaining module 302, configured to denoise the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image;

a denoised image generating module 303, configured to perform upsampling on the basis of the first denoised image and the second denoised image, so as to obtain a denoised image corresponding to the original image.

Alternatively, referring to fig. 15, on the basis of fig. 14, the original image includes: a raw image in bayer format; the down-sampling module 301 comprises:

a first sub-downsampling module 3011, configured to downsample each first bayer pattern data into a second pixel in the bayer pattern raw image to obtain a first downsampled image; the first bayer pattern data includes four first pixels in a 2 x 2 matrix;

a second sub-downsampling module 3012, configured to take every four first bayer pattern data as a group in the bayer pattern raw image, and downsample each group of first bayer pattern data into a second bayer pattern data to obtain a second downsampled image; wherein, each group of first Bayer format data is a matrix of 2 x 2.

Optionally, the second sub-downsampling module 3012 may include:

a grouping unit 30121 configured to divide every four first bayer pattern data into one group in the raw image;

a downsampling unit 30122, configured to downsample the first pixels included in each set of first bayer pattern data into four third pixels according to the color channel to which each pixel belongs;

a second downsampled image obtaining unit 30123 is configured to compose the second bayer pattern data by the four third pixels to obtain the second downsampled image.

Optionally, the denoised image obtaining module 302 may include:

a first denoised image obtaining unit 3021, configured to denoise the first downsampled image to obtain a first denoised image;

a second denoised image obtaining unit 3022, configured to denoise the second downsampled image based on the first denoised image, and obtain a second denoised image.

Optionally, the second denoised image obtaining unit 3022 may include:

a first inter-block similarity determining subunit, configured to determine a first inter-block similarity of the first denoised image;

a second inter-block similarity determining subunit, configured to determine, according to a correspondence between the first denoised image and the second downsampled image, a second inter-block similarity of the second downsampled image based on the first inter-block similarity;

and the second denoised image acquiring subunit is used for denoising the second down-sampled image by adopting a non-local mean value based on the second inter-block similarity to acquire a second denoised image.

Optionally, the denoised image generating module 303 may include:

an intermediate up-sampling data obtaining unit 3031, configured to replace four fourth pixels in the first denoised image with third bayer pattern data in the second denoised image, respectively, to obtain intermediate up-sampling data; the positions of the four fourth pixels in the first denoised image correspond to the positions of the fifth pixels in the third Bayer format data in the second denoised image;

a normalization unit 3032, configured to perform normalization processing on four fifth pixels of each third bayer pattern data in the intermediate up-sampled data to obtain four sixth pixels;

a denoised image generating unit 3033, configured to multiply the four sixth pixels with fourth pixels at corresponding positions, respectively, to obtain fourth bayer format data, so as to obtain a denoised image corresponding to the original image.

The image denoising device provided by the embodiment of the present invention can implement each process implemented by the image denoising device in the method embodiments of fig. 1 to fig. 13, and is not described herein again to avoid repetition.

In this way, in the embodiment of the present invention, the original image is down-sampled in a first preset manner and a second preset manner, respectively, to obtain a first down-sampled image and a second down-sampled image; denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image; and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image. In the embodiment of the invention, the original image is downsampled in two different preset modes, then is denoised respectively to obtain a first denoised image and a second denoised image, and then is upsampled based on the two denoised images to obtain the denoised image corresponding to the original image.

Fig. 16 is a schematic diagram of a hardware structure of a mobile terminal for implementing various embodiments of the present invention, where the mobile terminal 700 includes, but is not limited to: a radio frequency unit 701, a network module 702, a sound output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710, a power supply 711, and the like. Those skilled in the art will appreciate that the mobile terminal architecture shown in fig. 16 is not intended to be limiting of mobile terminals, and that a mobile terminal may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. In the embodiment of the present invention, the mobile terminal includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted terminal, a wearable device, a pedometer, and the like.

The processor 710 is configured to perform downsampling on an original image according to a first preset manner and a second preset manner, respectively, to obtain a first downsampled image and a second downsampled image;

According to the embodiment of the invention, the original image is respectively subjected to down-sampling according to a first preset mode and a second preset mode to obtain a first down-sampling image and a second down-sampling image; denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image; and performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image. In the embodiment of the invention, the original image is downsampled in two different preset modes, then is denoised respectively to obtain a first denoised image and a second denoised image, and then is upsampled based on the two denoised images to obtain the denoised image corresponding to the original image.

It should be understood that, in the embodiment of the present invention, the radio frequency unit 701 may be used for receiving and sending signals during a message transmission and reception process or a call process, and specifically, receives downlink data from a base station and then processes the received downlink data to the processor 710; in addition, the uplink data is transmitted to the base station. In general, radio frequency unit 701 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 701 may also communicate with a network and other devices through a wireless communication system.

The mobile terminal provides the user with wireless broadband internet access via the network module 702, such as helping the user send and receive e-mails, browse web pages, and access streaming media.

The sound output unit 703 may convert sound data received by the radio frequency unit 701 or the network module 702 or stored in the memory 709 into a sound signal and output as sound. Also, the sound output unit 703 may also provide sound output related to a specific function performed by the mobile terminal 700 (e.g., a call signal reception sound, a message reception sound, etc.). The sound output unit 703 includes a speaker, a buzzer, a receiver, and the like.

The input unit 704 is used to receive a sound or video signal. The input Unit 704 may include a Graphics Processing Unit (GPU) 7041 and a microphone 7042, and the Graphics processor 7041 processes image data of a still picture or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 706. The image frames processed by the graphic processor 7041 may be stored in the memory 709 (or other storage medium) or transmitted via the radio unit 701 or the network module 702. The microphone 7042 may receive sound, and may be capable of processing such sound into sound data. The processed voice data may be converted into a format output transmittable to a mobile communication base station via the radio frequency unit 701 in case of a phone call mode.

The mobile terminal 700 also includes at least one sensor 705, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor that adjusts the brightness of the display panel 7061 according to the brightness of ambient light, and a proximity sensor that turns off the display panel 7061 or a backlight when the mobile terminal 700 moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), detect the magnitude and direction of gravity when stationary, and can be used to identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), and vibration identification related functions (such as pedometer, tapping); the sensors 705 may also include fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which are not described in detail herein.

The display unit 706 is used to display information input by the user or information provided to the user. The Display unit 706 may include a Display panel 7061, and the Display panel 7061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The user input unit 707 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, the user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071, also referred to as a touch screen, may collect touch operations by a user on or near the touch panel 7071 (e.g., operations by a user on or near the touch panel 7071 using a finger, a stylus, or any other suitable object or attachment). The touch panel 7071 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 710, receives a command from the processor 710, and executes the command. In addition, the touch panel 7071 can be implemented by various types such as resistive, capacitive, infrared, and surface acoustic wave. The user input unit 707 may include other input devices 7072 in addition to the touch panel 7071. In particular, the other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described herein again.

Further, the touch panel 7071 may be overlaid on the display panel 7061, and when the touch panel 7071 detects a touch operation on or near the touch panel 7071, the touch operation is transmitted to the processor 710 to determine the type of the touch event, and then the processor 710 provides a corresponding visual output on the display panel 7061 according to the type of the touch event. Although the touch panel 7071 and the display panel 7061 are shown in fig. 16 as two separate components to implement the input and output functions of the mobile terminal, in some embodiments, the touch panel 7071 and the display panel 7061 may be integrated to implement the input and output functions of the mobile terminal, which is not limited herein.

The interface unit 708 is an interface through which an external device is connected to the mobile terminal 700. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, a sound input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 708 may be used to receive input (e.g., data information, power, etc.) from external devices and transmit the received input to one or more elements within the mobile terminal 700 or may be used to transmit data between the mobile terminal 700 and external devices.

The memory 709 may be used to store software programs as well as various data. The memory 709 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as voice data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 709 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The processor 710 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by operating or executing software programs or modules stored in the memory 709 and calling data stored in the memory 709, thereby integrally monitoring the mobile terminal. Processor 710 may include one or more processing units; preferably, the processor 710 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 110.

The mobile terminal 700 may also include a power supply 711 (e.g., a battery) for powering the various components, and the power supply 711 may be logically coupled to the processor 710 via a power management system that may enable managing charging, discharging, and power consumption by the power management system.

In addition, the mobile terminal 700 includes some functional modules that are not shown, and thus will not be described in detail herein.

Preferably, an embodiment of the present invention further provides a mobile terminal, including a processor 710, a memory 709, and a computer program stored in the memory 709 and capable of running on the processor 710, where the computer program is executed by the processor 710 to implement each process of the image denoising method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not described here again.

Based on the hardware structure of the mobile terminal, the following detailed description will be made of embodiments of the present invention.

The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the above-mentioned embodiment of the image denoising method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.

While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. An image denoising method, comprising:

performing upsampling on the basis of the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image;

wherein the step of performing upsampling based on the first denoised image and the second denoised image to obtain a denoised image corresponding to the original image comprises:

replacing four fourth pixels in the first denoised image with third Bayer format data in the second denoised image respectively to obtain intermediate up-sampling data; the third bayer pattern data includes four fifth pixels in a 2 x 2 matrix, the four fourth pixels are four pixels constituting a 2 x 2 matrix in the first denoised image, and the positions of the four fourth pixels in the first denoised image correspond to the positions of the fifth pixels in the third bayer pattern data in the second denoised image;

normalizing four fifth pixels of each third Bayer format data in the intermediate up-sampled data to obtain four sixth pixels, wherein the four sixth pixels are four pixels forming a 2 x 2 matrix form in the intermediate up-sampled data;

and multiplying the four sixth pixels with fourth pixels at corresponding positions respectively to obtain fourth Bayer format data comprising the four pixels in a 2 x 2 matrix form so as to obtain a denoised image corresponding to the original image.

2. The method of claim 1, wherein the original image comprises: a raw image in bayer format; the step of respectively performing down-sampling on the original image according to a first preset mode and a second preset mode to obtain a first down-sampled image and a second down-sampled image comprises:

in the Bayer-format original image, downsampling each first Bayer-format data into a second pixel to obtain a first downsampled image; the first bayer pattern data includes four first pixels in a 2 x 2 matrix;

taking every four first Bayer format data as a group in the Bayer format raw image, and downsampling each group of first Bayer format data into a second Bayer format data to obtain a second downsampled image; wherein, each group of first Bayer format data is a matrix of 2 x 2.

3. The method according to claim 2, wherein the step of obtaining the second downsampled image by taking every four first bayer pattern data as a group and downsampling each group of the first bayer pattern data into one second bayer pattern data in the raw image in the bayer pattern comprises:

dividing every four first Bayer format data into a group in the original image;

downsampling first pixels included in each group of first Bayer format data into four third pixels according to the color channel to which each pixel belongs;

composing the second Bayer format data with the four third pixels to obtain the second downsampled image.

4. The method of claim 1, wherein the step of denoising the first downsampled image and the second downsampled image respectively to obtain a first denoised image and a second denoised image comprises:

denoising the first downsampled image to obtain a first denoised image;

and denoising the second downsampled image based on the first denoised image to obtain a second denoised image.

5. An image denoising apparatus, comprising:

a de-noising image generating module, configured to perform upsampling based on the first de-noising image and the second de-noising image to obtain a de-noising image corresponding to the original image;

the denoised image generation module comprises:

an intermediate up-sampling data obtaining unit, configured to replace four fourth pixels in the first denoised image with third bayer pattern data in the second denoised image, respectively, to obtain intermediate up-sampling data; the third bayer pattern data includes four fifth pixels in a 2 x 2 matrix, the four fourth pixels are four pixels constituting a 2 x 2 matrix in the first denoised image, and the positions of the four fourth pixels in the first denoised image correspond to the positions of the fifth pixels in the third bayer pattern data in the second denoised image;

the normalization unit is used for performing normalization processing on four fifth pixels of each third bayer pattern data in the intermediate up-sampled data to obtain four sixth pixels, wherein the four sixth pixels are four pixels forming a 2 x 2 matrix form in the intermediate up-sampled data;

and the de-noised image generating unit is used for multiplying the four sixth pixels with fourth pixels at corresponding positions respectively to obtain fourth Bayer format data comprising the four pixels in a 2 x 2 matrix form so as to obtain a de-noised image corresponding to the original image.

6. The apparatus of claim 5, wherein the original image comprises: a raw image in bayer format; the down-sampling module comprises:

a first sub-down-sampling module, configured to down-sample each first bayer pattern data into a second pixel in the bayer pattern raw image, to obtain a first down-sampled image; the first bayer pattern data includes four first pixels in a 2 x 2 matrix;

the second sub-down-sampling module is used for taking every four first Bayer format data as a group in the Bayer format raw image, and down-sampling each group of the first Bayer format data into a second Bayer format data to obtain a second down-sampled image; wherein, each group of first Bayer format data is a matrix of 2 x 2.

7. The apparatus of claim 6, wherein the second sub-downsampling module comprises:

a grouping unit configured to divide every four first bayer pattern data into one group in the raw image;

a down-sampling unit, configured to down-sample a first pixel included in each set of first bayer pattern data into four third pixels according to a color channel to which each pixel belongs;

a second downsampled image acquisition unit configured to compose the second bayer pattern data with the four third pixels to obtain the second downsampled image.

8. A mobile terminal, characterized in that it comprises a processor, a memory and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, implements the steps of the image denoising method according to any one of claims 1 to 4.

9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image denoising method according to any one of claims 1 to 4.