CN112291446B - Non-uniformity correction method for large-area array CMOS image sensor - Google Patents

Abstract

The invention relates to a non-uniformity correction method of a large-area array CMOS image sensor, which effectively solves the non-uniformity problem of the CMOS image sensor, reduces parameter storage, is easy to realize a hardware system and ensures the real-time requirement. Firstly, under the uniform illumination of an integrating sphere, acquiring an original image of a CMOS image sensor, calculating the non-uniformity of the acquired image to obtain an image corresponding to the maximum value of the non-uniformity, then grouping the columns of the large-area array CMOS image according to the structural characteristics of the CMOS image sensor, solving parameters by utilizing the grouping, and finally establishing a correction model for carrying out the non-uniformity correction on the CMOS image sensor. The invention introduces the least square method, increases the correction precision, obtains parameters far smaller than the parameter quantity of the traditional scaling method, saves hardware resources, reduces the system power consumption, and has simple algorithm structure, small calculated quantity and easy hardware realization.

Description

Non-uniformity correction method for large-area array CMOS image sensor

Technical Field

The invention relates to the technical field of image preprocessing, in particular to a non-uniformity correction method of a large-area array CMOS image sensor.

Background

With the continuous development and progress of the integrated circuit technology, the imaging performance of the CMOS device is continuously improved, and the imaging quality of the CMOS device can meet most application scenarios. Meanwhile, the imaging system based on the CMOS design has the advantages of low cost, low power consumption, simple structure, easy realization of system-on-chip integration and the like, so that the imaging system is more and more widely applied. However, any type of image sensor has a problem of non-uniformity with different degrees, and in practical applications, it is mostly necessary to use a corresponding non-uniformity correction technique for correction.

The non-uniformity of the CMOS image sensor is mainly represented by stripe noise, which is mainly caused by the non-uniform response of the pixels of the CMOS detector, including the subtle difference in the size of the pixels detected by the detector, the influence of weak dark current, the incompleteness of the correction of the response function, the non-uniformity of the optical coating on the surface of the detector, the influence of the external environment and temperature variation on the photoelectric system of the detector, and the like. The non-uniformity correction is an effective method for reducing the fixed pattern noise of the CMOS image sensor, and has important significance for acquiring high-quality images.

In engineering, calibration methods are generally used for non-uniformity correction, and calibration method based on calibration methods mainly include two-point correction, piecewise linear correction, polynomial fitting correction and sigmoid curve correction. The S-shaped curve correction method has the best effect on the non-uniformity correction of the image, but the calculation correction algorithm is complex to realize, and the realization difficulty is high by adopting the FPGA. Although the two-point correction method is easy to implement, the correction effect is not good when the two-point correction method has a nonlinear response characteristic. The correction effect of the piecewise linear correction is related to the length of the piecewise interval, and the more the piecewise interval is, the better the correction effect is, but the more parameters need to be obtained. For hardware real-time operation, too many interval segmentation cannot be realized, so that the segmentation uniformity correction result cannot be ensured. The polynomial fitting correction method fits the coefficients of the polynomial through a least square method, and can balance between calculated amount and correction effect by combining with actual engineering requirements.

However, with the continuous increase of the number of pixels of a large-area array CMOS image sensor, some pixels reach tens of millions and even hundreds of millions, and the frame rate is also continuously increased, so that the original algorithm is difficult to implement in hardware, the storage space of the required correction parameter is increased, the correction time is long, and the real-time processing cannot be guaranteed.

Disclosure of Invention

The invention provides a non-uniformity correction method of a large-area array CMOS image sensor, which can effectively solve the non-uniformity problem of the CMOS image sensor, simultaneously reduces parameter storage, is easy to realize a hardware system and ensures the real-time requirement, in order to overcome the problems of difficult hardware realization, large parameter storage space required for correction, long correction time, incapability of ensuring real-time correction processing and the like of the non-uniformity correction method in the prior art.

In order to solve the technical problems, the invention adopts the following technical scheme:

a non-uniformity correction method of a large-area array CMOS image sensor comprises the following steps:

the method comprises the following steps: under the uniform illumination of the integrating sphere, original image acquisition is carried out on the CMOS image sensor to obtain N images I with different irradiances₁,I₂,…,I_N；

Step two: for the collected image I₁,I₂,…,I_NCalculating the non-uniformity to obtain an image I corresponding to the maximum value of the non-uniformity_max；

Step three: finding an image I_maxDetermining the average value of the gray scale of each column of pixels to determine the maximum average value of the gray scale T_maxAnd minimum mean value of gray scale T_min；

Step four: will [ T_min,T_max]The interval is divided into k sub-intervals on average;

step five: according to image I_maxThe number of pixel rows m and the number of pixel columns n establish k m x n zero matrices a₁,a₂,…,a_i,…,a_kWherein a is_iI is more than or equal to 1 and less than or equal to k and is a zero matrix corresponding to the ith subinterval;

step six: according to image I_maxImage I is obtained by averaging the gray levels of each column of pixels_maxColumn mean matrix A of₁Column mean matrix A₁A matrix | x of 1 × n₁,x₂,…,x_j,…,x_nL, where x_jAs an image I_maxJ is more than or equal to 1 and less than or equal to n of the gray level mean value of the jth row of pixels;

step seven: according to the column mean matrix A₁The subinterval corresponding to each element of (1), image I_maxAre respectively assigned to the corresponding zero matrix a₁,a₂,…,a_i,…,a_kIn the corresponding column position, the corresponding matrix beta is obtained₁,β₂,…,β_k；

Step eight: according to the step seven pairs of images I_maxFor the image I respectively₁,I₂,…,I_NGrouping to obtain k groups of matrix data:

step nine: respectively solving a gray level mean value for each non-zero matrix element in the k groups of matrix data to obtain k gray level mean value matrixes:

step ten: calculating the mean value of each gray mean value matrix to obtain an image I₁,I₂,…,I_NCorresponding image gray average value avg₁,avg₂,…,avg_N；

Step eleven: taking the gray level mean value obtained in the ninth step as a vertical coordinate, taking the corresponding image gray level mean value obtained in the tenth step as a horizontal coordinate, and establishing a data relation to obtain k groups of data for curve fitting:

step twelve: introducing a least square method, and respectively performing curve fitting on the k groups of data according to the residual square sum minimum principle to obtain k curves f containing correction parameters_i(x)，i＝1,2,…,k；

Step thirteen: will matrix beta₁,β₂,…,β_kThe elements with the internal non-zero value are assigned to be 1 to obtain a matrix theta₁,θ₂,…,θ_k；

Fourteen steps: according to curve f_i(x) And matrix theta₁,θ₂,…,θ_kEstablishing a correction model for carrying out non-uniformity correction on the CMOS image sensor:

wherein, I_preFor the input image of the CMOS image sensor before correction, I_afterAnd outputting an image for the corrected CMOS image sensor.

Compared with the prior art, the invention has the following beneficial effects:

(1) according to the structural characteristics of the CMOS image sensor, grouping the large-area array CMOS image columns, and solving parameters by utilizing the grouping;

(2) a least square method is introduced, so that the correction precision is increased;

(3) the obtained parameters are far smaller than the parameter quantity of the traditional scaling method, so that hardware resources are saved, and the system power consumption is reduced;

(4) the algorithm has simple structure, small calculation amount and easy hardware realization.

Drawings

Fig. 1 is a flowchart of a non-uniformity correction method for a large-area array CMOS image sensor according to an embodiment of the present invention;

FIG. 2 is a diagram of an image I according to an embodiment of the present invention_maxThe column division result is illustrated by 10 pixels × 10 pixels, k — 3.

Detailed Description

The technical solution of the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments.

Due to the influence of factors such as manufacturing process and the like, the photo-generated charge diffusion length, the photon absorption depth and the size of a photosensitive surface of each pixel photosensitive element are different, and each column of the CMOS image sensor is provided with an amplifier and an AD converter, so that thousands of amplifiers and AD converters cannot be guaranteed to have completely the same parameters in the manufacturing process. Therefore, when the same light intensity is input, the output signal of each pixel is different, so that the output image may have column stripes, which seriously affects the imaging quality.

The invention provides a non-uniformity correction method of a large-area array CMOS image sensor based on a polynomial fitting correction method by combining the structural characteristics of the CMOS image sensor, wherein in one embodiment, as shown in figure 1, the method comprises the following steps:

the method comprises the following steps: under the uniform illumination of the integrating sphere, original image acquisition is carried out on the CMOS image sensor to obtain N images I with different irradiances₁,I₂,…,I_N；

Step two: for the collected image I₁,I₂,…,I_NPerforming non-uniformity (PRNU) calculation to obtain image I corresponding to non-uniformity maximum value_max；

Alternatively, the image I is represented by₁,I₂,…,I_NFor each image of the series:

in the formula: PRNU is the image non-uniformity, Avg is the image gray level mean, V_ijM 'and N' are the number of pixel rows and pixel columns, respectively, of the image, which are the gray scale values at pixel (i, j).

Step three: from the resulting image I_maxObtaining an image I_maxDetermining the average value of the gray scales of each column of pixels, and expressing the average value of the maximum gray scales as T_maxThe minimum gray mean value is represented as T_min；

Step four: will [ T_min,T_max]The interval is divided into k sub-intervals on average;

step five: obtaining an image I_maxThe number of pixel rows m and the number of pixel columns n, and k m × n zero matrixes a are established according to the number of pixel rows m and the number of pixel columns n₁,a₂,…,a_i,…,a_kWherein a is_iI is more than or equal to 1 and less than or equal to k and is a zero matrix corresponding to the ith subinterval;

step six: according to image I_maxImage I is obtained by averaging the gray levels of each column of pixels_maxColumn mean matrix A of₁Column mean matrix A₁A matrix | x of 1 × n₁,x₂,…,x_j,…,x_nL, where x_jAs an image I_maxJ is more than or equal to 1 and less than or equal to n of the gray level mean value of the jth row of pixels; original image I_maxIs an m x n matrix, and after the gray average value of each row of pixels is obtained, a new matrix of 1 x n is formed, and a row average value matrix A is obtained₁；

Step seven: according to the column mean matrix A₁The subinterval corresponding to each element of (1), image I_maxIs assigned to the corresponding zero matrix a₁,a₂,…,a_kIn the corresponding column position, realize the image I_maxTo obtain a corresponding matrix beta₁,β₂,…,β_k；

The specific process is as follows: column mean matrix A₁Is a 1 × n matrix | x₁,x₂,…,x_j,…,x_nAccording to the column mean matrix A₁The size of each element in the matrix may determine the sub-interval corresponding to each element, such as element x in the matrix₁Is in the zero matrix a₁Within the sub-interval, the image I_maxIs assigned to the zero matrix a₁In the first column of (1), and so on, image I_maxAll columns of (a) are assigned to the corresponding zero matrix a₁,a₂,…,a_kIn (2), a corresponding matrix beta is obtained₁,β₂,…,β_k。

Step eight: according to the step seven pairs of images I_maxThe column division result can know that each column of the image gray level matrix is specifically divided into the zero matrix, the columns of the image are correspondingly grouped into the zero matrix of the corresponding gray level interval according to the division result, and by analogy, after the image is divided into the same zero matrix, each zero matrix is correspondingly provided with N new matrixes, such as beta₁₁Is I₁Gray matrix division into zero matrix a₁The new matrix, beta, obtained thereafter₁₁，β₂₁，β₃₁，…β_N1Is I₁,I₂,…,I_NIn the zero matrix a₁The data of (c); by analogy, k groups of matrix data are obtained:

step nine: respectively solving a gray level mean value for each non-zero matrix element in the k groups of matrix data to obtain k gray level mean value matrixes:

in the gray-scale mean matrix, avg₁₁Representing the non-zero matrix element beta₁₁And (5) solving the gray level average value, and so on.

Step ten: calculating the mean value of each gray mean value matrix to obtain an image I₁,I₂,…,I_NCorresponding image gray average value avg₁,avg₂,…,avg_NWherein avg is₁Is represented by₁And (5) averaging the gray levels of all pixels of the gray level image, and so on.

Step eleven: taking the gray level mean value obtained in the ninth step as a vertical coordinate, taking the corresponding image gray level mean value obtained in the tenth step as a horizontal coordinate, and establishing a data relation to obtain k groups of data for curve fitting:

step twelve: introducing a least square method, and respectively performing curve fitting on the k groups of data according to the residual square sum minimum principle to obtain k curves f containing correction parameters_i(x)，(i＝1,2,…k)；

Optionally, curve f_i(x) For a first order function y ═ a_ix+b_iWherein a is_iAnd b_iAre calibration parameters.

Step thirteen: will matrix beta₁,β₂,…,β_kThe elements with the internal non-zero value are assigned to be 1 to obtain a matrix theta₁,θ₂,…,θ_k。

Fourteen steps: according to curve f_i(x) And matrix theta₁,θ₂,…,θ_kEstablishing a correction model for carrying out non-uniformity correction on the CMOS image sensor:

wherein, I_preFor the input image of the CMOS image sensor before correction, I_afterAnd outputting an image for the corrected CMOS image sensor.

The non-uniformity correction method for the large-area array CMOS image sensor provided by the embodiment has the following beneficial effects:

(1) according to the structural characteristics of the CMOS image sensor, grouping the large-area array CMOS image columns, and solving parameters by utilizing the grouping;

(2) a least square method is introduced, so that the correction precision is increased;

(3) the obtained parameters are far smaller than the parameter quantity of the traditional scaling method, so that hardware resources are saved, and the system power consumption is reduced;

(4) the algorithm has simple structure, small calculation amount and easy hardware realization.

The technical solution of the present invention will be described in detail below with specific examples.

Step S1: collecting a plurality of images under uniform illumination of integrating spheres with different irradiances to obtain 10 images I with different irradiances₁,I₂,…,I₁₀；

Step S2: the acquired image I is processed by₁,I₂,…,I₁₀Performing non-uniformity (PRNU) calculation to obtain image I corresponding to PRNU maximum value_max；

In the formula: PRNU is the image non-uniformity, Avg is the image gray level mean, V_ijIs a pixel (i, j)The gray scale values M 'and N' are the number of pixel rows and the number of pixel columns of the image, respectively.

Step S3: from the resulting image I_maxObtaining an image I_maxDetermining the average value of the gray scale of each column of pixels to determine the maximum average value of the gray scale T_maxAnd minimum mean value of gray scale T_min；

Step S4: will [ T_min,T_max]The interval is divided into k subintervals with the period of t on average:

[T_min,T_min+t],[T_min+t,T_min+2t],…,[T_min+(k-1)t,T_max]

step S5: with image I_maxFor a 10pixel × 10pixel gray matrix (m is 10, n is 10), k is 3 for example, 3 10 × 10 zero matrices are established, each being a₁,a₂,a₃；

Step S6: according to image I_maxImage I is obtained by averaging the gray levels of each column of pixels_maxColumn mean matrix A of₁Column mean matrix A₁Is a matrix | x of 1 × 10₁,x₂,…,x₁₀|；

Step S7: according to the column mean matrix A₁The subinterval corresponding to each element of (1), image I_maxIs assigned to the corresponding zero matrix a₁,a₂,a₃In the corresponding column position, realize the image I_maxTo obtain a corresponding matrix beta₁,β₂,β₃(ii) a As shown in fig. 2, the corresponding column mean matrix a₁By dividing the image I_maxEach column of to a₁,a₂,a₃In (b) to obtain beta₁,β₂,β₃；

Step S8: for image I according to step S7_maxFor the image I respectively₁,I₂,…,I₁₀Grouping to obtain 3 groups of matrix data: beta is a_1,1,β_2,1,…,β_10,1，β_1,2,β_2,2,…,β_10,2，β_1,3,β_2,3,…,β_10,3；

Step S9: respectively solving a gray average value for each non-zero matrix element in the 3 groups of matrix data to obtain 3 gray average value matrixes:

with one set of matrix data beta_1,1,β_2,1,…,β_10,1For example, the gray level mean value of each non-zero matrix element is calculated respectively to obtain 1 gray level mean value matrix:

avg_1,1,avg_2,1,…,avg_10,1；

step S10: according to the gray level mean matrix avg_1,1,avg_2,1,…,avg_10,1Calculating the mean value to obtain an image I₁,I₂,…,I₁₀Corresponding image gray average value avg₁,avg₂,…,avg₁₀；

Step S11: according to (avg)₁,avg_1,1),(avg₂,avg_2,1),(avg₃,avg_3,1),…,(avg₁₀,avg_10,1) Drawing a curve;

step S12: introducing a least square method, and fitting a curve according to a minimum principle of residual square sum;

curve fitting is performed taking the first order function y as an example of ax + b:

computing the sum of squares of the residuals

Making a and b reasonably valued to minimize the value of M;

by calculating the partial derivatives of a and b:

obtaining correction parameters a and b by the above formula;

for the rest 2 sets of matrix data beta_1,2,β_2,2,…,β_10,2，β_1,3,β_2,3,…,β_10,3The calculations of steps S9 to S12 are performed, respectively, to obtain a total of 3 sets of correction parameters, a₁,b₁,a₂,b₂,a₃,b₃；

Step S13: will matrix beta₁,β₂,β₃The elements with the internal non-zero value are assigned to be 1 to obtain a matrix theta₁,θ₂,θ₃；

Step S14: according to curve f_i(x) And matrix theta₁,θ₂,θ₃Establishing a correction model for carrying out non-uniformity correction on the CMOS image sensor:

wherein, I_preFor the input image of the CMOS image sensor before correction, I_afterAnd outputting an image for the corrected CMOS image sensor.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (3)

1. A non-uniformity correction method of a large-area array CMOS image sensor is characterized by comprising the following steps:

the method comprises the following steps: under the uniform illumination of the integrating sphere, original image acquisition is carried out on the CMOS image sensor to obtain N images I with different irradiances₁,I₂,…,I_N；

Step two: for the collected image I₁,I₂,…,I_NCalculating the non-uniformity to obtain an image I corresponding to the maximum value of the non-uniformity_max；

Step three: finding an image I_maxDetermining the average value of the gray scale of each column of pixels to determine the maximum average value of the gray scale T_maxAnd minimum mean value of gray scale T_min；

Step four: will [ T_min,T_max]The interval is divided into k sub-intervals on average;

step five: according to image I_maxThe number of pixel rows m and the number of pixel columns n establish k m x n zero matrices a₁,a₂,…,a_i,…,a_kWherein a is_iI is more than or equal to 1 and less than or equal to k and is a zero matrix corresponding to the ith subinterval;

step six: according to image I_maxImage I is obtained by averaging the gray levels of each column of pixels_maxColumn mean matrix A of₁Column mean matrix A₁A matrix | x of 1 × n₁,x₂,…,x_j,…,x_nL, where x_jAs an image I_maxJ is more than or equal to 1 and less than or equal to n of the gray level mean value of the jth row of pixels;

step seven: according to the column mean matrix A₁The subinterval corresponding to each element of (1), image I_maxAre respectively assigned to the corresponding zero matrix a₁,a₂,…,a_i,…,a_kIn the corresponding column position, the corresponding matrix beta is obtained₁,β₂,…,β_k；

Step eight: according to the step seven pairs of images I_maxAs a result of column division of (1), respectively for the imageI₁,I₂,…,I_NGrouping to obtain k groups of matrix data:

step nine: respectively solving a gray level mean value for each non-zero matrix element in the k groups of matrix data to obtain k gray level mean value matrixes:

step ten: calculating the mean value of each gray mean value matrix to obtain an image I₁,I₂,…,I_NCorresponding image gray average value avg₁,avg₂,…,avg_N；

Step eleven: taking the gray level mean value obtained in the ninth step as a vertical coordinate, taking the corresponding image gray level mean value obtained in the tenth step as a horizontal coordinate, and establishing a data relation to obtain k groups of data for curve fitting:

step twelve: introducing a least square method, and respectively performing curve fitting on the k groups of data according to the residual square sum minimum principle to obtain k curves f containing correction parameters_i(x)，i＝1,2,…,k；

Step thirteen: will matrix beta₁,β₂,…,β_kThe elements with the internal non-zero value are assigned to be 1 to obtain a matrix theta₁,θ₂,…,θ_k；

Fourteen steps: according to curve f_i(x) And matrix theta₁,θ₂,…,θ_kEstablishing a correction model for carrying out non-uniformity correction on the CMOS image sensor:

wherein, I_preFor the input image of the CMOS image sensor before correction, I_afterAnd outputting an image for the corrected CMOS image sensor.

2. The method of claim 1, wherein the non-uniformity of the image I is corrected by₁,I₂,…,I_NFor each image of the series:

in the formula: PRNU is the image non-uniformity, Avg is the image gray level mean, V_ijM 'and N' are the number of pixel rows and pixel columns, respectively, of the image, which are the gray scale values at pixel (i, j).

3. The non-uniformity correction method for large-area CMOS image sensor according to claim 1 or 2,

curve f_i(x) For a first order function y ═ a_ix+b_i。

Images