CN113284196B - Camera distortion pixel-by-pixel calibration method - Google Patents
Camera distortion pixel-by-pixel calibration method Download PDFInfo
- Publication number
- CN113284196B CN113284196B CN202110819872.0A CN202110819872A CN113284196B CN 113284196 B CN113284196 B CN 113284196B CN 202110819872 A CN202110819872 A CN 202110819872A CN 113284196 B CN113284196 B CN 113284196B
- Authority
- CN
- China
- Prior art keywords
- pixel
- picture
- calibration plate
- camera
- mask
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013507 mapping Methods 0.000 claims abstract description 61
- 239000011159 matrix material Substances 0.000 claims abstract description 34
- 238000004364 calculation method Methods 0.000 claims abstract description 21
- 238000005457 optimization Methods 0.000 claims abstract description 20
- 238000006073 displacement reaction Methods 0.000 claims description 21
- 230000009466 transformation Effects 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 239000004576 sand Substances 0.000 claims description 4
- 241000208340 Araliaceae Species 0.000 claims description 3
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 claims description 3
- 235000003140 Panax quinquefolius Nutrition 0.000 claims description 3
- 238000003491 array Methods 0.000 claims description 3
- 235000008434 ginseng Nutrition 0.000 claims description 3
- 230000002068 genetic effect Effects 0.000 claims description 2
- 239000002245 particle Substances 0.000 claims description 2
- 238000002922 simulated annealing Methods 0.000 claims description 2
- 239000000654 additive Substances 0.000 claims 1
- 230000000996 additive effect Effects 0.000 claims 1
- 238000001514 detection method Methods 0.000 description 10
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 241000226585 Antennaria plantaginifolia Species 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for calibrating distortion of a camera pixel by pixel, which comprises the following steps: performing digital image correlation calculation on the shot calibration plate picture, the standard calibration plate picture and the mask thereof, extracting control points according to the calculation result, and performing initial estimation on internal parameters and external parameters of the camera; projecting a virtual calibration plate and a virtual mask plate on an image, performing digital image correlation calculation on a projection result and a calibration plate picture, taking the mean square error of non-zero values in a mapping matrix obtained by calculation as a target function, and taking internal parameters and external parameters of a camera as optimization variables to perform optimization calculation; and projecting the virtual calibration plate and the virtual mask plate onto the image by using the optimal values of the internal parameter and the external parameter of the camera, performing digital image correlation calculation on the projection result and the calibration plate image, and taking the average value of nonzero values of each element in each group of mapping matrixes obtained by calculation as the distortion of each point to obtain a distortion matrix. The method can obviously improve the accuracy of camera distortion and internal parameter calibration.
Description
Technical Field
The invention belongs to the technical field of computer vision, and relates to a method for calibrating distortion of a camera pixel by pixel.
Background
In binocular vision measurement and three-dimensional reconstruction technology, the accuracy of camera calibration has a decisive influence on the accuracy of measurement, matching and reconstruction. At present, a pinhole imaging model is widely used for calibrating camera internal parameters, and radial and tangential distortion models or other distortion models are used for calibrating distortion.
The distortion of a video camera is caused by adding a lens in front of the camera, the distortion caused by the shape of the lens is called radial distortion, and light rays are more curved far away from the center of the lens than near the center of the lens, and the distortion is mainly divided into two categories: barrel distortion and pincushion distortion. As shown in fig. 1, from left to right are the normal image, barrel distortion and pincushion distortion, respectively.
The radial distortion is described by equation (2):
wherein (x, y) is the coordinate of a point in the normalized image plane, (x)r,yr) To normalize the coordinates of corresponding points, k, in the image plane after distortion correction1、k2、k3For the distortion coefficient, r is the distance of point (x, y) from the center point of the normalized image plane, as shown in equation (1).
The distortion caused by the lens and image plane not being perfectly parallel during mechanical assembly is called tangential distortion, and is shown by equation (3):
wherein p is1、p2Is the distortion coefficient.
The values of the radial and tangential distortion coefficients are typically calculated according to the Zhang-friend scaling method.
Because the distortion model commonly used at present is an approximation of the true distortion relationship of the camera, the accuracy that can be achieved by using the model for calibration is very limited. How to calibrate the camera more accurately is an urgent problem to be solved.
Disclosure of Invention
The invention aims to provide a camera distortion calibration method with smaller error than that of a common method aiming at the defects of the prior art, which can calibrate the camera distortion pixel by pixel, and the calibration result is given in the form of two mapping tables.
The technical scheme adopted by the invention is as follows:
an image used for calibration is prepared, and a specific calibration plate is made based on the image. The speckle pattern is generated by equation (4) as the image used for calibration. Where I (x, y) represents the value at pixel coordinate (x, y) in the speckle pattern, n represents the number of speckles, k represents the speckle number, D represents the radius of the speckle (in pixels), (x)k,yk) Is a random number, indicates the location of the kth speckle,
also random number, represents the peak magnitude of the k speckle. The speckle pattern was set to dimensions 4000 x 4000 pixels, n =15000, D = 60. The parameter setting is not a unique scheme and can be increased or decreased, the principle is that the areas of black and white areas of the speckle pattern are approximately equal, and after the speckle pattern is printed on the surface of the calibration plate and shot by a camera (the shooting method is shown later and the picture part required by camera calibration is shot), the details of the speckle pattern are clear and visible. A black border is left around the speckle region, as shown in fig. 2, the width of which is greater than the radius of the sub-region set in the later described DIC detection step.
And (4) manufacturing a specific calibration board by using the speckle picture generated by the method in the last step. The size of a particular calibration plate is determined by the focal length and field angle of the calibrated camera. The particular calibration board size is selected such that when the calibration board is captured by the camera requiring calibration, the calibration board in the frame occupies approximately 2/3 inches of the entire frame. When a specific calibration plate is manufactured, a high-precision printer can be used for printing speckle pictures and pasting the speckle pictures on a square glass plate with the same size.
A calibration board standard picture (hereinafter referred to as "high pixel standard calibration board picture") having a resolution 2 times greater than the vertical resolution of the camera to be calibrated and a calibration board standard picture (hereinafter referred to as "standard calibration board picture") having a number of pixels equal to 3/4 of the vertical resolution of the camera to be calibrated are prepared. And the high-pixel standard calibration plate picture and the standard calibration plate picture are obtained by down-sampling the speckle picture generated by the method in the last step.
Two mask pictures, namely a mask picture of a high-pixel standard calibration board picture and a mask picture of a standard calibration board picture, are prepared. The mask image was used to mark the region of interest (speckle region) and other regions (border region) in the DIC inspection, the appearance of which is shown in fig. 3. The mask picture consists of regions of logic 0 and logic 1. The mask picture size of the high-pixel standard calibration board picture is the same as that of the high-pixel standard calibration board picture, the area range of logic 1 is the same as that of the speckle area in the high-pixel standard calibration board picture, and the area range of logic 0 is the same as that of the black frame area in the high-pixel standard calibration board picture. The mask picture size of the standard calibration board picture is the same as that of the standard calibration board picture, the area range of logic 1 is the same as that of the speckle area in the standard calibration board picture, and the area range of logic 0 is the same as that of the black frame area in the standard calibration board picture.
First, a calibration board is photographed by a camera to be calibrated to obtain n pictures (hereinafter referred to as "photographed calibration board pictures"). The number n of the pictures of the calibration plate is not less than 3 and is not uniquely determined, and n can be 15-30 generally to achieve good calibration effect. When shooting, the calibration board in the picture occupies about 2/3 area of the whole picture. The posture of the calibration plate is required during shooting. Taking the plane of the calibration plate opposite to the camera lens as a reference, as shown in fig. 4, in a cartesian coordinate system, the rotation angle α of the calibration plate in the plane x-y is as small as possible, the rotation angle β in the y-z plane and the rotation angle γ in the z-x plane can be arbitrarily set, and usually, the rotation angle β and γ is 0 to 30 degrees, which may cause the failure of the DIC detection if the angle is too large. Another requirement for the pose of the calibration plate when taken is to ensure that all positions of the camera view are covered by the calibration plate speckle pattern in at least one picture.
Taking a mask of a standard calibration board picture as an interested area, taking the standard calibration board picture as a reference picture, taking a shot calibration board picture as a current picture, and finding the displacement of a corresponding pixel in the shot calibration board picture relative to the pixel for each pixel in the standard calibration board picture by adopting a Digital Image Correlation (DIC) (hereinafter referred to as a detection DIC) process. The definition of "corresponding pixel" is if for a pixel (u) in picture a that is located at the center of a feature1,v1) If the same feature can be found in the picture B, the pixel (u) at the center position is called2,v2) Is a pixel (u) in picture A1,v1) The corresponding pixel in picture B. And respectively storing the displacements of all pixels in the x direction and the y direction obtained in the DIC method into two mapping matrixes. The method is applied to all n photographed calibration plate pictures to obtain n groups of 2n mapping matrices (hereinafter referred to as "standard picture mapping matrices").
Stored in each set of mapping matrices is each pixel in each captured calibration plate picture relative to its corresponding pixel (x) in the standard calibration plate picturer,yr) Displacement of (2)
Therefore, the corresponding pixel of each pixel on the k-th shooting calibration board picture on the standard calibration board picture can be calculated
As follows.
And extracting a point array of s rows and s columns from the mapping matrix to ensure that s is more than or equal to 2. The extraction mode of the point array is to take s rows and s columns of pixel points at equal intervals in a speckle area of a standard calibration plate picture, and ensure that s is more than or equal to 2. And according to the standard picture mapping matrix, taking the corresponding pixel coordinates of the pixel points on each shooting calibration board picture, and storing the n multiplied by s pixel coordinates. Correspondingly, the s x s point arrays at the same intervals on the plane of the calibration plate are taken, and the coordinates of the points in the world coordinate system are saved. As shown in fig. 5, the black dots in the figure are 7 × 7 control dot arrays on the calibration plate, and the number of rows and columns of the actual array is not fixed. The position of the array of dots taken on the calibration plate in the speckle pattern on the calibration plate is the same as the position of the array of dots taken on the standard calibration plate picture in the speckle pattern on the standard calibration plate picture. The saved pixel coordinates on the photographed calibration plate picture and world coordinates on the calibration plate are hereinafter referred to as "control points".
Using Zhangzhengyou scaling method to calibrate internal parameters K and n groups of external parameters R of camerak、tk(each picture corresponds to the rotation and translation of the calibration plate relative to the camera).
The high pixel standard calibration plate picture and its mask are projected from the world coordinate system to the pixel coordinate system according to the initial estimation of the camera internal and external parameters, the projection process is referred to formula (9) -formula (12) and its description. Because there are n groups of RkAnd tkFor each group RkAnd tkObtaining 1 projected high-pixel standard calibration board picture (hereinafter referred to as "high-pixel standard calibration board projection picture") and 1 projected mask (hereinafter referred to as "mask projection picture"), and sharing n projected high-pixel standard calibration board projection pictures and n mask projection pictures. The calibration plate posture in the projection picture of the kth high pixel standard calibration plate is almost the same as the calibration plate posture in the picture of the kth shooting calibration plate, and the difference is the initial estimation error of the internal and external parameters and the distortion quantity to be calibrated.
And carrying out DIC calculation for n times on all the n shot calibration plate pictures, the n high-pixel standard calibration plate projection pictures and the n mask projection pictures. Respectively storing the displacements of all pixels in the x direction and the y direction in each detection result into two mapping matrixes
And
thus, n sets of 2n mapping matrices (hereinafter referred to as "projection picture mapping matrices") are obtained.
Taking the mean square error of non-zero values in the 2n mapping matrixes as an objective function, and internal and external parameters K and R of the camerak、tkAs optimization variables, initial estimates of the camera internal and external parameters are used as initial values, and are calculated by an optimization method, as shown in formula (5) -formula (12). And (3) finding out the internal and external parameter values of the camera, which enable the formula (5) to reach the minimum value, namely the optimal value obtained by calculation of the optimization method.
In the formula (5) to the formula (13), i and j represent the row and column numbers of the mapping matrix, and are the pixel coordinates in the projection picture of the high pixel standard calibration plate. k represents the serial number of the kth group of mapping matrixes, and is also the picture serial number of the high-pixel standard calibration plate projection picture, the mask projection picture and the shooting calibration plate picture. i.e. ik、jkRepresenting the pixel coordinates in the kth set of high pixel standard calibration plate projection pictures. Equation (5) -equation (7), equation (5) represents the mean square error of the non-zero values in the 2n mapping matrices,
representing the pixel (i) in the k shooting calibration board picture for the value of the ith row and the jth column in the x-direction mapping matrix in the k set of mapping matrixesc,jc) Relative to its x-direction displacement of the corresponding pixel (i, j) in the k-th high pixel standard calibration plate projection picture,
the value of the ith row and the jth column in the y-direction mapping matrix in the kth group of mapping matrixes represents the pixel (i) in the kth shooting calibration board picturec,jc) Relative to its y-direction displacement of the corresponding pixel (i, j) in the kth high pixel standard calibration plate projection picture. Formula (6), i.e.
Means that all m's having a non-zero value in the ith row and jth columni,j,xIn the x-direction mapping matrix
Average value of (a). m isi,j,xRepresenting the number of matrices in all x-direction mapping matrices that have a non-zero value in row i and column j. Formula (7), i.e.
Means that all m's having a non-zero value in the ith row and jth columni,j,yIn y-direction mapping matrix
Average value of (a). m isi,j,yIndicating the number of matrices in all y-direction mapping matrices that have a non-zero value in the ith row and jth column.
F in formula (8)dicRepresenting DIC calculation with the result of 2n mapping matrix elements in n groups used in formula (5) to formula (7)
、
. In the formula (8), i and j represent pixel coordinates in the projection picture of the high pixel standard calibration board. k represents the serial number of the high-pixel standard calibration plate projection picture, the mask projection picture and the picture of the shooting calibration plate picture,
representing the kth high pixel standard calibration board projection picture,
indicating that the kth shot calibration plate picture,
representing the k mask projection picture. High pixel standard calibration board projection picture
And mask projection picture
Is obtained from the projection process, i.e., formula (10) -formula (13). Before the projection process, the panel picture needs to be calibrated according to the high pixel standard
Defining a virtual calibration board B in the world coordinate systemSAnd calibrating the mask of the board picture according to the high pixel standard
Defining a virtual mask B in the world coordinate systemm. The dimension of the calibration plate is set to be D multiplied by D, and the pixel number of the high-pixel standard calibration plate picture and the mask picture thereof is set to be W multiplied by W. Virtual calibration board BSAnd virtual mask BmAll are described by a point cloud consisting of W × W coordinate points, and the distance between the coordinate points is D/W. Equation (9) represents the virtual calibration board BSValue B of the middle coordinate point (u, v,0)S(u, v,0) and high pixel standard scaling panel picture pixels
Value of (A)
Same, for the virtual mask BmValue B of its coordinate point (u, v,0)m(u, v,0) and high pixel standard calibration plate picture pixel in mask
Value of (A)
The same is true.
In the formula (10)
Image representing pixel (i, j) in the kth high pixel standard calibration plate projection pictureElemental value, BS(u, v,0) represents the value of the midpoint (u, v,0) of the virtual calibration plate.
Representing the pixel value, B, of pixel (i, j) in the k-th mask projection picturem(u, v,0) represents the value of the point (u, v,0) in the virtual mask. Formula (10) represents the mapping between the high-pixel standard calibration plate projection picture and the virtual calibration plate and between the mask projection picture and the virtual mask plate, wherein the mapping relationship is obtained by formula (11) -formula (13), i.e. the corresponding relationship between the pixel (i, j) in the kth high-pixel standard calibration plate projection picture and the midpoint (u, v,0) of the virtual calibration plate, and the corresponding relationship between the pixel (i, j) in the kth mask projection picture and the midpoint (u, v,0) of the virtual mask plate.
Knowing the corresponding relationship between the pixel (i, j) in the projection picture of the kth high pixel standard calibration plate and the midpoint (u, v,0) of the virtual calibration plate, the pixel in the picture of the high pixel standard calibration plate can be calibrated according to the formula (9) -the formula (10)
Value of (A)
Obtaining the pixel value of the pixel (i, j) in the projection picture of the kth high pixel standard calibration board
. Knowing the corresponding relationship between the pixel (i, j) in the k-th mask projection picture and the midpoint (u, v,0) of the virtual mask plate, the pixel in the mask of the plate picture can be calibrated by the high pixel standard according to the formula (9) -formula (10)
Value of (A)
Obtaining the pixel value of the pixel (i, j) in the k mask projection picture
. Corresponding in general to (u, v,0)
Is a sub-pixel coordinate, i.e. a coordinate whose coordinate value is a decimal number, so that the k-th high pixel standard calibration board projects the pixel value of the pixel (i, j) in the picture
Scaling pixels in panel pictures by high pixel standards
The pixel values of the adjacent four integer coordinate pixels are obtained by bilinear interpolation, and the pixel value of the pixel (i, j) in the k-th mask projection picture
Scaling pixels in a mask for a panel picture by a high pixel standard
And carrying out bilinear interpolation on the pixel values of the adjacent four integer coordinate pixels.
Formula (11) -formula (13) represents the coordinate transformation relationship between the pixel (i, j) in the projection picture of the k-th high pixel standard calibration plate and the midpoint (u, v,0) in the virtual calibration plate. Equation (11) represents the transformation from the image coordinate system to the pixel coordinate system, (i)k,jk) Representing the pixel coordinates in the projection picture of the kth virtual calibration board, K representing the camera internal parameters,
indicating point (i)k,jk) Coordinates in the image coordinate system. Equation (12) represents the transformation from the camera coordinate system to the image coordinate system,
coordinates (x) representing the camera coordinate systemk,yk,zk) Is normalized to obtainAnd the coordinates of the image plane, namely the coordinates of the image coordinate system. Equation (13) represents the transformation from the world coordinate system to the camera coordinate system. Rk、tkRepresents the kth group of external parameters. (u, v,0) represents the coordinates of a point on the virtual calibration plate (or virtual mask). Because the virtual calibration plate (or virtual mask) plane coincides with the x-y plane of the world coordinate system, the third dimensional coordinate value of a point on the virtual calibration plate (or virtual mask) plane is 0. (x)k,yk,zk) Denotes coordinates of (u, v,0) in the camera coordinate system.
Finding K and Rk、tkAnd (3) projecting the virtual calibration plate and the virtual mask plate onto the image according to the optimal values by using a formula (10) -a formula (13), and performing DIC calculation as shown in a formula (8) to obtain n groups of projection image mapping matrixes. According to formula (6) and formula (7), taking m of each pixel point (i, j) in the n groups of projection picture mapping matrixesi,j,xAverage of non-0 shifts in x-direction
And mi,j,yAverage of non-0 shifts in y-direction
I.e. representing the calculated pixel-by-pixel distortion quantities, and arranging them in pixel order to obtain the distortion matrix.
The elements in the distortion matrix represent the displacement of each pixel in the distorted picture relative to the corresponding pixel in the undistorted picture. When distortion correction is carried out on a newly shot picture, according to a distortion matrix, the corresponding position of each pixel point of the corrected picture in the distorted picture is sequentially searched, because the position coordinate is generally decimal, and bilinear interpolation of pixel values of four integer coordinate pixels adjacent to the position in the distorted picture is taken as the value of the pixel point in the corrected picture.
In the invention, digital image correlation calculation is included in an optimization calculation process, the digital image correlation algorithm can be a FA-GN (Forward adaptive Gauss-Newton) method, an IC-GN (Inverse synthetic Gauss-Newton) method or a variant thereof, and the optimization method comprises but is not limited to a mathematical optimization method (a gradient method, a Gauss-Newton method, a Levenberg-Marquardt method, a linear programming method and the like) or an evolutionary calculation method (a particle swarm algorithm, an ant colony algorithm, a genetic algorithm, a simulated annealing algorithm and the like) or a combination of the two methods; when the Levenberg-Marquardt method is adopted, the optimization calculation effect is better.
The invention has the beneficial effects that:
the method applies a digital image correlation method to an optimization calculation process, takes the mean square error of the difference between each pixel point in a plurality of shot pictures and the ideal pixel coordinate obtained by the estimation calculation according to the current internal and external parameters as an objective function, breaks away from the limitation of a conventional distortion model, calibrates the distortion of a video camera point by point, and can completely describe the distortion of the camera by a calibration result instead of an approximate description, thereby effectively improving the calibration precision and further effectively improving the accuracy of visual measurement and three-dimensional reconstruction in practical application. Through inspection, the reprojection error of the method reaches 0.0786, which is less than 0.1085 of the reprojection error of Zhangyingyou calibration method.
Drawings
FIG. 1 is a schematic illustration of radial distortion;
FIG. 2 is a schematic illustration of a standard calibration plate picture;
FIG. 3 is a schematic illustration of a mask for a standard calibration plate picture;
FIG. 4 is a schematic view of the direction of rotation of the calibration plate when a picture of the calibration plate is taken;
FIG. 5 is a schematic diagram of a control point array for calibrating speckle regions of a plate;
FIG. 6 is a schematic flow diagram of the method of the present invention.
Detailed Description
Examples
Fig. 6 is a schematic flow chart of the method of the present invention, and the specific method is as follows:
an image used for calibration is prepared, and a calibration plate is manufactured according to the image. The speckle pattern is generated by equation (4) as the image used for calibration. The speckle pattern was set to dimensions 4000 x 4000 pixels, n =15000, D = 60. The parameter setting is not the only scheme and can be increased or decreased, the principle is that the areas of black and white areas of the speckle pattern are approximately equal, and after the speckle pattern is printed on the surface of the calibration plate and shot by a camera, the details of the speckle pattern are clear and visible. A black edge of 500 pixels width is left around the speckle area as shown in fig. 2.
The size of the calibration plate object is determined by the focal length and the field angle of the calibrated camera. The size of the calibration board is selected such that when the calibration board is photographed by a camera requiring calibration, the calibration board in the picture occupies about 2/3 of the entire picture. Calibration plates were made with dimensions 60mm by 60 mm. When the calibration plate is manufactured, a high-precision printer is used for printing the speckle pattern with the frame generated in the previous step and pasting the speckle pattern on a square glass plate with the same size.
Preparing a high-precision standard calibration plate picture with the pixel number of 2000 multiplied by 2000 and the texture identical to the real object of the calibration plate, wherein the frame width of the high-precision standard calibration plate picture is 200 pixels, and the speckle area size is 1600 multiplied by 1600 pixels. And preparing a standard calibration plate picture with the pixel number of 1000 multiplied by 1000 and the texture identical to the calibration plate real object, wherein the frame width of the standard calibration plate picture is 100 pixels, and the speckle area size is 800 multiplied by 800 pixels.
A high pixel standard calibration board picture and a mask of the standard calibration board picture are prepared as shown in fig. 3. The number of mask pixels of the high-pixel standard calibration board picture is 2000 multiplied by 2000, the area with the edge width of 200 pixels is logic 0, the other areas are logic 1, the number of mask pixels of the standard calibration board picture is 1000 multiplied by 1000, the area with the edge width of 100 pixels is logic 0, and the other areas are logic 1.
The calibration board was photographed using a camera having 1920 × 1080 pixels, and 20 photographs of the calibration board were obtained. The number n of the pictures of the calibration plate is not uniquely determined, n is more than or equal to 3, and n can be 15-30 generally. When shooting, the calibration board in the picture occupies about 2/3 area of the whole picture. The posture of the calibration plate is required during shooting. Taking the plane of the calibration plate opposite to the camera lens as a reference, as shown in fig. 4, in a cartesian coordinate system, the rotation angle α of the calibration plate in the plane x-y is as small as possible, the rotation angle β in the y-z plane and the rotation angle γ in the z-x plane can be arbitrarily set, and usually, the rotation angle β and γ is 0 to 30 degrees, which may cause the failure of the DIC detection if the angle is too large. Another requirement for the pose of the calibration plate when taken is to ensure that all positions of the camera view are covered by the calibration plate speckle pattern in at least one picture.
And performing DIC detection for 20 times on the 20 taken calibration plate pictures, the standard calibration plate pictures and the mask pictures thereof. Setting parameters for DIC detection, setting the radius of a subarea to be 40, setting the interval of the subareas to be 0, setting the convergence precision to be 1e-6, and setting the maximum iteration number to be 50, wherein the parameter setting is not a unique scheme and can be adjusted according to actual conditions. DIC detection is carried out on each pixel in the standard calibration plate picture, and the displacement of the corresponding pixel in the shooting calibration plate picture relative to the pixel is found. The displacements of these pixels in the x-direction and y-direction are stored as two mapping matrices, respectively. The method is applied to all 20 taken calibration plate pictures to obtain 20 groups of 40 mapping matrixes.
And the displacement of each pixel in each shooting calibration plate picture relative to the corresponding pixel in the standard calibration plate picture is stored in each group of mapping matrixes, so that the coordinates of the corresponding pixel of each pixel in the kth shooting calibration plate picture on the standard calibration plate picture can be calculated, as shown in formula (3) and formula (4).
A dot array of 7 rows and 7 columns is extracted in the mapping matrix. The extraction method of the point array is to extract pixel points at equal intervals in a speckle area of a standard calibration plate picture, extract 7 multiplied by 7 control points in the [200:100:800] th row and [200:100:800] th column of the standard calibration plate picture, calculate the coordinates of the corresponding pixels of the 7 multiplied by 7 control points on the standard calibration plate picture on the kth shooting calibration plate picture according to a mapping matrix, and store the coordinates of the 20 multiplied by 7 pixels. Correspondingly, a 7X 7 point array with the distance of 6mm is taken from the calibration plate real object, and the coordinates of the array in the world coordinate system are saved, and the specific point taking position is shown in figure 5. The saved pixel coordinates on the photographed calibration plate picture and world coordinates on the calibration plate are hereinafter referred to as "control points".
According to the positions of control points in the pixel coordinate system and the world coordinate system, the interior of the camera is calibrated by using a Zhang-Yongyou calibration methodGinseng K and 20 groups of external ginseng Rk、tk(each picture corresponds to the rotation and translation of the calibration plate relative to the camera) to obtain an internal reference matrix K and 20 external reference matrices Rk、tk。
The high pixel standard calibration plate picture and its mask are projected from the world coordinate system to the pixel coordinate system according to the initial estimation of the camera extrinsic reference matrix, the projection process is referred to formula (9) -formula (13) and its description. Because there are 20 groups RkAnd tkFor each group RkAnd tkAnd obtaining 1 projected high-pixel standard calibration board picture and 1 projected mask, and sharing 20 projected high-pixel standard calibration board pictures and 20 projected mask pictures. The calibration plate posture in the projection picture of the kth high pixel standard calibration plate is almost the same as the calibration plate posture in the picture of the kth shooting calibration plate, and the difference is the initial estimation error of the internal and external parameters and the distortion quantity to be calibrated.
DIC detection was performed 20 times on all 20 photographed calibration plate pictures, 20 high pixel standard calibration plate projection pictures and 20 mask projection pictures. And respectively storing the displacements of all pixels in the x direction and the y direction in each detection result into two mapping matrixes, thereby obtaining n groups of 2n mapping matrixes.
Taking the mean square error of non-zero values in the 2n mapping matrices as an objective function, as shown in formula (5) -formula (13), and the camera internal and external parameters K and Rk、tkAs optimization variables, initial estimates of the camera internal and external parameters are used as initial values and calculated by a Levenberg-Marquardt optimization method. And (5) finding out the internal and external parameter values of the camera, which enable the internal and external parameter values to reach the minimum value, namely the optimal value obtained by optimization calculation.
Finding K and Rk、tkAnd (3) obtaining the projections of the high-pixel standard calibration board picture and the mask picture thereof by using a formula (9) to a formula (13) according to the optimal values, and carrying out DIC calculation as shown in a formula (8) to obtain n groups of projection picture mapping matrixes. The calculated pixel-by-pixel distortion is represented by formula (6) and formula (7), and is obtained by arranging them in pixel orderA distortion matrix. The elements in the distortion matrix represent the displacement of each pixel in the distorted picture relative to the corresponding pixel in the undistorted picture. When distortion correction is carried out on a newly shot picture, according to a distortion matrix, the corresponding position of each pixel point of the corrected picture in the distorted picture is sequentially searched, because the position coordinate is generally decimal, and bilinear interpolation of pixel values of four integer coordinate pixels adjacent to the position in the distorted picture is taken as the value of the pixel point in the corrected picture.
Table 1 lists data obtained by repeating calibration experiments ten times using different groups of pictures, each group of 20 pictures, using the camera distortion pixel-by-pixel calibration method and the zhangnyou calibration method of the present invention. In the Zhangyingyou calibration method, calibration plates for feature point extraction include checkerboard, triangles, dots, and speckle calibration plates. In table 1, the first column is the average value of the reprojection errors of different calibration methods, and compared with the results of other control experiments, the average value of the reprojection errors of the camera distortion pixel-by-pixel calibration method of the present invention is the smallest. The last four columns are root mean square errors of the focal length and the main point position estimated value in the calibration result, and represent the stability of parameter estimation. The parameter value difference of each estimation in repeated calibration is smaller and more stable, and the parameter estimation of the method is most stable compared with the results of other control experiments.
TABLE 1 evaluation of calibration errors
Claims (8)
1. A method for calibrating distortion of a camera pixel by pixel is characterized by comprising the following steps:
preparing a specific calibration plate, a high-pixel standard calibration plate picture and a mask thereof, and a standard calibration plate picture and a mask thereof; shooting the calibration plate by using a camera to be calibrated to obtain n pictures of the shooting calibration plate, wherein n is more than or equal to 3;
taking a mask of a standard calibration board picture as an interested area, taking the standard calibration board picture as a reference picture, taking a shot calibration board picture as a current picture, finding the displacement of a corresponding point in the shot calibration board picture relative to each pixel for each pixel in the standard calibration board picture by adopting a digital image correlation method, and respectively storing the displacements of all the pixels in the x direction and the y direction into two mapping matrixes; namely, obtaining n groups of 2n standard picture mapping matrixes for all n photographed calibration plate pictures;
extracting control points, and performing initial estimation on internal parameters and external parameters of the camera by using a Zhangyingyou calibration method; obtaining a group of internal ginseng and n groups of external ginseng;
according to the obtained initial estimation of the internal and external parameters of the camera, projecting the high-pixel standard calibration plate picture and the mask thereof from the world coordinate system to the pixel coordinate system to obtain n standard calibration plate projection pictures and masks thereof;
taking a mask of a projection picture of a standard calibration plate as an interested area, taking the projection picture of the standard calibration plate as a reference picture, taking a picture of the calibration plate as a current picture, finding the displacement of a corresponding point in the picture of the calibration plate relative to a pixel for each pixel in the projection picture of the standard calibration plate by using a digital image correlation method, and respectively storing the displacements of all the pixels in the x direction and the y direction into two mapping matrixes, namely obtaining n groups of 2n projection picture mapping matrixes for all n pictures of the calibration plate;
taking the mean square error of the 2n projection picture mapping matrixes as a target function, taking the internal and external parameters of the camera as optimization variables, taking the initial estimation of the internal and external parameters of the camera as initial values, and calculating by using an optimization method to find the internal and external parameters of the camera, which enable the target function to reach the minimum value, namely an optimal value;
after finding the optimal value, taking respective average values of displacements in x and y directions in the corresponding n groups of projection picture mapping matrixes to obtain a distortion matrix, wherein elements represent the displacement of each pixel in the picture containing distortion relative to the corresponding pixel in the picture after distortion removal;
the specific method for extracting the control points comprises the following steps: extracting a point array of s rows and s columns from the mapping matrix, and ensuring that s is more than or equal to 2; the extraction mode of the point array is to take s rows and s columns of pixel points at equal intervals in a speckle area of a standard calibration plate picture, and ensure that s is more than or equal to 2; according to the standard picture mapping matrix, taking the corresponding pixel coordinates of the pixel points on each shooting calibration board picture, and storing the nxs x s pixel coordinates; correspondingly, the s multiplied by s point arrays at the same interval on the plane where the calibration plate is located are taken, and the coordinates of the points in the world coordinate system are stored; the saved pixel coordinates on the shot calibration plate picture and the world coordinates on the calibration plate are called control points.
2. The method for calibrating distortion of a camera pixel by pixel as claimed in claim 1, wherein the pattern of the specific calibration plate is a speckle pattern, and the high-pixel standard calibration plate picture and the standard calibration plate picture are speckle pictures.
3. The method for calibrating distortion of a camera pixel by pixel as claimed in claim 1, wherein n is 15-30.
4. The method for calibrating distortion of a camera pixel by pixel according to claim 1, wherein the objective function is specifically as follows:
in formula (1) -formula (9), i and j represent row and column numbers of the mapping matrix; k represents the sequence number of the kth group mapping matrix; i.e. ik、jkRepresenting pixel coordinates in a projection picture of a kth group of high pixel standard calibration plate; in the formula (1) -formula (3), formula (1) represents the mean square error of non-zero values in the 2n mapping matrices,
representing the pixel (i) in the k shooting calibration board picture for the value of the ith row and the jth column in the x-direction mapping matrix in the k set of mapping matrixesc,jc) Relative to its x-direction displacement of the corresponding pixel (i, j) in the k-th high pixel standard calibration plate projection picture,
the value of the ith row and the jth column in the y-direction mapping matrix in the kth group of mapping matrixes represents the pixel (i) in the kth shooting calibration board picturec,jc) The y-direction displacement of the corresponding pixel (i, j) in the projection picture of the k-th high pixel standard calibration plate is compared with the y-direction displacement of the corresponding pixel (i, j) in the projection picture of the k-th high pixel standard calibration plate; formula (2), i.e.
Means that all m's having a non-zero value in the ith row and jth columni,j,xIn the x-direction mapping matrix
Average value of (d); m isi,j,xRepresenting the number of matrixes with non-zero values in the ith row and the jth column in all the x-direction mapping matrixes; formula (3), i.e.
Means that all m's having a non-zero value in the ith row and jth columni,j,yIn y-direction mapping matrix
Average value of (d); m isi,j,yRepresenting the number of matrixes with non-zero values in the ith row and the jth column in all y-direction mapping matrixes;
f in formula (4)dicRepresenting DIC calculation with the result of 2n mapping matrix elements in n groups used in formula (1) to formula (3)
In the formula (4)
Representing the kth high pixel standard calibration board projection picture,
indicating that the kth shot calibration plate picture,
representing the k mask projection picture; high pixel standard calibration board projection picture
And maskFilm projection picture
Is obtained by a projection process, namely formula (6) -formula (9); before the projection process, the panel picture needs to be calibrated according to the high pixel standard
Defining a virtual calibration board B in the world coordinate systemSAnd calibrating the mask of the board picture according to the high pixel standard
Defining a virtual mask B in the world coordinate systemm(ii) a Setting the size of a calibration plate as D multiplied by D, and the number of pixels of a high-pixel standard calibration plate picture and a mask picture thereof as W multiplied by W; virtual calibration board BSAnd virtual mask BmAll the points are described by point clouds consisting of W multiplied by W coordinate points, and the distance between the coordinate points is D/W; equation (5) represents the virtual calibration board BSValue B of the middle coordinate point (u, v,0)S(u, v,0) and high pixel standard scaling panel picture pixels
Value of (A)
Same, for the virtual mask BmValue B of its coordinate point (u, v,0)m(u, v,0) and high pixel standard calibration plate picture pixel in mask
Value of (A)
The same;
in formula (6)
Represents the pixel value, B, of pixel (i, j) in the kth high pixel standard calibration plate projection pictureS(u, v,0) represents the value of the virtual calibration plate midpoint (u, v, 0);
representing the pixel value, B, of pixel (i, j) in the k-th mask projection picturem(u, v,0) represents the value of the point (u, v,0) in the virtual mask; equation (6) represents the mapping between the high pixel standard calibration plate projection picture and the virtual calibration plate and between the mask projection picture and the virtual mask plate;
knowing the corresponding relationship between the pixel (i, j) in the projection picture of the kth high pixel standard calibration plate and the midpoint (u, v,0) of the virtual calibration plate, the pixel in the picture of the high pixel standard calibration plate can be calibrated according to the formula (5) -the formula (6)
Value of (A)
Obtaining the pixel value of the pixel (i, j) in the projection picture of the kth high pixel standard calibration board
Knowing the corresponding relationship between the pixel (i, j) in the k-th mask projection picture and the midpoint (u, v,0) of the virtual mask plate, the pixel in the mask of the plate picture can be calibrated by the high pixel standard according to the formula (5) -formula (6)
Value of (A)
Obtaining the pixel value of the pixel (i, j) in the k mask projection picture
Pixel value of pixel (i, j) in kth high pixel standard calibration plate projection picture
Scaling pixels in panel pictures by high pixel standards
The pixel values of the adjacent four integer coordinate pixels are obtained by bilinear interpolation, and the pixel value of the pixel (i, j) in the k-th mask projection picture
Scaling pixels in a mask for a panel picture by a high pixel standard
Carrying out bilinear interpolation on pixel values of four adjacent integer coordinate pixels to obtain the pixel values;
formula (7) -formula (9) represents the coordinate transformation relationship between the pixel (i, j) in the projection picture of the kth high pixel standard calibration plate and the midpoint (u, v,0) of the virtual calibration plate; equation (7) represents the transformation from the image coordinate system to the pixel coordinate system (i)k,jk) Expressing the pixel coordinates in the projection picture of the kth virtual calibration board, K expressing the camera internal parameters,
indicating point (i)k,jk) Coordinates in an image coordinate system; equation (8) represents the transformation from the camera coordinate system to the image coordinate system,
coordinates (x) representing the camera coordinate systemk,yk,zk) Normalizing to obtain an image plane coordinate, namely an image coordinate system coordinate; equation (9) represents a transformation from the world coordinate system to the camera coordinate system; rk、tkRepresents the kth group of external parameters; (u, v,0) represents the coordinates of a point on the virtual calibration plate or virtual mask plate; (x)k,yk,zk) Denotes (u, v,0) in the camera coordinate systemThe coordinates of (a);
in the optimization process, internal and external parameters K and R of the camerak、tkAs optimization variables.
5. The method for calibrating distortion of camera pixel by pixel as claimed in claim 4, wherein in generating distortion matrix, the distortion matrix is generated according to K and Rk、tkProjecting the virtual calibration plate and the virtual mask plate on the image by using a formula (6) -a formula (9), and performing DIC calculation as shown in a formula (4) to obtain n groups of projection image mapping matrixes; according to formula (2) and formula (3), taking m of each pixel point (i, j) in n groups of projection picture mapping matrixesi,j,xAverage of non-0 shifts in x-direction
And mi,j,yAverage of non-0 shifts in y-direction
I.e. representing the calculated pixel-by-pixel distortion quantities, which are arranged in pixel order to obtain a distortion matrix.
6. A method for camera distortion pixel-by-pixel calibration as claimed in claim 1, wherein the digital image correlation method is a Forward Additive Gauss-Newton method, an Inverse compositional Gauss-Newton method or a variation thereof.
7. The method for calibrating distortion of a camera pixel by pixel according to claim 1, wherein the optimization method adopts a mathematical optimization method or an evolutionary computation method or a combination thereof, the mathematical optimization method is a gradient method, a Gauss-Newton method, a Levenberg-Marquardt method or a linear programming method, and the evolutionary computation method is a particle swarm algorithm, an ant colony algorithm, a genetic algorithm or a simulated annealing algorithm.
8. The method for calibrating distortion pixel by pixel of a camera according to claim 1, wherein when distortion correction is performed on a newly shot picture, the position of a corresponding pixel point in a distorted image is sequentially searched for each pixel point of the corrected image according to the distortion matrix, and bilinear interpolation of pixel values of four integer coordinate pixels adjacent to the position in the distorted image is taken as the value of the pixel point in the corrected picture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110819872.0A CN113284196B (en) | 2021-07-20 | 2021-07-20 | Camera distortion pixel-by-pixel calibration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110819872.0A CN113284196B (en) | 2021-07-20 | 2021-07-20 | Camera distortion pixel-by-pixel calibration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113284196A CN113284196A (en) | 2021-08-20 |
| CN113284196B true CN113284196B (en) | 2021-10-22 |
Family
ID=77286774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110819872.0A Active CN113284196B (en) | 2021-07-20 | 2021-07-20 | Camera distortion pixel-by-pixel calibration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113284196B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117576228B (en) * | 2024-01-16 | 2024-04-16 | 成都合能创越软件有限公司 | Real-time scene-based camera coordinate calibration method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803273A (en) * | 2017-01-17 | 2017-06-06 | 湖南优象科技有限公司 | A kind of panoramic camera scaling method |
| CN109272570A (en) * | 2018-08-16 | 2019-01-25 | 合肥工业大学 | A kind of spatial point three-dimensional coordinate method for solving based on stereoscopic vision mathematical model |
| CN109598762A (en) * | 2018-11-26 | 2019-04-09 | 江苏科技大学 | A kind of high-precision binocular camera scaling method |
| CN110517202A (en) * | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of vehicle body camera calibration method and its caliberating device |
| CN112258588A (en) * | 2020-11-13 | 2021-01-22 | 江苏科技大学 | Calibration method and system of binocular camera and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750697B (en) * | 2012-06-08 | 2014-08-20 | 华为技术有限公司 | Parameter calibration method and device |
| KR101921672B1 (en) * | 2014-10-31 | 2019-02-13 | 후아웨이 테크놀러지 컴퍼니 리미티드 | Image processing method and device |
| US11455745B2 (en) * | 2019-05-21 | 2022-09-27 | Sportlogiq Inc. | System and method for image registration and camera calibration using learned error functions |
-
2021
- 2021-07-20 CN CN202110819872.0A patent/CN113284196B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106803273A (en) * | 2017-01-17 | 2017-06-06 | 湖南优象科技有限公司 | A kind of panoramic camera scaling method |
| CN109272570A (en) * | 2018-08-16 | 2019-01-25 | 合肥工业大学 | A kind of spatial point three-dimensional coordinate method for solving based on stereoscopic vision mathematical model |
| CN109598762A (en) * | 2018-11-26 | 2019-04-09 | 江苏科技大学 | A kind of high-precision binocular camera scaling method |
| CN110517202A (en) * | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of vehicle body camera calibration method and its caliberating device |
| CN112258588A (en) * | 2020-11-13 | 2021-01-22 | 江苏科技大学 | Calibration method and system of binocular camera and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| A New Solution for Camera Calibration and;Rui Melo等;《IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING》;20120217;第634-643页 * |
| 面成型3D打印工艺中投影图像畸变校正技术研究;贾红帅;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170115;第1-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113284196A (en) | 2021-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109598762B (en) | High-precision binocular camera calibration method | |
| CN106780388B (en) | Optical distortion correction method for linear array camera | |
| CN109272570B (en) | Space point three-dimensional coordinate solving method based on stereoscopic vision mathematical model | |
| CN112013792B (en) | Surface scanning three-dimensional reconstruction method for complex large-component robot | |
| CN111351446B (en) | Light field camera calibration method for three-dimensional topography measurement | |
| CN109272574B (en) | Construction method and calibration method of linear array rotary scanning camera imaging model based on projection transformation | |
| CN109827521B (en) | Calibration method for rapid multi-line structured optical vision measurement system | |
| CN115861445B (en) | Hand-eye calibration method based on three-dimensional point cloud of calibration plate | |
| CN112258587B (en) | Camera calibration method based on gray wolf particle swarm mixing algorithm | |
| CN112614075B (en) | Distortion correction method and equipment for surface structured light 3D system | |
| CN111047649A (en) | Camera high-precision calibration method based on optimal polarization angle | |
| CN113920205B (en) | Calibration method of non-coaxial camera | |
| CN110349257B (en) | Phase pseudo mapping-based binocular measurement missing point cloud interpolation method | |
| CN112947885B (en) | Method and device for generating curved surface screen flattening image | |
| CN112258588A (en) | Calibration method and system of binocular camera and storage medium | |
| CN112150561A (en) | Multi-camera calibration method | |
| CN115457147A (en) | Camera calibration method, electronic device and storage medium | |
| CN113284196B (en) | Camera distortion pixel-by-pixel calibration method | |
| CN113920206A (en) | Calibration method of perspective tilt-shift camera | |
| CN113298886A (en) | Calibration method of projector | |
| CN113610929B (en) | Combined calibration method of camera and multi-line laser | |
| CN115100078B (en) | Method and related device for correcting and filling dot matrix coordinates in curved screen image | |
| CN113865514B (en) | Calibration method of line structured light three-dimensional measurement system | |
| CN110689582B (en) | Total station camera calibration method | |
| CN116277979B (en) | Optical machine distortion correction method for DLP printer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |