CN111080705B - Calibration method and device for automatic focusing binocular camera - Google Patents
Calibration method and device for automatic focusing binocular camera Download PDFInfo
- Publication number
- CN111080705B CN111080705B CN201910376476.8A CN201910376476A CN111080705B CN 111080705 B CN111080705 B CN 111080705B CN 201910376476 A CN201910376476 A CN 201910376476A CN 111080705 B CN111080705 B CN 111080705B
- Authority
- CN
- China
- Prior art keywords
- calibration
- camera
- lens
- positions
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention discloses a calibration method and a calibration device for an automatic focusing binocular camera. The invention has the following beneficial effects: one, only one planar calibration template is used and the calibration template does not need to be rotated. Compared with the current universal calibration method which needs a plurality of angle images, the calibration time of a production line is saved, the yield in unit time is improved, and the production cost is reduced. And secondly, calibrating the binocular cameras at the positions of the associated lenses respectively to obtain a plurality of groups of binocular camera calibration parameters in total. The multiple groups of calibration parameters effectively compensate parameter differences caused by lens position changes in the photographing process of the automatic focusing camera, so that the precision of depth calculation is improved, the 3D application effect of the camera can be optimized, the user experience is improved, and the product competitiveness is enhanced.
Description
Technical Field
The invention relates to the field of computer vision, optical measurement and camera manufacturing, in particular to a binocular camera calibration method and device.
Background
The automatic focusing is a function built in the camera which automatically completes focusing on a shot subject through an electronic and mechanical device and enables an image to be clear. The camera with the automatic focusing function is widely applied to more and more industries, including the fields of smart phones, unmanned aerial vehicles, video monitoring and the like, due to the characteristics of accurate focusing and convenient operation.
On the other hand, camera applications are moving from 2D into the 3D era based on binocular, multi-view cameras. For example, in the mobile phone industry, an optical zoom function is realized through the combination of a wide-angle camera and a long-focus camera, or the image quality is improved through the combination of a black-and-white camera and a color camera, or an infrared structure optical camera module is used for capturing a 3D point cloud of a human face, so that the human face recognition is realized. The unmanned aerial vehicle trade realizes that 3D environmental perception supports unmanned aerial vehicle to keep away the barrier in flight through two mesh cameras. In the automobile industry, distance measurement and 3D environment perception are completed through a binocular camera to assist driving and support automatic driving.
The calibration of the camera is the key to realize the 3D application of the binocular camera. In camera-based measurement and vision applications, the relationship between the three-dimensional position of an object point in space and its corresponding two-dimensional pixel point position in an image is described mathematically by a geometric projection model. The parameters of the model are generally known by dimensionsThe calibration pattern (such as a solid circle array or a black and white checkerboard) is obtained by photographing, image processing and calculation, and the process of determining the parameters of the camera projection model is called calibration. The parameters comprise internal parameters which refer to the main point position c when the camera forms imagesx、cyAnd focal length f of lensx、fySuch parameters are only relevant to the camera itself; the external parameters refer to the position of the camera in space, and generally refer to a rotation vector R and a translation vector T of the camera in a certain reference coordinate system; distortion parameter refers to the deviation of the corresponding pixel position of an object point in an image from a theoretical imaging point calculated based on a central projection model in the shooting process of a camera, and the deviation is generally determined by a radial distortion parameter k1、k2、k3And tangential distortion parameter p1、p2To describe.
The general calibration solution for the current camera mass production line has two limitations. Firstly, because the used calibration algorithm requires to shoot images of a plurality of calibration templates from different angles, a 2D plane calibration template is rotated or a calibration three-dimensional template spliced by a plurality of plane calibration templates (fixed angles are formed between the plane calibration templates) is used on the current production line. The device for rotating the calibration template is complex in mechanism, and the rotation lengthens the measurement time, reduces the yield per unit time and increases the cost. The three-dimensional template spliced by the plurality of plane calibration templates is increased in cost, or needs to be accurately fixed according to a certain angle, or needs to be accurately measured before use. Second, in a camera with an auto-focusing function, a lens is generally fixed to an electronic and mechanical moving mechanism, such as a VCM motor (voice coil motor). The lens is pushed forward and backward along the optical axis by a moving mechanism in the lens barrel to change the position. The distance from the lens to the surface of the imaging chip is changed (namely, the image distance is changed), the focusing depth of the camera is changed, and therefore the focusing function aiming at different scene depths can be achieved. However, for each scene depth, the lens has a corresponding position in the lens barrel, and the calibration parameter values are different at different positions, and theoretically, the camera needs to be recalibrated at each lens position, that is, for each scene depth. In calibration of the camera, generally, it is considered that in order to obtain an accurate calibration result, an image of a focused calibration template must be used, so that a calibration template is required at each corresponding scene depth, the size of the calibration template is in direct proportion to the depth, the binocular camera focuses on the calibration templates at the distance respectively during calibration, and an image of the accurately focused calibration template is photographed for calibration parameter calculation. In an actual production line, a calibration plate is not generally placed at each distance due to the consideration of cost and efficiency, the calibration at each distance results in huge size of the calibration machine, and the calibration machine needs to be provided with a moving mechanism to replace calibration templates at different depths. The calibration of each distance can greatly prolong the station time and improve the production cost, which can not be accepted by camera module manufacturers. Therefore, the calibration parameter change caused by the lens position change in the lens barrel cannot be considered comprehensively by the existing calibration scheme.
Disclosure of Invention
The invention aims to provide a calibration method and a calibration device for an automatic focusing binocular camera, which improve the calibration precision and efficiency.
Therefore, the calibration method of the automatic focusing binocular camera provided by the invention comprises the following steps: s1, moving the lens to a first lens position, and shooting an image of a calibration template corresponding to the lens position for the calibration template of a 2D plane placed at a standard distance; s2, moving the lens, and respectively shooting an image of the calibration template corresponding to the corresponding lens position at a plurality of different positions of the lens; and S3, analyzing and processing the single calibration template image shot at each lens position, and calculating to obtain the calibration parameters of the camera relative to the plane calibration template at each lens position.
In some embodiments, the invention also includes the following preferred features:
respectively calculating calibration parameters of the left camera and the right camera relative to the plane calibration template when each lens is positioned; calculating the relative positions of the left camera and the right camera corresponding to the scene depth based on the obtained calibration parameters of the left camera and the right camera relative to the plane calibration template at the relevant positions; corresponding to n groups of lens correlation positions, namely corresponding to n scene depths, and obtaining calibration parameters of n groups of binocular cameras in total; wherein n is an integer greater than 1.
The captured calibration image may be in focus or out of focus.
During depth testing, the set of calibration parameters corresponding to the associated position closest to the lens position during current shooting is selected to calculate the depth.
And a solid circular array is used as a calibration template.
Optimizing the calibration parameters of the camera to ensure that the square sum of the reprojection errors of the circle centers of all solid circles on the calibration template is minimum, namely:
wherein M is the position of the center of the solid circle on the calibration template, M is the pixel position corresponding to the center point in the image,
pixel positions calculated for the reprojection positions, i.e. the positions of pixels displaced by image distortion after the centre of a solid circle is projected at the center
And
including distortion offset), K is an internal parameter of the camera, the rotation vector R and the translational vector T are external parameters of the camera, and K is [ K ═ K [1 k2 k3]For radial distortion parameter, p ═ p1 p2]Is a tangential distortion parameter; the equation is optimized by using a Levenberg-Marquardt algorithm, after iteration, when the iteration error is smaller than a preset threshold value, the optimization iteration is finished, and the obtained results K, R, T, K and p are respectively an internal parameter, an external parameter and a distortion parameter corresponding to the calibration distance.
The method is used for calibrating the binocular camera and the three-eye, four-eye, five-eye and multi-eye cameras.
During the depth test, the positions of the lenses to be calibrated of the left camera and the right camera are determined according to the focusing points obtained by interpolation fitting of the focusing calibration results of the camera production line.
During depth test, after the left camera and the right camera focus on the same scene depth, the position of a lens to be calibrated is determined through the equivalent movement of the lens.
The invention also provides an automatic focusing binocular camera calibration device, which comprises: the template placing unit is used for placing a calibration template; the image acquisition unit is used for acquiring a calibration template image obtained by shooting a calibration template; the position determining unit is used for respectively obtaining the spatial positions of the two cameras relative to the same calibration template and determining the relative positions of the binocular cameras at the moment when the left camera lens and the right camera lens focused on the same scene depth are at the associated positions;
a lens control and moving unit for moving the lens by an electronic and mechanical device;
a calibration unit, in which a computer program is embodied, for implementing said calibration method. When the calibration template is in a fixed distance, the binocular camera is calibrated at different positions of the lens by moving the lens, and no matter the shot calibration image is focused or out of focus.
Compared with the prior art, the invention has the following beneficial effects: one, only one planar calibration template is used and the calibration template does not need to be rotated. Compared with the current universal calibration method which needs a plurality of angle images, the calibration time of a production line is saved, the yield in unit time is improved, and the production cost is reduced. And secondly, calibrating the binocular cameras at the positions of the associated lenses respectively to obtain a plurality of groups of binocular camera calibration parameters in total. The multiple groups of calibration parameters effectively compensate parameter differences caused by lens position changes in the photographing process of the automatic focusing camera, so that the precision of depth calculation is improved, the 3D application effect of the camera can be optimized, the user experience is improved, and the product competitiveness is enhanced. And besides precision improvement, the method only uses one fixed calibration template, and calibrates the binocular camera by defocused blurred images at different positions of the lens through moving the lens, thereby greatly improving calibration efficiency and reducing production cost.
Drawings
FIG. 1 is a schematic diagram of a calibration template used in an embodiment of the present invention.
Fig. 2 is a schematic diagram of calculating depth information of a point P on a scene object surface according to an embodiment of the present invention.
Fig. 3 is a structural diagram of a calibration device on a binocular camera production line according to an embodiment of the invention.
FIG. 4 is a still shot illustration of an embodiment of the invention at multiple lens positions.
Fig. 5 is a schematic overall flow chart of the calibration method according to the embodiment of the present invention.
FIG. 6 is a schematic flow chart of a calibration algorithm at each position according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention realizes the great reduction of the calibration time through theoretical breakthrough, thereby being capable of calibrating a plurality of scene depths aiming at a plurality of lens positions of the camera. The problem that the existing calibration scheme does not consider the calibration parameter change caused by the lens position change in the lens barrel, so that the depth calculation is not accurate is solved.
The following embodiment of the invention is a multi-lens position calibration scheme and a multi-lens position calibration device of a binocular camera with an automatic focusing function. A2D plane calibration template is placed at a standard distance on a production line of the binocular camera. In the whole image shooting process, the calibration template is fixed and does not need to rotate. And under the calibration distance, sequentially moving the positions of the left camera lens by controlling the VCM motor, wherein each pair corresponds to one lens position, and the left camera shoots an image of a calibration template corresponding to the position. And analyzing and processing a single calibration template image shot at each lens position, and calculating calibration parameters of the left camera relative to the plane calibration template at the lens position. Similarly, the positions of the lenses of the right camera are sequentially moved, and the calibration parameters of the right camera relative to the plane calibration template at each lens position are calculated.
When the left and right cameras are focused at the same scene depth, the left and right camera lens positions are associated positions corresponding to the scene depth. Based on the obtained calibration parameters of the left camera and the right camera at the relevant positions, the relative positions of the left camera and the right camera corresponding to the scene depth can be calculated. And obtaining calibration parameters of n groups of binocular cameras in total corresponding to n groups of lens association positions, namely corresponding to n scene depths. During depth testing, the set of calibration parameters corresponding to the associated position closest to the lens position during current shooting is selected to calculate the depth.
1. Calibration template used
Referring to fig. 1, the calibration scheme of the present embodiment uses a solid circle array as the calibration template pattern, the solid circle array is composed of C rows and xL columns of solid circles, and C and L are natural numbers. Each solid circle has the same size, the same radius and the same horizontal and vertical center distance. In other embodiments, other forms of calibration templates, such as solid square arrays, etc., may be used.
2. Monocular camera calibration initial value setting
The homogeneous coordinate of the center of the solid circle in the reference coordinate system is [ X Y Z1 ], and the homogeneous coordinate of the pixel obtained by photographing the point in the camera is assumed to be [ X Y1 ]. According to a projection model based on pinhole imaging, a light ray of a circle center [ X Y1 ] of a solid circle of a calibration template passes through a projection center, namely an optical center of a lens, and is projected onto an image to obtain a corresponding imaging pixel point [ x y 1] (here, for the case of a plane calibration template, a Z coordinate is assumed to be 0)
Where σ is a scale factor. The rotation vector R and the translation vector T are external parameters of the camera and describe the spatial position of the camera in a calibration template coordinate system. K is an internal parameter of the camera and is defined as
Wherein f isxAnd fyFocal length in horizontal and vertical directions, cxAnd cyIs the image principal point in the horizontal and vertical directions of the image.
Based on each shot image of the calibration template, the corresponding homography matrix can be calculated
Wherein h isjIs the column vector of the jth column (j ═ 1,2,3), hijIs the H matrix element in the ith row and jth column (i, j ═ 1,2, 3). According to the definition of the homography matrix, there are
[h1 h2 h3]＝K[r1 r2 T] (4)
According to the nature of the rotation matrix, r1And r2Is an orthogonal unit vector, thus having
Wherein:
from (5) and (6) respectively
h11h12·B11+(h31h12+h11h32)·B13+h21h22·B22+(h31h22+h21h32)·B23+ h31h32·B33＝0 (8)
In the production process of the camera, when the optical lens and the imaging chip are coupled and assembled, the main point of an image is required to be away from the central point of the imaging chip and not exceed a certain pixel range. Thus presetting the principal point cxAnd cyIs the image center. (8) In the formula (9), the homography matrix H can be obtained by shooting a calibration template image, and the element H of the homography matrix Hij(i, j ═ 1,2,3) is known; c. CxAnd cyAs the center point of the image, known, BijIs the camera intrinsic parameter focal length fx、fyAnd principal point cx、cyThe mathematical combination of (1), which calculates the intermediate quantity occurring in the process, i.e. BijHas only two unknowns f thereinxAnd fySo that the two equation sets (8) and (9) can solve two internal parameters, namely the focal length fxAnd fyIs started. The initial value of the spatial position of the binocular camera in the reference coordinate system can be estimated according to the position of the camera relative to the calibration template and the type of the binocular camera. Taking the binocular camera module of the mobile phone as an example, because the optical axes of the two cameras are approximately parallel, the initial value of the relative rotation matrix R is set as the unit matrix
3. Distortion parameter and initial value setting thereof
Because of the optical distortion of the lens, the actually projected pixel points generally have a small deviation in the radial direction and the tangential direction simultaneously on the image. Radial distortion refers to movement radially inward or outward. Tangential distortion refers to the shift of the actual image point in the direction perpendicular to the sagittal direction, i.e. the tangential direction. The aforementioned theoretical pixel position [ xy ] based on the projection model]Is affected by distortion, and is shifted to its actual projection position
Simulation with the following relationship
Wherein, [ k ]1 k2 k3]As a radial distortion parameter, [ p ]1 p2]As a tangential distortion parameter, r2＝x2+y2. The lenses typically used in cameras are relatively small in distortion, so the radial and tangential distortion parameters k1 k2 k3]And [ p ]1 p2]Is typically set to zero.
4. Monocular camera calibration parameter optimization
And generating a corresponding image pixel point in the image when the center of each solid circle is photographed. According to the projection model based on the pinhole imaging principle and image distortion, each circle center can also calculate a theoretical imaging position (including the offset generated by distortion). The deviation of the actual image point from this theoretical pixel position is called the reprojection error. The parameters of the geometric projection model, namely the calibration parameters of the camera, are to ensure that the square sum of the reprojection errors of the centers of all solid circles on the calibration template is minimum, and at the moment, the projection model most accurately describes the optical imaging projection process of the camera at the lens position
Wherein M is the position of the center of the solid circle on the calibration template, M is the pixel position corresponding to the center point in the image,
pixel positions calculated for the reprojection positions, i.e. the positions of pixels displaced by image distortion after the centre of a solid circle is projected at the center
And
including distortion offset), K is an internal parameter of the camera, the rotation vector R and the translational vector T are external parameters of the camera, and K is [ K ═ K [1 k2 k3]For radial distortion parameter, p ═ p1 p2]Is a tangential distortion parameter. The equation is optimized by using Levenberg-Marquardt and Levenberg-Marquardt algorithm, after a plurality of iterations, when the error of the iteration is smaller than a preset threshold value, the optimization iteration is finished, and the obtained results K, R, T, K and p are respectively the internal parameter, the external parameter and the distortion parameter corresponding to the calibration distance. The calibration parameters of the binocular camera obtained through the nonlinear optimization are more accurate.
5. Relative position relation of binocular camera
When the left and right cameras are focused at the same scene depth, the left and right camera lens positions are associated positions corresponding to the scene depth. When the binocular cameras are calibrated at the lens associated positions, after the left camera and the right camera are respectively calibrated at the lens associated positions, the positions of the two cameras relative to the same plane calibration template are known, and the spatial position of the left camera is Rl、TlThe spatial position of the right camera is Rr、TrSo that the relative positions of the two cameras can be calculated as
6. Depth information calculation
After the binocular cameras are calibrated at n lens associated positions (taking five associated positions as an example) respectively, n groups (five groups) of calibration parameters of the binocular cameras are obtained in total. And during depth test, recording the position of the VCM motor during current image shooting, comparing the current position of the VCM motor with the positions of the VCM motors corresponding to the five sets of calibration parameters, and selecting the set of calibration parameters with the position of the VCM motor closest to the position of the VCM motor to calculate the depth.
Referring to FIG. 2, taking a point P on the scene object surface as an example, the depth information of the point is calculatedIn the left camera coordinate system, the optical center C1The coordinate is the origin, and the point P is at the imaging corresponding point m of the left camera1Has the coordinate of [ x ]1 y1 f1]. In the right camera coordinate system, its optical center C2The coordinate is the origin, the point P is at the imaging corresponding point m of the right camera2Has the coordinate of [ x ]2 y2 f2]。C1And m1The coordinates of the right camera coordinate system are respectively
And
the coordinate of the point P is C1、m1Connecting line with C2、m2And connecting the line to the intersection point under the right camera coordinate system, wherein the depth information of the line is the Z-axis coordinate of the intersection point.
6) The structure of the calibration device and the flow of the calibration method are detailed in detail as follows:
fig. 3 is a structural diagram of the calibration device on the binocular camera production line. The calibration distance can be properly selected according to the requirements of precision, capacity, size and the like, and a 2D plane calibration template is placed at the selected standard calibration distance, wherein 0.5 m is taken as an example. The binocular camera module is fixed in the position just to the calibration template, and the binocular camera optical axis is perpendicular to the calibration template promptly. And the VCM motor pushes the lens to move to a new position, after the position is reached, the binocular camera takes a static picture, and the calibration algorithm calculates the calibration parameters of the camera. Meanwhile, the VCM motor pushes the lens to the next position, the binocular camera performs static image shooting, and calibration is performed, as shown in FIG. 4.
The calibration device disclosed by the invention calibrates the left camera and the right camera of the binocular camera module respectively once according to the following steps, as shown in figure 5:
first step, connecting and opening the camera
Secondly, the VCM motor pushes the lens to the position 1, a calibration template image is shot, and a calibration program calculates corresponding calibration parameters when the position 1 of the lens is positioned
Thirdly, the VCM motor pushes the lens to the position 2, a calibration template image is shot, and a calibration program calculates corresponding calibration parameters when the lens is at the position 2
Fourthly, the VCM motor pushes the lens to the position 3, a calibration template image is shot, and a calibration program calculates corresponding calibration parameters when the lens is at the position 3
The fifth step, the VCM motor pushes the lens to the position 4, shoots the calibration template image, and the calibration program calculates the corresponding calibration parameter when the lens is at the position 4
Sixthly, the VCM motor pushes the lens to the position 5, a calibration template image is shot, and a calibration program calculates corresponding calibration parameters when the lens is at the position 5
Seventhly, the camera is turned off and disconnected
The left camera or the right camera performs calibration at each lens position, and the corresponding calibration method has the following flow, as shown in fig. 6:
1. the camera shoots an image of the calibration template at the position of the corresponding lens;
2. extracting the pixel position of the solid circle center in a calibration template image shot by a camera through an image processing algorithm;
3. based on the calibration image, calculating the focal length of the lens of the camera, and obtaining the initial values (8), (9) of the internal parameter, the external parameter and the distortion parameter of the camera
4. And optimizing the internal parameters, the external parameters and the distortion parameters of the camera based on a Levenberg-Marquardt optimization algorithm to obtain accurate camera calibration parameters (12).
Calculating relative position parameters of the binocular camera when the associated lens position is calculated
When the left and right cameras are focused at the same scene depth, the left and right camera lens positions are associated positions corresponding to the scene depth. Based on the obtained calibration parameters of the left camera and the right camera at the relevant positions, relative position parameters (13) of the left camera and the right camera corresponding to the relevant lens positions, namely the scene depth can be calculated.
A binocular camera calibration device, the device includes:
the template placing unit is used for placing a calibration template;
the image acquisition unit is used for acquiring a calibration template image obtained by shooting a calibration template;
a lens control and moving unit for moving the lens by an electronic and mechanical device;
this binocular camera calibration device still includes following computer program to the realization is when maring the template at a fixed distance, through moving the camera lens, is markd the binocular camera in the different positions of camera lens, no matter the demarcation image of shooing is focus or out of focus:
the extraction unit is used for detecting a solid circle in the image of the calibration template so as to extract the pixel position of the circle center;
the calibration unit is used for calculating or setting initial values of internal parameters, external parameters (rotation vectors and translation vectors) and distortion parameters of the camera and preparing for next calibration parameter optimization;
and the optimization unit is used for optimizing the internal parameters, the external parameters and the distortion parameters of the camera by using a Levenberg-Marquardt algorithm based on the minimization of the quadratic sum of the reprojection errors of all the central points to obtain an accurate calibration result.
And the position determining unit is used for determining the relative positions of the binocular cameras at the moment based on the spatial positions of the two cameras relative to the same calibration template, which are respectively obtained by the previous calibration step, when the left camera lens and the right camera lens focused on the same scene depth are at the associated positions.
Application example one
On a production line of a binocular camera module of a smart phone, an automatic focusing calibration station is generally arranged in front of a calibration station, and the station calibrates the focusing position of a camera. The equipment for automatically focusing the calibration station places a 2D plane focusing calibration template at five distances of 0.07, 0.1, 0.2, 0.375 and 5.0 meters. The binocular camera modules are sequentially placed to positions opposite to the focus calibration templates. And facing each focusing calibration template, the VCM motor in the camera module drives the lens to move step by step in the whole travel range. At each stop position of the lens, the binocular camera statically shoots an image of the focusing calibration template, and the contrast of the image is calculated. And in the whole process of lens movement, searching for the highest image contrast position, and finishing final focusing. The lens position corresponding to the highest image contrast is the corresponding focus point of the scene depth (0.07, 0.1, 0.2, 0.375 or 5.0 meters).
In this embodiment, after the binocular camera module finishes measurement at the auto-focus calibration station, five respective focuses of the left and right cameras are selected as the lens positions for depth calibration according to the patent of the present invention.
Acquiring a calibration template image obtained by shooting a calibration template; (corresponding to S1)
The calibration template is typically a repeating pattern with fixed spacing, such as a black and white checkerboard calibration template, an equally spaced solid circular array calibration template, and the like.
As shown in fig. 1, the present calibration scheme employs a solid circle array consisting of 8 rows by 11 columns of solid circles as the calibration template pattern. Each solid circle has the same size, the same radius and the same horizontal and vertical center distance.
In the example, only one calibration template image needs to be shot by the left camera and the right camera at each lens position, and the optical axes of the left camera and the right camera are perpendicular to the calibration template during shooting, so that pattern deformation caused by shooting angles does not exist. In addition, because the calibration template is arranged on the LED panel lamp, the contrast between the calibration pattern and the white background is strong, and the calibration template is easy to extract under the conditions of no focusing and no blurring of the calibration pattern. In a calibration template image shot under an out-of-focus condition, the blurring of the dot edges is symmetrical about the center of a circle, and the position of the center of the circle is not influenced. Based on the above three advantageous factors, the solid circle array calibration template is suitable in the present embodiment.
According to the number of solid circles in the horizontal direction and the vertical direction in the calibration template and the center distance, the distribution of the solid circle array in the calibration template coordinate system can be determined, and the homogeneous coordinate of the center of the circle is obtained [ X Y1 ].
Detecting a solid circle for each calibration template image to extract a central point; (corresponding to S2)
In the field of computer vision such as three-dimensional scene reconstruction, repeated solid circles are often used for constructing calibration patterns, and the size of a calibration template is determined through fixed circle radius and circle center distance. The center of the solid circle has the advantages of easy detection, high position precision, reliable matching, real-time processing and the like. The current circle center detection algorithm comprises: circle center detection based on Blob area analysis, circle center detection based on edge extraction, circle center detection based on Hough transform and the like. In this example, after the calibration image is captured, a solid circle region is detected based on the gray value of the image, and a homogeneous coordinate [ x y 1] with the center of mass of the circle region as the center pixel position is obtained. The image processing steps are simple in calculation and strong in real-time performance. The same is true for the out-of-focus and fuzzy conditions of the calibration template image except for the condition that the calibration template image is in clear focus.
Calculating or setting initial values of respective internal parameters, external parameters (rotation vectors and translation vectors) and distortion parameters of the binocular camera; (corresponding to S3)
In computer vision, the interrelationship of a point on an object in space to its projected position on an image plane by an imaging system is generally described by a geometric projection model of a camera (or camera) system. A commonly used projection model is the central projection in optics based on the pinhole imaging principle. In the model, a point on an object passes through the projection center, namely the optical center of a lens, and is projected on an imaging chip along a straight line.
The homogeneous coordinate of the center of the solid circle in the reference coordinate system is [ X Y Z1 ], and the homogeneous coordinate of the pixel obtained by photographing the point in the camera is assumed to be [ X Y1 ]. According to a projection model based on pinhole imaging, the center [ X Y1 ] of a solid circle of a calibration template is projected on an image according to the formula (1) to obtain a corresponding imaging pixel point [ x y 1 ]. The rotation vector R and the translation vector T are external parameters of the camera and describe the spatial position of the camera in a calibration template coordinate system. K is an internal parameter of the camera and is defined as formula (2).
In the production process of the camera, when the optical lens and the imaging chip are coupled and assembled, the main point of an image is required to be away from the central point of the imaging chip and not exceed a certain pixel range. Thus presetting the principal point cxAnd cyIs an imageA center. (8) In the formula (9), the homography matrix H can be obtained by shooting the image of the calibration template, cxAnd cyThe image center point is known, so that the two equation sets (8), (9) can be solved to obtain two internal parameters, namely the focal length fxAnd fyIs started. The initial value of the spatial position of the binocular camera in the coordinate system of the calibration template can be estimated according to the position of the camera relative to the calibration template and the type of the binocular camera. Taking a mobile phone binocular camera module as an example, because the optical axes of the two cameras are approximately parallel, the initial value of the relative rotation matrix R is set as a unit matrix.
Because the lens has optical distortion, the actually projected pixel point generally has a small deviation on the image. The image distortion is mainly caused by the following reasons: the processing error of the lens surface causes the defect of the radial curvature; the optical center of each lens cannot be strictly kept collinear, and an eccentricity error is generated; due to tolerances in lens design, production and camera assembly processes, the lens and the imaging chip are not parallel and inclined. The above errors cause distortion of the image in both radial and tangential directions. Radial distortion refers to the fact that the actual image point moves inward or outward on its ideal location and the optical center line, i.e., radially. Tangential distortion refers to the fact that the actual image point is shifted in the direction perpendicular to the sagittal direction, i.e. in the tangential direction.
The theoretical pixel location based on the central projection model [ x y ] described above]Is affected by distortion, and is shifted to its actual projection position
The simulation is carried out by using the relations of the equations (10) and (11). The lenses typically used in cameras are relatively small in distortion, so the radial and tangential distortion parameters k1 k2 k3]And [ p ]1 p2]Is typically set to zero.
Optimizing respective internal parameters, external parameters and distortion parameters of the binocular camera to obtain an accurate calibration result; (corresponding to S4)
And generating a corresponding image pixel point in the image when the center of each solid circle is photographed. According to the projection model based on the pinhole imaging principle and the distortion model, a theoretical imaging position (including offset generated by distortion) can be calculated for each circle center. The deviation of the actual image point from this theoretical pixel position is called the reprojection error. The parameters of the geometric projection model, namely the calibration parameters of the camera, are such that the sum of squares of the reprojection errors of the centers of all solid circles on the calibration template is minimum (12), and at the moment, the projection model most accurately describes the optical imaging projection process of the camera at the lens position. The equation is optimized by using Levenberg-Marquardt and Levenberg-Marquardt algorithm, after a plurality of iterations, when the error of the iteration is smaller than a preset threshold value, the optimization iteration is finished, and the obtained results K, R, T, K and p are respectively the internal parameter, the external parameter and the distortion parameter corresponding to the calibration distance. The calibration parameters of the binocular camera obtained through the nonlinear optimization are more accurate.
Determining the relative position of the binocular camera;
when the left and right cameras are focused at the same scene depth, the left and right camera lens positions are associated positions corresponding to the scene depth. When the binocular cameras are calibrated at the associated positions of the lenses, after the left camera and the right camera are respectively calibrated at the associated lens positions of the left camera and the right camera, the positions of the two cameras relative to the same plane calibration template are known, so that the relative positions of the two cameras can be calculated according to the formula (13).
Depth information calculation
And after the binocular cameras are calibrated at the positions of the five lenses respectively, obtaining calibration parameters of the five groups of binocular cameras in total. And during depth test, recording the position of the VCM motor during current image shooting, comparing the current position of the VCM motor with the positions of the VCM motors corresponding to the five sets of calibration parameters, and selecting the set of calibration parameters with the position of the VCM motor closest to the position of the VCM motor to calculate the depth.
The parameter calibration method provided in the embodiment can be applied to camera calibration with an automatic focusing function, and compared with the existing general calibration method, the parameter calibration method has the advantages of simplicity and high efficiency because only one calibration template image is adopted for parameter calibration. In addition, the device provides camera calibration based on the out-of-focus blurred image at a plurality of lens positions, so that the difference of calibration parameters caused by the change of the lens positions can be better compensated, and the subsequent depth calculation is more accurate.
The method is also suitable for calibrating the three-view, four-view, five-view and multi-view cameras, although a plurality of cameras are provided, the cameras are regarded as the two-view cameras as long as the depth calculation principle is carried out based on the binocular trigonometry principle, and the relative positions of the cameras are calibrated and determined based on the multi-lens positions of a single plane calibration template, so that the method belongs to the protection scope of the invention.
Application example two
Similar to the first embodiment, the binocular camera module calibrates the focusing position of the camera at the automatic focusing calibration station to determine the corresponding focus point of the scene depth of 0.07, 0.1, 0.2, 0.375 or 5.0 meters.
Based on the obtained five focuses of the left camera and the right camera, the autofocus calibration device on the production line generally fits the focuses corresponding to the depths of other unmeasured scenes by an interpolation method (e.g., linear interpolation, etc.). In this embodiment, an appropriate main scene depth, for example, 0.5, 1.0, 1.5, 2.0 meters, may be selected according to the application of the subsequent binocular camera, and the respective lens positions to be calibrated of the left and right cameras may be determined according to the focusing points obtained by interpolation fitting of the result of the camera production line focusing calibration.
The following steps are repeated in the first embodiment and are not described again.
Application example three
A2D planar calibration template is placed at a selected standard calibration distance, here 0.5 meters for example. Firstly, the camera module to be tested finds the focusing point corresponding to the scene depth (namely 0.5 meter) through automatic focusing.
In a camera of a smartphone, a lens is generally pushed back and forth along an optical axis by a VCM motor (voice coil motor) in a lens barrel to change a position. There is generally a good linear relationship between the change in the control amount of the VCM motor and the displacement of the lens during the active stroke of the VCM motor. The change in the distance of the lens from the surface of the imaging chip (i.e., the change in the image distance) can be relatively accurately controlled by the VCM motor, so that different lens positions can be selected.
In this embodiment, the focuses of the left and right cameras can be used as their respective starting positions, and the lenses of the left and right cameras can be driven by controlling the VCM motors to displace equally. Assuming that the focal lengths of the lenses of the left and right cameras are the same, when the lenses are located at new positions, the left and right cameras focus on the same scene depth. According to the application of the subsequent binocular cameras, the displacement of the VCM motor can be properly selected to determine the positions of the left camera and the right camera to be calibrated respectively.
In this embodiment, after the left and right cameras focus on the same scene depth, the positions of the lenses to be calibrated are determined by the equivalent movement of the lenses. Embodiments one and two lens positions to be calibrated are determined based on the results of the autofocus calibration station, unlike the present embodiment.
The following steps are repeated in the first embodiment and are not described again.
Claims (10)
1. An automatic focusing binocular camera calibration method is characterized by comprising the following steps:
s1, moving the lenses of the left camera and the right camera to a plurality of different positions respectively, and shooting images of the calibration template corresponding to the corresponding lens positions by the left camera and the right camera at the plurality of different positions of the lens respectively aiming at the calibration template of the 2D plane placed at the standard distance;
s2, analyzing and processing a single calibration template image shot at each lens position, and respectively calculating calibration parameters of the left camera and the right camera relative to the plane calibration template at each lens position, wherein when the left camera and the right camera are focused at the same scene depth, the lens position of the left camera and the lens position of the right camera are associated positions corresponding to the scene depth; calculating the relative positions of the left camera and the right camera corresponding to the scene depth based on the obtained calibration parameters of the left camera and the right camera relative to the plane calibration template at the relevant positions; corresponding to n groups of lens correlation positions, namely corresponding to n scene depths, and obtaining calibration parameters of n groups of binocular cameras in total; wherein n is an integer greater than 1; during depth testing, the set of calibration parameters corresponding to the associated position closest to the lens position during current shooting is selected to calculate the depth.
2. The auto-focusing binocular camera calibration method according to claim 1, wherein the captured calibration image may be in focus or out of focus.
3. The method for calibrating an auto-focusing binocular camera according to claim 1, wherein a solid circular array is used as a calibration template.
4. The auto-focusing binocular camera calibration method of claim 3, wherein the camera calibration parameters are optimized such that a sum of squares of reprojection errors of centers of all solid circles on the calibration template is minimized, which is the following equation:
wherein M is the position of the center of the solid circle on the calibration template, M is the position of the pixel corresponding to the center of the circle in the image,
pixel positions calculated for the reprojection positions, i.e. the positions of pixels displaced by image distortion after the centre of a solid circle is projected at the center
And
including distortion offset), K is an internal parameter of the camera, the rotation vector R and the translational vector T are external parameters of the camera, and K is [ K ═ K [1 k2 k3]For radial distortion parameter, p ═ p1 p2]Is a tangential distortionChanging parameters;
the formula is optimized by using a Levenberg-Marquardt algorithm, after iteration, when the iteration error is smaller than a preset threshold value, the optimization iteration is finished, and the obtained results K, R, T, K and p are respectively an internal parameter, an external parameter and a distortion parameter corresponding to the calibration distance.
5. The method for calibrating an auto-focusing binocular camera according to claim 1, wherein the method is used for calibrating a binocular camera and a three-eye, four-eye, five-eye and multi-eye camera.
6. The auto-focusing binocular camera calibration method according to claim 1, wherein lens positions to be calibrated for each of the left and right cameras are determined based on a focus point obtained by interpolation fitting of a result of camera production line focus calibration at the time of depth test.
7. The auto-focusing binocular camera calibration method of claim 1, wherein in the depth test, the positions of the lenses to be calibrated are determined by an equal amount of movement of the lenses after the left and right cameras are focused at the same scene depth.
8. An auto-focus binocular camera calibration apparatus, comprising a computer program for implementing the calibration method according to any one of claims 1 to 7, wherein the binocular camera is calibrated at different positions of the lens by moving the lens when the calibration template is at a fixed distance, regardless of whether the captured calibration image is in focus or out of focus.
9. The utility model provides an automatic two mesh camera calibration device of focusing which characterized in that includes:
the template placing unit is used for placing a calibration template;
the image acquisition unit is used for acquiring a calibration template image obtained by shooting a calibration template;
the lens control and moving unit is used for moving the lenses of the left camera and the right camera through an electronic and mechanical device;
a calibration unit having a computer program embodied therein for implementing: when the calibration template is at a fixed distance, calibrating the binocular cameras at different positions of the lenses by moving the lenses of the left and right cameras to obtain calibration parameters of the left and right cameras relative to the plane calibration template when the lenses are at a plurality of positions; the calibration parameters of the left camera and the right camera relative to the plane calibration template when each lens position is calculated respectively, wherein when the left camera and the right camera focus on the same scene depth, the lens position of the left camera and the lens position of the right camera are associated positions corresponding to the scene depth; calculating the relative positions of the left camera and the right camera corresponding to the scene depth based on the obtained calibration parameters of the left camera and the right camera relative to the plane calibration template at the relevant positions; corresponding to n groups of lens correlation positions, namely corresponding to n scene depths, and obtaining calibration parameters of n groups of binocular cameras in total; wherein n is an integer greater than 1; during depth testing, the set of calibration parameters corresponding to the associated position closest to the lens position during current shooting is selected to calculate the depth.
10. The utility model provides an automatic two mesh camera calibration device of focusing which characterized in that includes:
the template placing unit is used for placing a calibration template;
the image acquisition unit is used for acquiring a calibration template image obtained by shooting a calibration template;
a lens control and moving unit for moving the lens by an electronic and mechanical device;
calibration unit, in which a computer program is embodied for implementing a calibration method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910376476.8A CN111080705B (en) | 2019-05-07 | 2019-05-07 | Calibration method and device for automatic focusing binocular camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910376476.8A CN111080705B (en) | 2019-05-07 | 2019-05-07 | Calibration method and device for automatic focusing binocular camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111080705A CN111080705A (en) | 2020-04-28 |
| CN111080705B true CN111080705B (en) | 2021-06-04 |
Family
ID=70310310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910376476.8A Active CN111080705B (en) | 2019-05-07 | 2019-05-07 | Calibration method and device for automatic focusing binocular camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111080705B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111932636B (en) * | 2020-08-19 | 2023-03-24 | 展讯通信（上海）有限公司 | Calibration and image correction method and device for binocular camera, storage medium, terminal and intelligent equipment |
| CN112381883A (en) * | 2020-11-12 | 2021-02-19 | 广东未来科技有限公司 | Design method of binocular 3D camera, computer readable medium and control system |
| CN113115017B (en) * | 2021-03-05 | 2022-03-18 | 上海炬佑智能科技有限公司 | 3D imaging module parameter inspection method and 3D imaging device |
| CN113587895B (en) * | 2021-07-30 | 2023-06-30 | 杭州三坛医疗科技有限公司 | Binocular distance measuring method and device |
| CN113838146A (en) * | 2021-09-26 | 2021-12-24 | 昆山丘钛光电科技有限公司 | Method and device for verifying calibration precision of camera module and method and device for testing camera module |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103559708A (en) * | 2013-10-30 | 2014-02-05 | 同济大学 | Industrial fixed-focus camera parameter calibration device based on square target model |
| WO2015000441A1 (en) * | 2013-07-05 | 2015-01-08 | Mediatek Inc. | On-line stereo camera calibration device and method for generating stereo camera parameters |
| CN105225224A (en) * | 2015-08-30 | 2016-01-06 | 大连理工大学 | Improve arrangements of cameras and the scaling method of depth of field measuring accuracy |
| CN106651859A (en) * | 2017-01-24 | 2017-05-10 | 长沙全度影像科技有限公司 | Multipath fisheye camera calibration device and method |
| CN108844459A (en) * | 2018-05-03 | 2018-11-20 | 华中科技大学无锡研究院 | A kind of scaling method and device of leaf digital template detection system |
| CN109483516A (en) * | 2018-10-16 | 2019-03-19 | 浙江大学 | A kind of mechanical arm hand and eye calibrating method based on space length and epipolar-line constraint |
| CN109859272A (en) * | 2018-12-18 | 2019-06-07 | 像工场(深圳)科技有限公司 | A kind of auto-focusing binocular camera scaling method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1260544C (en) * | 2004-07-14 | 2006-06-21 | 天津大学 | Compatible and accurate calibration method for double eye line structure photo-sensor and implementing apparatus |
-
2019
- 2019-05-07 CN CN201910376476.8A patent/CN111080705B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015000441A1 (en) * | 2013-07-05 | 2015-01-08 | Mediatek Inc. | On-line stereo camera calibration device and method for generating stereo camera parameters |
| CN103559708A (en) * | 2013-10-30 | 2014-02-05 | 同济大学 | Industrial fixed-focus camera parameter calibration device based on square target model |
| CN105225224A (en) * | 2015-08-30 | 2016-01-06 | 大连理工大学 | Improve arrangements of cameras and the scaling method of depth of field measuring accuracy |
| CN106651859A (en) * | 2017-01-24 | 2017-05-10 | 长沙全度影像科技有限公司 | Multipath fisheye camera calibration device and method |
| CN108844459A (en) * | 2018-05-03 | 2018-11-20 | 华中科技大学无锡研究院 | A kind of scaling method and device of leaf digital template detection system |
| CN109483516A (en) * | 2018-10-16 | 2019-03-19 | 浙江大学 | A kind of mechanical arm hand and eye calibrating method based on space length and epipolar-line constraint |
| CN109859272A (en) * | 2018-12-18 | 2019-06-07 | 像工场(深圳)科技有限公司 | A kind of auto-focusing binocular camera scaling method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111080705A (en) | 2020-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109767476B (en) | Automatic focusing binocular camera calibration and depth calculation method | |
| CN109859272B (en) | Automatic focusing binocular camera calibration method and device | |
| CN111080705B (en) | Calibration method and device for automatic focusing binocular camera | |
| CN110036410B (en) | Apparatus and method for obtaining distance information from view | |
| CN105716542B (en) | A kind of three-dimensional data joining method based on flexible characteristic point | |
| CN109919911B (en) | Mobile three-dimensional reconstruction method based on multi-view photometric stereo | |
| CN105225224A (en) | Improve arrangements of cameras and the scaling method of depth of field measuring accuracy | |
| CN111932636B (en) | Calibration and image correction method and device for binocular camera, storage medium, terminal and intelligent equipment | |
| CN109325981B (en) | Geometric parameter calibration method for micro-lens array type optical field camera based on focusing image points | |
| CN109712232B (en) | Object surface contour three-dimensional imaging method based on light field | |
| CN111189415B (en) | Multifunctional three-dimensional measurement reconstruction system and method based on line structured light | |
| Kim et al. | Performance analysis and validation of a stereo vision system | |
| CN112489137A (en) | RGBD camera calibration method and system | |
| EP3144894A1 (en) | Method and system for calibrating an image acquisition device and corresponding computer program product | |
| CN107063644B (en) | Finite object distance distortion measuring method and system | |
| CN116625258A (en) | Chain spacing measuring system and chain spacing measuring method | |
| Zhang et al. | Improved Camera Calibration Method and Accuracy Analysis for Binocular Vision | |
| CN111292380B (en) | Image processing method and device | |
| CN110708532B (en) | Universal light field unit image generation method and system | |
| Sardemann et al. | On the accuracy potential of focused plenoptic camera range determination in long distance operation | |
| CN112489141B (en) | Production line calibration method and device for single-board single-image strip relay lens of vehicle-mounted camera | |
| CN109682312B (en) | Method and device for measuring length based on camera | |
| Xu et al. | Mobile camera array calibration for light field acquisition | |
| CN113160393B (en) | High-precision three-dimensional reconstruction method and device based on large depth of field and related components thereof | |
| CN111862237B (en) | Camera calibration method for optical surface shape measurement and device for realizing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |