CN109427077B

CN109427077B - Processor for calibrating automobile all-round looking system

Info

Publication number: CN109427077B
Application number: CN201710757152.XA
Authority: CN
Inventors: 周牧
Original assignee: Anbofu Electronics Suzhou Co ltd
Current assignee: Anbofu Electronics Suzhou Co ltd
Priority date: 2017-08-29
Filing date: 2017-08-29
Publication date: 2021-10-12
Anticipated expiration: 2037-08-29
Also published as: CN109427077A

Abstract

The invention relates to a processor for calibrating an automobile all-round looking system, which comprises a plurality of fisheye cameras, wherein the processor is used for executing the following computer program steps to obtain a rotation matrix Rt of each camera: when the automobile is static, acquiring observation vectors v1, v2 and v3 from a camera to three vertexes of a known marker in a camera coordinate system, and calculating a normal vector z of an equilateral triangle in the camera coordinate system according to the side length and the space geometric relation of the triangle; when the automobile makes forward linear motion along the X axis of the vehicle coordinate system, calculating the X axis unit vector of the camera coordinate system; calculating a rotation matrix Rt of the camera: rt ═ X, X × Z, Z (X, X × Z, Z)^‑1. Compared with the prior art, the process adopted by the processor fully utilizes potential prior information, the calibration scheme is simple, the required calibration space is small, the calibration precision is high, the anti-interference performance is stronger, and the requirement on the skill of an operator is greatly reduced to the extent that the operator can do the process by himself.

Description

Processor for calibrating automobile all-round looking system

Technical Field

The invention relates to a processor with a computer program, in particular to a processor for calibrating an automobile all-round system.

Background

With the increasing holding capacity of automobiles and the increasing complexity of traffic environment, safe parking has become a difficult problem for drivers. In order to reduce potential safety risks in the parking process, various auxiliary parking systems come into play, such as a rearview reversing camera, a reversing radar and other systems, which reduce the parking burden of a driver to a certain extent, but still have the defects of being not intuitive and being incapable of completely eliminating blind areas. The 360-degree all-round looking system can eliminate visual blind areas by covering 360-degree areas around the vehicle body through the plurality of cameras, and meanwhile, images around the vehicle body are seamlessly spliced into a overlooking picture by the 360-degree all-round looking system to be presented to a driver, so that the performance is more visual.

Although the value ratio of the 360-degree look-around system in the whole vehicle is not high, the 360-degree look-around system occupies an important ring in factory calibration because the current calibration scheme also needs a large amount of manpower, which increases the cost ratio of the look-around system to the whole vehicle. The value/cost ratio is not large enough, which can affect the popularization of the panoramic system. The current mainstream factory calibration scheme is introduced as follows:

because the same type of automobile can have assembly deviation and can have installation deviation when the camera is installed all around to the automobile body, if adopt a set of fixed parameter to carry out 360 degrees all around system's image mosaicings, these deviations can influence the image mosaicings effect so, consequently must go off line and mark the influence of eliminating these deviations. The traditional factory calibration scheme has the following problems: (1) the requirement on the calibration environment is high, for example, the vehicle needs to be stopped at a fixed position, the precision requirement is high, the technical difficulty of operation of workers is increased, and more labor time cost is occupied; (2) the method is difficult to apply to 4S shop calibration and is more difficult for a driver to use personally, and potential car purchasing cost is increased indirectly. (3) A safe and convenient automatic calibration scheme is helpful for BEV products to move from the front-loading market to the back-loading market and transition from the incremental market to the stock market.

In fact, the six external parameters of the camera include the camera position parameter T: (t)_x,t_y,t_z) And a rotation parameter R: (r)_x,r_y,r_z) Wherein R: (r)_x,r_y,r_z) Equivalent to a rotation matrix Rt of 3 x 3. Their properties are different. From the measurement difficulty, t_x,t_y,t_zThe measurement difficulty is small, a common person can also measure by using a common tool, r_x,r_y,r_zThe measurement difficulty of (2) is large, and the measurement is basically impossible for ordinary people. From the requirement for precision, t_x,t_yThe effect of the error of (2) is small, the representation on the ring view is a fine translation; t is t_zSecond, the appearance on the ring view is a change in scale; r is_x,r_y,r_zThe error of (a) is very much affected, which may result in distortion of the image, which is not acceptable to the human eye. Based on the high degree of adaptability of the human eye, t_x,t_y,t_zThe final display effect and the guarantee of the safety cannot be influenced by the tiny error, and the only requirement is t of four cameras_x,t_yThe incremental error of (a) must be consistent, r_zThe multiplication errors of (a) must be consistent.

The existing on-line calibration scheme based on lane line detection has the defects of inaccurate lane line length and width, non-alignment of adjacent lane lines, difficult calibration of left and right cameras, potential safety hazards in the calibration process (generally required on an expressway) and the like.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a processor for calibrating an automobile environment-friendly system, which has low environmental requirements and low labor cost.

The purpose of the invention can be realized by the following technical scheme:

a processor for calibrating an automotive look-around system, said automotive look-around system comprising a plurality of fisheye cameras, said processor being adapted to perform the following computer program steps to obtain a rotation matrix Rt for each camera:

when the automobile is static, observation vectors v1, v2 and v3 from the camera to three vertexes of a known marker in a camera coordinate system are calculated according to pixel coordinates of the three vertexes on a shot image, connecting lines of the three vertexes of the marker form an equilateral triangle and are attached to the horizontal ground, observation distances s1, s2 and s3 from the camera to the three vertexes in the camera coordinate system are solved, and the distances between the three vertexes meet d₁₂＝d₂₃＝d₃₁D, wherein d is the side length of the equilateral triangle, and then the coordinates of the three vertexes under the camera coordinate system are obtained, so as to calculate the normal vector of the equilateral triangle, wherein the preferred solving method of the observation distances s1, s2 and s3 is as follows: giving initial values to s1, s2 and s3, determining the positions p1, p2 and p3 of the three vertexes under the camera coordinate system, and using d₁₂、d₂₃、d₃₁The sum of the squared errors with d to construct the energy function E, E ═ d₁₂–d)²+(d₂₃–d)²+(d₃₁–d)²Obtaining convergence values of s1, s2 and s3 by adopting an iterative algorithm, obtaining a normal vector of an equilateral triangle according to the convergence values of the observation distances s1, s2 and s3 in order to enable the convergence values to be correct, and calculating the height t of the camera under the vehicle coordinate system_zWill t_zComparing the measured height with the prior measured height of the camera, and if the comparison result is greater than the acceptable range, determining that the measured height is within the acceptable rangeRe-assigning initial values to s1, s2 and s3 and iteratively calculating convergence values until the comparison result is acceptable;

when the automobile makes forward linear motion along the X axis of the vehicle coordinate system, the coordinates of a certain vertex of a marker at two different moments under the camera coordinate system are respectively obtained, and the X axis unit vector of the camera coordinate system is calculated according to the difference of the two coordinates;

calculating a rotation matrix Rt of the camera: rt ═ X, X × Z, Z (X, X × Z, Z)^-1Wherein, X represents the unit vector of X axis of the vehicle coordinate system, Z represents the unit vector of Z axis of the vehicle coordinate system, X, Z are the unit vectors corresponding to X, Z under the camera coordinate system, X is actually parallel with the normal vector of the equilateral triangle.

After obtaining the rotation matrix Rt, the processor is further configured to execute the following computer program steps to obtain a horizontal position parameter t of the camera_xAnd t_y：

When the automobile turns around a turning center point C (0, C,0) under the vehicle coordinate with a fixed radius, acquiring coordinates of the peak of the marker at different moments under a camera coordinate system, acquiring multiple groups of data, converting the data into coordinates under the vehicle coordinate system according to the current calibration parameters, and calculating a horizontal position parameter t of the camera according to a principle equation that the distance from the same peak to the turning center point is kept unchanged if C is a known quantity meeting the requirement of precision_xAnd t_yError d of_xAnd d_yD is mixing_xAnd d_yRespectively adding the horizontal position parameters to the prior values of the horizontal position parameters of the camera to obtain the horizontal position parameter t_xAnd t_y(ii) a If c is an unknown quantity or a known quantity with the accuracy not meeting the requirement, a circle is determined by shooting the same vertex for three times and adopting a three-point method, and the difference between the coordinate value of the x axis of the circle center and 0 is taken as a horizontal position parameter t_xError e of_xE is to be_xAdding the horizontal position parameter t to the prior value of the horizontal position parameter of the camera to obtain the horizontal position parameter t_xNamely, the precise solution of the Y-axis direction is abandoned, but the precise solution of the X-axis direction is reserved, and the Y-axis direction is reserved for the subsequent overlapping region image registration technology.

Preferably, the vehicle is turned with a minimum turning radius, since in this state the turning radius is directly available; the collected data sets were: the coordinates of the same vertex at least 3 different times, or the coordinates of at least 2 different vertices at least 2 different times.

Obtaining horizontal position parameters t of each camera_xAnd t_yThe processor is then further configured to execute the following computer program steps to update the horizontal position parameter t of each camera_xAnd t_yAnd (3) keeping public errors and eliminating private errors:

when the automobile is static, acquiring images of two cameras i and j shooting the vertex P of the same marker, and respectively calculating the coordinate P of the vertex P under the automobile coordinate system by using the respective calibration parameters of the obtained camerasⁱAnd P^jChanging the automobile position or the top point of the marker, collecting at least two groups of coordinate data, substituting the coordinate data into the following formula, and simultaneously calculating the private error b of two cameras i and jⁱAnd b^j：

Pⁱ–P^j＝bⁱ-b^j

Adding the private errors to the prior values of the horizontal position parameters of the corresponding cameras respectively to obtain updated horizontal position parameters t_xAnd t_y。

Compared with the prior art, the invention has the following advantages:

(1) the program has low requirements on environment and technology in the calibration process, the vehicle does not need to be driven to a specific position, the vehicle can be driven to a common horizontal ground, the image acquired by the fisheye camera is simple, and the equilateral triangle for calibrating the marker is easy to arrange.

(2) Since the calibration work is an occasional work for the 4S shop, the staff is unlikely to be trained professionally, it is difficult to create a skillful experience, and the factory calibration solution is too demanding for the staff of the 4S shop. The scheme required by the program is simple to operate, does not have strict positioning requirements, is beneficial to maintenance of a 4S shop of the BEV around-looking system, even facilitates independent operation of a driver, is high in safety, and is beneficial to enabling the BEV around-looking system to move from a front-loading market to a rear-loading market and from an incremental market to a stock market.

(3) The equilateral triangle primitive scheme is adopted, and a series of advantages of the equilateral triangle primitive scheme are utilized: firstly, the vertexes and the edges of the triangles are not strictly distinguished in sequence, so that the detection is convenient; secondly, the environment identification degree of the triangular patterns is higher, few triangular patterns which can generate interference exist in a common field, the cost for detecting the triangular patterns by using an image algorithm is lower than the cost for detecting other patterns, and the error rate is lower; thirdly, due to the stability of the triangular structure, the complexity of the plane iterative algorithm is reduced, the risk that the algorithm converges to a wrong result is reduced, and the possibility that the algorithm converges to a non-plane is avoided; finally, the manufacturing (drawing) process of the equilateral triangle is simple and accurate.

(4) The potential prior information is fully utilized, the parameters sensitive to human eyes and the parameters insensitive to human eyes are distinguished, the calibration scheme is simple, the required calibration space is small, the calibration precision is high, the anti-interference performance is higher, and the requirement on the skill of an operator is greatly reduced to the extent that the operator can do the calibration.

Drawings

FIG. 1 is a diagram showing the arrangement of markers in the flow (1) of example 4;

FIG. 2 is a normal view from the camera to the vertex of the triangular marker in accordance with example 4;

FIG. 3 is a perspective view of a triangle observed in the coordinate system of the camera in example 4;

FIG. 4 is the movement of the apex of the triangle with respect to the turning center when the vehicle turns at the minimum turning radius according to example 4;

FIG. 5 is a layout of the identifiers of example 4 in an overlapping region;

FIG. 6 is a calibration flowchart of example 4.

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.

Example 1

A processor for calibrating an automotive look-around system comprising a plurality of fisheye cameras, the processor being configured to perform the following computer program steps to obtain a rotation matrix Rt for each camera:

when the automobile is static, observation vectors v1, v2 and v3 from the camera to three vertexes of a known marker in a camera coordinate system are calculated according to pixel coordinates of the three vertexes on a shot image, connecting lines of the three vertexes of the marker form an equilateral triangle and are attached to the horizontal ground, observation distances s1, s2 and s3 from the camera to the three vertexes in the camera coordinate system are solved, and the distances between the three vertexes meet d₁₂＝d₂₃＝d₃₁D, wherein d is the side length of the equilateral triangle, and then the coordinates of the three vertexes under the camera coordinate system are obtained, so as to calculate the normal vector z of the equilateral triangle, wherein the preferred solving method of the observation distances s1, s2 and s3 is as follows: giving initial values to s1, s2 and s3, determining the positions p1, p2 and p3 of the three vertexes under the camera coordinate system, and using d₁₂、d₂₃、d₃₁The sum of the squared errors with d to construct the energy function E, E ═ d₁₂–d)²+(d₂₃–d)²+(d₃₁–d)²Obtaining convergence values of s1, s2 and s3 by adopting an iterative algorithm, obtaining a normal vector z of an equilateral triangle according to the convergence values of the observation distances s1, s2 and s3 in order to enable the convergence values to be correct results, and calculating the height t of the camera under the vehicle coordinate system_z(generally, in a vehicle coordinate system, the direction of the Z axis is vertically downward, the direction of the X axis is directed straight ahead, and the direction of the Y axis is directed straight right), and t is set_zComparing the calculated height with the prior measurement height of the camera, if the comparison result is larger than the acceptable range, assigning initial values to s1, s2 and s3 again and calculating the convergence value in an iterative manner until the comparison result is acceptable;

calculating a rotation matrix of a cameraRt：Rt＝(x,x×z,z)*(X,X×Z,Z)^-1Wherein X represents a unit vector of an X axis of a world coordinate system, and Z represents a unit vector of a Z axis of a vehicle coordinate system.

When the automobile turns around a turning center point C (0, C,0) under the vehicle coordinate with the minimum turning radius, acquiring coordinates of the peak of the marker at different moments under a camera coordinate system, acquiring multiple groups of data (which can be coordinates of the same peak at least 3 different moments or coordinates of at least 2 different peaks at least 2 different moments), converting the data into the vehicle coordinate system according to the current calibration parameters, and calculating a horizontal position parameter t of the camera according to a principle equation that the distance from the same peak to the turning center point is kept unchanged_xAnd t_yError d of_xAnd d_yD is mixing_xAnd d_yRespectively adding the horizontal position parameters to the prior values of the horizontal position parameters of the camera to obtain the horizontal position parameter t_xAnd t_y。

Pⁱ–P^j＝bⁱ-b^j

In the turning center point C (0, C,0), the turning radius C is a known quantity or an unknown quantity, and when C is an unknown quantity, the same vertex is photographed three times, and a circle is determined by a three-point method to obtain the radius of the circle as the turning radius C.

The processor of the embodiment may be one or more of a combination of a video capture card and an industrial personal computer integrated in a fisheye camera, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, and a microprocessor.

Example 2

A calibration device for an automobile panoramic system comprises a processor and a marker arranged on a horizontal ground, wherein the marker is an equilateral triangle patch or an equilateral triangle frame attached to the horizontal ground.

The core idea of the invention is to adopt an equilateral triangle as a basic primitive, which breaks through the customary idea of using lines or squares for calibration in most of the prior calibration schemes. Equilateral triangles have some particular advantages: firstly, a user can use three equal-length line segments to make an absolute accurate equilateral triangle and then cut the equilateral triangle into plates, or the equilateral triangle can be easily painted with a certain color on the corresponding area of the ground, but any other shapes do not have the condition (the user does not need to additionally purchase accessories for calibrating the own looking-around system, but can use a ruler and a rope to operate by himself). Secondly, three vertexes and three sides of the triangle have no strict order requirement, and the three vertexes can uniquely determine a plane, so that the characteristic can avoid many ambiguities in the iterative process of the algorithm and can quickly converge to an effective solution. More than three points of primitives may face more problems in the optimization process, such as convergence to a locally optimal solution, or convergence to a non-planar result.

In summary, the method is the same whether the 4S shop is calibrated or the user calibrates himself, and the method can be any method as long as finding a flat area with the least texture, laying (drawing) a plurality of equilateral triangles with fixed side length, and the triangles have no strict requirements on position and angle. Then, the calibration can be finished by making straight lines and turns according to certain requirements.

Note that: in practical implementation, the shape of the calibration object is not limited to be triangular. Since the normal vector calculation algorithm mentioned in this patent does not specifically require that the target is an equilateral triangle, it is sufficient to clearly distinguish the distances between the three vertices. If other patterns are used and then the particular triangular sub-components are separated from the pattern, the method described in this patent can still be used. In this case, the same shall be considered within the scope of protection of the present patent application.

Example 3

An automobile all-round looking system with the calibration device of embodiment 2 comprises the calibration device and a plurality of fisheye cameras. The fisheye cameras can be four ultra-wide angle fisheye cameras arranged on the front, the back, the left and the right of the vehicle body, and during work, the four cameras simultaneously collect images around the vehicle, and finally form a seamless 360-degree panoramic top view around the vehicle through distortion reduction of an image processing unit → visual angle conversion → image splicing → image enhancement. The panoramic image and the single-view image of any one of the two images can be displayed, and the position and the distance of the obstacle can be accurately positioned by matching with the scale line. The calibration is carried out before the vehicle formally enters the road, as shown in figure 1, a top view of a certain arrangement mode of a triangle of the marker is shown, the triangle is also arranged at the front, the back, the left and the right of the vehicle body and is close to the camera as much as possible, the visibility is ensured, and the arrangement position and the angle are not strictly required.

Example 4

The embodiment is a panoramic system calibration method matched with embodiments 1 to 3, and before calibration, non-distortion correction is firstly performed on a fisheye camera. After correction of the non-distortion, the triangles can be detected by line detection and the vertices of the triangles can be positioned more accurately by lines than by corners.

The following is the calibration procedure:

(1) when the vehicle is stationary, the four triangles are swung at the positions close to the cameras

The four triangles are placed at approximate positions according to the top view angle of fig. 1, and the positions and angles of the four triangles are not strictly required, so long as three vertexes of the triangles are completely in the image. Fig. 2 is a normal view angle and fig. 3 is a perspective spatial view of a triangle as viewed in the camera coordinate system. In fig. 3, the origin O corresponds to the position of the camera, P1, P2 and P3 correspond to three vertices of a triangle, and the plane corresponding to the triangle is the ground. v1, v2, v3 correspond to the camera's observation vectors to the three vertices, which are known and calculated from the pixel positions of the three vertices on the image. The distances of the cameras from the three vertices, S1, S2, S3, are unknown. d12, d23, d31 correspond to the three sides of the triangle.

Let the coordinates of the triangle on the image be (Pix1, Pix2, Pix3), by which the camera's observation vectors (v1, v2, v3) to the three vertices in the camera's coordinate system can be calculated (known). Let (p1, p2, p3) be the positions of the triangle vertices observed in the camera coordinate system and (s1, s2, s3) be the observation distances (unknown). (p1, p2, p3) are unknown, but as long as (s1, s2, s3) are known, (p1, p2, p3) can be calculated. The three observed distances (s1, s2, s3) are solved below so that the distances between the three vertices (p1, p2, p3) satisfy: d₁₂＝d₂₃＝d₃₁Where d is the side length of an equilateral triangle.

Three equations can be listed based on the above conditions, but the system is difficult to solve analytically, and can be expressed as d₁₂,d₂₃,d₃₁And d to construct an energy function E (d) of (s1, s2, s3)₁₂–d)²+(d₂₃–d)²+(d₃₁–d)²F (s1, s2, s3), and then iteratively solved.

This energy formula is similar to the energy storage formula of a spring (if the connecting lines between p1, p2, p3 are imagined as a non-bendable spring), so the above energy formula can not be directly iterated, and a more intuitive physical model-based iteration scheme is provided:

in the camera coordinate system, the three vectors v1, v2, v3 are known and fixed. By assigning initial values to s1, s2, s3, p1, p2, p3 and their distances d are uniquely determined₁₂,d₂₃,d₃₁In the initial state, d is generally₁₂≠d₂₃≠d₃₁Imagine the connecting lines between the three vertices P1, P2, P3 as a set of rings on three supports v1, v2, v3, P1, P2, P3 as springs, which cannot bend but only stretch, have a length d when it is in a relaxed state, produce a pulling force when it is greater than d, produce a repelling force when it is less than d, and slide the rings on the supports under the action of the pulling force or the repelling force. P_iPoint pair P_jThe force generated by the point is denoted f_ij,f_ijRepulsion when greater than 0 and pulling when less than 0, it is evident that_ij＝f_ji。

Suppose that the forces generated by the P1 point and the P2 point on the P3 point are respectively f₁₃And f₂₃The forces projected on the support V3 can be calculated as F₁₃And F₂₃The current comprehensive acting force F3 ═ F at the point P3₁₃+F₂₃Similarly, the combined forces F1, F2, F3 currently experienced by each point, which are along the direction of the stent, are calculated. Adding a small learning rate to the force to make P1, P2 and P3 slightly displace under the action of force, and repeating until convergence, wherein d is in a convergence state₁₂＝d₂₃＝d₃₁＝d。

It is noted that on a complex primitive, such as a square, we face the following problems: the algorithm may converge on a non-plane, or although the algorithm converges on a plane, there are many local extreme points and many convergence results, which makes the subsequent processing very troublesome. The convergence result of an algorithm based on equilateral triangle primitives is always on a plane and at most converges to three different local minima, and in our practical experiments the algorithm almost always converges to the correct result.

Assuming convergence to a wrong result, the result can be excluded below using a priori information of the camera height. Because the observation distances from the three vertexes to the camera are knownFrom (s1, s2, s3), the plane of the triangle and the normal vector z of the plane are uniquely determined, and the height t of the observation position is calculated by the normal vector z of the plane_z(t_zIs the height of the camera and is also the vertical height from the observation point O in the figure to the plane of the triangle). If the height is significantly different from the a priori measured height of the camera, the result can be rejected and the initial value is reselected for iteration. Typically, a unique correct solution can be obtained through this step. Then, knowing that the rotation matrix Rt maps the Z-axis unit vector Z in the world coordinate system to (0,0,1) to the triangular normal vector Z in the camera coordinate system, where Z and Z are both known, we have obtained the first expression of the rotation matrix Rt: and Rt Z is Z.

The X-axis unit vectors (1,0,0) in the world coordinate system are determined below, and the Y-axis unit vectors (0,1,0) correspond to the vectors X and Y in the camera coordinate system, respectively.

(2) Steering wheel is righted to make vehicle go straight

And (3) enabling the vehicle to move straight for a short distance, and if the vehicle cannot move straight back and forth, ensuring that the triangle primitives are always in the corresponding camera view. Assume that the ground position at which the triangle is detected at time T0 is T⁰(P1⁰,P2⁰,P3⁰) At time T1, the position of the triangle is detected (tracked) as T¹(P1¹,P2¹,P3¹) Corresponding to a position t in the camera coordinate system⁰(p1⁰,p2⁰,p3⁰) And t¹(p1¹,p2¹,p3¹). Since the vehicle is moving straight, through P1⁰-P1¹The resulting vector VP is clearly parallel to the X-axis of the vehicle coordinate system. Correspondingly, let p1 be in the camera coordinate system⁰-p1¹Then we know the rotation matrix Rt to map VP to VP. By multiplying by a coefficient, VP is changed to the X-axis unit vector X (1,0,0) of the world coordinate system, VP is changed to vector X in the camera coordinate system, equivalent: rt X ═ X. Use (p 2)⁰,P2¹)，(p3⁰,P3¹) The effect of doing this is the same, so that multiple points of observation can be integrated,the result is made more accurate.

The corresponding vectors Z and X in the camera coordinate system corresponding to the positive directions of the Z axis and the X axis in the vehicle coordinate system are obtained, and if the rotation matrix is Rt, the following are obtained:

Rt*(X,X×Z,Z)＝(x,x×z,z)

Rt＝(x,x×z,z)*(X,X×Z,Z)^-1

where X Z is the outer product of vectors X and Z, and X Z is the outer product of X and Z, this term ensures that the calculation result is a rotation matrix, rather than a composite matrix of rotation and reflection transformations. To this end, we have obtained the rotation matrix Rt and the height t of the camera_zThere remain two parameters t_xAnd t_yUsually, the two parameters can meet the precision requirement by directly using the prior value at the time of factory production, but still need to be calibrated in the case of after-installation. In a state where the vehicle is running straight, it is impossible to calibrate t_xAnd t_yTwo parameters. If the two parameters are further calibrated, the turning radius of the vehicle needs to be known, and then a nonlinear transformation condition is created by turning with the maximum radius. Of course, if the minimum turning radius of the vehicle is not known, t_xAnd t_yAlready with a certain accuracy, t is used in step (4)_xAnd t_yThen the registration of four cameras in an overlapping region is carried out to minimize the measurement error so as to improve t_xAnd t_yEnsures that the four images are matched in the overlapping area.

(3) Vehicle turning with minimum turning radius

If we know the minimum turning radius of the vehicle and turn it with the minimum turning radius, we can go back and forth repeatedly with a small amplitude. Let the coordinates of the turning center C be (0, C,0), and divide the following into two cases a and B.

A. The accuracy of the minimum turning radius c is sufficient

Fig. 4 shows the movement of the apex of the triangle relative to the center of the turn observed when the vehicle is turning at a minimum turning radius. P is any vertex of the detected triangle, P: (t0) and P (t1) are their corresponding ground positions at times t0 and t 1. (since we have calculated the rotation matrix Rt, and the height of the camera t_zFor t_xAnd t_yWe temporarily use a priori measurements, thus obtaining a complete mapping formula, so it is feasible to map points on the image to coordinates on the ground)

Since C is known and accurate, the distance P (t0) to C should be equal to the distance P (t1) to C, let us assume that the coordinates of P (t0) are (P)_x ⁰,P_y ⁰0), the coordinates of P (t1) are (P)_x ¹,P_y ¹0), and the current external reference t_xAnd t_yRespectively having an error d_xAnd d_yThen, there are:

((P_x ⁰+d_x)-0)²+((P_y ⁰+d_y)-c)²＝((P_x ¹+d_x)-0)²+((P_y ¹+d_y)-c)²

to simplify the above formula, the quadratic term is reduced, and the result is d_xAnd d_yThe first order expression of (a). With such an expression, d can be solved for two_xAnd d_yHowever, our triangle has three vertexes, and more points can be selected during the vehicle turning process to form an oversaturation expression, and the solution with the minimum error sum of squares is solved by using the least square method. D obtained by decomposition_xAnd d_yAdding to the original t_xAnd t_yIn this way, accurate results can be obtained.

To this end, six external parameters T (T) have been calculated for each camera separately_x,t_y,t_z) And R (R)_x,r_y,r_z)

B. The minimum turning radius c is unknown, or the accuracy is not sufficient

At this time t_yCannot get an exact solution, but t_xIs possible. However, the algorithm is different, and continuous tracking of the vertices of the triangle is required to obtain the positions of at least three sequential points. Since the vehicle is turning, three points are not likely to be at oneThe bars are straight and therefore a unique circle can be determined. The accuracy of the circle center can be higher by performing superposition calculation on different observation positions of different vertexes. The difference between the circle center position and the theoretical turning center (0, c) of the vehicle is t_xAnd t_yCurrent error e of_xAnd e_y. Since c is not accurate enough, only the X-direction parameters can be updated here: t is t_x＝t_x+e_x.

(4) Image registration of the overlap region, putting four triangles in the overlap region

Regardless of whether step 3 is performed in the form of a or B, step 4 herein may be used to further increase t_xAnd t_yThe accuracy of (2). Four cameras, numbered 1,2,3,4, each having a pair of position parameters t_xAnd t_yTo be optimized, a total of 8 parameters to be optimized, t¹(t_x ¹,t_y ¹),t²(t_x ²,t_y ²),t³(t_x ³,t_y ³),t⁴(t_x ⁴,t_y ⁴) Let T¹,T²,T²,T⁴Are their precise values, eⁱ＝tⁱ–TⁱIs the current error. E is to beⁱDecomposed into common errors cⁱAnd private error bⁱ，eⁱ＝cⁱ+bⁱCommon error is consistent for four cameras, c¹＝c²＝c³＝c⁴C, and the private errors of the four cameras are independent of each other:

t_x ¹＝T_x ¹+c_x+b_x ¹；t_y ¹＝T_y ¹+c_y+b_y ¹；

t_x ²＝T_x ²+c_x+b_x ²；t_y ²＝T_y ¹+c_y+b_y ²；

t_x ³＝T_x ³+c_x+b_x ³；t_y ³＝T_y ¹+c_y+b_y ³；

t_x ⁴＝T_x ⁴+c_x+b_x ⁴；t_y ⁴＝T_y ¹+c_y+b_y ⁴；

their private error b can be determined by using image registration algorithm of the overlapping regionⁱ(b_x ⁱ,b_y ⁱ) Clean up, thereby converting these 8 random parameters (8 degrees of freedom) into two random parameters c (c)_x,c_y) (two degrees of freedom), and theoretically, the variance of this remaining common error must be smaller than the calibration parameter t of the previous four camerasⁱ(t_x ⁱ,t_y ⁱ) The mean of the variance of (a). The process is as follows:

according to the illustration in fig. 5, four triangles are respectively placed in four overlapping regions, so that each triangle can be simultaneously observed by two adjacent cameras, and there is no precise requirement on the position and angle of the triangle. Since the current calibration parameter is other than t_xAnd t_yBesides a small amount of errors, other parameters are accurate, and after the triangle on the image is projected into the ground coordinates, the triangle does not have obvious deformation and only slightly translates, so that the same triangle observed by different cameras can be accurately matched.

Let P be the ground coordinates of a vertex P of any triangle observed in two adjacent cameras i and jⁱAnd P^jThen, there are: pⁱ–P^j＝eⁱ–e^j＝(c+bⁱ)-(c+b^j)＝bⁱ-b^j

Any number of points can be obtained by transforming the positions of the triangles, so that the above expression can be sufficiently plural, juxtaposed them, and b is calculated by the least square method¹,b²,b³,b⁴Thus obtaining updated calibration parameters:

t_x ¹-b_x ¹＝T_x ¹+c_x；t_y ¹-b_y ¹＝T_y ¹+c_y；

t_x ²-b_x ²＝T_x ²+c_x；t_y ²-b_y ²＝T_y ²+c_y；

t_x ³-b_x ³＝T_x ³+c_x；t_y ³-b_y ³＝T_y ³+c_y；

t_x ⁴-b_x ⁴＝T_x ⁴+c_x；t_y ⁴-b_y ⁴＝T_y ⁴+c_y；

the new parameters are the same for the four cameras by subtracting the private error of each camera on the original basis and only remaining the public error, so that the requirements are generally met.

The whole calibration process can be completed in one go as long as a proper software interaction flow is designed. The calibration process can be generally summarized as: (1) placing a triangle as shown in figure 1, enabling the vehicle to go straight back and forth in a small amplitude, starting to collect the map, waiting for the software to feed back successfully (2) enabling the vehicle to turn at the maximum turning radius, driving in a small amplitude, starting to collect the map, waiting for the software to feed back successfully (3) enabling the vehicle to be static, rearranging the triangle primitives in an overlapping area as shown in figure 5, starting to collect the map, waiting for the software to feed back successfully, and then ending.

Claims

1. A processor for calibrating an automotive look-around system, said automotive look-around system comprising a plurality of fisheye cameras, characterized in that said processor is adapted to execute the following computer program steps to obtain a rotation matrix Rt for each camera:

when the automobile is static, acquiring observation vectors v1, v2 and v3 from the camera to three vertexes of a known marker in a camera coordinate system, wherein three vertex connecting lines of the marker form an equilateral triangle and are attached to the horizontal ground, and calculating normal vectors of the equilateral triangle in the camera coordinate system according to the side length and the space geometric relationship of the triangle;

when the automobile makes forward linear motion along the X axis of the vehicle coordinate system, the coordinates of a certain vertex of a marker at two different moments under the camera coordinate system are respectively obtained, and the corresponding X axis unit vector of the vehicle coordinate system under the camera coordinate system is calculated according to the difference of the two coordinates;

calculating a rotation matrix Rt of the camera: rt ═ X, X × Z, Z (X, X × Z, Z)^-1X, Z represents the unit vector of the X-axis and the unit vector of the Z-axis of the vehicle coordinate system, and X and Z represent the unit vectors corresponding to X, Z in the camera coordinate system.

2. The processor according to claim 1, wherein the observation vectors v1, v2, v3 from the camera to the three vertexes in the camera coordinate system are calculated according to pixel coordinates of the three vertexes on the shot image when the car is stationary.

3. The processor according to claim 1, wherein when the automobile is stationary, the normal vector z of the plane of the marker in the camera coordinate system is calculated by: solving the observation distances s1, s2 and s3 from the camera to the three vertexes under the camera coordinate system to enable the distances among the three vertexes to meet d₁₂＝d₂₃＝d₃₁D, wherein d is the known side length of the equilateral triangle, and then the coordinates of the three vertices in the camera coordinate system are obtained, so that the normal vector z of the equilateral triangle is calculated.

4. The processor of claim 3, wherein the observed distances s1, s2, s3 are solved by: giving initial values to s1, s2 and s3, determining the positions p1, p2 and p3 of the three vertexes under the camera coordinate system, and using d₁₂、d₂₃、d₃₁The sum of the squared errors with d to construct the energy function E, E ═ d₁₂–d)²+(d₂₃–d)²+(d₃₁–d)²An iterative algorithm is used to obtain convergence values for s1, s2, and s 3.

5. The processor of claim 4, wherein the camera height t in the vehicle coordinate system is calculated after obtaining the normal vector z of the equilateral triangle according to the convergence values of the observation distances s1, s2 and s3_zWill t_zAnd comparing the calculated height with the prior measurement height of the camera, and if the comparison result is larger than the acceptable range, assigning initial values to s1, s2 and s3 again and calculating the convergence value in an iterative manner.

6. The processor of claim 1, wherein after obtaining the camera rotation matrix Rt, the processor is further configured to execute the following computer program steps to obtain the horizontal position parameter t of the camera_xAnd t_y：

When the automobile turns around a turning center point C (0, C,0) under the vehicle coordinate with a fixed radius, acquiring coordinates of the vertex of the marker at different moments under a camera coordinate system, acquiring multiple groups of data, converting the data into coordinates under the vehicle coordinate system according to the current calibration parameters, and calculating a horizontal position parameter t of the camera according to a principle equation that the distance from the same vertex to the turning center point is kept unchanged if C is a known quantity with the accuracy meeting the set requirement_xAnd t_yError d of_xAnd d_yD is mixing_xAnd d_yRespectively adding the horizontal position parameters to the prior values of the horizontal position parameters of the camera to obtain the horizontal position parameter t_xAnd t_y。

7. The processor of claim 6, wherein the vehicle is turning with a minimum turning radius.

8. The processor of claim 6, wherein the plurality of sets of data collected are: the coordinates of the same vertex at least 3 different times, or the coordinates of at least 2 different vertices at least 2 different times.

9. The processor of claim 6, wherein a horizontal position parameter t of each camera is obtained_xAnd t_yThe processor is then further configured to execute the following computer program steps to update the horizontal position parameter t of each camera_xAnd t_y：

Pⁱ–P^j＝bⁱ-b^j

10. The processor according to claim 6, wherein if c is an unknown quantity or a known quantity with an accuracy not meeting the requirement, a circle is determined by three-point method by photographing the same vertex three times, and the difference between the x-axis coordinate value of the center of the circle and 0 is taken as a horizontal position parameter t_xError e of_xE is to be_xAdding the horizontal position parameter t to the prior value of the horizontal position parameter of the camera to obtain the horizontal position parameter t_x。