CN117372244A - Large scene feature stereo matching method based on two-dimensional array representation - Google Patents

Abstract

The invention relates to a large scene feature stereo matching method based on two-dimensional array representation, which is characterized by comprising the following steps: performing parameter calibration on the stereoscopic fisheye camera system, acquiring internal parameters of the stereoscopic fisheye camera, and acquiring fisheye images under different relative positions of the camera; constructing a extensible combined imaging model based on the spherical imaging model; calculating the mapping relation between the fish eye image pixels and the hemispherical image points; calculating the mapping relation between the hemispherical image point and the geodesic plane point of the regular icosahedron; PCA dimension reduction is carried out on the three-dimensional coordinates of each point on the ground wire plane; establishing a two-dimensional array model and generating an analog transformation element by utilizing optimal rigidity transformation; and carrying out affine transformation on the simulation transformation primitive to obtain a simulation image, extracting characteristic points, generating descriptors and matching the characteristic points. The invention can effectively solve the problem that accurate features are difficult to extract in the peripheral area due to severe nonlinear distortion, and improves the distribution range, quantity and accuracy of fisheye image feature matching.

Description

Large scene feature stereo matching method based on two-dimensional array representation

Technical Field

The invention belongs to the technical field of computer vision, and relates to a feature stereo matching method, in particular to a large scene feature stereo matching method based on two-dimensional array representation.

Background

For a long time, the research object of feature stereo matching mainly uses perspective images. The field of view of a pinhole perspective camera is typically only 40 ° to 60 °, and only very limited local scene information can be obtained by one shot. The pinhole perspective camera can meet the application requirements that part of target areas are limited in a local small range, such as the fields of defect detection, face recognition and the like. However, global information is more valuable for some applications without specific observation targets, such as intelligent driving, forest monitoring, autonomous navigation and obstacle avoidance of robots, pedestrian monitoring and tracking, and other large scene fields. Compared with a pinhole perspective camera, the fisheye camera has a wider field angle, and can generally exceed 140 degrees and even reach 270 degrees, and scene information covered by one fisheye image can be obtained by complex splicing of a plurality of perspective images. Therefore, in large scene applications, especially in the field of robot navigation and autopilot, fisheye cameras have significant advantages, and a wide field of view enables simultaneous visualization of objects in multiple directions.

In order to take advantage of the wide field angle of the fisheye camera, some stereo matching methods based on the characteristics of the fisheye image have been developed in recent years, wherein how to cope with severe nonlinear distortion of the fisheye image is a challenge, and each method needs a straight-forward challenge, and the nonlinear distortion seriously interferes with accurate extraction and description of the characteristics.

The most direct and common method is to correct the fisheye image into a perspective image, and then directly apply the perspective image matching method on the corrected image. Literature "Fiala M, roth g.automatic alignment and graph map building of panoramas [ C ]. IEEE International Workshop on Haptic Audio-Visual Environments and their Applications, ottawa, canada, 2005:104-109", "converts large scene fisheye images into cube surface maps, and then calculates SIFT features on each cube image plane. The literature, "Zhang J, yin X, luan J, et al, an improved vehicle panoramic image generation algorithm [ J ]. Multimedia Tools and Applications,2019,78 (19): 27663-27682". Based on the scanline concept, improves the spherical perspective projection algorithm and corrects the fisheye image, and then extracts SURF features on the corrected image. The literature "Lin Y, gao F, qin T, et al, autopomous aerial navigation using monocular visual-inertial fusion [ J ]. Journal of Field Robotics,2018,35 (1): 23-51." proposes a fisheye camera unmanned aerial vehicle navigation system, but only uses partial scene information of fisheye images: the two sub-areas in the horizontal direction are corrected to perspective images for subsequent processing and most of the field information in the vertical direction is discarded. The literature "Miiller M G, steidle F, schuster M J, et al, robust visual-inertial state estimation with multiple odometries and efficient mapping on an MAV with ultra-wide FOV stereo vision [ C ]. IEEE International Conference on Intelligent Robots and Systems, madrid, spain,2018:3701-3708," proposes a visual odometer and an omnidirectional three-dimensional mapping system, in order to overcome the distortion of the fisheye images, the central area of each fisheye image is corrected to two perspective images, and the peripheral scene information is also discarded. The perspective correction can effectively eliminate nonlinear distortion, but fundamentally eliminates the wide field angle advantage of the fisheye camera, and in addition, objects in the peripheral area of the fisheye image disc can be highly stretched, and objects near the center of the disc can be highly compressed.

Besides perspective correction, some students use equal rectangular images to represent the fisheye images, and the method can keep the wide field of view of the fisheye camera and can also alleviate distortion of the fisheye images to a certain extent. To generate a three-dimensional scene surround view, documents "Lo I, shih K, chen h.image stitching for dual fisheye cameras [ C ]. IEEE International Conference on Image Processing, athens, greene, 2018:3164-3168," correct overlapping field-of-view images of two fisheye cameras to an equal amount of rectangular images, then extract features and complete the entire view stitching process. The literature, "Zhao Q, feng W, wan L, et al sphorp: A fast and robust binary feature on the sphere [ J ]. International Journal of Computer Vision,2015,113 (2): 143-159," improves on the traditional ORB algorithm, constructs new binary features in spherical space for an equivalent rectangular image and proposes a spherical ORB algorithm. Document "Shan Y, li S.Descriptor matching for a discrete spherical image with a convolutional neural network [ J ]. IEEE Access,2018,6:20748-20755." converts fisheye images into equal rectangular images, extracts feature points using an accelerated segmentation feature test algorithm and describes features using a convolutional neural network. Recently, the literature "Pourian N, neores o.an end-to-end framework to high performance geometry-aware multi-scale keypoint detection and matching in fisheye images [ C ]. IEEE International Conference on Image Processing, taiwan,2019:1302-1306." proposes a multi-stage geometric perceptual feature point detection and matching framework to improve descriptor matching accuracy between multiple fisheye images, and adopts an equivalent rectangular image transformation in a global matching stage, so that new distortion is introduced in an edge region, thereby reducing framework performance. The equivalent rectangular image is obtained by taking a spherical model as a basic imaging model and expanding the spherical image, and secondary distortion, especially around spherical poles, is introduced in the process, so that serious image stretching deformation is caused, and the descriptor performance is reduced and the problem of serious mismatching is caused.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a large-scene characteristic stereo matching method based on two-dimensional array representation, which fully utilizes the wide field angle advantage of a fisheye camera and solves the characteristic stereo matching problem of a large-field fisheye image on the premise of not generating obvious image stretching and scene loss.

The invention solves the technical problems in the prior art by adopting the following technical scheme:

a large scene feature stereo matching method based on two-dimensional array representation comprises the following steps:

step 1, carrying out parameter calibration on a stereoscopic fisheye camera system by adopting a circular mark calibration plate, obtaining internal parameters of the stereoscopic fisheye camera, and acquiring fisheye images under different relative positions of the camera;

step 2, taking a spherical imaging model as a basic imaging model, and embedding the unit positive twenty surfaces into the unit spherical surface to construct an extensible combined imaging model;

step 3, based on the extensible combined imaging model, deriving a calculation formula of an azimuth angle sigma and a polar angle rho by using an equal solid angle projection model formula to obtain a mapping relation between fish-eye image pixels and hemispherical image points;

step 4, calculating the mapping relation between the hemispherical image point and each geodesic plane vertex of the regular icosahedron;

step 5, calculating a one-to-one correspondence relationship among the hemispherical image point, the fisheye image pixel and each geodesic plane point by using a positioning formula, and then performing PCA dimension reduction on the three-dimensional coordinates of each point on the geodesic plane;

step 6, establishing a two-dimensional array model, positioning array elements, and carrying out differential local representation according to the position information of each array element on the basis of directional array representation to generate an analog transformation element;

and 7, performing affine transformation on the simulation transformation primitive to obtain a simulation image, extracting characteristic points on the simulation image by adopting a SIFT algorithm, generating descriptors and matching the characteristic points, combining all the matching points into a group, and eliminating repeated matching point pairs.

Further, the spatial point imaging process of the extensible combined imaging model is as follows: defining O-XYZ as a camera coordinate system, and imaging a space point M on an image plane is realized through the following two steps: first, spatial point M is along a vectorLinear projection onto imaging model and obtaining projection points m' and m _Δ At this time, vector +.>Vector->Forming a polar angle ρ, vector +.>And->The projection vector on the XOY plane forms the azimuth angle σ; then, the projection points m' and m _Δ Non-linearly projected onto an imageThe image point m is obtained on the plane.

Further, the specific implementation method of the step 3 is as follows:

step 3.1, adopting an equal solid angle projection model according to internal parameters of the solid fisheye cameraDefining an image point m to an image center o _I Distance d of (2) _ima The relation between the polar angle rho and f is the focal length of the camera;

step 3.2, calculating azimuth angle sigma and polar angle rho corresponding to hemispherical point m';

and 3.3, calculating the coordinates of the image pixel m and the coordinates of the hemispherical image point m' by using the azimuth angle sigma and the polar angle rho.

Further, the calculation formula adopted in the step 3.2 is as follows:

wherein, ψ is the incident-X angle, ζ is the incident-Y angle;

the step 3.3 calculates the coordinates of the image pixel m using the following formula:

the step 3.3 calculates the coordinates of the hemispherical image point m' using the following formula:

wherein R is the radius of the hemispherical surface in the imaging model.

Further, the specific implementation method of the step 4 is as follows:

step 4.1, using projection point m _Δ The ground plane v ₁ ν ₂ ν ₃ Is v ₁ ν ₂ 、ν ₁ ν ₃ 、ν ₂ ν ₃ Calculating Plucker coordinates as directed line segments;

step 4.2, directed line segmentAnd a geodesic plane v ₁ ν ₂ ν ₃ Is->And->Executing an edge operator;

step 4.3, judging the projection point m _Δ The ground plane is located.

Further, the calculation method in the step 4.1 is as follows:

set projection point m _Δ The three vertexes of the ground wire plane are v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ )、ν ₂ ＝(X ₂ ,Y ₂ ,Z ₂ )、ν ₃ ＝(X ₃ ,Y ₃ ,Z ₃ ) Three directed line segmentsAnd->Plucker coordinates +.>And->Calculated by the following formula respectively:

the implementation method of the step 4.2 is as follows: let the origin of the imaging model camera coordinate system be O= (0, 0), the Plucker coordinate of the directed line Om' be Ω _Om' ＝[0,0,-X _m' ,0,-Z _m' ,-Y _m' ]Calculating directed line segments by adopting edge operation operatorsAnd-> And +.>Relation between->And->

The judging method in the step 4.3 is as follows:

if m is _Δ In the ground plane v ₁ ν ₂ ν ₃ In the above, there are three cases:

(1)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following condition is satisfied:

(2)v-shaped filter ₁ ν ₂ ν ₃ In this case, the following conditions are satisfied:

(3)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following conditions are satisfied:

further, the specific implementation method of the step 5 is as follows:

step 5.1, calculating the ground wire plane v ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)

Step 5.2, calculating the projection point m _Δ Is a three-dimensional coordinate of (2);

and 5.3, performing PCA dimension reduction on the three-dimensional coordinates of the geodesic plane points by using the fisheye image, the hemispherical surface and the mapping relation of each geodesic plane point.

Further, the step 5.1 calculates the geodesic plane v by using the following method ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)

The calculation method in the step 5.2 is as follows: by means of a geodesic plane v ₁ ν ₂ ν ₃ Vertex v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ ) Normalized normal vectorCorresponding hemispherical point m' = (X) _m' ,Y _m' ,Z _m' ) The projection point m is obtained by _Δ Is defined by three coordinate values X _Δ 、Y _Δ 、Z _Δ ：

Further, the specific implementation method of the step 6 is as follows:

step 6.1, selecting a reference geodesic plane, taking one vertex of the reference geodesic plane as an origin of a two-dimensional array model coordinate system, and expanding the regular icosahedron model into a two-dimensional array model;

step 6.2, calculating an optimal rigidity transformation matrix based on the two-dimensional array model;

and 6.3, establishing an analog transformation element by using the optimal rigid transformation matrix.

Further, the implementation method of the step 6.2 is as follows:

after the coordinates of the three-dimensional points on the geodesic plane are subjected to PCA dimension reduction, the geodesic plane alpha is set ₁ α ₂ O _p Are respectively at three vertexes ofα ₁ α ₂ O _p The corresponding array element is beta ₁ β ₂ O _p1 Let beta be ₁ β ₂ O _p1 Are respectively +.> To get alpha ₁ α ₂ O _p Conversion to array element beta ₁ β ₂ O _p1 Calculating an optimal rigid transformation matrix according to the following steps; first, alpha is ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Expressed by the following formula:

Tri _α ＝[α ₁ α ₂ O _p ]

Tri _β ＝[β ₁ β ₂ O _p1 ]

let set cen _α 、cen _β Alpha is respectively ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Center of mass, cen _α 、cen _β The coordinates of (2) are obtained by the following formula:

to find the optimal rotation matrix, it is necessary to align alpha ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Redirecting the coordinate data of all points in the plane alpha of the geodesic line ₁ α ₂ O _p And array element beta ₁ β ₂ O _p1 Is located at the origin, redirecting matrix H _→o Calculated by the following formula:

H _→o ＝(Tri_α-cen_α)×(Tri_β-cen_β) ^T

matrix H is aligned by _→o Singular value decomposition is performed and an optimal rotation matrix R is calculated _ET ：

H _→o ＝U _ET ×S _ET ×V _ET

R _ET ＝V _ET ×U _ET ^T

Using an optimal rotation matrix R _ET And centroid coordinates are used for calculating an optimal translation matrix T through the following steps _ET ：

T _ET ＝-R _ET ×cen _α +cen _β ；

And 6.3, establishing an analog transformation element by using the optimal rigid transformation matrix.

And obtaining a corresponding simulation transformation primitive array model by carrying out two-dimensional array representation on the geodesic plane and the neighborhood geodesic plane thereof. After PCA dimension reduction, the ground wire plane alpha is measured ₁ α ₂ O _p All points alpha of (a) ₁ 、α ₂ 、α ₃ 、…、α _n 、O _p May form a matrix αu of 3× (n+1) dimensions:

αU＝[α ₁ α ₂ α ₃ ... α _n O _p ]

wherein N is N;

based on simulation transformation primitive array model, the ground wire plane alpha is measured ₁ α ₂ O _p Performing optimal rigidity transformation to obtain array element beta ₁ β ₂ O _p1 ，β ₁ β ₂ O _p1 The coordinate set βu of all points in (a) is determined by:

βU＝R _ET ×αU+[T ₁ T ₂ T ₃ ... T _n T _n+1 ]

wherein matrix T ₁ 、T ₂ 、T ₃ 、…、T _n 、T _n+1 And T is _ET Equal.

The invention has the advantages and positive effects that:

1. compared with the method for carrying out equivalent rectangular expansion on the fisheye image based on the traditional single spherical model, the method for expanding the fisheye image by using the extensible model does not generate obvious image stretching deformation problem, and is more beneficial to improving the feature matching accuracy.

2. Compared with a perspective correction means, the invention does not generate scene loss problem, and can extract the characteristic of wider distribution no matter in the center area of the disk of the fisheye image with smaller distortion or in the peripheral area with severe distortion, thereby fully utilizing the large scene advantage of the fisheye camera.

3. According to the invention, the analog transformation element is constructed through the optimal rigid transformation, the problem that the characteristics near the edges of the saw teeth cannot be effectively described because of insufficient neighborhood information can be effectively solved, affine transformation is carried out on the analog transformation element, SIFT characteristic matching and repeated characteristic matching are eliminated on an analog image after transformation of each parameter, and compared with the existing fisheye image characteristic matching method, the method can obtain the characteristic matching results with abundant quantity and even thousands of pairs.

4. The invention has reasonable design, establishes a extensible combined imaging model and a two-dimensional array model, weakens fisheye image distortion on the premise of not generating obvious scene loss and image stretching, can effectively solve the problem that accurate features are difficult to extract in a peripheral area because of severe nonlinear distortion, and improves the distribution range of feature matching; the method comprises the steps of constructing analog transformation primitives and calculating feature matching on an analog image after affine transformation, so that the number of feature matching and the matching accuracy can be effectively improved.

Drawings

FIG. 1 is a flow chart of a large scene feature stereo matching method based on two-dimensional array representation of the present invention;

FIG. 2 is a schematic diagram of a malleable combined imaging model constructed in accordance with the present invention;

FIG. 3 is a schematic diagram of a two-dimensional array model constructed in accordance with the present invention;

FIG. 4 is a schematic diagram of an array model of an analog transformation primitive established by the present invention;

FIG. 5 is a schematic diagram of the matching result in a first scenario using the present invention;

FIG. 6 is a schematic diagram of the matching result in a second scenario using the present invention;

fig. 7 is a schematic diagram of a matching result in a third scenario using the present invention.

Detailed Description

Embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

A large scene feature stereo matching method based on two-dimensional array representation, as shown in figure 1, comprises the following steps:

and 1, carrying out parameter calibration on the stereoscopic fisheye camera system by adopting a circular mark calibration plate, acquiring internal parameters of the stereoscopic fisheye camera, and then acquiring fisheye images under different relative positions of the camera.

And 2, establishing a malleable combined imaging model based on the spherical imaging model.

At the bookIn the step, a unit positive twenty-face body is embedded into a unit spherical face structure extensible combined imaging model based on a spherical imaging model, and the imaging model is defined. Because the invention is aimed at the fisheye camera with the view angle not more than 180 degrees, the final extensible combined imaging model is formed by embedding half unit positive twenty planes in a hemispherical plane, and meanwhile, the space point imaging process of the extensible combined imaging model is required to be defined, as shown in fig. 2: O-XYZ is defined as the camera coordinate system. The spatial point M is imaged on the image plane in two steps: m is first along the vectorLinear projection onto imaging model and obtaining projection points m' and m _Δ . At this time, vector +.>Vector->Forming a polar angle ρ, vector +.>And->The projection vector on the XOY plane forms the azimuth angle σ; then points m' and m _Δ Non-linearly projected onto the image plane results in pixel m.

And 3, deriving a calculation formula of the azimuth angle sigma and the polar angle rho by using an equal solid angle projection model formula based on the extensible combined imaging model to obtain a mapping relation between fish-eye image pixels and hemispherical image points.

In this embodiment, the present step includes the steps of:

step 3.1, adopting an equal solid angle projection model according to camera parametersDefining pixel m to image center o _I Distance d of (2) _ima And the relation between the polar angle rho, f is the focal length of the camera.

Step 3.2, calculating azimuth angle sigma and polar angle rho corresponding to hemispherical point m', wherein the formula is as follows:

wherein, ψ is the incident-X angle, ζ is the incident-Y angle.

Step 3.3, calculating coordinates of the hemispherical image point m 'and the image pixel m using the azimuth angle σ and the polar angle ρ, respectively, wherein m' = (X _m' ,Y _m' ,Z _m' )，m＝(x _m ,y _m ) The formula is as follows:

wherein R is the radius of the hemispherical surface in the imaging model.

And 4, calculating the mapping relation between the hemispherical image point and the vertex of each geodesic plane of the regular icosahedron.

In this embodiment, the present step includes the steps of:

step 4.1, using projection point m _Δ The ground plane v ₁ ν ₂ ν ₃ Is v ₁ ν ₂ 、ν ₁ ν ₃ 、ν ₂ ν ₃ The Plucker coordinates are calculated as directed line segments.

Set point m _Δ The three vertexes of the geodesic plane are v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ )、ν ₂ ＝(X ₂ ,Y ₂ ,Z ₂ )、ν ₃ ＝(X ₃ ,Y ₃ ,Z ₃ ) Three directed line segmentsAnd->Plucker coordinates +.>And->Calculated by the following formula respectively:

step 4.2, directed line segmentAnd a geodesic plane v ₁ ν ₂ ν ₃ Is->And->An edge operator is performed.

The origin of the camera coordinate system of the imaging model is O= (0, 0), and the directed line segmentPlucker coordinates Ω _Om' ＝[0,0,-X _m' ,0,-Z _m' ,-Y _m' ]Calculating directed line segment by adopting edge operator>And->And +.>Relation between->And->

Step 4.3, judging point m _Δ The ground plane is located.

If m is _Δ In the ground plane v ₁ ν ₂ ν ₃ In the above, there are three cases:

1)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following condition is satisfied:

2)v-shaped filter ₁ ν ₂ ν ₃ In this case, the following conditions are satisfied:

3)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following conditions are satisfied:

and 5, calculating a one-to-one correspondence relation among the hemispherical image point, the fisheye image pixel and each geodesic plane point by using a positioning formula, and then performing PCA dimension reduction on the three-dimensional coordinates of each point on the geodesic plane.

In this embodiment, the present step includes the steps of:

step 5.1, calculating the ground wire plane v ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)

Ground plane v ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)The method can be obtained by the following formula:

step 5.2, calculating m _Δ Is a three-dimensional coordinate of (c).

By means of a geodesic plane v ₁ ν ₂ ν ₃ Vertex v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ ) Normalized normal vectorCorresponding hemispherical point m' = (X) _m' ,Y _m' ,Z _m' ) The projection point m is obtained by _Δ Is defined by three coordinate values X _Δ 、Y _Δ 、Z _Δ ：

And 5.3, performing PCA dimension reduction processing on the three-dimensional coordinates of the points on the geodesic plane by using the fisheye image, the hemispherical surface and the mapping relation of the points on each geodesic plane on the polyhedron.

And 6, establishing a two-dimensional array model, positioning array elements, and carrying out differential local representation according to the position information of each array element on the basis of directional array representation to generate an analog transformation element.

In this embodiment, the present step includes the steps of:

and 6.1, selecting a reference geodesic plane, taking one vertex of the reference geodesic plane as an origin of a two-dimensional array model coordinate system, and expanding the regular icosahedron model into a two-dimensional array model.

Assuming that the geodesic plane 1 is selected as the reference geodesic plane, a two-dimensional array model is created, as shown in fig. 3, in which one vertex of the reference geodesic plane is taken as the origin of the coordinate system of the two-dimensional array model, the geodesic planes 1, 2, 3, 4, 5, 6, … …, 17, 18, 19, 20 in the regular icosahedron correspond to the array elements 1', 2', 3', 4', 5', 6', … …, 17', 18', 19', 20' in the two-dimensional array model.

And 6.2, calculating an optimal rigidity transformation matrix based on the two-dimensional array model.

Tri _α ＝[α ₁ α ₂ O _p ]

Tri _β ＝[β ₁ β ₂ O _p1 ]

After the coordinates of the three-dimensional point on the geodesic plane are subjected to PCA dimension reduction, alpha is set ₁ α ₂ O _p Are respectively at three vertexes ofThe corresponding array element is beta ₁ β ₂ O _p1 Let beta be ₁ β ₂ O _p1 Are respectively +.>To get alpha ₁ α ₂ O _p Conversion to array element beta ₁ β ₂ O _p1 The optimal rigid transformation matrix is calculated as follows. First, alpha is ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Expressed by the following formula:

Tri _α ＝[α ₁ α ₂ O _p ]

Tri _β ＝[β ₁ β ₂ O _p1 ]

let set cen _α 、cen _β Alpha is respectively ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Center of mass, cen _α 、cen _β The coordinates of (2) can be obtained by the following formula:

to find the optimal rotation matrix, it is necessary to align alpha ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Redirecting the coordinate data of all points in the map, and alpha is caused to be ₁ α ₂ O _p And beta ₁ β ₂ O _p1 Is located at the origin, redirecting matrix H _→o The calculation can be made by the following formula:

H _→o ＝(Tri_α-cen_α)×(Tri_β-cen_β) ^T

for matrix H _→o Singular value decomposition is carried out, and an optimal rotation matrix R is obtained through calculation _ET ：

H _→o ＝U _ET ×S _ET ×V _ET

R _ET ＝V _ET ×U _ET ^T

Using an optimal rotation matrix R _ET And centroid coordinates are used for calculating an optimal translation matrix T through the following steps _ET ：

T _ET ＝-R _ET ×cen _α +cen _β

And 6.3, establishing an analog transformation element by using the optimal rigid transformation matrix.

The jagged edges that may occur during the two-dimensional array representation may be ignored or erroneously extracted due to lack of neighborhood information when computing feature points near the jagged edges. In order to solve the problem, the invention establishes an analog transformation primitive based on a two-dimensional array model. Taking the geodesic plane 1 as an example, the geodesic plane 1 and the neighborhood geodesic planes 2, 5 and 17 are subjected to two-dimensional array representation to obtain a corresponding analog transformation primitive array model, as shown in fig. 4.

After PCA dimension reduction, alpha ₁ α ₂ O _p All points alpha of (a) ₁ 、α ₂ 、α ₃ 、…、α _n 、O _p May form a matrix αu of 3× (n+1) dimensions:

αU＝[α ₁ α ₂ α ₃ ... α _n O _p ]

based on the simulation transformation primitive array model, for alpha ₁ α ₂ O _p Performing optimal rigidity transformation to obtain array element beta ₁ β ₂ O _p1 ，β ₁ β ₂ O _p1 The set of coordinates of all points in (a) is determined by:

βU＝R _ET ×αU+[T ₁ T ₂ T ₃ ... T _n T _n+1 ]

wherein matrix T ₁ 、T ₂ 、T ₃ 、…、T _n 、T _n+1 And T is _ET Equal.

The dimension reduction and optimal rigidity transformation are carried out on the geodesic planes 2, 5 and 17 by the same method, so that the simulation transformation element of the geodesic plane 1 is established. Similarly, analog conversion primitives corresponding to other geodesic planes may be created.

And 7, performing affine transformation on the simulation transformation primitive to obtain a simulation image, extracting characteristic points on the simulation image by adopting a SIFT algorithm, generating descriptors and matching the characteristic points, combining all the matching points into a group, and eliminating repeated matching point pairs.

The specific treatment process of the step is divided into two steps:

step 7.1, affine transformation is carried out on the analog transformation primitive: by changing two parameters that determine the direction of the camera axis: latitude angleAnd longitude angle->Affine transformation is carried out on the simulation transformation element under a plurality of different viewpoints to obtain a simulation image. In this step, affine transformation is performed on the analog transformation element by affine distortions caused by the analog camera optical axis direction change, which depend on the latitude angle +.>And longitude angle->Is a different value of (a). First making angle for analog transformation element>Is followed by an absolute tilt value +.>Is provided. Sampling interval of absolute tilt value t>The absolute tilt value t is +.>Accordingly, the latitude angle +.>Is the value of:longitude angle->Is +.>For each absolute tilt value t, the longitude angle takes the value 0 °,72 °/t,144 °/t,..180 °.

Step 7.2, executing a SIFT matching algorithm: since the distortion in each analog transformation element of the fisheye image is substantially uniform, affine distortion can be approximated. After affine transformation is carried out on the simulation transformation element, a series of simulation images are obtained, feature points are extracted on the simulation images by using a SIFT algorithm, descriptors are generated and matched, finally all matching pairs are combined into a group, repeated point pairs are removed, and the final matching results are shown in figures 5, 6 and 7. In three-dimensional fisheye images in three scenes shown in fig. 5, 6 and 7, each scene comprises four pairs of fisheye images acquired by different camera poses, the left two columns are original three-dimensional fisheye images, the right two columns are matching results obtained by the three-dimensional matching method for large scene features based on two-dimensional array representation, and the matching results can be observed from the images, so that the method can obtain abundant and corresponding accurate feature matching points in a central area with small distortion or a peripheral area with severe distortion.

It should be emphasized that the examples described herein are illustrative rather than limiting, and therefore the invention includes, but is not limited to, the examples described in the detailed description, as other embodiments derived from the technical solutions of the invention by a person skilled in the art are equally within the scope of the invention.

Claims (10)

1. A large scene feature stereo matching method based on two-dimensional array representation is characterized by comprising the following steps: the method comprises the following steps:

step 1, carrying out parameter calibration on a stereoscopic fisheye camera system by adopting a circular mark calibration plate, obtaining internal parameters of the stereoscopic fisheye camera, and acquiring fisheye images under different relative positions of the camera;

step 2, taking a spherical imaging model as a basic imaging model, and embedding the unit positive twenty surfaces into the unit spherical surface to construct an extensible combined imaging model;

step 3, based on the extensible combined imaging model, deriving a calculation formula of an azimuth angle sigma and a polar angle rho by using an equal solid angle projection model formula to obtain a mapping relation between fish-eye image pixels and hemispherical image points;

step 4, calculating the mapping relation between the hemispherical image point and each geodesic plane vertex of the regular icosahedron;

step 5, calculating a one-to-one correspondence relationship among the hemispherical image point, the fisheye image pixel and each geodesic plane point by using a positioning formula, and then performing PCA dimension reduction on the three-dimensional coordinates of each point on the geodesic plane;

step 6, establishing a two-dimensional array model, positioning array elements, and carrying out differential local representation according to the position information of each array element on the basis of directional array representation to generate an analog transformation element;

and 7, performing affine transformation on the simulation transformation primitive to obtain a simulation image, extracting characteristic points on the simulation image by adopting a SIFT algorithm, generating descriptors and matching the characteristic points, combining all the matching points into a group, and eliminating repeated matching point pairs.

2. The large scene feature stereo matching method based on two-dimensional array representation according to claim 1, wherein: the spatial point imaging process of the extensible combined imaging model is as follows: defining O-XYZ as a camera coordinate system, and imaging a space point M on an image plane is realized through the following two steps: first, spatial point M is along a vectorLinear projection onto imaging model and obtaining projection points m' and m _Δ At this time, vector +.>Vector->Forming a polar angle ρ, vector +.>And->In the XOY planeThe projection vector forms an azimuth angle sigma; then, the projection points m' and m _Δ Non-linearly projected onto the image plane gives the image point m.

3. The large scene feature stereo matching method based on two-dimensional array representation according to claim 1, wherein: the specific implementation method of the step 3 is as follows:

step 3.1, adopting an equal solid angle projection model according to internal parameters of the solid fisheye cameraDefining an image point m to an image center o _I Distance d of (2) _ima The relation between the polar angle rho and f is the focal length of the camera;

step 3.2, calculating azimuth angle sigma and polar angle rho corresponding to hemispherical point m';

and 3.3, calculating the coordinates of the image pixel m and the coordinates of the hemispherical image point m' by using the azimuth angle sigma and the polar angle rho.

4. A large scene feature stereo matching method based on two-dimensional array representation according to claim 3, characterized in that: the calculation formula adopted in the step 3.2 is as follows:

wherein, ψ is the incident-X angle, ζ is the incident-Y angle;

the step 3.3 calculates the coordinates of the image pixel m using the following formula:

the step 3.3 calculates the coordinates of the hemispherical image point m' using the following formula:

wherein R is the radius of the hemispherical surface in the imaging model.

5. The large scene feature stereo matching method based on two-dimensional array representation according to claim 1, wherein: the specific implementation method of the step 4 is as follows:

step 4.1, using projection point m _Δ The ground plane v ₁ ν ₂ ν ₃ Is v ₁ ν ₂ 、ν ₁ ν ₃ 、ν ₂ ν ₃ Calculating Plucker coordinates as directed line segments;

step 4.2, directed line segmentAnd a geodesic plane v ₁ ν ₂ ν ₃ Is->And->Executing an edge operator;

step 4.3, judging the projection point m _Δ The ground plane is located.

6. The large scene feature stereo matching method based on two-dimensional array representation according to claim 5, wherein: the calculation method in the step 4.1 is as follows:

set projection point m _Δ The three vertexes of the ground wire plane are v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ )、ν ₂ ＝(X ₂ ,Y ₂ ,Z ₂ )、ν ₃ ＝(X ₃ ,Y ₃ ,Z ₃ ) Three directed line segmentsAnd->Plucker coordinates +.>And->Calculated by the following formula respectively:

the implementation method of the step 4.2 is as follows: let the origin of the imaging model camera coordinate system be O= (0, 0), the Plucker coordinate of the directed line Om' be Ω _Om' ＝[0,0,-X _m' ,0,-Z _m' ,-Y _m' ]Calculating directed line segments by adopting edge operation operatorsAnd (3) with And +.>Relation between->

The judging method in the step 4.3 is as follows:

if m is _Δ In the ground plane v ₁ ν ₂ ν ₃ In the above, there are three cases:

(1)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following condition is satisfied:

(2)v-shaped filter ₁ ν ₂ ν ₃ In this case, the following conditions are satisfied:

(3)v-shaped filter ₁ ν ₂ ν ₃ At this time, the following conditions are satisfied:

7. the large scene feature stereo matching method based on two-dimensional array representation according to claim 1, wherein: the specific implementation method of the step 5 is as follows:

step 5.1, calculating the ground wire plane v ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)

Step 5.2, calculating the projection point m _Δ Is a three-dimensional coordinate of (2);

and 5.3, performing PCA dimension reduction on the three-dimensional coordinates of the geodesic plane points by using the fisheye image, the hemispherical surface and the mapping relation of each geodesic plane point.

8. The large scene feature stereo matching method based on two-dimensional array representation according to claim 7, wherein: the step 5.1 adopts the following calculation to measure the ground wire plane v ₁ ν ₂ ν ₃ Is a normalized normal vector of (2)

The calculation method in the step 5.2 is as follows: by means of a geodesic plane v ₁ ν ₂ ν ₃ Vertex v ₁ ＝(X ₁ ,Y ₁ ,Z ₁ ) Normalized normal vectorCorresponding hemispherical point m' = (X) _m' ,Y _m' ,Z _m' ) The projection point m is obtained by _Δ Is defined by three coordinate values X _Δ 、Y _Δ 、Z _Δ ：

9. The large scene feature stereo matching method based on two-dimensional array representation according to claim 1, wherein: the specific implementation method of the step 6 is as follows:

step 6.1, selecting a reference geodesic plane, taking one vertex of the reference geodesic plane as an origin of a two-dimensional array model coordinate system, and expanding the regular icosahedron model into a two-dimensional array model;

step 6.2, calculating an optimal rigidity transformation matrix based on the two-dimensional array model;

and 6.3, establishing an analog transformation element by using the optimal rigid transformation matrix.

10. The large scene feature stereo matching method based on two-dimensional array representation according to claim 9, wherein: the implementation method of the step 6.2 is as follows:

after the coordinates of the three-dimensional points on the geodesic plane are subjected to PCA dimension reduction, the geodesic plane alpha is set ₁ α ₂ O _p Are respectively at three vertexes ofα ₁ α ₂ O _p The corresponding array element is beta ₁ β ₂ O _p1 Let beta be ₁ β ₂ O _p1 Are respectively +.> To get alpha ₁ α ₂ O _p Conversion to array element beta ₁ β ₂ O _p1 Calculating an optimal rigid transformation matrix according to the following steps; first, alpha is ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Expressed by the following formula:

Tri _α ＝[α ₁ α ₂ O _p ]

Tri _β ＝[β ₁ β ₂ O _p1 ]

let set cen _α 、cen _β Alpha is respectively ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Center of mass, cen _α 、cen _β The coordinates of (2) are obtained by the following formula:

to find the optimal rotation matrix, it is necessary to align alpha ₁ α ₂ O _p 、β ₁ β ₂ O _p1 Redirecting the coordinate data of all points in the plane alpha of the geodesic line ₁ α ₂ O _p And array element beta ₁ β ₂ O _p1 Is located at the origin, redirecting matrix H _→o Calculated by the following formula:

H _→o ＝(Tri_α-cen_α)×(Tri_β-cen_β) ^T

matrix H is aligned by _→o Singular value decomposition is performed and an optimal rotation matrix R is calculated _ET ：

H _→o ＝U _ET ×S _ET ×V _ET

R _ET ＝V _ET ×U _ET ^T

Using an optimal rotation matrix R _ET And centroid coordinates are used for calculating an optimal translation matrix T through the following steps _ET ：

T _ET ＝-R _ET ×cen _α +cen _β ；

And 6.3, establishing an analog transformation element by using the optimal rigid transformation matrix.

And obtaining a corresponding simulation transformation primitive array model by carrying out two-dimensional array representation on the geodesic plane and the neighborhood geodesic plane thereof. After PCA dimension reduction, the ground wire plane alpha is measured ₁ α ₂ O _p All points alpha of (a) ₁ 、α ₂ 、α ₃ 、…、α _n 、O _p May form a matrix αu of 3× (n+1) dimensions:

αU＝[α ₁ α ₂ α ₃ ...α _n O _p ]

wherein N is N;

based on simulation transformation primitive array model, the ground wire plane alpha is measured ₁ α ₂ O _p Performing optimal rigidity transformation to obtain array element beta ₁ β ₂ O _p1 ，β ₁ β ₂ O _p1 The coordinate set βu of all points in (a) is determined by:

βU＝R _ET ×αU+[T ₁ T ₂ T ₃ ...T _n T _n+1 ]

wherein matrix T ₁ 、T ₂ 、T ₃ 、…、T _n 、T _n+1 And T is _ET Equal.