CN113465617A

CN113465617A - Map construction method and device and electronic equipment

Info

Publication number: CN113465617A
Application number: CN202110774668.1A
Authority: CN
Inventors: 宋立博; 汪海; 金忠孝
Original assignee: SAIC Motor Corp Ltd
Current assignee: SAIC Motor Corp Ltd
Priority date: 2021-07-08
Filing date: 2021-07-08
Publication date: 2021-10-01
Anticipated expiration: 2041-07-08
Also published as: CN113465617B

Abstract

The invention provides a map construction method, a map construction device and electronic equipment, wherein key frames in a key frame set in the embodiment are obtained by processing point features and line features of an acquired image according to a preset key frame screening rule, and after the key frame set is obtained, a matching feature combination comprising a first line feature in a target key frame and a second line feature in a common-view key frame is used for generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature. The method and the device not only use the point characteristics, but also introduce the line characteristics when generating the initial local map, and the line characteristics have the characteristic of accurately expressing the environmental information, so that the method and the device can more accurately construct the local map when constructing the local map by using the line characteristics, and improve the accuracy of generating the local map.

Description

Map construction method and device and electronic equipment

Technical Field

The invention relates to the field of vehicle positioning, in particular to a map construction method and device and electronic equipment.

Background

Vehicle simultaneous localization and mapping (SLAM) is a key technology for realizing self-service driving of vehicles. The technology enables the vehicle to be positioned by means of the carried sensor in the moving process of the unknown environment, and meanwhile, the map construction of the surrounding environment of the vehicle is completed.

After the sensor collects the image, the point characteristics of the image are extracted, and local map construction of the surrounding environment of the vehicle is carried out on the basis of the point characteristics. In an environment with sparse texture, such as an indoor parking lot, the number of point features of the image is small, so that the accuracy of local map construction of the surrounding environment of the vehicle is low.

Disclosure of Invention

In view of the above, the present invention provides a map construction method, an apparatus and an electronic device, so as to solve the problem that in an environment with sparse textures, the number of point features of an acquired image is small, so that the accuracy of constructing a local map of the vehicle surroundings is low.

In order to solve the technical problems, the invention adopts the following technical scheme:

a map construction method, comprising:

acquiring a key frame set; the key frame set comprises a plurality of key frames arranged according to a time sequence; the key frame is obtained by processing the point characteristics and the line characteristics of the collected image based on a preset key frame screening rule;

determining the key frame with the earliest corresponding time point in the key frame set as a target key frame, and determining a common-view key frame corresponding to the target key frame;

performing line feature matching operation on the target key frame and the common-view key frame to obtain a matched line feature combination; the match line feature combination comprises a first line feature in the target key frame and a second line feature in the common view key frame;

and generating an initial local map according to a first map line characteristic corresponding to the first line characteristic and a second map line characteristic corresponding to the second line characteristic, and performing nonlinear optimization operation on the initial local map to obtain a local map.

Optionally, generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, includes:

under the condition that the content of the first map line feature is empty and the second map line feature is empty, creating a new map line feature in a target local map, and establishing the corresponding relation between the first line feature and the new map line feature and the corresponding relation between the second line feature and the new map line feature;

setting the feature identifier of the new map line feature as a preset error identifier, and calculating a feature descriptor of the new map line feature to obtain an initial local map;

under the condition that the content of the first map line feature is the same as that of the second map line feature, if the feature identifier corresponding to the first map line feature is a preset error identifier, acquiring an observable frame corresponding to the first map line feature;

performing polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

performing triangulation processing on the first map line feature under the condition that the detection result of the polar plane detection and the detection result of the triangulation detection are both preset results to obtain a three-dimensional position of the first map line feature, and identifying the feature corresponding to the triangulated first map line feature as a preset correct identifier;

based on the three-dimensional location, an initial local map is generated.

under the condition that the content of the first map graph feature is different from the content of the second map graph feature, if the feature identification corresponding to the second map graph feature is a preset error identification and the feature identification corresponding to the first map graph feature is a preset correct identification, projecting the second map graph feature into the target key frame to obtain a projection result;

comparing the projection result with the first map line feature to obtain a comparison result, and deleting the first map line feature to obtain an initial local map under the condition that the comparison result is smaller than a preset threshold value;

setting the second map line feature as the first map line feature when the content of the first map line feature is not empty and the content of the second map line feature is empty;

acquiring an observable frame corresponding to the first map line feature under the condition that the feature identifier corresponding to the first map line feature is a preset error identifier;

based on the three-dimensional location, an initial local map is generated.

setting the first map line feature as the second map line feature when the content of the first map line feature is empty and the content of the second map line feature is not empty;

acquiring an observable frame corresponding to the second map features under the condition that the feature identifiers corresponding to the second map features are preset error identifiers;

performing triangulation processing on the second map feature to obtain a three-dimensional position of the second map feature under the condition that the detection result of the polar plane detection and the detection result of the triangulation detection are both preset results;

based on the three-dimensional location, an initial local map is generated.

Optionally, the key frame generation process:

acquiring an image acquired at the current moment, performing point feature extraction operation on the image to obtain point features, and calculating a feature descriptor of each point feature;

performing line feature extraction operation on the image to obtain line features, and calculating a feature descriptor of each line feature;

performing correlation operation on the line characteristics to obtain correlation line characteristics, and setting correlation identification for the correlation line characteristics;

performing point feature matching operation on the image acquired at the current moment and the image acquired at the previous moment to obtain a point feature matching result, and performing line feature matching operation to obtain a line feature matching result;

and determining whether the image acquired at the current moment meets a preset key frame screening rule or not based on the point feature matching result and the line feature matching result, and determining the image acquired at the current moment as a key frame under the condition of meeting the preset key frame screening rule.

Optionally, performing a correlation operation on the line feature to obtain a correlation line feature, and setting a correlation identifier for the correlation line feature, including:

determining line features, the distance between which and the position of any boundary line is within a preset distance range, based on the position of the image acquired by the line features at the current moment and the position of each boundary line in the image acquired at the current moment, and determining the line features as target line features;

determining the end point position of the target line characteristic according to a preset end point position determination rule;

and determining a plurality of target line features corresponding to the same entity boundary based on the end point position of the target line feature and the relative position relation between the target line feature and the cube center of the cube map corresponding to the image acquired at the current moment, respectively using the target line features as associated line features, and setting an associated identifier for the associated line features.

Optionally, determining whether the image acquired at the current time meets a preset key frame screening rule based on the point feature matching result and the line feature matching result, and determining the image acquired at the current time as a key frame if the image acquired at the current time meets the preset key frame screening rule includes:

acquiring a first number of matched point feature pairs in the point feature matching result;

obtaining a second number of matched line feature pairs in the line feature matching result;

and under the condition that any one of the first number and the second number meets a preset number rule, determining the image acquired at the current moment as a key frame.

A map building apparatus comprising:

the key frame acquisition module is used for acquiring a key frame set; the key frame set comprises a plurality of key frames arranged according to a time sequence; the key frame is obtained by processing the point characteristics and the line characteristics of the collected image based on a preset key frame screening rule;

a key frame determining module, configured to determine an earliest key frame at a corresponding time point in the key frame set as a target key frame, and determine a common-view key frame corresponding to the target key frame;

the matching module is used for performing line feature matching operation on the target key frame and the common-view key frame to obtain a matched line feature combination; the match line feature combination comprises a first line feature in the target key frame and a second line feature in the common view key frame;

and the map generation module is used for generating an initial local map according to a first map line characteristic corresponding to the first line characteristic and a second map line characteristic corresponding to the second line characteristic, and performing nonlinear optimization operation on the initial local map to obtain a local map.

A storage medium comprising a stored program, wherein the program, when executed, controls an apparatus in which the storage medium is located to perform the above-described map construction method.

An electronic device, comprising: a memory and a processor;

wherein the memory is used for storing programs;

the processor calls a program and is used to:

Compared with the prior art, the invention has the following beneficial effects:

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a flowchart of a method for constructing a map according to an embodiment of the present invention;

FIG. 2 is a flowchart of another mapping method according to an embodiment of the present invention;

fig. 3 is a schematic view of a calibration scenario according to an embodiment of the present invention;

fig. 4 is a schematic diagram of an image splitting scene according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an edge marking scene according to an embodiment of the present invention;

fig. 6 is a schematic view of a scene defined by a cube center according to an embodiment of the present invention;

fig. 7 is a scene schematic diagram of a map point and line feature result according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a map building apparatus according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The vision sensor has the advantages of small volume, low cost, easy installation and rich environmental information. And thus is widely used in SLAM.

In the current technical scheme of synchronous mapping and positioning of a monocular camera, tracking and positioning are mainly performed by a method based on point characteristics. The method mainly extracts points in the image as features to be positioned in map building, the point features are easy to extract, and the built map is a point cloud map. However, in an environment with sparse texture, such as an indoor parking lot environment, the wall texture is less, and the point feature is insufficient or unstable. And the map constructed by the method is often not dense enough, and the structure information of the surrounding environment is difficult to accurately depict.

In order to solve the technical problem, the inventor finds that the line features can more accurately express the environmental information compared with the point features, and particularly have obvious advantages in artificial scenes such as streets, indoor parking lots and the like. In addition, the characteristics are less influenced by the environment than the point characteristics, so that the SLAM system can be well assisted under a specific environment, and the robustness and the precision of the system are improved. Therefore, in the embodiment of the present invention, the SLAM technique can be implemented by combining the dot feature and the line feature.

In addition, the inventor finds that a common monocular camera uses a pinhole model, so that a fisheye camera is projected onto a camera plane of the pinhole model after distortion correction, and the size of the view angle of the fisheye camera is not fully utilized. When the vehicle moves, the speed is relatively high, the line features which are close to the camera can be cut off by the image edge or disappear in the image quickly, and the advantages of the line features cannot be fully utilized. Therefore, the inventor further proposes a scheme of fully utilizing the fisheye camera.

Specifically, on the basis of the above, the embodiment of the invention provides a map construction method, which is applied to a processor, and the processor can be an electronic control unit ECU on a vehicle. Referring to fig. 1, the map construction method may include:

and S11, acquiring a key frame set.

In practical applications, the key frame set comprises a key frame sequence, and the key frame sequence comprises a plurality of key frames arranged in a time sequence; the key frame is obtained by processing the point characteristics and the line characteristics of the collected image based on a preset key frame screening rule.

And S12, determining the key frame with the earliest corresponding time point in the key frame set as a target key frame, and determining the common-view key frame corresponding to the target key frame.

Specifically, in practical applications, as the processing time increases, more and more key frames are stored in the key frame set, and the key frames are sorted according to the determined time sequence.

More specifically, one frame with the earliest timestamp is obtained from the key frame set, the ORB descriptors of the three-dimensional points and lines matched in the earliest key frame and the local map are updated, and the common view relationship with the existing frames is updated, wherein the updates are respectively the common view (collaborative map) and minimum spanning tree (spanning tree) relationship for global optimization.

And calculating the rest common-view key frames having the common-view relation with the target key frame, and arranging the rest common-view key frames in a descending order according to the magnitude of the common-view relation degree, wherein the common-view relation degree refers to the feature quantity of the map points observed together in the example.

And S13, performing line feature matching operation on the target key frame and the common-view key frame to obtain a matched line feature combination.

The match line feature combination includes a first line feature in the target key frame and a second line feature in the common view key frame.

In practical application, the point features and the line features between the frames which are in the common view relation with the target key frame by frame are matched according to the arrangement sequence from high to low in the common view relation, and the matching mode in this example uses DBow2 dictionary acceleration matching.

And S14, generating an initial local map according to the first map line characteristic corresponding to the first line characteristic and the second map line characteristic corresponding to the second line characteristic, and performing nonlinear optimization operation on the initial local map to obtain a local map.

Specifically, each line feature in the target key frame is numbered, for each line feature in the target key frame, when the common-view frame queue is matched frame by frame, when each line feature obtains a corresponding match in a common-view frame, polar plane detection, triangulation detection, condition number detection of a triangulation equation set are performed, and processing such as identification change, triangulation, erasure and the like is performed on a visual detection result, in this example, a threshold value of overlapping degree of a triangulation detection line is set to be 0.85, a threshold value of polar plane detection is 5 degrees, and a condition number of the triangulation equation set is 15.

After the initial local map is obtained, nonlinear optimization is carried out on the local map, and optimization items of the local map comprise information such as key frame poses in a sliding window, map point feature three-dimensional coordinates, map line feature endpoint three-dimensional coordinates and the like.

In practical application, a nonlinear optimization tool package g2o is used to perform nonlinear optimization on a local map, the used pose vertex is lie algebra vertex vertexSE3Expmap in a g2o library, both line features and point features define errors by binary edges, the binary edges inherit a virtual class baseunary edge in the library, projection adjustment is performed on a cube map (cube map), the optimization item of the local map includes information such as key frame poses in the sliding window, three-dimensional coordinates of map point features, three-dimensional coordinates of map line feature end points, and the like, and the length of the sliding window is kept at 20 key frames in this example.

The process of graph optimization using the nonlinear optimization tool g2o is as follows: searching a sequence of key frames to be optimized, optimizing the pose of each key frame, wherein the vertex in the nonlinear optimization is a 6-degree-of-freedom vertex expressed by a lie algebra and expressed as a six-dimensional vector, namely a rotational displacement matrix

Wherein, T_cwRepresenting a 4 x 4 rotational displacement matrix, R_cw、t_cwRespectively representing a rotation matrix and a displacement vector, and by the formula of Rodrigues:

R＝I-n·sinθ+n·n^T(1-cosθ)

wherein, R is a 3 × 3 rotation matrix, I is a unit matrix, n is a rotation unit vector, and theta is a rotation angle.

The above-mentioned Rodrigue formula is converted into corresponding lie algebra, and the lie algebra is expressed as [ p ]^T,φ^T]Where φ is the lie algebra rotation vector and p is the displacement vector on the lie algebra.

All map point features and map line features observable by the key frame sequence are established as other vertexes, the vertex of the map point features is a space point represented by a three-dimensional coordinate, and the vertexes corresponding to the map line features are space three-dimensional coordinates corresponding to two end points of the map line features, namely each map line feature corresponds to two map optimization vertexes.

Vertices corresponding to each keyframe and map point features and map line features observable with the keyframeCorresponding vertexes are characterized to form a binary edge, for the binary edge formed by the key frame pose and the map point feature, specifically, the two-dimensional distance error between the map point feature projected to the key frame phase plane and the corresponding pixel point, and for a certain map point P_wCorresponding to the pixel point P in the key frame_uvThe rotational displacement matrix T of the key frame_cwThe corresponding energy equation is:

E_p＝‖P_uv-K·T_cw·P_w‖

E_pand expressing the corresponding value of the point energy equation, and K expresses a projection matrix of the camera.

The binary edge formed by the key frame pose and the map line feature is the vertical distance from the end point of the map line feature projected to the key frame phase plane to the corresponding line feature, namely the energy equation corresponding to the binary edge is as follows:

E_l＝‖F_kl·K·T_cw·l_ed‖₂

wherein, F_klRepresenting the weight of the energy function, K representing the camera projection matrix, l_edRepresenting the end points of the line, E_lAnd the corresponding value of the linear energy equation is represented.

Optimizing the binary edge set, wherein the nonlinear algorithm is a Lavenberg (Levenberg-Marquardt) algorithm, and solving:

(H+λI)Δx_k＝g

wherein H is Hessian matrix, λ is Lagrange multiplier, Δ x_kTo optimize the direction, I is the identity matrix and g is the incremental equation.

The Hessian matrix in the algorithm is approximately expressed by adopting a Jacobian matrix J which is set in advance, namely H is J.J.J^TThen, iterative solution is carried out, and the Jacobian matrix is represented by a Prock expression [ n ] of line characteristics^T,d^T]Derived by derivation, wherein d^TIs the direction vector of a straight line, n^TThe method is characterized in that a normal vector of a plane formed by a straight line and an optical center is used, error side screening is carried out by using an RANSAC mode, and a Huber kernel function is used for controlling the side with larger error.

After nonlinear optimization, detecting the key frame, judging whether the key frame forms a loop or not, and performing overall optimization on global point-line characteristics and tracks, wherein the specific sub-steps are as follows:

1. and comparing point feature descriptors in the key frames by using a DBow2 dictionary, and selecting a closed-loop reference frame by sharing the number of words.

2. And (3) carrying out point feature matching on the reference frame of the closed loop and the key frame, and if the number of the point feature matches exceeds a set threshold, determining that the closed loop is correct, and entering the step 3.

3. And calculating the relative pose of the key frame and the closed-loop reference frame through the matched feature points, correcting the absolute pose of the target key frame, eliminating drift and accumulated errors in a pose graph optimization mode, and correcting the track. And finally outputting the results of the track and map point characteristics and the map line characteristics, specifically referring to fig. 7.

The above embodiment mentioned obtaining the key frame set, and a specific obtaining process is now described, specifically, referring to fig. 2, the method may include:

s21, acquiring the image acquired at the current moment, performing point feature extraction operation on the image to obtain point features, and calculating a feature descriptor of each point feature.

During the moving process of the vehicle, images collected by a monocular camera, particularly a fisheye camera, of the vehicle are acquired. It should be noted that other types of cameras can be used in the present embodiment, and the present invention is not required.

After the image is acquired, the image is corrected to obtain a cubic projection map. The specific correction process is as follows:

referring to fig. 3, the fisheye image is corrected as shown in fig. 3 to obtain a cubic projection map. The size of the cube projection map obtained in this embodiment is 1536 × 1536, and the size of each corresponding face is 512 × 512.

After the correction is finished, point features and line features are extracted from the cube map. The specific substeps are as follows:

the cube projection map is divided into five face images with the numbers of 0, 1, 2, 3 and 4 as shown in fig. 4, and point features and line features are respectively extracted from the 5 face images. An image pyramid is established on each surface, the specific number of layers and the scaling ratio can be adjusted by combining the required operation speed and the experimental effect, the pyramid is divided into 8 layers in the embodiment, and the scaling ratio between the layers is 1.2.

For point features, each layer of the image pyramid extracts point features using an orb (organized Fast and rotaed brief) feature extraction algorithm and calculates a feature descriptor for each point feature. The ORB algorithm improves the problem of detecting the sub-directionality by FAST (features from accessed Segment test), and adopts a BRIEF (binary Robust Independent features) descriptor with extremely high speed, so that the image point feature extraction speed is greatly improved.

And S22, performing line feature extraction operation on the image to obtain line features, and calculating a feature descriptor of each line feature.

For Line features, Line features are extracted by using an LSD (Line Segment Detector) or EDL ine (EdgeDraw lines) algorithm at each layer of the image pyramid, and feature descriptors of each Line feature are calculated. Among them, the extraction speed of EDL ine is relatively faster.

And S23, performing correlation operation on the line characteristics to obtain correlation line characteristics, and setting correlation identifications for the correlation line characteristics.

In practical applications, since a solid boundary in real space, such as a line segment, may span multiple face images when projected into a cube map, it is necessary to find projections belonging to the same line segment on different faces. Therefore, it is necessary to correlate line features extracted on different plane images.

In another implementation manner of the present invention, step S23 may include:

1) and determining the line features with the distance from the position of any boundary line within a preset distance range based on the positions of the images acquired by the line features at the current moment and the positions of all boundary lines in the images acquired at the current moment, and determining the line features as target line features.

2) And determining the end point position of the target line characteristic according to a preset end point position determination rule.

Specifically, the 4 edges (i.e., the boundary lines) of each face image are respectively labeled as 0, 1, 2, and 3 as shown in fig. 5. And finding line features of line segment end points close to edges on each face image, and storing a position of the end point. For a line segment near the 0, 1 edge, the position is the x-coordinate of the endpoint, and for a line segment near the 2, 3 edge, the position is the y-coordinate of the endpoint. Wherein, close means that the distance is within a preset distance range.

3) And determining a plurality of target line features corresponding to the same entity boundary based on the end point positions of the target line features and the relative position relationship between the target line features and the cube center (figure 6) of the cube map corresponding to the image acquired at the current moment, respectively using the target line features as associated line features, and setting associated marks for the associated line features.

Specifically, the line segments close to the same edge are compared, and include edge 0 of surface 0 and edge 1 of surface 3, edge 1 of surface 0 and edge 0 of surface 4, edge 2 of surface 0 and edge 3 of surface 1, edge 3 of surface 0 and edge 2 of surface 2, edge 0 of surface 1 and edge 2 of surface 3, edge 1 of surface 1 and edge 2 of surface 4, edge 0 of surface 2 and edge 3 of surface 3, and edge 1 of surface 2 and edge 3 of surface 4.

If the line segments to be compared simultaneously satisfy:

1) the difference between the positions of the end points is less than a certain threshold; in this embodiment, the threshold is 4 pixels.

2) And the cube center of the cube projection map and the line segment for comparison respectively form two planes, and the included angle between the two planes is smaller than a certain threshold value. In this embodiment, the threshold is 3 °.

It can be considered as a projection of the same line segment in space. The projections which are considered to be the same space line segment are correlated on the data, and after the two line features are correlated, the two line features are called correlated line features, and the correlated line features are provided with correlated marks. In this embodiment, a variable Associate _ Lines is added to each line feature design in the associated line features. The variable is an array with a length of 5, and corresponds to line feature numbers on 5 surfaces respectively. If there is no relevant line feature on a certain face, it is marked as-1. For example, line feature number a on face 0 is associated with line feature number b on face 2, and the variables Association _ Lines for line feature a and line feature b are each designated as [ a, -1, b, -1, -1 ].

Taking a point on the surface 0 and a line feature on the surface 1 as an example, in the space coordinate system shown in fig. 5, it is assumed that the cube center O' has a space coordinate of (0,0,0), the side length of the cube is 2d, and the line feature l on the surface 0₀Two end points e₁、e₂Respectively is (x)₁,y₁)、(x₂,y₂) On the surface 1 features l₁Two end points e₃、e₄Respectively is (x)₃,y₃)、(x₄,y₄). Then

e₁、e₂、e₃、e₄Three-dimensional coordinates e corresponding to the space coordinate system₁′、e′₂、e₃′、e′₄Respectively as follows:

e′₁＝(x₁-d,y₁-d,d)

e′₂＝(x₂-d,y₂-d,d)

e′₃＝(-d,y₃-d,x₃-d)

e′₄＝(-d,y₄-d,x₄-d)

where d is 1/2 for the cube side length.

Then l₀The normal vector forming a plane with the optical center can be expressed as:

n₀＝(e′₁-O′)×(e′₂-O′)

then l₁The normal vector forming a plane with the optical center can be expressed as:

n₁＝(e′₃-O′)×(e′₄-O′)

the included angle of the plane is set as alpha, and calculation can be carried out through a normal vector:

|n₁||n₂|cosα＝n₁·n₂

wherein n is₀、n₁、n₂Respectively representing the normal vectors of the surface 0, the surface 1 and the surface 2, and alpha represents the included angle between the normal vectors of the surface 0 and the surface 1.

And S24, performing point feature matching operation on the image acquired at the current moment and the image acquired at the previous moment to obtain a point feature matching result, and performing line feature matching operation to obtain a line feature matching result.

Specifically, whether the processor is initialized or not is judged, if not, the processor is initialized, and if so, the image acquired at the current moment and the image acquired at the previous moment are subjected to point feature matching operation to obtain a point feature matching result, and line feature matching operation is performed to obtain a line feature matching result.

In practical application, the initialization process is as follows:

if the current frame is the first frame, the current frame is stored as the initial frame, and the next frame is waited for to enter. If the current frame is the second frame, traversing the point characteristics obtained by the previous frame of image, screening an area with proper size by taking the position on the current frame as the center, and selecting the point characteristics in the area as candidate points; and respectively calculating the descriptor distances between the candidate points and the characteristics of the previous frame points, wherein the point characteristic descriptors in the embodiment adopt binary BRIEF, so that the Hamming distances are adopted for calculation. If the distance is smaller than the required threshold, the threshold of the embodiment is set to 30, and the matching is successful. And sorting the successfully matched point features according to the descriptor distances from low to high. If the point feature logarithm successfully matched is greater than the threshold (the threshold of the matching logarithm set in this embodiment is 500), selecting feature point pairs with the required number according to the sorting. And if the logarithm of the feature points successfully matched is smaller than the threshold value, amplifying the screened area by one time, and performing matching again. If the value is still less than the threshold value, the initialization fails, and the initialization is restarted after waiting for the next frame.

And if the matched point feature pairs are not smaller than the threshold value, initializing by using the point feature pairs to obtain the poses of the first frame and the second frame, and establishing map points according to the point feature pairs.

After initialization, matching of the point feature and the line feature is performed according to the obtained point feature, line feature and associated line feature (wherein, one associated line feature is used for subsequent matching, and which line feature is specifically selected and can be selected at will). And performing key frame screening. The method comprises the following specific steps:

1) the initial pose of the current frame is set firstly, and can be assumed by uniform motion, or obtained by using other sensors such as imu (inertial measurement unit).

2) And for the obtained point features, tracking by using the descriptor distance and the point features in the previous frame to obtain matched point feature pairs.

Specifically, for a map point corresponding to a point feature in a previous frame, the coordinates of the map point are projected onto the image of the current frame according to the initial pose of the current frame. Screening out an area with a proper size on the current frame, and selecting point characteristics in the area as candidate points; and respectively calculating the descriptor distances between the candidate points and the characteristics of the previous frame point, and if the distances are smaller than a required threshold, the matching is successful. And sorting the successfully matched point features according to the descriptor distances from low to high. And if the successfully matched point feature pairs are greater than the requirements, selecting the point feature pairs with the required number according to the sorting. If the number of successfully matched point feature pairs is less than the required number, the screening area is enlarged (the specific enlargement can be one time of the enlargement), and the matching is performed again.

3) And for the obtained line features, tracking by using the descriptor distance and the line features in the previous frame to obtain matched line feature pairs.

Specifically, for map line features corresponding to the line features in the previous frame, three-dimensional coordinates of two end points of the map line features are projected onto the image of the current frame by using the camera pose.

And traversing the line features obtained from the current frame image, selecting a corresponding angle range (set by a technician according to an actual scene) on each face image, and taking the line features positioned in the angle range as candidate line features.

And calculating the sum of the distances from the two end points of each candidate line feature to the line segment projection in the candidate line features. And taking the line features of which the sum of the distances is less than the threshold value as new candidate line features. And respectively calculating the descriptor distances of the new candidate line features and the map line features, sorting the new candidate line features and the map line features from small to large according to the descriptor distances, and if the minimum descriptor distance is smaller than a threshold value, successfully matching the current frame line features corresponding to the minimum descriptor distance with the map line features.

4) And optimizing the pose of the current frame by combining the pose of the previous frame with the matched point feature pairs and line feature pairs.

And S25, determining whether the image acquired at the current moment meets a preset key frame screening rule or not based on the point feature matching result and the line feature matching result, and determining the image acquired at the current moment as a key frame under the condition that the preset key frame screening rule is met.

Specifically, whether the current frame is determined as the key frame is determined according to the point feature pair and the line feature pair. If it is determined to be a key frame, performing the subsequent step S12 for the key frame; if not, then wait for the next frame to be input.

In practical applications, step S25 may include:

1) acquiring a first number of matched point feature pairs in the point feature matching result;

2) obtaining a second number of matched line feature pairs in the line feature matching result;

3) and under the condition that any one of the first number and the second number meets a preset number rule, determining the image acquired at the current moment as a key frame.

Specifically, whether the current frame is judged as the key frame or not is judged according to the point feature pair and the line feature pair.

The judgment basis in this embodiment is:

and if the number of the point feature pairs matched is less than 80% of the number of the map point features corresponding to the last key frame, creating the key frame.

Or, the number of the line feature pairs on the match is less than 80% of the number of the map line features corresponding to the last key frame, and then the key frame is created.

If it is determined to be a key frame, performing the subsequent step S12 for the key frame; if not, then wait for the next frame to be input.

In this embodiment, a specific process for determining the key frame is given, and thus the local map can be constructed according to the determined key frame.

In the foregoing embodiment, an initial local map is generated according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, a non-linear optimization operation is performed on the initial local map to obtain a local map, and a map point and line feature result is obtained based on the local map, which is specifically referred to fig. 7.

In another embodiment of the present invention, a specific implementation process of "generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature" is provided, specifically, each line feature in the target key frame is numbered, and for each line feature in the target key frame, when performing frame-by-frame matching on the common-view frame queue, when performing corresponding matching on each line feature in a common-view frame, processing is performed according to the following several cases:

1) generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

1.1) under the condition that the content of the first map line feature is empty and the second map line feature is empty, creating a new map line feature in a target local map, and establishing the corresponding relation between the first line feature and the second line feature and the new map line feature respectively;

1.2) setting the feature identifier of the new map line feature as a preset error identifier, and calculating the feature descriptor of the new map line feature to obtain an initial local map.

Specifically, if the key frame and the corresponding match line feature in the common-view frame have no corresponding map line feature, a new map line feature (mapline) is established, the corresponding match line feature points to the new map line feature, the FLAG of the new map line feature mark is marked as FALSE (preset FALSE mark), a descriptor of the new map line feature is calculated, and the descriptor is inserted into a local map, so that an initial local map is obtained.

2) Generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

2.1) under the condition that the content of the first map line feature is the same as that of the second map line feature, if the feature identifier corresponding to the first map line feature is a preset error identifier, acquiring an observable frame corresponding to the first map line feature;

2.2) carrying out polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

2.3) carrying out triangularization processing on the first map line feature under the condition that the detection result of polar plane detection and the detection result of triangularization detection are both preset results to obtain the three-dimensional position of the first map line feature, and identifying the feature corresponding to the triangularization-processed first map line feature as a preset correct identifier;

2.4) generating an initial local map based on the three-dimensional position.

Specifically, if the corresponding match line features in the key frame and the common-view frame both correspond to the same map line feature, if the corresponding same map line feature identifier is FALSE, traversal is performed on an observable frame pointed by the map line feature, polar plane detection and triangularization detection are performed on the corresponding line features in the observable frame, the key frame and the common-view frame, if the corresponding line features pass (both a detection result of the polar plane detection and a detection result of the triangularization detection are smaller than a preset threshold), triangularization is performed on the first map line feature, the identifier FLAG thereof is set to TRUE (a preset correct identifier), and then the local map is adjusted based on the three-dimensional position to obtain an initial local map.

3) Generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

3.1) under the condition that the content of the first map linear feature is different from the content of the second map linear feature, if the feature identification corresponding to the second map linear feature is a preset wrong identification and the feature identification corresponding to the first map linear feature is a preset correct identification, projecting the second map linear feature into the target key frame to obtain a projection result;

and 3.2) comparing the projection result with the first map line feature to obtain a comparison result, and deleting the first map line feature to obtain an initial local map under the condition that the comparison result is smaller than a preset threshold value.

Specifically, if the key frame and the corresponding match line feature in the common-view frame correspond to different map line features, if the map line feature identifier corresponding to the second line feature in the common-view frame is TRUE, and the map line feature identifier corresponding to the first line feature in the key frame is FALSE, the map line feature corresponding to the second line feature in the common-view frame is projected into the target key frame and compared with the corresponding line feature in the target key frame, and if the error is smaller than a corresponding threshold value, the first map line feature observation pointed by the first line feature in the target key frame is erased, and the map line feature pointed by the corresponding line feature in the common-view frame is pointed to, so as to obtain an initial local map.

4) Generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

4.1) setting the second map line characteristic as the first map line characteristic when the content of the first map line characteristic is not empty and the content of the second map line characteristic is empty;

4.2) acquiring an observable frame corresponding to the first map line feature under the condition that the feature identifier corresponding to the first map line feature is a preset error identifier;

4.3) carrying out polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

4.4) carrying out triangularization processing on the first map line feature under the condition that the detection result of polar plane detection and the detection result of triangularization detection are both preset results to obtain the three-dimensional position of the first map line feature, and identifying the feature corresponding to the triangularization-processed first map line feature as a preset correct identifier;

4.5) generating an initial local map based on the three-dimensional position.

Specifically, if the map line feature corresponding to the line feature in the key frame exists and the map line feature in the common-view frame does not have the map line feature corresponding to the line feature in the key frame, the line feature in the common-view frame is directed to the map line feature corresponding to the line feature in the key frame, and if the map line feature is identified as FALSE, the observable frame to which the map line feature is directed is traversed, polar plane detection and triangularization detection are performed on the observable frame, the key frame, and the corresponding line feature in the common-view frame, and if the observable frame, the key frame, and the corresponding line feature in the common-view frame pass (both the detection result of the polar plane detection and the detection result of the triangularization detection are smaller than a preset threshold), the map line feature is triangulated and an identification FLAG thereof is set to TRUE, and then a local map is adjusted based on the three-dimensional position, so as to obtain an initial local map.

5) Generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

5.1) setting the first map features as the second map features under the condition that the content of the first map features is empty and the content of the second map features is not empty;

5.2) acquiring an observable frame corresponding to the second map characteristic under the condition that the characteristic identifier corresponding to the second map characteristic is a preset error identifier;

5.3) carrying out polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

5.4) carrying out triangularization processing on the second map feature under the condition that the detection result of the polar plane detection and the detection result of the triangularization detection are both preset results to obtain the three-dimensional position of the second map feature, and identifying the feature corresponding to the triangulated second map feature as a preset correct identifier;

5.5) generating an initial local map based on the three-dimensional position.

Specifically, if the line feature in the common-view frame has a corresponding map line feature and the key frame has no corresponding map line feature, the line feature in the key frame is pointed to the line feature in the common-view frame to have the corresponding map line feature, and if the map line feature is identified as FALSE, the observable frame pointed to by the map line feature is traversed, polar plane detection and triangularization detection are performed on the observable frame, the key frame and the corresponding line feature in the common-view frame, if the map line feature passes through the polar plane detection and the triangularization detection, the map line feature is triangulated, the identification FLAG thereof is set to TRUE, and then an initial local map is generated based on the three-dimensional position.

Specifically, if the map line feature of the key frame is detected to be FALSE and the number of observable frames is greater than or equal to 5, the map line feature is regarded as a poor map line feature and erased.

Wherein, the implementation manner of the polar plane detection is as follows: forming a polar plane by line features of the observable frames of the target keyframe, the common-view keyframe and the map line features and corresponding optical centers, wherein a normal vector n of the polar plane is^pFrom the optical center O to two end points l of the linear characteristic on the normalization plane_sp、l_epThe normalized vector of (a) is obtained by taking the cross product, that is:

n^p＝norm(O-l_sp)×norm(O-l_ep)

three straight lines formed by two-two intersection of the three planes are subjected to parallelism detection, and direction vectors l corresponding to the three straight linesⁿThe cross multiplication is carried out on every two polar planes, namely:

wherein, the two polar planes are intersected to form a direction vector lⁿStraight line, n₁ ^pIs the normal vector of one of the polar planes, n₂p is the normal vector of the other polar plane.

And if the cosine value of the included angle between any two straight lines exceeds a threshold value, the detection cannot be carried out through the polar plane.

In addition, the above-mentioned implementation of the triangulation detection: projecting the map line features obtained by triangulation to the line features of the target key frame, the common-view key frame and the observable frame of the map line features, wherein the overlapping degree of the line features is not less than a set threshold value, and for a key line feature equation F in each frame_kl1、F_kl2、F_kl3Wherein F is_kl1Key line feature equation for target Key frame, F_kl2For the key line feature equation in the common view key frame, F_kl3Is another common view keyframe's keyframe feature equation.

Representation l in world coordinate system for a certain endpoint of line feature in the key frame_edThe pixel coordinate representation P in the current key frame_edThe formula used for triangulation is as follows:

F_kl2·K·T_cw2·l_ed＝0

F_kl3·K·T_cw3·l_ed＝0

P_ed×(K·T_cw1)＝0

where K represents the camera projection matrix. T is_cw1Is a rotational displacement matrix, T, of the target key frame_cw2Rotational displacement matrix, T, for common view keyframes_cw3Is another co-view key frame rotational displacement matrix.

In order to improve the robustness of the triangulation mode, the ill-conditioned performance of the homogeneous equation is measured by using the condition number of the spectral norm of the homogeneous equation, when the condition number exceeds a certain threshold value, the equation is considered to be unstable, and the triangulation is abandoned, wherein for the matrix A corresponding to the equation set expressed as the above, the condition number of the spectral norm is expressed as:

cond₂(A)＝‖A‖₂‖A^-1‖₂

in this embodiment, by using the method for synchronously positioning and mapping point features and line features, it can be found that a large part of point features exist on a dynamic object in an artificial environment structure in the use process, and the point features are distributed more on a vehicle by taking an underground garage scene as an example. Whereas line features are more present in ambient static structures. Therefore, the line features can enhance the accuracy and robustness of positioning in such a scene. In addition, in the aspect of drawing construction, the line features have the significance of geometric representation, and a real scene can be better represented. The effect of the mapping according to an embodiment of the present invention is shown in fig. 7.

Optionally, on the basis of the embodiment of the map building method, another embodiment of the present invention provides a map building apparatus, and with reference to fig. 8, the map building apparatus may include:

a key frame acquisition module 11, configured to acquire a key frame set; the key frame set comprises a plurality of key frames arranged according to a time sequence; the key frame is obtained by processing the point characteristics and the line characteristics of the collected image based on a preset key frame screening rule;

a key frame determining module 12, configured to determine an earliest key frame in the key frame set at a corresponding time point as a target key frame, and determine a common view key frame corresponding to the target key frame;

a matching module 13, configured to perform line feature matching operation on the target key frame and the common-view key frame to obtain a matched line feature combination; the match line feature combination comprises a first line feature in the target key frame and a second line feature in the common view key frame;

and the map generation module 14 is configured to generate an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, and perform a non-linear optimization operation on the initial local map to obtain a local map.

Further, the map generation module includes:

the first creating sub-module is used for creating a new map line feature in the target local map and establishing the corresponding relation between the first line feature and the new map line feature and the corresponding relation between the second line feature and the new map line feature when the content of the first map line feature is empty and the content of the second map line feature is empty;

the first processing submodule is used for setting the feature identifier of the new map line feature as a preset error identifier, and calculating the feature descriptor of the new map line feature to obtain an initial local map;

a first frame obtaining sub-module, configured to, under a condition that content of the first map line feature is the same as content of the second map line feature, obtain an observable frame corresponding to the first map line feature if a feature identifier corresponding to the first map line feature is a preset error identifier;

the second processing submodule is used for carrying out polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

the third processing submodule is used for carrying out triangularization processing on the first map line feature under the condition that the detection result of the polar plane detection and the detection result of the triangularization detection are both preset results to obtain the three-dimensional position of the first map line feature, and identifying the feature corresponding to the triangularization-processed first map line feature as a preset correct identifier;

and the first map generation submodule is used for generating an initial local map based on the three-dimensional position.

Further, the map generation module includes:

the projection sub-module is used for projecting the second map line feature into the target key frame to obtain a projection result if the feature identifier corresponding to the second map line feature is a preset wrong identifier and the feature identifier corresponding to the first map line feature is a preset correct identifier under the condition that the content of the first map line feature is different from the content of the second map line feature;

the deleting submodule is used for comparing the projection result with the first map line characteristic to obtain a comparison result, and deleting the first map line characteristic to obtain an initial local map under the condition that the comparison result is smaller than a preset threshold value;

a fourth processing sub-module, configured to set the second map layout feature as the first map layout feature when the content of the first map layout feature is not empty and the content of the second map layout feature is empty;

the second frame obtaining submodule is used for obtaining an observable frame corresponding to the first map line feature under the condition that the feature identifier corresponding to the first map line feature is a preset error identifier;

the fifth processing submodule is used for carrying out polar plane detection and triangularization detection on the observable frame, the target key frame and the common-view frame;

the sixth processing submodule is used for carrying out triangularization processing on the first map line feature under the condition that the detection result of the polar plane detection and the detection result of the triangularization detection are both preset results to obtain the three-dimensional position of the first map line feature, and identifying the feature corresponding to the triangularization-processed first map line feature as a preset correct identifier;

and the second map generation submodule is used for generating an initial local map based on the three-dimensional position.

Further, the map generation module includes:

the setting sub-module is used for setting the first map line characteristic as the second map line characteristic under the condition that the content of the first map line characteristic is empty and the content of the second map line characteristic is not empty;

a third frame obtaining sub-module, configured to obtain an observable frame corresponding to the second map features when the feature identifier corresponding to the second map features is a preset error identifier;

a seventh processing submodule, configured to perform polar plane detection and triangularization detection on the observable frame, the target key frame, and the common-view frame;

the eighth processing submodule is used for carrying out triangularization processing on the second map characteristic under the condition that the detection result of the polar plane detection and the detection result of the triangularization detection are both preset results to obtain the three-dimensional position of the second map characteristic;

and the third map generation submodule is used for generating an initial local map based on the three-dimensional position.

Further, the system also comprises a key frame generating module, wherein the key frame generating module comprises:

the feature extraction submodule is used for acquiring an image acquired at the current moment, performing point feature extraction operation on the image to obtain point features, and calculating a feature descriptor of each point feature;

the characteristic processing submodule is used for carrying out line characteristic extraction operation on the image to obtain line characteristics and calculating a characteristic descriptor of each line characteristic;

the association submodule is used for performing association operation on the line characteristics to obtain association line characteristics and setting association identification for the association line characteristics;

the matching submodule is used for performing point feature matching operation on the image acquired at the current moment and the image acquired at the previous moment to obtain a point feature matching result and performing line feature matching operation to obtain a line feature matching result;

and the key frame determining submodule is used for determining whether the image acquired at the current moment meets a preset key frame screening rule or not based on the point feature matching result and the line feature matching result, and determining the image acquired at the current moment as a key frame under the condition of meeting the preset key frame screening rule.

Further, the association submodule is specifically configured to:

Further, the key frame determination sub-module is specifically configured to:

In this embodiment, the keyframes in the keyframe set are obtained by processing the point features and the line features of the acquired image according to a preset keyframe screening rule, and after the keyframe set is obtained, an initial local map is generated according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature by using a matching feature combination including a first line feature in a target keyframe and a second line feature in a common-view keyframe. The method and the device not only use the point characteristics, but also introduce the line characteristics when generating the initial local map, and the line characteristics have the characteristic of accurately expressing the environmental information, so that the method and the device can more accurately construct the local map when constructing the local map by using the line characteristics, and improve the accuracy of generating the local map.

It should be noted that, for the working processes of each module and sub-module in this embodiment, please refer to the corresponding description in the above embodiments, which is not described herein again.

Optionally, on the basis of the above embodiment of the map building method and apparatus, another embodiment of the present invention provides an electronic device, including: a memory and a processor;

wherein the memory is used for storing programs;

the processor calls a program and is used to:

Further, generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature, including:

based on the three-dimensional location, an initial local map is generated.

Further, the key frame generation process:

Further, performing association operation on the line features to obtain associated line features, and setting an association identifier for the associated line features, including:

Further, determining whether the image acquired at the current moment meets a preset key frame screening rule or not based on the point feature matching result and the line feature matching result, and determining the image acquired at the current moment as a key frame under the condition that the preset key frame screening rule is met, includes:

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A map construction method, comprising:

2. The map construction method according to claim 1, wherein generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature comprises:

based on the three-dimensional location, an initial local map is generated.

3. The map construction method according to claim 1, wherein generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature comprises:

based on the three-dimensional location, an initial local map is generated.

4. The map construction method according to claim 1, wherein generating an initial local map according to a first map line feature corresponding to the first line feature and a second map line feature corresponding to the second line feature comprises:

based on the three-dimensional location, an initial local map is generated.

5. The map construction method according to claim 1, wherein the generation process of the key frame:

6. The map construction method according to claim 5, wherein the associating operation is performed on the line feature to obtain an associated line feature, and an association identifier is set for the associated line feature, and the method comprises:

7. The map construction method according to claim 5, wherein determining whether the image acquired at the current time satisfies a preset key frame screening rule based on the point feature matching result and the line feature matching result, and determining the image acquired at the current time as a key frame if the preset key frame screening rule is satisfied comprises:

8. A map building apparatus, comprising:

9. A storage medium characterized by comprising a stored program, wherein a device on which the storage medium is located is controlled to execute the mapping method according to any one of claims 1 to 7 when the program is executed.

10. An electronic device, comprising: a memory and a processor;

wherein the memory is used for storing programs;

the processor calls a program and is used to: