CN114088083A

CN114088083A - Mapping method based on top view semantic object

Info

Publication number: CN114088083A
Application number: CN202111319475.3A
Authority: CN
Inventors: 周宏涛; 王璀; 范圣印; 李雪
Original assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Current assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Priority date: 2021-11-09
Filing date: 2021-11-09
Publication date: 2022-02-25
Anticipated expiration: 2041-11-09
Also published as: CN114088083B

Abstract

The invention discloses a mapping method based on top view semantic objects, which comprises the steps of carrying out initial mapping based on detected enhanced characteristic parking space information; based on a KM algorithm matching result, performing incremental map building; optimizing a map based on the observation information of the key frames; according to the method, the KM algorithm is used in the drawing construction process to complete matching of the characteristics of the current frame and the examples of the map, a key frame confirmation and elimination strategy is provided, position constraint between example parking spaces is used, the relative relation between the example parking spaces is adjusted according to multiple observation conditions, and a scale factor is added into an optimization function, so that the drawing construction precision is improved, and the calculation speed is accelerated. The invention solves the problems of difficult image correction, difficult matching algorithm, difficult key frame processing and difficult map optimization which are difficult to solve in the map building process based on the panoramic vision in the prior art.

Description

Mapping method based on top view semantic object

Technical Field

The invention relates to the technical field of autonomous parking and computer vision in the field of automatic driving, in particular to a mapping method based on a top view semantic object.

Background

In the application field of autonomous parking technology of automatic driving, various sensors are required to complete the functions of mapping and positioning the surrounding environment. At present, map construction and self-positioning technologies based on slam algorithm can be divided into laser slam, binocular slam and monocular slam according to different sensors, and the three map construction and self-positioning technologies are respectively insufficient: the laser slam is limited by equipment cost and is difficult to deploy into mass-production vehicle models; the binocular slam has high calculation complexity and is difficult to process in real time; monocular slam is affected by the problem of scale drift and is difficult to obtain absolute scale.

In order to overcome the defects and shortcomings, a scheme of a look-around vision fusion IMU (inertial measurement unit) and a wheel speed meter based on a slam algorithm is proposed at present, and the scheme influences the precision in the process of drawing construction and positioning due to the adoption of different processing methods of distorted pictures, different matching algorithms, different key frame processing strategies, the addition of different optimization factors and different semantic object constraints. How to select and design a more reasonable image correction method, a matching algorithm, a key frame processing strategy, semantic object constraint and map optimization becomes a crucial problem.

The scheme of the panoramic vision fusion IMU (inertial measurement unit) and the wheel speed meter based on the slam algorithm has the following implementation difficulties: compared with the laser mapping, the mapping based on the around vision method is greatly influenced by the environment, and the effect of laser mapping is difficult to achieve even if a plurality of remedial measures are taken. The laser mapping data processing amount is large, the point cloud matching needs a computing platform to have high computing power, but the mapping precision is high. Because image distortion, when the image is fuzzy, the matching difficulty is great in the look-around vision scheme, and the picture-building precision is difficult to improve under the influence of image quality. Therefore, mapping based on the around vision is much more difficult and complex than laser mapping. The method comprises the following specific steps:

the first difficulty is: difficulty of image rectification: in the process of drawing a drawing vehicle entering an underground parking lot, the position of a fisheye camera can move up and down along with the jolt of the vehicle at the joint of an up slope, a down slope and a parking space due to the fact that the drawing vehicle must pass through an up slope and a down slope jolt zone of the parking lot, so that the top view shot by the fisheye camera can be spliced wrongly, and further the drawing of the parking space near the up slope and the down slope of the parking lot based on the top view can be wrongly built. In the prior art, patent No. CN111862673A, "parking lot vehicle self-positioning and map construction method based on top view" adopts a real-time correction method in the map construction process: the method obtains the relative pose change of the current frame and the previous frame by calculating the homography matrix, and takes real-time correction by fully considering the condition that the vehicle bumps or goes up and down. However, the image correction process is complex, and a homography matrix and a splicing matrix of a fisheye camera need to be calculated in real time, so that the calculation amount is far higher than that of a flat road surface when a vehicle bumps;

the second difficulty is: difficulty of matching algorithm: any single matching algorithm is difficult to be matched without errors. When a new map parking space is created, the parking space seen in the current top view is matched with the parking space already established in the map, and if the matching is not achieved, the matching is established in the map as a new parking space, and the matching with the single condition is difficult to achieve accurate matching. If the identification condition is increased, 2 angular points of the current parking space plus the parking space number are matched, the risk of identification errors still exists, for example, the parking space numbers '1' and '7' are easily identified as the same type during identification, and when a '7' map parking space is established, if the position identification of the 2 angular points of the parking space is correct, but the parking space number is identified as '1', the current parking space still cannot be judged to be the '7' parking space.

The third difficulty: key frame processing difficulties. The important "frame" is generally called a key frame, and the "frame data" refers to two basic information, namely "parking space information" observed by the frame and "vehicle pose" holding the frame. The difficulty of processing the key frame is that the two basic information of the parking space information and the vehicle pose information holding the frame are dynamically changed, the parking space position comprises 2 angular points of the parking space and the middle point of a parking space number detection frame, the parking space information is dynamically changed, for example, the position of the same parking space on a map, because the vehicle is driven and the picture is taken at the same time, the positions of 3 points of the same parking space which are seen by the top view key frames at different positions are different, and a value which is most close to the actual parking space is finally obtained according to the different positions of the same parking space which are recorded by a plurality of key frames, it is difficult in fact because not only influenced by the distance between the current vehicle and the map parking space, but also influenced by the driving condition (vehicle bump condition) and illumination of the vehicle, and simultaneously influenced by the scale factor of the observation top view.

The fourth difficulty: difficulty in map optimization. The map optimization is to optimize the position of each parking space and the key frame vehicle attitude in the map again on the basis of the established map parking spaces. Even if an optimized matching method is found during incremental mapping and accurate mapping is realized, the accurate mapping cannot be realized only through one-time optimization, and a method of multiple optimization is required. For example, in the built map, the actual parking spaces are adjacent parking spaces, but two adjacent parking spaces on the map do not have a common angular point; three points of two adjacent parking spaces of the actual parking space are collinear, but three vertexes of the two adjacent parking spaces on the map form included angles, so that the error causes are related to image correction of the fisheye camera, matching algorithm and key frame data processing, and are also related to the pose of the key frame. The pose of a key frame corresponds to a group of observation values, the detection of the sufficient inflation and insufficient inflation of the tires of the vehicle has influence on the height position of a camera and the observation values of the key frame, and therefore the difficulty of map building based on vision is to realize deep optimization and multi-dimensional optimization.

Disclosure of Invention

The invention provides a mapping method based on a top view semantic object aiming at the defects of the prior art and aims to solve the problems of difficult image correction, difficult matching algorithm, difficult key frame processing and difficult map optimization which are difficult to solve in the mapping process based on the around vision in the prior art.

In order to solve the technical problems, the invention provides the following technical scheme

A mapping method based on top view semantic objects is characterized by comprising the following steps:

firstly, initializing and establishing a map based on the detected enhanced characteristic parking space information;

step two, based on a KM algorithm matching result, incremental map building is carried out;

thirdly, map optimization is carried out based on the observation information of the key frames;

the first step of establishing a map based on the detected enhanced characteristic parking space information is initialized, and the specific process is as follows:

1) completing the initialization step before the map building;

the initialization before the mapping is that the initial position and the preset track of the current vehicle are determined;

2) generating a top view based on image data shot by a plurality of fisheye cameras collected by a vehicle end, and finishing real-time correction of the top view;

3) obtaining enhanced characteristic parking space information based on the top view after real-time correction;

4) according to the enhanced characteristic parking space information, and combining the information of an inertia measurement unit and a wheel speed meter, finishing initialization map building; the initialized mapping is called an example map, and the example map is used as a basis for subsequent incremental mapping;

and step two, incremental map building is carried out based on the KM algorithm matching result, and the specific process is as follows:

1) repeating the process 2) and the process 3) of the step one to obtain a new top view after real-time correction and enhanced characteristic parking space information in the top view;

2) matching the enhanced characteristic parking spaces of the example map with the enhanced characteristic parking spaces of the new top view corrected in real time by using a KM algorithm;

3) the parking spaces of the new embodiment are as follows: converting the enhanced characteristic parking space information of the current top view from a top view coordinate system to a world coordinate system; if the new parking space image characteristics which are not matched appear in the matching process of the parking space image characteristics of the new top view and the example parking space of the initialized example map, establishing an example parking space in the example map, and converting the enhanced characteristic parking space information of the current top view into a world coordinate system from a top view coordinate system;

and step three, map optimization is carried out based on the key frame observation information, and the specific process is as follows:

1) finishing the angular point fusion between the newly added parking space and the adjacent parking space in the example map;

2) marking the collinear relative relation of the angular points of the adjacent parking spaces;

3) confirming key frames with parking spaces and key frames without parking spaces, and removing the key frames for later-stage map optimization;

4) local map optimization is carried out by utilizing the key frame with the parking places, and the position of each parking place of the adjacent parking places in the map, three points of the adjacent parking places are collinear, and image scale factors are optimized; each parking space position comprises two parking space angular points of each parking space and a midpoint of a parking space number detection frame;

in the first step, the top view is generated based on the image data shot by the plurality of fisheye cameras collected by the vehicle end, and the real-time correction of the top view is completed, and the specific process is as follows:

1) converting the fisheye camera coordinates to top view coordinates;

wherein the rightmost u and v represent the coordinates on the fisheye camera, passing through π^-1The bracket transformation converts the fisheye camera coordinates into corrected image coordinates, and then passes through [ R ]_p t_p]The inverse transformation of (a) transforms the corrected image coordinates to top view coordinates, i.e. leaves regions within the calibrated range in the corrected image, [ x ]_p y_p]Representing coordinates in a top view;

2) in accordance with formula (2) in comparison with formula (1)R in the same case_P、t_PSolving to obtain a corresponding external parameter matrix;

p＝HP....(2)

the correlation between the formula (1) and the formula (2) is: obtaining R of formula (1) by decomposing H in the matrix of formula (2)_P、t_P(ii) a Thereby obtaining [ R ] under different conditions_p t_p]In this different case [ R ]_p t_p]The method comprises the following steps:

A. a fisheye camera external parameter matrix under a flat road surface;

B. the fisheye camera external parameter matrix is used for vehicle under working conditions of different pitch angles and different roll angles;

3) generating a corresponding geometric lookup table based on the calculation result;

4) correcting the top view image information in bump in real time;

5) and obtaining a top view of the spliced plurality of fisheye camera pictures corrected in real time at each moment.

The top view based on real-time correction in the first step process 3) obtains enhanced characteristic parking space information, and the specific process is as follows:

1) obtaining a top view corrected in real time at each moment;

2) based on the deep neural network model, bit feature detection is performed on the top view: the method comprises the steps of detecting parking space position information and classifying parking space types; the parking space position information is two angular points of a parking space entrance line, and the parking space type is as follows: dividing the parking spaces into horizontal parking spaces, vertical parking spaces or inclined parking spaces according to the relative position relationship between the parking spaces and the road, wherein the parking spaces are specifically expressed as slots (x1, y1; x2, y 2; type); wherein, (x1, y1) and (x2, y2) are position information coordinates under a top view coordinate system of two angular points clockwise, and type is a parking space type;

3) and (3) performing bit number feature detection on the top view based on the deep neural network model: the method comprises the steps of detecting the parking space number and identifying the parking space number so as to obtain the position information of a detection frame of the parking space number characteristic and the identification result of the parking space number; the position information of the detection frame comprises the midpoint, the length and the width of the detection frame, and is specifically represented as number (x, y, w, h, alpha; NUM), wherein (x, y) is the midpoint of the parking space number detection frame, (w, h) is the length and the width in the parking space number detection, alpha is the clockwise rotation angle value of the detection frame relative to the vertical direction, and NUM represents the identification result of the parking space number;

4) integrate parking stall characteristic and parking stall number characteristic to reinforcing characteristic parking stall information on obtaining this top view: according to the position information of the two angular point coordinates of the parking space and the parking space number detection frame in the top view coordinate system, the parking space and the parking space number information are correlated, and accordingly the enhanced characteristic parking space with the parking space number information is obtained, and the enhanced characteristic parking space with the parking space number information is specifically represented as (x1, y1; x2, y 2; type; x, y, w, h, alpha; NUM).

In the first step, the initial map building is completed according to the enhanced characteristic parking space information and by combining the information of the inertia measurement unit and the wheel speed meter, and the method specifically comprises the following steps:

1) initializing and establishing a graph: based on the enhanced characteristic parking space information, combining the information of an inertia measurement unit and wheel speed meter to complete initialization map building, wherein the initialization map building is to obtain a key frame for observing the parking space for the first time and build an example map by using the key frame for observing the parking space for the first time; the key frame is a frame for observing a parking space for the first time, and the key frame comprises observed parking space information and vehicle pose information holding the frame.

2) The example map specifically projects the enhanced feature parking space under the top view coordinate system in the key frame where the parking space is observed for the first time to the vehicle coordinate system, and then converts the enhanced feature parking space under the vehicle coordinate system to the world coordinate system.

In the second step, the KM algorithm is used for matching the example parking space of the example map with the enhanced characteristic parking space of the new real-time corrected top view in the second step, and the specific steps are as follows:

1) carrying out multi-dimensional information matching on the current new top view enhanced characteristic parking space information corrected in real time and the example parking space of the example map, specifically comprising the following steps: matching was performed using the following 5-point information: the new top view after real-time correction is simply called a new top view, and the new top view contains enhanced characteristic parking space information;

a. matching by using the parking position information to obtain f_d,f_dRepresenting the coincidence degree of the current detected parking space position information based on the new top view and the parking space position information of the example map; f. of_dAlso known as the position error cost;

b. matching by using parking space category information to obtain f_t,f_tRepresenting whether the current detected parking space category based on the new top view is the same as the parking space category of the example map, f_tAlso known as parking space category cost;

c. matching by using the similarity of the parking spaces to obtain f_b,f_bRepresenting whether the current detected parking space number based on the new top view is similar to the parking space number of the example map; f. of_bAlso known as the parking space number similar cost;

d. matching by using the overlapping degree of the parking space number detection frames to obtain f_n，f_nRepresenting the overlapping degree of the currently detected parking space number detection frame based on the new top view and the parking space number detection frame of the example map; f. of_nAlso known as check box overlap cost;

e. matching by using the relative position information of the parking spaces to obtain f_r，f_rRepresenting the similarity degree of the current detected parking space relative position information based on the top view and the parking space relative position information of the example map; f. of_rAlso known as relative position cost;

the adjacent parking spaces refer to the situation that under the real world space, a shared angular point exists between two parking spaces, according to the clockwise direction, each parking space can possibly exist in an upper adjacent parking space and a lower adjacent parking space, and the specific judgment formula of the adjacent parking spaces is as follows:

||P_A-P_B||＜ΔS....(4)

wherein, P_AA certain corner position representing A parking space, i.e. PA_＝[x_A，y_A]，P_BA certain corner position representing B parking space, i.e. P_B＝[x_B，y_B]And Δ S represents the adjacent angle of two carsA point distance threshold;

2) performing optimal matching calculation through a KM algorithm: and (3) synthesizing the 5 kinds of information to obtain a total associated cost function for matching between the enhanced feature parking space of the example map and the enhanced feature parking space of the new top view, wherein the specific formula is as follows: (KM (Kuhn and Munkres) algorithm for optimal matching of bipartite graphs)

f＝ω_df_d+ω_tf_t+ω_bf_b+ω_nf_n+ω_rf_r....(5)

Where f is the total cost function, ω_d、ω_t、ω_b、ω_n、ω_rThe weight coefficients of the above five factors are respectively; according to the formula of the total cost function f, the cost between all the enhanced feature parking spaces with potential matching correlation in the current new top view and the enhanced feature parking spaces of the map example can be solved, so that a corresponding correlation matrix is constructed, and finally, a KM algorithm is substituted for calculation of optimal matching; the potential matching association is both: if the space distances are close or the parking space numbers are similar, the potential matching correlation is considered;

3) and projecting the enhanced characteristic parking space of the current new top view into a world coordinate system.

F is_d、f_t、f_b、f_n、f_rThe calculation formula of (a) is as follows:

wherein x is_a、y_aIs the midpoint position information, x, of two angular points of a parking space of an example map_b、y_bThe position information of the middle point of two angular points of the parking space in the new top view is projected to a world coordinate system;

wherein, type_aParking space being example map parking spaceType, type_bIs the type of stall in the new top view;

wherein a is the parking space number character string of the parking space of the example map, b is the parking space number character string of the parking space in the new top view,

indicating exclusive or, i is an index of the parking space number character string;

wherein A is the area of the position number detection frame in the example map, and B is the area of the position number detection frame projected to the world coordinate system in the new top view;

f_r＝ω_nlf_nl+ω_nnf_nn+ω_atf_at....(10)

wherein, ω is_nl、ω_nn、ω_atAre respectively corresponding weight coefficients, f_nlIs the similarity degree of the example map parking space of the adjacent parking spaces on the parking space and the parking space number of the observation characteristic parking space, f_nnThe similarity degree of the example map parking space of the adjacent parking space under the parking space and the parking space number of the observation characteristic parking space; f. of_atIndicating that the parking space is in the sliding windowWhether the parking space distribution type is the same as the example parking space distribution type in the map or not, and the type_arIs the type of the space distribution of the parking spaces of the example map_brThe parking space distribution type in the sliding window is adopted.

The third step 1) is to complete the corner point fusion between the newly added parking space and the adjacent parking space in the example map, and the specific process is as follows:

the common angular points of the adjacent parking spaces are fused: and adjusting the shared angular point between the parking spaces of the example map by using the observation result of the relative relation between the adjacent parking spaces in the new top view: if the situation that two adjacent parking spaces share one angular point exists in multiple observations in a sliding window in a new top view, two angular points with errors of the two corresponding adjacent parking spaces are fused in an example map, so that the two parking spaces both have the angular point, and only one angular point position information is optimized in the later optimization process;

marking the relative relation of the collinear angle points between the adjacent parking spaces in the process 2) in the third step, marking the relative relation of the collinear angle points between the parking spaces in the example map according to the relative relation between the parking spaces in the new top view, and specifically comprising the following steps of:

the parking spaces in the adjacent map in the new top view comprise the parking spaces which are matched and not matched in the example map, and the parking spaces which are not matched are the new parking spaces to be established in the map; the relative relation refers to whether angular points between adjacent parking spaces are collinear or not; the adjacent parking spaces refer to the fact that two parking spaces in a top view have a common angular point, and each parking space may have an upper adjacent parking space and a lower adjacent parking space in a clockwise direction, and the step e) in claim 5 is specifically defined;

the specific determination formula of the co-linear adjacent parking space angular points is as follows:

wherein x is_A1、y_A1And x_A2、y_A2Is the position information of two angular points of parking space A, x_B1、y_B1And x_B2、y_B2The angle points of the two parking spaces are collinear, and the angle points of the two parking spaces are marked as the collinear, if the absolute value of the included angle between the two angle point connecting lines of the two adjacent parking spaces is smaller than the threshold value of the included angle, the angle points of the two parking spaces are considered to be the collinear.

The key frames with parking spaces and the key frames without parking spaces are confirmed and the key frames are removed in the third step 3) for later-stage map optimization, and the specific process is as follows:

1) acknowledging key frames

a. Determining a key frame according to whether a new parking space exists or not

Judging whether a frame corresponding to the current new top view is a new parking space in the example map or not, and if so, confirming that the frame corresponding to the current new top view is a key frame and storing the key frame;

b. determining keyframes from distances

If the frame corresponding to the current new top view does not observe the parking space or the observed parking space is not a newly-built parking space in the example map but a historical parking space, when the distance or the course angle difference value between the current frame and the previous key frame is larger than a certain threshold value, the key frame is confirmed, if the key frame does not observe the parking space information, the key frame information does not include image data and only includes the position and attitude information of the current vehicle, and the position and attitude of the current vehicle is the position and attitude of the current vehicle when the current vehicle shoots the image;

the formula for inserting key frames according to distance is as follows:

‖P_k+1-P_k‖＞ΔP....(15)

wherein, P_kRepresenting the vehicle centre position at time k, i.e. P_k＝[x_k，y_k]，P_k+1Representing the vehicle centre position at time k +1, i.e. P_k+1＝[x_k+1，y_k+1]If the distance between the vehicle center at the moment K and the vehicle center at the moment K +1 is greater than the distance threshold, confirming a new key frame;

the formula for inserting key frames according to the heading angle difference value is as follows:

‖θ_k+1-θ_k‖＞Δθ....(16)

wherein, theta_kRepresenting the vehicle heading angle, theta, at time k_k+1Representing the vehicle course angle at the moment of K +1, wherein delta theta represents a set course angle threshold, and if the absolute value of the difference between the course angles at the moment of K and the moment of K +1 is greater than the course angle threshold, determining a new key frame;

2) removing key frames:

a. and calculating the observation score of the frame corresponding to the current new top view to the same example parking space: each parking space of the example map records the observation results of a plurality of key frames to the same parking space: the farthest observation and the closest observation were recorded: and recording observation data of the latest observation and the farthest observation according to the following formula, and calculating the observation score of each key frame to the parking space of the example:

L_min＝min(L_min,L)....(17)

L_max＝max(L_max,L)....(18)

wherein L is_minThe minimum distance from the center of the top view to the center of the top view is the nearest observation, namely the parking space angular point in the top view or the midpoint of the parking space number detection frame; l is_maxFor the farthest observation, namely the maximum distance between a parking space angular point or the midpoint of a parking space number detection frame in the top view and the center of the top view, L is the distance between the parking space angular point or the midpoint of the parking space number detection frame in the top view of the current key frame with the calculated score and the center of the top view, and g is the observation score in the observation weight of the current key frame;

b. according to the calculation result of the observation score, different observation weights are given;

c. if the observation weight of a certain key frame is lower than a set threshold value, the key frame is removed; the observation weight is calculated based on the distance between the angular point of the characteristic parking space and the center of the image and the distance between the midpoint of the parking space number detection frame and the center of the image, and is also influenced by the driving condition of the vehicle during observation and the illumination condition during observation;

the observation weight is calculated mainly in two parts: observing the angular point of the top view parking space, and observing the middle point of the top view parking space number detection frame; the shortest distance from the top view center to the image characteristic parking space angular point is taken as the angular point nearest observation, the shortest distance from the top view center to the midpoint of the image characteristic parking space number detection frame is taken as the parking space number nearest observation, and the specific calculation formula of the observation weight of the key frame is as follows:

f＝w_bw_l(∑(w_cg_c)+∑(w_ng_n))....(20)

wherein f represents the observation weight of the key frame, w_b、w_lWeight coefficient, g, representing vehicle bump, illumination effect_c、g_nRepresenting the angular point, the number of the parking space observation score, w_c、w_nWhether the angular point and the parking space number are the latest observation or not is represented, the latest observation is set to be 10, and the latest observation is not set to be 1;

in the third step, in the process 4), local map optimization is performed by using the key frame with the parking places, and the position of each parking place of the adjacent parking places in the map, three points of the adjacent parking places are collinear, and the image scale factor is optimized; each parking space position of the adjacent parking spaces comprises a parking space angular point of each parking space and a middle point of a parking space number detection frame;

the specific expression is as follows:

the above formula (21) is to find a difference value between the ith parking space coordinate information of the example map and the jth parking space observation information converted into the world coordinate system parking space coordinate information in the jth key frame, where the difference value is a difference value in the world coordinate system, and specifically is: the item 1 on the right of the equal sign of the formula (21) is the world coordinate information of the ith parking space in the example map, the initial value of the value is the coordinate information of the third step, namely the observation information with the highest observation score in the observation results of the same parking space by the plurality of reserved key frames is converted into the world coordinate system; the data in the parenthesis of the item 2 on the right of the equal sign of the formula (21) is the coordinate information of the ith parking space observed in the vehicle coordinate system by converting the ith parking space into the world coordinate system through the jth key frame, and the difference value of the coordinate information of the parking space of the example map and the coordinate information of the parking space of the ith parking space converted into the world coordinate system through the observation information of the jth key frame is obtained by subtracting the item 2 from the item 1 on the right of the equal sign of the formula (21).

The method comprises the following specific steps:

a. item 1 to the right of the equal sign

Representing the coordinates of two angular points in the parking space of the ith example map or the midpoint of the parking space number detection frame in the world coordinate system, wherein the value range of l is { a, b and c }, the value ranges respectively represent the two angular points of the parking space and the midpoint of the parking space number detection frame, and w represents the world coordinate system;

b. s in the parentheses of item 2 on the right of the equal sign_fRepresenting the image scale factor, and solving the formula (25) to obtain s_fA value;

c. the matrixes of

items

1 and 3 in the 2 nd brace on the right side of the equal sign represent the changes of rotation and translation of the vehicle, wherein the item 1 is the change of rotation, the item 3 is the change of translation, and the matrix [ x ]_j y_j θ_j]The pose of the j-th key frame vehicle in the sliding window in the world coordinate system, wherein the sliding window is a combination formed by continuous frames of key frame observation information and key frame pose information of the vehicle in a moving state;

d. the 2 nd matrix in the right brace with equal sign

Representing the coordinates of a vehicle coordinate system of a jth key frame in a sliding window of two angular points or the midpoint of a parking space number detection frame in the parking space of the ith example map, wherein the value range of l is { a, b and c }, and respectively representing the two angular points of the parking space and the midpoint of the parking space number detection frame;

representing that the vehicle observes coordinate information of different parking spaces in different pose states;

the above formula (23) and the formula (24) respectively represent vectors formed by 2 respective angle points of adjacent parking spaces, and the formula (22) represents cross multiplication between the two vectors of the formula (23) and the formula (24); if the vector cross multiplication result approaches zero, the three points tend to be collinear; wherein i and i-1 represent the ith parking space and the ith-1 parking space;

two angular point coordinates respectively representing the ith example parking space, and for the (i-1) th parking space, two angular point coordinates respectively representing the (i-1) th example parking space;

the above equation (25) is the overall optimization function, and the first part on the right of the equal sign is to sum the squares of the position errors observed for the current parking space for the different keyframes of equation (21), and then multiply by Λ_ijl，Λ_ijlFor assigning weights, Λ_ijlIs a diagonal matrix, elements of which are observation weights corresponding to the observation information used in formula (21) exist only at the diagonal lines; the second part on the right of the equal sign is that squares of three-point collinear errors observed on the current parking space by different key frames of the formula (22) are summed and then multiplied by lambada_k，Λ_kOnly diagonal elements exist, and the diagonal elements are all 1, and weight distribution is carried out on the square of the collinear error of each three point.

The third step further comprises a process 5): the method comprises the following steps of performing loop detection by using a key frame with parking space information, optimizing the pose of the key frame in a map by using the key frame with parking space information and a key frame without parking spaces, and optimizing parking space angular points and the middle points of a parking space number detection frame in the map again: the loop back detection comprises: the method comprises the following steps of searching parking spaces in a sliding window, wherein the parking spaces are similar to the parking spaces in a map or have close distances, matching and associating the parking spaces in the sliding window with example parking spaces in the map by using a KM algorithm, and if a certain number of matching exists, considering that loop occurs, wherein the specific process is as follows:

1) optimizing the pose of the key frame, wherein a specific formula for optimizing the pose of the key frame is as follows:

wherein, T_ijRepresenting relative motion between key frames of the ith and jth frames, T_iAnd T_jRespectively representing the poses of the ith and jth frame key frames;

is T_iThe multiplication is equal to the identity matrix;

wherein e is_ijRepresenting the error formed by pose transformation between the ith frame and the jth frame key frame;

is T_ijThe inverse matrix of (d); ln represents the logarithm operation; the V-shaped symbol represents the plum group form and is transformed into a plum algebraic form;

f is a total cost function and represents the square sum of errors formed by pose transformation; e.g. of the type_ijRepresenting the error formed by the pose transformation between the ith frame and the jth frame key frame,

is e_ijA diagonal transformation matrix;

in the above formula, i and j represent key frame number, T_iAnd T_jRespectively representing the poses of the ith and jth frame key frames; [ x ] of_j y_j θ_j]The pose of the j-th key frame vehicle in the sliding window in the world coordinate system is constructed into T through form transformation_jIn the form of (a);

2) and returning to the process 4) in the third step, and optimizing the parking space position information of the example map again, wherein the scale factor is not changed.

Advantageous effects of the invention

1. The method is based on the fact that the top view after splicing is corrected by using the pre-calculated fisheye camera external reference matrix under the condition that the top view of the vehicle has large errors when bumping. On the premise of ensuring the precision, the calculation speed is accelerated.

2. The invention uses KM (Kuhn-Munkras) algorithm to complete the matching of the characteristics of the current frame and the map example in the process of map building. Compared to ICP (iterative nearest neighbor) algorithms, KM algorithms can be based on more dimensional information, such as: the parking place type, the overlapping degree of the parking place number detection frames, the similarity degree of the parking place numbers, the relative position information of the parking places and the like. In the case of a poor initial value, a more robust result can still be obtained.

3. The invention provides a key frame confirming and removing strategy. When a frame of spliced top view is input, judging whether a new map example parking space is created or not and whether the observation is updated or not for the observed characteristic parking space, and calculating the weight of the new map example parking space; if new construction or updating is needed, the weight of the current frame is increased, and the weight of the key frame which is newly constructed or observed in the characteristic parking space is reduced. And if the weight of the key frame is lower than the threshold value, removing the key frame. And under the condition that the new map example parking space occurs in the local map example parking space and is not matched with the current frame characteristic parking space, updating and observing the Euclidean distance between the current frame characteristic parking space and the matched map example parking space, wherein the Euclidean distance is smaller than the previous observation distance. The accuracy is improved, and meanwhile, the scale of the calculated amount is reasonably controlled.

4. The invention uses the position constraint among the example parking spaces, adjusts the relative relation among the example parking spaces according to multiple observation conditions, and improves the accuracy of drawing construction.

5. When the mapping optimization function is constructed, the problem of scale change caused by the height change of a camera of a vehicle, the change of tire pressure and the error of a wheel speed meter is taken into consideration, so that the scale factor is added into the optimization function to complete mapping with higher precision.

Drawings

FIG. 1 is a schematic block diagram of the method of the present invention;

FIG. 1-1 shows a detailed process of step one of FIG. 1 according to the present invention;

FIGS. 1-2 illustrate a detailed process of step two of FIG. 1 according to the present invention;

FIGS. 1-3 illustrate a detailed process of step three of FIG. 1 according to the present invention;

FIG. 2-1 is a first schematic diagram illustrating an acknowledgement key frame according to a newly added parking space;

FIG. 2-2 is a schematic diagram of a new parking space confirmation key frame according to the present invention;

FIGS. 2-3 are schematic diagrams of keyframes being identified according to distance in accordance with the present invention;

FIGS. 2-4 are schematic diagrams of keyframe identification according to heading angle in accordance with the present invention;

FIGS. 2-5 are schematic diagrams of a parking space corresponding to a plurality of key frames according to the present invention;

FIG. 3-1 is a schematic diagram illustrating the overlapping degree matching of the parking space number detection frames according to the present invention;

FIG. 3-2 is a schematic diagram illustrating the matching of relative position information between parking spaces according to the present invention;

FIG. 4 is a schematic diagram illustrating pose optimization of a keyframe according to the present invention;

FIG. 5-1 is a top view of the neural network of the present invention prior to learning without including parking space position information;

fig. 5-2 is a top view of the neural network of the present invention after learning including the parking space position information.

Detailed Description

The design principle of the invention is as follows:

the invention is further explained below with reference to the drawings:

1. dynamic key frame design principle

First, overview: the key frame is the most important frame. The purpose of dynamically designing the key frames is to find the most elegant key frame in a plurality of key frames corresponding to the same parking space, and the purpose of finding the most elegant key frame is to optimize the skew and twisted parking spaces which do not accord with the actual parking space pose in the map by taking the parking space pose seen by the key frame as the reference, so that the optimal key frame reaches the degree which is the closest to the actual parking space pose.

And secondly, the principle that one parking space corresponds to a plurality of key frames. As shown in fig. 2-5, detecting the line of sight of the vehicle from finding a space to leaving the space is a process from far to near and then from near to far. This process will have multiple keyframes "seeing" the same slot, thus, one slot in the map retains multiple keyframes.

Third, design difficulty: the difficulty is that the score of the same key frame is changed along with the position of the vehicle when the vehicle is at different positions, and if the score is calculated for each key frame only 1 time, the key frame with the highest score can never be obtained. The key frame score is determined according to the distance between the key frame score and the target point, the maximum value is the value of the distance between the current all key frames and the target point, and the minimum value is the value of the distance between the current all key frames and the target point, taking the key frame number 2 of fig. 2-5 as an example: when the detection vehicle is at the position of the key frame No. 2, the "current all key frames" only include the key frames No. 1 and No. 2, the maximum value is the distance between the detection vehicle and the target point, and the minimum value is the distance between the key frame No. 1 and the target point, therefore, the score of the key frame No. 2 relative to the key frame No. 1 is high (assumed to be 0.8), but when the vehicle reaches the position of the key frame No. 3, the score of the key frame No. 3 should be higher than the score of the key frame No. 2 because the distance between the detection vehicle and the target point is closer, but the score of the key frame No. 2 may be the same as the score of the key frame No. 3, so that the conclusion that the score of the key frame No. 3 is higher than that No. 2 cannot be obtained. Therefore, the score for key frame number 2 cannot be calculated only once. Since the score of the key frame 2 is calculated relative to the key frame 1 when the key frame 2 is calculated for the first time, but the minimum value changes when the vehicle reaches the position of the key frame 3, the score of the key frame 2 is calculated again, and the minimum value is the distance from the key frame 3 to the target point, so that the score decreases when the score of the key frame 2 is calculated again. Similarly, when the vehicle reaches the position of key frame No. 7, the score of key frame No. 2 is recalculated, and the score is lower.

Fourth, the solution of the invention: as shown in fig. 2-5, in addition to calculating the score for the current key frame, the key frames for which scores have been calculated are recalculated. And setting a loop calculation interval for recalculating the key frame, wherein the loop calculation interval can be 10 frame data interval or 5 frame data interval. For example, the loop calculation interval of the key frame No. 2 in fig. 2-5 is 5 frames of data, which means that when the scores of the key frames No. 3, 4, 5, 6, 7 are calculated, the scores of the key frame No. 2 are calculated again. When the vehicle reaches the position of the No. 7 key frame from the position of the No. 2 key frame, the score of the No. 2 key frame is lower and lower along with the calculation of each time, and similarly, the same loop calculation method is also adopted for the No. 3, 4, 5 and 6 key frames, and the conclusion that the score of the No. 7 key frame is the highest can be obtained at the position of the No. 7 key frame.

And fifthly, removing key frames. The purpose of finding the best keyframe is to remove the lowest scoring keyframe, but the so-called best keyframe cannot be used for localization, since even very good data cannot be located by only one frame of data, and therefore, a better and best plurality of keyframes of data are retained after removing the worst keyframe.

2. Map optimization design principle

1) Necessity of map optimization. A plurality of key frames which are used for 'seeing' the parking space are reserved in each parking space of the map, and the key frames are good-quality key frames which are not removed. Although a plurality of parking spaces are already established in the map through incremental mapping, the positions of the parking spaces are not ideal, and may be skew, twisted and greatly different from the actual parking spaces, for example, the actual adjacent parking spaces have a common angular point and three points are collinear, but the adjacent parking spaces in the map are separated and do not have the common angular point, and an angle exists between two lines of the three points. At this time, the physical pose of each parking space needs to be optimized through a plurality of good key frames reserved for each parking space in the map.

2) A method for map optimization. Map optimization is divided into three steps, and the angular points of first, adjacent parking stall fuse, fuses two angular points of separation together and becomes a public angular point, but fuses two adjacent parking stalls and has only had public angular point, but the position of three angular point has not been optimized yet, the second, optimize the position of three angular point: firstly, as shown in formula (21), using the position and attitude of the parking space "seen" by the most excellent key frame with the highest score as the basis (x, y, angle), and then respectively comparing the position and attitude with the data of other key frames to respectively calculate a group of error values; secondly, as shown in the first part on the right of the equal sign of the formula (25), after a group of error values are respectively obtained by adopting the formula (21), the sum of squares is carried out, and then the sum of squares is multiplied by the lambda_ijl，(Λ_ijlFor assigning weights, Λ_ijlA diagonal matrix and a method for assigning weights) to obtain an optimal value, and finally taking the optimal value as the physical position of the current parking space closest to the actual parking space after optimization. Thirdly,And (4) optimizing three points in a collinear manner. Although the positions of three angular points of adjacent parking spaces are optimized to be more accurate, whether the positions of the three points are accurate or not is judged finally, whether the three points are collinear or not is judged, if the three points are not collinear and have angles, the positions of the three angular points are not accurate enough, at the moment, the optimization of the collinear angular points is carried out by adopting another method, the collinear three points are realized after the optimization, and the positions of the three angular points are further optimized. The three-point collinear optimization is divided into two steps, firstly, the errors of three-point collinear of a plurality of key frames are respectively solved, as shown in formulas (22), (23) and (24), then the second part on the right side of the equal sign of a formula (25) is passed, then the errors are passed through the square sum, and then multiplied by Λ_k，(Λ_kOnly diagonal lines have elements, and the diagonal line elements are all 1, and a method of weight distribution is performed on the square of the error of three-point collinearity of each item) to finally realize the optimization of three-point collinearity.

3. Multi-dimensional matching design principle

First, design difficulties. The incremental mapping is to match the currently seen parking space with the example map in the map from the top view, and if the current parking space is not matched, the current parking space is considered as a new parking space. The difficulty lies in that: it is difficult to judge whether the parking space is newly increased or the existing parking space or whether the parking space is neither newly increased nor the existing third undetermined parking space by only using any single information: 1) the position of the new parking space is not beyond or adjacent to the old parking space due to various factors, but one of 2 corner points of the new parking space is positioned on the existing parking space in the map, and the other is positioned beyond the existing parking space and accounts for 50 percent respectively, so that whether the new parking space is the new parking space cannot be judged only by the positions of the 2 corner points; 2) if the parking space number on the parking space is only used, the judgment cannot be carried out, because the parking space numbers of '1' and '7', '3' and '8' are often recognized wrongly; 3) even if 2 corners of the top view new parking space meet the conditions (adjacent corners or common corners) of the new parking space, the difference between the top view parking space posture and the example map parking space posture is 90 degrees, and the fact that the true posture of the new parking space is 90 degrees actually differs cannot be determined, or the difference is caused by misjudgment.

Second, solution: as shown in formula (6) -formula (14), 1) adopts 5 kinds of information comprehensive judgment. Except that the judgment is carried out by adopting three information of the parking position, the parking number and the parking posture (parking type), the fourth information is also adopted: and judging the information of adjacent parking spaces: when the parking space numbers "1" and "7", "3" and "8" are confused, the judgment can be carried out according to the parking space numbers of the adjacent parking spaces above, below, on the left and right of the current parking space except the position of the angular point, the left and right of the parking space number "7" should be "6" and "8", and the adjacent parking space of the parking space number "1" should be "3", so that the problem of wrong recognition of the parking spaces "1" and "7", and "3" and "8" is solved. A fifth type of information is also used: the overlapping degree of the parking space number detection frames, if the current top view is a new parking space, the parking space number detection frame (a square frame which is tightly attached to the design of the parking space number and is a parking space number detection frame) seen by the parking space cannot be overlapped with the existing parking space, the overlapping degree should be at the set position of the adjacent parking space, when one of 2 angular points of the new parking space falls on the existing example map parking space, the position of the new parking space can be further judged according to the position of the parking space number detection frame of the new parking space, because the area of the detection frame is smaller and more accurate than the area of the 2 angular points, if the detection frame falls on the position of the new parking space and is not at the position of the existing parking space of the example map, although one angular point position falls on the existing parking space of the example map, the parking space can be judged to be a new parking space, and incremental map building is performed. 2) The above 5 weight coefficients are allocated, the weight is not fixed, and the 5 weight coefficients can be dynamically adjusted according to the situation, which is not described herein.

Based on the principle of the invention, the invention designs a mapping method based on top view semantic objects.

A drawing method based on top view semantic objects is shown in figure 1, figure 1-2 and figure 1-3, and is characterized by comprising the following steps:

1) completing the initialization step before the map building;

supplementary explanation:

the top view of fig. 5-1 and 5-2 are different. The top view of fig. 5-1 is an initial top view formed by stitching of top view images of a range of multi-way fisheye cameras, the effect of which is shown in fig. 5-1. The default physical spatial range of the top view is 10 m x 10 m, the image resolution is 720 x 720, and the spatial dimension of each pixel is 13.88 mm, i.e., the Scale factor Scale is 13.88. At the moment, the top view keeps picture information, but does not have information such as detected parking space angular point coordinates, detected frame midpoint coordinates, parking space number identification results and the like. The initial top view includes only the image center coordinates, the top view vehicle rear axle center coordinates.

1) converting the fisheye camera coordinates to top view coordinates;

2) according to the formula (2), R under different conditions of the formula (1)_P、t_PSolving to obtain a corresponding external parameter matrix;

p＝HP....(2)

A. a fisheye camera external parameter matrix under a flat road surface;

4) correcting the top view image information in bump in real time;

1) obtaining a top view corrected in real time at each moment;

supplementary explanation:

as shown in fig. 5-2, the top view is a top view after neural network learning, and compared with the top view of fig. 5-1, the top view is added with parking space position information, which is recognized after deep neural network learning. Fig. 5-2 is a schematic diagram of a detection corner point of a detected parking space, a detection frame of a parking space number and an identification result in a top view, wherein a small circular point in the diagram represents a coordinate of the corner point of the parking space, adjacent parking spaces are relatively close to each other, and the corner points may be mutually shielded. The small triangle points in the figure represent the coordinates of the points in the detection box. The small diamond points in the figure represent the image center coordinates. The small hexagonal dots in the figure represent the coordinates of the center of the rear axle of the vehicle in a top view. The dotted line points to a part where the content of the detection box is recognized. After the detection of the deep neural network model, the information of the top view comprises not only the center coordinates of the image and the center coordinates of the rear axle of the vehicle in the top view, but also the coordinates of the parking space angular points, the center points of the parking space number detection frames in the top view, the parking space number identification result and the like.

Similarly, the neural network can also recognize top view parking space type information, the parking space types mainly include three types, namely a horizontal parking space, a vertical parking space and an inclined parking space, and fig. 5-2 only shows the parking space positions recognized after the deep neural network learning, but does not include the parking space types.

Supplementary explanation:

the patent relates to the transformation from the top view coordinate to the world coordinate system (namely the world coordinate system), and comprises a first step process 4), a second step process 2 and a second step process 3). Specifically, in the top view coordinate system, the top left corner of the top view is taken as the origin, the x-axis points to the rear of the vehicle, the y-axis points to the right of the vehicle, and the z-axis points upward. The vehicle coordinate system takes the center of the rear axle of the vehicle as an origin, the x-axis points to the front of the vehicle, the y-axis points to the left of the vehicle, and the z-axis points upwards. The transformation relationship between the two is as follows:

in the above formula, scale is a scale factor between the top view coordinate system and the vehicle coordinate system, describing the ratio between the picture pixels and the real world length; x is the number of_p、y_pIs the position of a point in the coordinate system of the top view, x_b、y_bIs the position of a point in the vehicle coordinate system, and Δ x is the absolute value of the difference in distance between the center point of the top view and the center of the rear axle of the vehicle in the top view. Formula equal sign right sideThe first term can convert the coordinate information of the point in the top view coordinate system represented by the second term into a vehicle coordinate system with the vehicle center as an origin, and then the third term which represents the distance from the top view center point to the vehicle rear axle center multiplied by the scale factor can obtain the coordinate in the vehicle coordinate system with the vehicle rear axle center as the origin.

Further, the enhanced characteristic parking space under the vehicle coordinate system is converted into the world coordinate system, and the conversion relationship between the enhanced characteristic parking space and the world coordinate system is as follows:

in the above formula, x_b、y_bIs the position of a point in the vehicle coordinate system, x_w、y_wIs the coordinate under the world coordinate system, and x, y and theta are the vehicle pose under the world coordinate system. The first item on the right side of the equation equal sign represents a rotation matrix obtained by calculation of a heading angle of a vehicle pose, the second item represents position information of a point under a vehicle coordinate system, translation is carried out through rotation of the first item and the third item, and finally coordinates under a world coordinate system can be obtained.

The two steps of coordinate transformation are carried out on the enhanced feature parking space of the top view, two angular points of the enhanced feature parking space in the top view and the middle point (x1, y1; x2, y 2; x, y) of the detection frame can be converted into a world coordinate system, and therefore the enhanced feature parking space observed for the first time is built into an example map, and initial map building is completed. And (3) only carrying out scale transformation on the length and width information (w, h) of the detection frame of the enhanced characteristic parking space, namely multiplying by a scale factor in a formula, and obtaining the length and width information in the global coordinate through the top view pixels and the scale factor. And the angle information alpha of the detection frame of the enhanced characteristic parking space is added with the vehicle heading angle theta to obtain the angle information in the global coordinate. And the recognition results of the parking place types and the parking place numbers of the characteristic parking places are enhanced to be used as non-spatial position information, and then the conversion is not carried out.

a. matching by using the parking position information to obtain f_d，f_dRepresenting the coincidence degree of the current detected parking space position information based on the new top view and the parking space position information of the example map; f. of_dAlso known as the position error cost;

c. matching by using the similarity of the parking spaces to obtain f_b,f_bRepresenting the similarity of the current detected parking space number based on the new top view and the parking space number of the example map; f. of_bAlso known as the parking space number similar cost;

||P_A-P_B||＜ΔS....(4)

wherein, P_AA certain corner position representing the A parking space, i.e. P_A＝[x_A，y_A]，P_BA certain corner position representing B parking space, i.e. P_B＝[x_B，y_B]The delta S represents a distance threshold value of adjacent angular points of two parking spaces;

f＝ω_df_d+ω_tf_t+ω_bf_b+ω_nf_n+ω_rf_r....(5)

F is_d、f_t、f_b、f_n、f_rThe calculation formula of (a) is as follows:

supplementary explanation:

example slot such as slot N123 has two corner coordinates of [2, 2%]、[2，4.5]，x_a＝(2+2)/2＝2，y_a(2+4.5)/2 ═ 3.25; parking space N123 is converted into two corner point coordinates [1.8, 2.1 ] in the world coordinate system in the new top view]、[1.8，4.6]，x_b＝(1.8+1.8)/2＝1.8，y_b＝(2.1+4.6)/2＝3.35；f_d＝0.22；

Wherein, type_aIs the type of parking space of the example map parking space_bIs the type of stall in the new top view;

supplementary explanation:

for example, the parking space number character string of the map parking space is N124, the recognition result in the top view is N123, where N is N, 1 is 1,2 is 2, 3 is not equal to 4, and the above results are accumulated, so that f is_nThe result of (2) is 3. Namely, the number of the parking space in the map of the example is compared with the number of the parking space in the top view one by one to judge whether the letter or the number of each parking space in the map of the example is the same.

supplementary explanation:

as shown in fig. 3-1, a is the area of the position-number detection box in the example map, and B is the area of the new top view where the position-number detection box is projected under the world coordinate system, assuming that the two areas are equal. The left side of the figure shows that A and B have no overlapping area at all, A.andU.B means that the area of the overlapping part of two detection frames is calculated, A.u.B means that the area of the two detection frames is summed, the repeated part is counted only once, and then the formula (9) is substituted, A.andU.B is 0, A.andU.B is 2 A.2B, f._bEqual to 0; in the middle of the diagram, a and B have half the overlapping area, so that a ≈ B equals to half the area of the detection frames, and a ═ B means the sum of the areas of the two detection frames, and the repeated part is counted only once, thereby substituting into formula (9), a ≈ B/2 ═ a/2 ═ B/2, a ═ B ═ 3A/2 ═ 3B/2, f ═ B ═ 3A/2 ═ 3B/2, and f ═ B ═ f_bEqual to 1/3; the right side of the diagram shows that A and B completely overlap, so that A ≈ B is equal to the area of the detection frame, and A ≈ B means that the sum of the areas of the two detection frames is obtained, and the repeated part is counted only once, thereby substituting into formula (9), wherein A ≈ B, A ═ B ═ A ═ B, f ═ B ═ A ═ B, and f ═ B_bEqual to 1.

f_r＝ω_nlf_nl+ω_nnf_nn+ω_atf_at....(10)

Wherein, ω is_nl、ω_nn、ω_atAre respectively corresponding weight coefficients, f_nlIs the similarity degree of the example map parking space of the adjacent parking spaces on the parking space and the parking space number of the observation characteristic parking space, f_nnThe similarity degree of the example map parking space of the adjacent parking space under the parking space and the parking space number of the observation characteristic parking space; f. of_atWhether the parking space distribution type of the parking space in the sliding window is the same as the example parking space distribution type in the map or not is shown, and the type_arIs the type of the space distribution of the parking spaces of the example map_brThe parking space distribution type in the sliding window is adopted.

Supplementary explanation:

the formula (10) is composed of the formulas (11), (12), and (13), and coefficients are assigned according to a certain weight. Formulas (11) and (12) represent the similarity between the sample parking space of the upper and lower adjacent parking spaces and the parking space number of the observation characteristic parking space, and the detailed calculation refers to the supplementary explanation of formula (8). The parking space spatial distribution type of the formula (13) indicates whether the parking space is located at the intersection, as shown in fig. 3-2, the dashed frame is an example parking space in the sliding window, the solid frame is an example parking space of the map, and the parking space spatial distribution type can be mainly divided into an intersection parking space and a non-intersection parking space, where N124, N125, N128, N111, N112, and N115 are intersection parking spaces, and N123, N126, N127, N110, N113, and N114 are non-intersection parking spaces. The sliding window is composed of position information of observation parking spaces of a plurality of nearest continuous key frames and position and posture information of the key frames.

the common angular points of the adjacent parking spaces are fused: and adjusting the shared angular point between the parking spaces of the example map by using the observation result of the relative relation between the adjacent parking spaces in the new top view: if the situation that two adjacent parking spaces share one angular point exists in multiple observations in a sliding window in a new top view, two angular points with errors of the two corresponding adjacent parking spaces are fused in an example map, so that the two parking spaces both have the angular point, and only one angular point position information is optimized in the later optimization process; marking the collinear relative relation of the angular points between the adjacent parking spaces in the process 2) of the third step,

according to the relative relationship between the parking spaces in the new top view, marking the collinear relative relationship between the angular points of the parking spaces in the example map, and the specific process is as follows:

1) acknowledging key frames

supplementary explanation:

as shown in fig. 2-1, assuming that the driving direction of the vehicle is from left to right, when the vehicle "sees" two new parking spaces, N123 and N124, in fig. 1, because the vehicle "sees" the new parking spaces, the frame of the current top view is determined as the key frame; as shown in fig. 2-2, when the vehicle continues to move forward from left to right, the original N123 and N124 slots are seen, and the new N125 slots are also seen, so that the current frame is determined to be the key frame.

b. Determining keyframes from distances

the formula for interpolating key frames according to distance is as follows:

‖P_k+1-P_k‖＞ΔP....(15)

supplementary explanation:

as shown in fig. 2-3, when the vehicle continues to move forward, there is no new "seen" space in front of the N125 space and the distance from the vehicle to move forward beyond the N125 space of fig. 3 exceeds a certain threshold, which is identified as the key frame by equation (15); but the key frame at the moment only has 'pose information of the frame' but no 'parking space position' information. The key frame without "slot position" information is confirmed because the process 5) of the subsequent step three is to optimize the position of the key frame first, not to optimize the slot. If the distance between the two frames is too far, the pose effect of the optimization key frame is poor, in order to keep the optimization effect, the key frame is determined according to the distance, and the key frame is determined as long as the distance is reached.

‖θ_k+1-θ_k‖＞Δθ....(16)

supplementary explanation:

as shown in fig. 2-4, the heading angle of the vehicle at different times changes, and the change of the heading angle affects the observation of the parking space position, so the pose of the "frame" at this time is important and marked as a key frame. The keyframes with a heading angle difference exceeding the threshold are confirmed because the process 5) of the next step three) is to optimize the vehicle attitude of the keyframe first, not the optimal slot. If the difference value of the course angles between the two frames exceeds the threshold value, the attitude effect of the optimized key frame is poor, in order to keep the optimization effect, the key frame is confirmed according to the difference value of the course angles, and the current frame is confirmed as the key frame as long as the difference value of the course angles between the current frame and the previous key frame exceeds the threshold value.

2) Removing key frames:

L_min＝min(L_min,L)....(17)

L_max＝max(L_max,L)....(18)

wherein L is_minThe minimum distance from the center of the top view to the center of the top view is the nearest observation, namely the parking space angular point in the top view or the midpoint of the parking space number detection frame; l is_maxFor the farthest observation, namely the maximum distance from the center of the top view to the center of the parking space angular point or the midpoint of the parking space number detection frame in the top view, L is the distance from the center of the top view to the center of the parking space angular point or the midpoint of the parking space number detection frame in the top view of the current key frame, and g is the observation score in the observation weight of the current key frame;

supplementary explanation:

1) the application conditions of the equations (17), (18), and (19) are that the current frame is the 2 nd or more frame, and if only the 1 st frame has not yet been the 2 nd frame, the minimum value L is_minMaximum value L_maxThe default is that the scores g are respectively maximum values (10, 10 and 10), and the default is that: when the first frame observes an N123 slot, there is no comparison of other frames, so it is considered to be the maximum. For example, the vehicle finds the space from a distance N123, which is a maximum value by default because it is only a long distance. However, as the vehicle moves forward, the distance is closer to the N123 parking space, the score is higher as the distance is closer, so the score of the first frame changes, and the score of the first frame is lower until the shortest distance is reached.

2) Minimum value L with respect to equation (19)_minMaximum value L_max: minimum value L_minMaximum value L_maxThe minimum value and the maximum value of the key frame of the parking space A class angular point A, A ' … …, the minimum value and the maximum value of the B class angular point B, B ' … … and the minimum value and the maximum value of the C class detection frame central point C, C ' … … (if the key frame of the N123 parking space is seen, 2 key frames are provided); for example: suppose that when N123 is observed in the first frame, the observed values of the two corner points of the parking space and the middle point of the detection frame are {2.8, 4.9, 4.4}, and when the second frame of N123 is observed,the observation values of the two corner points of the parking space and the midpoint of the detection frame are {3.2, 2 and 3.25}, and the minimum value L is calculated_minWhen comparing the same kind of corner points A, A 'and B, B' of the first frame and the second frame, and comparing the same kind of detection box midpoint C, C ', the minimum value relative to A, A' is 2.8 compared with 3.2 and 2.8, the minimum value relative to B, B 'is 4.9 compared with 2, the minimum value is 2, and the minimum value relative to C, C' is 3.25 compared with 4.4 and 3.25; similarly, the maximum value relative to A, A ' is 4.9 compared to 2.8 and 4.9, the maximum value relative to B, B ' is 4.9 compared to 4.9 and 2, and the maximum value relative to C, C ' is 4.4 compared to 3.25; if the key frames of the N123 parking spaces are 6, the minimum value of the A-type angular points is the minimum value of the 6A-type angular points, and is not the minimum value of the previous frame compared with the next frame.

3) The formulas (17), (18) and (19) are designed for each corner point and the central point of the detection frame, if only two frames of data exist, 3 minimum values, 3 maximum values and 3L values exist, and after the minimum values, the maximum values and the L values are respectively substituted into the formula (19), 3 fractional values are obtained;

4) regarding L: l is the distance from the center of the top view to the center of the parking space corner point or the midpoint of the parking space number detection frame in the top view of the current score-calculated key frame, which means: if the current calculated key frame score is the score of the previous frame, L is the distance from the center of the parking space angle point or the center of the parking space number detection frame to the center of the top view in the previous frame top view, and if the current calculated key frame score is the score of the latest generated key frame, L is the distance from the center of the parking space angle point or the center of the parking space number detection frame to the center of the top view in the latest generated top view.

5) Recalculate the meaning of the fraction of the last key frame: comparing the score of the latest key frame with the score of the last key frame after recalculation, assuming that the detected vehicle finds the cut-off from the point A to the point C and passes through A, B, C points, wherein the vehicle is closest to the point N123 at the point B, but the vehicle does not stop when reaching the point B but continues to move forward until reaching the point C, when the vehicle reaches the point C, the key frame score of the point B is known to be the nearest key frame only by comparing the key frame score of the point C with the key frame score of the point B, which requires recalculating the key frame score of the point B at the point C after the key frame score of the point C is calculated, and since the minimum value and the maximum value of the formula (19) are changed when the vehicle reaches the point C, the minimum value in the formula (19) is recalculated, the minimum value in the formula (19), the point C, The maximum value and the L value are changed, so the B point key frame score is recalculated differently from the first calculated value, and if the B point key frame score is not recalculated, the B point key frame score is possibly lower than the C point key frame score, and actually the score of the B point closest to the parking space is the highest. The reason is that: when the key frame score of the point B is calculated for the first time, the vehicle is positioned at the point B, and the point B is only compared with the point A to determine the minimum value, the maximum value and the L value, so that the score of the point B is possibly lower than the score of the vehicle reaching the point C, when the vehicle reaches the point C, the score of the point B is recalculated by using the new minimum value, the new maximum value and the new L value, because the distance between the point C and the point B is shortest, and the score of the point B is recalculated by using the new minimum value, the new maximum value and the new L value, and the key frame score of the point B is higher than the key frame score of the point C.

f＝w_bw_l(∑(w_cg_c)+∑(w_ng_n))....(20)

and after other key frames are confirmed, calculating the parking space observation scores of the key frames according to the steps, updating the observation weight of the key frames, and removing the key frames when the observation weight is smaller than a threshold value.

the specific expression is as follows:

the above formula (21) is to find a difference value between the ith parking space coordinate information of the example map and the jth parking space observation information converted into the world coordinate system parking space coordinate information in the jth key frame, where the difference value is a difference value in the world coordinate system, and specifically is: the item 1 on the right of the equal sign of the formula (21) is the world coordinate information of the ith parking space in the example map, and the initial value of the value is the coordinate information obtained by converting the observation information with the highest observation score in the observation results of the same parking space by the plurality of key frames reserved in the step eight into the world coordinate system; the data in the parenthesis of the 2 nd item on the right side of the equal sign of the formula (21) is that the ith parking space observed in the vehicle coordinate system is converted into coordinate information in the world coordinate system by the jth key frame, and the difference value of the ith parking space of the example map and the ith parking space in the jth key frame observation information converted into the coordinate information in the world coordinate system can be obtained by subtracting the 2 nd item from the 1 st item on the right side of the equal sign of the formula (21), which is concretely as follows:

a. item 1 to the right of the equal sign

c. the matrixes of

items

d. the 2 nd matrix in the right brace with equal sign

Supplementary explanation:

equation (21) the 2 nd column of the equal-sign right brace placed after the 1 st column rotation matrix is the desired expression: the 2 nd column matrix [ x ]_j y_j θ_j]^TIn the sliding window of (1)The pose of the keyframe vehicle in the world coordinate system is "seen" after the 1 st column rotation change, and the parenthetical 3 rd column translation matrix placed after the 2 nd column is the intended expression: and translating the pose of the j key frame vehicle in the world coordinate system in the rotated sliding window.

supplementary explanation:

the application premise of the step three process 5) is that the vehicle runs through a certain area A and a certain area B of the parking lot, the physical positions of the two areas are very close, and a closed loop can be formed. The physical positions of the area A and the area B are close, the observed physical positions of the parking spaces contained in the area A and the area B are very close, the two key frames can be fused, the poses of the two key frames after fusion are the same, the error is zero, after the error is zero, the key frame of the first parking space is reversely pushed back to reduce the error of each key frame one by one until the error of the last key frame is 0, so the design premise of the step three process 5) is that the pose optimization of the key frames can be carried out, and the process 4 of the step three can not be repeated after the pose optimization). If the physical positions of the parking spaces of the underground parking lot cannot form a closed loop, but two end points of a straight line are adopted, the key frames at one end and the other end cannot be fused, the precondition of pose optimization is not provided, and at the moment, the process 4) of the third step is repeated, so that the meaning is not provided.

in the above formula (26), T_ijRepresenting relative motion between key frames of the ith and jth frames, T_iAnd T_jRespectively representing the poses of the ith and jth frame key frames;

is T_iThe inverse matrix of (1), the multiplication of both being equal to the identity matrix;

in the above formula (27), e_ijRepresenting the error formed by pose transformation between the ith frame and the jth frame key frame;

is T_ijThe inverse matrix of (d); ln represents the logarithm operation of the result in the brackets; the V-shaped symbol represents the plum group form and is transformed into a plum algebraic form;

in the above formula (28), F is a total cost function, and represents the sum of squares of errors formed by pose transformation; e.g. of the type_ijRepresenting the error formed by the pose transformation between the ith frame and the jth frame key frame,

is e_ijA diagonal transformation matrix;

supplementary explanation:

as shown in fig. 4, the vehicle keeps the poses of 9 keyframes, T1 and T2.. T9, respectively, during the driving process. Corresponding to the formula (26),

then can pass through T₁Inversion to obtain, T_j＝T₂By passing

Then T can be found_ij＝T₁₂. During the running of the vehicle, T can be obtained by the formula (26)₁₂、T₂₃...T₈₉. At this time, T can be passed₉Directly calculate its sum T₁Relative pose change condition between them, because of T₁Can calculate the sum of T₂Situation of pose change, T₂Can calculate the sum of T₃The pose transformation situation of (1) is integrated to obtain T₁And T₃The pose of (1) is changed, so that T can be obtained step by step₁To T₉The pose change condition between. As shown by the dotted line in the figure, T₉₁Can be obtained by recursion of the relative relation among a plurality of poses, and simultaneously, T₉₁Or can pass through T₁And T₉The relative pose transformation is directly solved by observing the same parking space N123, and at the moment, consideration needs to be given to which pose transformation is more accurate. The former, relatively speaking, involves recursive, cumulative error comparisons between multiple posesLarge, so that the former needs to be adjusted by using the latter and thus will constitute T₉₁T of₁₂、T₂₃...T₈₉Are adjusted to form equation (27). The error function consisting of two poses and the relative pose is represented in equation (27) because

Is T_ijIs obtained from the formula (26)

The product of inverse matrices is the identity matrix, so

Therefore, it is not only easy to use

ln should go to zero after internal logarithm. Then, pose transformation T between every two poses between T1 and T2₁₂、T₂₃...T₈₉、T₉₁The sum of the squares of the formed error functions yields the total cost F in equation (28). Equations (29), (30) represent the two adjacent keyframe poses.

The above description is not meant to be limiting, it being noted that: it will be apparent to those skilled in the art that various changes, modifications, additions and substitutions can be made without departing from the true scope of the invention, and these improvements and modifications should also be construed as within the scope of the invention.

Claims

1. A mapping method based on top view semantic objects is characterized by comprising the following steps:

1) completing the initialization step before the map building;

4) local map optimization is carried out by utilizing the key frame with the parking places, and the position of each parking place of the adjacent parking places in the map, three points of the adjacent parking places are collinear, and image scale factors are optimized; and each parking space position comprises two parking space angular points of each parking space and the midpoint of the parking space number detection frame.

2. The mapping method based on the semantic objects of the top view according to claim 1, wherein in the step one, the top view is generated based on image data captured by a plurality of fisheye cameras collected by a vehicle end, and the real-time correction of the top view is completed in the following specific process:

1) converting the fisheye camera coordinates to top view coordinates;

p＝HP....(2)

correlation between formula (1) and formula (2)The association relationship is: obtaining R of formula (1) by decomposing H in the matrix of formula (2)_P、t_P(ii) a Thereby obtaining [ R ] under different conditions_p t_p]In this different case [ R ]_p t_p]The method comprises the following steps:

A. a fisheye camera external parameter matrix under a flat road surface;

4) correcting the top view image information in bump in real time;

3. The mapping method based on the top view semantic object according to claim 1, wherein the top view obtained in the first step process 3) based on the real-time correction is subjected to enhanced feature parking space information, and the specific process is as follows:

1) obtaining a top view corrected in real time at each moment;

4. The mapping method based on the top view semantic object according to claim 1, wherein in the first step, process 4), the initial mapping is completed according to the enhanced feature parking space information and by combining the information of the inertial measurement unit and the wheel speed meter, specifically as follows:

1) initializing and establishing a graph: based on the enhanced characteristic parking space information, combining the information of an inertia measurement unit and wheel speed meter to complete initialization map building, wherein the initialization map building is to obtain a key frame for observing the parking space for the first time and build an example map by using the key frame for observing the parking space for the first time; the key frame is a frame for observing the parking space for the first time, and comprises observed parking space information and vehicle pose information holding the frame;

5. The mapping method based on the top view semantic object according to claim 1, wherein the KM algorithm in the second step process 2) is used to match the example parking space of the example map with the enhanced feature parking space of the new real-time corrected top view, specifically:

b. matching by using parking space category information to obtain f_t，f_tRepresenting whether the current detected parking space category based on the new top view is the same as the parking space category of the example map, f_tAlso known as parking space category cost;

c. matching by using the similarity of the parking spaces to obtain f_b，f_bRepresenting whether the current detected parking space number based on the new top view is similar to the parking space number of the example map; f. of_bAlso known as the parking space number similar cost;

||P_A-P_B||＜ΔS....(4)

f＝ω_df_d+ω_tf_t+ω_bf_b+ω_nf_n+ω_rf_r....(5)

6. The method of claim 5, wherein f is the same as f_d、f_t、f_b、f_n、f_rThe calculation formula of (a) is as follows:

f_r＝ω_nlf_nl+ω_nnf_nn+ω_atf_at....(10)

7. The mapping method based on the top view semantic object according to claim 1, wherein the step three process 1) is performed to complete the corner point fusion between the new parking space and the adjacent parking space in the example map,

the specific process is as follows:

marking the collinear relative relation of the angular points between the adjacent parking spaces in the process 2) of the third step,

marking the collinear relative relation of the angular points of the parking spaces in the example map according to the relative relation of the parking spaces in the new top view,

the specific process is as follows:

8. The mapping method based on the top view semantic object according to claim 1, wherein the key frames with parking spaces and the key frames without parking spaces in the step three process 3) are determined and removed for later map optimization, and the specific process is as follows:

1) acknowledging key frames

b. determining keyframes from distances

the formula for interpolating key frames according to distance is as follows:

||P_k+1-P_k||＞ΔP....(15)

wherein, P_kRepresenting the vehicle centre position at time k, i.e. P_k＝[x_k，y_k]，P_k+1Representing the vehicle centre position at time k +1, i.e. P_k+1=[x_k+1，y_k+1]If the distance between the vehicle center at the moment K and the vehicle center at the moment K +1 is greater than the distance threshold, confirming a new key frame;

||θ_k+1-θ_k||＞Δθ....(16)

2) removing key frames:

L_min＝min(L_min，L)....(17)

L_max＝max(L_max，L)....(18)

wherein L is_minFor recent observation, i.e. the top viewThe minimum distance between the center point of the parking space corner or the middle point of the parking space number detection frame and the center of the top view; l is_maxFor the farthest observation, namely the maximum distance between a parking space angular point or the midpoint of a parking space number detection frame in the top view and the center of the top view, L is the distance between the parking space angular point or the midpoint of the parking space number detection frame in the top view of the current calculated key frame and the center of the top view, and g is the observation score in the observation weight of the current key frame;

wherein f represents the observation weight of the key frame, w_b、w_lWeight coefficient, g, representing vehicle bump, illumination effect_c、g_nRepresenting the angular point, the number of the parking space observation score, w_c、w_nWhether the angular point and the parking space number are the nearest observation or not is represented.

9. The mapping method based on the top view semantic object according to claim 1, wherein in the third step, process 4), local map optimization is performed by using a key frame with parking spaces, and the position of each parking space of adjacent parking spaces in the map, three points of the adjacent parking spaces are collinear, and image scale factors are optimized; each parking space position of the adjacent parking spaces comprises a parking space angular point of each parking space and a middle point of a parking space number detection frame;

the specific expression is as follows:

the above formula (21) is to find a difference value between the ith parking space coordinate information of the example map and the jth parking space observation information converted into the world coordinate system parking space coordinate information in the jth key frame, where the difference value is a difference value in the world coordinate system, and specifically is: the item 1 on the right of the equal sign of the formula (21) is the world coordinate information of the ith parking space in the example map, and the initial value of the value is the coordinate information converted from the observation information with the highest observation score in the observation results of the same parking space by the plurality of key frames reserved in the third step process 3) to the world coordinate system; the data in the parenthesis of the 2 nd item on the right side of the equal sign of the formula (21) is that the ith parking space observed in the vehicle coordinate system is converted into coordinate information in the world coordinate system by the jth key frame, and the difference value of the ith parking space of the example map and the ith parking space in the jth key frame observation information converted into the coordinate information in the world coordinate system can be obtained by subtracting the 2 nd item from the 1 st item on the right side of the equal sign of the formula (21), which is concretely as follows:

a. item 1 to the right of the equal sign

c. the 1 st item and the 3 rd item in the 2 nd item brace on the right of the equal sign represent the rotation and translation changes of the vehicle, wherein, the 1 st item is the rotationItem 3 is a change in translation, matrix [ x ]_j y_j θ_j]The pose of the j-th key frame vehicle in the sliding window in the world coordinate system, wherein the sliding window is a combination formed by continuous frames of key frame observation information and key frame pose information of the vehicle in a moving state;

d. the 2 nd matrix in the right brace with equal sign

to be provided with

The formula (23) and the formula (24) represent vectors formed by 2 corner points of adjacent parking spaces respectively, and the formula (22) represents cross multiplication between the two vectors of the formula (23) and the formula (24); if the vector cross multiplication result approaches zero, the three points tend to be collinear; wherein i and i-1 represent the ith parking space and the ith-1 parking space;

10. The mapping method based on top view semantic objects according to claim 1, wherein the third step further comprises a process 5): the method comprises the following steps of performing loop detection by using a key frame with parking space information, optimizing the pose of the key frame in a map by using the key frame with parking space information and a key frame without parking spaces, and optimizing parking space angular points and the middle points of a parking space number detection frame in the map again: the loop back detection comprises: the method comprises the following steps of searching parking spaces in a sliding window, wherein the parking spaces are similar to the parking spaces in a map or have close distances, matching and associating the parking spaces in the sliding window with example parking spaces in the map by using a KM algorithm, and if a certain number of matching exists, considering that loop occurs, wherein the specific process is as follows:

is T_iThe multiplication is equal to the identity matrix;

is e_ijA diagonal transformation matrix;

in the above formula, i and j represent key frame number, T_iAnd T_jRespectively representing the poses of the ith and jth frame key frames; [ x ] of_j y_jθ_j]The pose of the j-th key frame vehicle in the sliding window in the world coordinate system is constructed into T through form transformation_jIn the form of (a);