CN115421158B - Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device - Google Patents

Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device Download PDF

Info

Publication number
CN115421158B
CN115421158B CN202211387608.5A CN202211387608A CN115421158B CN 115421158 B CN115421158 B CN 115421158B CN 202211387608 A CN202211387608 A CN 202211387608A CN 115421158 B CN115421158 B CN 115421158B
Authority
CN
China
Prior art keywords
point cloud
dimensional
semantic
state
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211387608.5A
Other languages
Chinese (zh)
Other versions
CN115421158A (en
Inventor
陈昶昊
李方言
王雄飞
何晓峰
褚超群
潘献飞
毛军
张礼廉
范晨
胡小平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN202211387608.5A priority Critical patent/CN115421158B/en
Publication of CN115421158A publication Critical patent/CN115421158A/en
Application granted granted Critical
Publication of CN115421158B publication Critical patent/CN115421158B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3833—Creation or updating of map data characterised by the source of data
    • G01C21/3841—Data obtained from two or more sources, e.g. probe vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Optics & Photonics (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

The invention discloses a solid-state laser radar three-dimensional semantic map building method and device for self-supervised learning. The light-weight laser radar/inertia combined mapping algorithm based on Kalman filtering carries out scene three-dimensional reconstruction, and the three-dimensional point cloud map is semantically segmented through the point cloud semantic segmentation model, so that the semantic segmentation effect on the three-dimensional coordinate point cloud is good, and the point cloud segmentation understanding capability and the semantic segmentation precision are improved.

Description

Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device
Technical Field
The invention relates to the technical field of three-dimensional point cloud semantic segmentation, in particular to a solid-state laser radar three-dimensional semantic mapping method and device for self-supervised learning.
Background
Semantic segmentation is a typical computer vision problem that involves taking some raw data (e.g., flat images) as input and converting them into segmented regions with semantic meaning, forming a semantic map that facilitates understanding by robots and humans. The appearance of lightweight solid-state lidar has reduced lidar's cost and weight size, and is different with rotation type lidar, and small-size solid-state lidar's the angle of vision is less, and the scanning mode is irregular. The invention relates to a method and a device for constructing a graph aiming at dense three-dimensional point cloud semantics of a light and small solid-state laser radar. The method and apparatus may be used for mobile robots as well as hand-held devices.
Along with the rapid popularization of robots, the application of logistics distribution, household robots, medical robots and the like requires that the robots can independently navigate and establish images in various indoor and outdoor complex environments. The execution of the robot task needs to have the capabilities of autonomous navigation, map building and scene understanding. The laser radar can directly acquire information such as three-dimensional geometric position and reflection intensity of a target, the measuring distance can reach hundreds of meters, and the measuring precision is high. However, it is not enough for an indoor robot autonomous navigation to obtain only a three-dimensional point cloud map. The human body can distinguish roads, walls and other sundries from the point cloud map, but for the robot, the point cloud map in the eyes is a group of points with unknown meanings. And when the robot is required to effectively identify objects and regions with semantic meanings in the point cloud, semantic segmentation and image building are required. In recent years, the rapid increase of computational power enables the three-dimensional point cloud semantic segmentation algorithm based on deep learning to rapidly develop and improve the performance, and the research on the aspect also becomes a research hotspot in recent years.
The three-dimensional semantic mapping can be divided into two steps of semantic segmentation understanding and three-dimensional dense mapping of three-dimensional point cloud. In recent years, relevant research results focus on three-dimensional point cloud mapping and point cloud semantic segmentation. In the aspect of three-dimensional point cloud mapping, zhang et al provides a laser radar odometer and mapping algorithm based on batch optimization, and real-time three-dimensional point cloud mapping is realized. Lin et al further provides a positioning and mapping algorithm suitable for the small solid-state laser radar, and realizes feature extraction under a limited field angle and motion compensation and point cloud matching under an irregular sampling condition. And Xu et al, fusing the point cloud characteristics and inertial vision of the laser radar by adopting extended Kalman filtering, and realizing positioning and mapping with higher efficiency and robustness. In the aspect of three-dimensional point cloud matching, qi et al. And (3) projecting the point cloud into a strip picture, performing semantic segmentation on the picture and mapping the point cloud back to realize the construction and segmentation of the point cloud, wherein the algorithm is only suitable for the rotary laser radar. At present, a three-dimensional dense semantic mapping method for small solid-state lidar does not exist.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a method and a device for building a three-dimensional semantic map of a solid-state laser radar for self-supervision learning, which can be applied to mobile robots and handheld equipment. The method constructs a coding-decoding deep neural network, semantically segments real-time input three-dimensional point cloud based on a self-supervision learning mode, and adopts Kalman filtering to fuse laser radar point cloud data, semantic segmentation results and inertial data to construct a map so as to realize scene three-dimensional reconstruction. In addition, the reflectivity of the laser radar signal is introduced into a point cloud segmentation network of self-supervision learning, and therefore the point cloud segmentation understanding capability is effectively improved.
In order to achieve the purpose, the invention provides a solid-state laser radar three-dimensional semantic mapping method for self-supervision learning, which comprises the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and carrying out self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time inertia data through a small solid-state laser radar and an inertia measurement element, fusing the real-time three-dimensional point cloud and the real-time inertia data based on a Kalman filter, and outputting a dense three-dimensional point cloud map composed of the real-time point cloud data under global coordinates;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on computing equipment with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, carrying out three-dimensional point cloud, inertia data acquisition and map updating according to fixed frequency, carrying out real-time semantic segmentation, and generating a three-dimensional semantic map. Because the point cloud segmentation network is trained, the synchronous acquisition of RGB images is not required.
In order to achieve the above object, the present invention further provides a solid-state lidar three-dimensional semantic graph building apparatus for self-supervised learning, comprising:
the system comprises an RGB camera, a laser radar and an inertia measuring device, wherein the RGB camera, the laser radar and the inertia measuring device are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring RGB images, three-dimensional point cloud data and inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertia data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
and the three-dimensional semantic map building module is connected with the point cloud semantic division model and the three-dimensional point cloud map building module and is used for generating a three-dimensional semantic map by corresponding the point cloud division result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate.
The invention provides a method and a device for building a three-dimensional semantic map of a solid-state laser radar for self-supervised learning. In addition, the reflectivity of the laser radar signal is introduced into the point cloud segmentation network, so that the point cloud segmentation understanding capability is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
FIG. 1 is a flow chart of a solid-state lidar three-dimensional semantic mapping method in an embodiment of the invention;
fig. 2 is a block diagram of a three-dimensional semantic map building apparatus for solid-state lidar in an embodiment of the invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In addition, descriptions such as "first", "second", etc. in the present invention are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.
Example 1
Fig. 1 shows a three-dimensional semantic mapping method for a solid-state lidar for self-supervised learning disclosed in this embodiment, which specifically includes the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and carrying out self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time Inertial data through a small solid-state laser radar and an Inertial Measurement Unit (IMU), fusing the real-time three-dimensional point cloud and the real-time Inertial data based on a Kalman filter, and outputting a dense three-dimensional point cloud map composed of the real-time point cloud data under the global coordinate;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on computing equipment with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, carrying out three-dimensional point cloud and inertia data acquisition and map updating according to the frequency of 50Hz, carrying out real-time semantic segmentation, and generating a three-dimensional semantic map.
In this embodiment, the point cloud semantic segmentation model includes an encoding layer, a decoding layer, and a semantic prediction layer. The point cloud semantic segmentation model is input as
Figure 950697DEST_PATH_IMAGE001
Is collected and evaluated>
Figure 868975DEST_PATH_IMAGE002
Is the number of points, is greater or less>
Figure 238776DEST_PATH_IMAGE003
Is the characteristic dimension of each point, in this example 4 dimensions, including the three-dimensional coordinates ≥>
Figure 473579DEST_PATH_IMAGE004
And a reflectivity>
Figure 801793DEST_PATH_IMAGE005
。
The coding layer adopts a 3-layer coder, is mainly used for carrying out feature extraction on the input three-dimensional point cloud, sequentially processes point cloud data, reduces the number of the point cloud, increases the dimension of each point, and finally counts the point cloud
Figure 941787DEST_PATH_IMAGE006
Is reduced to be->
Figure 397170DEST_PATH_IMAGE007
And promoting the dimension of the point cloud feature from 4 dimensions to 128 dimensions. The encoder consists of a local feature aggregator and a random sampling layer, and specifically comprises the following parts:
the local feature aggregator consists of a local space encoder and an attention mechanism and aims to expand the receptive field of each point so as to extract more effective features;
the random sampling layer is used for accelerating the extraction of point features and improving the operation efficiency.
And the decoding layer designs a decoder with 3 layers, the nearest K adjacent points are found for each input point by adopting a K nearest neighbor algorithm (K nearest neighbors are found to represent) through the encoder of each layer, and the point cloud characteristic set is up-sampled by a nearest neighbor difference algorithm. Finally, the features obtained by the up-sampling are connected with the features obtained by the encoder to obtain the feature vector output by the decoding layer
Figure 673431DEST_PATH_IMAGE008
。
The semantic prediction layer is used for mapping the feature vector obtained by the decoding layer to the full-link layer and the linear occipital flow function ReLU
Figure 969283DEST_PATH_IMAGE009
，/>
Figure 347306DEST_PATH_IMAGE010
Is a semantic category, having>
Figure 324489DEST_PATH_IMAGE011
Class, is>
Figure 720835DEST_PATH_IMAGE012
The number of the midpoints of the three-dimensional point cloud is shown.
In the specific implementation process, the self-supervision training of the point cloud semantic segmentation model specifically comprises the following steps:
step 1.1, corresponding three-dimensional point cloud coordinates and RGB image pixels one by one through a geometric relationship, obtaining a semantic segmentation result of each pixel in an RGB image by adopting a trained RGB image segmentation model, and obtaining a point cloud semantic label through a corresponding relationship between the three-dimensional point cloud coordinates and the RGB image pixels, wherein the label obtained through the corresponding relationship of the RGB image is defined as an automatic supervision semantic label in the embodiment;
step 1.2, point characteristics output by point cloud segmentation network
Figure 938321DEST_PATH_IMAGE013
Corresponding to images by RGBComparing the obtained self-supervision semantic labels, and iteratively adjusting parameters of each level of the point cloud semantic segmentation model to enable semantic results output by the point cloud semantic segmentation model to approach the self-supervision semantic labels, wherein the process of iteratively adjusting the parameters is self-supervision learning training;
and step 1.3, after iteration for a certain number of times, a model with an output score closer to the self-supervision semantic label is left, so that a point cloud semantic segmentation model is formed.
It should be noted that the RGB image is only used for self-supervised training of the point cloud semantic segmentation model, and the trained point cloud semantic segmentation model can perform point cloud semantic segmentation only by inputting point cloud data, so as to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current time.
In this embodiment, the map building process of the dense three-dimensional point cloud map specifically includes:
step 2.1, performing inertial integration on the real-time inertial data to obtain a first initial state quantity (including position, attitude, speed and IMU error quantity) of the system, wherein the implementation process comprises the following steps:
IMU measurement noise
Figure 52908DEST_PATH_IMAGE014
0, the integral of inertia is:
Figure 833782DEST_PATH_IMAGE015
in the formula
Figure 350214DEST_PATH_IMAGE016
Indicate a relation>
Figure 941863DEST_PATH_IMAGE017
、/>
Figure 543746DEST_PATH_IMAGE018
、/>
Figure 128311DEST_PATH_IMAGE019
(here ` Harbour `>
Figure 499249DEST_PATH_IMAGE020
An inertia integration function of 0), based on a predetermined criterion>
Figure 261800DEST_PATH_IMAGE021
Representing an exponential function, the combination of which represents the inertial integration state update process; />
Figure 85400DEST_PATH_IMAGE022
I.e. two adjacent IMU sample times ∑ in one radar scan>
Figure 208076DEST_PATH_IMAGE023
And/or>
Figure 433521DEST_PATH_IMAGE024
Difference of (IMU sampling interval), ->
Figure 366974DEST_PATH_IMAGE025
Is the label of the ith IMU measurement data sample.
Figure 943448DEST_PATH_IMAGE026
Is the measurement data (i.e. </or >>
Figure 135395DEST_PATH_IMAGE027
Inertial data representing the ith IMU sample); />
Figure 215347DEST_PATH_IMAGE028
Indicates a system state, wherein>
Figure 585279DEST_PATH_IMAGE029
Indicating that the system is in the ^ th->
Figure 586733DEST_PATH_IMAGE030
The first status quantity at the subsampling instant->
Figure 51213DEST_PATH_IMAGE031
Presentation system in the fifth or fifth place>
Figure 251250DEST_PATH_IMAGE032
A first state quantity at the sub-sampling time;
step 2.2, based on the real-time three-dimensional point cloud, performing state updating on the first state quantity by adopting Kalman filtering to obtain a second state quantity of the system, wherein the implementation process comprises the following steps:
firstly, calculating the point cloud residual error of the laser radar three-dimensional point cloud
Figure 306931DEST_PATH_IMAGE033
The method comprises the following steps:
Figure 77572DEST_PATH_IMAGE034
in the formula (I), the compound is shown in the specification,
Figure 345742DEST_PATH_IMAGE035
is the nearest characteristic point position on the map, is>
Figure 400286DEST_PATH_IMAGE036
Is point->
Figure 643179DEST_PATH_IMAGE037
The normal vector (or edge orientation) of the respective plane (or edge) in which it lies, or->
Figure 415963DEST_PATH_IMAGE038
Estimating the point cloud position by the IMU;
iterative updating is carried out on the state estimation by adopting an iterative Kalman filter until the point cloud residual error
Figure 487824DEST_PATH_IMAGE039
And converging, wherein the process of iteratively updating the state estimation comprises the following steps:
Figure 131295DEST_PATH_IMAGE040
in the formula
Figure 810670DEST_PATH_IMAGE041
Is represented by>
Figure 8433DEST_PATH_IMAGE042
Moment (>
Figure 883985DEST_PATH_IMAGE042
Indicates the fifth->
Figure 913121DEST_PATH_IMAGE043
Scan end time of a sub lidar scan) th>
Figure 701079DEST_PATH_IMAGE044
Status after sub-kalman filtering>
Figure 448455DEST_PATH_IMAGE045
Is generated value of->
Figure 862119DEST_PATH_IMAGE046
Is indicated to be at>
Figure 11341DEST_PATH_IMAGE047
Is at a moment->
Figure 958482DEST_PATH_IMAGE048
Status after sub-kalman filtering>
Figure 927575DEST_PATH_IMAGE049
The generated value of (a) is,Irepresents a unit matrix, <' > based on>
Figure 348192DEST_PATH_IMAGE050
Represents an observation matrix, <' > based on>
Figure 86341DEST_PATH_IMAGE051
Represents->
Figure 730949DEST_PATH_IMAGE052
In connection with>
Figure 203650DEST_PATH_IMAGE053
Is based on the partial derivative of (4)>
Figure 959116DEST_PATH_IMAGE054
In an exponential function,>
Figure 20613DEST_PATH_IMAGE055
is a logarithmic function>
Figure 836122DEST_PATH_IMAGE056
Represents->
Figure 514228DEST_PATH_IMAGE057
Status of the moment>
Figure 89697DEST_PATH_IMAGE058
Is true and>
Figure 802438DEST_PATH_IMAGE059
and the generating value->
Figure 788849DEST_PATH_IMAGE060
Based on the error status of the status flag, based on the status flag>
Figure 423092DEST_PATH_IMAGE061
Indicates a generation value->
Figure 520361DEST_PATH_IMAGE062
And the evaluation value->
Figure 838341DEST_PATH_IMAGE063
An error therebetween;
Figure 261232DEST_PATH_IMAGE064
and representing a Kalman filtering gain matrix, wherein the calculation process is as follows:
Figure 913931DEST_PATH_IMAGE065
Figure 814891DEST_PATH_IMAGE066
Figure 705486DEST_PATH_IMAGE067
in the formula (I), the compound is shown in the specification,
Figure 784432DEST_PATH_IMAGE068
represents a covariance matrix, <' > based on a covariance matrix>
Figure 924426DEST_PATH_IMAGE069
Represents a fifth or fifth party>
Figure 629077DEST_PATH_IMAGE070
IMU state covariance at sub-sampling time, <' >>
Figure 170917DEST_PATH_IMAGE071
Is shown as
Figure 420764DEST_PATH_IMAGE072
IMU state covariance at sub-sampling time, <' >>
Figure 251316DEST_PATH_IMAGE073
Represents the covariance matrix pick>
Figure 494079DEST_PATH_IMAGE074
And partial derivative matrix->
Figure 156004DEST_PATH_IMAGE075
The intermediate variable which is generated, is taken>
Figure 91599DEST_PATH_IMAGE076
And &>
Figure 691339DEST_PATH_IMAGE077
Respectively represent->
Figure 737792DEST_PATH_IMAGE078
To (X)>
Figure 457487DEST_PATH_IMAGE079
And &>
Figure 298404DEST_PATH_IMAGE080
In a partial derivative matrix of>
Figure 634707DEST_PATH_IMAGE081
Indicates IMU noise->
Figure 235584DEST_PATH_IMAGE082
Covariance of (2), superscript
Figure 606523DEST_PATH_IMAGE083
Represents a transpose of a matrix;
in the above iterative update of the state estimate, the cloud residual is present at the point
Figure 618341DEST_PATH_IMAGE084
After convergence, an optimal state estimate, i.e. a second state quantity, can be obtained as follows:
Figure 910782DEST_PATH_IMAGE085
in the formula (I), the compound is shown in the specification,
Figure 33459DEST_PATH_IMAGE086
indicating that the system is in the ^ th->
Figure 275215DEST_PATH_IMAGE087
A second state quantity at the sub-sampling time;
and 2.3, updating the reverse state of the second state quantity to obtain a third state quantity of the system so as to optimize the estimation of the position and the attitude and improve the positioning accuracy, wherein the implementation process of the reverse state updating comprises the following steps:
Figure 457935DEST_PATH_IMAGE088
/>
in the formula (I), the compound is shown in the specification,
Figure 34410DEST_PATH_IMAGE089
presentation system in the fifth or fifth place>
Figure 960778DEST_PATH_IMAGE087
A third state quantity at the subsampling instant->
Figure 57041DEST_PATH_IMAGE090
Indicating that the system is in the ^ th->
Figure 613924DEST_PATH_IMAGE091
A third state quantity at the sub-sampling time;
step 2.4, obtaining a conversion matrix from the laser radar coordinate system to the IMU coordinate system based on the third state quantity
Figure 412116DEST_PATH_IMAGE092
And
Figure 142174DEST_PATH_IMAGE093
IMU coordinate system at a time instant>
Figure 342211DEST_PATH_IMAGE094
To the global coordinate system>
Figure 883045DEST_PATH_IMAGE095
Update the transition matrix->
Figure 168533DEST_PATH_IMAGE096
And pass through>
Figure 702283DEST_PATH_IMAGE097
And/or>
Figure 960089DEST_PATH_IMAGE098
Converting point cloud coordinates of each frame of self coordinate system in one scanning of the laser radar into coordinates of a coordinate system at the scanning end time to obtain global coordinates, wherein the global coordinates are as follows:
Figure 655512DEST_PATH_IMAGE099
in the formula (I), the compound is shown in the specification,
Figure 179029DEST_PATH_IMAGE100
represents the coordinates of the laser point cloud, and is used for judging whether the laser point is in the dark or not>
Figure 516469DEST_PATH_IMAGE101
Represents->
Figure 425519DEST_PATH_IMAGE102
Sub-scanning lidar frame>
Figure 760686DEST_PATH_IMAGE103
A global coordinate system is represented, and,
Figure 755186DEST_PATH_IMAGE104
indicates the fifth->
Figure 381471DEST_PATH_IMAGE105
A characteristic point->
Figure 145028DEST_PATH_IMAGE106
Represents a fifth or fifth party>
Figure 447833DEST_PATH_IMAGE107
A characteristic point is at>
Figure 195209DEST_PATH_IMAGE108
Sub-scanning the coordinates of the lidar frame>
Figure 359605DEST_PATH_IMAGE109
Represents a fifth or fifth party>
Figure 243248DEST_PATH_IMAGE110
Global coordinates of the feature points after coordinate conversion; />
Figure 716954DEST_PATH_IMAGE111
A coordinate conversion matrix representing the coordinate system from the lower right to the upper left>
Figure 889310DEST_PATH_IMAGE112
Represents the IMU coordinate system, is>
Figure 106664DEST_PATH_IMAGE113
Represents the lidar coordinate system>
Figure 595546DEST_PATH_IMAGE114
Represents a transformation matrix from the lidar coordinate system to the IMU coordinate system, based on the evaluation of the measured values>
Figure 505733DEST_PATH_IMAGE115
Represents->
Figure 962122DEST_PATH_IMAGE116
IMU coordinate system at a time->
Figure 717588DEST_PATH_IMAGE117
To the global coordinate system>
Figure 779085DEST_PATH_IMAGE118
Updating the transformation matrix;
Figure 79748DEST_PATH_IMAGE119
the number of point clouds;
and 2.5, adding all the feature points at each time to the existing map according to the global coordinate to obtain the three-dimensional point cloud map under the global coordinate system.
In the global mapping process of step 2.5, each point cloud has a fixed serial number and a feature quantity, and the feature quantity is a point cloud segmentation result obtained by the semantic segmentation model in step 3. And (4) corresponding the semantic segmentation result of the local point cloud obtained in the step (3) with the generated global three-dimensional point cloud map coordinate, thus obtaining a global three-dimensional semantic map.
Example 2
On the basis of the solid-state laser radar three-dimensional semantic map building method for the self-supervised learning in the embodiment 1, the embodiment also discloses a solid-state laser radar three-dimensional semantic map building device for the self-supervised learning, and referring to fig. 2, the device mainly comprises an RGB camera, a laser radar, an inertia measuring device, a data acquisition module, a point cloud semantic segmentation model, a self-supervised training module, a three-dimensional point cloud map building module and a three-dimensional semantic map building module. Specifically, the method comprises the following steps:
the system comprises an RGB camera, a laser radar and an inertia measuring device, wherein the RGB camera, the laser radar and the inertia measuring device are arranged on carriers such as a mobile robot or handheld equipment and are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring the acquired RGB image, the three-dimensional point cloud data and the inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertial data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
the three-dimensional semantic map building module is connected with the point cloud semantic segmentation model and the three-dimensional point cloud map building module and is used for enabling the point cloud segmentation result corresponding to each point cloud coordinate to correspond to the dense three-dimensional point cloud map coordinate to generate the three-dimensional semantic map.
In a specific application process, each functional module of the solid-state lidar three-dimensional semantic map building device is the same as that in embodiment 1, and therefore, the description thereof is omitted in this embodiment.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (8)

1. A solid-state laser radar three-dimensional semantic mapping method for self-supervision learning is characterized by comprising the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and carrying out self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time inertial data through a small solid-state laser radar and an inertial measurement element, fusing the real-time three-dimensional point cloud and the real-time inertial data based on a Kalman filter, and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under a global coordinate, wherein the dense three-dimensional point cloud map specifically comprises the following steps:
step 2.1, performing inertia integration on the real-time inertia data to obtain a first initial state quantity of the system;
2.2, based on the real-time three-dimensional point cloud, performing state updating on the first state quantity by adopting Kalman filtering to obtain a second state quantity;
step 2.3, updating the reverse state of the second state quantity to obtain a third state quantity of the system;
step 2.4, obtaining a conversion matrix from the laser radar coordinate system to an inertial coordinate system and an estimation updating conversion matrix from the inertial coordinate system to a global coordinate system based on the third state quantity, and converting point cloud coordinates of each frame of self coordinate system in one scanning of the laser radar to coordinates of a coordinate system at the last scanning moment to obtain global coordinates;
step 2.5, adding all the feature points at each time of adoption to the existing map according to the global coordinate to obtain a dense three-dimensional point cloud map under the global coordinate system;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on computing equipment with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, carrying out three-dimensional point cloud, inertia data acquisition and map updating according to fixed frequency, carrying out real-time semantic segmentation, and generating a three-dimensional semantic map.
2. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 1, wherein in the step 1, the point cloud semantic segmentation model comprises:
the encoding layer is used for extracting the characteristics of the input three-dimensional point cloud;
the decoding layer is used for finding K nearest adjacent points for each input point on the basis of the characteristics extracted by the coding layer, then carrying out up-sampling on the point cloud characteristic set through a nearest neighbor difference algorithm, and connecting the characteristics obtained by the up-sampling with the characteristics obtained by the coder to obtain a characteristic vector output by the decoding layer;
a semantic prediction layer for mapping the feature vectors obtained from the decoding layer to the full-link layer and the linear anchor function ReLU
Figure QLYQS_1
，/>
Figure QLYQS_2
Is a semantic category, having>
Figure QLYQS_3
Class, is>
Figure QLYQS_4
Is threeThe number of the midpoints of the dimensional point cloud.
3. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 2, wherein in step 1, the self-supervised training is performed on the point cloud semantic segmentation model, specifically:
step 1.1, corresponding three-dimensional point cloud coordinates and RGB image pixels one by one through a geometric relationship, obtaining a semantic segmentation result of each pixel in an RGB image by adopting a trained RGB image segmentation model, and obtaining a self-supervision semantic label through the corresponding relationship between the three-dimensional point cloud coordinates and the RGB image pixels;
step 1.2, point characteristics output by point cloud segmentation network
Figure QLYQS_5
Comparing with the self-supervision semantic label obtained through the corresponding relation of the RGB image, and enabling the semantic result output by the point cloud semantic segmentation model to approach the self-supervision semantic label by iteratively adjusting the parameters of each level of the point cloud semantic segmentation model;
and step 1.3, after iteration for a certain number of times, a model with an output score closer to the self-supervision semantic label is left, so that a point cloud semantic segmentation model is formed.
4. The method for self-supervised learning solid-state lidar three-dimensional semantic mapping according to claim 1, 2 or 3, wherein in step 2.1, inertial measurement noise is set
Figure QLYQS_6
0, the integral of inertia is:
Figure QLYQS_7
in the formula (I), the compound is shown in the specification,
Figure QLYQS_20
indicate a relation>
Figure QLYQS_8
、/>
Figure QLYQS_14
、/>
Figure QLYQS_13
Is integrated by inertia function of->
Figure QLYQS_18
Representing an exponential operator, the combination of which represents the inertia integral state update process; />
Figure QLYQS_22
I.e. two adjacent sampling instants->
Figure QLYQS_24
And/or>
Figure QLYQS_11
The difference between the two; />
Figure QLYQS_17
Is the first->
Figure QLYQS_12
The label of the sub-inertial data sample,>
Figure QLYQS_19
indicates the fifth->
Figure QLYQS_10
Real-time inertial data sampled by the secondary IMU; />
Figure QLYQS_15
Indicating the state of the system in which, among other things,
Figure QLYQS_21
indicating that the system is in the ^ th->
Figure QLYQS_23
The first status quantity at the subsampling instant->
Figure QLYQS_9
Presentation system in the fifth or fifth place>
Figure QLYQS_16
A first state quantity at the sub-sampling instant.
5. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 4, wherein the step 2.2 is specifically as follows:
firstly, calculating the point cloud residual error of the real-time three-dimensional point cloud
Figure QLYQS_25
The method comprises the following steps:
Figure QLYQS_26
in the formula (I), the compound is shown in the specification,
Figure QLYQS_27
is the location of the nearest characteristic point on the map, in combination with the characteristic point on the map>
Figure QLYQS_28
Is point->
Figure QLYQS_29
In the normal vector or edge orientation of the respective plane or edge in which it lies>
Figure QLYQS_30
Estimating the point cloud position by the IMU;
iterative updating is carried out on the state estimation by adopting an iterative Kalman filter until the point cloud residual error
Figure QLYQS_31
Convergence, wherein,the process of iteratively updating the state estimate is:
Figure QLYQS_32
in the formula (I), the compound is shown in the specification,
Figure QLYQS_46
is represented by>
Figure QLYQS_38
Is at a moment->
Figure QLYQS_40
Status after sub-kalman filtering>
Figure QLYQS_45
Is generated value of->
Figure QLYQS_51
Is indicated to be at>
Figure QLYQS_48
Moment k-th Carlman filtered status->
Figure QLYQS_52
Is generated value of->
Figure QLYQS_34
Represents a fifth or fifth party>
Figure QLYQS_42
The scan end time of the secondary lidar scan,Irepresents a unit matrix, <' > based on>
Figure QLYQS_33
Represents an observation matrix, <' > is selected>
Figure QLYQS_39
Represents->
Figure QLYQS_35
About>
Figure QLYQS_43
Is based on the partial derivative of (4)>
Figure QLYQS_37
In the form of a logarithmic function of the function,
Figure QLYQS_44
represents->
Figure QLYQS_49
Status of the moment>
Figure QLYQS_53
Is true and>
Figure QLYQS_50
and the generating value->
Figure QLYQS_54
Based on the error status of the status flag, based on the status flag>
Figure QLYQS_36
Indicates a generation value->
Figure QLYQS_41
And evaluation value->
Figure QLYQS_47
The error between;
Figure QLYQS_55
and representing a Kalman filtering gain matrix, wherein the calculation process is as follows:
Figure QLYQS_56
Figure QLYQS_57
Figure QLYQS_58
in the formula (I), the compound is shown in the specification,
Figure QLYQS_71
represents a covariance matrix, based on the covariance matrix>
Figure QLYQS_61
Represents a fifth or fifth party>
Figure QLYQS_69
IMU state covariance at sub-sampling time, <' >>
Figure QLYQS_64
Indicates the fifth->
Figure QLYQS_68
IMU state covariance at sub-sampling time, <' >>
Figure QLYQS_73
Represents the covariance matrix pick>
Figure QLYQS_74
And partial derivative matrix->
Figure QLYQS_66
The intermediate variable which is generated, is taken>
Figure QLYQS_72
And &>
Figure QLYQS_59
Respectively represent->
Figure QLYQS_65
Is paired and/or matched>
Figure QLYQS_62
And &>
Figure QLYQS_67
Based on the partial derivative matrix, is greater than or equal to>
Figure QLYQS_63
Indicates IMU noise->
Figure QLYQS_70
Covariance of (4), upper corner flag->
Figure QLYQS_60
Represents a transpose of a matrix;
in the above iterative update of the state estimate, the cloud residual is present at the point
Figure QLYQS_75
After convergence, an optimal state estimate, i.e. a second state quantity, can be obtained as follows:
Figure QLYQS_76
in the formula (I), the compound is shown in the specification,
Figure QLYQS_77
presentation system at the fifth place>
Figure QLYQS_78
A second state quantity at the sub-sampling instant. />
6. The method for building a three-dimensional semantic map of a solid-state lidar for autonomous learning according to claim 5, wherein in step 2.3, the updating of the reverse state of the second state quantity specifically comprises:
Figure QLYQS_79
in the formula (I), the compound is shown in the specification,
Figure QLYQS_80
indicating that the system is in the ^ th->
Figure QLYQS_81
A third state quantity at the subsampling instant->
Figure QLYQS_82
Indicating that the system is in the ^ th->
Figure QLYQS_83
A third state quantity at the sub-sampling instant.
7. The method for building a three-dimensional semantic map of a solid-state lidar according to claim 1, 2 or 3, wherein in step 2.4, converting the point cloud coordinates of the own coordinate system of each frame in one scan of the lidar into the coordinates of the global coordinate system is specifically:
Figure QLYQS_84
in the formula (I), the compound is shown in the specification,
Figure QLYQS_97
represents the coordinates of the laser point cloud, and is used for judging whether the laser point is in the dark or not>
Figure QLYQS_89
Represents->
Figure QLYQS_93
Sub-scanning lidar frame>
Figure QLYQS_88
Represents a global coordinate system, is>
Figure QLYQS_92
Indicates the fifth->
Figure QLYQS_98
A characteristic point->
Figure QLYQS_102
Indicates the fifth->
Figure QLYQS_87
Individual characteristic point is in>
Figure QLYQS_91
Sub-scanning the coordinates of the lidar frame>
Figure QLYQS_85
Represents a fifth or fifth party>
Figure QLYQS_94
Global coordinates of the feature points after coordinate conversion; />
Figure QLYQS_86
A coordinate transformation matrix representing the coordinate system from the lower right corner mark to the upper left corner mark,
Figure QLYQS_96
represents an inertial frame, is present>
Figure QLYQS_99
Represents the lidar coordinate system>
Figure QLYQS_103
Represents a transformation matrix from the lidar coordinate system to the inertial coordinate system, and>
Figure QLYQS_95
represents->
Figure QLYQS_101
Inertial frame of time->
Figure QLYQS_100
To a global coordinate system>
Figure QLYQS_104
Updating the transformation matrix; />
Figure QLYQS_90
Is the number of lidar point clouds.
8. An apparatus for building a three-dimensional semantic map of a solid-state lidar for self-supervised learning, wherein the method of any one of claims 1 to 7 is adopted, and the apparatus comprises:
the system comprises an RGB camera, a laser radar and an inertia measuring device, wherein the RGB camera, the laser radar and the inertia measuring device are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring RGB images, three-dimensional point cloud data and inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertial data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
and the three-dimensional semantic map building module is connected with the point cloud semantic division model and the three-dimensional point cloud map building module and is used for generating a three-dimensional semantic map by corresponding the point cloud division result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate.
CN202211387608.5A 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device Active CN115421158B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211387608.5A CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211387608.5A CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Publications (2)

Publication Number Publication Date
CN115421158A CN115421158A (en) 2022-12-02
CN115421158B true CN115421158B (en) 2023-04-07

Family

ID=84207166

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211387608.5A Active CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Country Status (1)

Country Link
CN (1) CN115421158B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116229057B (en) * 2022-12-22 2023-10-27 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning
CN115638788B (en) * 2022-12-23 2023-03-21 安徽蔚来智驾科技有限公司 Semantic vector map construction method, computer equipment and storage medium
CN116778162A (en) * 2023-06-25 2023-09-19 南京航空航天大学 Weak supervision large aircraft appearance point cloud semantic segmentation method based on geometric feature guidance
CN117517864B (en) * 2023-11-08 2024-04-26 南京航空航天大学 Laser radar-based power transmission line near electricity early warning method and device

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106688017B (en) * 2016-11-28 2019-03-01 深圳市大疆创新科技有限公司 Generate method, computer system and the device of point cloud map
CN110097553B (en) * 2019-04-10 2023-05-02 东南大学 Semantic mapping system based on instant positioning mapping and three-dimensional semantic segmentation
CN111274976B (en) * 2020-01-22 2020-09-18 清华大学 Lane detection method and system based on multi-level fusion of vision and laser radar
CN112634451B (en) * 2021-01-11 2022-08-23 福州大学 Outdoor large-scene three-dimensional mapping method integrating multiple sensors
CN113128591B (en) * 2021-04-14 2023-12-05 中山大学 Rotary robust point cloud classification method based on self-supervision learning
CN113763423A (en) * 2021-08-03 2021-12-07 中国北方车辆研究所 Multi-mode data based systematic target recognition and tracking method
CN114926469A (en) * 2022-04-26 2022-08-19 中南大学 Semantic segmentation model training method, semantic segmentation method, storage medium and terminal
CN114898322A (en) * 2022-06-13 2022-08-12 中国第一汽车股份有限公司 Driving environment identification method and device, vehicle and storage medium
CN115222919A (en) * 2022-07-27 2022-10-21 徐州徐工矿业机械有限公司 Sensing system and method for constructing color point cloud map of mobile machine

Also Published As

Publication number Publication date
CN115421158A (en) 2022-12-02

Similar Documents

Publication Publication Date Title
CN115421158B (en) Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device
CN112014857B (en) Three-dimensional laser radar positioning and navigation method for intelligent inspection and inspection robot
Velas et al. CNN for IMU assisted odometry estimation using velodyne LiDAR
JP7236565B2 (en) POSITION AND ATTITUDE DETERMINATION METHOD, APPARATUS, ELECTRONIC DEVICE, STORAGE MEDIUM AND COMPUTER PROGRAM
CN110675418B (en) Target track optimization method based on DS evidence theory
CN113393522B (en) 6D pose estimation method based on monocular RGB camera regression depth information
CN109947097B (en) Robot positioning method based on vision and laser fusion and navigation application
JP6456141B2 (en) Generating map data
US7747106B2 (en) Method and system for filtering, registering, and matching 2.5D normal maps
CN104881029B (en) Mobile Robotics Navigation method based on a point RANSAC and FAST algorithms
CN115655262A (en) Deep learning perception-based multi-level semantic map construction method and device
CN115900710A (en) Dynamic environment navigation method based on visual information
CN116222577B (en) Closed loop detection method, training method, system, electronic equipment and storage medium
CN112183247A (en) Laser point cloud data classification method based on multispectral image
CN117036300A (en) Road surface crack identification method based on point cloud-RGB heterogeneous image multistage registration mapping
CN117808689A (en) Depth complement method based on fusion of millimeter wave radar and camera
CN113804182B (en) Grid map creation method based on information fusion
CN117570968A (en) Map construction and maintenance method and device based on visual road sign and storage medium
CN114384486A (en) Data processing method and device
CN116704032A (en) Outdoor visual SLAM method based on monocular depth estimation network and GPS
US11741631B2 (en) Real-time alignment of multiple point clouds to video capture
CN113554705A (en) Robust positioning method for laser radar in changing scene
CN115908564A (en) Storage line inspection method of automatic transportation equipment and automatic transportation equipment
CN116895014A (en) Semantic map construction method and device, electronic equipment and storage medium
CN117765025A (en) Motion trail prediction method, medium and system for long-distance bird target

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant