CN112712589A - Plant 3D modeling method and system based on laser radar and deep learning - Google Patents
Plant 3D modeling method and system based on laser radar and deep learning Download PDFInfo
- Publication number
- CN112712589A CN112712589A CN202110023105.9A CN202110023105A CN112712589A CN 112712589 A CN112712589 A CN 112712589A CN 202110023105 A CN202110023105 A CN 202110023105A CN 112712589 A CN112712589 A CN 112712589A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- point
- cloud data
- matrix
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000013135 deep learning Methods 0.000 title claims abstract description 19
- 230000000007 visual effect Effects 0.000 claims abstract description 86
- 238000001914 filtration Methods 0.000 claims abstract description 23
- 238000009499 grossing Methods 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 20
- 238000000926 separation method Methods 0.000 claims abstract description 12
- 230000003068 static effect Effects 0.000 claims abstract description 5
- 239000011159 matrix material Substances 0.000 claims description 92
- 230000009466 transformation Effects 0.000 claims description 38
- 238000000605 extraction Methods 0.000 claims description 17
- 238000006243 chemical reaction Methods 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 10
- 238000013507 mapping Methods 0.000 claims description 6
- 238000012952 Resampling Methods 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 2
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Computer Graphics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
A plant 3D modeling method based on laser radar and deep learning comprises the following steps: 1) performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles; 2) carrying out preprocessing of filtering, denoising and downsampling on each laser radar original point cloud data obtained in the step 1; 3) extracting semantic feature points from the laser radar point cloud data processed in the step 2, and identifying each target in the environment; 4) selecting the targets identified in the step 3, and separating point cloud information of the plants; 5) splicing the plurality of point cloud data processed in the step 4; 6) carrying out grid smoothing on the point cloud data processed in the step 5; 7) and (4) performing three-dimensional modeling on the point cloud data processed in the step (6). The invention also provides a plant 3D modeling system based on the laser radar and the deep learning. The plant of the invention has good authenticity, high geometric precision and good separation effect.
Description
Technical Field
The invention relates to the field of three-dimensional modeling of plants, which is mainly applied to the field of high-precision modeling of standing trees, in particular to a method and a system for denoising, area segmentation, feature extraction, registration and splicing of laser radar point cloud data and forming a high-precision 3D plant model based on a deep learning method.
Background
At present, the main method for performing 3D modeling on plants in the industry is to perform plant reconstruction in a mode based on image sequences. The reconstruction mode based on the image sequence has low precision, few applicable scenes and short scanning distance. And image sequence based methods require the gradual addition of new images starting from the matching and reconstruction of two images until the entire sequence is reconstructed. Since this method cannot simultaneously utilize all image information, the accumulated error is inevitable.
The laser radar is well known for the characteristics of long scanning distance, high spatial data acquisition speed, high precision, large data volume, wide applicable scene and the like, and has become a preferred sensor in many mapping fields. With the further acceleration of the forestry informatization process, active remote sensing technologies represented by the laser radar technology are gradually emerging in the field of vegetation modeling. And with the development of computer hardware and deep learning technology in recent years, deep learning has been applied to processing laser radar point cloud data, so that a deep learning-based method for processing plant 3D point cloud data acquired by laser radar becomes possible.
The lidar 3D point cloud data is closer to the original representation of the scene without further discretization and an object is represented by only one matrix of N x D. Qi et al propose a deep learning network PointNet for segmenting, classifying and identifying point cloud data, and the network can directly perform feature extraction, semantic segmentation, classification and identification on the original point cloud data. The proposal of the network provides a theoretical basis for 3D object modeling.
Disclosure of Invention
The invention aims to provide a plant 3D modeling method and system based on laser radar and deep learning, aiming at the defects of insufficient plant model reality sense, low geometric precision of a model, unobvious detail texture characteristics, unsatisfactory noise processing, environmental limitation (such as light) of modeling and the like of the conventional plant modeling method.
The technical scheme adopted by the invention for realizing the purposes of the invention is as follows:
a plant 3D modeling method based on laser radar and deep learning comprises the following steps:
s1: performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles;
s2: preprocessing filtering, denoising and downsampling each laser radar original point cloud data obtained in the step S1;
s3: extracting semantic feature points from the laser radar point cloud data processed in the step S2, and identifying each target in the environment;
s4: selecting the targets identified in the step S3, and separating point cloud information of plants;
s5: splicing the plurality of point cloud data processed in step S4;
s6: performing mesh smoothing on the point cloud data processed in the step S5;
s7: and (4) performing three-dimensional display on the point cloud processed in the step (S6).
Further, the step S2 specifically includes:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
Further, the step S3 specifically includes:
s3.1: extracting semantic feature points from the point cloud data processed in the step S2:
inputting N x 3 point cloud data processed by S2 into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of a model to specific space conversion is ensured, then, the cloud data of each point are subjected to feature extraction by a multi-layer perceptron, then, one T-Net is used for aligning the features, then, the maximum pooling operation is performed on each dimension of the features to obtain the final global features, the global features and the cloud local features of each point obtained before are connected in series, and then, the classification result of each data point is obtained by the multi-layer perceptron;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
and (4) performing global feature extraction on the classified point cloud processed in the S3.1, and predicting a final classification score through a multilayer perceptron. The recognition result of each target is given by the classification score.
Further, the step S5 specifically includes:
s5.1: selecting one view as a main view:
selecting a visual angle from the single plant point clouds at the multiple visual angles acquired in the step S4 as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view angle P is a matrix S of D x 3pAs follows:
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P={p1,p2,...,pn} (5)
pifor a single set of points, the mean is first found and the covariance matrix c of the set of points is calculatedov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are respectively as follows:
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
S5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
and (3) carrying out space transformation on the point cloud sets P and Q after the rough registration in the S5.3 to enable the distance between the point clouds P and Q to be the minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative calculation, and if not, recalculating until a convergence condition is met.
The invention also provides a plant 3D modeling system based on the laser radar and the deep learning, which comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a plant point cloud information separation module, a splicing module, a grid smoothing module and a display module which are sequentially connected.
The technical conception of the invention is as follows: based on laser radar and deep learningThe plant 3D modeling method comprises the steps of obtaining point cloud data of the whole environment from laser radars with multiple visual angles, carrying out grid filtering on the original point cloud data to remove noise points of smaller clusters, and carrying out Hash downsampling on the original point cloud to remove redundant points, so that useless and repeated points in the original data are as few as possible. And respectively inputting the processed point cloud data into a PointNet neural network, aligning the input data by multiplying the input data by a conversion matrix learned by T-Net, then carrying out global feature extraction on the input data, and predicting the final classification score by a multilayer perceptron. And giving out the identification result of each target through the classification score, identifying all the targets from the identification result, and segmenting specific plant point cloud data according to the requirements. Selecting one visual angle from a plurality of visual angles as a main visual angle, and according to point cloud data of the main visual angle and the included angle theta between the X-axis positive half axis and the main visual angle X-axis positive half axis of other visual angleskAnd carrying out coordinate transformation on the single plant point cloud data of other visual angles, wherein all coordinates after the coordinate transformation are based on the main visual angle. And then, splicing and smoothly filtering the point cloud data of all the visual angles.
The invention has the beneficial effects that: 1) the point cloud information of the plant to be modeled is acquired from multiple visual angles, and the reality of the modeled plant is better and the geometric accuracy is higher due to the multiple visual angles. 2) The original point cloud data is subjected to operations such as filtering, denoising and downsampling, noise and redundant information points are removed, interference can be reduced, and the calculation amount of a neural network in the next step can be reduced. 3) And separating the plant point clouds to be modeled from the point clouds by using a PointNet neural network, wherein the separation effect and efficiency of the method are better and the generalization to different plants is better compared with that of a machine learning method or a manual separation method such as DBSCAN (direct species learning) and the like by using the PointNet neural network to separate targets. 4) Using a base of thetakThe method has small calculated amount, is suitable for splicing multi-view point cloud data, and can correctly integrate the data of each view into the main view.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a cloud point at a certain viewing angle;
FIG. 3 is a point cloud image of FIG. 2 after view denoising and downsampling;
FIG. 4 is a cloud point map of a plant isolated from the point cloud data;
FIG. 5 is a cloud point of a plant isolated from another perspective;
fig. 6 is a cloud point diagram after the completion of the two-view stitching.
Detailed Description
The following detailed description of embodiments of the invention is provided in connection with the accompanying drawings.
Referring to fig. 1 to 6, a plant 3D modeling method based on laser radar and deep learning includes the following steps:
1) acquiring multi-view point cloud information of a target by using a laser radar:
1.1) obtaining target point cloud information:
according to the actual situation of the standing tree to be measured, a plurality of visual angles are selected to carry out static scanning on the target, and the scanning time of the static scanning is set to be 1S which can cover 99% of a visual field. Fig. 2 is a point cloud information diagram of the whole target environment acquired from a certain view angle.
1.2) acquiring the position relation among radars at various visual angles:
marking included angle theta formed by positive half shafts of X axes of all adjacent radar view angle coordinate systemsk。
2) Preprocessing the acquired point cloud information:
2.1) carrying out filtering and denoising on the original point cloud data of each visual angle:
the method comprises the steps of firstly, carrying out filtering and denoising on point cloud data of each visual angle, filtering out non-target clusters formed by few points according to a preset threshold value, and then smoothing and resampling the point cloud data by using a moving least square method.
2.2): carrying out down-sampling on the point cloud data after denoising:
dividing the denoised point cloud data into a plurality of equal lattices in space, calculating the number of small lattices divided in each direction, calculating the specific lattice in which each point falls, and mapping each point to different containers by using a hash table. Only one point is kept for the point cloud in each grid. The corresponding hash function is:
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z. The point cloud data map after the preprocessing is shown in fig. 3.
3) Carrying out feature extraction and target detection on point cloud data:
3.1): extracting semantic feature points from the preprocessed point cloud data:
and inputting the preprocessed N x 3 point cloud data into a PointNet network as all data of one frame, and aligning the input data by multiplying the input data by a conversion matrix learned by T-Net, so that the invariance of the model to specific space conversion is ensured. And then, extracting the features of the cloud data of each point through a multi-layer perceptron, and aligning the features by using a T-Net. And then performing maximum pooling operation on all dimensions of the features to obtain final global features. And connecting the global features with the previously obtained local features of the clouds of each point in series, and obtaining the classification result of each data point through a multi-layer perceptron.
3.2): and (3) performing target extraction and identification on point cloud data by using PointNet:
and performing global feature extraction on the point cloud after semantic feature extraction, and predicting the final classification score through a multilayer perceptron. The recognition result of each target is given by the classification score.
4): separating the point cloud information of the plants:
4.1): according to the identified target, selecting a plant to be modeled, and separating point cloud data of the plant:
and (4) selecting results from each target point cloud frame according to the identification result in the step (3) and the plant to be modeled. And separating all point cloud data in the frame according to the 3D coordinates of the selected frame and storing the point cloud data as a point cloud file. Fig. 4 and 5 are point cloud information graphs of the same plant separated from two different visual angles.
5): point cloud data coordinate transformation and splicing:
5.1): selecting one view as a main view:
and 4, selecting one visual angle from the single plant point clouds at the multiple visual angles acquired in the step 4 as a main visual angle P.
5.2): and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view angle P is a matrix S of D x 3pAs follows:
wherein Xi、Yi、ZiRespectively representing the values of the X-axis, the Y-axis and the Z-axis of the corresponding point cloud coordinates. The matrix corresponding to the point cloud set of other visual angles is SkAs follows:
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskAnd calculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle. Corresponding relative matrix SkAs follows:
5.3): carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting PCA algorithm, grouping the point cloud data processed by 5.2 in pairs, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein the point cloud set of one point cloud is assumed as follows:
P={p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
5.4): and (3) carrying out fine registration on the point cloud image after the coarse registration:
and performing spatial transformation on the point cloud sets P and Q after coarse registration to minimize the distance between the point clouds P and Q, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, stopping iterative computation if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold, and otherwise, recalculating until a convergence condition is met. The point cloud after the fine registration is completed is shown in fig. 6.
6): and (3) smoothing the complete point cloud data:
6.1): and 5, carrying out grid smoothing on the basis of the point cloud data in the step 5:
and 5, moving the concave-convex points to the average position of the adjacent point weighting strategy by adopting a Laplace smoothing algorithm on the complete point cloud data finally generated in the step 5, wherein the influence weight is the reciprocal of the distance.
A plant 3D modeling system based on laser radar and deep learning comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a plant point cloud information separation module, a splicing module, a grid smoothing module and a display module which are sequentially connected;
the point cloud image acquisition module is connected with the laser radar, and the laser radar is used for statically scanning plants at multiple visual angles to acquire original point cloud data at different visual angles;
the preprocessing module is used for preprocessing the filtering, denoising and downsampling of each laser radar original point cloud data obtained in the step S1, and specifically comprises the following steps:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
The target identification module is used for extracting semantic feature points from the laser radar point cloud data input by the preprocessing module and identifying each target in the environment, and specifically comprises the following steps:
s3.1: extracting semantic feature points from the point cloud data processed by the preprocessing module:
inputting N x 3 point cloud data processed by a preprocessing module into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of the model to specific space conversion is ensured, then extracting the characteristics of each point cloud data by a plurality of multilayer perceptrons, aligning the characteristics by using one T-Net, then performing maximum pooling operation on each dimension of the characteristics to obtain final global characteristics, connecting the global characteristics with the cloud local characteristics of each point obtained previously in series, and obtaining the classification result of each data point by using the multilayer perceptrons;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
performing global feature extraction on the classified point cloud processed in the S3.1, predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score;
the point cloud information separation module of the plant selects the targets identified by the preprocessing module and separates the point cloud information of the plant;
the splicing module splices a plurality of point cloud data processed by the point cloud information separation module of the plant, and specifically comprises:
s5.1: selecting one view as a main view:
selecting a visual angle from single plant point clouds at a plurality of visual angles acquired by a point cloud information separation module of a plant as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P={p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
x, Y, Z and X ', Y ' and Z ' are respectively 3 column vectors of a target point set matrix and a source point set matrix, 3 eigenvectors of a covariance matrix cov are used as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is used as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration;
s5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
performing space transformation on the point cloud sets P and Q after S5.3 coarse registration to enable the distance between the point clouds P and Q to be minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative computation, and if not, recalculating until a convergence condition is met;
the mesh smoothing module is used for carrying out mesh smoothing on the point cloud data processed by the splicing module;
and the display module is used for carrying out three-dimensional display on the point cloud processed by the grid smoothing module.
The embodiments described in this specification are merely illustrative of implementations of the inventive concept and the scope of the present invention should not be considered limited to the specific forms set forth in the embodiments but rather by the equivalents thereof as may occur to those skilled in the art upon consideration of the present inventive concept.
Claims (5)
1. A plant 3D modeling method and system based on laser radar and deep learning are characterized by comprising the following steps:
s1: performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles;
s2: preprocessing filtering, denoising and downsampling each laser radar original point cloud data obtained in the step S1;
s3: extracting semantic feature points from the laser radar point cloud data processed in the step S2, and identifying each target in the environment;
s4: selecting the targets identified in the step S3, and separating point cloud information of plants;
s5: splicing the plurality of point cloud data processed in step S4;
s6: performing mesh smoothing on the point cloud data processed in the step S5;
s7: and (4) performing three-dimensional display on the point cloud processed in the step (S6).
2. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S2 specifically includes:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
E.g.,hash(hx+hy*Dx+hz*Dx*Dy)%container_size (1)
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
3. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S3 specifically includes:
s3.1: extracting semantic feature points from the point cloud data processed in the step S2:
inputting N x 3 point cloud data processed by S2 into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of a model to specific space conversion is ensured, then, the cloud data of each point are subjected to feature extraction by a multi-layer perceptron, then, one T-Net is used for aligning the features, then, the maximum pooling operation is performed on each dimension of the features to obtain the final global features, the global features and the cloud local features of each point obtained before are connected in series, and then, the classification result of each data point is obtained by the multi-layer perceptron;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
and (3) performing global feature extraction on the classified point cloud processed in the step (S3.1), predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score.
4. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S5 specifically includes:
s5.1: selecting one view as a main view:
selecting a visual angle from the single plant point clouds at the multiple visual angles acquired in the step S4 as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P={p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
S5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
and (3) carrying out space transformation on the point cloud sets P and Q after the rough registration in the S5.3 to enable the distance between the point clouds P and Q to be the minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative calculation, and if not, recalculating until a convergence condition is met.
5. The utility model provides a plant 3D modeling's system based on laser radar and deep learning which characterized in that: the system comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a point cloud information separation module of plants, a splicing module, a grid smoothing module and a display module which are connected in sequence;
the point cloud image acquisition module is connected with the laser radar, and the laser radar is used for statically scanning plants at multiple visual angles to acquire original point cloud data at different visual angles;
the preprocessing module is used for preprocessing the filtering, denoising and downsampling of each laser radar original point cloud data obtained in the step S1, and specifically comprises the following steps:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
E.g.,hash(hx+hy*Dx+hz*Dx*Dy)%container_size (1)
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
The target identification module is used for extracting semantic feature points from the laser radar point cloud data input by the preprocessing module and identifying each target in the environment, and specifically comprises the following steps:
s3.1: extracting semantic feature points from the point cloud data processed by the preprocessing module:
inputting N x 3 point cloud data processed by a preprocessing module into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of the model to specific space conversion is ensured, then extracting the characteristics of each point cloud data by a plurality of multilayer perceptrons, aligning the characteristics by using one T-Net, then performing maximum pooling operation on each dimension of the characteristics to obtain final global characteristics, connecting the global characteristics with the cloud local characteristics of each point obtained previously in series, and obtaining the classification result of each data point by using the multilayer perceptrons;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
performing global feature extraction on the classified point cloud processed in the S3.1, predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score;
the point cloud information separation module of the plant selects the targets identified by the preprocessing module and separates the point cloud information of the plant;
the splicing module splices a plurality of point cloud data processed by the point cloud information separation module of the plant, and specifically comprises:
s5.1: selecting one view as a main view:
selecting a visual angle from single plant point clouds at a plurality of visual angles acquired by a point cloud information separation module of a plant as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs shown below:
S5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P={p1,p2,...,pn} (5)
pi is a single point set, the mean value is firstly calculated and the covariance matrix cov of the point set is calculated, and the target point set point cloud covariance matrix and the source point cloud covariance matrix are respectively as follows:
x, Y, Z and X ', Y ' and Z ' are respectively 3 column vectors of a target point set matrix and a source point set matrix, 3 eigenvectors of a covariance matrix cov are used as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is used as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration;
s5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
performing space transformation on the point cloud sets P and Q after S5.3 coarse registration to enable the distance between the point clouds P and Q to be minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative computation, and if not, recalculating until a convergence condition is met;
the mesh smoothing module is used for carrying out mesh smoothing on the point cloud data processed by the splicing module;
and the display module is used for carrying out three-dimensional display on the point cloud processed by the grid smoothing module.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110023105.9A CN112712589A (en) | 2021-01-08 | 2021-01-08 | Plant 3D modeling method and system based on laser radar and deep learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110023105.9A CN112712589A (en) | 2021-01-08 | 2021-01-08 | Plant 3D modeling method and system based on laser radar and deep learning |
Publications (1)
Publication Number | Publication Date |
---|---|
CN112712589A true CN112712589A (en) | 2021-04-27 |
Family
ID=75548572
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110023105.9A Withdrawn CN112712589A (en) | 2021-01-08 | 2021-01-08 | Plant 3D modeling method and system based on laser radar and deep learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112712589A (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113256640A (en) * | 2021-05-31 | 2021-08-13 | 北京理工大学 | Method and device for partitioning network point cloud and generating virtual environment based on PointNet |
CN113409301A (en) * | 2021-07-12 | 2021-09-17 | 上海精劢医疗科技有限公司 | Point cloud segmentation-based femoral neck registration method, system and medium |
CN113421290A (en) * | 2021-07-05 | 2021-09-21 | 山西大学 | Power plant boiler interior three-dimensional reconstruction method based on unmanned aerial vehicle image acquisition |
CN113434936A (en) * | 2021-06-28 | 2021-09-24 | 北京工业大学 | Road geometric element estimation method and device |
CN113447953A (en) * | 2021-06-29 | 2021-09-28 | 山东高速建设管理集团有限公司 | Background filtering method based on road traffic point cloud data |
CN113640772A (en) * | 2021-09-16 | 2021-11-12 | 招商局检测车辆技术研究院有限公司 | Method and system for realizing target perception in vehicle-road cooperation |
-
2021
- 2021-01-08 CN CN202110023105.9A patent/CN112712589A/en not_active Withdrawn
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113256640A (en) * | 2021-05-31 | 2021-08-13 | 北京理工大学 | Method and device for partitioning network point cloud and generating virtual environment based on PointNet |
CN113256640B (en) * | 2021-05-31 | 2022-05-24 | 北京理工大学 | Method and device for partitioning network point cloud and generating virtual environment based on PointNet |
CN113434936A (en) * | 2021-06-28 | 2021-09-24 | 北京工业大学 | Road geometric element estimation method and device |
CN113447953A (en) * | 2021-06-29 | 2021-09-28 | 山东高速建设管理集团有限公司 | Background filtering method based on road traffic point cloud data |
CN113421290A (en) * | 2021-07-05 | 2021-09-21 | 山西大学 | Power plant boiler interior three-dimensional reconstruction method based on unmanned aerial vehicle image acquisition |
CN113409301A (en) * | 2021-07-12 | 2021-09-17 | 上海精劢医疗科技有限公司 | Point cloud segmentation-based femoral neck registration method, system and medium |
CN113640772A (en) * | 2021-09-16 | 2021-11-12 | 招商局检测车辆技术研究院有限公司 | Method and system for realizing target perception in vehicle-road cooperation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113128405B (en) | Plant identification and model construction method combining semantic segmentation and point cloud processing | |
CN112712589A (en) | Plant 3D modeling method and system based on laser radar and deep learning | |
CN102236794B (en) | Recognition and pose determination of 3D objects in 3D scenes | |
EP2720171B1 (en) | Recognition and pose determination of 3D objects in multimodal scenes | |
Guerry et al. | Snapnet-r: Consistent 3d multi-view semantic labeling for robotics | |
CN110648361A (en) | Real-time pose estimation method and positioning and grabbing system of three-dimensional target object | |
CN111723721A (en) | Three-dimensional target detection method, system and device based on RGB-D | |
CN113128610B (en) | Industrial part pose estimation method and system | |
CN110942515A (en) | Point cloud-based target object three-dimensional computer modeling method and target identification method | |
Rubio et al. | Efficient monocular pose estimation for complex 3D models | |
CN110490915B (en) | Point cloud registration method based on convolution-limited Boltzmann machine | |
Ansari | A review on SIFT and SURF for underwater image feature detection and matching | |
CN114219855A (en) | Point cloud normal vector estimation method and device, computer equipment and storage medium | |
CN114022542A (en) | Three-dimensional reconstruction-based 3D database manufacturing method | |
CN111127556B (en) | Target object identification and pose estimation method and device based on 3D vision | |
CN113592015B (en) | Method and device for positioning and training feature matching network | |
CN117765039A (en) | Point cloud coarse registration method, device and equipment | |
CN110634160B (en) | Method for constructing target three-dimensional key point extraction model and recognizing posture in two-dimensional graph | |
CN112001954A (en) | Polar curve constraint-based underwater PCA-SIFT image matching method | |
CN113409392A (en) | 6DoF pose estimation method of reflective workpiece | |
Ren et al. | Robust object pose estimation based on improved point pair features method | |
Joshi et al. | A training free technique for 3D object recognition using the concept of vibration, energy and frequency | |
CN113543307B (en) | Visual information feature fingerprint database construction method, positioning method and device | |
Wang et al. | Enhanced histogram feature descriptor for automated point cloud registration | |
Liu et al. | 6D Pose Estimation Method Using Curvature-Enhanced Point-Pair Features |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210427 |
|
WW01 | Invention patent application withdrawn after publication |