CN112712589A - Plant 3D modeling method and system based on laser radar and deep learning - Google Patents

Plant 3D modeling method and system based on laser radar and deep learning Download PDF

Info

Publication number
CN112712589A
CN112712589A CN202110023105.9A CN202110023105A CN112712589A CN 112712589 A CN112712589 A CN 112712589A CN 202110023105 A CN202110023105 A CN 202110023105A CN 112712589 A CN112712589 A CN 112712589A
Authority
CN
China
Prior art keywords
point cloud
point
cloud data
matrix
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202110023105.9A
Other languages
Chinese (zh)
Inventor
宣琦
郑俊杰
朱城超
朱振强
刘壮壮
翔云
邱君瀚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University of Technology ZJUT
Original Assignee
Zhejiang University of Technology ZJUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University of Technology ZJUT filed Critical Zhejiang University of Technology ZJUT
Priority to CN202110023105.9A priority Critical patent/CN112712589A/en
Publication of CN112712589A publication Critical patent/CN112712589A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10044—Radar image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30181—Earth observation
    • G06T2207/30188—Vegetation; Agriculture

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

A plant 3D modeling method based on laser radar and deep learning comprises the following steps: 1) performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles; 2) carrying out preprocessing of filtering, denoising and downsampling on each laser radar original point cloud data obtained in the step 1; 3) extracting semantic feature points from the laser radar point cloud data processed in the step 2, and identifying each target in the environment; 4) selecting the targets identified in the step 3, and separating point cloud information of the plants; 5) splicing the plurality of point cloud data processed in the step 4; 6) carrying out grid smoothing on the point cloud data processed in the step 5; 7) and (4) performing three-dimensional modeling on the point cloud data processed in the step (6). The invention also provides a plant 3D modeling system based on the laser radar and the deep learning. The plant of the invention has good authenticity, high geometric precision and good separation effect.

Description

Plant 3D modeling method and system based on laser radar and deep learning
Technical Field
The invention relates to the field of three-dimensional modeling of plants, which is mainly applied to the field of high-precision modeling of standing trees, in particular to a method and a system for denoising, area segmentation, feature extraction, registration and splicing of laser radar point cloud data and forming a high-precision 3D plant model based on a deep learning method.
Background
At present, the main method for performing 3D modeling on plants in the industry is to perform plant reconstruction in a mode based on image sequences. The reconstruction mode based on the image sequence has low precision, few applicable scenes and short scanning distance. And image sequence based methods require the gradual addition of new images starting from the matching and reconstruction of two images until the entire sequence is reconstructed. Since this method cannot simultaneously utilize all image information, the accumulated error is inevitable.
The laser radar is well known for the characteristics of long scanning distance, high spatial data acquisition speed, high precision, large data volume, wide applicable scene and the like, and has become a preferred sensor in many mapping fields. With the further acceleration of the forestry informatization process, active remote sensing technologies represented by the laser radar technology are gradually emerging in the field of vegetation modeling. And with the development of computer hardware and deep learning technology in recent years, deep learning has been applied to processing laser radar point cloud data, so that a deep learning-based method for processing plant 3D point cloud data acquired by laser radar becomes possible.
The lidar 3D point cloud data is closer to the original representation of the scene without further discretization and an object is represented by only one matrix of N x D. Qi et al propose a deep learning network PointNet for segmenting, classifying and identifying point cloud data, and the network can directly perform feature extraction, semantic segmentation, classification and identification on the original point cloud data. The proposal of the network provides a theoretical basis for 3D object modeling.
Disclosure of Invention
The invention aims to provide a plant 3D modeling method and system based on laser radar and deep learning, aiming at the defects of insufficient plant model reality sense, low geometric precision of a model, unobvious detail texture characteristics, unsatisfactory noise processing, environmental limitation (such as light) of modeling and the like of the conventional plant modeling method.
The technical scheme adopted by the invention for realizing the purposes of the invention is as follows:
a plant 3D modeling method based on laser radar and deep learning comprises the following steps:
s1: performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles;
s2: preprocessing filtering, denoising and downsampling each laser radar original point cloud data obtained in the step S1;
s3: extracting semantic feature points from the laser radar point cloud data processed in the step S2, and identifying each target in the environment;
s4: selecting the targets identified in the step S3, and separating point cloud information of plants;
s5: splicing the plurality of point cloud data processed in step S4;
s6: performing mesh smoothing on the point cloud data processed in the step S5;
s7: and (4) performing three-dimensional display on the point cloud processed in the step (S6).
Further, the step S2 specifically includes:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
Figure BDA0002889394520000031
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
Further, the step S3 specifically includes:
s3.1: extracting semantic feature points from the point cloud data processed in the step S2:
inputting N x 3 point cloud data processed by S2 into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of a model to specific space conversion is ensured, then, the cloud data of each point are subjected to feature extraction by a multi-layer perceptron, then, one T-Net is used for aligning the features, then, the maximum pooling operation is performed on each dimension of the features to obtain the final global features, the global features and the cloud local features of each point obtained before are connected in series, and then, the classification result of each data point is obtained by the multi-layer perceptron;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
and (4) performing global feature extraction on the classified point cloud processed in the S3.1, and predicting a final classification score through a multilayer perceptron. The recognition result of each target is given by the classification score.
Further, the step S5 specifically includes:
s5.1: selecting one view as a main view:
selecting a visual angle from the single plant point clouds at the multiple visual angles acquired in the step S4 as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view angle P is a matrix S of D x 3pAs follows:
Figure BDA0002889394520000041
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
Figure BDA0002889394520000042
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
Figure BDA0002889394520000043
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P＝{p1,p2,...,pn} (5)
pifor a single set of points, the mean is first found and the covariance matrix c of the set of points is calculatedov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are respectively as follows:
Figure BDA0002889394520000044
Figure BDA0002889394520000051
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
S5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
and (3) carrying out space transformation on the point cloud sets P and Q after the rough registration in the S5.3 to enable the distance between the point clouds P and Q to be the minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative calculation, and if not, recalculating until a convergence condition is met.
The invention also provides a plant 3D modeling system based on the laser radar and the deep learning, which comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a plant point cloud information separation module, a splicing module, a grid smoothing module and a display module which are sequentially connected.
The technical conception of the invention is as follows: based on laser radar and deep learningThe plant 3D modeling method comprises the steps of obtaining point cloud data of the whole environment from laser radars with multiple visual angles, carrying out grid filtering on the original point cloud data to remove noise points of smaller clusters, and carrying out Hash downsampling on the original point cloud to remove redundant points, so that useless and repeated points in the original data are as few as possible. And respectively inputting the processed point cloud data into a PointNet neural network, aligning the input data by multiplying the input data by a conversion matrix learned by T-Net, then carrying out global feature extraction on the input data, and predicting the final classification score by a multilayer perceptron. And giving out the identification result of each target through the classification score, identifying all the targets from the identification result, and segmenting specific plant point cloud data according to the requirements. Selecting one visual angle from a plurality of visual angles as a main visual angle, and according to point cloud data of the main visual angle and the included angle theta between the X-axis positive half axis and the main visual angle X-axis positive half axis of other visual angleskAnd carrying out coordinate transformation on the single plant point cloud data of other visual angles, wherein all coordinates after the coordinate transformation are based on the main visual angle. And then, splicing and smoothly filtering the point cloud data of all the visual angles.
The invention has the beneficial effects that: 1) the point cloud information of the plant to be modeled is acquired from multiple visual angles, and the reality of the modeled plant is better and the geometric accuracy is higher due to the multiple visual angles. 2) The original point cloud data is subjected to operations such as filtering, denoising and downsampling, noise and redundant information points are removed, interference can be reduced, and the calculation amount of a neural network in the next step can be reduced. 3) And separating the plant point clouds to be modeled from the point clouds by using a PointNet neural network, wherein the separation effect and efficiency of the method are better and the generalization to different plants is better compared with that of a machine learning method or a manual separation method such as DBSCAN (direct species learning) and the like by using the PointNet neural network to separate targets. 4) Using a base of thetakThe method has small calculated amount, is suitable for splicing multi-view point cloud data, and can correctly integrate the data of each view into the main view.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention;
FIG. 2 is a cloud point at a certain viewing angle;
FIG. 3 is a point cloud image of FIG. 2 after view denoising and downsampling;
FIG. 4 is a cloud point map of a plant isolated from the point cloud data;
FIG. 5 is a cloud point of a plant isolated from another perspective;
fig. 6 is a cloud point diagram after the completion of the two-view stitching.
Detailed Description
The following detailed description of embodiments of the invention is provided in connection with the accompanying drawings.
Referring to fig. 1 to 6, a plant 3D modeling method based on laser radar and deep learning includes the following steps:
1) acquiring multi-view point cloud information of a target by using a laser radar:
1.1) obtaining target point cloud information:
according to the actual situation of the standing tree to be measured, a plurality of visual angles are selected to carry out static scanning on the target, and the scanning time of the static scanning is set to be 1S which can cover 99% of a visual field. Fig. 2 is a point cloud information diagram of the whole target environment acquired from a certain view angle.
1.2) acquiring the position relation among radars at various visual angles:
marking included angle theta formed by positive half shafts of X axes of all adjacent radar view angle coordinate systemsk。
2) Preprocessing the acquired point cloud information:
2.1) carrying out filtering and denoising on the original point cloud data of each visual angle:
the method comprises the steps of firstly, carrying out filtering and denoising on point cloud data of each visual angle, filtering out non-target clusters formed by few points according to a preset threshold value, and then smoothing and resampling the point cloud data by using a moving least square method.
2.2): carrying out down-sampling on the point cloud data after denoising:
dividing the denoised point cloud data into a plurality of equal lattices in space, calculating the number of small lattices divided in each direction, calculating the specific lattice in which each point falls, and mapping each point to different containers by using a hash table. Only one point is kept for the point cloud in each grid. The corresponding hash function is:
Figure BDA0002889394520000071
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z. The point cloud data map after the preprocessing is shown in fig. 3.
3) Carrying out feature extraction and target detection on point cloud data:
3.1): extracting semantic feature points from the preprocessed point cloud data:
and inputting the preprocessed N x 3 point cloud data into a PointNet network as all data of one frame, and aligning the input data by multiplying the input data by a conversion matrix learned by T-Net, so that the invariance of the model to specific space conversion is ensured. And then, extracting the features of the cloud data of each point through a multi-layer perceptron, and aligning the features by using a T-Net. And then performing maximum pooling operation on all dimensions of the features to obtain final global features. And connecting the global features with the previously obtained local features of the clouds of each point in series, and obtaining the classification result of each data point through a multi-layer perceptron.
3.2): and (3) performing target extraction and identification on point cloud data by using PointNet:
and performing global feature extraction on the point cloud after semantic feature extraction, and predicting the final classification score through a multilayer perceptron. The recognition result of each target is given by the classification score.
4): separating the point cloud information of the plants:
4.1): according to the identified target, selecting a plant to be modeled, and separating point cloud data of the plant:
and (4) selecting results from each target point cloud frame according to the identification result in the step (3) and the plant to be modeled. And separating all point cloud data in the frame according to the 3D coordinates of the selected frame and storing the point cloud data as a point cloud file. Fig. 4 and 5 are point cloud information graphs of the same plant separated from two different visual angles.
5): point cloud data coordinate transformation and splicing:
5.1): selecting one view as a main view:
and 4, selecting one visual angle from the single plant point clouds at the multiple visual angles acquired in the step 4 as a main visual angle P.
5.2): and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view angle P is a matrix S of D x 3pAs follows:
Figure BDA0002889394520000091
wherein Xi、Yi、ZiRespectively representing the values of the X-axis, the Y-axis and the Z-axis of the corresponding point cloud coordinates. The matrix corresponding to the point cloud set of other visual angles is SkAs follows:
Figure BDA0002889394520000092
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskAnd calculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle. Corresponding relative matrix SkAs follows:
Figure BDA0002889394520000093
5.3): carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting PCA algorithm, grouping the point cloud data processed by 5.2 in pairs, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein the point cloud set of one point cloud is assumed as follows:
P＝{p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
Figure BDA0002889394520000094
Figure BDA0002889394520000101
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
5.4): and (3) carrying out fine registration on the point cloud image after the coarse registration:
and performing spatial transformation on the point cloud sets P and Q after coarse registration to minimize the distance between the point clouds P and Q, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, stopping iterative computation if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold, and otherwise, recalculating until a convergence condition is met. The point cloud after the fine registration is completed is shown in fig. 6.
6): and (3) smoothing the complete point cloud data:
6.1): and 5, carrying out grid smoothing on the basis of the point cloud data in the step 5:
and 5, moving the concave-convex points to the average position of the adjacent point weighting strategy by adopting a Laplace smoothing algorithm on the complete point cloud data finally generated in the step 5, wherein the influence weight is the reciprocal of the distance.
A plant 3D modeling system based on laser radar and deep learning comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a plant point cloud information separation module, a splicing module, a grid smoothing module and a display module which are sequentially connected;
the point cloud image acquisition module is connected with the laser radar, and the laser radar is used for statically scanning plants at multiple visual angles to acquire original point cloud data at different visual angles;
the preprocessing module is used for preprocessing the filtering, denoising and downsampling of each laser radar original point cloud data obtained in the step S1, and specifically comprises the following steps:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
Figure BDA0002889394520000111
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
The target identification module is used for extracting semantic feature points from the laser radar point cloud data input by the preprocessing module and identifying each target in the environment, and specifically comprises the following steps:
s3.1: extracting semantic feature points from the point cloud data processed by the preprocessing module:
inputting N x 3 point cloud data processed by a preprocessing module into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of the model to specific space conversion is ensured, then extracting the characteristics of each point cloud data by a plurality of multilayer perceptrons, aligning the characteristics by using one T-Net, then performing maximum pooling operation on each dimension of the characteristics to obtain final global characteristics, connecting the global characteristics with the cloud local characteristics of each point obtained previously in series, and obtaining the classification result of each data point by using the multilayer perceptrons;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
performing global feature extraction on the classified point cloud processed in the S3.1, predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score;
the point cloud information separation module of the plant selects the targets identified by the preprocessing module and separates the point cloud information of the plant;
the splicing module splices a plurality of point cloud data processed by the point cloud information separation module of the plant, and specifically comprises:
s5.1: selecting one view as a main view:
selecting a visual angle from single plant point clouds at a plurality of visual angles acquired by a point cloud information separation module of a plant as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
Figure BDA0002889394520000121
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
Figure BDA0002889394520000122
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
Figure BDA0002889394520000131
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P＝{p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
Figure BDA0002889394520000132
Figure BDA0002889394520000133
x, Y, Z and X ', Y ' and Z ' are respectively 3 column vectors of a target point set matrix and a source point set matrix, 3 eigenvectors of a covariance matrix cov are used as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is used as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration;
s5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
performing space transformation on the point cloud sets P and Q after S5.3 coarse registration to enable the distance between the point clouds P and Q to be minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative computation, and if not, recalculating until a convergence condition is met;
the mesh smoothing module is used for carrying out mesh smoothing on the point cloud data processed by the splicing module;
and the display module is used for carrying out three-dimensional display on the point cloud processed by the grid smoothing module.
The embodiments described in this specification are merely illustrative of implementations of the inventive concept and the scope of the present invention should not be considered limited to the specific forms set forth in the embodiments but rather by the equivalents thereof as may occur to those skilled in the art upon consideration of the present inventive concept.

Claims (5)

1. A plant 3D modeling method and system based on laser radar and deep learning are characterized by comprising the following steps:
s1: performing static scanning on the plants at a plurality of visual angles by using a laser radar to obtain original point cloud data at different visual angles;
s2: preprocessing filtering, denoising and downsampling each laser radar original point cloud data obtained in the step S1;
s3: extracting semantic feature points from the laser radar point cloud data processed in the step S2, and identifying each target in the environment;
s4: selecting the targets identified in the step S3, and separating point cloud information of plants;
s5: splicing the plurality of point cloud data processed in step S4;
s6: performing mesh smoothing on the point cloud data processed in the step S5;
s7: and (4) performing three-dimensional display on the point cloud processed in the step (S6).
2. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S2 specifically includes:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
hash(hx,hy,hz):
Figure FDA0002889394510000021
E.g.，hash(hx+hy*Dx+hz*Dx*Dy)％container_size (1)
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
3. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S3 specifically includes:
s3.1: extracting semantic feature points from the point cloud data processed in the step S2:
inputting N x 3 point cloud data processed by S2 into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of a model to specific space conversion is ensured, then, the cloud data of each point are subjected to feature extraction by a multi-layer perceptron, then, one T-Net is used for aligning the features, then, the maximum pooling operation is performed on each dimension of the features to obtain the final global features, the global features and the cloud local features of each point obtained before are connected in series, and then, the classification result of each data point is obtained by the multi-layer perceptron;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
and (3) performing global feature extraction on the classified point cloud processed in the step (S3.1), predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score.
4. The method and system for 3D modeling of plants based on lidar and deep learning of claim 1, wherein: the step S5 specifically includes:
s5.1: selecting one view as a main view:
selecting a visual angle from the single plant point clouds at the multiple visual angles acquired in the step S4 as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
Figure FDA0002889394510000031
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
Figure FDA0002889394510000032
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs follows:
Figure FDA0002889394510000033
s5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P＝{p1,p2,...,pn} (5)
pifor a single point set, the mean is first calculated and the covariance matrix of the point set is calculated cov, the target point set point cloud covariance matrix and the source point cloud covariance matrix are shown below:
Figure FDA0002889394510000041
Figure FDA0002889394510000042
x, Y, Z and X ', Y ' and Z ' are 3 column vectors of a target point set matrix and a source point set matrix respectively, 3 eigenvectors of a covariance matrix cov are taken as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is taken as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration.
S5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
and (3) carrying out space transformation on the point cloud sets P and Q after the rough registration in the S5.3 to enable the distance between the point clouds P and Q to be the minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of the data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative calculation, and if not, recalculating until a convergence condition is met.
5. The utility model provides a plant 3D modeling's system based on laser radar and deep learning which characterized in that: the system comprises a point cloud image acquisition module, a preprocessing module, a target identification module, a point cloud information separation module of plants, a splicing module, a grid smoothing module and a display module which are connected in sequence;
the point cloud image acquisition module is connected with the laser radar, and the laser radar is used for statically scanning plants at multiple visual angles to acquire original point cloud data at different visual angles;
the preprocessing module is used for preprocessing the filtering, denoising and downsampling of each laser radar original point cloud data obtained in the step S1, and specifically comprises the following steps:
s2.1: carrying out filtering and denoising on the original point cloud data of each visual angle:
firstly, filtering and denoising point cloud data of each visual angle, filtering clusters formed by few points, and smoothing and resampling the point cloud data by using a moving least square method;
s2.2: carrying out down-sampling on the point cloud data after denoising:
dividing the point cloud data processed in the step S2.1 into a plurality of equal grids in space, calculating how many small grids are divided in each direction, calculating in which specific grid each point specifically falls, mapping each point into different containers by using a hash table, and only reserving one point for the point cloud in each grid, wherein the corresponding hash function is as follows:
hash(hx,hy,hz):
Figure FDA0002889394510000051
E.g.，hash(hx+hy*Dx+hz*Dx*Dy)％container_size (1)
wherein Dx、Dy、DzDimension of the voxel grid, h, being X, Y, Z dimensions, respectivelyx、hy、hzIs an index of voxel points in the direction X, Y, Z.
The target identification module is used for extracting semantic feature points from the laser radar point cloud data input by the preprocessing module and identifying each target in the environment, and specifically comprises the following steps:
s3.1: extracting semantic feature points from the point cloud data processed by the preprocessing module:
inputting N x 3 point cloud data processed by a preprocessing module into a PointNet network as all data of one frame, wherein the input data are aligned by multiplying with a conversion matrix learned by T-Net, the invariance of the model to specific space conversion is ensured, then extracting the characteristics of each point cloud data by a plurality of multilayer perceptrons, aligning the characteristics by using one T-Net, then performing maximum pooling operation on each dimension of the characteristics to obtain final global characteristics, connecting the global characteristics with the cloud local characteristics of each point obtained previously in series, and obtaining the classification result of each data point by using the multilayer perceptrons;
s3.2: and (3) performing target extraction and identification on point cloud data by using PointNet:
performing global feature extraction on the classified point cloud processed in the S3.1, predicting a final classification score through a multilayer perceptron, and giving an identification result of each target through the classification score;
the point cloud information separation module of the plant selects the targets identified by the preprocessing module and separates the point cloud information of the plant;
the splicing module splices a plurality of point cloud data processed by the point cloud information separation module of the plant, and specifically comprises:
s5.1: selecting one view as a main view:
selecting a visual angle from single plant point clouds at a plurality of visual angles acquired by a point cloud information separation module of a plant as a main visual angle P;
s5.2: and performing coordinate transformation on the point clouds of other visual angles according to the main visual angle:
the point cloud coordinate set of the main view P is a matrix Sp of D × 3, as follows:
Figure FDA0002889394510000061
wherein Xi、Yi、ZiRespectively representing the values of X-axis, Y-axis and Z-axis of the corresponding point cloud coordinates, and the matrix corresponding to the point cloud set of other visual angles is SkAs follows:
Figure FDA0002889394510000071
then according to the included angle theta between the positive half axis of the X axis at the main visual angle and the positive half axis of the X axis at other visual angleskCalculating the point cloud set matrix of other visual angles relative to the point cloud set matrix of the main visual angle and the corresponding relative matrix SkAs shown below：
Figure FDA0002889394510000072
S5.3: carrying out coarse registration on the point clouds after coordinate transformation in pairs:
adopting a PCA algorithm, grouping the point cloud data processed in S5.2 pairwise, and then obtaining a conversion matrix of the two-point cloud data according to the main shaft trend and the main morphological characteristics of the two-point cloud data, wherein a point cloud set of one point cloud is assumed as follows:
P＝{p1,p2,...,pn} (5)
pi is a single point set, the mean value is firstly calculated and the covariance matrix cov of the point set is calculated, and the target point set point cloud covariance matrix and the source point cloud covariance matrix are respectively as follows:
Figure FDA0002889394510000073
Figure FDA0002889394510000074
x, Y, Z and X ', Y ' and Z ' are respectively 3 column vectors of a target point set matrix and a source point set matrix, 3 eigenvectors of a covariance matrix cov are used as 3 coordinate axes of a space rectangular coordinate system of a point set P, the mean value is used as a coordinate origin of the coordinate system, coordinate transformation matrix parameters of source point set point cloud data corresponding to the target point set point cloud data are calculated, the source point set data are uniformly transformed to the space rectangular coordinate system of the target point set data by using the coordinate transformation matrix parameters, and the two point cloud sets are substantially unified to the same coordinate system after two-two coarse registration;
s5.4: and (3) performing fine registration on the point cloud image after the S5.3 coarse registration:
performing space transformation on the point cloud sets P and Q after S5.3 coarse registration to enable the distance between the point clouds P and Q to be minimum, respectively selecting two point sets to represent a source point set and a target point set in an overlapping area of data of the two point clouds to be registered, calculating the error between the target point set and the source point set of the point clouds P and Q under a rotation matrix, calculating an optimal transformation matrix, if the calculated transformation matrix meets the requirement of a target function, namely the average distance between the two point sets is smaller than a given threshold value, stopping iterative computation, and if not, recalculating until a convergence condition is met;
the mesh smoothing module is used for carrying out mesh smoothing on the point cloud data processed by the splicing module;
and the display module is used for carrying out three-dimensional display on the point cloud processed by the grid smoothing module.
CN202110023105.9A 2021-01-08 2021-01-08 Plant 3D modeling method and system based on laser radar and deep learning Withdrawn CN112712589A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110023105.9A CN112712589A (en) 2021-01-08 2021-01-08 Plant 3D modeling method and system based on laser radar and deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110023105.9A CN112712589A (en) 2021-01-08 2021-01-08 Plant 3D modeling method and system based on laser radar and deep learning

Publications (1)

Publication Number Publication Date
CN112712589A true CN112712589A (en) 2021-04-27

Family

ID=75548572

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110023105.9A Withdrawn CN112712589A (en) 2021-01-08 2021-01-08 Plant 3D modeling method and system based on laser radar and deep learning

Country Status (1)

Country Link
CN (1) CN112712589A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113256640A (en) * 2021-05-31 2021-08-13 北京理工大学 Method and device for partitioning network point cloud and generating virtual environment based on PointNet
CN113409301A (en) * 2021-07-12 2021-09-17 上海精劢医疗科技有限公司 Point cloud segmentation-based femoral neck registration method, system and medium
CN113421290A (en) * 2021-07-05 2021-09-21 山西大学 Power plant boiler interior three-dimensional reconstruction method based on unmanned aerial vehicle image acquisition
CN113434936A (en) * 2021-06-28 2021-09-24 北京工业大学 Road geometric element estimation method and device
CN113447953A (en) * 2021-06-29 2021-09-28 山东高速建设管理集团有限公司 Background filtering method based on road traffic point cloud data
CN113640772A (en) * 2021-09-16 2021-11-12 招商局检测车辆技术研究院有限公司 Method and system for realizing target perception in vehicle-road cooperation

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113256640A (en) * 2021-05-31 2021-08-13 北京理工大学 Method and device for partitioning network point cloud and generating virtual environment based on PointNet
CN113256640B (en) * 2021-05-31 2022-05-24 北京理工大学 Method and device for partitioning network point cloud and generating virtual environment based on PointNet
CN113434936A (en) * 2021-06-28 2021-09-24 北京工业大学 Road geometric element estimation method and device
CN113447953A (en) * 2021-06-29 2021-09-28 山东高速建设管理集团有限公司 Background filtering method based on road traffic point cloud data
CN113421290A (en) * 2021-07-05 2021-09-21 山西大学 Power plant boiler interior three-dimensional reconstruction method based on unmanned aerial vehicle image acquisition
CN113409301A (en) * 2021-07-12 2021-09-17 上海精劢医疗科技有限公司 Point cloud segmentation-based femoral neck registration method, system and medium
CN113640772A (en) * 2021-09-16 2021-11-12 招商局检测车辆技术研究院有限公司 Method and system for realizing target perception in vehicle-road cooperation

Similar Documents

Publication Publication Date Title
CN113128405B (en) Plant identification and model construction method combining semantic segmentation and point cloud processing
CN112712589A (en) Plant 3D modeling method and system based on laser radar and deep learning
CN102236794B (en) Recognition and pose determination of 3D objects in 3D scenes
EP2720171B1 (en) Recognition and pose determination of 3D objects in multimodal scenes
Guerry et al. Snapnet-r: Consistent 3d multi-view semantic labeling for robotics
CN110648361A (en) Real-time pose estimation method and positioning and grabbing system of three-dimensional target object
CN111723721A (en) Three-dimensional target detection method, system and device based on RGB-D
CN113128610B (en) Industrial part pose estimation method and system
CN110942515A (en) Point cloud-based target object three-dimensional computer modeling method and target identification method
Rubio et al. Efficient monocular pose estimation for complex 3D models
CN110490915B (en) Point cloud registration method based on convolution-limited Boltzmann machine
Ansari A review on SIFT and SURF for underwater image feature detection and matching
CN114219855A (en) Point cloud normal vector estimation method and device, computer equipment and storage medium
CN114022542A (en) Three-dimensional reconstruction-based 3D database manufacturing method
CN111127556B (en) Target object identification and pose estimation method and device based on 3D vision
CN113592015B (en) Method and device for positioning and training feature matching network
CN117765039A (en) Point cloud coarse registration method, device and equipment
CN110634160B (en) Method for constructing target three-dimensional key point extraction model and recognizing posture in two-dimensional graph
CN112001954A (en) Polar curve constraint-based underwater PCA-SIFT image matching method
CN113409392A (en) 6DoF pose estimation method of reflective workpiece
Ren et al. Robust object pose estimation based on improved point pair features method
Joshi et al. A training free technique for 3D object recognition using the concept of vibration, energy and frequency
CN113543307B (en) Visual information feature fingerprint database construction method, positioning method and device
Wang et al. Enhanced histogram feature descriptor for automated point cloud registration
Liu et al. 6D Pose Estimation Method Using Curvature-Enhanced Point-Pair Features

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20210427

WW01 Invention patent application withdrawn after publication