CN113240594B - High-quality point cloud completion method based on multiple hierarchies - Google Patents
High-quality point cloud completion method based on multiple hierarchies Download PDFInfo
- Publication number
- CN113240594B CN113240594B CN202110457260.1A CN202110457260A CN113240594B CN 113240594 B CN113240594 B CN 113240594B CN 202110457260 A CN202110457260 A CN 202110457260A CN 113240594 B CN113240594 B CN 113240594B
- Authority
- CN
- China
- Prior art keywords
- point cloud
- local
- complete
- incomplete
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000011084 recovery Methods 0.000 claims abstract description 21
- 238000012549 training Methods 0.000 claims abstract description 13
- 230000004927 fusion Effects 0.000 claims abstract description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims 1
- 238000013528 artificial neural network Methods 0.000 abstract description 4
- 230000008439 repair process Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 208000009119 Giant Axonal Neuropathy Diseases 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 201000003382 giant axonal neuropathy 1 Diseases 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Architecture (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Image Generation (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a high-quality point cloud completion method based on multiple levels, which belongs to the technical field of computer vision, and is characterized in that original incomplete point cloud is generated into a plurality of incomplete areas through a random area divider, then the global characteristics of the incomplete point cloud and the local characteristics of the incomplete areas are extracted, a complete area after pre-recovery is generated through a pre-recovery module, the complete area after pre-recovery is utilized to obtain fusion characteristics through a multi-level characteristic aggregator, then a strategy from coarse to fine is adopted to generate complete point cloud with rich details, the loss of the obtained local and global point cloud and the corresponding real complete point cloud is calculated, a network is trained according to loss values, a training model is stored, and the method can be applied to incomplete point cloud data for completion operation; the method aims to enable the neural network to pay attention to a local structure during point cloud completion, build a completion frame from the local part to the whole part, and perform key repair on a missing local area, so that the quality of a detailed structure of the point cloud after completion is improved.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a high-quality point cloud completion method based on multiple levels.
Background
The cloud completion means that the computer completes the missing point cloud area according to the input incomplete three-dimensional point cloud shape, so that a point cloud shape with higher quality is obtained. The traditional point cloud completion method is used for performing shape repair based on prior structure information of a three-dimensional object, such as symmetry information or semantic information. However, the conventional method can only process incomplete point cloud shapes with low point cloud missing rate, symmetrical missing, simple structure, and the like. In recent years, with the development of deep learning, more and more researches adopt a completion method based on a neural network.
Groueix et al learn a manifold mapping relationship from a two-dimensional Euclidean space to a three-dimensional curved surface through a constructed decoder, thereby completing a point cloud completion task. Yang et al propose to two-step folding operation on the European style plane, map the two-dimensional grid to the three-dimensional space coordinate through the three-layer perceptron model, improve the smoothness of the point cloud surface after the completion. Yuan et al keeps the arrangement independence attribute of the point cloud data by using a non-convolution feature extractor, and introduces a point cloud generation strategy with a multi-resolution structure, thereby smoothing the point cloud data from coarse to fine. Tchapmi et al designed a tree-based decoder TopNet to generate geometric information of sub-point clouds, thereby exploring potential topological space in the task of point cloud data completion. Liu et al, to generate smooth completion details, introduce a penalty mechanism to prevent overlap between multiple three-dimensional point cloud area surfaces. Wang et al propose a method for increasing the density of a point cloud based on multiple iterations, and introduce a countermeasure network (GANs) to improve the quality of details generated from the point cloud. Wen et al introduce a Skip-Connection operation to transfer feature information between different layers, and refine a local region generated by point cloud while maintaining a complete point cloud structure, thereby reducing information loss in a network learning process. Huang et al designs an unsupervised learning task to learn only the missing local regions to improve the ability of the network to generate differential details in point cloud data.
Although the point cloud completion performance based on deep learning is gradually improved, researchers find that networks tend to focus more on global features to generate point cloud shapes with common features, and ignore point cloud local detail structures.
Disclosure of Invention
The invention mainly aims to provide a high-quality point cloud completion method based on multiple layers, so that a neural network focuses on a local structure during point cloud completion, a high-quality and detailed three-dimensional shape is recovered, and the problems in the background technology can be effectively solved.
In order to achieve the purpose, the invention adopts the technical scheme that:
a high-quality point cloud completion method based on multiple levels mainly comprises the following steps:
s1, generating the original incomplete point cloud into a plurality of incomplete areas through a random area divider;
s2, extracting global features of incomplete point cloud and obtaining local features of incomplete areas in S1;
s3, generating a complete area after pre-recovery through a pre-recovery module;
s4, obtaining fusion characteristics through the multi-level characteristic aggregator through the pre-recovered complete region generated in S3;
s5, generating a complete point cloud with abundant details by adopting a strategy from coarse to fine;
s6, calculating loss of the local and global point clouds and the corresponding real and complete point clouds obtained in S3 and S5, and training a network according to the loss value;
and S7, storing the training model, and performing completion operation on the incomplete point cloud data.
Preferably, in S2, the PCN network structure is used to extract features.
Preferably, a two-dimensional grid is introduced in S3 and S5 to improve the smoothness of the generated point cloud coordinates.
Preferably, the distance between the generated point cloud and the real complete point cloud is calculated by using the chamfer loss in S6 to realize the constraint.
Preferably, the partial loss of the part S3 introduces a dynamic training strategy to raise an additional constraint of the network on the missing region.
Compared with the prior art, the invention has the following beneficial effects:
1. the model has good robustness, adopts a local to global recovery strategy, can effectively cope with point cloud shapes with different deletion degrees, can recover a complete three-dimensional shape, and can generate complex and real local details.
2. The accuracy is high, and the model can obtain 4.39 multiplied by 10 on data of ShapeNet and KITTI respectively by complementing based on a local-to-global network structure-4And 2.49X 10-3The chamfering distance of (2) is excellent in all the subdivided object categories.
3. The method has a good development prospect, the size of the network model is not increased and the operation complexity is not increased, but a delicate structure and training strategy are designed aiming at the practical problems of the problem, the requirement of point cloud completion of a higher standard can be met, and the method becomes an effective preprocessing mode.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a schematic diagram of a point cloud completion network according to the present invention;
FIG. 3 is a diagram of the completion effect of the present invention on different objects;
FIG. 4 is a detailed diagram of the network structure in step S2 according to the present invention;
FIG. 5 is a detailed diagram of the network structure in step S3 according to the present invention;
FIG. 6 is a detailed diagram of the network structure in step S4 according to the present invention;
fig. 7 is a detailed diagram of the network structure in step S5 of the present invention.
Detailed Description
In order to make the technical means, the creation characteristics, the achievement purposes and the effects of the invention easy to understand, the invention is further described with the specific embodiments.
As shown in fig. 1-7, a high quality point cloud completion method based on multiple levels includes the following steps:
s1, generating the original incomplete point cloud into a plurality of incomplete areas through a random area divider;
s2, extracting global features of incomplete point cloud and obtaining local features of incomplete areas in S1;
s3, generating a complete area after pre-recovery through a pre-recovery module;
s4, obtaining fusion characteristics through the multi-level characteristic aggregator through the pre-recovered complete region generated in S3;
s5, generating a complete point cloud with abundant details by adopting a strategy from coarse to fine;
s6, calculating loss of the local and global point clouds and the corresponding real and complete point clouds obtained in S3 and S5, and training a network according to the loss value;
and S7, storing the training model to obtain the data for complementing incomplete point clouds.
In this embodiment, for each input incomplete point cloud, 10 central points are randomly selected in step S1 by using a Random Sampling method, and then 256 points are randomly selected as incomplete areas in an area with a radius of 0.3. Because each sample can be selected from different central points in iteration, various local areas are finally obtained for pre-recovery, the operation effect is similar to data enhancement, so that the network can train various samples, and the model robustness is improved.
In this embodiment, in S2, the network structure of PCN is used to extract global and regional features. The PCN network consists of two PointNet (PN) layers { H1,H2And (c) composition. The first layer of PN layer obtains point-by-point characteristics through a shared multilayer perceptron (MLP) and obtains global characteristics through maximum pooling operation. And the second PN layer fuses the point-by-point features and the global features, and finally, the global features of the point cloud shape and the global features and the point-by-point features of each region are obtained through coding. The specific network structure is shown in fig. 4.
In the present embodiment, S3 is a pre-recovery module to learn local geometry and recover local regions with rich details from partial objects. Following the principles of predecessor verification, fully connected layers are key to decoders in point cloud completion, which have a strong ability to generate sparse points, and we developed similar decoders to recover regions in the pre-recovery module. Finally, each three-dimensional shape is composed of a set of previously restored complete regions. The specific network structure is shown in fig. 5.
In this embodiment, in S4, we design a multi-level feature aggregator, which fuses the local features of a set of pre-restored regions to obtain the three-dimensional shape feature. Conventional encoders typically apply maximal pooling to extract global features, inevitably losing local structure information. Our goal is to preserve the geometric information of the pre-restored regions and translate it into global features to improve the quality of the generated detail. Inspired by previous approaches, the hierarchy has the potential to filter and pass valuable information. The specific network structure is shown in fig. 6, and the entire multi-level feature aggregation level includes a point level, a local level, and a global level. By skillfully designing a multilayer perceptron and performing maximum pooling and expansion operations, local and global information is continuously transmitted and fused, and finally highly concentrated point cloud characteristics are obtained for recovering a complete shape.
In this embodiment, we adopt a coarse-to-fine recovery strategy in S5, and the specific structure is shown in fig. 7. Recovering coarse-grained complete point cloud with lower resolution in the first stage, and recovering the shape, contour and structure of the three-dimensional point cloud; and correcting the space coordinate at the second stage, and outputting the offset to generate a fine-grained complete point cloud. In order to fully utilize potential features of different levels and improve local smoothness of recovered point cloud, the method for fusing two-dimensional grids, point features, global features extracted in the first stage and original global features obtained by a feature extractor is provided. Furthermore, since neural networks have a stronger ability to learn the remainder of the mapping function, a similar principle is applied to predict the position shift of each point. We predict the position offset to refine the existing coarse-grained points and output complete and fine-grained point clouds with discriminative details.
In this embodiment, S6 calculates the chamfer loss from the local and global point clouds and the corresponding real complete point cloud obtained in S3 and S5, so as to know the neural network model training. First, a chamfer distance is defined, which for the point cloud { X, Y }, can be calculated as:
wherein n isX,nYThe number of points in the point cloud { X, Y }. And (4) for the local point cloud pre-recovered in the step S3, adopting a dynamic training strategy, namely endowing different weights to incomplete areas with different deletion degrees, and focusing attention on the point cloud area with extreme deletion for recovery. Remember that each incomplete shape is represented by the pre-recovery region { R1,R2,...,RnIs composed of the corresponding real complete area
The corresponding distance is obtained by the chamfer distance formula as d1,d2,...,dn}. Dynamic weight w of ith Pre-recovery regioniCan be calculated as:
where k is a parameter for adjusting the chamfer distance. The loss of each incomplete point cloud in the final S3 step can be expressed as:
for the coarse-grained complete point cloud Y obtained in the step S5CAnd coarse-grained complete point cloud YFWhich corresponds to the same real complete point cloud
The loss can be expressed as:
where λ is a hyper-parameter that balances the two losses. The final step S6 calculates the total loss as:
L＝LS3+β·LS5 (5)
wherein β is a hyperparameter for balancing the losses obtained in steps S3 and S5.
According to the high-quality point cloud completion method based on multiple layers, by introducing the local recovery task, the network model focuses on local structure information, and generates the three-dimensional point cloud which has complete shape and abundant details, and has strong generalization performance. It should be noted that the loss of the pre-recovery area calculated in step S5 only occurs in the training stage, and the testing stage only needs to generate a complete point cloud through the pre-recovery area.
The foregoing shows and describes the general principles and broad features of the present invention and advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (4)
1. A high-quality point cloud completion method based on multiple levels is characterized by comprising the following steps: the method mainly comprises the following steps:
s1, generating the original incomplete point cloud into a plurality of incomplete areas through a random area divider; randomly selecting 10 central points by using a Random Sampling method, randomly selecting 256 points in an area with a radius of 0.3 as an incomplete area, and finally pre-recovering various local areas obtained due to the fact that different central points can be selected in each sample in iteration;
s2, extracting global features of incomplete point cloud and obtaining local features of incomplete areas in S1; feature extraction using PCN network architecture, PCN network consisting of two PointNet (PN) layers { H }1,H 2The first PN layer obtains point-by-point characteristics through a shared multilayer perceptron (MLP), and then obtains global characteristics through maximum pooling operation, the second PN layer fuses the point-by-point characteristics and the global characteristics, and finally, the global characteristics of the point cloud shape and the global characteristics and the point-by-point characteristics of each area are obtained through coding;
s3, generating a complete area after pre-recovery through a pre-recovery module; learning local geometric structures and recovering local areas with rich details from partial objects, wherein each three-dimensional shape consists of a group of complete areas recovered in advance;
s4, obtaining fusion characteristics through the pre-recovered complete region generated in the S3 through a multi-level characteristic aggregator; designing a multi-level feature aggregator, fusing local features of a group of pre-recovery regions to obtain the three-dimensional shape feature, wherein the whole multi-level feature aggregator comprises a point level, a local level and a global level, continuously transmitting and fusing local and global information, and finally obtaining a highly concentrated point cloud feature for recovering a complete shape;
s5, generating a complete point cloud with abundant details by adopting a strategy from coarse to fine; recovering coarse-grained complete point clouds with lower resolution in a first stage, and recovering the shape, outline and structure of the three-dimensional point cloud; correcting the space coordinate in the second stage, and outputting the offset to generate a fine-grained complete point cloud;
s6, calculating loss of the local and global point clouds and the corresponding real and complete point clouds obtained in S3 and S5, and training a network according to the loss value;
and S7, storing the training model, and performing completion operation on the incomplete point cloud data.
2. The method of claim 1, wherein the method comprises: two-dimensional grids for improving the smoothness of the generated point cloud coordinates are introduced in the S3 and S5.
3. The method of claim 1, wherein the method comprises: in the step S6, the distance between the generated point cloud and the real complete point cloud is calculated by using the chamfer loss.
4. The method of claim 1, wherein the method comprises: the partial loss of the S3 section introduces a dynamic training strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110457260.1A CN113240594B (en) | 2021-04-27 | 2021-04-27 | High-quality point cloud completion method based on multiple hierarchies |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110457260.1A CN113240594B (en) | 2021-04-27 | 2021-04-27 | High-quality point cloud completion method based on multiple hierarchies |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113240594A CN113240594A (en) | 2021-08-10 |
| CN113240594B true CN113240594B (en) | 2022-06-24 |
Family
ID=77129821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110457260.1A Active CN113240594B (en) | 2021-04-27 | 2021-04-27 | High-quality point cloud completion method based on multiple hierarchies |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113240594B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113902061A (en) * | 2021-11-15 | 2022-01-07 | 深圳先进技术研究院 | Point cloud completion method and device |
| CN117274454A (en) * | 2023-08-29 | 2023-12-22 | 西交利物浦大学 | Three-dimensional point cloud completion method, device and storage medium based on component information |
-
2021
- 2021-04-27 CN CN202110457260.1A patent/CN113240594B/en active Active
Non-Patent Citations (4)
| Title |
|---|
| PCN: Point Completion Network;Wentao Yuan 等;《International Conference on 3D Vision》;20181015;第728-737页 * |
| TopNet: Structural Point Cloud Decoder;Lyne P. Tchapmi 等;《CVF Conference on Computer Vision and Pattern Recognition (CVPR)》;20191231;全文 * |
| 基于多分支结构的点云补全网络;罗开乾等;《激光与光电子学进展》;20201231;第57卷(第24期);第241019-1-241019-7页 * |
| 基于多尺度特征和注意力机制的航空图像分割;宁芊等;《控制理论与应用》;20200630;第37卷(第6期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113240594A (en) | 2021-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022151535A1 (en) | Deep learning-based face feature point detection method | |
| CN108038906B (en) | Three-dimensional quadrilateral mesh model reconstruction method based on image | |
| CN113240594B (en) | High-quality point cloud completion method based on multiple hierarchies | |
| Zhu et al. | AdaFit: Rethinking learning-based normal estimation on point clouds | |
| CN108010103A (en) | The quick fine generation method of river with complicated landform | |
| CN108664885B (en) | Human body key point detection method based on multi-scale cascade Hourglass network | |
| CN111046917B (en) | Object-based enhanced target detection method based on deep neural network | |
| CN111401151B (en) | Accurate three-dimensional hand posture estimation method | |
| CN105118090A (en) | Adaptive point-cloud filtering method for complex terrain structure | |
| CN114463492B (en) | Self-adaptive channel attention three-dimensional reconstruction method based on deep learning | |
| CN110473140A (en) | A kind of image dimension reduction method of the extreme learning machine based on figure insertion | |
| CN110348299A (en) | The recognition methods of three-dimension object | |
| Zhao et al. | Joint learning of salient object detection, depth estimation and contour extraction | |
| US11514660B1 (en) | Scene recognition method, training method and device based on pyramid attention | |
| CN115223017B (en) | Multi-scale feature fusion bridge detection method based on depth separable convolution | |
| Jia et al. | Collaborative contrastive learning for hyperspectral and LiDAR classification | |
| CN110633706B (en) | Semantic segmentation method based on pyramid network | |
| CN117115359B (en) | Multi-view power grid three-dimensional space data reconstruction method based on depth map fusion | |
| CN116935226B (en) | HRNet-based improved remote sensing image road extraction method, system, equipment and medium | |
| CN116129118B (en) | Urban scene laser LiDAR point cloud semantic segmentation method based on graph convolution | |
| CN114937153B (en) | Visual characteristic processing system and method based on neural network in weak texture environment | |
| CN103823973B (en) | Target tracking method based on MeanShift algorithm | |
| CN112308089A (en) | Attention mechanism-based capsule network multi-feature extraction method | |
| CN115393735A (en) | Remote sensing image building extraction method based on improved U-Net | |
| CN114596488A (en) | Lightweight remote sensing target detection method based on dense feature fusion network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |