WO2023173243A1 - Semantic label generation for two-dimensional lidar scanning graph - Google Patents

Semantic label generation for two-dimensional lidar scanning graph Download PDF

Info

Publication number
WO2023173243A1
WO2023173243A1 PCT/CN2022/080561 CN2022080561W WO2023173243A1 WO 2023173243 A1 WO2023173243 A1 WO 2023173243A1 CN 2022080561 W CN2022080561 W CN 2022080561W WO 2023173243 A1 WO2023173243 A1 WO 2023173243A1
Authority
WO
WIPO (PCT)
Prior art keywords
lidar
point cloud
depth
semantic
dimensional
Prior art date
Application number
PCT/CN2022/080561
Other languages
French (fr)
Chinese (zh)
Inventor
闫志鑫
黄智骁
黄新宇
任骝
Original Assignee
罗伯特·博世有限公司
闫志鑫
黄智骁
黄新宇
任骝
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 罗伯特·博世有限公司, 闫志鑫, 黄智骁, 黄新宇, 任骝 filed Critical 罗伯特·博世有限公司
Priority to PCT/CN2022/080561 priority Critical patent/WO2023173243A1/en
Publication of WO2023173243A1 publication Critical patent/WO2023173243A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging

Definitions

  • This application relates to the field of LiDAR, and in particular to a method, device and computer program product for generating semantic labels for two-dimensional LiDAR scans.
  • 2D lidar scans have become widely used.
  • a lidar scanner can be used to obtain very accurate 2D lidar scans.
  • very few deep learning solutions exist for 2D lidar scans This is mainly due to the fact that most 2D lidar scans do not have corresponding semantic annotations, resulting in a lack of training data for training neural network models that can extract semantic information from input 2D lidar scans. Therefore, it is desirable to provide a technical solution for generating semantic annotations of two-dimensional lidar scans.
  • this application aims to propose a method, device and computer program product for generating semantic annotations for two-dimensional lidar scans.
  • This application provides a method for generating semantic annotations for two-dimensional lidar scans.
  • the method includes: obtaining a two-dimensional lidar scan; obtaining a panoramic image corresponding to the two-dimensional lidar scan; and using the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
  • a two-dimensional depth point cloud corresponding to the depth map can be generated. For example, a set of depth points with appropriate heights can be selected from a plurality of depth points included in the depth map to construct a depth point cloud.
  • the two-dimensional lidar point cloud corresponding to the two-dimensional lidar scan image, the mapping relationship between the depth point cloud and the lidar point cloud, and a set of semantic information extracted from the depth map can be generated.
  • the mapping relationship between the depth point cloud and the lidar point cloud can be obtained by registering the depth point cloud to the lidar point cloud.
  • Depth point clouds can be registered to lidar point clouds through coarse matching and precise matching.
  • coarse matching the depth point cloud can be initially registered to the lidar point cloud by estimating the coarse rotation parameters between the depth point cloud and the lidar point cloud through an exhaustive search.
  • precise matching the depth point cloud can be further registered to the lidar point cloud through the iterative closest point (ICP) algorithm.
  • ICP iterative closest point
  • the panoramic image corresponding to the two-dimensional lidar scan can be used to generate a semantic annotation set of the two-dimensional lidar scan.
  • training data for training the semantic information extraction model can be generated based on the two-dimensional lidar scan and the semantic annotation set corresponding to the two-dimensional lidar scan.
  • the neural network model used to extract a set of semantic information from the input two-dimensional lidar scan can be called a semantic information extraction model.
  • This application also provides a device for generating semantic annotations for two-dimensional lidar scans.
  • the device includes: a two-dimensional lidar scan image acquisition module, used to obtain a two-dimensional lidar scan image; a panoramic image acquisition module, used to obtain a panoramic image corresponding to the two-dimensional lidar scan image; and annotation generation A module configured to use the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
  • This application also provides a device for generating semantic annotations for two-dimensional lidar scans.
  • the apparatus includes: at least one processor; and a memory storing computer-executable instructions that, when executed, cause the at least one processor to implement the method described above for a two-dimensional lidar scan. Methods for generating semantic annotations.
  • This application also provides a computer program product for generating semantic annotations of two-dimensional lidar scans.
  • the computer program product includes a computer program executed by at least one processor for implementing the above-described method for semantic annotation generation of two-dimensional lidar scans.
  • FIG. 1 illustrates an exemplary process for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
  • Figure 2 illustrates an exemplary process for generating a depth point cloud from a depth map according to an embodiment of the present disclosure.
  • FIG. 3 illustrates an exemplary process for obtaining a mapping relationship between a depth point cloud and a lidar point cloud according to an embodiment of the present disclosure.
  • Figure 4 illustrates an exemplary process for registering a depth point cloud to a lidar point cloud in accordance with an embodiment of the present disclosure.
  • 5A to 5D illustrate examples of semantic annotation generation of two-dimensional lidar scans according to embodiments of the present disclosure.
  • FIG. 6 illustrates an exemplary apparatus for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
  • FIG. 7 shows another exemplary apparatus for semantic annotation generation of two-dimensional lidar scans according to an embodiment of the present disclosure.
  • FIG. 1 illustrates an exemplary process 100 for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
  • a two-dimensional lidar scan 102 and a panoramic image 104 corresponding to the two-dimensional lidar scan are obtained, and the panoramic image 104 can be used to generate a semantic annotation set of the two-dimensional lidar scan 102 124.
  • the two-dimensional lidar scan 102 can be collected by a two-dimensional lidar scanner.
  • the two-dimensional lidar scan 102 may have a corresponding lidar point cloud 106 .
  • Semantic annotation set 124 may include a set of semantic annotations corresponding to a set of lidar points in lidar point cloud 106 .
  • the panoramic image 104 corresponding to the two-dimensional lidar scan 102 may be captured by a camera capable of photographing the panoramic image in the space where the two-dimensional lidar scan 102 is collected.
  • the camera may be independent of the two-dimensional lidar scan 102. devices other than dimensional lidar scanners.
  • Panoramic image 104 may also be referred to as a 360-degree image.
  • a depth map 110 and a set of semantic information 112 may be extracted from the panoramic image 104 .
  • the depth map 110 and the set of semantic information 112 may be extracted from the panoramic image 104 by a single neural network model 108 .
  • the single neural network model 108 may be a pretrained multi-task neural network model, such as a HoHoNet model.
  • the depth map 110 and the semantic information set 112 can be respectively extracted from the panoramic image 104 through two neural network models 108 .
  • the depth map 110 can be extracted from the panoramic image 104 through the OmniDepth model or the PanoDept model
  • the semantic information set 112 can be extracted from the panoramic image 104 through the Mask R-CNN model.
  • the extracted depth map 110 and the set of semantic information 112 may be utilized to generate a set of semantic annotations 124 for the two-dimensional lidar scan 104 .
  • the depth point cloud 116 may be generated according to the depth map 110 by the depth point cloud generation unit 114 .
  • Depth point cloud 116 may be a two-dimensional point cloud. An exemplary process of generating depth point cloud 116 will be described later in conjunction with FIG. 2 .
  • the mapping relationship 120 between the depth point cloud 116 and the lidar point cloud 106 corresponding to the two-dimensional lidar scan 102 may be obtained by the mapping relationship obtaining unit 118 .
  • a matching depth point in depth point cloud 116 that matches that lidar point may be determined.
  • An exemplary process of obtaining the mapping relationship 120 between the depth point cloud 116 and the lidar point cloud 106 will be described later with reference to FIG. 3 .
  • the semantic annotation set 124 of the two-dimensional lidar scan 102 may be generated by the semantic annotation set generation unit 122 based on the lidar point cloud 106, the mapping relationship 120 and the semantic information set 112.
  • Semantic information set 112 may include a set of semantic information corresponding to depth point cloud 116 .
  • the semantic information of the matching depth point of the lidar point can be assigned to the lidar point based on the mapping relationship 120 to obtain the semantics corresponding to the lidar point. Label.
  • a set of semantic annotations corresponding to lidar point cloud 106 may be combined into semantic annotation set 124 .
  • lidar point cloud 106 may be updated by removing outlier lidar points from lidar point cloud 106 to obtain an updated lidar point cloud.
  • An abnormal lidar point may, for example, be a lidar point whose distance from the center of the lidar point cloud 106 exceeds a predetermined distance threshold.
  • mapping relationship between the depth point cloud 116 and the updated lidar point cloud can be obtained through the mapping relationship obtaining unit 118 .
  • specific order or hierarchy of steps in process 100 is exemplary only, and the process for semantic annotation generation of two-dimensional lidar scans may be performed in an order different from that described.
  • the semantic annotation set generated according to embodiments of the present disclosure can be used to generate training data for training a semantic information extraction model.
  • Training data for training a semantic information extraction model may be generated based on a two-dimensional lidar scan and a set of semantic annotations corresponding to the two-dimensional lidar scan.
  • the semantic information extraction model trained using such training data can be adapted to extract semantic information sets from the input two-dimensional lidar scans.
  • FIG. 2 illustrates an exemplary process 200 for generating a depth point cloud from a depth map in accordance with an embodiment of the present disclosure.
  • a depth point cloud corresponding to the depth map may be generated.
  • Process 200 may correspond to the operations at depth point cloud generation unit 114 in FIG. 1 .
  • a depth map can include multiple depth points. Each depth point can have a height.
  • a depth point cloud can be constructed by selecting a set of depth points with appropriate heights from multiple depth points.
  • a predetermined height range of depth points can be determined.
  • a predetermined height of the depth point may be determined.
  • the predetermined height of the depth point may be labeled h D .
  • the predetermined height h D of the depth point may coincide with the height of the two-dimensional lidar scanner in the three-dimensional model.
  • the two-dimensional lidar scanner may be the scanner used to acquire the two-dimensional lidar scan 102 .
  • the actual height of the 2D lidar scanner can be labeled h L
  • the height of the 2D lidar scanner in the 3D model can be labeled h L′ .
  • the height h L′ of the two-dimensional lidar scanner in the three-dimensional model can be determined by, for example, the following formula:
  • H is the height of the three-dimensional model
  • h R is the height of the room.
  • the height H of the three-dimensional model can be set to 3.26 meters
  • the actual height h of the two-dimensional lidar scanner can be set to 1.2 meters
  • the height h of the room can be set to 3 meters.
  • the height h L′ of the 2D lidar scanner in the 3D model can be approximately 1.3 meters.
  • the number of lidar points in the lidar point cloud can be determined.
  • a predetermined height range of the depth point may be determined based on the predetermined height h D of the depth point and the number of lidar points in the lidar point cloud.
  • the predetermined height range of the depth points will later be used to select the depth points.
  • the predetermined height range of the depth points can be adjusted so that the height of the selected depth point is close to the predetermined height h D , and the number of depth points included in this range is close to or located at the number of lidar points in the lidar point cloud. Same order of magnitude. That is, the predetermined height range can be determined as the number of depth points falling within it is close to or in the same order of magnitude as the number of lidar points in the lidar point cloud, and is a range surrounding the predetermined height h D .
  • the predetermined height range can be determined as a range in which the number of depth points falling within is about 500 and surrounding the predetermined height h D .
  • the predetermined height range may be (1.29, 1.31).
  • a set of depth points whose heights are within a predetermined height range may be selected from a plurality of depth points included in the depth map.
  • a depth point cloud can be constructed based on the selected set of depth points.
  • the depth point cloud can be a two-dimensional point cloud that can be constructed by removing the height information of each depth point from a selected set of depth points.
  • process for generating a depth point cloud from a depth map described above in connection with FIG. 2 is only exemplary.
  • the steps in the process for generating a depth point cloud can be replaced or modified in any way, and the process can include more or fewer steps, depending on the actual application requirements.
  • specific order or hierarchy of steps in process 200 is exemplary only, and the process for generating a depth point cloud may be performed in an order different than that depicted.
  • FIG. 3 illustrates an exemplary process 300 for obtaining a mapping relationship between a depth point cloud and a lidar point cloud according to an embodiment of the present disclosure.
  • the process 300 may correspond to the operations at the mapping relationship obtaining unit 118 in FIG. 1 .
  • the depth point cloud can be registered to the lidar point cloud to obtain a registered depth point cloud.
  • Depth point clouds can be registered to lidar point clouds through coarse matching and precise matching. An exemplary process of registering the depth point cloud to the lidar point cloud will be explained later in conjunction with Figure 4.
  • a matching depth point in the registered depth point cloud that matches the lidar point can be determined.
  • the closest depth point to the lidar point can be identified from the registered depth point cloud, and the closest identified depth point can be determined to be the closest depth point to the lidar point.
  • LiDAR points are matched to matching depth points.
  • a mapping relationship PL ⁇ PD may be generated based on the lidar point cloud and a set of matching depth points corresponding to the lidar point cloud, where PL is the lidar point cloud and PD is the depth point cloud .
  • the process for obtaining the mapping relationship described above in conjunction with FIG. 3 is only exemplary. According to actual application requirements, the steps in the process for obtaining the mapping relationship can be replaced or modified in any way, and the process can include more or fewer steps. Furthermore, the specific order or hierarchy of steps in process 300 is only exemplary, and the process for obtaining the mapping relationship may be performed in an order different from that described.
  • FIG. 4 illustrates an exemplary process 400 for registering a depth point cloud to a lidar point cloud in accordance with an embodiment of the present disclosure.
  • Process 400 may correspond to the operations at step 302 in FIG. 3 .
  • the depth point cloud can be registered to the lidar point cloud through coarse matching and precise matching.
  • the coarse rotation parameters between the depth point cloud and the lidar point cloud can be estimated, thereby preliminarily registering the depth point cloud to the lidar point cloud.
  • This rough rotation parameter can be used as the initial parameter for subsequent precise matching.
  • the coarse rotation parameters between the depth point cloud and the lidar point cloud can be estimated through an exhaustive search.
  • the center of the depth point cloud and the center of the lidar point cloud can be calculated separately.
  • the center of the point cloud can be calculated by calculating the average of the two-dimensional coordinates of all points in the point cloud.
  • the depth point cloud can be transformed by aligning the center of the depth point cloud to the center of the lidar point cloud, as shown in the following formula:
  • c L (x L , y L ) is the center of the lidar point cloud PL
  • c D (x D , y D ) is the center of the depth point cloud PD .
  • the lidar point closest to the depth point can be identified from the lidar point cloud.
  • the closest lidar point can be used as the matching lidar point for that depth point.
  • the distance between each depth point and the corresponding matching lidar point can be calculated, thereby obtaining a set of distances, and the average distance Loss of the set of distances can be calculated, as shown in the following formula:
  • n is the number of depth points included in the depth point cloud, is a point in the depth point cloud, and is a point in the lidar point cloud.
  • the average distance Loss can be recorded.
  • each lidar point can be weighted. For example, the distance between each lidar point and the center of the lidar point cloud can be calculated. Lidar points that are closer to the center of the lidar point cloud can have greater weight.
  • the current depth point cloud can be rotated by a predetermined rotation angle in a clockwise direction, as shown in the following formula:
  • RCM is the rough matching rotation matrix, as shown in the following formula:
  • process 400 may return to 406. By iteratively performing steps 406 to 412, multiple average distance losses can be obtained.
  • process 400 may proceed to 414.
  • a minimum average distance may be selected from the plurality of average distances Loss, and a rotation angle corresponding to the minimum average distance may be identified. This angle can be used as a rough rotation parameter.
  • the depth point cloud may be rotated based on the coarse rotation parameters to obtain a coarsely registered depth point cloud.
  • the coarsely registered depth point cloud can be further registered to the lidar point cloud through precise matching.
  • the coarsely registered depth point cloud can be further registered to the lidar point cloud through an ICP algorithm.
  • the precise rotation parameters and translation parameters between the coarsely registered depth point cloud and lidar point cloud can be estimated through the ICP algorithm.
  • the closest lidar point to the depth point can be identified from the lidar point cloud.
  • the closest lidar point can be used as the matching lidar point for that depth point.
  • the distance between each depth point and the corresponding matching lidar point can be calculated to obtain a set of distances, and the root mean square of the average of the squares of the individual distances in the set can be calculated.
  • Square, RMS) difference Error as shown in the following formula:
  • each lidar point can be weighted. For example, the distance between each lidar point and the center of the lidar point cloud can be calculated. Lidar points that are closer to the center of the lidar point cloud can have greater weight.
  • the current coarsely registered depth point cloud may be transformed using the ICP transformation matrix T icp .
  • the ICP transformation matrix T icp may include precise rotation parameters and translation parameters. This operation can be shown as the following formula:
  • the process 400 may return to 418.
  • the process 400 may proceed to 426.
  • the current coarsely registered depth point cloud may be used as the final registered depth point cloud, and process 400 may end.
  • the process described above in connection with Figure 4 for registering a depth point cloud to a lidar point cloud is exemplary only.
  • the steps in the process for registering a depth point cloud to a lidar point cloud can be replaced or modified in any way, and the process can include more or fewer steps, depending on the actual application requirements.
  • the depth point cloud in addition to estimating rough rotation parameters between the depth point cloud and the lidar point cloud through exhaustive search, the depth point cloud can also be estimated through principal component analysis (PCA) Coarse rotation parameters between LiDAR point clouds.
  • PCA principal component analysis
  • the specific order or hierarchy of steps in process 400 is exemplary only, and the process for registering a depth point cloud to a lidar point cloud may be performed in an order different from that described.
  • 5A to 5D illustrate examples of semantic annotation generation of two-dimensional lidar scans according to embodiments of the present disclosure.
  • FIG. 5A is a schematic diagram 500a of a lidar point cloud corresponding to a two-dimensional lidar scan.
  • a lidar point cloud 502 is presented in diagram 500a.
  • Lidar point cloud 502 may include lidar point cloud 504 and lidar point cloud 506 .
  • the lidar point cloud 504 may be, for example, a lidar point cloud corresponding to an indoor portion acquired by a two-dimensional lidar scanner.
  • the lidar point cloud 506 may be, for example, a lidar point cloud corresponding to an outdoor portion acquired by a two-dimensional lidar scanner.
  • each point in the lidar point cloud 506 is far away from the center of the lidar point cloud 502, these points may be regarded as abnormal lidar points and removed. Accordingly, only semantic annotations of the lidar point cloud 504 may be generated.
  • Figure 5B is a schematic diagram 500b of a depth point cloud corresponding to a panoramic image.
  • the panoramic image may be a panoramic image corresponding to the two-dimensional lidar scan in FIG. 5A.
  • Depth point cloud 508 is presented in diagram 500b.
  • Depth point cloud 508 may have a set of semantic information.
  • different semantic information is indicated through different graphics.
  • the mapping relationship between the depth point cloud 508 and the lidar point cloud 504 can be obtained.
  • the mapping relationship between the depth point cloud 508 and the lidar point cloud 504 can be obtained through the processes 300 to 400 described above in conjunction with FIGS. 3 to 4 .
  • depth point cloud 508 may be registered to lidar point cloud 504 to obtain registered depth point cloud 508'.
  • a matching depth point in the registered depth point cloud 508' that matches the lidar point may be determined.
  • a mapping relationship PL ⁇ PD can be generated based on the lidar point cloud 504 and a set of matching depth points corresponding to the lidar point cloud 504.
  • Figure 5C is a schematic diagram 500c showing a lidar point cloud 504 and a registered depth point cloud 508'.
  • the semantic annotation of the two-dimensional lidar scan image can be generated based on the lidar point cloud 504, the mapping relationship, and the semantic information set of the depth point cloud 508.
  • the set of semantic information of depth point cloud 508 may include a set of semantic information corresponding to depth point cloud 508 .
  • the semantic information of the matching depth point of the lidar point can be assigned to the lidar point based on the mapping relationship to obtain a semantic annotation corresponding to the lidar point.
  • a set of semantic annotations corresponding to lidar point cloud 504 may be combined into a set of semantic annotations for lidar point cloud 504 .
  • Figure 5D is a schematic diagram 500d illustrating a lidar point cloud 504 and a corresponding set of semantic annotations.
  • FIG. 5A to FIG. 5D is just an example of semantic annotation generation of a two-dimensional lidar scan.
  • the 2D lidar scan can have any other form and can have different semantic annotations.
  • FIG. 6 illustrates an exemplary apparatus 600 for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
  • the device 600 may include: a two-dimensional lidar scan image acquisition module 610, used to obtain a two-dimensional lidar scan image; a panoramic image acquisition module 620, used to obtain a panoramic image corresponding to the two-dimensional lidar scan image; and
  • An annotation generation module 630 is configured to use the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
  • the annotation generation module 630 may utilize the panoramic image obtained through the panoramic image obtaining module 620 to generate a semantic annotation set of the two-dimensional lidar scan image obtained through the two-dimensional lidar scan image obtaining module 610 .
  • the apparatus 600 may also include any other module configured for semantic annotation generation of two-dimensional lidar scans according to the above-described embodiments of the present disclosure.
  • the two-dimensional lidar scan acquisition module 610 can be used to obtain a two-dimensional lidar scan collected by a two-dimensional lidar scanner
  • the panoramic image acquisition module 620 can be used to obtain a two-dimensional lidar scan obtained by a two-dimensional lidar scanner.
  • the camera that captures the panoramic image collects the panoramic image corresponding to the two-dimensional lidar scan.
  • the camera may be a device independent of the two-dimensional lidar scanner.
  • FIG. 7 shows another exemplary apparatus 700 for semantic annotation generation of two-dimensional lidar scans according to an embodiment of the present disclosure.
  • Apparatus 700 may include: at least one processor 710; and memory 720 storing computer-executable instructions.
  • the computer-executable instructions when executed, may cause the at least one processor 710 to perform any of the operations of the method for semantic annotation generation of a two-dimensional lidar scan as described above.
  • Embodiments of the present disclosure provide a computer program product for semantic annotation generation of two-dimensional lidar scans, including a computer program executed by at least one processor for implementing the method described above for two-dimensional laser scanning. Any operation of the method for generating semantic annotations of radar scans.
  • modules in the device described above can be implemented in various ways. These modules may be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules may be functionally further divided into sub-modules or combined together.
  • processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system.
  • a processor, any portion of a processor, or any combination of processors presented in this disclosure may be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) ), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure.
  • DSP digital signal processor
  • FPGA field programmable gate array
  • PLDs programmable logic devices
  • state machines gate logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure.
  • the functions of a processor, any part of a processor, or any combination of processors given in this disclosure may be implemented as software executed by a microprocessor, microcontroller, DSP, or other suitable platform.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

The present disclosure provides a semantic label generation method and apparatus for a two-dimensional lidar scanning graph, and a computer program product. A two-dimensional lidar scanning graph can be obtained. A panoramic image corresponding to the two-dimensional lidar scanning graph can be obtained. A semantic label set for the two-dimensional lidar scanning graph can be generated using the panoramic image.

Description

二维激光雷达扫描图的语义标注生成Semantic annotation generation for 2D lidar scans 技术领域Technical field
本申请涉及激光雷达(LiDAR)领域，特别涉及一种用于二维激光雷达扫描图的语义标注(semantic label)生成的方法、装置及计算机程序产品。This application relates to the field of LiDAR, and in particular to a method, device and computer program product for generating semantic labels for two-dimensional LiDAR scans.
背景技术Background technique
随着扫地机器人和测量设备市场的发展，二维激光雷达扫描图获得了广泛的应用。在许多场景中，可以利用激光雷达扫描仪获得非常准确的二维激光雷达扫描图。然而，仅存在极少的针对二维激光雷达扫描图的深度学习解决方案。这主要是由于大多数的二维激光雷达扫描图不具有相应的语义标注，从而导致缺乏用于训练能够从输入的二维激光雷达扫描图中提取语义信息的神经网络模型的训练数据。因此，提供一种生成二维激光雷达扫描图的语义标注的技术方案是期望的。With the development of the sweeping robot and measurement equipment market, 2D lidar scans have become widely used. In many scenarios, a lidar scanner can be used to obtain very accurate 2D lidar scans. However, only very few deep learning solutions exist for 2D lidar scans. This is mainly due to the fact that most 2D lidar scans do not have corresponding semantic annotations, resulting in a lack of training data for training neural network models that can extract semantic information from input 2D lidar scans. Therefore, it is desirable to provide a technical solution for generating semantic annotations of two-dimensional lidar scans.
发明内容Contents of the invention
针对以上问题，本申请旨在提出一种用于二维激光雷达扫描图的语义标注生成的方法、装置及计算机程序产品。In response to the above problems, this application aims to propose a method, device and computer program product for generating semantic annotations for two-dimensional lidar scans.
本申请提供了一种用于二维激光雷达扫描图的语义标注生成的方法。所述方法包括：获得二维激光雷达扫描图；获得与所述二维激光雷达扫描图相对应的全景图像；以及利用所述全景图像来生成所述二维激光雷达扫描图的语义标注集合。This application provides a method for generating semantic annotations for two-dimensional lidar scans. The method includes: obtaining a two-dimensional lidar scan; obtaining a panoramic image corresponding to the two-dimensional lidar scan; and using the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
根据本申请的一个方面，可以生成与深度图相对应的二维深度点云。例如，可以从深度图所包括的多个深度点中选择具有合适高度的一组深度点来构建深度点云。According to one aspect of the present application, a two-dimensional depth point cloud corresponding to the depth map can be generated. For example, a set of depth points with appropriate heights can be selected from a plurality of depth points included in the depth map to construct a depth point cloud.
根据本申请的另一个方面，可以基于对应于二维激光雷达扫描图的激光雷达点云、深度点云与激光雷达点云之间的映射关系以及从深度图中提取的语义信息集合来生成二维激光雷达扫描图的语义标注集合。According to another aspect of the present application, the two-dimensional lidar point cloud corresponding to the two-dimensional lidar scan image, the mapping relationship between the depth point cloud and the lidar point cloud, and a set of semantic information extracted from the depth map can be generated. A collection of semantic annotations for 3D lidar scans.
根据本申请的又一个方面，可以通过将深度点云配准(register)到激光雷达点云来获得深度点云与激光雷达点云之间的映射关系。可以通过粗略匹配和精准匹配来将深度点云配准到激光雷达点云。在粗略匹配中，可以通过穷举搜索来估计深度点云与激光雷达点云之间的粗略旋转参数，从而将深度点云初步配准到激光雷达点云。在精准匹配中，可以通过迭代最近点(Iterative Closest Point，ICP)算法来将深度点云进一步配准到激光雷达点云。According to yet another aspect of the present application, the mapping relationship between the depth point cloud and the lidar point cloud can be obtained by registering the depth point cloud to the lidar point cloud. Depth point clouds can be registered to lidar point clouds through coarse matching and precise matching. In coarse matching, the depth point cloud can be initially registered to the lidar point cloud by estimating the coarse rotation parameters between the depth point cloud and the lidar point cloud through an exhaustive search. In precise matching, the depth point cloud can be further registered to the lidar point cloud through the iterative closest point (ICP) algorithm.
采用本申请的上述技术手段，可以利用与二维激光雷达扫描图相对应的全景图像来生成二维激光雷达扫描图的语义标注集合。更进一步的，可以基于二维激光雷达扫描图和与该二维激光雷达扫描图相对应的语义标注集合来生成用于训练语义信息提取模型的训练数据。在本文中，可以将用于从输入的二维激光雷达扫描图中提取语义信息集合的神经网络模型称为语义信息提取模型。Using the above technical means of the present application, the panoramic image corresponding to the two-dimensional lidar scan can be used to generate a semantic annotation set of the two-dimensional lidar scan. Furthermore, training data for training the semantic information extraction model can be generated based on the two-dimensional lidar scan and the semantic annotation set corresponding to the two-dimensional lidar scan. In this paper, the neural network model used to extract a set of semantic information from the input two-dimensional lidar scan can be called a semantic information extraction model.
本申请还提供了一种用于二维激光雷达扫描图的语义标注生成的装置。所述装置包括：二维激光雷达扫描图获得模块，用于获得二维激光雷达扫描图；全景图像获得模块，用于获得与所述二维激光雷达扫描图相对应的全景图像；以及标注生成模块，用于利用所述全景图像来生成所述二维激光雷达扫描图的语义标注集合。This application also provides a device for generating semantic annotations for two-dimensional lidar scans. The device includes: a two-dimensional lidar scan image acquisition module, used to obtain a two-dimensional lidar scan image; a panoramic image acquisition module, used to obtain a panoramic image corresponding to the two-dimensional lidar scan image; and annotation generation A module configured to use the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
本申请还提供了一种用于二维激光雷达扫描图的语义标注生成的装置。所述装置包括：至少一个处理器；以及存储计算机可执行指令的存储器，所述计算机可执行指令在被执行时使得所述至少一个处理器实施上文所述的用于二维激光雷达扫描图的语义标注生成的方法。This application also provides a device for generating semantic annotations for two-dimensional lidar scans. The apparatus includes: at least one processor; and a memory storing computer-executable instructions that, when executed, cause the at least one processor to implement the method described above for a two-dimensional lidar scan. Methods for generating semantic annotations.
本申请还提供了一种用于二维激光雷达扫描图的语义标注生成的计算机程序产品。所述计算机程序产品包括计算机程序，所述计算机程序被至少一个处理器执行用于实施上文所述的用于二维激 光雷达扫描图的语义标注生成的方法。This application also provides a computer program product for generating semantic annotations of two-dimensional lidar scans. The computer program product includes a computer program executed by at least one processor for implementing the above-described method for semantic annotation generation of two-dimensional lidar scans.
附图说明Description of the drawings
从后述的详细说明并结合下面的附图将能更全面地理解本申请的前述及其他方面。需要指出的是，各附图的比例出于清楚说明的目的有可能不一样，但这并不会影响对本申请的理解。The foregoing and other aspects of the present application will be more fully understood from the following detailed description in conjunction with the following drawings. It should be noted that the proportions of the drawings may be different for the purpose of clear explanation, but this will not affect the understanding of the present application.
图1示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的示例性过程。FIG. 1 illustrates an exemplary process for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
图2示出了根据本公开实施例的用于根据深度图来生成深度点云的示例性过程。Figure 2 illustrates an exemplary process for generating a depth point cloud from a depth map according to an embodiment of the present disclosure.
图3示出了根据本公开实施例的用于获得深度点云与激光雷达点云之间的映射关系的示例性过程。FIG. 3 illustrates an exemplary process for obtaining a mapping relationship between a depth point cloud and a lidar point cloud according to an embodiment of the present disclosure.
图4示出了根据本公开实施例的用于将深度点云配准到激光雷达点云的示例性过程。Figure 4 illustrates an exemplary process for registering a depth point cloud to a lidar point cloud in accordance with an embodiment of the present disclosure.
图5A至图5D示出了根据本公开实施例的二维激光雷达扫描图的语义标注生成的示例。5A to 5D illustrate examples of semantic annotation generation of two-dimensional lidar scans according to embodiments of the present disclosure.
图6示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的示例性装置。FIG. 6 illustrates an exemplary apparatus for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure.
图7示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的另一示例性装置。FIG. 7 shows another exemplary apparatus for semantic annotation generation of two-dimensional lidar scans according to an embodiment of the present disclosure.
具体实施方式Detailed ways
现在将参考多种示例性实施方式来讨论本公开。应当理解，这些实施方式的讨论仅仅用于使得本领域技术人员能够更好地理解并从而实施本公开的实施例，而并非教导对本公开的范围的任何限制。The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is merely to enable those skilled in the art to better understand and thereby implement embodiments of the disclosure and is not intended to teach any limitation on the scope of the disclosure.
下面给将结合附图详细描述本公开的各个实施例。Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
图1示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的示例性过程100。在过程100中，可以获得二维激光雷达扫描图102以及获得与该二维激光雷达扫描图相对应的全景图像104，并且可以利用全景图像104来生成二维激光雷达扫描图102的语义标注集合124。FIG. 1 illustrates an exemplary process 100 for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure. In the process 100 , a two-dimensional lidar scan 102 and a panoramic image 104 corresponding to the two-dimensional lidar scan are obtained, and the panoramic image 104 can be used to generate a semantic annotation set of the two-dimensional lidar scan 102 124.
可以通过二维激光雷达扫描仪来采集二维激光雷达扫描图102。二维激光雷达扫描图102可以具有相对应的激光雷达点云106。语义标注集合124可以包括与激光雷达点云106中的一组激光雷达点相对应的一组语义标注。The two-dimensional lidar scan 102 can be collected by a two-dimensional lidar scanner. The two-dimensional lidar scan 102 may have a corresponding lidar point cloud 106 . Semantic annotation set 124 may include a set of semantic annotations corresponding to a set of lidar points in lidar point cloud 106 .
可以由能够拍摄全景图像的相机在采集二维激光雷达扫描图102的空间中进行拍摄，从而采集与二维激光雷达扫描图102相对应的全景图像104，所述相机可以是独立于所述二维激光雷达扫描仪以外的设备。全景图像104也可以被称为360度图像。可以从全景图像104中提取深度图110和语义信息集合112。在一种实施方式中，可以通过单个神经网络模型108来从全景图像104中提取深度图110和语义信息集合112。该单个神经网络模型108可以是经预训练的多任务神经网络模型，例如HoHoNet模型。在另一种实施方式中，可以通过两个神经网络模型108来从全景图像104中分别提取深度图110和语义信息集合112。例如，可以通过OmniDepth模型或者PanoDept模型来从全景图像104中提取深度图110，并且可以通过Mask R-CNN模型来从全景图像104中提取语义信息集合112。The panoramic image 104 corresponding to the two-dimensional lidar scan 102 may be captured by a camera capable of photographing the panoramic image in the space where the two-dimensional lidar scan 102 is collected. The camera may be independent of the two-dimensional lidar scan 102. devices other than dimensional lidar scanners. Panoramic image 104 may also be referred to as a 360-degree image. A depth map 110 and a set of semantic information 112 may be extracted from the panoramic image 104 . In one implementation, the depth map 110 and the set of semantic information 112 may be extracted from the panoramic image 104 by a single neural network model 108 . The single neural network model 108 may be a pretrained multi-task neural network model, such as a HoHoNet model. In another implementation, the depth map 110 and the semantic information set 112 can be respectively extracted from the panoramic image 104 through two neural network models 108 . For example, the depth map 110 can be extracted from the panoramic image 104 through the OmniDepth model or the PanoDept model, and the semantic information set 112 can be extracted from the panoramic image 104 through the Mask R-CNN model.
可以利用所提取的深度图110和语义信息集合112来生成二维激光雷达扫描图104的语义标注集合124。可以通过深度点云生成单元114，根据深度图110来生成深度点云116。深度点云116可以是二维点云。后面将结合图2来说明生成深度点云116的示例性过程。随后，可以通过映射关系获得单元118来获得深度点云116与对应于二维激光雷达扫描图102的激光雷达点云106之间的映射关系120。例如，对于激光雷达点云106中的每个激光雷达点，可以确定深度点云116中的与该激光雷达点相匹配的匹配深度点。后面将结合图3来说明获得深度点云116与激光雷达点云106之间的映射关系120的示例性过程。The extracted depth map 110 and the set of semantic information 112 may be utilized to generate a set of semantic annotations 124 for the two-dimensional lidar scan 104 . The depth point cloud 116 may be generated according to the depth map 110 by the depth point cloud generation unit 114 . Depth point cloud 116 may be a two-dimensional point cloud. An exemplary process of generating depth point cloud 116 will be described later in conjunction with FIG. 2 . Subsequently, the mapping relationship 120 between the depth point cloud 116 and the lidar point cloud 106 corresponding to the two-dimensional lidar scan 102 may be obtained by the mapping relationship obtaining unit 118 . For example, for each lidar point in lidar point cloud 106, a matching depth point in depth point cloud 116 that matches that lidar point may be determined. An exemplary process of obtaining the mapping relationship 120 between the depth point cloud 116 and the lidar point cloud 106 will be described later with reference to FIG. 3 .
随后，可以通过语义标注集合生成单元122，基于激光雷达点云106、映射关系120和语义信息集合112来生成二维激光雷达扫描图102的语义标注集合124。语义信息集合112可以包括与深度点云116相对应的一组语义信息。对于激光雷达点云106中的每个激光雷达点，可以基于映射关系120，将该激光雷达点的匹配深度点的语义信息指派给该激光雷达点，以获得与该激光雷达点相对应的语义标注。可以将与激光雷达点云106相对应的一组语义标注组合成语义标注集合124。Subsequently, the semantic annotation set 124 of the two-dimensional lidar scan 102 may be generated by the semantic annotation set generation unit 122 based on the lidar point cloud 106, the mapping relationship 120 and the semantic information set 112. Semantic information set 112 may include a set of semantic information corresponding to depth point cloud 116 . For each lidar point in the lidar point cloud 106, the semantic information of the matching depth point of the lidar point can be assigned to the lidar point based on the mapping relationship 120 to obtain the semantics corresponding to the lidar point. Label. A set of semantic annotations corresponding to lidar point cloud 106 may be combined into semantic annotation set 124 .
应当理解，上文结合图1描述的用于二维激光雷达扫描图的语义标注生成的过程仅是示例性的。根据实际应用需求，可以以任意方式对用于二维激光雷达扫描图的语义标注生成的过程中的步骤进行替换或修改，并且该过程可以包括更多或更少的步骤。例如，在过程100中，可以通过从激光雷达点云106中移除异常激光雷达点来更新激光雷达点云106，以获得经更新的激光雷达点云。异常激光雷达点可以例如是与激光雷达点云106的中心之间的距离超过预定距离阈值的激光雷达点。相应地，可以通过映射关系获得单元118来获得深度点云116与经更新的激光雷达点云之间的映射关系。此外，过程100中的步骤的具体顺序或层级仅是示例性的，可以按照与所描述顺序不同的顺序来执行用于二维激光雷达扫描图的语义标注生成的过程。It should be understood that the process for generating semantic annotations of two-dimensional LiDAR scans described above in conjunction with FIG. 1 is only exemplary. The steps in the process for generating semantic annotations of two-dimensional LiDAR scans can be replaced or modified in any way, and the process can include more or fewer steps, depending on the actual application requirements. For example, in process 100 , lidar point cloud 106 may be updated by removing outlier lidar points from lidar point cloud 106 to obtain an updated lidar point cloud. An abnormal lidar point may, for example, be a lidar point whose distance from the center of the lidar point cloud 106 exceeds a predetermined distance threshold. Correspondingly, the mapping relationship between the depth point cloud 116 and the updated lidar point cloud can be obtained through the mapping relationship obtaining unit 118 . Furthermore, the specific order or hierarchy of steps in process 100 is exemplary only, and the process for semantic annotation generation of two-dimensional lidar scans may be performed in an order different from that described.
根据本公开实施例生成的语义标注集合可以用于生成用于训练语义信息提取模型的训练数据。可以基于二维激光雷达扫描图和与该二维激光雷达扫描图相对应的语义标注集合来生成用于训练语义信息提取模型的训练数据。利用这样的训练数据训练出的语义信息提取模型可以适用于从输入的二维激光雷达扫描图中提取语义信息集合。The semantic annotation set generated according to embodiments of the present disclosure can be used to generate training data for training a semantic information extraction model. Training data for training a semantic information extraction model may be generated based on a two-dimensional lidar scan and a set of semantic annotations corresponding to the two-dimensional lidar scan. The semantic information extraction model trained using such training data can be adapted to extract semantic information sets from the input two-dimensional lidar scans.
图2示出了根据本公开实施例的用于根据深度图来生成深度点云的示例性过程200。在过程200中，可以生成与深度图相对应的深度点云。过程200可以对应于图1中的深度点云生成单元114处的操作。深度图可以包括多个深度点。每个深度点可以具有高度。可以从多个深度点选择具有合适高度的一组深度点来构建深度点云。FIG. 2 illustrates an exemplary process 200 for generating a depth point cloud from a depth map in accordance with an embodiment of the present disclosure. In process 200, a depth point cloud corresponding to the depth map may be generated. Process 200 may correspond to the operations at depth point cloud generation unit 114 in FIG. 1 . A depth map can include multiple depth points. Each depth point can have a height. A depth point cloud can be constructed by selecting a set of depth points with appropriate heights from multiple depth points.
首先，可以确定深度点的预定高度范围。在202处，可以确定深度点的预定高度。可以将深度点的预定高度标记为h D。深度点的预定高度h D可以与二维激光雷达扫描仪在三维模型中的高度相一致。在过程200对应于图1中的深度点云生成单元114处的操作的情况下，二维激光雷达扫描仪可以是用于采集二维激光雷达扫描图102的扫描仪。可以将二维激光雷达扫描仪的实际高度标记为h L，并且将二维激光雷达扫描仪在三维模型中的高度标记为h L′。可以通过例如以下公式来确定二维激光雷达扫描仪在三维模型中的高度h L′: First, a predetermined height range of depth points can be determined. At 202, a predetermined height of the depth point may be determined. The predetermined height of the depth point may be labeled h D . The predetermined height h D of the depth point may coincide with the height of the two-dimensional lidar scanner in the three-dimensional model. Where process 200 corresponds to operations at depth point cloud generation unit 114 in FIG. 1 , the two-dimensional lidar scanner may be the scanner used to acquire the two-dimensional lidar scan 102 . The actual height of the 2D lidar scanner can be labeled h L , and the height of the 2D lidar scanner in the 3D model can be labeled h L′ . The height h L′ of the two-dimensional lidar scanner in the three-dimensional model can be determined by, for example, the following formula:
Figure PCTCN2022080561-appb-000001
Figure PCTCN2022080561-appb-000001
其中，H是三维模型的高度，并且h R是房间的高度。作为示例，可以将三维模型的高度H设置为3.26米，将二维激光雷达扫描仪的实际高度h L设置为1.2米，并且将房间的高度h R设置为3米。在这种情况下，二维激光雷达扫描仪在三维模型中的高度h L′可以约为1.3米。深度点的预定高度h D可以与二维激光雷达扫描仪在三维模型中的高度h L′相一致，即，h D＝h L′。 where H is the height of the three-dimensional model, and h R is the height of the room. As an example, the height H of the three-dimensional model can be set to 3.26 meters, the actual height h of the two-dimensional lidar scanner can be set to 1.2 meters, and the height h of the room can be set to 3 meters. In this case, the height h L′ of the 2D lidar scanner in the 3D model can be approximately 1.3 meters. The predetermined height h D of the depth point may be consistent with the height h L′ of the two-dimensional lidar scanner in the three-dimensional model, ie, h D =h L′ .
在204处，可以确定激光雷达点云中的激光雷达点的数量。At 204, the number of lidar points in the lidar point cloud can be determined.
在206处，可以根据深度点的预定高度h D和激光雷达点云中的激光雷达点的数量，确定深度点的预定高度范围。深度点的预定高度范围将在后续用于选择深度点。可以通过调整深度点的预定高度范围，使得所选择的深度点的高度接近预定高度h D，且该范围内包括的深度点的数量与激光雷达点云中的激光雷达点的数量相接近或者位于相同数量级。也就是说，可以将预定高度范围确定为落在其中的深度点的数量与激光雷达点云中的激光雷达点的数量相接近或者位于相同数量级，且是围绕着预定高度h D的范围。例如，当激光雷达点云中的激光雷达点的数量为500时，可以将预定高度范围确定为落在其中的深度点的数量为500左右，且围绕着预定高度h D的范围。作为示例，当深度点的预定高度h D是1.3米时，预定高度范围可以是(1.29,1.31)。 At 206, a predetermined height range of the depth point may be determined based on the predetermined height h D of the depth point and the number of lidar points in the lidar point cloud. The predetermined height range of the depth points will later be used to select the depth points. The predetermined height range of the depth points can be adjusted so that the height of the selected depth point is close to the predetermined height h D , and the number of depth points included in this range is close to or located at the number of lidar points in the lidar point cloud. Same order of magnitude. That is, the predetermined height range can be determined as the number of depth points falling within it is close to or in the same order of magnitude as the number of lidar points in the lidar point cloud, and is a range surrounding the predetermined height h D . For example, when the number of lidar points in the lidar point cloud is 500, the predetermined height range can be determined as a range in which the number of depth points falling within is about 500 and surrounding the predetermined height h D . As an example, when the predetermined height h D of the depth point is 1.3 meters, the predetermined height range may be (1.29, 1.31).
在208处，可以从深度图所包括的多个深度点中选择高度在预定高度范围内的一组深度点。At 208, a set of depth points whose heights are within a predetermined height range may be selected from a plurality of depth points included in the depth map.
在210处，可以基于所选择的一组深度点来构建深度点云。深度点云可以是二维点云，可以通过从所选择的一组深度点中移除每个深度点的高度信息来构建二维深度点云。At 210, a depth point cloud can be constructed based on the selected set of depth points. The depth point cloud can be a two-dimensional point cloud that can be constructed by removing the height information of each depth point from a selected set of depth points.
应当理解，上文结合图2描述的用于根据深度图来生成深度点云的过程仅是示例性的。根据实际应用需求，可以以任意方式对用于生成深度点云的过程中的步骤进行替换或修改，并且该过程可以包括更多或更少的步骤。此外，过程200中的步骤的具体顺序或层级仅是示例性的，可以按照与所描述顺序不同的顺序来执行用于生成深度点云的过程。It should be understood that the process for generating a depth point cloud from a depth map described above in connection with FIG. 2 is only exemplary. The steps in the process for generating a depth point cloud can be replaced or modified in any way, and the process can include more or fewer steps, depending on the actual application requirements. Furthermore, the specific order or hierarchy of steps in process 200 is exemplary only, and the process for generating a depth point cloud may be performed in an order different than that depicted.
图3示出了根据本公开实施例的用于获得深度点云与激光雷达点云之间的映射关系的示例性过程300。过程300可以对应于图1中的映射关系获得单元118处的操作。FIG. 3 illustrates an exemplary process 300 for obtaining a mapping relationship between a depth point cloud and a lidar point cloud according to an embodiment of the present disclosure. The process 300 may correspond to the operations at the mapping relationship obtaining unit 118 in FIG. 1 .
在302处，可以将深度点云配准到激光雷达点云，以获得经配准的深度点云。可以通过粗略匹配和精准匹配来将深度点云配准到激光雷达点云。后面将结合图4来说明将深度点云配准到激光雷达点云的示例性过程。At 302, the depth point cloud can be registered to the lidar point cloud to obtain a registered depth point cloud. Depth point clouds can be registered to lidar point clouds through coarse matching and precise matching. An exemplary process of registering the depth point cloud to the lidar point cloud will be explained later in conjunction with Figure 4.
在304处，对于激光雷达点云中的每个激光雷达点，可以确定经配准的深度点云中的与该激光雷达点相匹配的匹配深度点。在一种实施方式中，对于特定激光雷达点，可以从经配准的深度点云中识别与该激光雷达点最接近的深度点，并且可以将所识别的最接近的深度点确定为与该激光雷达点相匹配的匹配深度点。At 304, for each lidar point in the lidar point cloud, a matching depth point in the registered depth point cloud that matches the lidar point can be determined. In one embodiment, for a particular lidar point, the closest depth point to the lidar point can be identified from the registered depth point cloud, and the closest identified depth point can be determined to be the closest depth point to the lidar point. LiDAR points are matched to matching depth points.
在306处，可以基于激光雷达点云以及与该激光雷达点云对应的一组匹配深度点来生成映射关系P L→P D，其中P L是激光雷达点云，并且P D是深度点云。 At 306, a mapping relationship PL → PD may be generated based on the lidar point cloud and a set of matching depth points corresponding to the lidar point cloud, where PL is the lidar point cloud and PD is the depth point cloud .
应当理解，上文结合图3描述的用于获得映射关系的过程仅是示例性的。根据实际应用需求，可以以任意方式对用于获得映射关系的过程中的步骤进行替换或修改，并且该过程可以包括更多或更少的步骤。此外，过程300中的步骤的具体顺序或层级仅是示例性的，可以按照与所描述顺序不同的顺序来执行用于获得映射关系的过程。It should be understood that the process for obtaining the mapping relationship described above in conjunction with FIG. 3 is only exemplary. According to actual application requirements, the steps in the process for obtaining the mapping relationship can be replaced or modified in any way, and the process can include more or fewer steps. Furthermore, the specific order or hierarchy of steps in process 300 is only exemplary, and the process for obtaining the mapping relationship may be performed in an order different from that described.
图4示出了根据本公开实施例的用于将深度点云配准到激光雷达点云的示例性过程400。过程400可以对应于图3中的步骤302处的操作。在过程400中，可以通过粗略匹配和精准匹配来将深度点云配准到激光雷达点云。Figure 4 illustrates an exemplary process 400 for registering a depth point cloud to a lidar point cloud in accordance with an embodiment of the present disclosure. Process 400 may correspond to the operations at step 302 in FIG. 3 . In process 400, the depth point cloud can be registered to the lidar point cloud through coarse matching and precise matching.
在粗略匹配中，可以估计深度点云与激光雷达点云之间的粗略旋转参数，从而将深度点云初步配准到激光雷达点云。该粗略旋转参数可以用作后续的精准匹配的初始参数。在一种实施方式中，可以通过穷举搜索来估计深度点云与激光雷达点云之间的粗略旋转参数。In coarse matching, the coarse rotation parameters between the depth point cloud and the lidar point cloud can be estimated, thereby preliminarily registering the depth point cloud to the lidar point cloud. This rough rotation parameter can be used as the initial parameter for subsequent precise matching. In one implementation, the coarse rotation parameters between the depth point cloud and the lidar point cloud can be estimated through an exhaustive search.
在402处，可以分别计算深度点云的中心和激光雷达点云的中心。在一种实施方式中，对于每个点云，可以通过计算该点云中所有点的二维坐标的平均值来计算该点云的中心。At 402, the center of the depth point cloud and the center of the lidar point cloud can be calculated separately. In one embodiment, for each point cloud, the center of the point cloud can be calculated by calculating the average of the two-dimensional coordinates of all points in the point cloud.
在404处，可以通过将深度点云的中心对齐到激光雷达点云的中心来变换深度点云，如以下公式所示：At 404, the depth point cloud can be transformed by aligning the center of the depth point cloud to the center of the lidar point cloud, as shown in the following formula:
(x 0,y 0)＝c L-c D                             (2) (x 0 ,y 0 )＝c L -c D (2)
P D＝P D+(x 0,y 0)                            (3) P D = P D + (x 0 ,y 0 ) (3)
其中，c L＝(x L,y L)是激光雷达点云P L的中心，并且c D＝(x D,y D)是深度点云P D的中心。 Among them, c L =(x L , y L ) is the center of the lidar point cloud PL , and c D =(x D , y D ) is the center of the depth point cloud PD .
在406处，对于当前深度点云中的每个深度点，可以从激光雷达点云中识别与该深度点最接近的激光雷达点。该最接近的激光雷达点可以作为该深度点的匹配激光雷达点。At 406, for each depth point in the current depth point cloud, the lidar point closest to the depth point can be identified from the lidar point cloud. The closest lidar point can be used as the matching lidar point for that depth point.
在408处，可以计算每个深度点与相应的匹配激光雷达点之间的距离，从而获得一组距离，并且可以计算该组距离的平均距离Loss，如以下公式所示：At 408, the distance between each depth point and the corresponding matching lidar point can be calculated, thereby obtaining a set of distances, and the average distance Loss of the set of distances can be calculated, as shown in the following formula:
Figure PCTCN2022080561-appb-000002
Figure PCTCN2022080561-appb-000002
其中，n是深度点云所包括的深度点的数量，
Figure PCTCN2022080561-appb-000003
是深度点云中的点，并且
Figure PCTCN2022080561-appb-000004
是激光雷达点云中的点。可以记录该平均距离Loss。 Among them, n is the number of depth points included in the depth point cloud,
Figure PCTCN2022080561-appb-000003
is a point in the depth point cloud, and
Figure PCTCN2022080561-appb-000004
is a point in the lidar point cloud. The average distance Loss can be recorded.
优选地，在计算平均距离Loss时，可以对各个激光雷达点进行加权。例如，可以计算各个激光雷达点与激光雷达点云的中心之间的距离。与激光雷达点云的中心越接近的激光雷达点可以具有越大的权重。Preferably, when calculating the average distance Loss, each lidar point can be weighted. For example, the distance between each lidar point and the center of the lidar point cloud can be calculated. Lidar points that are closer to the center of the lidar point cloud can have greater weight.
在410处，可以按照顺时针方向，将当前深度点云转动预定转动角度，如以下公式所示：At 410, the current depth point cloud can be rotated by a predetermined rotation angle in a clockwise direction, as shown in the following formula:
Figure PCTCN2022080561-appb-000005
Figure PCTCN2022080561-appb-000005
其中，R CM是粗略匹配旋转矩阵，如以下公式所示： Among them, RCM is the rough matching rotation matrix, as shown in the following formula:
Figure PCTCN2022080561-appb-000006
Figure PCTCN2022080561-appb-000006
其中，θ i是预定转动角度，例如10度，并且θ i+1-θ i＝10°。 Wherein, θ i is a predetermined rotation angle, for example 10 degrees, and θ i+1 - θ i =10°.
在412处，可以确定是否已经旋转了360度。At 412, it can be determined whether it has been rotated 360 degrees.
如果在412处确定了尚未旋转360度，则过程400可以返回至406处。通过迭代地执行步骤406至步骤412，可以获得多个平均距离Loss。If it is determined at 412 that 360 degrees has not been rotated, process 400 may return to 406. By iteratively performing steps 406 to 412, multiple average distance losses can be obtained.
如果在412处确定了已经旋转了360度，则过程400可以进行至414处。在414处，可以从多个平均距离Loss中选择最小平均距离，并且识别与该最小平均距离相对应的旋转角度。该角度可以作为粗略旋转参数。If it is determined at 412 that 360 degrees have been rotated, process 400 may proceed to 414. At 414, a minimum average distance may be selected from the plurality of average distances Loss, and a rotation angle corresponding to the minimum average distance may be identified. This angle can be used as a rough rotation parameter.
在416处，可以基于粗略旋转参数来旋转深度点云，以获得经粗配准的深度点云。At 416, the depth point cloud may be rotated based on the coarse rotation parameters to obtain a coarsely registered depth point cloud.
在获得了经粗配准的深度点云之后，可以通过精准匹配来将经粗配准的深度点云进一步配准到激光雷达点云。在一种实施方式中，可以通过ICP算法来将经粗配准的深度点云进一步配准到激光雷达点云。通过ICP算法可以估计经粗配准的深度点云与激光雷达点云之间的精准旋转参数和平移参数。After obtaining the coarsely registered depth point cloud, the coarsely registered depth point cloud can be further registered to the lidar point cloud through precise matching. In one implementation, the coarsely registered depth point cloud can be further registered to the lidar point cloud through an ICP algorithm. The precise rotation parameters and translation parameters between the coarsely registered depth point cloud and lidar point cloud can be estimated through the ICP algorithm.
在418处，对于当前的经粗配准的深度点云中的每个深度点，可以从激光雷达点云中识别与该深度点最接近的激光雷达点。该最接近的激光雷达点可以作为该深度点的匹配激光雷达点。At 418, for each depth point in the current coarsely registered depth point cloud, the closest lidar point to the depth point can be identified from the lidar point cloud. The closest lidar point can be used as the matching lidar point for that depth point.
在420处，可以计算每个深度点与相应的匹配激光雷达点之间的距离，从而获得一组距离，并且可以计算该组距离中的各个距离的平方的平均值的均方根(Root Mean Square，RMS)差Error，如以下公式所示：At 420, the distance between each depth point and the corresponding matching lidar point can be calculated to obtain a set of distances, and the root mean square of the average of the squares of the individual distances in the set can be calculated. Square, RMS) difference Error, as shown in the following formula:
Figure PCTCN2022080561-appb-000007
Figure PCTCN2022080561-appb-000007
优选地，在计算RMS差Error时，可以对各个激光雷达点进行加权。例如，可以计算各个激光雷达点与激光雷达点云的中心之间的距离。与激光雷达点云的中心越接近的激光雷达点可以具有越大的权重。Preferably, when calculating the RMS difference Error, each lidar point can be weighted. For example, the distance between each lidar point and the center of the lidar point cloud can be calculated. Lidar points that are closer to the center of the lidar point cloud can have greater weight.
在422处，可以使用ICP变换矩阵T icp来对当前的经粗配准的深度点云进行变换。ICP变换矩阵T icp可以包括精准旋转参数和平移参数。该操作可以如以下公式所示： At 422, the current coarsely registered depth point cloud may be transformed using the ICP transformation matrix T icp . The ICP transformation matrix T icp may include precise rotation parameters and translation parameters. This operation can be shown as the following formula:
Figure PCTCN2022080561-appb-000008
Figure PCTCN2022080561-appb-000008
在424处，可以确定RMS差Error是否已经收敛。例如，可以通过确定RMS差Error是否低于阈值来确定RMS差Error是否已经收敛。At 424, it can be determined whether the RMS difference Error has converged. For example, it may be determined whether the RMS difference Error has converged by determining whether the RMS difference Error is below a threshold.
如果在424处确定了RMS差Error尚未收敛，则过程400可以返回至418处。If it is determined at 424 that the RMS difference Error has not converged, the process 400 may return to 418.
如果在424处确定了RMS差Error已经收敛，则过程400可以进行至426处。在426处，可以将当前的经粗配准的深度点云作为最终的经配准的深度点云，并且过程400可以结束。If it is determined at 424 that the RMS difference Error has converged, the process 400 may proceed to 426. At 426, the current coarsely registered depth point cloud may be used as the final registered depth point cloud, and process 400 may end.
应当理解，上文结合图4描述的用于将深度点云配准到激光雷达点云的过程仅是示例性的。根据实际应用需求，可以以任意方式对用于将深度点云配准到激光雷达点云的过程中的步骤进行替换或修改，并且该过程可以包括更多或更少的步骤。例如，在过程400中，除了可以通过穷举搜索来估计深度点云与激光雷达点云之间的粗略旋转参数之外，还可以通过主成分分析(Principal Component Analysis，PCA)来估计深度点云与激光雷达点云之间的粗略旋转参数。此外，过程400中的步骤的具体顺序或层级仅是示例性的，可以按照与所描述顺序不同的顺序来执行用于将深度点云配准到激光雷达点云的过程。It should be understood that the process described above in connection with Figure 4 for registering a depth point cloud to a lidar point cloud is exemplary only. The steps in the process for registering a depth point cloud to a lidar point cloud can be replaced or modified in any way, and the process can include more or fewer steps, depending on the actual application requirements. For example, in process 400, in addition to estimating rough rotation parameters between the depth point cloud and the lidar point cloud through exhaustive search, the depth point cloud can also be estimated through principal component analysis (PCA) Coarse rotation parameters between LiDAR point clouds. Furthermore, the specific order or hierarchy of steps in process 400 is exemplary only, and the process for registering a depth point cloud to a lidar point cloud may be performed in an order different from that described.
图5A至图5D示出了根据本公开实施例的二维激光雷达扫描图的语义标注生成的示例。5A to 5D illustrate examples of semantic annotation generation of two-dimensional lidar scans according to embodiments of the present disclosure.
图5A是与二维激光雷达扫描图相对应的激光雷达点云的示意图500a。在示意图500a中呈现了激光雷达点云502。激光雷达点云502可以包括激光雷达点云504和激光雷达点云506。激光雷达点云504可以例如是由二维激光雷达扫描仪获取的、对应于室内部分的激光雷达点云。激光雷达点云506可以例如是由二维激光雷达扫描仪获取的、对应于室外部分的激光雷达点云。根据本公开的实施例，由于激光雷达点云506中的各个点与激光雷达点云502的中心之间的距离较远，因此这些点可以被视为异常激光雷达点而被移除。相应地，可以仅生成激光雷达点云504的语义标注。Figure 5A is a schematic diagram 500a of a lidar point cloud corresponding to a two-dimensional lidar scan. A lidar point cloud 502 is presented in diagram 500a. Lidar point cloud 502 may include lidar point cloud 504 and lidar point cloud 506 . The lidar point cloud 504 may be, for example, a lidar point cloud corresponding to an indoor portion acquired by a two-dimensional lidar scanner. The lidar point cloud 506 may be, for example, a lidar point cloud corresponding to an outdoor portion acquired by a two-dimensional lidar scanner. According to embodiments of the present disclosure, since each point in the lidar point cloud 506 is far away from the center of the lidar point cloud 502, these points may be regarded as abnormal lidar points and removed. Accordingly, only semantic annotations of the lidar point cloud 504 may be generated.
图5B是与全景图像相对应的深度点云的示意图500b。该全景图像可以是与图5A中的二维激光雷达扫描图相对应的全景图像。在示意图500b中呈现了深度点云508。深度点云508可以具有一组语义信息。在示意图500b中，通过不同的图形指示了不同的语义信息。Figure 5B is a schematic diagram 500b of a depth point cloud corresponding to a panoramic image. The panoramic image may be a panoramic image corresponding to the two-dimensional lidar scan in FIG. 5A. Depth point cloud 508 is presented in diagram 500b. Depth point cloud 508 may have a set of semantic information. In the schematic diagram 500b, different semantic information is indicated through different graphics.
可以获得深度点云508与激光雷达点云504之间的映射关系。可以通过上文结合图3至图4描述的过程300至过程400来获得深度点云508与激光雷达点云504之间的映射关系。例如，可以将深度点云508配准到激光雷达点云504，以获得经配准的深度点云508’。随后，对于激光雷达点云504中的每个激光雷达点，可以确定经配准的深度点云508’中的与该激光雷达点相匹配的匹配深度点。接着，可以基于激光雷达点云504以及与该激光雷达点云504对应的一组匹配深度点来生成映射关系P L→P D。图5C是示出了激光雷达点云504和经配准的深度点云508’的示意图500c。 The mapping relationship between the depth point cloud 508 and the lidar point cloud 504 can be obtained. The mapping relationship between the depth point cloud 508 and the lidar point cloud 504 can be obtained through the processes 300 to 400 described above in conjunction with FIGS. 3 to 4 . For example, depth point cloud 508 may be registered to lidar point cloud 504 to obtain registered depth point cloud 508'. Subsequently, for each lidar point in the lidar point cloud 504, a matching depth point in the registered depth point cloud 508' that matches the lidar point may be determined. Next, a mapping relationship PL → PD can be generated based on the lidar point cloud 504 and a set of matching depth points corresponding to the lidar point cloud 504. Figure 5C is a schematic diagram 500c showing a lidar point cloud 504 and a registered depth point cloud 508'.
在获得了深度点云508与激光雷达点云504之间的映射关系之后，可以基于激光雷达点云504、映射关系和深度点云508的语义信息集合来生成二维激光雷达扫描图的语义标注集合。深度点云508的语义信息集合可以包括与深度点云508相对应的一组语义信息。对于激光雷达点云504中的每个激光雷达点，可以基于映射关系，将该激光雷达点的匹配深度点的语义信息指派给该激光雷达点，以获得与该激光雷达点相对应的语义标注。可以将与激光雷达点云504相对应的一组语义标注组合成激光雷达点云504的语义标注集合。图5D是示出了激光雷达点云504和相应的语义标注集合的示意图500d。After obtaining the mapping relationship between the depth point cloud 508 and the lidar point cloud 504, the semantic annotation of the two-dimensional lidar scan image can be generated based on the lidar point cloud 504, the mapping relationship, and the semantic information set of the depth point cloud 508. gather. The set of semantic information of depth point cloud 508 may include a set of semantic information corresponding to depth point cloud 508 . For each lidar point in the lidar point cloud 504, the semantic information of the matching depth point of the lidar point can be assigned to the lidar point based on the mapping relationship to obtain a semantic annotation corresponding to the lidar point. . A set of semantic annotations corresponding to lidar point cloud 504 may be combined into a set of semantic annotations for lidar point cloud 504 . Figure 5D is a schematic diagram 500d illustrating a lidar point cloud 504 and a corresponding set of semantic annotations.
应当理解，图5A至图5D仅仅是二维激光雷达扫描图的语义标注生成的一个示例。根据实际应用需求，二维激光雷达扫描图可以具有任意其他形式，并且可以具有不同的语义标注。It should be understood that FIG. 5A to FIG. 5D is just an example of semantic annotation generation of a two-dimensional lidar scan. Depending on the actual application requirements, the 2D lidar scan can have any other form and can have different semantic annotations.
图6示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的示例性装置600。装置600可以包括：二维激光雷达扫描图获得模块610，用于获得二维激光雷达扫描图；全景图像获得模块620，用于获得与所述二维激光雷达扫描图相对应的全景图像；以及标注生成模块630，用于利用所述全景图像来生成所述二维激光雷达扫描图的语义标注集合。标注生成模块630可以利用通过全景图像获得模块620获得全景图像来生成通过二维激光雷达扫描图获得模块610获得的二维激光雷达扫描图的语义标注集合。此外，装置600还可以包括根据上述本公开的实施例的被配置用于二维激光雷达扫描图的语义标注生成的任何其他模块。在一些实施例中，所述二维激光雷达扫描图获得模块610可以用于获得由二维激光雷达扫描仪采集的二维激光雷达扫描图，所述全景图像获得模块620可以用于获得由能够拍摄全景图像的相机采集的与二维激光雷达扫描图相对应的全景图像，所述相机可以是独立于所述二维激光雷达扫描仪以外的设备。FIG. 6 illustrates an exemplary apparatus 600 for semantic annotation generation of a two-dimensional lidar scan according to an embodiment of the present disclosure. The device 600 may include: a two-dimensional lidar scan image acquisition module 610, used to obtain a two-dimensional lidar scan image; a panoramic image acquisition module 620, used to obtain a panoramic image corresponding to the two-dimensional lidar scan image; and An annotation generation module 630 is configured to use the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan. The annotation generation module 630 may utilize the panoramic image obtained through the panoramic image obtaining module 620 to generate a semantic annotation set of the two-dimensional lidar scan image obtained through the two-dimensional lidar scan image obtaining module 610 . In addition, the apparatus 600 may also include any other module configured for semantic annotation generation of two-dimensional lidar scans according to the above-described embodiments of the present disclosure. In some embodiments, the two-dimensional lidar scan acquisition module 610 can be used to obtain a two-dimensional lidar scan collected by a two-dimensional lidar scanner, and the panoramic image acquisition module 620 can be used to obtain a two-dimensional lidar scan obtained by a two-dimensional lidar scanner. The camera that captures the panoramic image collects the panoramic image corresponding to the two-dimensional lidar scan. The camera may be a device independent of the two-dimensional lidar scanner.
图7示出了根据本公开实施例的用于二维激光雷达扫描图的语义标注生成的另一示例性装置700。装置700可以包括：至少一个处理器710；以及存储计算机可执行指令的存储器720。所述计算机可执行指令在被执行时可以使得所述至少一个处理器710实施如上所述的用于二维激光雷达扫描图的语义标注生成的方法的任何操作。FIG. 7 shows another exemplary apparatus 700 for semantic annotation generation of two-dimensional lidar scans according to an embodiment of the present disclosure. Apparatus 700 may include: at least one processor 710; and memory 720 storing computer-executable instructions. The computer-executable instructions, when executed, may cause the at least one processor 710 to perform any of the operations of the method for semantic annotation generation of a two-dimensional lidar scan as described above.
本公开的实施例提出了用于二维激光雷达扫描图的语义标注生成的计算机程序产品，包括计算机程序，所述计算机程序被至少一个处理器执行用于实施如上所述的用于二维激光雷达扫描图的语义标注生成的方法的任何操作。Embodiments of the present disclosure provide a computer program product for semantic annotation generation of two-dimensional lidar scans, including a computer program executed by at least one processor for implementing the method described above for two-dimensional laser scanning. Any operation of the method for generating semantic annotations of radar scans.
应当理解，以上描述的装置中的所有模块都可以通过各种方式来实施。这些模块可以被实施为硬件、软件、或其组合。此外，这些模块中的任何模块可以在功能上被进一步划分成子模块或组合在一起。It should be understood that all modules in the device described above can be implemented in various ways. These modules may be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules may be functionally further divided into sub-modules or combined together.
已经结合各种装置和方法描述了处理器。这些处理器可以使用电子硬件、计算机软件或其任意组合来实施。这些处理器是实施为硬件还是软件将取决于具体的应用以及施加在***上的总体设计约束。作为示例，本公开中给出的处理器、处理器的任意部分、或者处理器的任意组合可以实施为微处理器、微控制器、数字信号处理器(DSP)、现场可编程门阵列(FPGA)、可编程逻辑器件(PLD)、状态机、门逻辑、分立硬件电路、以及配置用于执行在本公开中描述的各种功能的其他适合的处理部件。本公开给出的处理器、处理器的任意部分、或者处理器的任意组合的功能可以实施为由微处理器、微控制器、DSP或其他适合的平台所执行的软件。The processor has been described in connection with various devices and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in this disclosure may be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) ), programmable logic devices (PLDs), state machines, gate logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described in this disclosure. The functions of a processor, any part of a processor, or any combination of processors given in this disclosure may be implemented as software executed by a microprocessor, microcontroller, DSP, or other suitable platform.
本领域技术人员应当理解，以上公开的各个实施例可以在不偏离发明实质的情况下做出各种修改和变形，这些修改和变形都应当落入本发明的保护范围之内，并且，本发明的保护范围应当由权利要求书来限定。Those skilled in the art should understand that various modifications and variations can be made to the various embodiments disclosed above without departing from the essence of the invention, and these modifications and variations should fall within the protection scope of the present invention. Furthermore, the present invention The scope of protection should be defined by the claims.

Claims (15)

  1. 一种用于二维激光雷达扫描图的语义标注生成的方法，包括：A method for generating semantic annotations for two-dimensional lidar scans, including:
    获得二维激光雷达扫描图；Obtain a 2D lidar scan;
    获得与所述二维激光雷达扫描图相对应的全景图像；以及Obtaining a panoramic image corresponding to the two-dimensional lidar scan; and
    利用所述全景图像来生成所述二维激光雷达扫描图的语义标注集合。The panoramic image is used to generate a semantic annotation set of the two-dimensional lidar scan.
  2. 根据权利要求1所述的方法，其中，所述生成语义标注集合包括：The method according to claim 1, wherein generating a semantic annotation set includes:
    从所述全景图像中提取深度图和语义信息集合；以及Extract a depth map and semantic information set from the panoramic image; and
    利用所述深度图和所述语义信息集合来生成所述二维激光雷达扫描图的所述语义标注集合。The depth map and the semantic information set are used to generate the semantic annotation set of the two-dimensional lidar scan.
  3. 根据权利要求2所述的方法，其中，所述提取深度图和语义信息集合包括：The method according to claim 2, wherein the extracting the depth map and the semantic information set includes:
    通过单个神经网络模型来从所述全景图像中提取所述深度图和所述语义信息集合；或者Extract the depth map and the set of semantic information from the panoramic image through a single neural network model; or
    通过两个神经网络模型来从所述全景图像中分别提取所述深度图和所述语义信息集合。The depth map and the semantic information set are respectively extracted from the panoramic image through two neural network models.
  4. 根据权利要求2所述的方法，其中，所述二维激光雷达扫描图具有相对应的激光雷达点云，并且所述生成所述语义标注集合包括：The method of claim 2, wherein the two-dimensional lidar scan has a corresponding lidar point cloud, and generating the semantic annotation set includes:
    根据所述深度图来生成深度点云；Generate a depth point cloud according to the depth map;
    获得所述深度点云与所述激光雷达点云之间的映射关系；以及Obtain the mapping relationship between the depth point cloud and the lidar point cloud; and
    基于所述激光雷达点云、所述映射关系和所述语义信息集合来生成所述二维激光雷达扫描图的所述语义标注集合。The semantic annotation set of the two-dimensional lidar scan is generated based on the lidar point cloud, the mapping relationship and the semantic information set.
  5. 根据权利要求4所述的方法，其中，所述深度图包括多个深度点，所述深度点具有高度，并且所述生成深度点云包括：The method of claim 4, wherein the depth map includes a plurality of depth points, the depth points have heights, and the generating a depth point cloud includes:
    确定深度点的预定高度范围；determining a predetermined height range of depth points;
    从所述多个深度点中选择高度在所述预定高度范围内的一组深度点；以及Select a set of depth points with a height within the predetermined height range from the plurality of depth points; and
    基于所述一组深度点来构建所述深度点云。The depth point cloud is constructed based on the set of depth points.
  6. 根据权利要求5所述的方法，其中，所述确定预定高度范围包括：The method of claim 5, wherein determining the predetermined height range includes:
    确定深度点的预定高度；Determine the predetermined height of the depth point;
    确定所述激光雷达点云中的激光雷达点的数量；以及determining a number of lidar points in the lidar point cloud; and
    根据所述预定高度和所述激光雷达点的数量，确定所述预定高度范围。The predetermined height range is determined based on the predetermined height and the number of lidar points.
  7. 根据权利要求4所述的方法，其中，所述获得映射关系包括：The method according to claim 4, wherein said obtaining the mapping relationship includes:
    将所述深度点云配准到所述激光雷达点云，以获得经配准的深度点云；Register the depth point cloud to the lidar point cloud to obtain a registered depth point cloud;
    对于所述激光雷达点云中的每个激光雷达点，确定所述经配准的深度点云中的与所述激光雷达点相匹配的匹配深度点；以及For each lidar point in the lidar point cloud, determining a matching depth point in the registered depth point cloud that matches the lidar point; and
    基于所述激光雷达点云以及与所述激光雷达点云对应的一组匹配深度点来生成所述映射关系。The mapping relationship is generated based on the lidar point cloud and a set of matching depth points corresponding to the lidar point cloud.
  8. 根据权利要求7所述的方法，其中，所述将所述深度点云配准到所述激光雷达点云包括：The method of claim 7, wherein registering the depth point cloud to the lidar point cloud includes:
    估计粗略旋转参数；Estimate coarse rotation parameters;
    基于所述粗略旋转参数来旋转所述深度点云，以获得经粗配准的深度点云；以及Rotate the depth point cloud based on the coarse rotation parameter to obtain a coarsely registered depth point cloud; and
    通过迭代最近点算法来将所述经粗配准的深度点云进一步配准到所述激光雷达点云。The coarsely registered depth point cloud is further registered to the lidar point cloud by an iterative closest point algorithm.
  9. 根据权利要求7所述的方法，其中，所述确定匹配深度点包括：The method of claim 7, wherein determining matching depth points includes:
    从所述经配准的深度点云中识别与所述激光雷达点最接近的深度点；以及Identify the closest depth point to the lidar point from the registered depth point cloud; and
    将所述最接近的深度点确定为与所述激光雷达点相匹配的所述匹配深度点。The closest depth point is determined as the matching depth point that matches the lidar point.
  10. 根据权利要求4所述的方法，还包括：The method of claim 4, further comprising:
    通过从所述激光雷达点云中移除异常激光雷达点来更新所述激光雷达点云，以获得经更新的激光雷达点云，并且updating the lidar point cloud by removing abnormal lidar points from the lidar point cloud to obtain an updated lidar point cloud, and
    其中，所述获得所述深度点云与所述激光雷达点云之间的映射关系包括：Wherein, obtaining the mapping relationship between the depth point cloud and the lidar point cloud includes:
    获得所述深度点云与所述经更新的激光雷达点云之间的映射关系。A mapping relationship between the depth point cloud and the updated lidar point cloud is obtained.
  11. 根据权利要求4所述的方法，其中，所述语义信息集合包括与所述深度点云相对应的一组语义信息，并且所述生成所述语义标注集合包括：The method of claim 4, wherein the set of semantic information includes a set of semantic information corresponding to the depth point cloud, and the generating the set of semantic annotations includes:
    对于所述激光雷达点云中的每个激光雷达点，基于所述映射关系，将所述深度点云中与所述激光雷达点相匹配的匹配深度点的语义信息指派给所述激光雷达点，以获得与所述激光雷达点相对应的语义标注；以及For each lidar point in the lidar point cloud, based on the mapping relationship, the semantic information of the matching depth point in the depth point cloud that matches the lidar point is assigned to the lidar point , to obtain semantic annotations corresponding to the lidar points; and
    将与所述激光雷达点云相对应的一组语义标注组合成所述语义标注集合。A set of semantic annotations corresponding to the lidar point cloud is combined into the semantic annotation set.
  12. 根据权利要求1所述的方法，还包括：The method of claim 1, further comprising:
    基于所述二维激光雷达扫描图和所述语义标注集合来生成用于训练语义信息提取模型的训练数据，所述语义信息提取模型适用于从输入的二维激光雷达扫描图中提取语义信息集合。Training data for training a semantic information extraction model is generated based on the two-dimensional lidar scan and the semantic annotation set, and the semantic information extraction model is suitable for extracting a semantic information set from the input two-dimensional lidar scan. .
  13. 一种用于二维激光雷达扫描图的语义标注生成的装置，包括：A device for generating semantic annotations for two-dimensional lidar scans, including:
    二维激光雷达扫描图获得模块，用于获得二维激光雷达扫描图；The two-dimensional lidar scan image acquisition module is used to obtain the two-dimensional lidar scan image;
    全景图像获得模块，用于获得与所述二维激光雷达扫描图相对应的全景图像；以及a panoramic image acquisition module, used to obtain a panoramic image corresponding to the two-dimensional lidar scan; and
    标注生成模块，用于利用所述全景图像来生成所述二维激光雷达扫描图的语义标注集合。An annotation generation module, configured to use the panoramic image to generate a semantic annotation set of the two-dimensional lidar scan.
  14. 一种用于二维激光雷达扫描图的语义标注生成的装置，包括：A device for generating semantic annotations for two-dimensional lidar scans, including:
    至少一个处理器；以及at least one processor; and
    存储计算机可执行指令的存储器，所述计算机可执行指令在被执行时使得所述至少一个处理器实施根据权利要求1至12中任一项所述的方法。A memory storing computer-executable instructions which, when executed, cause the at least one processor to implement a method according to any one of claims 1 to 12.
  15. 一种用于二维激光雷达扫描图的语义标注生成的计算机程序产品，包括计算机程序，所述计算机程序被至少一个处理器执行用于实施根据权利要求1至12中任一项所述的方法。A computer program product for semantic annotation generation of two-dimensional lidar scans, comprising a computer program executed by at least one processor for implementing the method according to any one of claims 1 to 12 .
PCT/CN2022/080561 2022-03-14 2022-03-14 Semantic label generation for two-dimensional lidar scanning graph WO2023173243A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/080561 WO2023173243A1 (en) 2022-03-14 2022-03-14 Semantic label generation for two-dimensional lidar scanning graph

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/080561 WO2023173243A1 (en) 2022-03-14 2022-03-14 Semantic label generation for two-dimensional lidar scanning graph

Publications (1)

Publication Number Publication Date
WO2023173243A1 true WO2023173243A1 (en) 2023-09-21

Family

ID=88021913

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/080561 WO2023173243A1 (en) 2022-03-14 2022-03-14 Semantic label generation for two-dimensional lidar scanning graph

Country Status (1)

Country Link
WO (1) WO2023173243A1 (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108415032A (en) * 2018-03-05 2018-08-17 中山大学 A kind of point cloud semanteme map constructing method based on deep learning and laser radar
CN110297491A (en) * 2019-07-02 2019-10-01 湖南海森格诺信息技术有限公司 Semantic navigation method and its system based on multiple structured light binocular IR cameras
US20200018852A1 (en) * 2018-07-12 2020-01-16 Toyota Research Institute, Inc. System and method for calibrating a lidar and a camera together using semantic segmentation
CN111461245A (en) * 2020-04-09 2020-07-28 武汉大学 Wheeled robot semantic mapping method and system fusing point cloud and image
CN111486855A (en) * 2020-04-28 2020-08-04 武汉科技大学 Indoor two-dimensional semantic grid map construction method with object navigation points
CN111609852A (en) * 2019-02-25 2020-09-01 北京奇虎科技有限公司 Semantic map construction method, sweeping robot and electronic equipment
CN112767485A (en) * 2021-01-26 2021-05-07 哈尔滨工业大学（深圳） Point cloud map creating and scene identifying method based on static semantic information
CN113902828A (en) * 2021-09-26 2022-01-07 武汉科技大学 Construction method of indoor two-dimensional semantic map with corner as key feature

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108415032A (en) * 2018-03-05 2018-08-17 中山大学 A kind of point cloud semanteme map constructing method based on deep learning and laser radar
US20200018852A1 (en) * 2018-07-12 2020-01-16 Toyota Research Institute, Inc. System and method for calibrating a lidar and a camera together using semantic segmentation
CN111609852A (en) * 2019-02-25 2020-09-01 北京奇虎科技有限公司 Semantic map construction method, sweeping robot and electronic equipment
CN110297491A (en) * 2019-07-02 2019-10-01 湖南海森格诺信息技术有限公司 Semantic navigation method and its system based on multiple structured light binocular IR cameras
CN111461245A (en) * 2020-04-09 2020-07-28 武汉大学 Wheeled robot semantic mapping method and system fusing point cloud and image
CN111486855A (en) * 2020-04-28 2020-08-04 武汉科技大学 Indoor two-dimensional semantic grid map construction method with object navigation points
CN112767485A (en) * 2021-01-26 2021-05-07 哈尔滨工业大学（深圳） Point cloud map creating and scene identifying method based on static semantic information
CN113902828A (en) * 2021-09-26 2022-01-07 武汉科技大学 Construction method of indoor two-dimensional semantic map with corner as key feature

Similar Documents

Publication Publication Date Title
CN111063021B (en) Method and device for establishing three-dimensional reconstruction model of space moving target
JP6216508B2 (en) Method for recognition and pose determination of 3D objects in 3D scenes
JP5677798B2 (en) 3D object recognition and position and orientation determination method in 3D scene
Läbe et al. Automatic relative orientation of images
JP6506483B1 (en) System and method for estimating the pose of a textureless object
JP2010079453A (en) Apparatus for generating position-and-orientation estimation model, position and orientation calculation device, image processor, and method for the same
JP2011133273A (en) Estimation apparatus and control method thereof, and program
JP6055435B2 (en) Subject recognition apparatus, subject recognition method, and subject recognition program
JP7033439B2 (en) Ishigaki management system, stone wall management method and computer program
CN116229007A (en) Four-dimensional digital image construction method, device, equipment and medium using BIM modeling
Gupta et al. Augmented reality system using lidar point cloud data for displaying dimensional information of objects on mobile phones
CN110866945A (en) Method for generating three-dimensional tree by automatic identification of oblique photography model
WO2024037562A1 (en) Three-dimensional reconstruction method and apparatus, and computer-readable storage medium
CN110969650B (en) Intensity image and texture sequence registration method based on central projection
WO2023173243A1 (en) Semantic label generation for two-dimensional lidar scanning graph
Zhong et al. Triple screening point cloud registration method based on image and geometric features
Li et al. Automatic Keyline Recognition and 3D Reconstruction For Quasi‐Planar Façades in Close‐range Images
Li-Chee-Ming et al. Generation of three dimensional photo-realistic models from LiDAR and image data
Weinmann et al. Fast and accurate point cloud registration by exploiting inverse cumulative histograms (ICHs)
CN112967398B (en) Three-dimensional data reconstruction method and device and electronic equipment
CN109029365B (en) Method, system, medium and device for extracting different-side image connection points of electric power corridor
Zhang et al. Research on binocular real-time ranging method in window area
CN115222912A (en) Target pose estimation method and device, computing equipment and storage medium
CN112686962A (en) Indoor visual positioning method and device and electronic equipment
CN117233782A (en) Automatic calibration method for multiple sensors of train, electronic equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22931283

Country of ref document: EP

Kind code of ref document: A1