WO2022147960A1 - 点云的标注方法及标注设备 - Google Patents

点云的标注方法及标注设备 Download PDF

Info

Publication number
WO2022147960A1
WO2022147960A1 PCT/CN2021/099073 CN2021099073W WO2022147960A1 WO 2022147960 A1 WO2022147960 A1 WO 2022147960A1 CN 2021099073 W CN2021099073 W CN 2021099073W WO 2022147960 A1 WO2022147960 A1 WO 2022147960A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
target
frame
target object
frames
Prior art date
Application number
PCT/CN2021/099073
Other languages
English (en)
French (fr)
Inventor
王伟宝
Original Assignee
新石器慧通（北京）科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 新石器慧通（北京）科技有限公司 filed Critical 新石器慧通（北京）科技有限公司
Publication of WO2022147960A1 publication Critical patent/WO2022147960A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Definitions

  • the present application relates to the technical field of unmanned driving and automatic driving, for example, to a point cloud labeling method and labeling device.
  • the point cloud frames collected for the surrounding environment during the driving of the unmanned vehicle are all black and white images.
  • the background in the point cloud frame is black and the point cloud is white, that is, the point cloud frame is an image without color information, which is necessary for the original need.
  • objects at different positions correspond to point cloud data with different degrees of sparseness
  • objects farther away from the unmanned vehicle correspond to sparser point cloud data.
  • the target object will be difficult to be labeled due to insufficient point cloud information.
  • the present application provides a point cloud labeling method and labeling device, which can extend the labeling area and reduce the difficulty of labeling.
  • a point cloud labeling method including:
  • target matching in multiple point cloud frames is performed through target tracking starting from the key frame, so that the point cloud in which the target object is not identified in the multiple point cloud frames for target matching is performed.
  • the matched target is marked in the frame.
  • memory configured to store computer instructions
  • a processor coupled to the memory, is configured to execute, based on computer instructions stored in the memory, implementing the above-described method of labeling a point cloud.
  • FIG. 1 shows a schematic diagram of an unmanned vehicle collecting point cloud frames
  • Fig. 2 shows a point cloud frame in the environment collected by the unmanned vehicle while driving
  • FIG. 3 shows a flowchart of a method for labeling a point cloud provided by Embodiment 1 of the present application
  • FIG. 5 shows a flowchart of a method for extracting key frames provided by Embodiment 3 of the present application
  • FIG. 6 shows a flowchart of a method for target matching provided in Embodiment 4 of the present application
  • FIG. 7 shows a structural block diagram of a point cloud labeling device provided by an embodiment of the present application.
  • Figure 1 shows a self-driving unmanned vehicle.
  • the unmanned vehicle obtains the point cloud data of the surrounding environment through the detection of lidar (the point cloud data can be a part of the point cloud frame after segmentation), and then determines the position of the target object from the point cloud data to ensure the smooth completion of the journey.
  • Lidar detects point cloud data corresponding to the surrounding environment.
  • the principle is as follows: Lidar includes a detection unit and a processing unit.
  • the detection unit includes a transmitter module and a receiver module.
  • the transmitter module emits multiple laser beams around, and one laser beam corresponds to one direction.
  • the receiving module receives the echo returned by the laser beam after encountering the target;
  • the processing unit determines the position of the reflection point of the laser beam according to the interval between the echo reception time and the laser beam emission time and the emission direction of the laser beam.
  • the point position is a cloud point in the point cloud frame shown in Figure 2.
  • a point cloud frame is the data collected after the lidar detects the surrounding objects.
  • the unmanned vehicle determines the position of the target from the above point cloud data. It identifies the target from the point cloud data through the trained target recognition model.
  • training samples are required to participate in the training of the target recognition model. .
  • the training sample here is the point cloud data collected in advance associated with the target object information; after the point cloud data is recognized by the target object recognition model under training, the target object information associated with the point cloud data is used to determine the recognition results. Whether the results are correct and whether it is necessary to continue training the target recognition model. Therefore, it is of great significance to label the target information on the point cloud data.
  • the point cloud frame has no color information, which undoubtedly further increases the difficulty of labeling for manual labeling that would have required a lot of time and labor costs.
  • objects at different positions correspond to point cloud data with different degrees of sparseness, and objects farther away from the unmanned vehicle correspond to sparser point cloud data.
  • the target object will be difficult to be labeled due to insufficient point cloud information.
  • the present application provides a labeling method and labeling device that can reduce the difficulty of standardization and extend the labeling area.
  • Figure 3 shows a flowchart of the point cloud labeling method.
  • the labeling method includes:
  • Step S110 acquiring a plurality of point cloud frames continuously collected during the driving of the unmanned vehicle.
  • multiple point cloud frames are continuously collected by the lidar installed on the unmanned vehicle. Assuming that the multiple point cloud frames are sorted as P 1 , P 2 , . 1 , P 2 , ..., P n , there is no point cloud frame in which the position of the target object jumps, that is to say, the point cloud frame P i is between the point cloud frame P i-1 and the point cloud frame P i+1 .
  • the transition point cloud frame of , the target object from the position indicated by the point cloud frame P i-1 passes through the position indicated by the point cloud frame P i to reach the position indicated by the point cloud frame P i+1 .
  • Lidar will continuously collect point cloud frames according to preset rules. For example, point cloud frames are usually collected at intervals of 0.1s during the driving of the unmanned vehicle, so as to ensure that the point cloud frame P i is the point cloud frame P i-1 and the point cloud frame P i-1. Transition point cloud frames between point cloud frames P i+1 .
  • step S120 target recognition is performed on a plurality of point cloud frames respectively to obtain a recognition result.
  • the target By running a target recognition model (such as the point cloud three-dimensional detection model Pointpillar), the target is automatically recognized in sequence on multiple point cloud frames, so as to quickly obtain multiple point cloud frames associated with target information.
  • the association between the point cloud frame and the target object information described here can be achieved by establishing an association list between the target object information and the position information of the cloud point corresponding to the target object in the point cloud frame, and one association list corresponds to one point cloud frame;
  • the association list includes one association data or multiple association data, and one association data corresponds to a target object and is an association between the target object and a cloud point position information or a plurality of cloud point position information.
  • a target object is identified in the corresponding point cloud frame of the association list; when an association list includes multiple association data, multiple objects are identified in the corresponding point cloud frame of the association list.
  • a target belongs to one of the targets identified in all point cloud frames, the value range of j is 0-m, m is the number of targets identified in all point cloud frames, and the target X j identifies all point clouds the same target within the frame.
  • the above target object recognition model is pre-trained in the following manner: constructing a set of point cloud data samples, where the point cloud data samples in the set include point cloud data collected by an unmanned vehicle and the point cloud data is associated with a pre-identified target object label;
  • the point cloud data samples in the set are input into the target object recognition model, and the target object in the point cloud data sample is identified by the target object recognition model, and compared with the associated target object label; If the sample ratio of the target object and the target object label is consistent with the target object label exceeds the predetermined ratio threshold, it is considered that the target object recognition model is successfully trained; if the target object identified in the set and the target object label consistent sample ratio does not exceed, then adjust the target object recognition model
  • the coefficients of the model such that the ratio of the samples identified in the set that are consistent with the target label exceeds a predetermined ratio threshold.
  • the point cloud labeling method cannot label the long-distance objects in the point cloud data. That is, for the point cloud data for training the target recognition model, there may be a situation that contains a long-distance target but lacks the target label of the target in the long-distance state. In the process of automatic target recognition for each point cloud frame in sequence, if the target X j in the point cloud frame Pi belongs to a long-distance target, the target recognition model cannot identify the target in the point cloud frame Pi object X j .
  • Step S130 according to the recognition result, extract the key frame corresponding to the target object from the plurality of point cloud frames, wherein the key frame corresponding to the target object is the point cloud frame in which the target object is recognized.
  • the point cloud frame in which the target X j is recognized is selected as the key frame.
  • the final selected key frame is a point cloud frame with a high point cloud density or a point cloud frame with easy identification of the target X j , wherein the point cloud density of the key frame is usually greater than 1000/m 3 , that is to say , if there is a target in an area in the space where the unmanned vehicle is located, the corresponding point cloud density of the area in the key frame is greater than 1000/m 3 .
  • the key frame is a point cloud frame corresponding to the target. There is only one key frame for the same target, and different targets may correspond to different key frames. If the association between the point cloud frame and the target object information is represented by the association list in step S120, the key frame can be determined through the association list where the target object information of the target object is located.
  • Step S140 for the target object, target matching in multiple point cloud frames is performed through target tracking starting from the key frame, so as to mark the point cloud frame in which the target object is not identified among the multiple point cloud frames for target matching. matched target.
  • the above point cloud frames for which the target object is not identified are included in the multiple point cloud frames collected in step S110, which may be one frame or multiple frames.
  • the above target matching can use a target tracking algorithm, such as SORT algorithm.
  • target matching in multiple point cloud frames is performed through target tracking, so that if one of the multiple point cloud frames does not recognize the target object If there is a target in the point cloud frame, it can match the target and then mark the target.
  • the point cloud frame of the unidentified target object is completed by combining the key frame to complete the labeling of the target object, and the difficulty of labeling is effectively reduced.
  • this labeling method can label the long-distance target by combining the key frame, so that the labeling area can be extended.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the above-mentioned first embodiment, and thus will not be repeated here.
  • step S130 according to the recognition result, extract the key frame corresponding to the target object from the multiple point cloud frames, including:
  • step S131a according to the identification result, each point cloud frame in which the target object is identified among the plurality of point cloud frames is determined as a candidate frame, and multiple candidate frames are obtained.
  • step S132a the separation distance between the target and the unmanned vehicle in each candidate frame is obtained, and a plurality of separation distances corresponding to the plurality of candidate frames are obtained.
  • step S133a the candidate frame corresponding to the minimum value among the plurality of separation distances is determined as the key frame.
  • each candidate frame corresponds to a separation distance. If the separation distance corresponding to a candidate frame is the separation distance with the smallest value among the plurality of separation distances, the candidate frame are identified as keyframes.
  • association between the point cloud frame and the target object information is represented by the association list in step S120, then if there are multiple candidate frames, there are multiple association lists including the target object information of the target object.
  • This embodiment provides a method from multiple candidate frames. A method of selecting one of the multiple candidate frames corresponding to each association list as a key frame.
  • the candidate frame corresponding to the minimum value among the plurality of separation distances is determined as the key frame, so the key frame has relatively rich information on the target object, which is beneficial to the point cloud frame in which the target object is not identified. to accurately frame all the cloud points belonging to the target.
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the above-mentioned first embodiment, and thus will not be repeated here.
  • step S130 according to the recognition result, extract the key frame corresponding to the target object from the multiple point cloud frames, including:
  • step S131a according to the identification result, each point cloud frame in which the target object is identified among the plurality of point cloud frames is determined as a candidate frame, and multiple candidate frames are obtained.
  • Step S132b obtaining the interval duration between each candidate frame and the same point cloud frame with no target identified in the collection time, and obtaining multiple interval durations corresponding to multiple candidate frames.
  • step S133b the candidate frame corresponding to the minimum value among the plurality of interval durations is determined as the key frame.
  • each candidate frame corresponds to an interval duration. If the interval duration corresponding to a candidate frame is the interval duration with the smallest value among multiple interval durations, the candidate frame are identified as keyframes.
  • association between the point cloud frame and the target object information is represented by the association list in step S120, then if there are multiple candidate frames, there are multiple association lists including the target object information of the target object.
  • This embodiment provides another method from A method of selecting one of multiple candidate frames corresponding to multiple association lists as a key frame.
  • the candidate frame corresponding to the minimum value among the multiple interval durations is determined as the key frame, so that the target object in the point cloud frame where the target object is not identified can be obtained through the evolution of the route with a shorter duration predicted location.
  • Embodiment 4 is a diagrammatic representation of Embodiment 4:
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the above-mentioned first embodiment, and thus will not be repeated here.
  • step S140 target matching of multiple point cloud frames is performed through target tracking starting from the key frame, including:
  • Step S141 determining the driving parameters of the target object according to the position of the target object in the multiple candidate frames, wherein the candidate frame is the point cloud frame in which the target object is identified among the multiple point cloud frames.
  • Step S142 based on the driving parameters of the target object and the driving parameters of the unmanned vehicle, starting from the key frame, deduce the predicted position of the target object in the point cloud frame in which the target object is not identified in the consecutive multiple point cloud frames.
  • Step S143 Identify the target at the predicted position to determine whether the target is matched in the point cloud frame corresponding to the predicted position.
  • the driving parameters of the target can include the linear velocity and angular velocity of the target. After knowing the linear velocity and angular velocity of the target, the driving route of the target can be inferred; similarly, the driving parameters of the unmanned vehicle can include the line of the unmanned vehicle. Speed and angular velocity. After knowing the linear velocity and angular velocity of the unmanned vehicle, the driving route of the unmanned vehicle can be inferred.
  • the above-mentioned multiple candidate frames are the point cloud frames that identify the target corresponding to the transition period between the key frame collection time and the collection time of a point cloud frame that does not recognize the target, and the driving parameters of the target are the key frame collection time.
  • the driving parameters of the target object in the transition period between the acquisition moments of the point cloud frame without the target object similarly, the driving parameters of the unmanned vehicle are also the driving parameters of the unmanned vehicle in the above-mentioned transition period, so that the target object
  • the driving route and the driving route of the unmanned vehicle are the routes during the above transition period. According to the respective driving routes of the target object and the unmanned vehicle and their respective initial positions in the key frame, it can be deduced that the target object is not recognized.
  • the driving parameters of the target object are determined according to the positions of the target object in multiple candidate frames, and then the target in the point cloud frame without the target object is deduced based on the driving parameters of the target object and the driving parameters of the unmanned vehicle.
  • the labeling of point cloud frames of unmanned vehicles is more autonomous for unmanned vehicles, and the entire labeling process is more convenient.
  • Embodiment 5 is a diagrammatic representation of Embodiment 5:
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the fourth embodiment, and thus will not be repeated.
  • step S143 identifying the target object at the predicted position to determine whether the target object is matched in the point cloud frame corresponding to the predicted position, including: obtaining the point cloud density of the predicted position; judging whether the point cloud density is greater than a predetermined threshold value ; In the case that the point cloud density is greater than the predetermined threshold, it is determined that the target object is matched in the point cloud frame corresponding to the predicted position.
  • the above-mentioned predicted position is a spatial range, and the point cloud density of the predicted position can be selected from the average density of point cloud data in the predicted position.
  • the above-mentioned predetermined threshold may be determined according to disturbances that cause noise data, and the disturbances such as larger particles of dust are exemplified. Larger particles of dust will cause noise data to appear in the point cloud frame collected by lidar, but the larger particles of dust that cause noise data tend to have a smaller density in space, and these dusts mostly appear in open places, so The noise data caused by the larger particles of dust has a smaller point cloud density. In this way, in the case that the interferer causing the noise data is the larger particles of dust, the above-mentioned predetermined threshold can be set according to the density of the larger particles of dust in the space. After the predetermined threshold is set, if the point cloud density is not greater than the predetermined threshold, it means that the point cloud at the predicted position is the point cloud data corresponding to the interference object, and the predicted position does not match the target object.
  • the echo of the laser beam will weaken in light intensity or deviate in the propagation path, so that the distant target corresponds to the point cloud data with a smaller density in the point cloud frame. However, this smaller density is still far greater than the noise caused by the data. The corresponding point cloud density of the distractor in the point cloud frame.
  • the marking is more accurate.
  • Embodiment 6 is a diagrammatic representation of Embodiment 6
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the above-mentioned first embodiment, and thus will not be repeated here.
  • the labeling method further includes outputting the key frame to receive the first verification result input after the manual verification of the key frame; wherein, if the manual verification determines that the key frame has an incorrect recognition result for the target, then The first verification result is a manually input replacement frame, and the labeling method further includes: updating the key frame to the replacement frame.
  • the manually input replacement frame is a point cloud frame among multiple point cloud frames, and the point cloud information of the target object in the replacement frame is relatively sufficient, and the replacement frame is used to replace the key frame determined in step S130, and the key frame determined in step S130 After the frame is updated to the replacement frame, the replacement frame is used as the key frame used in step S140.
  • the selection of the key frame is manually intervened through the first verification result, so that the selection of the key frame is more accurate, and then the labeling of the target object in the point cloud frame in which the target object is not identified in step S140 is more accurate.
  • Embodiment 7 is a diagrammatic representation of Embodiment 7:
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that of the above-mentioned first embodiment, and thus will not be repeated here.
  • the labeling method further includes: obtaining a dynamic model of the target based on the type of the target; using the dynamic model of the target and a set of cloud points marked as the target in a plurality of target point cloud frames, The orientation of the target object is adjusted for each target point cloud frame, so as to mark the orientation information of the target object in each target point cloud frame; wherein, the target point cloud frame includes: identifying the target in multiple point cloud frames The point cloud frame of the object, and the point cloud frame of the point cloud frame where the target object is not identified, and the point cloud frame matched to the target object through the target tracking.
  • the above-mentioned orientation information includes the position of the target object and/or the direction of the target object.
  • the above kinetic model is a kinetic model corresponding to the type of the target. If the types of the two targets are the same, the two targets can use the same kinetic model. If the types of the two targets are different, the two targets can use the same kinetic model. The kinetic model is also different.
  • the dynamic model Bicycle model gives the bicycle's state equation, that is, the expression equation of the bicycle's center of mass in two mutually perpendicular directions (scalar) and angular velocity (vector) in two-dimensional space.
  • the linear velocity (vector) of the center of mass of the vehicle is represented by the linear velocity (vector) of the center of mass of the vehicle, the angle between the longitudinal axis of the body and the front of the setting, and the distance from the center of mass of the vehicle to the front and rear wheels, so the linear velocity (vector) and angular velocity ( vector), the angle between the longitudinal axis of the body and the front of the setting and the distance from the center of mass of the vehicle to the front and rear wheels can be inferred.
  • the distance represents the space occupied by the bicycle.
  • Bicycle model is an ideal model.
  • the angular velocity of the center of mass of the vehicle and the speed in two mutually perpendicular directions in two-dimensional space are also slightly affected by other parameters, such as the slip angle that characterizes the angle between the linear velocity of the vehicle's center of mass and the longitudinal axis of the vehicle. That is, the Bicycle model cannot completely and accurately determine the orientation information of the target.
  • the orientation information of the target object is given by the distribution range of the point cloud data in the target point cloud frame meeting. Because some cloud points of the target object are missing in the point cloud frame, the orientation information provided by the point cloud frame is also inaccurate.
  • the orientation of the target object is adjusted for each target point cloud frame through the dynamic model of the target object and the set of cloud points marked as the target object in multiple target point cloud frames, so that the final determined orientation
  • the information has a high degree of fit with the dynamic model of the target and the target point cloud frame.
  • the finally determined orientation information needs to have a high degree of fit with the dynamic model of the target object and all target point cloud frames, so the determined orientation information is obtained after global tuning. As a result, the actual orientation of the target can be accurately represented.
  • Embodiment 8 is a diagrammatic representation of Embodiment 8
  • the method for labeling a point cloud provided in this embodiment basically adopts the same process as that in the seventh embodiment, and thus will not be repeated here.
  • the labeling method further includes: determining the size information of the target object through the dynamic model of the target object and a set of cloud points marked as the target object in a plurality of target point cloud frames, so as to display the size information of the target object in each target point cloud.
  • the size information of the target is annotated in the frame.
  • the size information of the target object can be inferred from the dynamic model of the target object and the driving parameters of the target object; the size information of the target object can also be inferred from the set of cloud points marked as the target object in the target point cloud frame.
  • the dynamic model of the target object is an ideal model, the size information of the inferred target object will deviate from the actual size of the target object; and some cloud points of the target object in the target point cloud frame will be missing, so the target point is passed through the target point.
  • the size information of the target object inferred from the cloud frame will also deviate from the actual size of the target object.
  • the size information of the target object is determined through the dynamic model of the target object and the cloud point sets marked as the target object in multiple target point cloud frames, that is, the determined size information of the target object is related to the dynamic model of the target object and more Each target point cloud frame has a high degree of fit.
  • the size information of the target object is determined through the dynamic model of the target object and the cloud point sets marked as the target object in multiple target point cloud frames, so that the determined size information of the target object is subject to the comprehensive constraints of multiple reference conditions , so it can accurately characterize the actual size of the target.
  • Embodiment 9 is a diagrammatic representation of Embodiment 9:
  • the method for labeling the point cloud provided in this embodiment basically adopts the same process as that of the eighth embodiment above, and thus will not be repeated here.
  • the labeling method further includes: outputting the target point cloud frame with the labeling result to receive the second checking result input after manually checking the labeling result; wherein, if the labeling result is determined to be incorrect after manual checking , the second verification result is a manually input replacement result, and the labeling method further includes: updating the labeling result to the replacement result.
  • the replacement result of manual input is the label information for accurately labeling the point cloud frame, and the replacement result of manual input may include label information such as type information, orientation information and size information.
  • the labeling of the point cloud frame is manually intervened through the second verification result, so that the labeling of the point cloud frame is more accurate.
  • the present application also discloses a point cloud labeling device, comprising: a memory configured to store computer instructions; a processor coupled to the memory, the processor configured to execute the computer instructions stored in the memory to implement any of the above embodiments The labeling method of the point cloud.
  • the labeling device is a device that executes the labeling method, and is configured to label multiple point cloud frames continuously collected during the driving of the unmanned vehicle.
  • the labeling information includes, for example, the type of the target object, the size of the target object, and the orientation of the target object, among which,
  • the types of objects include pedestrians, bicycles, and roadside transformer boxes, for example; the size of objects includes the length of bicycles; the orientation of objects includes the orientation of the objects and the position of the objects relative to the unmanned vehicle.
  • the marking device may include not only the above-mentioned memory and processor, but also one or more of the following components: a power supply component, an input/output interface, and a communication component.
  • the memory is also configured to store multiple types of data to support the operation of the labeling device, and the memory can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (Static Random Memory) Access Memory, SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic disk, etc.
  • the power supply components are configured to provide power to various components of the annotation device, and the power supply components may include a power management system, one or more power supplies, and other components associated with power generation, management, and distribution.
  • the input/output interface is configured to provide an interface for the connection between the marking device and peripheral modules, and the above-mentioned peripheral modules may be keyboards, mobile hard disks, and the like.
  • the communication component is configured to facilitate wired or wireless communication between the tagging device and other devices, and for example, the communication component includes a Near Field Communication (NFC) module to facilitate short-range communication.
  • NFC Near Field Communication
  • the processing components are generally configured to control the overall operation of the annotation device, such as operations associated with display, data communication, and computing and recording operations; the processing components may include one or more processors to execute the instructions.
  • FIG. 7 is a schematic diagram showing an optional structure of the labeling device.
  • the labeling device includes an acquisition unit 110, an identification unit 120, an extraction unit 130, and an annotation unit 140, wherein the acquisition unit 110 is configured to acquire a plurality of point cloud frames continuously collected during the driving of the unmanned vehicle; the identification unit 120 is configured by It is configured to perform target recognition on a plurality of point cloud frames respectively to obtain a recognition result; the extraction unit 130 is configured to extract key frames corresponding to the target objects from the plurality of point cloud frames according to the recognition results; the labeling unit 140 is configured to target the target The target matching in multiple point cloud frames is carried out through target tracking starting from the key frame, and the matched target object is marked in the point cloud frame where the target object is not identified among the multiple point cloud frames for target matching. .
  • the acquisition unit 110 included in the labeling device can be constructed through an input/output interface, so that the acquisition unit 110 can input the point cloud frame through a peripheral module (such as a keyboard) connected to the input/output interface; the acquisition unit 110 can also be constructed through a communication component, so that the acquisition unit 110 110 The point cloud frame is uploaded through the lidar connected by the communication component.
  • a peripheral module such as a keyboard
  • the labeling unit 140 included in the labeling device may include an input/output interface, so that the labeling unit 140 outputs the labelled point cloud frame through the input/output interface.
  • the identification unit 120 , the extraction unit 130 and part of the annotation unit 140 included in the labeling device can be constructed by a processor, and the processor implements all the functions of the identification unit 120 and the extraction unit 130 and some functions of the labeling unit 140 by executing instructions.
  • the acquiring unit 110, the identifying unit 120, the extracting unit 130 and the labeling unit 140 included in the labeling device are configured to execute the method for labeling the point cloud in the above-mentioned embodiment in combination. and the function of the labeling unit 140 will not be described again.
  • the labeling device may also include some other units to perform functions not performed by the acquisition unit 110 , the identification unit 120 , the extraction unit 130 , and the labeling unit 140 in the above-described embodiments.
  • the point cloud labeling device after determining the point cloud frame where the target is located as the key frame, performs target matching in multiple point cloud frames through target tracking, so that if one of the multiple point cloud frames If there is a target object in the point cloud frame where the target object is not recognized, it can match the target object and then mark the target object.
  • the point cloud frame of the unidentified target object is completed by combining the key frame to complete the labeling of the target object, and the difficulty of labeling is effectively reduced.
  • this labeling method can label the long-distance target by combining the key frame, so that the labeling area can be extended.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)

Abstract

一种点云的标注方法及标注设备，涉及无人驾驶、自动驾驶的技术领域。该标注方法包括：获取无人车行驶中连续采集的多个点云帧（S110）；对多个点云帧分别进行目标识别，得到识别结果（S120）；根据识别结果，从多个点云帧中提取目标物对应的关键帧，目标物对应的关键帧为识别出目标物的点云帧（S130）；针对目标物，通过从关键帧出发的目标追踪进行多个点云帧内的目标匹配，以在进行目标匹配的多个点云帧中的未识别出目标物的点云帧内标注匹配到的目标物（S140）。

Description

点云的标注方法及标注设备
本申请要求在2021年01月05日提交中国专利局、申请号为202110005211.4的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及无人驾驶、自动驾驶的技术领域，例如涉及一种点云的标注方法及标注设备。
背景技术
对于无人车来说，识别周围目标物是必不可少的功能，该功能是基于深度学习模型实现的。为了生成深度学习模型，需要关联有目标物信息的点云数据参与训练深度学习模型，因而，对点云数据进行目标物信息的标注具有重要意义。
无人车行驶中针对周围环境采集的一幅幅点云帧皆为黑白图像，点云帧中的背景为黑色而点云为白色，即点云帧是无颜色信息的图像，这对于本来需要耗费大量时间与人力成本的人工标注来说，无疑加大了标注难度。尤其是，一个点云帧内，不同位置处的目标物对应不同稀疏程度的点云数据，并且距离无人车较远的目标物对应较稀疏的点云数据，因而，距离无人车较远的目标物会因点云信息不足而难以被标注。
发明内容
本申请提供了一种点云的标注方法及标注设备，能够延长标注区域，降低标注难度。
提供一种点云的标注方法，包括：
获取无人车行驶中连续采集的多个点云帧；
对所述多个点云帧分别进行目标识别，得到识别结果；
根据所述识别结果，从所述多个点云帧中提取目标物对应的关键帧，所述目标物对应的关键帧为识别出所述目标物的点云帧；
针对所述目标物，通过从所述关键帧出发的目标追踪进行多个点云帧内的目标匹配，以在进行目标匹配的多个点云帧中的未识别出所述目标物的点云帧内标注匹配到的所述目标物。
还提供一种点云的标注设备，包括：
存储器，被配置为存储计算机指令；
处理器，耦合到所述存储器，所述处理器被配置为基于所述存储器存储的计算机指令执行实现上述的点云的标注方法。
附图说明
图1示出采集点云帧的无人车的示意图；
图2示出无人车在行驶中采集的环境中的一个点云帧；
图3示出本申请实施例一所提供的一种点云的标注方法的流程图；
图4示出本申请实施例二所提供的一种提取关键帧的方法的流程图；
图5示出本申请实施例三所提供的一种提取关键帧的方法的流程图；
图6示出本申请实施例四所提供的一种目标匹配的方法的流程图；
图7示出本申请实施例所提供的一种点云的标注设备的结构框图。
具体实施方式
为了便于理解本申请，下面将参照相关附图对本申请进行描述。附图中给出了本申请的实施例。但是，本申请可以通过不同的形式来实现，并不限于本文所描述的实施例。
图1所示为一辆自动行进的无人车，该自动行进过程得以平稳完成的重要原因在于无人车上安装的多种传感器，例如激光雷达。无人车通过激光雷达的探测获得周围环境的点云数据(点云数据可以是点云帧分割后的一部分)，然后从点云数据中确定目标物的位置以保证行进的平稳完成。
激光雷达探测周围环境对应的点云数据，原理如下：激光雷达包括探测单元和处理单元，其中，探测单元包括发射模块和接收模块，发射模块对周围发射多个激光束，一个激光束对应一个方向；接收模块接收激光束遇到目标物后返回的回波；处理单元是根据回波接收时刻与激光束发射时刻之间的间隔时长以及激光束的发射方向确定激光束的反射点位置，一个反射点位置即为图2所示点云帧中一个云点。一个点云帧是激光雷达对周围一圈的目标物进行探测后采集的数据。
无人车从上述点云数据中确定目标物的位置，是通过训练好的目标物识别模型从点云数据中识别出目标物，为了生成目标物识别模型，需要训练样本参与训练目标物识别模型。这里的训练样本为关联有目标物信息的提前采集的点 云数据；在点云数据被训练中的目标物识别模型识别出一个结果后，与点云数据关联的目标物信息用于确定识别出的结果是否正确以及是否需要继续训练目标物识别模型。因而，对点云数据进行目标物信息的标注具有重要意义。
然而，点云帧无颜色信息，对于本来需要耗费大量时间与人力成本的人工标注来说，无疑进一步加大了标注难度。尤其是，一个点云帧内，不同位置处的目标物对应不同稀疏程度的点云数据，并且距离无人车较远的目标物对应较稀疏的点云数据，因而，距离无人车较远的目标物会因点云信息不足而难以被标注。针对此，本申请提供了一种能够降低标准难度、延长标注区域的标注方法及标注设备。
以下先基于实施例对本申请提供的标注方法进行描述。
实施例一：
图3所示为点云的标注方法的流程图。参照图3，标注方法包括：
步骤S110，获取无人车行驶中连续采集的多个点云帧。
这里通过无人车上安装的激光雷达连续采集了多个点云帧，假设多个点云帧按采集时间的先后排序为P 1，P 2，…，P n，则多个点云帧P 1，P 2，…，P n中不存在目标物位置发生跳变的点云帧，也就是说，点云帧P i是点云帧P i-1与点云帧P i+1之间的过渡点云帧，目标物从点云帧P i-1所示的位置经过点云帧P i所示的位置而到达点云帧P i+1所示的位置。
激光雷达会根据预设规则进行点云帧的连续采集，例如通常在无人车行驶中间隔0.1s进行一次点云帧的采集，这样确保点云帧P i是点云帧P i-1与点云帧P i+1之间的过渡点云帧。
步骤S120，对多个点云帧分别进行目标识别，得到识别结果。
通过运行目标物识别模型(例如点云的三维检测模型Pointpillar)对多个点云帧顺次进行目标物的自动识别，从而快速得到多个关联有目标物信息的点云帧。这里所述点云帧和目标物信息的关联，可以是通过目标物信息和点云帧中目标物对应云点的位置信息建立关联列表而实现，一个关联列表对应一个点云帧；以及，一个关联列表包括一个关联数据或多个关联数据，一个关联数据对应一个目标物且为该目标物和一个云点位置信息或多个云点位置信息之间的关联。
一个关联列表包括一个关联数据的情况下，该关联列表的对应点云帧内识别出一个目标物；一个关联列表包括多个关联数据的情况下，该关联列表的对应点云帧内识别出多个目标物。目标物X j属于所有点云帧内识别出的目标物的一个，j的取值范围为0-m，m为所有点云帧内识别出的目标物数量，目标物X j 标识所有点云帧内的同一个目标物。
上述目标物识别模型通过以下方式预先训练：构造点云数据样本集合，所述集合中的点云数据样本包括无人车采集的点云数据且点云数据关联有预先识别出的目标物标签；将所述集合中的点云数据样本输入目标物识别模型，由目标物识别模型识别出点云数据样本中的目标物，并与关联的目标物标签进行比较；如果所述集合中识别出的目标物与目标物标签一致的样本比率超过预定比率阈值，则认为目标物识别模型训练成功；如果所述集合中识别出的目标物与目标物标签一致的样本比率不超过，则调整目标物识别模型的系数，使得所述集合中识别出的目标物与目标物标签一致的样本比率超过预定比率阈值。
由于颜色信息以及远距离目标物点云信息的缺乏，点云标注方法无法对点云数据中存在的远距离目标物进行标注。即，对于训练目标物识别模型的点云数据来说，会存在包含一远距离目标物但缺乏该目标物在远距离状态下的目标物标签的情况，这样上述通过运行目标物识别模型对多个点云帧顺次进行目标物自动识别的过程中，若点云帧P i中的目标物X j属于远距离目标物，则目标物识别模型无法识别出来该点云帧P i内的目标物X j。
步骤S130，根据识别结果，从多个点云帧中提取目标物对应的关键帧，其中，目标物对应的关键帧为识别出目标物的点云帧。
多个点云帧P 1，P 2，…，P n中不存在目标物位置发生跳变的点云帧，因而点云帧P i中目标物X j属于远距离目标物，点云帧P i之前或之后必有一个点云帧使得目标物X j属于近距离目标物且能够被识别，这里即选择目标物X j被识别的点云帧为关键帧。通常情况下，最后选取出的关键帧为点云密度较大的点云帧或者易识别目标物X j的点云帧，其中，关键帧的点云密度通常大于1000/m 3，也就是说，无人车所在空间内若一区域存在目标物则该区域在关键帧内对应的点云密度大于1000/m 3。
关键帧为一个与目标物对应的点云帧，同一个目标物只有一个关键帧，不同的目标物有可能对应不同的关键帧。若步骤S120中通过关联列表表示点云帧和目标物信息的关联，那么可以通过目标物的目标物信息所在的关联列表来确定关键帧。
步骤S140，针对目标物，通过从关键帧出发的目标追踪进行多个点云帧内的目标匹配，以在进行目标匹配的多个点云帧中的未识别出目标物的点云帧内标注匹配到的目标物。
上述未识别出目标物的点云帧包含在步骤S110采集的多个点云帧内，可以是一帧也可以是多帧。上述目标匹配可以采用目标追踪算法，例如有SORT算 法。
本实施例中，在将目标物所在的点云帧确定为关键帧后，通过目标追踪进行多个点云帧内的目标匹配，这样若该多个点云帧中的一个未识别出目标物的点云帧内存在目标物则能够匹配到目标物进而标注目标物。其中，未识别出目标物的点云帧通过结合关键帧完成了目标物的标注，标注难度有效降低。对于目标物在未识别出目标物的点云帧内属于远距离目标物的情况，该标注方法能够通过结合关键帧的方式对远距离目标物进行标注，使得标注区域得以延长。
实施例二：
本实施例所提供的点云的标注方法基本采用与上述实施例一相同的流程，因此不再赘述。
区别之处在于：参照图4，步骤S130，根据识别结果，从多个点云帧中提取目标物对应的关键帧，包括：
步骤S131a，根据识别结果，将多个点云帧中识别出目标物的每个点云帧确定为一个候选帧，得到多个候选帧。
步骤S132a，获取每个候选帧中目标物与无人车的间隔距离，得到多个候选帧对应的多个间隔距离。
步骤S133a，将多个间隔距离中最小值对应的候选帧，确定为关键帧。
上述间隔距离和候选帧之间是一一对应的关系，即每个候选帧对应一个间隔距离，若一候选帧对应的间隔距离是多个间隔距离中具有最小值的间隔距离，则该候选帧被确定为关键帧。
若步骤S120中通过关联列表表示点云帧和目标物信息的关联，那么有多个候选帧的情况下存在多个关联列表中包含目标物的目标物信息，该实施例提供了一种从多个关联列表对应的多个候选帧中选择一个作为关键帧的方法。
本实施例中，将多个间隔距离中最小值对应的候选帧确定为关键帧，因而关键帧内对目标物具有较为丰富的信息，这些信息利于在未识别出所述目标物的点云帧中准确框住属于所述目标物的所有云点。
实施例三：
本实施例所提供的点云的标注方法基本采用与上述实施例一相同的流程，因此不再赘述。
区别之处在于：参照图5，步骤S130，根据识别结果，从多个点云帧中提取目标物对应的关键帧，包括：
步骤S131a，根据识别结果，将多个点云帧中识别出目标物的每个点云帧确定为一个候选帧，得到多个候选帧。
步骤S132b，获取每个候选帧与同一未识别出目标物的点云帧在采集时间上的间隔时长，得到多个候选帧对应的多个间隔时长。
步骤S133b，将多个间隔时长中最小值对应的候选帧，确定为关键帧。
上述间隔时长和候选帧之间是一一对应的关系，即每个候选帧对应一个间隔时长，若一候选帧对应的间隔时长是多个间隔时长中具有最小值的间隔时长，则该候选帧被确定为关键帧。
若步骤S120中通过关联列表表示点云帧和目标物信息的关联，那么有多个候选帧的情况下存在多个关联列表中包含目标物的目标物信息，该实施例提供了另一种从多个关联列表对应的多个候选帧中选择一个作为关键帧的方法。
本实施例中，将多个间隔时长中最小值对应的候选帧确定为关键帧，因而能够通过较短时长的路线演变即可得到未识别出所述目标物的点云帧中所述目标物的预测位置。
实施例四：
本实施例所提供的点云的标注方法基本采用与上述实施例一相同的流程，因此不再赘述。
区别之处在于：参照图6，步骤S140，通过从关键帧出发的目标追踪进行多个点云帧的目标匹配，包括：
步骤S141，根据目标物在多个候选帧中的位置，确定目标物的行驶参数，其中，候选帧为多个点云帧中识别出目标物的点云帧。
步骤S142，基于目标物的行驶参数和无人车的行驶参数，从关键帧出发推演连续的多个点云帧中未识别出目标物的点云帧内目标物的预测位置。
步骤S143，在预测位置识别目标物，以确定是否在预测位置对应的点云帧内匹配到目标物。
目标物的行驶参数可以包括目标物的线速度、角速度，在得知目标物的线速度和角速度后则能够推知目标物的行驶路线；同样，无人车的行驶参数可以包括无人车的线速度、角速度，在得知无人车的线速度和角速度后则能够推知无人车的行驶路线。
上述多个候选帧为关键帧采集时刻与一未识别出目标物的点云帧的采集时刻之间的过渡时段对应的识别出目标物的点云帧，目标物的行驶参数为关键帧采集时刻与该未识别出目标物的点云帧的采集时刻之间的过渡时段内目标物的行驶参数，同样，无人车的行驶参数也为上述过渡时段内无人车的行驶参数，这样目标物的行驶路线以及无人车的行驶路线皆为上述过渡时段内行驶的路线，根据目标物和无人车各自的行驶路线以及各自在关键帧中的初始位置，则能够推演出未识别出目标物的点云帧内目标物的预测位置。
本实施例中，根据目标物在多个候选帧中的位置确定目标物的行驶参数，然后基于目标物的行驶参数和无人车的行驶参数来推演未识别出目标物的点云帧内目标物的预测位置，这样目标物的行驶参数无需依赖车联网等无人车以外的辅助***获取，即，预测位置的确定无需依赖车联网，只需及时获取无人车自身的行驶参数即可实现无人车的点云帧的标注，对于无人车来说其点云标注的自主性更强，整个标注过程更加便捷。
实施例五：
本实施例所提供的点云的标注方法基本采用与上述实施例四相同的流程，因此不再赘述。
区别之处在于：步骤S143，在预测位置识别目标物，以确定是否在预测位置对应的点云帧内匹配到目标物，包括：获取预测位置的点云密度；判断点云密度是否大于预定阈值；在点云密度大于预定阈值的情况下，确定在预测位置对应的点云帧内匹配到目标物。
由于目标物具有空间体积，因而上述预测位置是一个空间范围，预测位置的点云密度可以选择预测位置内点云数据的平均密度。
上述预定阈值可以是根据引起噪声数据的干扰物来确定，以较大颗粒灰尘这种干扰物进行示例性说明。较大颗粒的灰尘会使得激光雷达采集的点云帧内出现噪声数据，但达到引起噪声数据的较大颗粒灰尘往往在空间里具有较小的密度，并且这些灰尘多出现在空旷的位置，因而较大颗粒灰尘引起的噪声数据具有较小的点云密度，这样在引起噪声数据的干扰物为较大颗粒灰尘的情况下，可以根据空间内较大颗粒灰尘的密度设置上述预定阈值。预定阈值设定后，若点云密度不大于预定阈值，则说明预测位置的点云为干扰物对应的点云数据，预测位置没有匹配到目标物。
激光束的回波会在光强方面减弱或在传播路线上偏离，这样远距离目标物在点云帧内对应较小密度的点云数据，然而这个较小密度仍然远远大于引起噪 声数据的干扰物在点云帧内对应的点云密度。
本实施例中，在点云密度大于预定阈值的情况下才确定在预测位置对应的点云帧内匹配到目标物，这样有利于将噪声数据和远距离目标物的点云数据区分开，避免在噪声数据处标注远距离目标物对应的目标物信息，即使得标注免受干扰物的影响，标注更加准确。
实施例六：
本实施例所提供的点云的标注方法基本采用与上述实施例一相同的流程，因此不再赘述。
区别之处在于：所述标注方法还包括将关键帧输出以接收人工对关键帧进行核验后输入的第一核验结果；其中，若人工核验后确定关键帧对目标物的识别结果不正确，则第一核验结果为人工输入的替换帧，标注方法还包括：将关键帧更新为替换帧。
人工输入的替换帧为多个点云帧中的一个点云帧，且该替换帧内目标物的点云信息较为充足，该替换帧用于替换步骤S130确定的关键帧，步骤S130确定的关键帧更新为替换帧后替换帧作为步骤S140中使用的关键帧。
本实施例中，人工通过第一核验结果干预了关键帧的选择，使得关键帧的选择更精确，继而使得步骤S140中在未识别出目标物的点云帧内对目标物的标注更精确。
实施例七：
本实施例所提供的点云的标注方法基本采用与上述实施例一相同的流程，因此不再赘述。
区别之处在于：所述标注方法还包括：基于目标物的类型，获取目标物的动力学模型；通过目标物的动力学模型和多个目标点云帧内标注为目标物的云点集合，对每个目标点云帧进行目标物方位的调优，以在每个目标点云帧内对目标物进行方位信息的标注；其中，目标点云帧包括：多个点云帧中识别出目标物的点云帧，以及，未识别出目标物的点云帧中通过目标追踪匹配到目标物的点云帧。
上述方位信息包括目标物的位置和/或目标物的方向。上述动力学模型为与目标物类型对应的动力学模型，若两个目标物的类型相同则该两个目标物能够使用同一动力学模型，若两个目标物的类型不同则该两个目标物的动力学模型 也不同。
以目标物是自行车为例进行示例性说明。首先获取自行车的动力学模型Bicycle model，动力学模型Bicycle model给出了自行车的状态方程，即自行车质心在二维空间内两个相互垂直方向上的速度(标量)以及角速度(矢量)的表达方程，例如是通过车辆质心的线速度(矢量)、车身纵轴与设定正前方的夹角、车辆质心到前后轮的距离来表示，因而在得知自行车质心的线速度(矢量)和角速度(矢量)后，则能够推知车身纵轴与设定正前方的夹角以及车辆质心到前后轮的距离，其中，车身纵轴与设定正前方的夹角表示自行车的方向，车辆质心到前后轮的距离表示了自行车所占的空间位置。Bicycle model是一个理想的模型，车辆质心角速度以及二维空间内两个相互垂直方向上的速度还会受到其它参量的稍许影响，例如表征车辆质心线速度与车辆纵轴夹角的滑移角，即Bicycle model并不能完全精确地确定目标物的方位信息。
通过目标点云帧会中点云数据的分布范围给出目标物的方位信息。由于点云帧内会缺失目标物的一些云点，因而点云帧提供的方位信息也具有不精确性。
本实施例中，通过目标物的动力学模型和多个目标点云帧内标注为目标物的云点集合，对每个目标点云帧进行目标物方位的调优，是使得最后确定的方位信息与目标物的动力学模型以及目标点云帧都有较高的契合度。其中，目标点云帧为多个，最后确定的方位信息需要与目标物的动力学模型以及所有目标点云帧都有较高的契合度，这样确定的方位信息是经过全局调优后得到的结果，能够准确地表征目标物的实际方位。
实施例八：
本实施例所提供的点云的标注方法基本采用与上述实施例七相同的流程，因此不再赘述。
区别之处在于：所述标注方法还包括：通过目标物的动力学模型和多个目标点云帧内标注为目标物的云点集合，确定目标物的大小信息，以在每个目标点云帧内对目标物进行大小信息的标注。
通过目标物的动力学模型以及目标物的行驶参数可以推知目标物的大小信息；通过目标点云帧内标注为目标物的云点集合同样可以推知目标物的大小信息。但由于目标物的动力学模型是一个理想模型，因而推知的目标物的大小信息和目标物的实际大小会有偏差；而目标点云帧中目标物的部分云点会缺失，因而通过目标点云帧推知的目标物的大小信息也会与目标物的实际大小有偏差。而通过目标物的动力学模型和多个目标点云帧内标注为目标物的云点集合，确 定目标物的大小信息，即使得确定的目标物的大小信息与目标物的动力学模型以及多个目标点云帧都有较高的契合度。
本实施例中，通过目标物的动力学模型和多个目标点云帧内标注为目标物的云点集合确定目标物的大小信息，使得确定的目标物大小信息受多个参考条件的综合约束，因而能准确地表征目标物的实际大小。
实施例九：
本实施例所提供的点云的标注方法基本采用与上述实施例八相同的流程，因此不再赘述。
区别之处在于：所述标注方法还包括：将标注了标注结果的目标点云帧输出以接收人工对标注结果进行核验后输入的第二核验结果；其中，若人工核验后确定标注结果不正确，则第二核验结果为人工输入的替换结果，标注方法还包括：将标注结果更新为替换结果。
人工输入的替换结果为对点云帧进行准确标注的标注信息，人工输入的替换结果可以包括类型信息、方位信息和大小信息这些标注信息。
本实施例中，人工通过第二核验结果干预了点云帧的标注，使得点云帧的标注更精确。
本申请还公开了一种点云的标注设备，包括：存储器，被配置为存储计算机指令；处理器，耦合到存储器，处理器被配置为基于存储器存储的计算机指令执行实现如以上任一实施例所述点云的标注方法。
标注设备为执行标注方法的设备，被配置为对无人车行驶中连续采集的多个点云帧进行目标物标注，标注的信息例如有目标物类型、目标物大小、目标物方位，其中，目标物类型例如有行人、自行车、路旁的变电箱等；目标物大小例如有自行车的长；目标物方位包括目标物的朝向和目标物相对于无人车的位置。
标注设备可以不仅包括上述存储器和处理器，还包括以下一个或多个组件：电源组件、输入/输出接口以及通信组件。其中，存储器还被配置为存储多种类型的数据以支持标注设备的操作，存储器可以由任何类型的易失性或非易失性存储设备或者他们的组合实现，如静态随机存储存储器(Static Random Access Memory，SRAM)、电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)，磁盘等。电源组件被配置为对 标注设备的多种组件提供电力，电源组件可以包括电源管理***、一个或多个电源以及其它与电力生成、管理和分配相关联的组件。输入/输出接口被配置为对标注设备和***模块之间连接提供接口，上述***模块可以是键盘和移动硬盘等。通信组件被配置为便于标注设备和其它设备之间进行有线或无线通信，示例性地，通信组件包括近场通信(Near Field Communication，NFC)模块以促进短程通信。处理组件通常被配置为控制标注设备的整体操作，诸如与显示、数据通信以及运算和记录操作相关联的操作；处理组件可以包括一个或多个处理器来执行指令。
图7所示为标注设备的一种可选结构示意图。参照图7，标注设备包括获取单元110、识别单元120、提取单元130和标注单元140，其中，获取单元110被配置为获取无人车行驶中连续采集的多个点云帧；识别单元120被配置为对多个点云帧分别进行目标识别，得到识别结果；提取单元130被配置为根据识别结果，从多个点云帧中提取目标物对应的关键帧；标注单元140被配置为针对目标物，通过从关键帧出发的目标追踪进行多个点云帧内的目标匹配，以在进行目标匹配的多个点云帧中的未识别出目标物的点云帧内标注匹配到的目标物。
标注设备包括的获取单元110可以通过输入/输出接口搭建，这样获取单元110通过输入/输出接口连接的***模块(例如键盘)录入点云帧；获取单元110也可以通过通信组件搭建，这样获取单元110通过通信组件连接的激光雷达上传点云帧。
标注设备包括的标注单元140可以包括输入/输出接口，这样标注单元140通过输入/输出接口输出标注后的点云帧。
标注设备包括的识别单元120、提取单元130和部分标注单元140可以通过处理器搭建，处理器通过执行指令实现识别单元120和提取单元130的全部功能以及标注单元140的部分功能。
该标注设备包括的获取单元110、识别单元120、提取单元130和标注单元140相结合被配置为执行上述实施例中点云的标注方法，因而这里对获取单元110、识别单元120、提取单元130和标注单元140的功能不再进行描述。
标注设备还可以包括一些其他单元来执行上述实施例中没有被获取单元110、识别单元120、提取单元130和标注单元140执行的功能。
本实施例中，点云的标注设备在将目标物所在的点云帧确定为关键帧后，通过目标追踪进行多个点云帧内的目标匹配，这样若该多个点云帧中的一个未识别出目标物的点云帧内存在目标物则能够匹配到目标物进而标注目标物。其 中，未识别出目标物的点云帧通过结合关键帧完成了目标物的标注，标注难度有效降低。对于目标物在未识别出目标物的点云帧内属于远距离目标物的情况，该标注方法能够通过结合关键帧的方式对远距离目标物进行标注，使得标注区域得以延长。
附图中的流程图、框图图示了本申请实施例的***、方法、装置的可能的体系框架、功能和操作，流程图和框图上的方框可以代表一个模块、程序段或仅仅是一段代码，所述模块、程序段和代码都是用来实现规定逻辑功能的可执行指令。也应当注意，所述实现规定逻辑功能的可执行指令可以重新组合，从而生成新的模块和程序段。因此附图的方框以及方框顺序只是用来更好的图示实施例的过程和步骤，而不应以此作为对申请本身的限制。
在本文中，所含术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。

Claims (10)

  1. 一种点云的标注方法，包括：
    获取无人车行驶中连续采集的多个点云帧；
    对所述多个点云帧分别进行目标识别，得到识别结果；
    根据所述识别结果，从所述多个点云帧中提取目标物对应的关键帧，所述目标物对应的关键帧为识别出所述目标物的点云帧；
    针对所述目标物，通过从所述关键帧出发的目标追踪进行多个点云帧内的目标匹配，以在进行目标匹配的多个点云帧中的未识别出所述目标物的点云帧内标注匹配到的所述目标物。
  2. 根据权利要求1所述的标注方法，其中，所述根据所述识别结果，从所述多个点云帧中提取目标物对应的关键帧，包括：
    根据所述识别结果，将所述多个点云帧中识别出所述目标物的每个点云帧确定为一个候选帧，得到多个候选帧；
    获取每个候选帧中所述目标物与所述无人车的间隔距离，得到所述多个候选帧对应的多个间隔距离；
    将所述多个间隔距离中的最小值对应的候选帧，确定为所述关键帧。
  3. 根据权利要求1所述的标注方法，其中，所述根据所述识别结果，从所述多个点云帧中提取目标物对应的关键帧，包括：
    根据所述识别结果，将所述多个点云帧中识别出所述目标物的每个点云帧确定为一个候选帧，得到多个候选帧；
    获取每个候选帧与同一未识别出所述目标物的点云帧在采集时间上的间隔时长，得到所述多个候选帧对应的多个间隔时长；
    将所述多个间隔时长中最小值对应的候选帧，确定为所述关键帧。
  4. 根据权利要求1所述的标注方法，其中，所述通过从所述关键帧出发的目标追踪进行多个点云帧的目标匹配，包括：
    根据所述目标物在多个候选帧中的位置，确定所述目标物的行驶参数，其中，所述候选帧为所述多个点云帧中识别出所述目标物的点云帧；
    基于所述目标物的行驶参数和所述无人车的行驶参数，从所述关键帧出发推演连续的多个点云帧中未识别出所述目标物的点云帧内所述目标物的预测位置；
    在所述预测位置识别所述目标物，以确定是否在所述预测位置对应的点云帧内匹配到所述目标物。
  5. 根据权利要求4所述的标注方法，其中，在所述预测位置识别所述目标物，以确定是否在所述预测位置对应的点云帧内匹配到所述目标物，包括：
    获取所述预测位置的点云密度；
    判断所述点云密度是否大于预定阈值；
    响应于所述点云密度大于预定阈值，确定在所述预测位置对应的点云帧内匹配到所述目标物。
  6. 根据权利要求1所述的标注方法，还包括：
    将所述关键帧输出以接收人工对所述关键帧进行核验后输入的第一核验结果；
    其中，在人工核验后由人工确定所述关键帧对所述目标物的识别结果不正确的情况下，所述第一核验结果为人工输入的替换帧；
    所述标注方法还包括：将所述关键帧更新为所述替换帧。
  7. 根据权利要求1所述的标注方法，还包括：
    基于所述目标物的类型，获取所述目标物的动力学模型；
    通过所述目标物的动力学模型和多个目标点云帧内标注为所述目标物的云点集合，对每个目标点云帧进行所述目标物方位的调优，以在所述每个目标点云帧内对所述目标物进行方位信息的标注；
    其中，所述目标点云帧包括：所述多个点云帧中识别出所述目标物的点云帧，以及，所述多个点云帧中未识别出所述目标物的点云帧中通过目标追踪匹配到所述目标物的点云帧。
  8. 根据权利要求7所述的标注方法，还包括：
    通过所述目标物的动力学模型和所述多个目标点云帧内标注为所述目标物的云点集合，确定所述目标物的大小信息，以在每个目标点云帧内对所述目标物进行大小信息的标注。
  9. 根据权利要求8所述的标注方法，还包括：
    将标注了标注结果的目标点云帧输出以接收人工对所述标注结果进行核验后输入的第二核验结果；
    其中，在人工核验后由人工确定所述标注结果不正确的情况下，所述第二核验结果为人工输入的替换结果；
    所述标注方法还包括：将所述标注结果更新为所述替换结果。
  10. 一种点云的标注设备，包括：
    存储器，被配置为存储计算机指令；
    处理器，耦合到所述存储器，所述处理器被配置为基于所述存储器存储的计算机指令执行实现如权利要求1-9中任一项所述点云的标注方法。
PCT/CN2021/099073 2021-01-05 2021-06-09 点云的标注方法及标注设备 WO2022147960A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110005211.4A CN112329749B (zh) 2021-01-05 2021-01-05 点云的标注方法及标注设备
CN202110005211.4 2021-01-05

Publications (1)

Publication Number Publication Date
WO2022147960A1 true WO2022147960A1 (zh) 2022-07-14

Family

ID=74302173

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/099073 WO2022147960A1 (zh) 2021-01-05 2021-06-09 点云的标注方法及标注设备

Country Status (2)

Country Link
CN (1) CN112329749B (zh)
WO (1) WO2022147960A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116012624A (zh) * 2023-01-12 2023-04-25 阿波罗智联(北京)科技有限公司 定位方法、装置、电子设备、介质以及自动驾驶设备
CN116363557A (zh) * 2023-03-17 2023-06-30 杭州再启信息科技有限公司 一种用于连续帧的自学习标注方法、***及介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112329749B (zh) * 2021-01-05 2021-04-27 新石器慧通（北京）科技有限公司 点云的标注方法及标注设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190156507A1 (en) * 2016-10-10 2019-05-23 Tencent Technology (Shenzhen) Company Limited Method and apparatus for processing point cloud data and storage medium
CN111191600A (zh) * 2019-12-30 2020-05-22 深圳元戎启行科技有限公司 障碍物检测方法、装置、计算机设备和存储介质
CN111553302A (zh) * 2020-05-08 2020-08-18 深圳前海微众银行股份有限公司 关键帧选取方法、装置、设备及计算机可读存储介质
CN111626217A (zh) * 2020-05-28 2020-09-04 宁波博登智能科技有限责任公司 一种基于二维图片和三维点云融合的目标检测和追踪方法
CN112329749A (zh) * 2021-01-05 2021-02-05 新石器慧通（北京）科技有限公司 点云的标注方法及标注设备

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180136332A1 (en) * 2016-11-15 2018-05-17 Wheego Electric Cars, Inc. Method and system to annotate objects and determine distances to objects in an image
CN107239794B (zh) * 2017-05-18 2020-04-28 深圳市速腾聚创科技有限公司 点云数据分割方法和终端
CN109521756B (zh) * 2017-09-18 2022-03-08 阿波罗智能技术(北京)有限公司 用于无人驾驶车辆的障碍物运动信息生成方法和装置
CN108446585B (zh) * 2018-01-31 2020-10-30 深圳市阿西莫夫科技有限公司 目标跟踪方法、装置、计算机设备和存储介质
CN108917761B (zh) * 2018-05-07 2021-01-19 西安交通大学 一种无人车在地下车库中的精确定位方法
WO2020154968A1 (en) * 2019-01-30 2020-08-06 Baidu.Com Times Technology (Beijing) Co., Ltd. A point clouds ghosting effects detection system for autonomous driving vehicles
CN110221603B (zh) * 2019-05-13 2020-08-14 浙江大学 一种基于激光雷达多帧点云融合的远距离障碍物检测方法
CN110647658A (zh) * 2019-08-02 2020-01-03 惠州市德赛西威汽车电子股份有限公司 一种基于云计算的车载图像特征自动识别方法与***
CN110927742A (zh) * 2019-11-19 2020-03-27 杭州飞步科技有限公司 障碍物跟踪方法、装置、设备及存储介质
CN111311684B (zh) * 2020-04-01 2021-02-05 亮风台（上海）信息科技有限公司 一种进行slam初始化的方法与设备
CN111476822B (zh) * 2020-04-08 2023-04-18 浙江大学 一种基于场景流的激光雷达目标检测与运动跟踪方法
CN111583337B (zh) * 2020-04-25 2023-03-21 华南理工大学 一种基于多传感器融合的全方位障碍物检测方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190156507A1 (en) * 2016-10-10 2019-05-23 Tencent Technology (Shenzhen) Company Limited Method and apparatus for processing point cloud data and storage medium
CN111191600A (zh) * 2019-12-30 2020-05-22 深圳元戎启行科技有限公司 障碍物检测方法、装置、计算机设备和存储介质
CN111553302A (zh) * 2020-05-08 2020-08-18 深圳前海微众银行股份有限公司 关键帧选取方法、装置、设备及计算机可读存储介质
CN111626217A (zh) * 2020-05-28 2020-09-04 宁波博登智能科技有限责任公司 一种基于二维图片和三维点云融合的目标检测和追踪方法
CN112329749A (zh) * 2021-01-05 2021-02-05 新石器慧通（北京）科技有限公司 点云的标注方法及标注设备

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116012624A (zh) * 2023-01-12 2023-04-25 阿波罗智联(北京)科技有限公司 定位方法、装置、电子设备、介质以及自动驾驶设备
CN116012624B (zh) * 2023-01-12 2024-03-26 阿波罗智联(北京)科技有限公司 定位方法、装置、电子设备、介质以及自动驾驶设备
CN116363557A (zh) * 2023-03-17 2023-06-30 杭州再启信息科技有限公司 一种用于连续帧的自学习标注方法、***及介质
CN116363557B (zh) * 2023-03-17 2023-09-19 杭州再启信息科技有限公司 一种用于连续帧的自学习标注方法、***及介质

Also Published As

Publication number Publication date
CN112329749B (zh) 2021-04-27
CN112329749A (zh) 2021-02-05

Similar Documents

Publication Publication Date Title
WO2022147960A1 (zh) 点云的标注方法及标注设备
US11681746B2 (en) Structured prediction crosswalk generation
US11885910B2 (en) Hybrid-view LIDAR-based object detection
US20190310651A1 (en) Object Detection and Determination of Motion Information Using Curve-Fitting in Autonomous Vehicle Applications
EP3639241B1 (en) Voxel based ground plane estimation and object segmentation
CN110675307B (zh) 基于vslam的3d稀疏点云到2d栅格图的实现方法
US20180349746A1 (en) Top-View Lidar-Based Object Detection
US20180348374A1 (en) Range-View Lidar-Based Object Detection
CN111291697B (zh) 用于识别障碍物的方法和装置
CN108764187A (zh) 提取车道线的方法、装置、设备、存储介质以及采集实体
CN112949366B (zh) 障碍物识别方法和装置
US11967103B2 (en) Multi-modal 3-D pose estimation
CN106845496B (zh) 精细目标识别方法和***
CN112905849A (zh) 一种车辆数据处理的方法及装置
CN111295666A (zh) 一种车道线检测方法、装置、控制设备及存储介质
CN114511077A (zh) 使用基于伪元素的数据扩增来训练点云处理神经网络
CN111126327B (zh) 车道线检测方法、***、车载***及车辆
CN113835102A (zh) 车道线生成方法和装置
Carson et al. Predicting to improve: Integrity measures for assessing visual localization performance
Kim et al. Semantic point cloud-based adaptive multiple object detection and tracking for autonomous vehicles
CN113469045B (zh) 无人集卡的视觉定位方法、***、电子设备和存储介质
CN112395956A (zh) 一种面向复杂环境的可通行区域检测方法及***
CN116164758B (zh) 高精度点云地图的更新方法、装置、介质、设备及***
US20230394694A1 (en) Methods and apparatus for depth estimation using stereo cameras in a vehicle system
CN108944945A (zh) 用于辅助驾驶的状态预测方法、装置、电子设备和车辆

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21917009

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21917009

Country of ref document: EP

Kind code of ref document: A1