CN116359908A - Point cloud data enhancement method, device, computer equipment, system and storage medium - Google Patents

Point cloud data enhancement method, device, computer equipment, system and storage medium Download PDF

Info

Publication number
CN116359908A
CN116359908A CN202111615244.7A CN202111615244A CN116359908A CN 116359908 A CN116359908 A CN 116359908A CN 202111615244 A CN202111615244 A CN 202111615244A CN 116359908 A CN116359908 A CN 116359908A
Authority
CN
China
Prior art keywords
point cloud
millimeter wave
cloud data
data
radar
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111615244.7A
Other languages
Chinese (zh)
Inventor
杨炎龙
李娟娟
孟凡志
邓永强
吴雷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Wanji Technology Co Ltd
Original Assignee
Beijing Wanji Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Wanji Technology Co Ltd filed Critical Beijing Wanji Technology Co Ltd
Priority to CN202111615244.7A priority Critical patent/CN116359908A/en
Priority to PCT/CN2022/136303 priority patent/WO2023124780A1/en
Publication of CN116359908A publication Critical patent/CN116359908A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/865—Combination of radar systems with lidar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/50—Systems of measurement based on relative movement of target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/50—Systems of measurement based on relative movement of target
    • G01S13/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/50—Systems of measurement based on relative movement of target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/50—Systems of measurement based on relative movement of target
    • G01S17/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/10—Image enhancement or restoration using non-spatial domain filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10044—Radar image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Radar Systems Or Details Thereof (AREA)

Abstract

The application is applicable to the technical field of data processing and provides a point cloud data enhancement method, a point cloud data enhancement device, computer equipment, a point cloud data enhancement system and a point cloud data storage medium. The point cloud data enhancement method comprises the following steps: acquiring point cloud data of a laser radar and millimeter wave signals of the millimeter wave radar; acquiring three-dimensional cube data according to the millimeter wave signals; the three-dimensional cube data comprises information of distance, speed and angle dimensions; acquiring a hematmap distribution diagram according to the three-dimensional cubic data; extracting targets from the Heatm ap distribution map, and determining the targets in the Heatm ap distribution map; determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap profile based on the target; and carrying out data enhancement on the target point cloud set. The method and the device can solve the problem that the point cloud data of the laser radar are sparse, and the detection accuracy of the target is affected.

Description

Point cloud data enhancement method, device, computer equipment, system and storage medium
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a point cloud data enhancement method, a point cloud data enhancement device, computer equipment, a point cloud data enhancement system and a point cloud data storage medium.
Background
The lidar is a radar system that detects a characteristic quantity such as a position, a speed, etc. of a target by emitting a laser beam. The laser radar is mainly based on point cloud sensing at present, and can detect targets such as vehicles and pedestrians on roads and detect information such as positions and speeds of the targets in a radar self coordinate system. However, the point cloud data of the laser radar is sparse, and the detection accuracy of the target is affected.
Disclosure of Invention
The embodiment of the application provides a point cloud data enhancement method, a point cloud data enhancement device, computer equipment, a point cloud data enhancement system and a point cloud data enhancement storage medium, and aims to solve the problem that point cloud data of a laser radar are sparse and detection accuracy of a target is affected.
In a first aspect, an embodiment of the present application provides a point cloud data enhancement method, where the point cloud data enhancement method includes:
acquiring point cloud data of a laser radar and millimeter wave signals of the millimeter wave radar, wherein the point cloud data and the millimeter wave signals are obtained by respectively carrying out data sampling on the same area by the laser radar and the millimeter wave radar;
acquiring three-dimensional cube data according to the millimeter wave signals; the three-dimensional cube data comprises information of distance, speed and angle dimensions;
Acquiring a hematmap distribution diagram according to the three-dimensional cubic data; the hematmap is an information map of the distance and angle dimensions, and the information map comprises the energy of points corresponding to the distance and the angle;
extracting targets from the Heatm ap distribution map, and determining the targets in the Heatm ap distribution map;
determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap profile based on the target;
and carrying out data enhancement on the target point cloud set.
In a second aspect, an embodiment of the present application provides a point cloud data enhancement apparatus, where the point cloud data enhancement apparatus includes:
the system comprises a signal acquisition module, a data acquisition module and a data acquisition module, wherein the signal acquisition module is used for acquiring point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar, and the point cloud data and the millimeter wave signals are obtained by respectively carrying out data sampling on the same area through the laser radar and the millimeter wave radar;
the data acquisition module is used for acquiring three-dimensional data according to the millimeter wave signals; the three-dimensional data comprises information of distance, speed and angle dimensions;
the distribution map acquisition module is used for acquiring a hematmap distribution map according to the three-dimensional stereo data; the hematmap is an information map of the distance and angle dimensions, and the information map comprises the energy of points corresponding to the distance and the angle;
The target extraction module is used for extracting targets from the Heatm ap distribution map and determining the targets in the Heatm ap distribution map;
the set determining module is used for determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap distribution diagram based on the target;
and the data enhancement module is used for enhancing the data of the target point cloud set.
In a third aspect, embodiments of the present application provide a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to the first aspect described above when the computer program is executed.
In a fourth aspect, embodiments of the present application provide a multi-sensor system comprising a lidar, a millimeter-wave radar, and a computer device as described in the third aspect above;
the laser radar and the millimeter wave radar acquire data of the same area;
the laser radar is used for outputting the collected point cloud data to the computer equipment;
the millimeter wave radar is used for outputting the collected millimeter wave signals to the computer equipment.
In a fifth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method according to the first aspect described above.
In a sixth aspect, embodiments of the present application provide a computer program product for, when run on a computer device, causing the computer device to perform the steps of the method as described in the first aspect above.
From the above, according to the method, the point cloud data and the millimeter wave signals obtained by sampling the data of the same area by the laser radar and the millimeter wave radar can be obtained firstly according to the millimeter wave signals, then the hetmap distribution diagram is obtained according to the three-dimensional data, the target in the hetmap distribution diagram is extracted, so that the target in the hetmap distribution diagram is determined, the corresponding target point cloud set in the point cloud data spatially synchronous with the hetmap distribution diagram can be determined based on the target in the hetmap distribution diagram, and the density of the point cloud data (especially the point cloud data of the far end of the laser radar) of the laser radar can be increased by carrying out data enhancement on the target point cloud set, so that the detection accuracy of the laser radar on the target is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required for the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a multi-sensor system according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an implementation of a point cloud data enhancement method according to an embodiment of the present application;
fig. 3 is a flowchart illustrating a processing flow of the millimeter wave signal;
FIG. 4 is an example diagram of a vehicle traveling on a roadway;
FIG. 5 is an exemplary diagram of a hematmap profile;
fig. 6 is a schematic flowchart of an implementation of a point cloud data enhancement method according to another embodiment of the present application;
fig. 7 is a schematic structural diagram of a point cloud data enhancement device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
It should be understood that the sequence number of each step in this embodiment does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
In order to illustrate the technical solutions described in the present application, the following description is made by specific examples.
Referring to fig. 1, which is a schematic diagram of the architecture of a multi-sensor system according to an embodiment of the present application, for convenience of explanation, only a portion related to the embodiment of the present application is shown.
The above-described multi-sensor system includes a laser radar 11, a millimeter wave radar 12, and a computer device 13.
The type of the lidar 11 is not limited in this application. For example, the lidar 11 may be an 8-wire, 16-wire, 24-wire, 32-wire, 64-wire, 128-wire radar, or the like.
The present application does not limit the type of millimeter wave radar 12. For example, millimeter-wave radar 12 may be a 77G, 24G fundamental radar, or the like.
In the application scenario of the multi-sensor system, in order to realize that at least partially overlapped sensing ranges exist in the laser radar 11 and the millimeter wave radar 12, the installation of the laser radar 11 and the millimeter wave radar 12 needs to consider the field matching problem of the two. For example, millimeter-wave radar 12 may employ an area array antenna array that covers a field of view that is 180 ° forward or less. The lidar 11 has a limited field of view and a 360 ° look around field of view. During forward scanning, the laser radar 11 with limited field of view can complete scanning imaging of the forward field of view by matching with the millimeter wave radar 12 adopting an area array. The laser radar 11 with 360-degree looking around view field can be matched with 3 to 4 millimeter wave radars 12 with area arrays according to actual requirements, and the view fields of the millimeter wave radars 12 can be partially overlapped. Of course, the laser radar 11 with 360 ° looking around view may be matched with a millimeter wave radar 12 with 360 ° looking around view, which is not limited herein.
The laser radar 11 and the millimeter wave radar 12 sample data of the same area in the sensing range, point cloud data of the laser radar 11 and millimeter wave signals of the millimeter wave radar 12 can be obtained respectively, the laser radar 11 can send the point cloud data to the computer equipment 13, the millimeter wave radar 12 can send the millimeter wave signals to the computer equipment 13, after the computer equipment 13 receives the point cloud data and the millimeter wave signals, three-dimensional stereo data can be acquired firstly according to the millimeter wave signals, then a hetmap distribution diagram is acquired according to the three-dimensional stereo data, and targets in the hetmap distribution diagram are extracted, so that targets in the hetmap distribution diagram are determined, a corresponding target point cloud set in the point cloud data in spatial synchronization with the hetmap can be determined based on the targets in the hetmap distribution diagram, and the density of the point cloud data (particularly the point cloud data of the far end of the laser radar) of the laser radar 11 can be increased by carrying out data enhancement on the target point cloud set, so that the detection accuracy of the laser radar 11 on the targets is improved.
The millimeter wave radar 12 is used for sensing environment by electromagnetic waves, has a long detection distance and is suitable for moving targets. The lidar 11 is based mainly on point cloud perception, and the stereoscopic impression and the target class are easier to distinguish. The point cloud data of the laser radar 11 is sparse at the far end, so that the detection accuracy of the target is affected, and the detection distance of the millimeter wave radar 12 is larger than that of the laser radar 11, so that the laser radar 11 is assisted at the far end by utilizing the detection capability of the millimeter wave radar 12, the density of the far-end point cloud data of the laser radar 11 can be increased, and the detection accuracy of the laser radar 11 to the target is improved.
It should be noted that, the application scenario of the multi-sensor system is not limited in this application. For example, the application scenario may be an automatic driving scenario of a vehicle, the laser radar 11 is a vehicle-mounted laser radar 11, the millimeter wave radar 12 is a vehicle-mounted millimeter wave radar, and the computer device 13 may be a vehicle-mounted terminal. The application scene can also be an auxiliary driving scene of the vehicle, and after the targets are detected, the information such as the size, the position, the movement speed, the course angle and the like of the detected targets can be displayed on a high-precision map so as to better assist a driver to safely drive the vehicle.
Referring to fig. 2, a flowchart of an implementation of a point cloud data enhancement method according to an embodiment of the present application is shown, where the point cloud data enhancement method is applied to a computer device in the multi-sensor system shown in fig. 1. As shown in fig. 2, the point cloud data enhancement method may include the steps of:
step 201, acquiring point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar.
The point cloud data and the millimeter wave signals are obtained by respectively carrying out data sampling on the same area through a laser radar and a millimeter wave radar.
The same region may be any region within the sensing range of the lidar and the millimeter wave radar, and is not limited herein. For example, when the lidar is an in-vehicle lidar and the millimeter wave radar is an in-vehicle millimeter wave radar, the same area may be a forward travel area of the vehicle.
Taking 24G baseband radar as an example, millimeter wave radar adopts frequency modulation continuous wave (Frequency Modulated Continuous Wave, FMCW) to perform space measurement on the same area, sends and receives FMCW signals with the baseband frequency of 24GHz from a radio frequency antenna (including a sending antenna and a receiving antenna), performs AD sampling on echoes (i.e. received continuous modulated electromagnetic waves) received by the receiving antenna, stores the echoes after AD sampling in a register, outputs from a network interface after completing the reception of a frame of millimeter wave signals, and outputs millimeter wave signals to a computer device.
After the laser radar scans the same area to obtain the point cloud data, noise filtering can be performed on the point cloud data to reduce noise interference and improve the detection accuracy of the target.
And 202, acquiring three-dimensional cube data according to the millimeter wave signals.
The three-dimensional cube data comprises information of distance, angle and speed dimensions, namely the three-dimensional cube data comprises information of three dimensions of distance, angle and speed.
In one embodiment, the millimeter wave signal may be subjected to fourier transform in distance, speed and angle dimensions to obtain three-dimensional cubic data, that is, three-dimensional cubic data is obtained by performing three-dimensional fast fourier transform (Three Dimensional Fast Fourier Transform,3 DFFT) processing on the millimeter wave signal; the millimeter wave signal may also be processed by algorithms such as FFT and Beam Forming (BF), capon, MUSIC, etc. to obtain three-dimensional stereo data, which is not limited herein. Among them, 3DFFT includes a distance-dimensional fast fourier transform (range FFT), a velocity-dimensional fast fourier transform (doppler FFT), and an angle-dimensional fast fourier transform (angle FFT).
An exemplary diagram of the processing flow of millimeter wave signals is shown in fig. 3. Millimeter wave radar detects targets in the sensing range by sending and receiving FMCW signals. The FMCW signal transmitted by the transmitting antenna of the millimeter wave radar is divided into more chirps (only 4 chirps are shown in fig. 3), the AD sampling result of the intermediate frequency (Intermediate Frequency, IF) signal obtained by each returned chirp is subjected to FFT processing, and the position of the target can be calculated according to the obtained peak frequency, and the process is called range FFT; based on the doppler shift caused by the motion speed of the target, performing FFT (inter-chip FFT) on the results of all chips in a single antenna after range FFT, namely, doppler FFT, and calculating the speed of the target according to the obtained peak frequency (doppler FFT result); based on the phase difference between different receiving antennas caused by the angle of the target, FFT (angular FFT) between different antenna signals is performed, and the angle of the target can be calculated according to the obtained peak frequency (angular FFT result). The above procedure can be considered as one 3DFFT, depending on the separability of the fourier transform. The angle of the target may be calculated by an algorithm such as BF, capon, MUSIC.
And 203, acquiring a hematmap distribution diagram according to the three-dimensional cubic data.
Wherein the hetmap is an information map of the distance and angle dimensions, the information map comprising energies of points corresponding to the distance and angle. The energy of the point corresponding to the distance and the angle can be expressed using a Signal-to-Noise Ratio (SNR) of the point corresponding to the distance and the angle.
The three-dimensional stereo data comprises information of three dimensions such as distance, speed and angle, and the information of the two dimensions such as the distance and the angle can be extracted from the information of the three dimensions, so that a hectmap distribution diagram is obtained.
As shown in fig. 4, an example diagram of a vehicle traveling on a road is shown, a laser radar (i.e., an on-vehicle laser radar) and a millimeter wave radar (i.e., an on-vehicle millimeter wave radar) are mounted on a vehicle a, a target 1, a target 2 and a target 3 are present in a front traveling area of the vehicle a, the laser radar and the millimeter wave radar on the vehicle a perform data acquisition on the front traveling area, point cloud data and millimeter wave signals can be obtained, 3DFFT processing is performed on the millimeter wave signals, three-dimensional stereo data can be obtained, and an example diagram of a hetmap map as shown in fig. 5 can be obtained from the three-dimensional stereo data, and as can be seen from fig. 5, the hetmap contains signal to noise ratios of points corresponding to distances and angles, and the points with relatively high signal to noise ratio in the hetmap are generally the targets. In fig. 4, X and Y are two axes of a rectangular planar coordinate system established with the vehicle a as the origin, 46.7dB is the signal-to-noise ratio of the target 2, 51.6dB is the signal-to-noise ratio of the target 1, and 37.7dB is the signal-to-noise ratio of the target 3.
And 204, extracting targets from the Heatm ap distribution map, and determining the targets in the Heatm ap distribution map.
When the millimeter wave radar transmits the FMCW signal to the same area, the target in the same area will reflect the FMCW signal, thereby generating higher energy. The target in the same region can be extracted from the hetmap profile based on the energy in the hetmap profile.
As an alternative embodiment, the energy peaks may be targeted in the hematmap profile.
Because the energy of the target is high, all energy peaks in the hetmap can be targeted. For example, in the hetmap profile of fig. 5, there are four energy peaks, which can be targeted, i.e., four targets in the hetmap profile.
Step 205, determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap profile based on the target.
Wherein the set of target points cloud may be a set of point clouds that constitute a target in the hetmap profile.
Under the condition that the spatial synchronization of the Heatm ap distribution diagram and the point cloud data is completed, a target point cloud set corresponding to the target in the point cloud data can be obtained based on the target in the Heatm ap distribution diagram.
As an alternative embodiment, determining a corresponding set of target point clouds in the point cloud data spatially synchronized with the hetmap profile based on the targets, comprises:
under the condition that the space synchronization of the point cloud data and the hetmap distribution diagram is completed, judging whether at least two target point clouds in the point cloud data exist in the 3sigma distribution of the target;
if at least two target point clouds in the point cloud data exist in the 3sigma distribution of the target, determining that the at least two target point clouds form a target point cloud set.
As can be seen from fig. 5, the target in the hematmap distribution diagram can be described by using a 3sigma distribution, so that in the case that the spatial synchronization of the point cloud data and the hematmap distribution diagram is completed, by determining whether at least two target point clouds in the point cloud data exist in the 3sigma distribution of the target, a corresponding target point cloud set can be obtained.
Since a target is usually composed of a large number of target point clouds, whether the target actually exists can be detected by setting a number of thresholds. Specifically, when there is a target point cloud in the point cloud data in the 3sigma distribution of the target, the number of the target point clouds may be counted, if the number of the target point clouds is greater than the number threshold, it may be determined that the target is actually present, and the subsequent steps may be continuously performed to perform data enhancement on the target point cloud set of the target; if the number of target point clouds is less than or equal to the number threshold, it may be determined that the target is not present (i.e., the target is a false target) without performing subsequent steps to reduce the occupancy of computing resources of the computer device by the target.
As an alternative embodiment, markers are arranged in the perception range of the laser radar and the millimeter wave radar; further comprises:
acquiring position information of a marker in a laser radar coordinate system and position information of a millimeter wave radar coordinate system respectively;
acquiring the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system according to the position information of the marker in the laser radar coordinate system and the position information in the millimeter wave radar coordinate system respectively;
and according to the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system, performing spatial synchronization on the point cloud data and the millimeter wave signals so as to complete the spatial synchronization of the point cloud data and the hetmap distribution diagram.
The marker may be a metal marker having a strong reflection echo characteristic, such as a corner reflector.
In order to reduce the influence of other targets with strong reflection echo characteristics on the spatial calibration of millimeter wave radars and laser radars, in the environment of the spatial calibration of millimeter wave radars and laser radars, other FMCW signals in other positions in the environment except for the markers cannot generate echoes once absorbed. In the lidar, the marker is white, and the other positions are made of light-absorbing black materials.
The lidar coordinate system may refer to a coordinate system established with the lidar as the origin. The millimeter wave radar coordinate system may refer to a coordinate system established with the millimeter wave radar as an origin.
The position information of the marker in the lidar coordinate system includes a distance of the marker from the lidar and an angle of the marker in a perspective of the lidar.
The position information of the marker in the millimeter wave radar coordinate system includes a distance of the marker from the millimeter wave radar and an angle of the marker in a view angle of the millimeter wave radar.
According to the method, the marker is used as a datum point, according to the position information of the marker in the millimeter wave radar coordinate system and the position information of the marker in the laser radar coordinate system, the marker can be rotated, translated and aligned in space to obtain a rotation matrix and a translation vector, the rotation matrix and the translation vector are parameters in a coordinate transformation relation between the laser radar and the millimeter wave radar, the coordinate transformation relation is the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system, manual participation is not needed in the process, and the efficiency and the precision of space synchronization of the millimeter wave radar and the laser radar are improved. After the coordinate transformation relation between the laser radar and the millimeter wave radar is obtained, all point clouds in the point cloud data can be transformed into a hetmap distribution diagram based on the coordinate transformation relation, and a target point cloud set corresponding to the target can be obtained based on the position of the target in the hetmap distribution diagram (for example, the point cloud in the position of the target in the hetmap distribution diagram is the target point cloud corresponding to the target).
It should be noted that, after the computer device obtains the point cloud data and the millimeter wave signal, the computer device may perform spatial synchronization on the point cloud data and the millimeter wave signal first, and then obtain three-dimensional stereo data according to the millimeter wave signal after spatial synchronization is completed, or may obtain three-dimensional stereo data according to the millimeter wave signal first, and then perform spatial synchronization on the point cloud data and the millimeter wave signal, which is not limited herein.
As an alternative embodiment, after spatially synchronizing the point cloud data and the millimeter wave signal, the method further includes:
regularization processing is carried out on the point cloud data and the millimeter wave signals respectively so as to unify the amplitude ranges of the point cloud data and the millimeter wave signals.
Through regularization processing of the point cloud data and the millimeter wave signals, the point cloud data and the millimeter wave signals can be unified in the same amplitude range, so that the computer equipment can quickly find the target point cloud from the point cloud data.
As an alternative embodiment, before the spatial synchronization of the point cloud data and the hetmap profile is completed, the method further comprises:
performing frame number alignment on the point cloud data and the millimeter wave signals under the same time coordinate axis so as to perform time synchronization on the point cloud data and the millimeter wave signals; the point cloud data and the millimeter wave signals are respectively and simultaneously output by a laser radar and a millimeter wave radar, the first radar is used for transmitting and receiving electromagnetic waves after receiving trigger pulses transmitted by the second radar at fixed time intervals, the first radar is any one of the laser radar and the millimeter wave radar, and the second radar is the radar except the first radar in the laser radar and the millimeter wave radar.
When the time synchronization of the point cloud data and the millimeter wave signals is realized, a mode of combining software synchronization and hardware synchronization can be used, so that the precision of the time synchronization is improved. The hardware synchronization may be that the second radar sends a trigger pulse to the first radar at fixed time intervals, the first radar sends and receives electromagnetic waves after receiving the trigger pulse, and the first radar and the second radar output sampled signals (i.e., point cloud data and millimeter wave signals) at the same time. The software synchronization is based on hardware synchronization, and frame number alignment is carried out on point cloud data output by the laser radar and millimeter wave signals output by the millimeter wave radar under the same time coordinate axis.
In an application scene of hardware synchronization, the first radar is a millimeter wave radar, the second radar is a laser radar, when the laser radar rotates and scans, a trigger pulse is sent to the millimeter wave radar every time a motor of the laser radar rotates to cross a zero point, and when the trigger pulse is received by the millimeter wave radar, FMCW with the fundamental frequency of 24GHz or 77GHz starts to be sent. Meanwhile, the millimeter wave radar receiving antenna starts to receive electromagnetic wave echoes, after the transmission is finished, the reception is basically finished, the laser radar also rotationally scans the same field of view, and at the moment, sampling signals of the millimeter wave radar receiving antenna and the laser radar are output as the same frame content.
And 206, carrying out data enhancement on the target point cloud set.
Wherein, the data enhancement on the target point cloud set can be understood as increasing the density of the point cloud data.
The data enhancement of the target point cloud set mainly solves the problems of sparse point cloud, limited detection distance and the like of the laser radar by utilizing the capability of the millimeter wave radar to detect a longer distance. And the point cloud data enhancement scheme based on the hetmap distribution diagram can solve the problem of missed detection or false detection of point cloud level fusion or target level fusion.
In an exemplary diagram, the target 2 in fig. 4 is a far-end target of the lidar of the vehicle a, the most distant energy peak in fig. 5 is the target 2, and as can be seen from fig. 5, the position probability distribution of the millimeter wave signal of the target 2 covers a range from-20 ° to 20 °, the distance is about 80 meters, and the probability distribution of the distance is more concentrated than the probability distribution of the angle. When the laser radar detects the target 2 at the same time, due to the sparsity of the laser radar at the far end, the scanning angle of 0.2 degrees, the point distance at 80 meters is 28cm, and the width of one vehicle is about 1.8m, so that for the vehicle (namely the target 2), only 6 to 7 target point clouds exist. At this time, since the millimeter wave radar and the laser radar detect the existence of the target 2 at the same time, the cloud of the target point of the target 2 appears in the peak of the millimeter wave signal and also appears in the energy peak in the hectmap profile. By data enhancement of 6 to 7 target point clouds of the target 2, denser point cloud data can be obtained.
According to the embodiment of the application, the point cloud data and the millimeter wave signals obtained by sampling the data of the same area by the laser radar and the millimeter wave radar are obtained, three-dimensional stereo data can be obtained firstly according to the millimeter wave signals, then the hetmap distribution diagram is obtained according to the three-dimensional stereo data, and the target in the hetmap distribution diagram is determined by extracting the target, the corresponding target point cloud set in the point cloud data which is spatially synchronous with the hetmap distribution diagram can be determined based on the target in the hetmap distribution diagram, and the density of the point cloud data (particularly the point cloud data of the far end of the laser radar) of the laser radar can be increased by carrying out data enhancement on the target point cloud set, so that the detection accuracy of the laser radar on the target is improved.
Referring to fig. 6, a flowchart of an implementation of a point cloud data enhancement method according to another embodiment of the present application is shown, where the point cloud data enhancement method is applied to a computer device in the multi-sensor system shown in fig. 1. As shown in fig. 6, the point cloud data enhancement method may include the steps of:
and 601, acquiring point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar.
This step is the same as step 201, and specific reference may be made to the related description of step 201, which is not repeated here.
And step 602, acquiring three-dimensional cube data according to the millimeter wave signals.
The step is the same as step 202, and the detailed description of step 202 is omitted here.
And 603, acquiring a hematmap distribution map according to the three-dimensional cubic data.
This step is the same as step 203, and specific reference may be made to the related description of step 203, which is not repeated here.
And step 604, extracting targets from the Heatm ap distribution map, and determining the targets in the Heatm ap distribution map.
The step is the same as step 204, and the detailed description of step 204 is omitted here.
Step 605, based on the targets, determining a corresponding set of target point clouds in the point cloud data spatially synchronized with the hetmap profile.
This step is the same as step 205, and specific reference may be made to the description of step 205, which is not repeated here.
Step 606, equidistant fitting of the target point clouds in the target point cloud set.
By carrying out equidistant fitting on the target point cloud, reasonable enhancement of the density of the point cloud can be ensured. Wherein the equidistant fitting may also be referred to as equidistant interpolation, which belongs to the linear interpolation method. Of course, other interpolation methods may be used to enhance the point cloud density, which is not limited herein.
As an alternative embodiment, the target point clouds in the target point cloud set may be fitted equally spaced using a power function. I.e. an equidistant fitting is achieved with a power function.
According to the embodiment of the application, the point cloud data and the millimeter wave signals obtained by sampling the data of the same area by the laser radar and the millimeter wave radar can be obtained firstly, three-dimensional stereo data can be obtained according to the millimeter wave signals, then the hematmap distribution map is obtained according to the three-dimensional stereo data, and the target in the hematmap distribution map is determined by extracting the target, so that the corresponding target point cloud set in the point cloud data which is spatially synchronous with the hematmap distribution map can be determined based on the target in the hematmap distribution map, reasonable enhancement of the point cloud density can be ensured by fitting the target point cloud at equal intervals, and the detection accuracy of the laser radar to the target is improved.
Referring to fig. 7, a schematic structural diagram of a point cloud data enhancement device according to an embodiment of the present application is shown, for convenience of explanation, only a portion related to the embodiment of the present application is shown.
The point cloud data enhancement device comprises:
the signal acquisition module 71 is configured to acquire point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar, where the point cloud data and the millimeter wave signals are obtained by respectively performing data sampling on the same area by the laser radar and the millimeter wave radar;
A data acquisition module 72, configured to acquire three-dimensional data according to the millimeter wave signal; the three-dimensional data comprises information of distance, speed and angle dimensions;
a profile acquisition module 73 for acquiring a hetmap profile from the three-dimensional stereo data; the hematmap is an information map of the distance and angle dimensions, and the information map comprises the energy of points corresponding to the distance and the angle;
a target extraction module 74 for performing target extraction on the hetmap profile to determine a target in the hetmap profile;
a set determining module 75, configured to determine, based on the target, a set of target points cloud corresponding to the point cloud data spatially synchronized with the hetmap profile;
and the data enhancement module 76 is used for enhancing the data of the target point cloud set.
Optionally, the target extraction module 74 is specifically configured to:
the energy peak is targeted in the hematmap profile.
Optionally, the set determining module 75 is specifically configured to:
judging whether at least two target point clouds in the point cloud data exist in the 3sigma distribution of the target under the condition that the point cloud data and the hetmap distribution diagram are spatially synchronized;
And if at least two target point clouds exist in the 3sigma distribution of the target and exist in the point cloud data, determining that the at least two target point clouds form the target point cloud set.
Optionally, markers are arranged in the perception ranges of the laser radar and the millimeter wave radar; the point cloud data enhancement device further comprises:
the information acquisition module is used for acquiring the position information of the marker in a laser radar coordinate system and the position information of the marker in a millimeter wave radar coordinate system respectively;
the pose acquisition module is used for acquiring the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system according to the position information of the marker in the laser radar coordinate system and the position information in the millimeter wave radar coordinate system respectively;
and the space synchronization module is used for performing space synchronization on the point cloud data and the millimeter wave signals according to the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system so as to complete the space synchronization of the point cloud data and the hetmap distribution diagram.
Optionally, the point cloud data enhancement device further includes:
and the amplitude processing module is used for regularizing the point cloud data and the millimeter wave signals respectively so as to unify the amplitude ranges of the point cloud data and the millimeter wave signals.
Optionally, the point cloud data enhancement device further includes:
the time synchronization module is used for aligning the frame numbers of the point cloud data and the millimeter wave signals under the same time coordinate axis so as to perform time synchronization on the point cloud data and the millimeter wave signals; the point cloud data and the millimeter wave signals are respectively and simultaneously output by the laser radar and the millimeter wave radar, and a first radar transmits and receives electromagnetic waves after receiving trigger pulses transmitted by a second radar every fixed time, wherein the first radar is any one of the laser radar and the millimeter wave radar, and the second radar is the radar except the first radar in the laser radar and the millimeter wave radar.
Optionally, the data acquisition module 72 is specifically configured to:
and carrying out Fourier transformation on the distance, the speed and the angle dimension of the millimeter wave signal to obtain the three-dimensional cubic data.
Optionally, the data enhancement module 76 is specifically configured to:
and equally fitting the target point clouds in the target point cloud set.
Optionally, the data enhancement module 76 is specifically configured to:
and equally fitting the target point clouds in the target point cloud set by using a power function.
The point cloud data enhancement device provided in the embodiment of the present application may be applied to the foregoing method embodiment, and details refer to descriptions of the foregoing method embodiment, which are not repeated herein.
Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 8, the computer device of this embodiment includes: one or more processors 80 (only one shown), a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. The processor 80, when executing the computer program 82, implements the steps of the point cloud data enhancement method embodiments described above.
The computer device may include, but is not limited to, a processor 80, a memory 81. It will be appreciated by those skilled in the art that fig. 8 is merely an example of a computer device and is not intended to be limiting, and that more or fewer components than shown may be included, or certain components may be combined, or different components may be included, for example, in a computer device that may also include input-output devices, network access devices, buses, etc. Optionally, the computer device further comprises an antenna array.
The processor 80 may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 81 may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory 81 may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like. Further, the memory 81 may also include both an internal storage unit and an external storage device of the computer device. The memory 81 is used for storing the computer program and other programs and data required by the computer device. The memory 81 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above device may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
The embodiments of the present application also provide a computer readable storage medium storing a computer program, where the computer program when executed by a processor implements steps of the foregoing method embodiments.
The present application also provides a computer program product which, when run on a computer device, causes the computer device to perform the steps that can be carried out in the respective method embodiments described above.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in this application, it should be understood that the disclosed apparatus/computer device and method may be implemented in other ways. For example, the apparatus/computer device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each method embodiment described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (13)

1. The point cloud data enhancement method is characterized by comprising the following steps of:
acquiring point cloud data of a laser radar and millimeter wave signals of the millimeter wave radar, wherein the point cloud data and the millimeter wave signals are obtained by respectively carrying out data sampling on the same area by the laser radar and the millimeter wave radar;
acquiring three-dimensional cube data according to the millimeter wave signals; the three-dimensional cube data comprises information of distance, speed and angle dimensions;
acquiring a hematmap distribution diagram according to the three-dimensional cubic data; the hematmap is an information map of the distance and angle dimensions, and the information map comprises the energy of points corresponding to the distance and the angle;
Extracting targets from the Heatm ap distribution map, and determining the targets in the Heatm ap distribution map;
determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap profile based on the target;
and carrying out data enhancement on the target point cloud set.
2. The method of claim 1, wherein performing target extraction on the hetmap profile, determining targets in the hetmap profile, comprises:
the energy peak is targeted in the hematmap profile.
3. The method of claim 1, wherein determining a corresponding set of target point clouds in point cloud data spatially synchronized with the hetmap profile based on the target comprises:
judging whether at least two target point clouds in the point cloud data exist in the 3sigma distribution of the target under the condition that the point cloud data and the hetmap distribution diagram are spatially synchronized;
and if at least two target point clouds exist in the 3sigma distribution of the target and exist in the point cloud data, determining that the at least two target point clouds form the target point cloud set.
4. A method according to claim 3, wherein markers are provided in the range of perception of the lidar and the millimeter wave radar; further comprises:
Acquiring position information of the marker in a laser radar coordinate system and position information of the marker in a millimeter wave radar coordinate system respectively;
acquiring the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system according to the position information of the marker in the laser radar coordinate system and the position information in the millimeter wave radar coordinate system respectively;
and according to the relative pose between the laser radar coordinate system and the millimeter wave radar coordinate system, performing spatial synchronization on the point cloud data and the millimeter wave signals so as to complete the spatial synchronization of the point cloud data and the hetmap distribution diagram.
5. The method of claim 4, further comprising, after spatially synchronizing the point cloud data and the millimeter wave signal:
and regularizing the point cloud data and the millimeter wave signals respectively to unify the amplitude ranges of the point cloud data and the millimeter wave signals.
6. The method of claim 3, further comprising, prior to the point cloud data and the hetmap profile completing spatial synchronization:
performing frame number alignment on the point cloud data and the millimeter wave signals under the same time coordinate axis so as to perform time synchronization on the point cloud data and the millimeter wave signals; the point cloud data and the millimeter wave signals are respectively and simultaneously output by the laser radar and the millimeter wave radar, and a first radar transmits and receives electromagnetic waves after receiving trigger pulses transmitted by a second radar every fixed time, wherein the first radar is any one of the laser radar and the millimeter wave radar, and the second radar is the radar except the first radar in the laser radar and the millimeter wave radar.
7. The method of any of claims 1 to 6, wherein acquiring three-dimensional cubic data from the millimeter wave signal comprises:
and carrying out Fourier transformation on the distance, the speed and the angle dimension of the millimeter wave signal to obtain the three-dimensional cubic data.
8. The method of any of claims 1 to 6, wherein data enhancement of the set of target point clouds comprises:
and equally fitting the target point clouds in the target point cloud set.
9. The method of claim 8, wherein equally fitting the target point clouds in the set of target point clouds comprises:
and equally fitting the target point clouds in the target point cloud set by using a power function.
10. A point cloud data enhancement apparatus, characterized in that the point cloud data enhancement apparatus comprises:
the system comprises a signal acquisition module, a data acquisition module and a data acquisition module, wherein the signal acquisition module is used for acquiring point cloud data of a laser radar and millimeter wave signals of a millimeter wave radar, and the point cloud data and the millimeter wave signals are obtained by respectively carrying out data sampling on the same area through the laser radar and the millimeter wave radar;
the data acquisition module is used for acquiring three-dimensional data according to the millimeter wave signals; the three-dimensional data comprises information of distance, speed and angle dimensions;
The distribution map acquisition module is used for acquiring a hematmap distribution map according to the three-dimensional stereo data; the hematmap is an information map of the distance and angle dimensions, and the information map comprises the energy of points corresponding to the distance and the angle;
the target extraction module is used for extracting targets from the Heatm ap distribution map and determining the targets in the Heatm ap distribution map;
the set determining module is used for determining a corresponding target point cloud set in the point cloud data spatially synchronized with the hetmap distribution diagram based on the target;
and the data enhancement module is used for enhancing the data of the target point cloud set.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 9 when the computer program is executed.
12. A multi-sensor system comprising a lidar, a millimeter wave radar, and the computer device of claim 11;
the laser radar and the millimeter wave radar acquire data of the same area;
The laser radar is used for outputting the collected point cloud data to the computer equipment;
the millimeter wave radar is used for outputting the collected millimeter wave signals to the computer equipment.
13. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 9.
CN202111615244.7A 2021-12-27 2021-12-27 Point cloud data enhancement method, device, computer equipment, system and storage medium Pending CN116359908A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202111615244.7A CN116359908A (en) 2021-12-27 2021-12-27 Point cloud data enhancement method, device, computer equipment, system and storage medium
PCT/CN2022/136303 WO2023124780A1 (en) 2021-12-27 2022-12-02 Point cloud data augmentation method and apparatus, and computer device, system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111615244.7A CN116359908A (en) 2021-12-27 2021-12-27 Point cloud data enhancement method, device, computer equipment, system and storage medium

Publications (1)

Publication Number Publication Date
CN116359908A true CN116359908A (en) 2023-06-30

Family

ID=86925191

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111615244.7A Pending CN116359908A (en) 2021-12-27 2021-12-27 Point cloud data enhancement method, device, computer equipment, system and storage medium

Country Status (2)

Country Link
CN (1) CN116359908A (en)
WO (1) WO2023124780A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523568B (en) * 2024-01-08 2024-03-29 陕西欧卡电子智能科技有限公司 Bridge height detection method, device, equipment and medium based on unmanned ship

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509972A (en) * 2018-01-16 2018-09-07 天津大学 A kind of barrier feature extracting method based on millimeter wave and laser radar
KR20200144862A (en) * 2019-06-19 2020-12-30 삼성전자주식회사 Method and device to improve resolution of radar
CN113536850B (en) * 2020-04-20 2022-09-30 长沙莫之比智能科技有限公司 Target object size testing method and device based on 77G millimeter wave radar
CN111352112B (en) * 2020-05-08 2022-11-29 泉州装备制造研究所 Target detection method based on vision, laser radar and millimeter wave radar
CN112401856B (en) * 2020-11-16 2022-07-08 武汉烽火凯卓科技有限公司 Nursing home monitoring method and system based on millimeter wave radar

Also Published As

Publication number Publication date
WO2023124780A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
Gao et al. MIMO-SAR: A hierarchical high-resolution imaging algorithm for mmWave FMCW radar in autonomous driving
TWI834771B (en) Early fusion of camera and radar frames
US9746554B2 (en) Radar imaging system and related techniques
CN112639519B (en) Apparatus and method for two-stage signal processing in radar systems
CN111896947A (en) Rapid super-resolution tracking system and method for automotive millimeter wave radar
CN111679266A (en) Automobile millimeter wave radar sparse array grating lobe false target identification method and system
CN116027318A (en) Method, device, electronic equipment and storage medium for multi-sensor signal fusion
WO2023124780A1 (en) Point cloud data augmentation method and apparatus, and computer device, system and storage medium
Ram Fusion of inverse synthetic aperture radar and camera images for automotive target tracking
US20210323560A1 (en) Vehicle speed calculation method, system, device, and storage medium
WO2022183408A1 (en) Lane line detection method and lane line detection apparatus
CN111175715A (en) Auxiliary driving system and method capable of restraining short-distance harmonic waves of radar
CN116027288A (en) Method and device for generating data, electronic equipment and storage medium
Hoffmann et al. Filter-based segmentation of automotive sar images
CN114924246A (en) Target detection method, target detection device, millimeter wave radar and storage medium
WO2022097158A2 (en) Efficient direction of arrival estimation using low rank approximation
CN112689773B (en) Radar signal processing method and radar signal processing device
CN116068503A (en) Combined calibration method and device for millimeter wave radar and laser radar and terminal equipment
CN116047442A (en) Method and device for detecting target angle and electronic equipment
CN111175717A (en) Auxiliary driving method capable of restraining short-distance harmonic waves of radar and scene application
Zaumseil et al. Radar-based near field environment perception using back projection algorithm
JP7223197B2 (en) ELECTRONIC DEVICE, ELECTRONIC DEVICE CONTROL METHOD, AND PROGRAM
JP7113878B2 (en) ELECTRONIC DEVICE, ELECTRONIC DEVICE CONTROL METHOD, AND PROGRAM
CN118169689A (en) Method and device for determining passable area, millimeter wave radar and medium
Lei et al. Millimetre-Wave Automotive Radar Imaging Enhancement via Doppler Beam Sharpening

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination