CN111563398A - Method and device for determining information of target object - Google Patents
Method and device for determining information of target object Download PDFInfo
- Publication number
- CN111563398A CN111563398A CN201910114242.6A CN201910114242A CN111563398A CN 111563398 A CN111563398 A CN 111563398A CN 201910114242 A CN201910114242 A CN 201910114242A CN 111563398 A CN111563398 A CN 111563398A
- Authority
- CN
- China
- Prior art keywords
- target object
- point cloud
- rectangular frame
- image
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Traffic Control Systems (AREA)
Abstract
The embodiment of the application discloses a method and a device for determining information of a target object. One embodiment of the method for determining information of a target object comprises: acquiring an image of the rectangular frame of the identified target object; based on the external parameters of the camera and the external parameters of the laser radar, projecting the laser point cloud data points into the image; screening point cloud data points projected into a target object rectangular frame in the image; determining a filtering window based on the target object rectangular frame; based on the point cloud information in the filtering window, filtering interference data points with the distance from the laser radar larger than a preset threshold value from the point cloud data points in the rectangular frame of the target object to obtain point cloud on the target object in the rectangular frame of the target object; information of the target object is determined based on the point cloud above the target object and the target object. The method and the device can fuse two-dimensional information of the target object on the image and three-dimensional information of the point cloud, and extract accurate and effective information of the target object.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for determining information of a target object.
Background
In recent years, the automatic driving technology has received a high degree of attention from the automatic control industry. In the automatic driving equipment taking the automatic driving technology as the technical core, the pedestrian detection technology plays an important role, and is a key technology for protecting vehicles and pedestrians during the operation of the whole vehicle. The general method of the pedestrian detection technology is to detect and identify pedestrians by using image information, but the distance of the pedestrians can be obtained only by adopting a binocular or laser radar mode, the whole information extraction process is complex, and the calculation amount is large, so that how to detect the information of the pedestrians more gradually and accurately is a technical problem to be solved by the technical personnel in the field.
At present, the mainstream way for detecting the information of the pedestrian is to adopt binocular to simultaneously obtain point cloud and image information, extract the image information of the pedestrian based on image identification, extract distance information from a corresponding depth map, or identify the characteristics of the pedestrian and extract the information of the pedestrian on three-dimensional laser point cloud by using a clustering method.
However, the distance of binocular detection is short, and pedestrians beyond a certain distance from the automatic driving device cannot recognize: the laser radar has a large identification distance, but the pedestrian identification difficulty based on point cloud is large, and the accuracy is difficult to guarantee.
Disclosure of Invention
The embodiment of the application provides a method and a device for determining information of a target object.
In a first aspect, an embodiment of the present application provides a method for determining information of a target object, including: acquiring an image of the rectangular frame of the identified target object; projecting the laser point cloud data points into the image based on external parameters of a camera and external parameters of a laser radar; screening point cloud data points projected into the target object rectangular frame in the image; determining a filtering window based on the target object rectangular frame; based on the point cloud information in the filtering window, filtering interference data points with the distance from the laser radar larger than a preset threshold value from the point cloud data points in the rectangular frame of the target object to obtain point cloud on the target object in the rectangular frame of the target object; determining information of the target object based on the point cloud above the target object and the image of the target object.
In some embodiments, the acquiring the image of the rectangular frame of the identified target object comprises: detecting the image collected by the camera by adopting a target detection algorithm to obtain a rectangular frame to be identified; and identifying and framing the object in the rectangular frame to be identified by adopting a target identification algorithm to obtain an image of the rectangular frame of the identified target object.
In some embodiments, the projecting the laser point cloud data points into the image based on the camera extrinsic parameters and the lidar extrinsic parameters comprises: determining a rotation matrix and a translation matrix of a coordinate system based on external parameters of a camera and external parameters of a laser radar; based on the rotation matrix and the translation matrix, projecting laser point cloud data points into the image.
In some embodiments, the determining a filtered window based on the target object rectangular box comprises: equally dividing the rectangular frame of the target object into an upper rectangular frame and a lower rectangular frame by taking the transverse axis as a symmetry axis; and determining the upper rectangular frame of the upper and lower rectangular frames as a filtering window.
In some embodiments, the determining a filtered window based on the target object rectangular box comprises: and determining a window of the rectangular frame of the target object, which comprises a preset key part of the target object, as a filtering window.
In some embodiments, the filtering, based on the point cloud information in the filtered window, interference data points whose distance from the lidar is greater than a predetermined threshold from the point cloud data points within the rectangular frame of the target object to obtain the point cloud above the target object within the rectangular frame of the target object includes: determining a data point closest to the laser radar based on the point cloud information in the filtered window; and searching in the target object rectangular frame by taking the nearest data point as a reference, and filtering out interference point clouds of which the distance from the laser radar is greater than a preset threshold value to obtain the point clouds on the target object in the target object rectangular frame.
In some embodiments, the target includes: pedestrians, cyclists, vehicles, utility poles, and trash cans.
In some embodiments, the determining information of the target object based on the point cloud over the target object and the image of the target object comprises: determining image information, distance information, and speed information of the target object based on the point cloud above the target object and the target object.
In a second aspect, an embodiment of the present application provides an apparatus for determining information of a target object, including: an image acquisition unit configured to acquire an image in which a rectangular frame of the target object has been recognized; a point cloud projection unit configured to project laser point cloud data points into the image based on external parameters of a camera and external parameters of a lidar; a data point screening unit configured to screen point cloud data points projected into the target object rectangular frame in the image; a window determination unit configured to determine a filtered window based on the target object rectangular frame; a point cloud determination unit configured to filter interference data points, the distance from which to the laser radar is greater than a predetermined threshold value, from the point cloud data points in the target object rectangular frame based on the point cloud information in the filtered window, to obtain a point cloud above a target object in the target object rectangular frame; an information determination unit configured to determine information of the target object based on the point cloud above the target object and the image of the target object.
In some embodiments, the image acquisition unit comprises: the image detection unit is configured to detect the image acquired by the camera by adopting a target detection algorithm to obtain a rectangular frame to be identified; and the image recognition unit is configured to recognize and frame the object in the rectangular frame to be recognized by adopting a target recognition algorithm to obtain an image of the rectangular frame of the recognized target object.
In some embodiments, the point cloud projection unit comprises: a matrix determination unit configured to determine a rotation matrix and a translation matrix of a coordinate system based on external parameters of the camera and external parameters of the lidar; a data point projection unit configured to project a laser point cloud data point into the image based on the rotation matrix and the translation matrix.
In some embodiments, the window determination unit is further configured to: equally dividing the rectangular frame of the target object into an upper rectangular frame and a lower rectangular frame by taking the transverse axis as a symmetry axis; and determining the upper rectangular frame of the upper and lower rectangular frames as a filtering window.
In some embodiments, the window determination unit is further configured to: and determining a window of the rectangular frame of the target object, which comprises a preset key part of the target object, as a filtering window.
In some embodiments, the point cloud determination unit further comprises: a data point determination unit configured to determine a closest data point to a lidar based on the point cloud information in the filtered window; and the point cloud filtering unit is configured to search in the target object rectangular frame by taking the nearest data point as a reference, filter out interference point clouds of which the distance from the laser radar is greater than a preset threshold value, and obtain the point cloud above the target object in the target object rectangular frame.
In some embodiments, the target in the point cloud determination unit comprises: pedestrians, cyclists, vehicles, utility poles, and trash cans.
In some embodiments, the information determination unit is further configured to: determining image information, distance information, and speed information of the target object based on the point cloud above the target object and the target object.
In a third aspect, an embodiment of the present application provides an unmanned automobile, including: a camera, a lidar and an apparatus as described in any of the above.
In a fourth aspect, an embodiment of the present application provides a robot, including: a camera, a lidar and an apparatus as described in any of the above.
In a fifth aspect, an embodiment of the present application provides an apparatus, including: one or more processors; storage means for storing one or more programs; when executed by one or more processors, cause the one or more processors to implement a method as described in any above.
In a sixth aspect, embodiments of the present application provide a computer-readable medium, on which a computer program is stored, which when executed by a processor implements the method as described in any one of the above.
According to the method and the device for determining the information of the target object, firstly, the image of the rectangular frame of the identified target object is obtained; then, based on the external parameters of the camera and the external parameters of the laser radar, projecting the laser point cloud data points into the image; then, screening point cloud data points projected into a rectangular frame of a target object in the image; then, determining a filtering window based on a target object rectangular frame in the image; then based on the point cloud information in the filtering window, filtering interference data points with the distance from the laser radar larger than a preset threshold value from the point cloud data points in the rectangular frame of the target object to obtain point cloud on the target object in the rectangular frame of the target object; finally, information of the target object is determined based on the point cloud above the target object and the image of the target object. In the process, the method can fuse the two-dimensional information of the target object on the image and the three-dimensional information on the point cloud, so that the information of the target object can be accurately and effectively extracted, and a data source is provided for perception decision.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a schematic flow chart diagram illustrating one embodiment of a method for determining information for a target object in accordance with embodiments of the present application;
FIG. 3 is a schematic diagram of an application scenario of a method for determining information of a target object according to an embodiment of the present application;
FIG. 4 is a schematic flow chart diagram illustrating yet another embodiment of a method for determining information for a target object in accordance with an embodiment of the present application;
FIG. 5 is a schematic diagram of an embodiment of an apparatus for determining information of a target object according to the present application;
FIG. 6 is a schematic block diagram of a computer system suitable for use in implementing a server according to embodiments of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 shows an exemplary system architecture 100 to which embodiments of the present method for determining information of an object or an apparatus for determining information of an object may be applied.
Fig. 1 shows an exemplary system architecture 100 to which embodiments of the present method for controlling a vehicle or of an apparatus for determining information of a target object can be applied.
As shown in fig. 1, the system architecture 100 may include terminals 101, 102, 103, a network 104, and servers 105, 106. The network 104 serves as a medium for providing communication links between the terminals 101, 102, 103 and the servers 105, 106. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user 110 may use the terminals 101, 102, 103 to interact with the servers 105, 106 via the network 104 to receive or send messages or the like. The terminals 101, 102, 103 may have various communication client applications installed thereon, such as shopping applications, instant messaging tools, mailbox clients, social platform software, video playing applications, and the like.
The terminals 101, 102, 103 may be hardware or software. When the terminals 101, 102, 103 are hardware, they may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like. When the terminals 101, 102, 103 are software, they can be installed in the electronic devices listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The servers 105, 106 may be servers providing various services, such as background servers providing support for the terminals 101, 102, 103. The background server can analyze, store or calculate the data submitted by the terminal and push the analysis, storage or calculation result to the terminal.
It should be noted that, in practice, the method for determining information of an object provided by the embodiment of the present application may be executed by the servers 105 and 106, and accordingly, the apparatus for determining information of an object is generally disposed in the servers 105 and 106. However, when the performance of the terminal can satisfy the execution condition of the method or the setting condition of the device, the method for determining the information of the object provided by the embodiment of the present application may also be executed by the terminal 101, 102, 103, and the means for determining the information of the object may also be provided in the terminal 101, 102, 103.
It should be understood that the number of terminals, networks, and servers in fig. 1 are merely illustrative. There may be any number of terminals, networks, and servers, as desired for an implementation.
With continued reference to FIG. 2, FIG. 2 illustrates a flow 200 of one embodiment of a method for determining information of a target object according to the present application. The method for determining the information of the target object comprises the following steps:
In this embodiment, an execution subject (for example, a terminal or a server shown in fig. 1) on which the above-described method for determining information of an object operates may acquire an image of a rectangular frame of an identified object from a local or server. Here, the image in which the rectangular frame of the target object has been recognized may be obtained by detecting and recognizing an image captured by the camera.
In some optional implementations of the embodiment, acquiring the image of the rectangular frame of the identified target object includes: detecting an image acquired by a camera by adopting a target detection algorithm to obtain a rectangular frame to be identified; and identifying and framing the object in the rectangular frame to be identified by adopting a target identification algorithm to obtain an image of the rectangular frame of the identified target object.
In the implementation mode, the target detection algorithm is adopted to detect the image acquired by the camera, so that the predicted rectangular object frames and the probability that the rectangular object frames are accurate prediction results can be obtained. Based on the predicted object rectangular boxes and the probabilities that these object rectangular boxes are accurate prediction results, the rectangular box to be recognized that is finally used for recognizing the target can be determined. For example, the rectangular frames of the objects are all used as the rectangular frames to be recognized of the recognition target, or the rectangular frame of the object with higher probability is used as the rectangular frame to be recognized of the recognition target.
The target detection algorithm may be implemented by an algorithm for detecting a target in the prior art or a technology developed in the future, which is not limited in the present application. For example, the Object Detection algorithm may be implemented by using a cascade Detection network (CascadedRCNN), a relationship network for Object Detection (relationship Networks for Object Detection), a single-shot Object detector (RefineDet), Scale Normalization of image pyramids (Scale Normalization for image templates), an Object Detection method (R-FCN), enrichment with EnichedSemantics, scalable Object Detection (Scale-scalable Object Detection), and the like.
Then, for the object in the rectangular frame to be recognized, the class of the object and the probability that the object is in the class can be recognized by adopting a target recognition algorithm. And finally, according to the recognized object class and the probability, the class with the highest probability is used as the object class, and the image of the recognized target object rectangular frame is obtained.
The target recognition algorithm may also be implemented by an algorithm for recognizing a target in the prior art or a technology developed in the future, which is not limited in the present application. For example, the method is implemented by using a classifier for target recognition, and the classifier can comprise a support vector machine, a K-nearest neighbor, a neural network, a random forest and the like.
Detecting an image acquired by a camera by adopting a target detection algorithm to obtain a rectangular frame to be identified; and then, the object in the rectangular frame to be recognized is recognized and framed by adopting a target recognition algorithm to obtain the image of the rectangular frame of the recognized target object, so that the image of the rectangular frame of the recognized target object can be quickly and accurately obtained, and the efficiency of subsequently determining the information of the target object is improved.
And 202, projecting the laser point cloud data points into an image based on the external parameters of the camera and the external parameters of the laser radar.
In this embodiment, the external parameters of the camera and the external parameters of the lidar may be determined according to the detected target object by an algorithm for calculating the external parameters, and then the external parameters of the camera and the external parameters of the lidar are determined by the algorithm.
After determining the external parameters of the camera and the external parameters of the lidar, a transformation relationship between the coordinate system of the camera and the coordinate system of the lidar may be determined according to the external parameters of the camera and the external parameters of the lidar, and then the laser point cloud data points may be projected into the image using the transformation relationship. The laser point cloud data herein generally includes three-dimensional coordinates (XYZ), and some laser point cloud data may further include laser reflection Intensity (Intensity) and/or color information (RGB).
In this embodiment, based on the laser point cloud data projected into the image in step 202, the point cloud data points projected into the rectangular frame of the target object in the image can be screened. At this time, the determined information of the target object includes point cloud information in a rectangular frame of the target object and image information in the rectangular frame of the target object.
And step 204, determining a filtering window based on the rectangular frame of the target object in the image.
In this embodiment, the target object rectangular frame may be directly determined as the filtered window, or a part of the target object rectangular frame may be determined as the filtered window. Such as a predetermined area within the rectangular frame of the object, a window within the rectangular frame of the object having predetermined characteristics, etc.
In some optional implementation manners of this embodiment, a window in the rectangular frame of the target object, which includes a preset key portion of the target object, may be determined as a filtered window.
In this implementation manner, the target object included in the target object rectangular frame generally has a key part, and then the window of the target object including the key part is determined as a filtering window, so that an area with dense point clouds and rich information can be determined, so as to simplify search content in the following and improve search efficiency.
In this embodiment, according to the filtered window determined in step 204, distance information of the point cloud in the filtered window may be obtained. When the laser radar scans, the point cloud formed by the scanned target object is closer to the laser radar, and the point cloud farther from the laser radar is usually an interference point cloud such as a background. Therefore, interference data points with the distance from the laser radar larger than a preset threshold value can be filtered from the point cloud data points in the rectangular frame of the target object, and the point cloud above the target object in the rectangular frame of the target object is obtained. The point cloud above the target in the target rectangular frame is the point cloud generated by scanning the target.
Here, the object may be an object captured midway through the image capture, and the object itself is not limited to this. For example, the target object may be a pedestrian, a cyclist, a vehicle, a utility pole, a trash can, and the like, of interest while the vehicle is traveling.
In the embodiment, based on the point cloud above the target object, three-dimensional information of the target object in the camera coordinates can be acquired; based on the image of the target object, two-dimensional information of the target object in a camera coordinate system can be acquired; based on the three-dimensional information and the two-dimensional information in the camera coordinate system, more comprehensive three-dimensional information of the target object can be obtained through fusion.
In an optional implementation manner of this embodiment, determining the information of the target object based on the point cloud above the target object and the image of the target object may include: image information, distance information, and velocity information of the target object are determined based on the point cloud above the target object and the target object.
In the implementation mode, two-dimensional information of the target object, namely image information of the target object can be determined based on three-dimensional information in a camera coordinate system, and the image information is fused with point cloud information, so that the details are richer and more complete; based on the three-dimensional information in the camera coordinate system, the distance information of the target object from the laser radar can be further determined; based on the tracking of the camera image and the laser point cloud, the moving speed of the target object can also be determined.
By the method for determining the information of the target object in the implementation mode, the details of the image information of the target object can be improved, and the distance and the moving speed of the target object can be determined.
An exemplary application scenario of the method for determining information of a target object of the present application is described below with reference to fig. 3.
Fig. 3 shows a schematic flow diagram of an application scenario of the method for determining information of an object according to the present application, as shown in fig. 3.
As shown in fig. 3, a method 300 for determining information of a target object operates in an electronic device 310 and may include:
firstly, acquiring an image 302 of a rectangular frame 301 of an identified target object;
then, based on the camera extrinsic parameters 303 and the lidar extrinsic parameters 304, projecting a laser point cloud data point 305 into the image 302;
then, point cloud data points 306 projected into the target object rectangular box 301 in the image 302 are determined;
then, based on the rectangular frame 301 of the target object in the image 302, determining a filtered window 307;
then, based on the point cloud information in the filtering window 307, filtering an interference data point 308 with a distance from the laser radar greater than a predetermined threshold value from the point cloud data point in the rectangular frame of the target object, and obtaining a point cloud 309 on the target object in the rectangular frame of the target object;
finally, based on the point cloud 309 above the target and the image 310 of the target in the target rectangular box 301, information 311 of the target is determined.
It should be understood that the application scenario of the method for determining information of an object shown in fig. 3 is only an exemplary description of the method for determining information of an object, and does not represent a limitation of the method. For example, the steps shown in fig. 3 above may be implemented in further detail.
According to the method for determining the information of the target object, the response information content can be directly pushed to the responder based on the current inquiry information, and the responder does not need to select the dialect group and select the dialect, so that the average time of the responder for responding to the responder is reduced, the traditional response process is optimized, the communication efficiency is improved, and the labor cost is saved.
Referring to FIG. 4, a flow diagram of yet another embodiment of a method for determining information of a target object is shown, in accordance with the present application.
As shown in fig. 4, a process 400 of the method for determining information of a target object according to the present embodiment may include the following steps:
In this embodiment, an execution subject (for example, a terminal or a server shown in fig. 1) on which the above-described method for determining information of an object operates may acquire an image of a rectangular frame of an identified object from a local or server. Here, the image in which the rectangular frame of the target object has been recognized may be obtained by detecting and recognizing an image captured by the camera.
In this embodiment, the external parameters of the camera and the external parameters of the lidar may be determined according to the detected target object by an algorithm for calculating the external parameters, and then the external parameters of the camera and the external parameters of the lidar are determined by the algorithm. After determining the extrinsic parameters of the camera and the extrinsic parameters of the lidar, a rotation matrix and a translation matrix of the coordinate system may be determined from the extrinsic parameters of the camera and the extrinsic parameters of the lidar.
And 403, projecting the laser point cloud data points into the image based on the rotation matrix and the translation matrix.
In this embodiment, after the rotation matrix and the translation matrix of the coordinate system are determined in step 402, the laser point cloud data may be projected into the image using the rotation matrix and the translation matrix. The laser point cloud data herein generally includes three-dimensional coordinates (XYZ), and some laser point cloud data may further include laser reflection Intensity (Intensity) and/or color information (RGB).
In this embodiment, based on the laser point cloud data projected into the image in step 403, the point cloud data points projected into the rectangular frame of the target object in the image can be screened. At this time, the determined information of the target object includes point cloud information in a rectangular frame of the target object and image information in the rectangular frame of the target object.
In this embodiment, the transverse axis of the rectangular frame of the target object is taken as the symmetry axis, and the rectangular frame of the target object is equally divided into an upper rectangular frame and a lower rectangular frame, so that the characteristic dense areas of the rectangular frames can be rapidly distinguished.
In step 406, the upper rectangular frame of the upper and lower rectangular frames is determined as the filtered window.
In this embodiment, considering that the upper rectangular frame of the upper and lower rectangular frames generally includes richer information, the upper rectangular frame of the upper and lower rectangular frames may be determined as the filtered window. For example, the upper half of a pedestrian is a head and a trunk, and the point cloud of the upper half is denser and more informative.
In this embodiment, based on the point cloud information in the filtered window, a data point closest to the laser radar in the point cloud may be determined according to the three-dimensional coordinates of the point cloud information.
And step 408, searching in the target object rectangular frame by taking the nearest data point as a reference, and filtering out interference point clouds with the distance from the laser radar larger than a preset threshold value to obtain the point clouds on the target object in the target object rectangular frame.
In this embodiment, when the laser radar scans, the point cloud formed by the scanned target object is closer to the laser radar, and the point cloud farther from the laser radar is usually an interference point cloud such as a background. Therefore, interference data points with the distance from the laser radar larger than a preset threshold value can be filtered from the point cloud data points in the rectangular frame of the target object, and the point cloud above the target object in the rectangular frame of the target object is obtained. The point cloud above the target in the target rectangular frame is the point cloud generated by scanning the target.
In the embodiment, based on the point cloud above the target object, three-dimensional information of the target object in the camera coordinates can be acquired; based on the image of the target object, two-dimensional information of the target object in a camera coordinate system can be acquired; based on the three-dimensional information and the two-dimensional information in the camera coordinate system, more comprehensive three-dimensional information of the target object can be obtained through fusion.
Compared with the embodiment shown in fig. 2, the method for determining the information of the target object in the above embodiment of the application adopts the rotation matrix and the translation matrix of the coordinate system to project the laser point cloud data points into the value image, so that the projection efficiency is improved. In addition, the rectangular frame above the transverse axis of the rectangular frame of the target object is used as the filtering window, so that the calculation amount for determining the filtering window is reduced, and the efficiency for determining the information of the target object is improved. In addition, the data point closest to the laser radar is used as a reference, searching is carried out in the rectangular frame of the target object, interference point clouds with the distance from the laser radar larger than a preset threshold value are filtered, the point clouds on the target object can be obtained, the calculation amount required for determining the information of the target object is reduced, and the efficiency for determining the information of the target object is improved.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, an embodiment of the present application provides an embodiment of an apparatus for determining information of a target object, where the embodiment of the apparatus corresponds to the embodiments of the methods shown in fig. 2 to fig. 4, and the apparatus may be applied to various electronic devices.
As shown in fig. 5, the apparatus 500 for determining information of a target object of the present embodiment may include: an image acquisition unit 510 configured to acquire an image in which a rectangular frame of the target object has been recognized; a point cloud projection unit 520 configured to project the laser point cloud data points into the image based on the external parameters of the camera and the external parameters of the laser radar; a data point screening unit 530 configured to screen point cloud data points projected into a rectangular frame of a target in an image; a window determining unit 540 configured to determine a filtered window based on the target object rectangular frame; a point cloud determining unit 550 configured to filter, based on the point cloud information in the filtering window, interference data points whose distance from the laser radar is greater than a predetermined threshold from the point cloud data points in the target rectangular frame, to obtain a point cloud on the target in the target rectangular frame; an information determination unit 560 configured to determine information of the target object based on the point cloud above the target object and the image of the target object.
In some optional implementations of the present embodiment, the obtained image obtaining unit 510 includes (not shown in the figure): the image detection unit is configured to detect the image acquired by the camera by adopting a target detection algorithm to obtain a rectangular frame to be identified; and the image recognition unit is configured to recognize and frame the object in the rectangular frame to be recognized by adopting a target recognition algorithm to obtain an image of the rectangular frame of the recognized target object.
In some optional implementations of the present embodiment, the point cloud projection unit 520 includes (not shown in the figure): a matrix determination unit configured to determine a rotation matrix and a translation matrix of a coordinate system based on external parameters of the camera and external parameters of the lidar; a data point projection unit configured to project the laser point cloud data points into the image based on the rotation matrix and the translation matrix.
In some optional implementations of the present embodiment, the window determining unit 540 is further configured to: equally dividing a rectangular frame of the target into an upper rectangular frame and a lower rectangular frame by taking a transverse axis as a symmetry axis; and determining the upper rectangular frame of the upper and lower rectangular frames as a filtering window.
In some optional implementations of the present embodiment, the window determining unit 540 is further configured to: and determining a window of the rectangular frame of the target object, which comprises the preset key part of the target object, as a filtering window.
In some optional implementations of the present embodiment, the point cloud determining unit 550 further includes (not shown in the figure): a data point determination unit configured to determine a closest data point to the lidar based on the point cloud information in the filtered window; and the point cloud filtering unit is configured to search in the target object rectangular frame by taking the nearest data point as a reference, filter out interference point clouds of which the distance from the laser radar is greater than a preset threshold value, and obtain the point cloud above the target object in the target object rectangular frame.
In some optional implementations of the present embodiment, the target object in the point cloud determining unit 550 includes: pedestrians, cyclists, vehicles, utility poles, and trash cans.
In some optional implementations of the present embodiment, the information determining unit 560 is further configured to: image information, distance information, and velocity information of the target object are determined based on the point cloud above the target object and the target object.
It should be understood that the elements recited in apparatus 500 may correspond to various steps in the methods described with reference to fig. 2-4. Thus, the operations and features described above for the method are equally applicable to the apparatus 500 and the units included therein, and are not described in detail here.
The embodiment of the present application further provides an unmanned vehicle, including: camera, lidar and an apparatus as in any of the embodiments above.
An embodiment of the present application further provides a robot, including: camera, lidar and an apparatus as in any of the embodiments above.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing a server according to embodiments of the present application. The terminal or server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the method of the present application when executed by a Central Processing Unit (CPU) 601. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an image acquisition unit, a point cloud projection unit, a data point screening unit, a window determination unit, a point cloud determination unit, and an information determination unit. Here, the names of these units do not constitute a limitation to the unit itself in some cases, and for example, the image acquisition unit may also be described as a "unit that acquires an image of a rectangular frame of the object that has been recognized".
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: acquiring an image of the rectangular frame of the identified target object; projecting the laser point cloud data points into the image based on external parameters of a camera and external parameters of a laser radar; screening point cloud data points projected into the target object rectangular frame in the image; determining a filtering window based on the target object rectangular frame; based on the point cloud information in the filtering window, filtering interference data points with the distance from the laser radar larger than a preset threshold value from the point cloud data points in the rectangular frame of the target object to obtain point cloud on the target object in the rectangular frame of the target object; determining information of the target object based on the point cloud above the target object and the image of the target object.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (13)
1. A method for determining information of a target object, comprising:
acquiring an image of the rectangular frame of the identified target object;
projecting the laser point cloud data points into the image based on external parameters of a camera and external parameters of a laser radar;
screening point cloud data points projected into the target object rectangular frame in the image;
determining a filtering window based on the target object rectangular frame;
based on the point cloud information in the filtering window, filtering interference data points with the distance from the laser radar larger than a preset threshold value from the point cloud data points in the rectangular frame of the target object to obtain point cloud on the target object in the rectangular frame of the target object;
determining information of the target object based on the point cloud above the target object and the image of the target object.
2. The method of claim 1, wherein the acquiring of the image of the rectangular frame of the identified target object comprises:
detecting the image collected by the camera by adopting a target detection algorithm to obtain a rectangular frame to be identified;
and identifying and framing the object in the rectangular frame to be identified by adopting a target identification algorithm to obtain an image of the rectangular frame of the identified target object.
3. The method of claim 1, wherein the projecting laser point cloud data points into the image based on camera and lidar extrinsic parameters comprises:
determining a rotation matrix and a translation matrix of a coordinate system based on external parameters of a camera and external parameters of a laser radar;
based on the rotation matrix and the translation matrix, projecting laser point cloud data points into the image.
4. The method of claim 1, wherein the determining a filtered window based on the object rectangle box comprises:
equally dividing the rectangular frame of the target object into an upper rectangular frame and a lower rectangular frame by taking the transverse axis as a symmetry axis;
and determining the upper rectangular frame of the upper and lower rectangular frames as a filtering window.
5. The method of claim 1, wherein the determining a filtered window based on the object rectangle box comprises:
and determining a window of the rectangular frame of the target object, which comprises a preset key part of the target object, as a filtering window.
6. The method of claim 1, wherein the filtering interference data points having a distance from the lidar that is greater than a predetermined threshold from the point cloud data points within the rectangular frame of the target object based on the point cloud information in the filtered window to obtain the point cloud above the target within the rectangular frame of the target object comprises:
determining a data point closest to the laser radar based on the point cloud information in the filtered window;
and searching in the target object rectangular frame by taking the nearest data point as a reference, and filtering out interference point clouds of which the distance from the laser radar is greater than a preset threshold value to obtain the point clouds on the target object in the target object rectangular frame.
7. The method of claim 1, wherein the target comprises: pedestrians, cyclists, vehicles, utility poles, and trash cans.
8. The method of any of claims 1-7, wherein the determining information of the object based on the point cloud over the object and the image of the object comprises:
determining image information, distance information, and speed information of the target object based on the point cloud above the target object and the target object.
9. An apparatus for determining information of a target object, comprising:
an image acquisition unit configured to acquire an image in which a rectangular frame of the target object has been recognized;
a point cloud projection unit configured to project laser point cloud data points into the image based on external parameters of a camera and external parameters of a lidar;
a data point screening unit configured to screen point cloud data points projected into the target object rectangular frame in the image;
a window determination unit configured to determine a filtered window based on the target object rectangular frame;
a point cloud determination unit configured to filter interference data points, the distance from which to the laser radar is greater than a predetermined threshold value, from the point cloud data points in the target object rectangular frame based on the point cloud information in the filtered window, to obtain a point cloud above a target object in the target object rectangular frame;
an information determination unit configured to determine information of the target object based on the point cloud above the target object and the image of the target object.
10. An unmanned vehicle comprising: a camera, a lidar and an apparatus as claimed in claim 9.
11. A robot, comprising: a camera, a lidar and an apparatus as claimed in claim 9.
12. A server, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
13. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910114242.6A CN111563398A (en) | 2019-02-13 | 2019-02-13 | Method and device for determining information of target object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910114242.6A CN111563398A (en) | 2019-02-13 | 2019-02-13 | Method and device for determining information of target object |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111563398A true CN111563398A (en) | 2020-08-21 |
Family
ID=72074112
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910114242.6A Pending CN111563398A (en) | 2019-02-13 | 2019-02-13 | Method and device for determining information of target object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563398A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112270673A (en) * | 2020-11-11 | 2021-01-26 | 珠海格力智能装备有限公司 | Method and apparatus for treating garbage |
| CN112766241A (en) * | 2021-04-07 | 2021-05-07 | 北京三快在线科技有限公司 | Target object identification method and device |
| CN113160324A (en) * | 2021-03-31 | 2021-07-23 | 北京京东乾石科技有限公司 | Bounding box generation method and device, electronic equipment and computer readable medium |
| WO2022067647A1 (en) * | 2020-09-30 | 2022-04-07 | 华为技术有限公司 | Method and apparatus for determining pavement elements |
-
2019
- 2019-02-13 CN CN201910114242.6A patent/CN111563398A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022067647A1 (en) * | 2020-09-30 | 2022-04-07 | 华为技术有限公司 | Method and apparatus for determining pavement elements |
| CN112270673A (en) * | 2020-11-11 | 2021-01-26 | 珠海格力智能装备有限公司 | Method and apparatus for treating garbage |
| CN113160324A (en) * | 2021-03-31 | 2021-07-23 | 北京京东乾石科技有限公司 | Bounding box generation method and device, electronic equipment and computer readable medium |
| CN112766241A (en) * | 2021-04-07 | 2021-05-07 | 北京三快在线科技有限公司 | Target object identification method and device |
| CN112766241B (en) * | 2021-04-07 | 2021-07-30 | 北京三快在线科技有限公司 | Target object identification method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11037305B2 (en) | Method and apparatus for processing point cloud data | |
| CN111563398A (en) | Method and device for determining information of target object | |
| JP2020504358A (en) | Image-based vehicle damage evaluation method, apparatus, and system, and electronic device | |
| CN110379020B (en) | Laser point cloud coloring method and device based on generation countermeasure network | |
| CN111222395A (en) | Target detection method and device and electronic equipment | |
| KR102002024B1 (en) | Method for processing labeling of object and object management server | |
| CN111242097A (en) | Face recognition method and device, computer readable medium and electronic equipment | |
| CN111784774B (en) | Target detection method, target detection device, computer readable medium and electronic equipment | |
| CN103077236B (en) | Portable set realizes the system and method for video knowledge acquisition and marking Function | |
| CN112200067B (en) | Intelligent video event detection method, system, electronic equipment and storage medium | |
| CN110390706B (en) | Object detection method and device | |
| CN111209431A (en) | Video searching method, device, equipment and medium | |
| CA3052846A1 (en) | Character recognition method, device, electronic device and storage medium | |
| CN112270309A (en) | Vehicle access point equipment snapshot quality evaluation method and device and readable medium | |
| WO2024016752A1 (en) | Vehicle damage identification method and apparatus, and electronic device and storage medium | |
| CN111160410B (en) | Object detection method and device | |
| CN111382695A (en) | Method and apparatus for detecting boundary points of object | |
| CN111126159A (en) | Method, apparatus, electronic device, and medium for tracking pedestrian in real time | |
| CN114663871A (en) | Image recognition method, training method, device, system and storage medium | |
| CN113378768A (en) | Garbage can state identification method, device, equipment and storage medium | |
| CN111797266A (en) | Image processing method and apparatus, storage medium, and electronic device | |
| CN110377776B (en) | Method and device for generating point cloud data | |
| CN111310595A (en) | Method and apparatus for generating information | |
| CN110634155A (en) | Target detection method and device based on deep learning | |
| CN113344121B (en) | Method for training a sign classification model and sign classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |