CN115063760A - Vehicle travelable area detection method, device, equipment and storage medium - Google Patents
Vehicle travelable area detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115063760A CN115063760A CN202210554342.2A CN202210554342A CN115063760A CN 115063760 A CN115063760 A CN 115063760A CN 202210554342 A CN202210554342 A CN 202210554342A CN 115063760 A CN115063760 A CN 115063760A
- Authority
- CN
- China
- Prior art keywords
- ray
- dimensional
- travelable
- vehicle
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 89
- 238000005070 sampling Methods 0.000 claims abstract description 114
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000002372 labelling Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 6
- 230000011218 segmentation Effects 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the technical field of automatic driving, and discloses a method, a device, equipment and a storage medium for detecting a vehicle travelable area. The method comprises the following steps: acquiring an image to be detected corresponding to a driving scene, and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene; projecting the sampling point to an image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray; and performing boundary detection on each three-dimensional ray based on ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in the driving scene according to the detection result. The invention improves the detection precision of the suspended object and the object with a longer distance in the automatic driving scene.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to a method, a device, equipment and a storage medium for detecting a vehicle travelable area.
Background
Detecting 3D travelable regions from images is an important component of vision-based unmanned driving scenarios. The 3D travelable region is defined as how far away in the 3D space the obstacle will come to the outside in any direction.
Current detection schemes include: two schemes of image segmentation + projection and direct segmentation under a top view are adopted. Aiming at the image pavement segmentation and projection, firstly, the image is segmented at a pixel level to obtain the separation between the ground and the non-ground. And then directly projecting the edge of the ground to the lower part of the 3D space to form the road surface edge in the 3D space. This approach requires a large amount of computation, requires segmentation of all pixels of the full map, and cannot handle flying objects (e.g., ETC crossbars, etc.) where the detected travelable area is far from the actual travelable area. For the road surface segmentation under the top view, image information is projected to a 3D space, and then the road surface/non-road surface segmentation is carried out under the 3D space (top view). This method is limited by depth estimation errors and the limitations of the method itself, which degrades rapidly at longer distances (e.g., >50m), and does not give effective detection. In summary, the detection method of the vehicle driving area has the problem of insufficient detection accuracy for the suspended object and the object far away.
Disclosure of Invention
The invention mainly aims to solve the technical problem that the detection precision of a detection method for a vehicle travelable area on a suspended object and a distant object is insufficient.
The first aspect of the invention provides a vehicle travelable region detection method, including: acquiring an image to be detected corresponding to a driving scene, and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene; projecting the sampling points to the image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray; and performing boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in a driving scene according to the detection result.
Optionally, in a first implementation manner of the first aspect of the present invention, the acquiring sample points corresponding to each three-dimensional ray emitted by a target vehicle in a driving scene includes: determining a corresponding number of three-dimensional rays emitted by a target vehicle in a driving scene based on a preset resolution, and determining a three-dimensional coordinate of each sampling point on each three-dimensional ray; and selecting a preset number of sampling points from the sampling points on the three-dimensional rays according to the three-dimensional coordinates and the distance of each three-dimensional ray from near to far.
Optionally, in a second implementation manner of the first aspect of the present invention, the projecting the sampling point to the image to be detected by using a preset camera parameter to obtain a ray feature corresponding to each three-dimensional ray includes: extracting multi-scale features of the image to be detected and extracting a position tensor of the preset resolution of the sampling point; and projecting the sampling point to the image to be detected by using preset camera parameters, and fusing the multi-scale features and the position tensor according to a projection result to obtain ray features corresponding to each three-dimensional ray.
Optionally, in a third implementation manner of the first aspect of the present invention, the fusing the multi-scale feature and the position tensor according to the projection result, and obtaining the ray feature corresponding to each three-dimensional ray includes: determining the projection position information of each sampling point in the image to be detected according to the projection result; and according to the projection position information, fusing the multi-scale features and the position tensor of which the projection positions are overlapped to obtain the ray features corresponding to each three-dimensional ray.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the fusing, according to the projection position information, the multi-scale feature and the position tensor whose projection positions are overlapped, and obtaining the ray feature corresponding to each three-dimensional ray includes: combining the multi-scale features to corresponding sampling points according to the projection position information to obtain the content features of each sampling point; fusing the content characteristics of each sampling point with the position tensor to obtain a final characteristic tensor of each sampling point; and generating ray characteristics corresponding to each three-dimensional ray based on the characteristic tensor of each sampling point in each three-dimensional ray.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the performing, based on the ray feature, boundary detection on each three-dimensional ray to obtain a detection result includes: detecting a travelable boundary point corresponding to each three-dimensional ray based on the ray characteristics, and predicting the travelable distance and the boundary parameters of each three-dimensional ray according to the travelable boundary point; and obtaining a detection result corresponding to each three-dimensional ray based on the travelable distance and the boundary parameter.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after the detecting, based on the ray feature, a travelable boundary point corresponding to each three-dimensional ray, the method further includes: acquiring a labeling boundary point corresponding to each three-dimensional ray; calculating the relative distance between the labeling boundary point corresponding to each three-dimensional ray and the travelable boundary point; and adjusting the travelable boundary point corresponding to each three-dimensional ray according to the relative distance to obtain a new travelable boundary point.
A second aspect of the present invention provides a vehicle travelable region detection apparatus including: the acquisition module is used for acquiring an image to be detected corresponding to a driving scene and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene; the projection module is used for projecting the sampling points to the image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray; and the detection module is used for carrying out boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in the driving scene according to the detection result.
Optionally, in a first implementation manner of the second aspect of the present invention, the acquisition module includes: the determining unit is used for determining a corresponding number of three-dimensional rays emitted by a target vehicle in a driving scene based on a preset resolution, and determining the three-dimensional coordinates of each sampling point on each three-dimensional ray; and the selecting unit is used for selecting a preset number of sampling points from the sampling points on the three-dimensional rays according to the three-dimensional coordinates and from near to far of each three-dimensional ray.
Optionally, in a second implementation manner of the second aspect of the present invention, the projection module includes: the extraction unit is used for extracting the multi-scale features of the image to be detected and extracting the position tensor of the preset resolution of the sampling point; and the projection unit is used for projecting the sampling points to the image to be detected by using preset camera parameters, and fusing the multi-scale features and the position tensor according to the projection result to obtain the ray features corresponding to each three-dimensional ray.
Optionally, in a third implementation manner of the second aspect of the present invention, the projection unit is further configured to: determining the projection position information of each sampling point in the image to be detected according to the projection result; and according to the projection position information, fusing the multi-scale features and the position tensor overlapped at the projection position to obtain the ray features corresponding to each three-dimensional ray.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the projection unit is further configured to: combining the multi-scale features to corresponding sampling points according to the projection position information to obtain the content features of each sampling point; fusing the content characteristics of each sampling point with the position tensor to obtain a final characteristic tensor of each sampling point; and generating ray characteristics corresponding to each three-dimensional ray based on the characteristic tensor of each sampling point in each three-dimensional ray.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the detecting module includes: the prediction unit is used for detecting a travelable boundary point corresponding to each three-dimensional ray based on the ray characteristics and predicting the travelable distance and the boundary parameters of each three-dimensional ray according to the travelable boundary point; and the generating unit is used for obtaining a detection result corresponding to each three-dimensional ray based on the travelable distance and the boundary parameter.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the detection module further includes an adjusting unit, configured to: acquiring a labeling boundary point corresponding to each three-dimensional ray; calculating the relative distance between the labeling boundary point corresponding to each three-dimensional ray and the travelable boundary point; and adjusting the travelable boundary point corresponding to each three-dimensional ray according to the relative distance to obtain a new travelable boundary point.
A third aspect of the invention provides a vehicle travelable region detection apparatus including: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the vehicle travelable region detection apparatus to execute the vehicle travelable region detection method described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to execute the vehicle travelable region detection method described above.
According to the technical scheme provided by the invention, a novel neural network architecture is designed, image characteristics and 3D ray projection are combined, and the sampling points of a 3D space are directly projected onto an image by utilizing internal and external parameter information of a camera, so that the overall characteristics of the 3D ray are obtained. And regressing the distance of one travelable region for the ray of each direction based on the overall characteristics. The travelable area can be directly predicted end to end, and the accuracy of predicting the travelable area of the suspended object and the distant object is improved.
Drawings
Fig. 1 is a schematic view of an embodiment of a method for detecting a travelable area of a vehicle according to an embodiment of the present invention;
fig. 2 is a schematic view of another embodiment of the vehicle travelable region detection method in the embodiment of the invention;
fig. 3 is a schematic view of an embodiment of a vehicle travelable region detection apparatus according to an embodiment of the present invention;
fig. 4 is a schematic view of another embodiment of the vehicle travelable region detection apparatus in the embodiment of the invention;
fig. 5 is a schematic view of an embodiment of a vehicle travelable region detection apparatus according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a method, a device, equipment and a storage medium for detecting a vehicle travelable area, which are used for acquiring an image to be detected corresponding to a driving scene and acquiring sampling points corresponding to three-dimensional rays emitted by a target vehicle in the driving scene; projecting the sampling point to an image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray; and performing boundary detection on each three-dimensional ray based on ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in the driving scene according to the detection result. The invention improves the detection precision of the suspended object and the object with a longer distance in the automatic driving scene.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a detailed flow of an embodiment of the present invention is described below, and referring to fig. 1, a first embodiment of a method for detecting a vehicle travelable area according to an embodiment of the present invention includes:
101. acquiring an image to be detected corresponding to a driving scene, and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene;
it is to be understood that the execution subject of the present invention may be a vehicle travelable region detection device, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
In the embodiment, in the driving process of the vehicle, the current driving scene is shot through a camera arranged on the vehicle, and the image to be detected can be obtained; or the image to be detected corresponding to the driving scene obtained by vehicle history shooting; or an image to be detected generated by simulating a driving scene. The image to be detected can be a plurality of frames of images shot by different cameras and different directions in a driving scene, and can also be a single frame of image.
In this embodiment, the vehicle is used as an origin, a plurality of three-dimensional rays are emitted outwards, an entity object in a driving scene is detected, a sampling point in the driving scene can be determined according to a detection result, and the detected sampling point is collected. The sampling points are feedback points corresponding to the entity object in the driving scene scanned by the ray of the laser radar, so that the acquired entity three-dimensional space distribution condition of the driving scene can be restored,
specifically, the target vehicle adopts the arranged multi-line laser radar to emit three-dimensional rays to acquire the overall space profile of the whole driving scene. The multi-line laser radar with 4 lines, 8 lines, 16 lines, 32 lines, 64 lines and 128 lines can be adopted, namely, a plurality of transmitters and receivers are arranged in the vertical direction, and a plurality of rays are obtained through the rotation of the motor, and the more the number of rays is, the more perfect the surface profile of the object is.
102. Projecting the sampling points to the image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray;
in this embodiment, in order to increase the information depth of the relevant features of the travelable region, the laser radar and the camera are combined according to the camera parameters, and the spatial sampling point acquired by the three-dimensional ray is fused with the two-dimensional information of the to-be-seen image, so that the acquired ray features contain richer information, and the detection of a distant object and a suspended object is subsequently performed to predict the travelable region of the vehicle.
In this embodiment, according to the camera parameters, including internal and external parameters, of the image to be detected, the sampling point is converted from the three-dimensional space coordinate to the two-dimensional plane coordinate corresponding to the image to be detected, a projection relationship between the sampling point and the two-dimensional plane coordinate is established, and the sampling point is projected into the image to be detected according to the projection relationship. And then fusing image information of the sampling point and the projection position, and combining the image information by taking the same three-dimensional ray as a unit to obtain the ray characteristics corresponding to each three-dimensional ray.
It should be noted that, a single three-dimensional ray is used as a minimum unit for subsequent detection, and after the sampling points and the image to be detected are projected, the three-dimensional rays to which each sampling point belongs need to be combined to obtain the ray characteristics of each three-dimensional ray.
In addition, after the sampling points are projected to the image to be detected, the ray characteristics of each three-dimensional ray can be obtained by combining the corresponding two-dimensional information; or projecting the sampling points to the image to be detected, fusing to obtain image plane information, and converting into three-dimensional information. I.e. here the ray characteristics can be two-dimensional information as well as three-dimensional information.
103. And performing boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in a driving scene according to the detection result.
In this embodiment, a convolution neural network is used to perform convolution processing and dimension reduction on the ray features, and the relevant features of the travelable region in the ray features are gradually extracted and dimension reduced to determine the boundary of the three-dimensional ray and use the boundary to predict the travelable region for each three-dimensional ray.
Specifically, the ray characteristics of each three-dimensional ray are sequentially input into a convolutional neural network to detect the boundary of each three-dimensional ray, and the obtained detection result includes the position, type, speed and the like of the boundary point. And combining the detection results obtained by each three-dimensional ray, and drawing a boundary outline to obtain a travelable area.
Specifically, if two-dimensional ray characteristics are output, finally detecting the boundary of each three-dimensional ray on an image plane so as to predict a travelable area of a driving scene on the image plane; the travelable region on the image plane can be further converted into a travelable region on the three-dimensional space as a final travelable region.
Specifically, if three-dimensional ray characteristics are input, detecting to obtain a boundary of each three-dimensional ray on a three-dimensional space, and subsequently predicting a travelable area of a driving scene on the three-dimensional space; the travelable region in the three-dimensional space can be further converted into a travelable region in the two-dimensional space to serve as a final travelable region.
In the embodiment of the invention, a novel neural network architecture is designed, image characteristics and 3D ray projection are combined, and the sampling point of a 3D space is directly projected onto an image by utilizing external and internal parameter information of a camera, so that the overall characteristics of the 3D ray are obtained. And regressing the distance of one travelable region for the ray of each direction based on the overall characteristics. The travelable area can be directly predicted end to end, and the accuracy of predicting the travelable area of the suspended object and the distant object is improved.
Referring to fig. 2, a second embodiment of the method for detecting a driving area of a vehicle according to the embodiment of the present invention includes:
201. acquiring an image to be detected corresponding to a driving scene, determining a corresponding number of three-dimensional rays emitted by a target vehicle in the driving scene based on a preset resolution, and determining a three-dimensional coordinate of each sampling point on each three-dimensional ray;
202. according to the three-dimensional coordinates, selecting a preset number of sampling points from the sampling points on each three-dimensional ray from near to far;
in this embodiment, the number of the three-dimensional rays to be emitted is controlled by setting the resolution of the three-dimensional rays scanned by the laser radar, so that the scanning efficiency and the scanning detail degree of the three-dimensional rays are controlled. The lower the resolution, the smaller the number of three-dimensional rays, the higher the scanning efficiency, the lower the scanning detail, and the opposite is true. And finally, acquiring corresponding sampling points with three-dimensional coordinates according to the entity object existing in the three-dimensional ray scanning area.
Specifically, the resolution set here includes a distance, an orientation, a velocity, and an angle, and the number of three-dimensional rays emitted can be controlled by setting the angular resolution and the orientation resolution. For example, an azimuth resolution of 360 ° and an angular resolution of 0.2 ° are provided, and a total of 1800 rays can be emitted (1800 for 360 °/0.2 °).
In this embodiment, the distance range of the three-dimensional ray acquisition is controlled by the distance resolution, and for each three-dimensional ray, the farther the sampling point is from the vehicle, the lower the predicted availability of the sampling point to the current travelable area, and conversely, the higher the availability. In order to reduce the amount of calculation, only the available sampling points with high availability are taken, the number K can be preset, and K sampling points are selected from near to far as sampling points for subsequent detection.
203. Extracting multi-scale features of the image to be detected and extracting a position tensor of the preset resolution of the sampling point;
204. projecting the sampling point to the image to be detected by using preset camera parameters, and fusing the multi-scale features and the position tensor according to a projection result to obtain ray features corresponding to each three-dimensional ray;
in the embodiment, the sampling points are projected on the image to be detected, and the essence of the method lies in that three-dimensional information on the sampling points is converted into two-dimensional information and is fused with image information in the image to be detected so as to enrich the information enrichment degree of each sampling point, so that the accuracy is higher when the travelable region is predicted on each three-dimensional ray in the follow-up process. Therefore, multi-scale features of the image to be detected are extracted at first, and the position tensor of the sampling point is extracted.
Specifically, multi-scale image features including multi-scale features such as size scale, plane scale and semantic scale are extracted through a convolutional neural network, such as a residual neural network and a feature pyramid network. Then, the position tensors of the sampling points are extracted, the position tensors with different sizes are extracted according to different resolution settings, for example, the set azimuth resolution is 360 degrees, the angular resolution is 0.2 degrees, and the preset number is K, the size of the position tensor extracted here is N x1800x K x3, where N is the batch size and 3 represents the x, y, z three-dimensional coordinates of each sampling point.
Specifically, an external reference matrix and an internal reference matrix of camera parameters can be directly cascaded, and projection of sampling points on an image to be detected is directly realized. Wherein, in the projection process, the specific coordinate system conversion process comprises: the laser radar world coordinate system-camera coordinate system-image plane coordinate system-pixel coordinate system, the corresponding coordinate system is converted into: euclidean coordinate system- > homogeneous coordinate system- > euclidean coordinate system.
In addition, the fusing the multi-scale features and the position tensor specifically includes the following steps:
1) determining the projection position information of each sampling point in the image to be detected according to the projection result;
2) and according to the projection position information, fusing the multi-scale features and the position tensor of which the projection positions are overlapped to obtain the ray features corresponding to each three-dimensional ray.
In this embodiment, the projection of the sampling point in the image to be detected needs to be determined first, and then the corresponding multi-scale features and the position tensor are fused. The conversion from the sampling points to the two-dimensional pixels on the image to be detected can be realized through the intrinsic parameters of the camera, then the transformation of the coordinate system is carried out on the whole external coordinate system of the driving scene and the external coordinate system of the camera through the extrinsic parameters of the camera, and the two-dimensional pixels after the conversion of the sampling points are projected onto the coordinate system where the camera is located, so that the projection of the sampling points in the image to be detected is realized. And finally, directly fusing the corresponding multi-scale features and the position tensor according to the projected pixel point of each sampling point.
Furthermore, when the multi-scale features and the position tensor are fused, the multi-scale features can be combined to corresponding sampling points according to the projection position information to obtain the content features of each sampling point; fusing the content characteristics of each sampling point with the position tensor to obtain a final characteristic tensor of each sampling point; and generating ray characteristics corresponding to each three-dimensional ray based on the characteristic tensor of each sampling point in each three-dimensional ray.
In this embodiment, after the sampling points are projected onto the image to be detected, the row and column coordinates of the sampling points on the image plane are obtained. The size is N x1800x K x3, where 3 denotes u, v, flag, where u denotes the column coordinates, v denotes the row coordinates, and flag denotes whether it is within the image to be detected.
Then, the multi-scale features can be collected for each sample point and combined using a collection operator, such as gather _ nd, to finally obtain the content features of N x1800x K × C dimensions. Meanwhile, the position tensor of each sampling point is superposed on the content characteristic, and finally the characteristic tensor of each sampling point of N x1800x K x C dimension is obtained, namely the ray characteristic of each three-dimensional ray.
Specifically, for example, when the multi-scale features are collected on each sampling point by using gather _ nd, params can be adopted to determine the tensor of the search, indeces determine the search of the search, and can be obtained by tf/where (), axis determine the search axis, name determines the operation name, and batch _ dims determines the batch dimension of controlling the indeces.
205. Detecting a travelable boundary point corresponding to each three-dimensional ray based on the ray characteristics, and predicting the travelable distance and the boundary parameters of each three-dimensional ray according to the travelable boundary point;
206. and obtaining a detection result corresponding to each three-dimensional ray based on the travelable distance and the boundary parameters, and determining a travelable area of the target vehicle in a driving scene according to the detection result.
In this embodiment, the travelable boundary point of each three-dimensional ray is detected by using the aforementioned convolutional neural network such as the residual neural network and the feature pyramid network, and the ray features are extracted and reduced in dimension step by step, so that an N x1800x1x 1-dimensional tensor can be obtained finally, and the travelable boundary point of each three-dimensional ray is detected.
In this embodiment, the formal distance from the target vehicle to the boundary point of the travelable area is predicted with the target vehicle as the origin according to the coordinate information of the plane or space where the travelable boundary point is located, and the boundary parameters of each three-dimensional ray, such as the boundary category (static object or dynamic object) and the boundary speed (which may be static, that is, 0), are predicted according to the N x1800x1x 1-dimensional tensor corresponding to the travelable boundary point, so as to obtain the detection result. Finally, reversely drawing the boundary contour of the travelable area according to the predicted travelable distance and the boundary parameters, and determining the travelable area of the target vehicle in the driving scene according to the position information of the boundary contour.
In addition, if the image to be detected is obtained by shooting a driving scene for the vehicle history and is used for training the aforementioned convolutional neural network, the convolutional neural network is iteratively updated here, specifically as follows:
1) acquiring a labeling boundary point corresponding to each three-dimensional ray;
2) calculating the relative distance between the labeling boundary point corresponding to each three-dimensional ray and the travelable boundary point;
3) and adjusting the travelable boundary point corresponding to each three-dimensional ray according to the relative distance to obtain a new travelable boundary point.
In this embodiment, since each three-dimensional ray detects one travelable boundary point in each direction, all travelable boundary points can be regarded as one prediction point set. And simultaneously, marking boundary points as marking point sets, comparing the two point sets, and calculating a relative distance to adjust the travelable boundary points.
In particular, the relative distance between two sets of points is used as a loss function for any point P in the set of predicted points i Taking it and the most true value point in the marked point setSmall L i And taking the distance as loss, and finally obtaining the total loss between the prediction point set and the labeling point set. And meanwhile, setting a loss threshold value to judge whether the convolutional neural network is updated or not so as to adjust the travelable boundary point. The relative distance between the point sets is used as a loss function, so that the network can tolerate the influence of internal and external reference noise.
With reference to fig. 3, the method for detecting a vehicle travelable area in the embodiment of the present invention is described above, and a device for detecting a vehicle travelable area in the embodiment of the present invention is described below, where one embodiment of the device for detecting a vehicle travelable area in the embodiment of the present invention includes:
the acquisition module 301 is configured to acquire an image to be detected corresponding to a driving scene and acquire sampling points corresponding to three-dimensional rays emitted by a target vehicle in the driving scene;
the projection module 302 is configured to project the sampling point to the image to be detected by using preset camera parameters to obtain a ray feature corresponding to each three-dimensional ray;
the detection module 303 is configured to perform boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determine a drivable area of the target vehicle in the driving scene according to the detection result.
In the embodiment of the invention, a novel neural network architecture is designed, image characteristics and 3D ray projection are combined, and the sampling point of a 3D space is directly projected onto an image by utilizing external and internal parameter information of a camera, so that the overall characteristics of the 3D ray are obtained. And regressing the distance of one travelable region for the ray of each direction based on the overall characteristics. The travelable area can be directly predicted end to end, and the accuracy of predicting the travelable area of the suspended object and the distant object is improved.
Referring to fig. 4, another embodiment of the vehicle travelable region detecting apparatus according to the embodiment of the present invention includes:
the acquisition module 301 is configured to acquire an image to be detected corresponding to a driving scene and acquire sampling points corresponding to three-dimensional rays emitted by a target vehicle in the driving scene;
the projection module 302 is configured to project the sampling point to the image to be detected by using preset camera parameters to obtain a ray feature corresponding to each three-dimensional ray;
the detection module 303 is configured to perform boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determine a drivable area of the target vehicle in the driving scene according to the detection result.
Specifically, the acquisition module 301 includes:
the determining unit 3011 is configured to determine, based on a preset resolution, a corresponding number of three-dimensional rays emitted by a target vehicle in a driving scene, and determine a three-dimensional coordinate of each sampling point on each three-dimensional ray;
and the selecting unit 3012 is configured to select, according to the three-dimensional coordinates and according to each three-dimensional ray from near to far, a preset number of sampling points from among the sampling points on the three-dimensional ray.
Specifically, the projection module 302 includes:
the extracting unit 3021 is configured to extract a multi-scale feature of the image to be detected and extract a position tensor of the preset resolution of the sampling point;
and the projection unit 3022 is configured to project the sampling point to the image to be detected by using preset camera parameters, and fuse the multi-scale features and the position tensor according to a projection result to obtain a ray feature corresponding to each three-dimensional ray.
Specifically, the projection unit 3022 is further configured to:
determining the projection position information of each sampling point in the image to be detected according to the projection result;
and according to the projection position information, fusing the multi-scale features and the position tensor of which the projection positions are overlapped to obtain the ray features corresponding to each three-dimensional ray.
Specifically, the projection unit 3022 is further configured to:
combining the multi-scale features to corresponding sampling points according to the projection position information to obtain the content features of each sampling point;
fusing the content characteristics of each sampling point with the position tensor to obtain a final characteristic tensor of each sampling point;
and generating ray characteristics corresponding to each three-dimensional ray based on the characteristic tensor of each sampling point in each three-dimensional ray.
Specifically, the detecting module 303 includes:
a prediction unit 3031, configured to detect a travelable boundary point corresponding to each three-dimensional ray based on the ray characteristics, and predict a travelable distance and a boundary parameter of each three-dimensional ray according to the travelable boundary point;
a generating unit 3032, configured to obtain a detection result corresponding to each three-dimensional ray based on the travelable distance and the boundary parameter.
Specifically, the detecting module 303 further includes an adjusting unit 3033, configured to:
acquiring a labeling boundary point corresponding to each three-dimensional ray;
calculating the relative distance between the labeling boundary point corresponding to each three-dimensional ray and the travelable boundary point;
and adjusting the travelable boundary point corresponding to each three-dimensional ray according to the relative distance to obtain a new travelable boundary point.
Fig. 3 and 4 above describe the vehicle travelable region detection apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the vehicle travelable region detection apparatus in the embodiment of the present invention is described in detail from the perspective of the hardware processing.
Fig. 5 is a schematic structural diagram of a vehicle driving-capable area detection apparatus 500 according to an embodiment of the present invention, where the vehicle driving-capable area detection apparatus 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations in the vehicle travelable region detection apparatus 500. Further, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the vehicle travelable region detection apparatus 500.
The vehicle drivable area detecting apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows service, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the vehicle travelable region detection apparatus structure shown in fig. 5 does not constitute a limitation of the vehicle travelable region detection apparatus, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The present invention also provides a vehicle travelable region detection apparatus, which includes a memory and a processor, the memory having stored therein computer-readable instructions, which, when executed by the processor, cause the processor to execute the steps of the vehicle travelable region detection method in the above-described embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions that, when executed on a computer, cause the computer to execute the steps of the vehicle travelable region detection method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A vehicle travelable region detection method characterized by comprising:
acquiring an image to be detected corresponding to a driving scene, and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene;
projecting the sampling points to the image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray;
and performing boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determining a drivable area of the target vehicle in a driving scene according to the detection result.
2. The method for detecting the travelable area of the vehicle according to claim 1, wherein the step of collecting the sampling points corresponding to the three-dimensional rays emitted by the target vehicle in the traveling scene comprises the following steps:
determining a corresponding number of three-dimensional rays emitted by a target vehicle in a driving scene based on a preset resolution, and determining a three-dimensional coordinate of each sampling point on each three-dimensional ray;
and selecting a preset number of sampling points from the sampling points on the three-dimensional rays according to the three-dimensional coordinates and the distance of each three-dimensional ray from near to far.
3. The method for detecting the travelable area of the vehicle according to claim 1, wherein the projecting the sampling point to the image to be detected by using the preset camera parameters to obtain the ray characteristics corresponding to each three-dimensional ray comprises:
extracting multi-scale features of the image to be detected and extracting a position tensor of the preset resolution of the sampling point;
and projecting the sampling point to the image to be detected by using preset camera parameters, and fusing the multi-scale features and the position tensor according to a projection result to obtain ray features corresponding to each three-dimensional ray.
4. The method according to claim 3, wherein the fusing the multi-scale feature and the position tensor according to the projection result to obtain the ray feature corresponding to each three-dimensional ray comprises:
determining the projection position information of each sampling point in the image to be detected according to the projection result;
and according to the projection position information, fusing the multi-scale features and the position tensor of which the projection positions are overlapped to obtain the ray features corresponding to each three-dimensional ray.
5. The method according to claim 4, wherein the step of fusing the multi-scale feature and the position tensor, which are overlapped at the projection positions, according to the projection position information to obtain the ray feature corresponding to each three-dimensional ray includes:
combining the multi-scale features to corresponding sampling points according to the projection position information to obtain the content features of each sampling point;
fusing the content characteristics of each sampling point with the position tensor to obtain a final characteristic tensor of each sampling point;
and generating ray characteristics corresponding to each three-dimensional ray based on the characteristic tensor of each sampling point in each three-dimensional ray.
6. The vehicle drivable area detection method as claimed in any one of claims 1 to 5, wherein said boundary detection of each of said three-dimensional rays based on said ray features, obtaining a detection result comprises:
detecting a travelable boundary point corresponding to each three-dimensional ray based on the ray characteristics, and predicting a travelable distance and a boundary parameter of each three-dimensional ray according to the travelable boundary point;
and obtaining a detection result corresponding to each three-dimensional ray based on the travelable distance and the boundary parameter.
7. The vehicle travelable region detection method according to claim 6, characterized by, after said detecting the travelable boundary point corresponding to each of the three-dimensional rays based on the ray feature, further comprising:
acquiring a labeling boundary point corresponding to each three-dimensional ray;
calculating the relative distance between the labeling boundary point corresponding to each three-dimensional ray and the travelable boundary point;
and adjusting the travelable boundary point corresponding to each three-dimensional ray according to the relative distance to obtain a new travelable boundary point.
8. A vehicle travelable region detection apparatus characterized by comprising:
the acquisition module is used for acquiring an image to be detected corresponding to a driving scene and acquiring sampling points corresponding to all three-dimensional rays emitted by a target vehicle in the driving scene;
the projection module is used for projecting the sampling points to the image to be detected by using preset camera parameters to obtain ray characteristics corresponding to each three-dimensional ray;
and the detection module is used for carrying out boundary detection on each three-dimensional ray based on the ray characteristics to obtain a detection result, and determining a travelable area of the target vehicle in a driving scene according to the detection result.
9. A vehicle travelable region detection apparatus characterized by comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the vehicle travelable region detection apparatus to execute the steps of the vehicle travelable region detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the steps of the vehicle travelable region detection method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210554342.2A CN115063760A (en) | 2022-05-20 | 2022-05-20 | Vehicle travelable area detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210554342.2A CN115063760A (en) | 2022-05-20 | 2022-05-20 | Vehicle travelable area detection method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115063760A true CN115063760A (en) | 2022-09-16 |
Family
ID=83198763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210554342.2A Pending CN115063760A (en) | 2022-05-20 | 2022-05-20 | Vehicle travelable area detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115063760A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116258822A (en) * | 2023-05-16 | 2023-06-13 | 山东捷瑞数字科技股份有限公司 | Three-dimensional engine boundary defining method, device and storage medium based on meta universe |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109683170A (en) * | 2018-12-27 | 2019-04-26 | 驭势科技（北京）有限公司 | A kind of image traveling area marking method, apparatus, mobile unit and storage medium |
| CN112560774A (en) * | 2020-12-25 | 2021-03-26 | 广州文远知行科技有限公司 | Obstacle position detection method, device, equipment and storage medium |
| CN113160330A (en) * | 2021-04-16 | 2021-07-23 | 上海交通大学 | End-to-end-based camera and laser radar calibration method, system and medium |
| JP2021144677A (en) * | 2020-03-11 | 2021-09-24 | ベイジン バイドゥ ネットコム サイエンス アンド テクノロジー カンパニー リミテッド | Obstacle detection method, device, electronic apparatus, storage medium, and computer program |
-
2022
- 2022-05-20 CN CN202210554342.2A patent/CN115063760A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109683170A (en) * | 2018-12-27 | 2019-04-26 | 驭势科技（北京）有限公司 | A kind of image traveling area marking method, apparatus, mobile unit and storage medium |
| JP2021144677A (en) * | 2020-03-11 | 2021-09-24 | ベイジン バイドゥ ネットコム サイエンス アンド テクノロジー カンパニー リミテッド | Obstacle detection method, device, electronic apparatus, storage medium, and computer program |
| CN112560774A (en) * | 2020-12-25 | 2021-03-26 | 广州文远知行科技有限公司 | Obstacle position detection method, device, equipment and storage medium |
| CN113160330A (en) * | 2021-04-16 | 2021-07-23 | 上海交通大学 | End-to-end-based camera and laser radar calibration method, system and medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116258822A (en) * | 2023-05-16 | 2023-06-13 | 山东捷瑞数字科技股份有限公司 | Three-dimensional engine boundary defining method, device and storage medium based on meta universe |
| CN116258822B (en) * | 2023-05-16 | 2023-08-11 | 山东捷瑞数字科技股份有限公司 | Three-dimensional engine boundary defining method, device and storage medium based on meta universe |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108279670B (en) | Method, apparatus and computer readable medium for adjusting point cloud data acquisition trajectory | |
| US9157757B1 (en) | Methods and systems for mobile-agent navigation | |
| CN106529587B (en) | Vision course recognition methods based on object detection | |
| CN110879994A (en) | Three-dimensional visual inspection detection method, system and device based on shape attention mechanism | |
| KR20180044279A (en) | System and method for depth map sampling | |
| CN111380510B (en) | Repositioning method and device and robot | |
| CN108519102B (en) | Binocular vision mileage calculation method based on secondary projection | |
| CN115049700A (en) | Target detection method and device | |
| CN112739983B (en) | Method for correcting point cloud data and related device | |
| CN111598952A (en) | Multi-scale cooperative target design and online detection and identification method and system | |
| CN109214254B (en) | Method and device for determining displacement of robot | |
| CN114088099A (en) | Semantic relocation method and device based on known map, electronic equipment and medium | |
| CN111798453A (en) | Point cloud registration method and system for unmanned auxiliary positioning | |
| CN115063760A (en) | Vehicle travelable area detection method, device, equipment and storage medium | |
| CN115100616A (en) | Point cloud target detection method and device, electronic equipment and storage medium | |
| CN112219225A (en) | Positioning method, system and movable platform | |
| Wang et al. | Improved LeGO-LOAM method based on outlier points elimination | |
| CN116105721B (en) | Loop optimization method, device and equipment for map construction and storage medium | |
| CN114627395B (en) | Multi-rotor unmanned aerial vehicle angle analysis method, system and terminal based on nested targets | |
| CN115409861A (en) | Laser radar ground point cloud segmentation method, device, equipment and medium based on image processing | |
| CN110728176B (en) | Unmanned aerial vehicle visual image feature rapid matching and extracting method and device | |
| CN107392209B (en) | Device and method for extracting line segments | |
| CN108917768B (en) | Unmanned aerial vehicle positioning navigation method and system | |
| CN113393501A (en) | Method and system for determining matching parameters of road image and point cloud data and related equipment | |
| CN111462321A (en) | Point cloud map processing method, processing device, electronic device and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |