CN117037103A

CN117037103A - Road detection method and device

Info

Publication number: CN117037103A
Application number: CN202311159844.6A
Authority: CN
Inventors: 张建; 周时莹; 王超; 李扬; 谢飞; 洪日; 闫善鑫; 李雅欣; 赵凤凯; 高勇
Original assignee: FAW Group Corp
Current assignee: FAW Group Corp
Priority date: 2023-09-08
Filing date: 2023-09-08
Publication date: 2023-11-10

Abstract

The invention discloses a road detection method and a road detection device. The method comprises the following steps: acquiring an original road image shot by a shooting device arranged on a vehicle, and determining a road area in the original road image to obtain a first road area image; acquiring three-dimensional point cloud data scanned by laser radar equipment, and determining a second road area image based on the three-dimensional point cloud data; determining a target road area image based on a target travel path of the vehicle, the first road area image, and the second road area image; classifying the target road region image based on a pre-trained target classification model to obtain road types, wherein the road types comprise structured roads and unstructured roads. The problem of the road detection accuracy is lower based on the neural network is solved, the accuracy of the road type is improved, and the safety of automatic driving is guaranteed.

Description

Road detection method and device

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a road detection method and apparatus.

Background

In the automatic driving process, a road feasible in front of the road needs to be detected, but objects such as trees and the like near the road can cause great interference to the road in the detected image. Current road detection is mainly based on road characteristics and road models. The road area can be obtained mainly through visual features such as colors, gray scales, textures and the like of roads and by combining clustering and other methods in road area detection based on the road features. However, this method is insensitive to road shape and is susceptible to road shading, water track, and the like. Although road model-based methods are effective in overcoming the effects of road shadows and water marks, they are susceptible to irregular road boundaries and are therefore commonly used for the detection of structured roads.

In the related road detection technical scheme, when the road image is identified, for unstructured roads, the interested areas in the road image are dynamically adjusted based on the road boundaries at two sides, and then the road type is determined through the image classification algorithm of the neural network. However, the road boundaries in unstructured roads are difficult to determine because they are blurred and irregularly shaped.

Disclosure of Invention

The invention provides a road detection method and a road detection device, which are used for improving the accuracy of road type detection.

According to an aspect of the present invention, there is provided a road detection method, the method comprising:

acquiring an original road image shot by a shooting device arranged on a vehicle, and determining a road area in the original road image to obtain a first road area image;

acquiring three-dimensional point cloud data scanned by laser radar equipment, and determining a second road area image based on the three-dimensional point cloud data;

determining a target road area image based on a target travel path of the vehicle, the first road area image, and the second road area image;

classifying the target road region image based on a pre-trained target classification model to obtain road types, wherein the road types comprise structured roads and unstructured roads.

According to another aspect of the present invention, there is provided a road detection apparatus including:

the first road area image determining module is used for acquiring an original road image shot by a shooting device arranged on the vehicle, determining a road area in the original road image and obtaining a first road area image;

the second road area image determining module is used for acquiring three-dimensional point cloud data scanned by the laser radar equipment and determining a second road area image based on the three-dimensional point cloud data;

a target road area image determining module for determining a target road area image based on a target travel path of the vehicle, the first road area image, and the second road area image;

the road type determining module is used for classifying the target road area image based on the pre-trained target classification model to obtain the road type, wherein the road type comprises a structured road and an unstructured road.

According to another aspect of the present invention, there is provided a vehicle including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the road detection method of any one of the embodiments of the invention.

According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a road detection method according to any one of the embodiments of the present invention.

According to the technical scheme provided by the embodiment of the invention, the original road image shot by the shooting device arranged on the vehicle is obtained, the road area in the original road image is determined, the first road area image is obtained, and the occupation ratio of the road area in the image can be increased. And acquiring three-dimensional point cloud data scanned by the laser radar equipment, and determining a second road area image based on the three-dimensional point cloud data, so that the road area can be determined more intuitively and accurately. The target road area image is determined based on the target travel path of the vehicle, the first road area image and the second road area image, and the position of the region of interest is adaptively adjusted in consideration of a possible future travel direction of the vehicle. The target road area image is classified based on a pre-trained target classification model, so that road types are obtained, the road types comprise structured roads and unstructured roads, the problem that the accuracy of road detection based on a neural network is low is solved, the accuracy of the road types is improved, and the safety of automatic driving is guaranteed.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a road detection method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a particular median filtering template window provided in accordance with an embodiment of the present invention;

FIG. 3 is a schematic illustration of a specific initial road grayscale image provided in accordance with an embodiment of the present invention;

FIG. 4 is a schematic illustration of a specific road grayscale image provided in accordance with an embodiment of the present invention;

FIG. 5 is a schematic diagram of a specific road vanishing line and a gray value mean distribution image thereof according to an embodiment of the present invention;

Fig. 6 is a schematic diagram of a target driving path in a specific road fusion image according to an embodiment of the present invention;

FIG. 7 is a flow chart of another road detection method provided according to an embodiment of the present invention;

FIG. 8 is a flowchart of a method for determining a road boundary grid based on an alpha shape algorithm, provided in accordance with an embodiment of the present invention;

FIG. 9 is a schematic diagram of a border wire provided according to an embodiment of the present invention;

FIG. 10 is a schematic illustration of flood filling a binarized projection image according to an embodiment of the present invention;

FIG. 11 is a flowchart of a specific road detection method according to an embodiment of the present invention;

fig. 12 is a block diagram of a road detection apparatus according to an embodiment of the present invention;

fig. 13 is a block diagram of a vehicle according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first" and "second," "target" and "initial," etc. in the description and claims of the present invention and the above-described drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Fig. 1 is a flowchart of a road detection method according to an embodiment of the present invention, where the embodiment is applicable to a scene of road detection based on a road image, and may be executed by a road detection device, and the road detection device may be implemented in a form of hardware and/or software and configured in a processor of a vehicle.

As shown in fig. 1, the road detection method includes the steps of:

s110, acquiring an original road image shot by a shooting device arranged on the vehicle, and determining a road area in the original road image to obtain a first road area image.

The original road image is an original image captured by a capturing device provided in the vehicle, and may be, for example, a front camera provided in the vehicle head. Optionally, the camera can also be a binocular camera, so that more road information can be captured, and the accuracy of a road area is improved. The vehicle logo or the grille mounted in front of the vehicle can also be used for looking around the cameras, images of the cameras are spliced, 360-degree imaging is provided for the vehicle, road conditions are determined, detection of surrounding conditions of the vehicle body can be achieved, and real-time confirmation of the conditions in front of the road is facilitated.

Specifically, in order to detect a road ahead of the vehicle, the photographing device may be provided on the vehicle. The method comprises the steps that a shooting device shoots the front of a vehicle to obtain an original road image, wherein the original road image comprises a road on which the vehicle is running; and detecting the road in the original road image by using methods such as image classification, semantic segmentation, target detection or instance segmentation, and the like, and determining a road area to obtain a first road area image.

In a specific embodiment, determining a road area in an original road image to obtain a first road area image includes:

firstly, carrying out graying treatment and filtering treatment on an original road image to obtain a road gray image.

Specifically, in order to reduce the influence of light intensity and vehicle vibration, the original road image is subjected to graying treatment, and the gray image still reflects the distribution and characteristics of the chromaticity and the highlighting level of the whole and part of the original road image compared with the color image. This has the advantage that the contrast between the road area and the non-road area can be visually enhanced, and the image matrix can be simplified, thereby improving the operation speed.

Optionally, the graying processing method includes any one of a gamma (gamma) correction method, a maximum value method, an average value method, and a weighted average method, which is not specifically limited in this embodiment.

Illustratively, a weighted average is employedAnd carrying out graying treatment on the original road image by using the method to obtain an original road gray image. Specifically, the original road image includes three channels of red (R), green (G) and blue (B), and for each pixel p in the original road image, the Gray value (Gray) of the pixel at the same position in the original road Gray image is determined based on the formula (1) _p )：

Gray _p ＝0.2990×R _p +0.5780×G _p +0.1140×B _p (1)；

Wherein R is _p Is the component value of the pixel p in the R channel in the original road image, G _p Is the component value of the pixel p in the G channel in the original road image, B _p Is the component value of pixel p in the B-channel in the original road image.

Furthermore, because noise exists in both the color image and the gray image, in order to reduce the influence of the noise, the filtering method is used for filtering the original road gray image, and the method has the advantages that the noise of the original road gray image can be restrained under the condition that the detail characteristics of the image are reserved as much as possible, the effectiveness and the reliability of subsequent processing and analysis are enhanced, and the accuracy of road detection is further improved.

Optionally, the filtering method includes any one of mean filtering, gaussian filtering and median filtering, which is not specifically limited in this embodiment.

In a specific embodiment, the initial road gray image is filtered based on median filtering, and for each pixel point in the initial road gray image, the gray value of the pixel point is set as the median of the gray values of all the pixel points in the neighborhood window set by the point, so as to obtain the road gray image. This has the advantage that the median filtering has a good filtering effect on isolated noise pixels (e.g. pretzel noise and impulse noise), maintains the edge characteristics of the original road image, does not significantly obscure the image, and facilitates the determination of the road area.

Illustratively, the initial road gray image (fig. 3) is median filtered based on a median filter template window (see fig. 2) of size 3×3 to obtain the road gray image (see fig. 4). Specifically, for the initial roadEach pixel (p') in the Gray image, the Gray value (Gray) of the pixel at that position is updated based on equation (2) _p,filt )：

Gray _p,filt ＝median{Gray _p' },p'∈W (2)；

Wherein Gray _p’ And W is a median filtering template window taking p' as a central pixel.

Then, a road vanishing line is determined based on the road gradation image, and an initial road area image is determined based on the road vanishing line.

Specifically, for the pixel points in the road gray image, calculating the average value of gray values of the pixel points in each row by row unitsArranging the gray value mean value from small to large, and determining the minimum value of the gray value mean value +.>A corresponding row of pixel points (see fig. 5) to obtain a road vanishing line; and determining all pixel points below the road vanishing line in the road gray level image based on the road vanishing line to obtain an initial road area image. The road detection method has the advantages that the road in the original road image is separated from the part above the road based on the road vanishing line, redundant information is removed, the image area is reduced, the road area image with high accuracy is obtained, the operation time of the target classification model is reduced, and the accuracy of road detection is further improved.

And finally, carrying out connected domain analysis on the initial road region image, and determining the connected domain with the maximum pixel number to obtain a first road region image.

Specifically, the connected domain analysis is performed on the initial road region image, an image region composed of pixel points which are in a preset pixel value range and are adjacent in position in the initial road region image is determined, the connected domain is obtained, the connected domain with the largest pixel number (largest area) is determined, and the connected domain is used as a road region in the initial road region image, so that a first road region image is obtained.

For each pixel point in the initial road area image, the pixel value of the image area in the neighborhood (for example, 4 neighborhood/8 neighborhood) range of the pixel point is determined, if all the pixel values in the neighborhood range are the same as the pixel value of the pixel point, the image area and the area formed by the pixel point are regarded as being communicated, each communicated area in the initial road area image is obtained, the communicated areas with the number of pixels being less than the set number of pixels (for example, 60) are deleted, and the image of the communicated area with the largest number of pixels is taken as the first road area image.

Optionally, the binarization algorithm may be used to perform binarization processing on the initial road area image, and then perform connected domain analysis on the binarized image, where the binarization algorithm includes a bimodal method, a P-parameter method, an iterative method, an Oxford (OTSU) algorithm, and the like. The method has the advantages that the influence of pixel value fluctuation on extraction of different connected domains can be prevented, the accuracy of the first road region image is improved, and the accuracy of road detection is further improved.

Illustratively, the binarization threshold is determined based on an oxford algorithm (also called a maximum inter-class variance algorithm or a maximum inter-class threshold algorithm), the initial road region image is subjected to binarization processing, and the initial road region image is updated with the binarized image. Specifically, the binarization threshold is determined based on the formula (3), assuming that the initial road region image includes only two types of pixels, i.e., a foreground region (road region) pixel and a background region (non-road region) pixel, it is necessary to calculate an optimal threshold that can separate the two types of pixels such that the intra-class variance of the two types of pixels is minimized, i.e., such that the inter-class variance of the two types of pixels is maximized (argmaxtr (S _B ) A) is provided; and then, carrying out binarization processing on pixel values in the initial road area image based on the binarization threshold value, and updating the initial road area image by using the binarized image. The method has the advantage that the connected domain analysis is carried out on the binarized image, so that the influence of the fluctuation of the pixel value on the extraction of different connected domains can be prevented.

Wherein S is _B Is a weighted sum matrix of variances of two classes, ω ₀ The proportion omega of the number of the pixels belonging to the road area in the initial road area image to the total number of the pixels of the initial road area image ₁ For the ratio of the number of pixels belonging to the non-road area in the initial road area image to the total number of pixels in the initial road area image,is the average value of the gray values of the pixels belonging to the road area in the initial road area image, and is +.>Is the average value of gray values of pixels belonging to non-road areas in the initial road area image, ++>The average value of gray values of all pixel points in the initial road area image is obtained.

S120, acquiring three-dimensional point cloud data scanned by the laser radar equipment, and determining a second road area image based on the three-dimensional point cloud data.

Specifically, a laser radar device is further arranged near the shooting device, and the laser radar device scans the surrounding environment of the vehicle to obtain three-dimensional point cloud data comprising the surrounding environment of the vehicle; modeling the three-dimensional point cloud data, and converting the point cloud data into a three-dimensional model with a structure and a shape; and projecting the road region in the three-dimensional model into the pixel coordinate system of the first road region image according to the conversion relation among the laser radar coordinate system, the three-dimensional model coordinate system, the equipment coordinate system of the shooting device and the pixel coordinate system of the first road region image, so as to obtain a second road region image.

Optionally, preprocessing operations such as denoising and registering are performed on the three-dimensional point cloud data, and the three-dimensional point cloud data is updated. The method has the advantages that noise and outliers in the point cloud can be effectively removed, point cloud data simplification is achieved on the basis of maintaining geometric characteristics, and a more accurate three-dimensional model is generated.

Optionally, a Bird's Eye View (BEV) sensor may be used to convert the three-dimensional point cloud data into BEV features, and further identify a road area according to the BEV features, and determine an image corresponding to the road area, so as to obtain a second road area image.

Alternatively, three-dimensional road point cloud data corresponding to the road region in the three-dimensional point cloud data can be directly determined, and the three-dimensional road point cloud data is projected to a pixel coordinate system of the first road region image to obtain the second road region image.

S130, determining a target road area image based on the target running path of the vehicle, the first road area image and the second road area image.

Wherein the target travel path is a path on which the vehicle is about to travel.

Specifically, taking a position at a first set distance from the front of the vehicle as the end point of a current driving road area, adopting a preset target detection algorithm to determine whether an obstacle exists in the first set distance range, removing sampling points falling on the obstacle, connecting the rest points with points in a second set distance range, and deleting connecting lines crossing the obstacle to form an undirected graph; searching in the undirected graph by using a preset path searching algorithm (such as Dijkstra algorithm or A-type algorithm) to determine a feasible path between the current position of the vehicle and the terminal point, and obtaining a target driving path.

In a specific embodiment, the turning radius of the vehicle is determined according to the front wheel angle and the wheelbase of the vehicle; and determining a target running path of the vehicle according to the turning radius, the front wheel rotation angle and the transverse coordinates of the vehicle in a vehicle coordinate system.

First, based on the formula (4), according to the front wheel rotation angle (δ _f ) And the wheelbase (L) _w ) Calculating a turning radius (R) of the vehicle:

then, according to the formula (5)Lateral coordinates (Y _trace ) And longitudinal coordinates (X) _trace ) Mapping relation of (c):

specifically, based on a first road area image and a second road area image, determining an intersection of a road pixel point in the first road area image and a road pixel point in the second road area image to obtain a road intersection image; mapping a target driving path of the vehicle to a pixel coordinate system where the road intersection image is located, and determining the road intersection image comprising the target driving path to obtain a road planning image; in the road planning image, an area image of a preset size with a target driving path as a central line is determined, and a target road area image is obtained. This has the advantage that the determination of the road area based on the future direction of travel of the vehicle enables the position of the road area to be adaptively adjusted taking into account the possible future direction of travel of the vehicle, which can improve the robustness of the road type determination for unstructured roads such as tortuous mountain roads.

In one embodiment, determining a target road area image based on a target travel path of a vehicle, a first road area image, and a second road area image includes: fusing the first road area image and the second road area image to obtain a road fused image; and determining an area of interest corresponding to the road fusion image according to a target driving path of the vehicle, and generating a target road area image based on an area corresponding to the road pixel point in the area of interest.

Specifically, fusing the first road area image and the second road area image to obtain a road fused image; mapping the target driving path to a pixel coordinate system where the road fusion image is located to obtain a path fusion image; the size of the region of interest is preset, and the region of preset size taking the target driving path as the central line in the path fusion image is determined to obtain the region of interest; and determining a region corresponding to the road pixel point of the region of interest in the path fusion image to obtain a target road region image.

Illustratively, the dimensions (i.e., height H and width W) of the rectangular region of interest are determined according to the mounting location of the camera; projecting the target travel path of the vehicle to the pixel coordinate system of the road fusion image (see fig. 6) to obtain the track point coordinates (x _image (y), y); according to the size of the rectangular region of interest and the track point coordinates (x _image The method comprises the steps of (y), y), determining the position (O) of the central position (O (x, y)) of a rectangular region of interest in a road fusion image in a pixel coordinate system of the road fusion image based on a formula (6), and obtaining the region of interest corresponding to the road fusion image; and determining the region corresponding to the road pixel point in the region of interest to obtain a target road region image.

Optionally, binarization processing can be performed on the first road area image and the second road area image respectively to obtain a first road area binary image and a second road area binary image; and fusing the first road area binary image and the second road area binary image, and updating the road fusion image.

Specifically, binarizing the first road area image and the second road area image respectively to obtain a first road area binary image and a second road area binary image; and respectively carrying out filtering treatment on the first road region binary image and the second road region binary image, fusing the filtered first road region binary image and the filtered second road region binary image, and updating the road fused image.

In a specific embodiment, filtering is performed on the first road area binary image and the second road area binary image based on mean filtering, and for each pixel point in the binary image, setting the gray value of the pixel point as the mean value of gray values of all pixel points in the neighborhood window set by the point, so as to obtain a mean filtering image. The larger the window, the better the filtering effect, but the more blurred the image will become, so the size of the neighborhood window needs to be set according to the specific application scenario. The method has the advantages that the mean filtering algorithm is simple and easy to realize, does not need to spend a great deal of time and space, can effectively eliminate noise and improve the image quality.

Exemplary, based on a mean filtering template with a preset size (3×3), mean filtering processing is performed on the first road area binary image and the second road area binary image, and image fusion is performed on the first road area binary image and the second road area binary image after mean filtering to obtain a road fusion image. Specifically, first, for each pixel (p ') in the first road area binary image and the second road area binary image, an average value of Gray values of 9 pixels (including the pixel (p ')) around each pixel is determined based on the formula (7) as a Gray value (Gray after filtering of the pixel (p ') _p,avgflt ) Obtaining a first mean filtered image (B ₁ ) And a second mean filtered image (B ₂ )：

Gray _p,avgflt ＝mean{Gray _p' },p'∈W(7)。

Then, the first mean value filtered image (B) is based on the formula (8) ₁ ) And a second mean filtered image (B ₂ ) Weighting fusion is carried out to obtain a road fusion image (B) _f )：

Wherein w is ₁ And w ₂ Respectively the first mean filtered image (B ₁ ) And a second mean filtered image (B ₂ ) B is a fusion threshold, and the road fusion image (B _f ) And marking the pixels exceeding the threshold value as road pixels, and determining an area image formed by all the road pixels to obtain a target road area image.

Optionally, the connected domain analysis can be performed on the road fusion image to obtain an image area composed of pixel points which are in a preset pixel value range and are adjacent in position in the road fusion image, and the connected domain with the largest pixel number (largest area) is determined and used as the road area in the road fusion image to obtain the target road area image.

For each pixel point in the road fusion image, the pixel value of the image area in the neighborhood 8 neighborhood range of the pixel point is determined, if all the pixel values in the neighborhood 8 range are the same as the pixel value of the pixel point, the image area and the area formed by the pixel point are regarded as being communicated, each communication area in the road fusion image is obtained, the communication areas with the pixel number less than 60 are deleted, and the image of the communication area with the largest pixel number is taken as the target road area image.

And S140, classifying the target road area image based on the pre-trained target classification model to obtain the road type.

Road types include structured roads and unstructured roads.

The object classification model includes at least one of a Residual Network (ResNet), a Network in the Network (Network In Network), an Alexax Network (AlexaNet) visual geometry group (Visual Geometry Group, VGG), a Google learning Network (Google LeNet), a Google learning Network version 4 (Google LeNet-v4, also known as acceptance-v 4), and an X onset (Xacceptance).

In a specific embodiment, the target classification model includes a preset feature extraction module and a preset classifier, and classifies the target road area image based on the pre-trained target classification model to obtain a road type, including:

firstly, extracting texture features of a target road area image based on a preset feature extraction module to obtain texture feature data of the target road area image.

The preset feature extraction module is constructed based on a gray level co-occurrence matrix algorithm. A Gray-level co-occurrence matrix, GLCM, is used to describe the texture of an image by studying the spatial correlation properties of Gray. Specifically, since the texture features are formed by repeatedly appearing gray scale distribution at spatial positions, a certain gray scale relationship exists between gray scale values of two pixel points separated by a second preset distance in the image space, that is, the spatial correlation characteristic of gray scales in the image.

It should be noted that, because the data size of the gray level co-occurrence matrix is large, it is generally not directly used as a texture feature, but a statistic constructed based on the gray level co-occurrence matrix is used as a texture classification feature. Wherein the statistics include energy, entropy, contrast, uniformity, correlation, variance, sum average, sum variance, sum entropy, difference variance, difference average, difference entropy, correlation information measure, and maximum correlation coefficient.

In one embodiment, the texture feature data includes contrast, inverse differential moment, angular second moment, correlation, and entropy.

Wherein the contrast (contrast) is used to determine the distribution of the values of the gray level co-occurrence matrix and the degree of local variation. Calculating the contrast (C) of the gray co-occurrence matrix P (i, j) by the formula (9):

where t=i-j, (L-1) is the number of gradations of the target road area image.

The contrast is used for reflecting the definition of the image and the depth of the grooves of the texture, and the deeper the grooves of the texture, the larger the contrast, the clearer the effect; otherwise, if the contrast value is small, the grooves are shallow, and the effect is blurred. If the gray values of two adjacent pixels in the target road area image are very different (i.e. the absolute value of t is very large), the more compact the texture of the image is, the larger the contrast value is.

The inverse differential moment (inverse different moment), also called inverse differential moment, is used to quantify the local variation of the image texture, reflecting the homogeneity of the image texture. The larger the inverse difference moment, the lack of variation between different areas of texture in the image, and the local uniformity of the image. Calculating an inverse differential moment (D) of the gray level co-occurrence matrix P (i, j) by the formula (10):

the angular second moment (angular second moment, ASM), also known as energy, is used to quantify the image gray distribution uniformity and texture thickness. If the element values of the gray level co-occurrence matrix are similar, the energy is smaller, and the graying is fine; if some of the values are large and others are small, the energy value is large. Calculating the square sum of the values of each matrix element in the gray level co-occurrence matrix through a formula (11) to obtain an angular second moment (E) of the gray level co-occurrence matrix P (i, j):

autocorrelation (autocorrelation) is used to quantify the degree of similarity of the gray levels of an image in the row or column direction, reflecting the local gray level correlation of an image, the greater the value, the greater the correlation. Calculating the correlation (R) of the gray level co-occurrence matrix P (i, j) by the formula (12):

entropy (entropy) is used to quantify the randomness of the amount of information contained in an image. I.e. the complexity of the gray distribution in the image, the greater the entropy value, the more complex the image. Calculating entropy of the gray level co-occurrence matrix by the formula (13):

Illustratively, a gray level co-occurrence matrix of the target road region image in four generation directions of 0 °, 45 °, 90 °, and 135 ° is determined, the gray level co-occurrence matrix including probabilities of occurrence of pairs of pixel points having gray values i and j in the corresponding generation directions, respectively.

Further, the position offset of the pixel point pair is determined based on the formula (14) to obtain the position relation of the pixel point pair,

where θ represents the direction of generation of the gradation co-occurrence matrix.

Specifically, for each gray level co-occurrence matrix in the generated direction, calculating preset indexes of the gray level co-occurrence matrix, wherein the preset indexes comprise entropy, contrast, inverse differential moment, angular second moment and correlation, averaging characteristic values in the four directions, and determining a final preset index to obtain texture characteristic data of the target road region image.

And inputting the texture feature data into a preset classifier to obtain the road type corresponding to the road area image.

The preset classifier is used for classifying the texture feature data and comprises algorithms such as decision trees, logistic regression, naive Bayes, neural networks and the like.

In a specific embodiment, firstly, training a target classification model by taking various road area images as samples and road types as labels to obtain a trained target classification model; inputting the target road area image into a trained target classification model, and extracting texture features of the target road area image by a preset feature extraction module based on a gray level co-occurrence matrix algorithm to obtain texture feature data of the target road area image; inputting the texture feature data into a preset classifier, wherein the preset classifier is used for classifying the texture feature data, determining the road type and outputting the road type. Optionally, a network structure for image scaling may be further provided in the target classification model, where the network is specifically configured to scale the target road area image to a preset size, and update the target road area image. The method has the advantages that when the size of the first image is larger than the preset size, the first image is scaled to obtain the image with the preset size, and therefore calculation cost and prediction time of the model can be reduced.

Illustratively, the preset classifier includes a support vector machine (Support Vector Machine, SVM) classifier. Specifically, a public data set of road images is obtained, 80% of road images in the public data set are used as training sets, and 20% of road images in the public data set are used as verification sets; taking the road images in the training set as samples, taking the road type corresponding to each road image as a label, and training the target classification model; verifying the target classification model by using the verification set until the accuracy of the target classification model reaches a preset accuracy to obtain a pre-trained target classification model; inputting the target road area image into a trained target classification model, scaling the target road area image to the same size as the road image in the training set, and updating the target road area image based on the scaled image; inputting the target road region image into a target classification model, and extracting texture features of the target road region image in four directions of 0 degree, 45 degree, 90 degree and 135 degree based on a gray level co-occurrence matrix algorithm by a preset feature extraction module to obtain texture feature data; and inputting the texture feature data in the four directions into an SVM classifier, and outputting the road type.

Optionally, determining and selecting the road image with the same size as the image of the target road area in the public data set to train the target classification model, and ensuring the consistency of the resolution of the road image. The method has the advantages that influences of different sizes of road areas in images of different sizes and interference of noise distribution are reduced, and accuracy of the target classification model is improved.

Alternatively, the small-size road image is used for training the target classification model, and the large-size road image is inferred based on the pre-trained target classification model, so that the training speed of the model can be improved.

According to the technical scheme, the road area is determined based on the three-dimensional point cloud data and the road image, the target driving path of the vehicle is predicted by combining the steering angle, the final road area is determined according to the future driving direction of the vehicle, the position of the road area can be adaptively adjusted according to the future possible driving direction of the vehicle, the problem of low road detection accuracy based on the neural network is solved through the accuracy of the target road area image, the accuracy of the road type is improved, and the safety of automatic driving is guaranteed.

Fig. 7 is a flowchart of another road detection method according to an embodiment of the present invention, where the embodiment is applicable to a scene of road detection based on a road image, and the embodiment belongs to the same inventive concept as the road detection method in the above embodiment, and further describes a process of determining a second road area image based on three-dimensional point cloud data based on the above embodiment.

As shown in fig. 7, the road detection method includes:

s210, acquiring an original road image shot by a shooting device arranged on a vehicle, and determining a road area in the original road image to obtain a first road area image.

S220, acquiring three-dimensional point cloud data scanned by the laser radar equipment, constructing a grid map based on the three-dimensional point cloud data, and determining road boundary grids in the grid map.

Specifically, a three-dimensional space is determined, wherein the three-dimensional space comprises three-dimensional spaces of all three-dimensional point cloud data scanned by the laser radar equipment, and a three-dimensional grid is constructed aiming at the three-dimensional space to obtain a grid map; determining a normal vector of point cloud in each grid in the grid map to obtain a normal vector angle; and determining grids with the dispersion degree exceeding a threshold value according to the dispersion degree (such as standard deviation) of the normal vector angles in the grids, and obtaining the road boundary grids.

In a specific embodiment, the front of the vehicle is taken as the positive direction of the X axis, the right side of the vehicle is taken as the positive direction of the Y axis, and the upper side of the vehicle is taken as the positive direction of the Z axis, so that the three-dimensional space of the first preset distance range is determined; and determining coordinates of the three-dimensional point cloud data in the three-dimensional space according to the three-dimensional space corresponding to the grid map and the laser radar coordinate system. The values of the X axis, the Y axis and the Z axis are respectively [ -30, 30] in meters, and the three-dimensional space is divided into grids, wherein the size of each grid is 0.2X 0.2m, so as to obtain a grid map; determining the normal vector of point cloud in each grid aiming at the three-dimensional point cloud data in each grid to obtain a normal vector angle; and determining grids with the dispersion degree exceeding a threshold value according to the dispersion degree (such as standard deviation) of the normal vector angles in the grids, and obtaining the road boundary grids.

Further, determining a road boundary grid in the grid map includes: and determining grids belonging to the road boundary according to the three-dimensional point cloud data corresponding to the grids and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grids aiming at each grid in the grid map to obtain the road boundary grid.

Specifically, for each grid in a grid map, determining a normal vector of point cloud in the grid to obtain a normal vector angle mean value of the point cloud in the grid; determining a difference value between a normal vector angle mean value of the point cloud in the grid and a normal vector angle mean value of at least one point cloud in the grid adjacent to the grid; when the difference exceeds a preset angle difference, the grid belongs to the road boundary grid.

Optionally, determining the grid belonging to the road boundary according to the three-dimensional point cloud data corresponding to the grid and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grid, and obtaining the road boundary grid includes: determining grid height differences between the three-dimensional point cloud data corresponding to the grid and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grid; and if the grid height difference is larger than the preset difference value, taking the grid as a road boundary grid.

Specifically, for each grid in the grid map, determining the height values of all point clouds in the grid, and obtaining a preset statistical index (mean/median/mode) of the point cloud height values as the grid height value of the grid; determining a difference value between the grid height value of the grid and the grid height value of at least one adjacent grid of the grid to obtain a grid height difference; and if the grid height difference is larger than the preset difference value, taking the grid as a road boundary grid.

In a specific embodiment, for each grid (i) in the grid map, determining the height values of all point clouds in the grid to obtain a mean value of the point cloud height values, and taking the mean value as the height value of the grid; arranging all grids from small to large according to the height values of the grids to obtain a sequencing result; traversing each grid (i) in the order of the grids in the ranking result, determining the height value (Z) of the next grid (i+1) of the grid in the ranking result _i+1 ) To obtain the height value (Z _i ) Height value (Z) to next grid (i+1) _i+1 ) Is a grid height difference of (2); if the grid height difference is smaller than or equal to the preset height difference (for example, 1.5 m), taking the grid as an initial road boundary grid; for each grid in the grid map, determining the height values of four adjacent (up, down, left, right) grids of the grid to obtain the height value (Z _i ) Grid height difference from the height value of the adjacent grid in each direction; if the grid height difference in one direction is larger than the preset difference, the grid is used as an initial road boundary grid, and all the initial road boundary grids form the road boundary grid. The method has the advantages that the road boundary grids are determined according to the height difference, trees near the road can be eliminated, the accuracy of the road boundary grids is improved, and the accuracy of road detection is further improved.

Optionally, the road boundary grid is further screened based on a point cloud contour extraction algorithm, which includes an edge detection algorithm and an alpha shape algorithm (also called a rolling sphere method).

Illustratively, the road boundary grid is further filtered based on an alpha shape algorithm. Specifically, a circle of a set radius is determined, around which point cloud data in the road boundary grid is scrolled. If the set radius is small enough, each point in the point cloud is a boundary point; if properly increased to a certain extent, it only scrolls on the point to the road boundary, and the track of the scroll is the road boundary. The method has the advantages that the road boundary can be extracted rapidly and accurately, and the influence of the shape of the boundary point in the point cloud data is reduced.

FIG. 8 is a flowchart of a method for determining a road boundary grid based on an alpha shape algorithm according to an embodiment of the present invention. As shown in fig. 8, the method includes:

s1, a first grid set P formed by the grids of the road boundary ₀ The threshold value α is set in advance.

S2, traversing each grid in the road boundary grids to serve as a first grid p ₀ 。

S3, calculating a first grid p ₀ With the first grid set P ₀ Other grids inDetermining all the grids with the distance less than 2α to obtain a second grid set Q ₀ 。

Specifically, construct with p ₀ Determining all road boundary grids in the circle as the circle center and with the radius of 2 alpha to obtain a second grid set Q ₀ 。

S4, determining a second grid set Q ₀ Dividing the first grid p ₀ A road boundary grid other than the first one, a second grid p is obtained ₁ 。

S5, determining the first grid p to pass through ₀ And a second grid p ₁ And the circle centers of two circles with the radius alpha are obtained to obtain a first circle center O ₁ And a second center of circle O ₂ 。

S6, determining a second grid set Q ₀ Except for the first grid p ₀ And a second grid p ₁ To obtain a third grid set S ₀ 。

S7, aiming at the third grid set S ₀ Each road boundary grid in the road map, determining the road boundary grid to the first circle center O ₁ Is greater than the first distance d ₁ And the road boundary grid to the second circle center O ₂ Is a second distance d of (2) ₂ 。

S8, respectively determining the first distance d ₁ And a second distance d ₂ Whether greater than a threshold alpha.

S9, determining a first grid p ₀ And a second grid p ₁ The road boundary grid is updated for the road boundary grid.

Specifically, if the first distance d ₁ From a second distance d ₂ Is greater than a threshold value alpha, determines a first grid p ₀ And a second grid p ₁ The road boundary grid is updated for the road boundary grid.

And S230, dividing the grid map according to the road boundary grids to obtain the road area grids.

Specifically, the road boundary grids are used as cutting lines to divide the grid map; and determining the regional grids comprising the road boundary grids and the grid map coordinate origin grids to obtain the road regional grids.

And S240, projecting the road area grid to a pixel coordinate system where the original road image is located to obtain a projection image, and determining a second road area image based on the projection image.

Specifically, according to the installation positions of the laser radar and the shooting device, determining a first coordinate conversion relation between a laser radar coordinate system and a camera coordinate system; determining a second coordinate conversion relation between a pixel coordinate system of the original road image and a camera coordinate system according to the size of a single pixel in the original road image; determining a target conversion relation between a laser radar coordinate system and a pixel coordinate system of an original road image according to the first coordinate conversion relation and the second coordinate conversion relation; based on the target conversion relation, projecting the road area grid into a pixel coordinate system where an original road image is positioned to obtain a projection image; and determining a road area in the projection image to obtain a second road area image.

Illustratively, first, for each point, a three-dimensional coordinate (X _C ,Y _C ,Z _C ) And the three-dimensional coordinates (X _L ,Y _L ,Z _L ) The method comprises the steps of carrying out a first treatment on the surface of the And (3) calibrating a laser radar coordinate system and a 3 x 3 rotation matrix (R) and a 3 x 1 translation matrix (T) of the camera based on a formula (15) according to the installation positions of the camera and the laser radar, so as to obtain a first coordinate conversion relation of the laser radar coordinate system and the camera coordinate system, and realize coordinate conversion of the laser radar coordinate system and the camera coordinate system.

Next, for each point, the coordinates (x, y) of that point in the pixel coordinate system of the camera imaging plane (original road image) are determined, the focal length f of the camera, the length 2x of the original road image is determined ₀ Width 2y of original road image ₀ A single pixel is at the length dx and width dy of the original road image; and (3) carrying out coordinate conversion on the camera coordinate system and the original road image pixel coordinate system based on the formula (16) to obtain a second coordinate conversion relation.

Then, a target conversion relationship between the lidar coordinate system and the pixel coordinate system of the original road image is determined based on the formula (17):

finally, based on the target conversion relation, projecting the road area grid into a pixel coordinate system where the original road image is positioned to obtain a projection image; and determining a road area in the projection image to obtain a second road area image.

In a specific embodiment, determining the second road area image based on the projection image comprises:

first, road boundary pixel points corresponding to a road boundary grid in a projection image are acquired.

It will be appreciated that the road boundary in the projected image may not be continuous in the pixel coordinate system in which the road area grid is projected onto the original road image, and therefore, in order to make the road boundary in the projected image continuous, it is necessary to determine the road boundary pixel points in the acquired projected image corresponding to the road boundary grid.

Specifically, according to the coordinate conversion relation between the pixel coordinate system of the projection image and the laser radar coordinate system, determining the pixel point corresponding to each road boundary grid in the projection image to obtain the road boundary pixel point.

And then, establishing a connection line of two end point pixel points of the road boundary pixel points to obtain at least one boundary connection line.

It will be appreciated that a road boundary has a set of road boundary pixels. Specifically, at least one group of road boundary pixel points is determined according to the positions of the road boundary pixel points; for each group of road boundary pixel points, two end point pixel points of the road boundary pixel points are determined, and a connecting line of the two end point pixel points is established, so that a boundary connecting line is obtained (see fig. 9).

Illustratively, two groups of road boundary pixel points are obtained according to the positions of the road boundary pixel points; and determining two end point pixel points of each group of road boundary pixel points, and establishing a connecting line of the two end point pixel points to obtain a boundary connecting line.

Alternatively, two end points of the road boundary grid can be directly connected by a straight line, a boundary connecting line is determined, and an image pixel point through which the boundary connecting line passes is taken as a boundary pixel point.

And finally, performing image filling on the boundary connecting lines based on a preset image filling algorithm to obtain a second road area image, wherein the preset image filling algorithm comprises a flood filling algorithm.

It will be appreciated that for a scene where only one road boundary or irregular road boundary shape exists in the projected image, the boundary lines need to be filled by an image filling algorithm.

Specifically, in the projection image, determining seed points (x ', y') of a flood filling algorithm according to the installation position of the shooting device; and filling flood aiming at the boundary connecting lines and the seed points, determining a road area, and obtaining a second road area image.

In a specific embodiment, fig. 10 is a schematic diagram of flood filling a binary projection image according to an embodiment of the present invention. Specifically, for pixel points in the binarized projection image, sequentially labeling the pixel points from the upper left corner to the lower right corner to obtain a labeling result; determining a pixel point corresponding to the median of the label in the label result to obtain a seed point; starting from the seed point, extracting all pixel points which are communicated with the seed point in a preset range (4 neighborhood/8 neighborhood) near the seed point until all pixel points in a closed area corresponding to the boundary connecting line are filled, and separating (or respectively dyeing into different colors) a communication area corresponding to the boundary connecting line from other adjacent areas to obtain a filling area; and determining an image formed by all pixel points of the filling area to obtain a second road area image.

S250, determining a target road area image based on the target travel path of the vehicle, the first road area image, and the second road area image.

S260, classifying the target road area image based on the pre-trained target classification model to obtain road types, wherein the road types comprise structured roads and unstructured roads.

Fig. 11 is a flowchart of a specific road detection method according to an embodiment of the present invention, as shown in fig. 11, S300, S302, and S311 may be performed simultaneously, where the road detection method includes:

s300, acquiring an original road image, preprocessing the original road image, and updating the original road image.

Wherein the preprocessing comprises graying processing and median filtering processing.

S301, determining a road vanishing line based on an original road image; and dividing the original road image according to the road vanishing line to obtain a first road region image.

S302, acquiring three-dimensional point cloud data, and constructing a three-dimensional grid based on the three-dimensional point cloud data to obtain a grid map.

S303, determining a road boundary grid based on the elevation of the three-dimensional point cloud data in the grid map; and dividing the grid map according to the road boundary grids to obtain the road area grids.

S304, determining a coordinate conversion relation between the laser radar coordinate system and a pixel coordinate system of the original road image.

S305, projecting the road area grid into the pixel coordinate system of the original road image according to the coordinate conversion relation between the laser radar coordinate system and the pixel coordinate system of the original road image, and obtaining a projection image.

And S306, determining a road boundary line based on the road boundary pixel points in the projection image, and filling flood into the road boundary line to obtain a second road area image.

S307, performing image fusion on the first road area image and the second road area image to obtain a road fusion image.

S308, determining a target travel path of the vehicle based on the steering angle of the vehicle.

S309, determining an interested region in the road fusion image based on the target driving path, and removing non-road pixel points in the interested region to obtain a target road region image.

S310, extracting texture features of the target road area image to obtain texture feature data of the target road area image.

S311, acquiring a public data set of the road image, and determining a training data set.

S312, extracting texture features from the road images in the training data set to obtain sample feature data.

S313, training the initial SVM classifier based on the sample feature data and the road type label file of the road image corresponding to the sample feature data to obtain the target SVM classifier.

S314, inputting the texture feature data of the target road area image into a target SVM classifier, and outputting the road type.

According to the technical scheme, the grid map is constructed aiming at the three-dimensional point cloud data, the three-dimensional point cloud is projected into the pixel coordinate system where the road image is located based on the road region segmentation method of the point cloud elevation in the grid map, the problem that the accuracy is low in the existing road detection method is solved, the periphery of the power transmission line can be monitored in real time, the range of feature matching is reduced, the data volume of feature matching is reduced, the image registration speed is increased, and the speed and accuracy of power transmission line detection are further increased.

Fig. 12 is a block diagram of a road detection device according to an embodiment of the present invention, which is applicable to a scene of road detection based on road images, and the device may be implemented in hardware and/or software, and integrated into a processor of a vehicle.

As shown in fig. 12, the road detection apparatus includes:

The first road area image determining module 601 is configured to obtain an original road image captured by a capturing device provided in a vehicle, determine a road area in the original road image, and obtain a first road area image; the second road area image determining module 602 is configured to obtain three-dimensional point cloud data scanned by the laser radar device, and determine a second road area image based on the three-dimensional point cloud data; a target road area image determination module 603 for determining a target road area image based on a target travel path of the vehicle, the first road area image and the second road area image; the road type determining module 604 is configured to classify the target road area image based on the pre-trained target classification model, so as to obtain a road type, where the road type includes a structured road and an unstructured road. The problem of the road detection accuracy is lower based on the neural network is solved, the accuracy of the road type is improved, and the safety of automatic driving is guaranteed.

Optionally, the first road area image determining module 601 includes a first road area image determining sub-module, and the first road area image determining sub-module is specifically configured to:

carrying out graying treatment and filtering treatment on the original road image to obtain a road gray image;

Determining a road vanishing line based on the road gray image, and determining an initial road area image based on the road vanishing line;

and carrying out connected domain analysis on the initial road region image, and determining the connected domain with the maximum pixel number to obtain a first road region image.

Optionally, the second road area image determining module 602 includes a second road area image determining sub-module, which is specifically configured to:

constructing a grid map based on the three-dimensional point cloud data, and determining road boundary grids in the grid map;

dividing the grid map according to the road boundary grids to obtain road area grids;

and projecting the road area grid to a pixel coordinate system where the original road image is positioned to obtain a projection image, and determining a second road area image based on the projection image.

Optionally, the second road area image determining sub-module includes a road boundary grid determining unit, which is specifically configured to:

and determining grids belonging to the road boundary according to the three-dimensional point cloud data corresponding to the grids and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grids aiming at each grid in the grid map, and obtaining the road boundary grids.

Optionally, the road-boundary-grid determining unit includes a road-boundary-grid determining subunit specifically configured to:

determining grid height differences between the three-dimensional point cloud data corresponding to the grid and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grid;

and if the grid height difference is smaller than or equal to the preset difference value, taking the grid as the road boundary grid.

Optionally, the second road area image determining sub-module includes a road boundary grid projection unit, and the road boundary grid projection unit is specifically configured to:

obtaining road boundary pixel points corresponding to the road boundary grids in the projection image;

establishing a connection line of two endpoint pixel points of the road boundary pixel points to obtain at least one boundary connection line;

and performing image filling on the boundary connecting lines based on a preset image filling algorithm to obtain a second road area image, wherein the preset image filling algorithm comprises a flood filling algorithm.

Optionally, the target road area image determining module 603 includes a target road area image unit, where the target road area image unit is specifically configured to:

fusing the first road area image and the second road area image to obtain a road fused image;

And determining an area of interest corresponding to the road fusion image according to a target driving path of the vehicle, and generating a target road area image based on an area corresponding to the road pixel point in the area of interest.

Optionally, the target road area image determining module 603 further includes a target travel path determining unit, which is specifically configured to:

determining the turning radius of the vehicle according to the front wheel corner and the wheelbase of the vehicle;

and determining a target running path of the vehicle according to the turning radius, the front wheel rotation angle and the transverse coordinates of the vehicle in a vehicle coordinate system.

Optionally, the road type determination module 604 includes a texture feature classification sub-module, which is specifically configured to:

extracting texture features of the target road area image based on a preset feature extraction module to obtain texture feature data of the target road area image, wherein the preset feature extraction module is constructed based on a gray level co-occurrence matrix algorithm;

The road detection device provided by the embodiment of the invention can execute the road detection method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Fig. 13 is a block diagram of a vehicle according to an embodiment of the present invention, and as shown in fig. 13, the vehicle 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., in which a computer program executable by the at least one processor is stored, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the vehicle 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

Various components in the vehicle 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard or a mouse; an output unit 17 such as various types of displays or speakers, etc.; a storage unit 18 such as a magnetic disk or an optical disk; and a communication unit 19 such as a network card, modem or wireless communication transceiver, etc. The communication unit 19 allows the vehicle 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunications networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), any suitable processor, controller or microcontroller, and the like. The processor 11 performs the respective methods and processes described above, such as a road detection method.

In some embodiments, the road detection method may be implemented as a computer program, which is tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the vehicle 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the road detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the road detection method in any other suitable way (e.g. by means of firmware).

Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine or partly on the machine, partly on the machine and partly on a remote machine or entirely on the remote machine or server as a stand-alone software package.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here may be implemented on a vehicle 10 having: a display device (e.g., CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display ) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the vehicle 10. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.

The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. A road detection method, characterized by comprising:

And classifying the target road region image based on a pre-trained target classification model to obtain road types, wherein the road types comprise structured roads and unstructured roads.

2. The method of claim 1, wherein the determining the road region in the original road image to obtain a first road region image comprises:

and carrying out connected domain analysis on the initial road region image, and determining the connected domain with the largest pixel number to obtain a first road region image.

3. The method of claim 1, wherein the determining the second road region image based on the three-dimensional point cloud data comprises:

and projecting the road area grid to a pixel coordinate system where the original road image is located to obtain a projection image, and determining the second road area image based on the projection image.

4. A method according to claim 3, wherein said determining a road boundary grid in said grid map comprises:

and determining grids belonging to a road boundary according to the three-dimensional point cloud data corresponding to the grids and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grids aiming at each grid in the grid map, so as to obtain the road boundary grid.

5. The method according to claim 4, wherein the determining a grid belonging to a road boundary according to the three-dimensional point cloud data corresponding to the grid and the three-dimensional point cloud data corresponding to at least one neighboring grid of the grid, and obtaining the road boundary grid includes:

determining a grid height difference between the three-dimensional point cloud data corresponding to the grid and the three-dimensional point cloud data corresponding to at least one adjacent grid of the grid;

and if the grid height difference is larger than a preset difference value, taking the grid as the road boundary grid.

6. A method according to claim 3, wherein said determining said second road area image based on said projection image comprises:

Acquiring road boundary pixel points corresponding to the road boundary grids in the projection image;

and performing image filling on the boundary connecting lines based on a preset image filling algorithm to obtain the second road area image, wherein the preset image filling algorithm comprises a flood filling algorithm.

7. The method of claim 1, wherein the determining a target road area image based on the target travel path of the vehicle, the first road area image, and the second road area image comprises:

and determining an area of interest corresponding to the road fusion image according to the target driving path of the vehicle, and generating a target road area image based on an area corresponding to the road pixel point in the area of interest.

8. The method of claim 1, wherein determining a target travel path of the vehicle comprises:

determining a turning radius of the vehicle according to the front wheel corner and the wheelbase of the vehicle;

And determining a target running path of the vehicle according to the turning radius, the front wheel corner and the transverse coordinate of the vehicle in a vehicle coordinate system where the vehicle is located.

9. The method of claim 1, wherein the target classification model includes a preset feature extraction module and a preset classifier, and the classifying the target road region image based on the pre-trained target classification model to obtain a road type includes:

10. A road detection apparatus, characterized by comprising:

A target road area image determining module configured to determine a target road area image based on a target travel path of a vehicle, the first road area image, and the second road area image;

the road type determining module is used for classifying the target road area image based on a pre-trained target classification model to obtain road types, wherein the road types comprise structured roads and unstructured roads.