CN116958939A - Method, system and storage medium for detecting road under rainy and foggy weather - Google Patents
Method, system and storage medium for detecting road under rainy and foggy weather Download PDFInfo
- Publication number
- CN116958939A CN116958939A CN202311092272.4A CN202311092272A CN116958939A CN 116958939 A CN116958939 A CN 116958939A CN 202311092272 A CN202311092272 A CN 202311092272A CN 116958939 A CN116958939 A CN 116958939A
- Authority
- CN
- China
- Prior art keywords
- network
- road
- model
- image
- feature extraction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000001514 detection method Methods 0.000 claims abstract description 60
- 230000011218 segmentation Effects 0.000 claims abstract description 55
- 238000000605 extraction Methods 0.000 claims abstract description 39
- 238000012545 processing Methods 0.000 claims abstract description 24
- 238000003062 neural network model Methods 0.000 claims abstract description 14
- 238000011176 pooling Methods 0.000 claims description 27
- 230000004927 fusion Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 11
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000013527 convolutional neural network Methods 0.000 description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 5
- 238000002834 transmittance Methods 0.000 description 5
- 230000016776 visual perception Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000004438 eyesight Effects 0.000 description 4
- 238000013139 quantization Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 230000002146 bilateral effect Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/766—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using regression, e.g. by projecting features on hyperplanes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The application discloses a method, a system and a storage medium for detecting a road under a rainy and foggy weather, wherein the method comprises the following steps: inputting the obtained foggy first road image into an image rain and fog removing neural network model for processing to obtain a second road image; inputting the second road image into a feature extraction network for feature extraction of the second road image; constructing a road target detection model and a travelable region semantic segmentation model; respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result. The application can save calculation resources and ensure the real-time performance and accuracy of calculation.
Description
Technical Field
The application relates to the technical field of intelligent driving, in particular to a method, a system and a storage medium for detecting a road under rainy and foggy weather.
Background
In the field of intelligent driving, vehicles can visually detect a road surface and objects in the road during driving, which is an important function, and development of the field is greatly promoted. The fact that whether a vehicle can detect in real time and accurately in a driving scene under the conditions of rain and fog is still a challenging problem is still a great deal of attention and research.
With the development of the artificial intelligence field, the development of intelligent driving is further promoted by the innovation of computer vision including deep learning. Particularly in the visual perception technology, the deep learning method is applied, and the robot vision can be used for replacing human vision in the running process, so that an intelligent driving system can automatically analyze scenes and perceive objects, and the intelligent driving system is developed towards the directions of more intelligence, practicability and convenience. From the perspective of people's need for intelligent driving vision perception, it has wide research prospect.
The current intelligent driving visual perception technology has some difficulties. The intelligent driving vehicle is usually required to carry a multi-task system, and parallel tasks will cause larger pressure on calculation resources during driving, so that the situation of poor timeliness and accuracy in the detection process is caused, and reliable support cannot be provided for subsequent tasks such as obstacle avoidance and path planning. In addition, the visual perception technology is mostly affected by rain and fog weather, the obtained original image is unclear, and the subsequent detection has larger deviation.
At present, a visual perception algorithm which is relatively commonly used in the intelligent driving field mainly comprises semantic segmentation and target detection. In the running process of the intelligent driving vehicle, the two algorithms are usually operated simultaneously, and then the detection results of the two algorithms are mutually complemented, so that a more reliable detection effect is achieved. However, intelligent driving vehicles often encounter special weather, such as heavy fog, rain and snow, etc., so that the images obtained by the cameras are greatly distorted. In addition, the travelling crane needs to be provided with a multi-task system, and parallel tasks in the travelling process can cause great stress on calculation resources, further cause poor detection timeliness and accuracy, and can not provide reliable support for subsequent tasks such as obstacle avoidance, path planning and the like.
Disclosure of Invention
The application provides a road detection method, a system and a storage medium under rainy and foggy weather, which are used for solving the problems of poor detection timeliness and accuracy in the visual perception algorithm in the prior art, saving calculation resources and ensuring calculation timeliness and accuracy.
In order to achieve the above purpose, the technical scheme adopted by the application is as follows:
a method for detecting a road under a rainy and foggy weather, comprising the following steps:
inputting the obtained foggy first road image into an image rain and fog removing neural network model for processing to obtain a second road image;
inputting the second road image into a feature extraction network for feature extraction of the second road image;
constructing a road target detection model and a travelable region semantic segmentation model;
respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result.
Preferably, the image defogging neural network model adopts an MSCNN neural network, or an image defogging model based on a convolution network DehazeNet, or an image defogging model based on an all-in-one convolution network AOD-Net.
Preferably, the feature extraction network performs feature extraction by combining a residual network with a feature pyramid network, and obtains more detail information through multi-scale feature fusion.
Preferably, the road target detection model comprises a regional suggestion network, a pooling operation network, a target classification network and a regression prediction network;
the regional suggestion network adopts an anchor frame mechanism with a direction and is used for processing a feature map output by the feature extraction network;
the pooling operation network is used for processing the feature graph output by the feature extraction network and the suggestion frame output by the regional suggestion network;
the target classification network is used for predicting object types of the output of the pooling operation network;
the regression prediction network is used for processing the output of the pooling operation network to obtain the coordinates, width, height and direction angle of the center point of the rectangular frame of the first target position.
Preferably, the anchor frame in the regional suggestion network sets a corresponding angle to acquire the target position information.
Further, the pooling operation network adopts a RoI Align operation or a RoIPooling operation.
Preferably, the travelable region semantic segmentation model adopts a Mini-deep Lab module or a deep Lab module; the Mini-deep Lab module comprises three branches, wherein the first branch carries out convolution operation on an input feature map; the second branch also carries out convolution operation on the input feature map, and adopts cavity convolution to enlarge the receptive field; the third branch firstly carries out average pooling operation on the input feature map and then carries out convolution operation; and finally, processing the output of the three branches through a characteristic connecting layer, then entering a convolution operation, and finally outputting a first pixel level foreground segmentation result.
Preferably, the method further comprises:
and carrying out fusion finishing treatment on a first target position output by the road target detection model and a first pixel level foreground segmentation result output by the drivable region semantic segmentation model by an information fusion finishing module based on the multi-layer perceptron MLP to obtain a second target position and a second pixel level foreground segmentation result.
A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, the processor implementing the steps of the method as described above when the computer program is executed.
A computer readable storage medium having stored thereon a computer program which when executed by a processor realizes the steps of the method as described above.
The application has the beneficial effects that:
the application combines the image rain and fog removing neural network model with the road target detection model and the drivable region semantic segmentation model to form a rain and fog weather road detection method. Aiming at instantaneity, the travelable region semantic segmentation model and the road target detection model use the same feature extraction network, wherein the travelable region of the road is extracted through the travelable region semantic segmentation model so as to save calculation resources and ensure instantaneity; for accuracy, firstly, an image rain and fog removing neural network model is adopted to firstly carry out rain and fog removing treatment on a foggy road image, and then detection is carried out through a road target detection model, so that the accuracy of a detection effect is ensured.
Drawings
FIG. 1 is a flow chart of a road detection method under rain and fog weather.
Fig. 2 is an overall block diagram of the rain and fog road target detection model according to the present application.
Fig. 3 is an overall block diagram of the Mini-deep lab semantic segmentation module according to the present application.
Fig. 4 is a block diagram of a parallel network of a road target detection model and a drivable region semantic segmentation model according to the present application.
Detailed Description
Further advantages and effects of the present application will become readily apparent to those skilled in the art from the disclosure herein, by referring to the accompanying drawings and the preferred embodiments. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be understood that the preferred embodiments are presented by way of illustration only and not by way of limitation.
It should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present application by way of illustration, and only the components related to the present application are shown in the drawings and are not drawn according to the number, shape and size of the components in actual implementation, and the form, number and proportion of the components in actual implementation may be arbitrarily changed, and the layout of the components may be more complicated.
Intelligent driving vehicles often encounter special weather, such as heavy fog, rain and snow, etc., so that the images obtained by the cameras are greatly distorted. In addition, the intelligent driving vehicle is also required to be provided with a multi-task system, and a plurality of perception tasks consume a large amount of calculation resources in parallel in the driving process, so that the problems of hysteresis, low accuracy and the like in the road detection process are caused.
According to the embodiment, the image rain and fog removing neural network model is added into the road detection method and is effectively combined with the deep semantic segmentation and the target detection algorithm, so that the network calculation amount is reduced, and the accuracy and timeliness are ensured. Therefore, a method for detecting a road under a rainy and foggy weather in an intelligent rainy and foggy driving scene is provided, and as shown in fig. 1, the method comprises the following steps:
inputting the obtained foggy first road image into an image rain and fog removing neural network model for processing to obtain a second road image;
inputting the second road image into a feature extraction network for feature extraction of the second road image;
constructing a road target detection model and a travelable region semantic segmentation model;
respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result.
The embodiment provides an end-to-end road detection method under the rain and fog weather by combining pixel-level semantic segmentation, which comprises the steps of firstly carrying out pre-processing on an obtained image by using an image rain and fog removing neural network model, and then combining the advantages of the two methods of target detection and semantic segmentation, so that the related task requirements of intelligent driving vehicles can be met in the detection process, and real-time performance and accuracy are considered under the condition of saving computing resources, so as to achieve a more accurate detection result.
In this embodiment, the foggy road image obtained by the camera disposed on the vehicle is an RGB image.
According to the method, the MSCNN neural network with better performance is adopted for the image rain and fog removing neural network model according to the requirement of road detection performance in the intelligent rain and fog driving scene. The MSCNN neural network is a multi-scale convolutional neural network, and comprises a coarse-scale convolutional network and a fine-scale convolutional network; and inputting the foggy road image as an input image into the MSCNN neural network, and outputting the foggy road image as a transmissivity graph corresponding to the input image.
The coarse scale convolutional neural network is used for predicting the overall transmittance map of the scene. The coarse-scale convolutional neural network consists of a multi-scale convolutional layer, a pooling layer, an up-sampling layer and a linear connecting layer.
The fine-scale convolutional neural network is used for refining the rough transmittance map output by the rough-scale convolutional neural network, and a more accurate transmittance map can be obtained by inputting the foggy image and the rough transmittance map into the fine-scale convolutional neural network.
In this embodiment, the image defogging neural network model may also adopt an image defogging model based on a convolutional network DehazeNet or an image defogging model based on an all-in-one convolutional network AOD-Net.
The image defogging model based on the convolutional network DehazeNet comprises the following steps: an input layer, a feature extraction layer, a multi-scale mapping layer, a nonlinear regression layer and an output layer;
wherein the input layer is a hazy image block of size 16 x 16;
the feature extraction layer adopts a structure of combining a convolution layer with an activation layer to realize feature extraction of an input image block;
the multi-scale is composed of convolution layers with different convolution kernel sizes and is used for fully extracting haze features of receptive fields in different ranges in an input feature map;
the nonlinear regression layer adopts a bilateral correction linear unit; the bilateral correction linear unit not only can ensure that bilateral correction is carried out, but also can ensure local linearity;
the output layer is used for outputting a transmissivity graph corresponding to the foggy road image.
The image defogging model based on the all-in-one convolution network AOD-Net is an end-to-end defogging model constructed by combining a plurality of cascaded convolution layers and full connection layers. The image defogging model based on the all-in-one convolution network AOD-Net inputs a foggy road image, outputs a defogged clear image, and does not need to estimate a transmittance value and an atmospheric illumination value independently.
In this embodiment, the feature extraction network performs feature extraction by combining a residual network with a feature pyramid network, and obtains more detail information through multi-scale feature fusion, so as to improve the detection accuracy of the road target detection model in a small object or a object in a visual field far distance.
Wherein, the residual networks are ResNet-50 and ResNet-101, and the characteristic pyramid networks are Featurized image pyramid, single feature map, pyramidal feature hierarchy and Feature Pyramid Network.
In a specific embodiment, the feature extraction network uses ResNet-101 in combination with feature pyramid network FPN for feature extraction.
Wherein, resNet-101 is a depth residual network, representing a residual network with 101 layers, wherein the first layer is a 7×7 convolution layer, followed by 4 stages, each stage comprising several residual blocks; finally, a global average pooling layer and a full connection layer are arranged; each residual block consists of two 3 x 3 convolutional layers, each followed by a batch normalization and ReLU activation function. There are also batch normalization and ReLU activation functions between residual blocks. The first residual block of each stage converts the input channel number to the output channel number using a 1 x 1 convolutional layer for addition with the subsequent residual block.
In this embodiment, the feature extraction network performs feature extraction by combining ResNet-101 with feature pyramid network FPN, so that more detailed information can be obtained, which is very critical for detecting small objects or objects far from the visual field in the intelligent driving scene, and the intelligent driving system of the vehicle can react in advance according to the detected information, so that the safety is improved.
In this embodiment, as shown in fig. 2, the road target detection model includes a regional suggestion network, a pooling operation network, a target classification network, and a regression prediction network;
the regional suggestion network adopts an anchor frame mechanism with a direction and is used for processing a feature map output by the feature extraction network;
the pooling operation network is used for processing the feature graph output by the feature extraction network and the suggestion frame output by the regional suggestion network;
the target classification network is used for predicting object types of the output of the pooling operation network;
the regression prediction network is used for processing the output of the pooling operation network to obtain coordinates (x, y) of a center point of a rectangular frame of the first target position, a width w, a height h and a direction angle theta.
In this embodiment, the anchor frame in the regional suggestion network sets a corresponding angle to obtain the target location information. The anchor frame is arranged at an angle of + -15 DEG, -45 DEG and + -75 DEG, so that more accurate position information is obtained.
In this embodiment, the pooling network uses the RoI Align operation or the RoI streaming operation. The general pooling network uses the RoIPooling operation, but each quantization operation of the RoIPooling operation corresponds to a slight region feature misalignment, and the quantization operations introduce a deviation between the RoI and the extracted features. These quantifications may not affect the classification task, but they have a significant negative impact on the prediction pixel precision mask. The present embodiment therefore prioritizes the use of the RoI Align operation to reduce quantization errors in the suggested frame size transform operation.
The specific process of the RoI alignment operation is as follows: firstly, traversing each candidate region of the candidate, keeping floating point number boundaries unquantified (not aligned with grid cells), dividing an average grid into H multiplied by W sub-grid regions, and keeping boundaries of each cell unquantified;
next, 4 regular sampling points are selected for each region (i.e., the regions are further divided equally into four regions, respectively, taking the midpoint of each sub-region).
Then, the pixel value of four adopted points is calculated by using a bilinear interpolation algorithm.
Finally, an aggregation operation is respectively carried out on each sub-region by utilizing maximum pooling (max pooling) or average pooling (average pooling), and a final feature map is obtained.
Therefore, the embodiment adopts the ROI Align to carry out pooling operation, so that quantization errors are reduced, and further guarantee is provided for outputting accurate and reliable detection results.
The target classification network and the regression prediction network respectively process the final feature images output by the pooling operation network to respectively obtain object types, and rectangular frame center point coordinates (x, y), width w, height h and direction angle theta of the first target position.
In this embodiment, the travelable region semantic segmentation model uses a light Mini-deep lab module or a deep lab module as a main body portion for processing. The Mini-deep Lab semantic segmentation module is extracted from a deep Lab semantic segmentation algorithm. Compared with the traditional deep Lab algorithm, the structure is simpler, the occupation of resources is smaller, the detection accuracy can be ensured, the method is more suitable for being parallel to other algorithm modules, and the advantages of the method and the device are combined to achieve a better detection effect. As shown in fig. 3, the network model structure of the Mini-deep lab semantic segmentation module comprises three branches, wherein the first branch carries out convolution operation on an input feature map; the second branch also carries out convolution operation on the input characteristic diagram, but adopts a cavity convolution (Dilated Convolution) mode to enlarge the receptive field; the third branch first performs an average pooling operation on the input feature map and then performs a convolution operation. Finally, the outputs of the three branches are subjected to a characteristic connection process (Concatenation Operation) and then enter a convolution operation, so that timeliness and accuracy are guaranteed.
In this embodiment, the method further includes:
carrying out fusion finishing treatment on a first target position output by a road target detection model and a first pixel level foreground segmentation result output by a drivable region semantic segmentation model by an information fusion finishing module based on a multi-layer perceptron MLP to obtain a second target position and a second pixel level foreground segmentation result; the rectangular box center point coordinates (x, y '), width w', height h ', and direction angle θ' of the second target location.
In this embodiment, in order to obtain more accurate road target detection and drivable region semantic segmentation results, a fusion refinement process is performed on a first target position output by a road target detection model and a first pixel level foreground segmentation result output by a drivable region semantic segmentation model through an information fusion refinement module based on a multi-layer perceptron MLP, so as to output detection results with higher precision.
As shown in fig. 4, in the embodiment, a complete network block diagram of the road detection method under the rainy and foggy weather is provided, and a parallel branch of a drivable region semantic segmentation model is newly added on the basis of a road target detection model, so as to perform image segmentation prediction on a drivable region of a road surface. The drivable region semantic segmentation model and the road target detection model share a feature extraction network, so that the calculation resources are saved, and the timeliness and the accuracy are ensured.
In a specific embodiment, there is also provided a computer device, including a memory and a processor, where the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method for detecting a road in rainy and foggy weather are as follows:
inputting the obtained foggy road image into an image rain and fog removing neural network model for processing to obtain a second road image;
inputting the second road image into a feature extraction network for feature extraction of the second road image;
constructing a road target detection model and a travelable region semantic segmentation model;
respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result.
Where the memory and the processor are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses connecting the various circuits of the one or more processors and the memory together. The bus may also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or may be a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over the wireless medium via the antenna, which further receives the data and transmits the data to the processor.
In a specific embodiment, there is also provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the method steps of:
inputting the obtained foggy road image into an image rain and fog removing neural network model for processing to obtain a second road image;
inputting the second road image into a feature extraction network for feature extraction of the second road image;
constructing a road target detection model and a travelable region semantic segmentation model;
respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result.
Those skilled in the art will appreciate that all or part of the steps in implementing the methods of the embodiments described above may be implemented by a program stored in a storage medium, including instructions for causing a device (which may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are merely preferred embodiments for fully explaining the present application, and the scope of the present application is not limited thereto. Equivalent substitutions and modifications will occur to those skilled in the art based on the present application, and are intended to be within the scope of the present application.
Claims (10)
1. A method for detecting a road under a rainy and foggy weather is characterized by comprising the following steps of: the method comprises the following steps:
inputting the obtained foggy first road image into an image rain and fog removing neural network model for processing to obtain a second road image;
inputting the second road image into a feature extraction network for feature extraction of the second road image;
constructing a road target detection model and a travelable region semantic segmentation model;
respectively inputting a feature map output by a feature extraction network into a road target detection model and a travelable region semantic segmentation model for parallel processing, wherein the road target detection model outputs an object category and a first target position; and the drivable region semantic segmentation model outputs a first pixel-level foreground segmentation result.
2. The method for detecting a road in rainy and foggy weather according to claim 1, wherein: the image defogging neural network model adopts an MSCNN neural network, or an image defogging model based on a convolution network Dehazenet, or an image defogging model based on an all-in-one convolution network AOD-Net.
3. The method for detecting a road in rainy and foggy weather according to claim 1, wherein: the feature extraction network adopts a residual network and a feature pyramid network to perform feature extraction, and more detail information is obtained through multi-scale feature fusion.
4. The method for detecting a road in rainy and foggy weather according to claim 1, wherein: the road target detection model comprises a regional suggestion network, a pooling operation network, a target classification network and a regression prediction network;
the regional suggestion network adopts an anchor frame mechanism with a direction and is used for processing a feature map output by the feature extraction network;
the pooling operation network is used for processing the feature graph output by the feature extraction network and the suggestion frame output by the regional suggestion network;
the target classification network is used for predicting object types of the output of the pooling operation network;
the regression prediction network is used for processing the output of the pooling operation network to obtain the coordinates, width, height and direction angle of the center point of the rectangular frame of the first target position.
5. The method for detecting a road in rainy and foggy weather according to claim 4, wherein: and setting a corresponding angle of an anchor frame in the regional suggestion network to acquire target position information.
6. The method for detecting a road in rainy and foggy weather according to claim 4, wherein: the pooling operation network adopts RoI Align operation or RoIPooling operation.
7. The method for detecting a road in rainy and foggy weather according to claim 1, wherein: the driving area semantic segmentation model adopts a Mini-deep Lab module or a deep Lab module; the Mini-deep Lab module comprises three branches, wherein the first branch carries out convolution operation on an input characteristic diagram; the second branch also carries out convolution operation on the input feature map, and adopts cavity convolution to enlarge the receptive field; the third branch firstly carries out average pooling operation on the input feature map and then carries out convolution operation; and finally, processing the output of the three branches through a characteristic connecting layer, then entering a convolution operation, and finally outputting a first pixel level foreground segmentation result.
8. The method for detecting a road in rainy and foggy weather according to claim 1, wherein: the method further comprises the following steps:
and carrying out fusion finishing treatment on a first target position output by the road target detection model and a first pixel level foreground segmentation result output by the drivable region semantic segmentation model by an information fusion finishing module based on the multi-layer perceptron to obtain a second target position and a second pixel level foreground segmentation result.
9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that the processor implements the steps of the method of any one of claims 1 to 8 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311092272.4A CN116958939A (en) | 2023-08-28 | 2023-08-28 | Method, system and storage medium for detecting road under rainy and foggy weather |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311092272.4A CN116958939A (en) | 2023-08-28 | 2023-08-28 | Method, system and storage medium for detecting road under rainy and foggy weather |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116958939A true CN116958939A (en) | 2023-10-27 |
Family
ID=88451385
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311092272.4A Pending CN116958939A (en) | 2023-08-28 | 2023-08-28 | Method, system and storage medium for detecting road under rainy and foggy weather |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116958939A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315446A (en) * | 2023-11-29 | 2023-12-29 | 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) | Reservoir spillway abnormity intelligent identification method oriented to complex environment |
-
2023
- 2023-08-28 CN CN202311092272.4A patent/CN116958939A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117315446A (en) * | 2023-11-29 | 2023-12-29 | 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) | Reservoir spillway abnormity intelligent identification method oriented to complex environment |
| CN117315446B (en) * | 2023-11-29 | 2024-02-09 | 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) | Reservoir spillway abnormity intelligent identification method oriented to complex environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200272835A1 (en) | Intelligent driving control method, electronic device, and medium | |
| Ghanem et al. | Lane detection under artificial colored light in tunnels and on highways: an IoT-based framework for smart city infrastructure | |
| CN108875603B (en) | Intelligent driving control method and device based on lane line and electronic equipment | |
| CN109740465B (en) | Lane line detection algorithm based on example segmentation neural network framework | |
| Mu et al. | Lane detection based on object segmentation and piecewise fitting | |
| Haque et al. | A computer vision based lane detection approach | |
| CN105989334B (en) | Road detection method based on monocular vision | |
| CN116188999B (en) | Small target detection method based on visible light and infrared image data fusion | |
| CN116958939A (en) | Method, system and storage medium for detecting road under rainy and foggy weather | |
| Wang et al. | The research on edge detection algorithm of lane | |
| CN115035295A (en) | Remote sensing image semantic segmentation method based on shared convolution kernel and boundary loss function | |
| CN113095152A (en) | Lane line detection method and system based on regression | |
| CN111079634B (en) | Method, device and system for detecting obstacle in running process of vehicle and vehicle | |
| CN115147809B (en) | Obstacle detection method, device, equipment and storage medium | |
| CN115376082B (en) | Lane line detection method integrating traditional feature extraction and deep neural network | |
| CN116030364A (en) | Unmanned aerial vehicle lightweight target detection method, system, medium, equipment and terminal | |
| Liu et al. | URTSegNet: A real-time segmentation network of unstructured road at night based on thermal infrared images for autonomous robot system | |
| CN112001378A (en) | Lane line processing method and device based on feature space, vehicle-mounted terminal and medium | |
| CN114998883A (en) | License plate recognition method and device, electronic equipment and intelligent transportation equipment | |
| CN114820679A (en) | Image annotation method and device, electronic equipment and storage medium | |
| Xing et al. | Traffic sign recognition from digital images by using deep learning | |
| CN110738668A (en) | method and system for intelligently controlling high beam and vehicle | |
| Shen et al. | Lane line detection and recognition based on dynamic ROI and modified firefly algorithm | |
| Gong et al. | Complex lane detection based on dynamic constraint of the double threshold | |
| CN116189150B (en) | Monocular 3D target detection method, device, equipment and medium based on fusion output |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |