CN114782915B - Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation - Google Patents
Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation Download PDFInfo
- Publication number
- CN114782915B CN114782915B CN202210373936.3A CN202210373936A CN114782915B CN 114782915 B CN114782915 B CN 114782915B CN 202210373936 A CN202210373936 A CN 202210373936A CN 114782915 B CN114782915 B CN 114782915B
- Authority
- CN
- China
- Prior art keywords
- lane line
- decoder
- polynomial
- line detection
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 107
- 238000013140 knowledge distillation Methods 0.000 title claims abstract description 29
- 230000011218 segmentation Effects 0.000 claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 54
- 238000012549 training Methods 0.000 claims abstract description 32
- 238000000605 extraction Methods 0.000 claims abstract description 25
- 238000012360 testing method Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000003062 neural network model Methods 0.000 claims description 37
- 230000008569 process Effects 0.000 claims description 23
- 238000005070 sampling Methods 0.000 claims description 20
- 230000004927 fusion Effects 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 9
- 238000010586 diagram Methods 0.000 claims description 5
- 238000005259 measurement Methods 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 abstract description 24
- 238000004364 calculation method Methods 0.000 abstract description 6
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 238000011897 real-time detection Methods 0.000 abstract 1
- 238000005728 strengthening Methods 0.000 abstract 1
- 238000012800 visualization Methods 0.000 description 15
- 230000008447 perception Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004873 anchoring Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000011478 gradient descent method Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
An intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation belong to the technical field of artificial intelligence and detection. The method aims to solve the problem that no detection method capable of giving consideration to both algorithm precision and calculation efficiency exists in the existing lane line detection algorithm. Firstly, extracting the lane line characteristics in an image by using a main network, obtaining the lane line polynomial model parameters by using the polynomial decoder decoding characteristics, and adding a semantic segmentation branch for strengthening the characteristic extraction capability of the main network; meanwhile, a teaching module and a testing module are provided, and information is transmitted from the semantic segmentation decoder to the polynomial decoder by a knowledge distillation-like strategy, so that the detection precision of the polynomial decoder is improved; all structures of the model participate in processing the image during model training; when the model is deployed, only a backbone network model and a polynomial decoder are deployed, and the requirement of real-time detection of the lane lines of the intelligent automobile is met. The method is mainly used for detecting the lane line.
Description
Technical Field
The invention relates to a lane line detection system, and belongs to the technical field of artificial intelligence and detection.
Background
Perception, decision and control are core technologies for realizing real landing application of the intelligent automobile, and environmental perception is used as a leading technology for decision and control, so that the perception, decision and control method plays an important role in safety, reliability and comfort of driving of the intelligent automobile. The lane line detection is one of the tasks of intelligent automobile environment perception, and is a basic task of environment perception, wherein a vehicle-mounted camera or a laser radar is used for detecting and identifying lane line marks on the surface of a road, constraint information is provided for subsequent planning and decision making, and the environment perception is achieved. However, lane line detection in real road traffic scenes not only requires that the algorithm ensure accuracy and speed, but also realizes accurate and robust detection in environments such as shading, dazzling light, severe weather and the like, which is very difficult.
After the development of the deep learning technology in 2012, deep learning becomes a mainstream method for detecting lane lines, and the methods can be divided into four types according to a description mode of the lane lines: 1) Segmentation based on images (segmentation); 2) Based on an anchor model (anchor); 3) Line-based classification (row-wise); 4) Based on a polynomial model (polynomial). The method 1) classifies all pixels on a picture by image segmentation, and extracts pixels belonging to lane lines, and the method has high precision, but needs to classify all pixels and has low efficiency; the method 2) describes the lane line by using a preset anchoring model, and predicts the deviation between the real lane line and the anchoring model, so that the efficiency is improved compared with the method 1), but the degree of freedom is low, and a non-maximum inhibition algorithm is often needed for post-processing; the method 3) firstly groups the images according to the rows, only predicts whether the pixels of the lane lines are in the group, and has larger improvement on the calculation speed compared with the method 1). According to the model designed in the first three lane line description modes, the output of the model cannot be directly sent to a subsequent planning and decision-making module, but after-treatment such as clustering and regression is needed to generate a lane line parameter model required by planning. Method 4) can directly predict a geometric parameter model of a lane line, such as a quadratic polynomial or a cubic polynomial, but the accuracy of this method is not sufficient to be applied to a practical scene compared to other methods.
In a complex intelligent automobile vehicle system, end-to-end realization and direct application of algorithm output to subsequent modules are the target directions of algorithm design. However, the existing end-to-end lane line detection method is low in precision, and the non-end-to-end algorithm is high in precision and low in calculation efficiency. Therefore, designing a high-precision end-to-end lane line detection algorithm is a difficult problem to be solved by the lane line detection algorithm in the current intelligent automobile unmanned scene.
Disclosure of Invention
The invention aims to solve the problem that the existing lane line detection algorithm does not have a detection method which can give consideration to both algorithm precision and calculation efficiency.
Intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation comprises:
the road condition image acquisition unit is used for acquiring road condition images and providing the road condition images for the detection unit of the lane line to process;
the detection unit of the lane line, utilize the neural network model of lane line detection to carry on the detection of the lane line to the road conditions picture;
the lane line detection neural network model includes:
a feature extraction network: the system is used for extracting the characteristics of an input image to obtain lane line characteristics with different scales, and the lane line characteristics are respectively input to a polynomial decoder and a segmentation decoder;
a polynomial decoder: taking the highest dimensional features obtained by the feature extraction network as input, and finally obtaining curve parameters of the lane lines through two-layer convolution and two-layer full connection operation, wherein the output parameters of the polynomial decoder are N x (a + b + 1), wherein N represents the number of the lane lines in the image, a represents the coefficient number of the polynomial, b represents a definition domain of the lane lines in the image coordinates, and 1 represents the probability of the lane lines; determining a lane line model by outputting curve parameters of the lane line, wherein the lane line model is a polynomial, and the lane line curve is a curve corresponding to the polynomial;
a segmentation decoder: taking feature graphs of different scales of the feature extraction network as input, respectively performing convolution and up-sampling operations on the feature graphs, then splicing the feature graphs into a fusion feature graph, wherein the fusion feature graph is a feature graph with the same size as an output segmentation mask, and performing convolution on the feature graph to obtain a final segmentation mask;
a teaching module: the method comprises the steps that characteristics output by a first layer of a polynomial decoder are subjected to up-sampling to obtain characteristic graphs of three scales, and the sizes of the characteristic graphs after up-sampling corresponding to the polynomial decoder are respectively the same as the sizes of the characteristic graphs of different scales in a segmentation decoder; then, carrying out feature similarity measurement with features of different scales of a segmentation decoder, carrying out global average pyramid pooling on the features of different scales during the feature similarity measurement process, enabling the features of each scale to generate three new features of different scales respectively, and judging the similarity between the new features at the polynomial decoder side and the features corresponding to the new features at the segmentation decoder side by utilizing a mean square error loss function;
a test module: taking the second layer output characteristic of the polynomial decoder and the fusion characteristic graph of the segmentation decoder as input, performing up-sampling on the second layer output characteristic of the polynomial decoder, wherein the size of the characteristic graph after up-sampling is the same as that of the fusion characteristic graph in the segmentation decoder; then, channel summation and spatial softmax operation are respectively carried out on the feature graph after the polynomial decoder is up-sampled and the fusion feature graph in the segmentation decoder, and then the similarity between the output features is judged by utilizing the mean square error loss;
the lane line detection neural network model is divided into a processing mode under a training mode and a processing mode under a deployment mode:
the training mode is used for the training process of the lane line detection neural network model, training is carried out by using a training set for lane line detection, the training set is input into the model in batches in the training process, images are processed according to the whole structure of the lane line detection neural network model, and lane line parameters and a segmentation mask are output; in the training process, the output of the polynomial decoder and the output of the segmentation decoder are used as the input of the teaching module and the test module, so that the polynomial decoder can learn the capability of the segmentation decoder;
the deployment mode is used for detecting lane lines, only a feature extraction network and a polynomial decoder of a lane line detection neural network model are deployed under model deployment, and lane line parameters are output to determine a lane line model.
Furthermore, in the training process of the lane line detection neural network model, weighting and summing are carried out on loss functions of the teaching module, the testing module, the polynomial decoder and the segmentation decoder to serve as final loss of the model, and the lane line detection neural network model is trained according to the final loss.
Further, the feature extraction network obtains lane line features of three different scales.
Further, the feature extraction network is a backbone network of a ResNet model.
Further, a quadratic polynomial or a cubic polynomial is used to describe the lane line model.
Further, the road condition image acquiring unit includes:
the image acquisition module is used for acquiring road condition images shot by the vehicle-mounted camera;
and the resolution adjustment module is used for adjusting the resolution of the road condition image shot by the vehicle-mounted camera and taking the adjusted image as the input of the lane line detection neural network model.
Further, the input image size of the lane marking detection neural network model is 288 × 800.
The intelligent automobile end-to-end lane line detection method based on auxiliary supervision and knowledge distillation comprises the steps of shooting road condition images by using a vehicle-mounted camera, and inputting the road condition images shot by the vehicle-mounted camera into the intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation to detect lane lines.
A storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement the intelligent vehicle end-to-end lane line detection system based on assisted surveillance and knowledge distillation.
An intelligent vehicle end-to-end lane line detection device based on auxiliary supervision and knowledge distillation comprises a processor and a memory, wherein at least one instruction is stored in the memory, and is loaded and executed by the processor to realize the intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation.
Has the advantages that:
1. the lane line detection model provided by the invention is an end-to-end model, and can directly output lane line parameters for subsequent planning and decision-making algorithms according to input images. The deployment process is simple, and the influence of post-processing on the real-time performance and complexity of the algorithm is eliminated.
2. The lane line detection algorithm provided by the invention utilizes semantic segmentation multitask branches as auxiliary supervision, and enhances the extraction capability of a main network on the relevant features of the lane line; in addition, the provided teaching and testing module gives full play to the advantage that the lane line detection accuracy of the segmentation algorithm is high, and the knowledge of the segmentation decoder is transmitted to the polynomial decoder, so that the problem that the lane line detection algorithm based on the polynomial is low in accuracy is solved.
3. The method provided by the invention can be used for improving the model precision by means of multi-task auxiliary supervision, knowledge distillation and the like in the training stage, and only using the trunk model and the polynomial decoder in the deployment stage to ensure the algorithm calculation efficiency, thereby realizing the balance of accuracy and real-time performance.
Drawings
FIG. 1 is a diagram of a lane line detection neural network model architecture;
FIG. 2 is a visualization result of a lane line visualization result detected in a normal scene of the CULane data set;
fig. 3 is a visualization result of a lane line detected in a crowded scene of the CULane data set;
fig. 4 is a lane line visualization result detected in a glaring scene in the CULane data set;
FIG. 5 is a lane line visualization result detected in a shadow scene in the CULane data set;
fig. 6 is a lane line visualization result detected in a scene where an identification arrow identifies the CULane data set;
fig. 7 is a lane line visualization result detected in a scene without identification arrow identification in the CULane dataset;
fig. 8 is a lane line visualization result detected in a curved road scene in the CULane data set;
fig. 9 is a lane line visualization result detected in a night scene in the CULane data set;
FIG. 10 is a visualization result of lane lines detected in the test set of the TuSimple data set;
FIG. 11 is a visualization result of a lane line detected under a test set of a TuSimple data set;
FIG. 12 is a visualization result of lane lines detected in the test set of the TuSimple data set;
FIG. 13 is a visualization result of lane lines detected in the test set of the TuSimple data set.
Detailed Description
The invention aims to design an end-to-end lane line detection neural network model, which is applied to an intelligent automobile unmanned scene and solves the problem that the existing lane line detection field is difficult to meet the requirements of speed and precision at the same time. After the algorithm is applied, the intelligent automobile can realize high-precision end-to-end lane line detection under 1080p video at the speed of more than 200 frames/second. The algorithm is less influenced by environmental factors, can realize an accurate detection function under conventional roads, congested roads (sheltered), dark light environments and dazzling light environments, and provides safe and reliable sensing output for intelligent automobiles. The present invention will be described with reference to specific embodiments.
The first specific implementation way is as follows:
the intelligent automobile end-to-end lane line detection method based on auxiliary supervision and knowledge distillation is realized by depending on an intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation.
This embodiment is intelligent car end-to-end lane line detecting system based on supplementary supervision and knowledge distillation, includes:
the road condition image acquisition unit is used for acquiring road condition images and providing the road condition images for the detection unit of the lane line to process;
the road condition image acquisition unit comprises:
the image acquisition module is used for acquiring road condition images shot by the vehicle-mounted camera;
the resolution adjusting module is used for adjusting the resolution of the road condition image shot by the vehicle-mounted camera and taking the adjusted image as the input of the lane line detection neural network model; in this embodiment, the resolution of the image is compressed to 288 × 800 to reduce the amount of computation of the model;
the lane line detection unit is used for detecting the lane lines on the road condition images by using the lane line detection neural network model;
the structure of the lane line detection neural network model is shown in fig. 1, and the processing process of the lane line detection neural network model is as follows:
the input image is subjected to feature extraction by a feature extraction network, in the embodiment, the feature extraction network selects a main network (light blue cuboid in fig. 1) of a ResNet model to obtain lane line features (arrows output downwards or rightwards after the last three cuboid blocks are stacked in fig. 1) with three different scales, and the features are respectively input into a polynomial decoder and a segmentation decoder to complete respective lane line prediction tasks; wherein,
the polynomial decoder is used for obtaining curve parameters of the lane lines by using the highest dimensional features obtained by the feature extraction network as input through two layers of convolution and two layers of full connection operation, and further determining and describing a lane line model, wherein the lane line model is a polynomial, and the lane line curve is a curve corresponding to the polynomial; in the invention, a quadratic polynomial or a cubic polynomial is used for describing a lane line model;
taking a quadratic polynomial as an example, the output parameters of the polynomial decoder are N (3 +2+ 1), where N represents the number of lane lines in the image, 3 represents three coefficients of the quadratic polynomial, 2 represents the domain of the lane line in the image coordinates, and 1 represents the probability of the lane line existing.
The segmentation decoder takes three feature maps with different scales as input, respectively performs convolution and up-sampling operations on the feature maps, then splices the feature maps into a fused feature map, wherein the fused feature map is a feature map with the same size as an output segmentation mask (mask), and then performs convolution on the feature map to obtain a final segmentation mask.
The lane line detection neural network model needs to be trained to be deployed in practical application, so the training and deployment processes of the lane line detection neural network model are respectively introduced here.
(1) Training: the lane marking detection has many public data sets, such as CULane, tuSimple, etc., and the CULane data set is taken as an example for explanation here.
And inputting the training set of the data set into the model in batches, processing the image according to the lane line detection neural network model structure, and outputting lane line parameters and a segmentation mask. In the training process, the output characteristics of the first layer of the polynomial decoder and the characteristics of three different scales of the segmentation decoder are used as the input of a teaching module, so that the polynomial decoder learns the capability of the segmentation decoder, and the second layer of output characteristics of the polynomial decoder and the fusion characteristic diagram of the segmentation decoder are used as the input of a testing module to ensure that the polynomial decoder actually learns the relevant characteristics of the segmentation decoder;
the teaching module performs up-sampling on the intermediate features output by the first layer of the polynomial decoder to obtain feature maps of three scales, and the size of the feature map after up-sampling is respectively the same as the size of the feature maps of three different scales in the segmentation decoder; then, the similarity of the features is measured with the features of the segmentation decoder in three different scales, global average pyramid pooling is firstly carried out on the features, new features of three different scales are respectively generated by the features of each scale, and then the similarity between the new features at the polynomial decoder side and the features corresponding to the new features at the segmentation decoder side is judged by utilizing a mean square error loss function.
The test module takes the second layer output characteristic of the polynomial decoder and the fusion characteristic graph of the segmentation decoder as input, and firstly carries out up-sampling on the second layer output characteristic of the polynomial decoder, wherein the size of the characteristic graph after the up-sampling is the same as that of the fusion characteristic graph in the segmentation decoder; and then channel summation and spatial softmax operation are respectively carried out on the feature graph after the polynomial decoder is up-sampled and the fusion feature graph in the segmentation decoder, and then the similarity between the output features is judged by utilizing the mean square error loss, so that the information of the two encoders is really transmitted.
And (3) performing weighted summation on the loss functions of the four parts aiming at a teaching module, a testing module, a polynomial decoder and a segmentation decoder to serve as the final loss of the model, and completing the updating of model parameters by utilizing an optimization algorithm (such as a gradient descent method), thereby completing the training of the lane line detection neural network model.
(2) A deployment process: in order to ensure the real-time performance of the model during actual operation, when the model is deployed, only the feature extraction network and the polynomial decoder (namely the light blue cuboid part and the dashed box part where the polynomial decoder is located) of the lane line detection neural network model are deployed, because the polynomial decoder at the moment has the capability of accurately detecting the lane lines in the image, the decoder does not need to be segmented for auxiliary supervision. At this time, the operation process of the lane line detection neural network model is as follows: video streams of the vehicle-mounted camera are input into the model in a form of continuous image frames, and the images are subjected to down-sampling to 288 × 800, and then are subjected to a feature extraction network and a polynomial decoder, and a lane line parameter model is directly output to a subsequent planning module, so that end-to-end lane line detection is realized.
The second embodiment is as follows:
the intelligent automobile end-to-end lane line detection method based on auxiliary supervision and knowledge distillation comprises the following steps:
firstly, acquiring a road condition image shot by a vehicle-mounted camera, and then inputting the road condition image shot by the vehicle-mounted camera into a lane line detection neural network model to detect a lane line; in the embodiment, the specific process of detecting the lane line by using the lane line detection neural network model comprises the following steps of:
firstly, preparing a lane line detection public data set, such as TuSimple, CULane and the like, dividing the lane line detection public data set into a training set, a verification set and a test set according to the specification of the data set, and preparing matched labels for the training set and the verification set;
step two, performing down-sampling on the original image obtained in the step one, reducing the resolution to 288 × 800, and performing data enhancement processing such as random rotation, cutting, horizontal turning and the like on a training set for training a neural network model;
and step three, designing an end-to-end neural network model, wherein the model is a main part of the invention different from the prior art and is also a main innovation embodiment of the invention. The structure of the lane line detection neural network model provided by the invention is shown in the attached figure 1, and the structure introduction and the working principle of the model are as follows. The road condition image shot by the vehicle-mounted camera is used as input, after the preprocessing of the second step, the road condition image is input into a main network (light blue cuboid in the attached drawing 1) of a ResNet model to realize the feature extraction of the input image, three lane line features (downward or rightward output arrows after the last three cuboid blocks are stacked in the attached drawing 1) with different scales are obtained, and the features are respectively input into a polynomial decoder and a segmentation decoder to finish respective lane line prediction tasks. The polynomial decoder uses the highest dimensional feature as input, and finally obtains the curve parameters of the lane line through two layers of convolution and two layers of full connection operation. In the present invention, a quadratic polynomial or a cubic polynomial is used to describe the lane lines, in this embodiment, a quadratic polynomial is used, and the output parameters of the polynomial decoder are N (3 +2+ 1), where N represents the number of lane lines in the image, 3 represents three coefficients of the quadratic polynomial, 2 represents a defined field of the lane line in the image coordinates, and 1 represents the probability of the lane line existing. The segmentation decoder takes three feature maps with different scales as input, performs convolution and up-sampling operations on the feature maps respectively, splices the feature maps into feature maps with the same size as an output segmentation mask (mask), and performs convolution on the feature maps to obtain a final segmentation mask.
And fourthly, training an end-to-end lane line detection model. Inputting the training set of the data set into the model in batches, processing the image according to the model structure described in the third step, and outputting lane line parameters and the segmentation mask. And in the training process, the intermediate output of the polynomial decoder and the intermediate output of the segmentation decoder are used as the input of the teaching module and the test module, so that the polynomial decoder can learn the capability of the segmentation decoder. The teaching module is used for measuring the feature similarity between the intermediate features of the segmented decoders after up-sampling the intermediate features and the intermediate features of the segmented decoders with three different scales, the method is used for carrying out global average pooling on the features with three different scales, and the similarity between the features is judged by utilizing a mean square error loss function. The test module judges the similarity between the output characteristics by using the final characteristics of the polynomial decoder and the segmentation decoder and using the mean square error loss after the channel summation and the spatial softmax operation, thereby ensuring that the information of the two encoders is really transmitted. And finally, the teaching module, the testing module, the polynomial decoder and the segmentation decoder perform weighted summation on the loss functions of the four parts to serve as the final loss of the model, and the model parameters are updated by utilizing an optimization algorithm (such as a gradient descent method).
And step five, deploying an end-to-end lane line detection model. In order to ensure the real-time performance of the model in actual operation, when the model is deployed, only a backbone network (i.e., a feature extraction network) and a polynomial decoder (i.e., a light blue cuboid part and a dashed box part where the polynomial decoder is located in fig. 1) of the model are deployed, because the polynomial decoder at this time already has the capability of accurately detecting lane lines in an image, the decoder does not need to be segmented for auxiliary supervision. The operation process of the model at this time is as follows: video streams of the vehicle-mounted camera are input into the model in a form of continuous image frames, and the images are subjected to down sampling to 288 × 800, and then are directly output to a subsequent planning module through a backbone network and a polynomial decoder, so that end-to-end lane line detection is realized.
The lane line detection model provided by the invention can change the detailed structure according to the specific actual requirements. Among the modifiable content are feature extraction networks and lane line models (i.e., curves of lane lines). The main network model used by the method is a characteristic extraction part of ResNet-34, if other requirements on speed or precision actually exist, the main network model can be replaced by main networks of conventional neural networks such as ResNet-18, shuffleNet, mobileNet and the like, and the model can be finely adjusted after the main network model is replaced; the lane line model can be replaced by a polynomial model of a first order, a third order or higher order according to the actual road condition, the specific replacement mode is to replace the output dimensionality of the last fully-connected layer of the polynomial decoder by N (M +1+2+ 1), wherein M is the highest order of the polynomial model, and the model is finely adjusted after the replacement.
The lane line detection algorithm provided by the invention utilizes semantic segmentation multi-task branches as auxiliary supervision, so that the extraction capability of a main network on the relevant features of the lane line is enhanced; in addition, the provided teaching and testing module gives full play to the advantage that the lane line detection precision of the segmentation algorithm is high, and the knowledge of the segmentation decoder is transferred to the polynomial decoder, so that the precision of the predicted lane line model of the polynomial decoder is improved. When the system is actually deployed, only the backbone network and the polynomial decoder work, the curve model of the lane line can be detected end to end, and high accuracy is kept.
In the embodiment, the invention is compared with the existing method on the two lane line detection data sets of TuSimple and curane. Wherein TuSimple uses precision (Accuracy) as an evaluation index, and CULane uses F1-Measure as an evaluation index. The calculation formulas of the two are as follows:
wherein S is clip Is the number of true value points, C, of pixels belonging to a lane line in a sequence clip The number of the pixel points with correct prediction results is shown, precision is shown, and Recall is shown. The results of the method are compared with the best performance method in the existing four types of lane line detection algorithms, and the comparison results are shown in tables 1 and 2.
TABLE 1 Tusimple comparison results
TABLE 2 CULane comparison results
As can be seen from tables 1 and 2, the end-to-end lane line detection model provided by the present invention achieves the level of the optimal method in terms of both accuracy and speed, and is more suitable for deployment in an actual unmanned driving scenario. In the test result of the CULane data set, the method achieves the highest F1-Measure, and meanwhile, the running speed of the model is the highest among all the methods, so that the superiority of the method compared with the existing method is reflected. The visualization results of the CULane data set are shown in fig. 2 to 9, and fig. 2 to 9 show the lane line detection conditions of the method under eight different scenes, so that the lane line can be accurately detected under the environments of crowding, dazzling, dark light and the like. The visualization results of the invention under the test set of TuSimple dataset are shown in fig. 10 to 13.
The third concrete implementation mode:
the embodiment is a storage medium, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to realize the intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation.
It should be understood that any method described herein, including any methods described herein, may correspondingly be provided as a computer program product, software, or computerized method, which may include a non-transitory machine-readable medium having stored thereon instructions, which may be used to program a computer system, or other electronic device, to perform a process. Storage media may include, but is not limited to, magnetic storage media, optical storage media; a magneto-optical storage medium comprising: read only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM), and flash memory layers; or other type of media suitable for storing electronic instructions.
The fourth concrete implementation mode:
the embodiment is the intelligent automobile end-to-end lane line detection device based on auxiliary supervision and knowledge distillation, the device comprises a processor and a memory, and it should be understood that any device described in the invention, which comprises the processor and the memory, may also comprise other units and modules for displaying, interacting, processing, controlling and the like through signals or instructions;
the memory stores at least one instruction that is loaded and executed by the processor to implement the intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation.
The present invention is capable of other embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention.
Claims (10)
1. Intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation is characterized by comprising:
the road condition image acquisition unit is used for acquiring road condition images and providing the road condition images for the detection unit of the lane line to process;
the lane line detection unit is used for detecting the lane lines on the road condition images by using the lane line detection neural network model;
the lane line detection neural network model includes:
a feature extraction network: the system is used for extracting the characteristics of an input image to obtain lane line characteristics with different scales, and the lane line characteristics are respectively input into a polynomial decoder and a segmentation decoder;
a polynomial decoder: taking the highest dimensional features obtained by the feature extraction network as input, and finally obtaining curve parameters of the lane lines through two-layer convolution and two-layer full connection operation, wherein the output parameters of the polynomial decoder are N x (a + b + 1), wherein N represents the number of the lane lines in the image, a represents the coefficient number of the polynomial, b represents the definition domain of the lane lines in the image coordinates, and 1 represents the probability of the lane lines; determining a lane line model by outputting curve parameters of the lane line, wherein the lane line model is a polynomial, and the lane line curve is a curve corresponding to the polynomial;
a segmentation decoder: taking feature graphs of different scales of the feature extraction network as input, respectively performing convolution and up-sampling operations on the feature graphs, then splicing the feature graphs into a fusion feature graph, wherein the fusion feature graph is a feature graph with the same size as an output segmentation mask, and performing convolution on the feature graph to obtain a final segmentation mask;
a teaching module: the method comprises the steps that characteristics output by a first layer of a polynomial decoder are subjected to up-sampling to obtain characteristic graphs of three scales, and the sizes of the characteristic graphs after up-sampling corresponding to the polynomial decoder are respectively the same as the sizes of the characteristic graphs of different scales in a segmentation decoder; then, carrying out characteristic similarity measurement with the characteristics of different scales of the segmentation decoder, carrying out global average pyramid pooling on the characteristics of different scales during the characteristic similarity measurement process, enabling the characteristics of each scale to generate three new characteristics of different scales respectively, and judging the similarity between the new characteristics at the polynomial decoder side and the characteristics corresponding to the new characteristics at the segmentation decoder side by utilizing a mean square error loss function;
a test module: taking the second-layer output characteristic of the polynomial decoder and the fused characteristic diagram of the segmentation decoder as input, performing up-sampling on the second-layer output characteristic of the polynomial decoder, wherein the size of the characteristic diagram after up-sampling is the same as that of the fused characteristic diagram in the segmentation decoder; then, channel summation and spatial softmax operation are respectively carried out on the feature graph after the polynomial decoder is up-sampled and the fusion feature graph in the segmentation decoder, and then the similarity between the output features is judged by utilizing the mean square error loss;
the lane line detection neural network model is divided into a processing mode under a training mode and a processing mode under a deployment mode:
the training mode is used for the training process of the lane line detection neural network model, training is carried out by utilizing a training set for lane line detection, the training set is input into the model in batches in the training process, images are processed according to all structures of the lane line detection neural network model, and lane line parameters and segmentation masks are output; in the training process, the output of the polynomial decoder and the output of the segmentation decoder are used as the input of the teaching module and the test module, so that the polynomial decoder can learn the capability of the segmentation decoder;
the deployment mode is used for detecting lane lines, only a feature extraction network and a polynomial decoder of a lane line detection neural network model are deployed under model deployment, and lane line parameters are output to determine a lane line model.
2. The intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation as claimed in claim 1, wherein in the training process of the lane line detection neural network model, the loss functions of the teaching module, the testing module, the polynomial decoder and the segmentation decoder are subjected to weighted summation to serve as the final loss of the model, and the lane line detection neural network model is trained according to the final loss.
3. An intelligent vehicle end-to-end lane marking detection system based on auxiliary supervision and knowledge distillation as claimed in claim 2, wherein the feature extraction network obtains lane marking features of three different scales.
4. The intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation of claim 3, wherein the feature extraction network is a backbone network of a ResNet model.
5. The intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation of claim 4, wherein a quadratic polynomial or a cubic polynomial is used to describe the lane line model.
6. The intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation as claimed in one of claims 1 to 5, wherein the road condition image acquisition unit comprises:
the image acquisition module is used for acquiring road condition images shot by the vehicle-mounted camera;
and the resolution adjusting module is used for adjusting the resolution of the road condition image shot by the vehicle-mounted camera and taking the adjusted image as the input of the lane line detection neural network model.
7. An intelligent vehicle end-to-end lane marking detection system based on assisted supervision and knowledge distillation as claimed in claim 6, characterized in that the input image size of the lane marking detection neural network model is 288 x 800.
8. The intelligent automobile end-to-end lane line detection method based on auxiliary supervision and knowledge distillation is characterized in that a vehicle-mounted camera is used for shooting road condition images, and then the road condition images shot by the vehicle-mounted camera are input into the intelligent automobile end-to-end lane line detection system based on auxiliary supervision and knowledge distillation, which is disclosed by one of claims 1 to 7, so that lane line detection is carried out.
9. A storage medium, wherein at least one instruction is stored, and wherein the at least one instruction is loaded and executed by a processor to implement the intelligent vehicle end-to-end lane line detection system based on aided supervision and knowledge distillation of any one of claims 1 to 7.
10. An intelligent vehicle end-to-end lane line detection device based on auxiliary supervision and knowledge distillation, characterized in that the device comprises a processor and a memory, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to realize the intelligent vehicle end-to-end lane line detection system based on auxiliary supervision and knowledge distillation as claimed in one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210373936.3A CN114782915B (en) | 2022-04-11 | 2022-04-11 | Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210373936.3A CN114782915B (en) | 2022-04-11 | 2022-04-11 | Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114782915A CN114782915A (en) | 2022-07-22 |
| CN114782915B true CN114782915B (en) | 2023-04-07 |
Family
ID=82429362
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210373936.3A Active CN114782915B (en) | 2022-04-11 | 2022-04-11 | Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114782915B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115376091A (en) * | 2022-10-21 | 2022-11-22 | 松立控股集团股份有限公司 | Lane line detection method assisted by image segmentation |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066986A (en) * | 2017-04-21 | 2017-08-18 | 哈尔滨工业大学 | A kind of lane line based on monocular vision and preceding object object detecting method |
| CN111242037A (en) * | 2020-01-15 | 2020-06-05 | 华南理工大学 | Lane line detection method based on structural information |
| CN112528878A (en) * | 2020-12-15 | 2021-03-19 | 中国科学院深圳先进技术研究院 | Method and device for detecting lane line, terminal device and readable storage medium |
| CN112883807A (en) * | 2021-01-22 | 2021-06-01 | 中汽创智科技有限公司 | Lane line detection method and system |
| CN113158768A (en) * | 2021-03-03 | 2021-07-23 | 中山大学 | Intelligent vehicle lane line detection method based on ResNeSt and self-attention distillation |
| CN113591670A (en) * | 2021-07-27 | 2021-11-02 | 中国科学院合肥物质科学研究院 | Lane line detection method based on convolutional neural network |
| CN113780132A (en) * | 2021-08-31 | 2021-12-10 | 武汉理工大学 | Lane line detection method based on convolutional neural network |
| CN113920100A (en) * | 2021-10-19 | 2022-01-11 | 成都一尧科技有限公司 | Knowledge distillation-based weak supervision bone scanning image hot spot segmentation method and system |
| CN114120069A (en) * | 2022-01-27 | 2022-03-01 | 四川博创汇前沿科技有限公司 | Lane line detection system, method and storage medium based on direction self-attention |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11003920B2 (en) * | 2018-11-13 | 2021-05-11 | GM Global Technology Operations LLC | Detection and planar representation of three dimensional lanes in a road scene |
-
2022
- 2022-04-11 CN CN202210373936.3A patent/CN114782915B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066986A (en) * | 2017-04-21 | 2017-08-18 | 哈尔滨工业大学 | A kind of lane line based on monocular vision and preceding object object detecting method |
| CN111242037A (en) * | 2020-01-15 | 2020-06-05 | 华南理工大学 | Lane line detection method based on structural information |
| CN112528878A (en) * | 2020-12-15 | 2021-03-19 | 中国科学院深圳先进技术研究院 | Method and device for detecting lane line, terminal device and readable storage medium |
| CN112883807A (en) * | 2021-01-22 | 2021-06-01 | 中汽创智科技有限公司 | Lane line detection method and system |
| CN113158768A (en) * | 2021-03-03 | 2021-07-23 | 中山大学 | Intelligent vehicle lane line detection method based on ResNeSt and self-attention distillation |
| CN113591670A (en) * | 2021-07-27 | 2021-11-02 | 中国科学院合肥物质科学研究院 | Lane line detection method based on convolutional neural network |
| CN113780132A (en) * | 2021-08-31 | 2021-12-10 | 武汉理工大学 | Lane line detection method based on convolutional neural network |
| CN113920100A (en) * | 2021-10-19 | 2022-01-11 | 成都一尧科技有限公司 | Knowledge distillation-based weak supervision bone scanning image hot spot segmentation method and system |
| CN114120069A (en) * | 2022-01-27 | 2022-03-01 | 四川博创汇前沿科技有限公司 | Lane line detection system, method and storage medium based on direction self-attention |
Non-Patent Citations (4)
| Title |
|---|
| Flexible lane detection using CNNs;Li Haixia，et al;《2021 international conference on computer technology and media convergence design(CTMCD)》;20210716;235-238 * |
| 一种采用区域知识蒸馏网络的车道线检测方法;叶飞等;《小型微型计算机***》;20211025;第43卷(第11期);2348-2353 * |
| 基于卷积神经网络的车道线检测与车道保持控制研究;刘泽凌;《中国优秀硕士学位论文全文数据库工程科技II辑》;20220315;C035-579 * |
| 基于目标特征蒸馏的车道线检测;龙建武等;《重庆理工大学学报(自然科学)》;20200930;第34卷(第9期);198-208 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114782915A (en) | 2022-07-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110163187B (en) | F-RCNN-based remote traffic sign detection and identification method | |
| CN108694386B (en) | Lane line detection method based on parallel convolution neural network | |
| CN111738111A (en) | Road extraction method of high-resolution remote sensing image based on multi-branch cascade void space pyramid | |
| CN113126115B (en) | Semantic SLAM method and device based on point cloud, electronic equipment and storage medium | |
| CN111738037B (en) | Automatic driving method, system and vehicle thereof | |
| CN113468967A (en) | Lane line detection method, device, equipment and medium based on attention mechanism | |
| CN108520238B (en) | Scene prediction method of night vision image based on depth prediction coding network | |
| CN109902610A (en) | Traffic sign recognition method and device | |
| CN113591617B (en) | Deep learning-based water surface small target detection and classification method | |
| CN117765373B (en) | Lightweight road crack detection method and system with self-adaptive crack size | |
| CN115285143A (en) | Automatic driving vehicle navigation method based on scene classification | |
| CN114913498A (en) | Parallel multi-scale feature aggregation lane line detection method based on key point estimation | |
| CN114120272A (en) | Multi-supervision intelligent lane line semantic segmentation method fusing edge detection | |
| CN114743119A (en) | High-speed rail contact net dropper nut defect detection method based on unmanned aerial vehicle | |
| CN114782915B (en) | Intelligent automobile end-to-end lane line detection system and equipment based on auxiliary supervision and knowledge distillation | |
| CN114067142A (en) | Method for realizing scene structure prediction, target detection and lane level positioning | |
| KR20220045762A (en) | System for automatic recognition and monitoring of vessel using artificial intelligence image processing and method for providing the same | |
| CN115273032A (en) | Traffic sign recognition method, apparatus, device and medium | |
| CN115049948A (en) | Unmanned aerial vehicle inspection method and device based on neural network model and related equipment | |
| CN115049130A (en) | Automatic driving track prediction method based on space-time pyramid | |
| CN114550023A (en) | Traffic target static information extraction device | |
| CN114048536A (en) | Road structure prediction and target detection method based on multitask neural network | |
| CN114549970A (en) | Night small target fruit detection method and system fusing global fine-grained information | |
| CN115294548B (en) | Lane line detection method based on position selection and classification method in row direction | |
| CN111160282A (en) | Traffic light detection method based on binary Yolov3 network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |