CN114445801A - Lane line detection method based on cross-layer optimization - Google Patents

Lane line detection method based on cross-layer optimization Download PDF

Info

Publication number
CN114445801A
CN114445801A CN202210088294.2A CN202210088294A CN114445801A CN 114445801 A CN114445801 A CN 114445801A CN 202210088294 A CN202210088294 A CN 202210088294A CN 114445801 A CN114445801 A CN 114445801A
Authority
CN
China
Prior art keywords
lane line
level
feature map
pyramid
line detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210088294.2A
Other languages
Chinese (zh)
Other versions
CN114445801B (en
Inventor
郑途
黄亦非
刘洋
唐文剑
杨政
何晓飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Fabu Technology Co Ltd
Original Assignee
Hangzhou Fabu Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Fabu Technology Co Ltd filed Critical Hangzhou Fabu Technology Co Ltd
Priority to CN202210088294.2A priority Critical patent/CN114445801B/en
Publication of CN114445801A publication Critical patent/CN114445801A/en
Application granted granted Critical
Publication of CN114445801B publication Critical patent/CN114445801B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a lane line detection method based on cross-layer optimization. Inputting a road picture and obtaining a pyramid hierarchical feature map by using a convolutional neural network; preliminarily detecting and positioning lane line parameters by using a high-level pass detector, and then taking the detected and positioned lane line parameters as low-level input for continuous detection; continuously repeating the steps to obtain a finally predicted lane line; training a lane line detection model, and repeating the steps to optimize by using a loss function until convergence; and inputting the road picture to be detected in real time into the trained lane line detection model to obtain the lane line position in the road picture. The invention combines the high-level and low-level network characteristics to detect the lane line, improves the detection precision and has superiority.

Description

Lane line detection method based on cross-layer optimization
Technical Field
The invention relates to a lane line image processing method in the field of computer vision, in particular to a lane line detection method based on cross-layer optimization.
Background
Lane line detection is an important task in the field of computer vision. The method is in the field of mutual promotion and development with deep learning, can be applied to automatic driving or auxiliary driving, provides road lane line information for the automatic driving or the auxiliary driving, and further helps an intelligent vehicle to better position the vehicle.
Lane line detection is a very challenging task in computer vision. A lane line is a traffic sign with very high level semantic information. The shape of lane lines is similar to some road signs, but they have different semantic information. The high-level feature is very important for lane line detection. But the appearance of the lane lines is simple and requires low-level local features to be accurately located. Therefore, in the lane line detection task, it is very important to predict the position of the lane line using high-level and low-level features. Previous approaches have also been directed to improving lane line detection methods using high-level or low-level features. The Ultra Fast Structure-aware Deep Lane Detection published in the International Top-meeting ECCV2020 takes advantage of advanced global features to detect Lane lines. While a method for Detecting and reuniting a whole section of Lane line based on local line segment detection is provided in detection Lane Marker from Bottom Up via Key Point published in the international top-level conference CVPR 2021. The former method only focuses on high-level global features, and the latter method only focuses on local features. These methods do not combine the use of advanced and low-level features and therefore the detection accuracy is not high.
Disclosure of Invention
In order to solve the problems in the background art, the invention provides a lane line detection method based on cross-layer optimization. The method well combines the high-level features and the low-level features to optimize the process of lane line detection and improve the detection effect.
The technical scheme adopted by the invention is as follows:
(1) inputting a road picture and extracting a pyramid hierarchical feature map in the road picture by using a convolutional neural network;
the road picture is acquired by one or more combined cameras in front, side and back of the vehicle head towards the driving front.
(2) Preliminarily detecting and positioning lane line parameters by using a detector by using a higher-level pyramid level feature map, and then continuously detecting the detected and positioned lane line parameters as the input of relatively lower-level features;
(3) continuously repeating the optimization process of the step (2) for a preset number of times to obtain a finally predicted lane line;
(4) training a lane line detection model formed by the steps (1) to (3), repeating the steps (1) to (3) and optimizing the lane line detection model by using a loss function until the lane line detection model is converged;
(5) and after the training of the lane line detection model is finished, using the trained lane line detection model for a lane line detection task, and obtaining the lane line position in the road picture for the road picture to be detected in real time.
In the step (1), the convolutional neural network includes a plurality of convolution modules, and the input road image is processed by a plurality of continuous convolution modules and then results processed by different adjacent convolution modules are transmitted and superimposed to obtain a plurality of pyramid hierarchical feature maps.
The convolution module is mainly formed by sequentially connecting convolution, nonlinear change, batch normalization and the like.
In the step (1), the convolutional neural network specifically processes the input road picture through a plurality of consecutive convolutional modules, and obtains the backbone characteristic diagram after each convolutional module processing.
The method comprises three convolution modules, wherein the three convolution modules are taken as an example, three backbone feature maps of a high level, a middle level and a low level are sequentially obtained according to the sequence of the three convolution modules, the backbone feature map of the high level is processed by the convolution module for one time to obtain a pyramid level feature map of the high level, a result obtained by processing the backbone feature map of the middle level by the convolution module for one time is added with the pyramid level feature map of the high level to obtain a pyramid level feature map of the middle level, and a result obtained by processing the backbone feature map of the low level by the convolution module for one time is added with the pyramid level feature map of the middle level to obtain a pyramid level feature map of the low level.
The resolution of the three pyramid level feature maps at the high level, the middle level and the low level is gradually increased, and the semantic information is gradually reduced.
In the step (2), the three pyramid level feature maps of the high level, the middle level and the low level are transmitted through a plurality of detectors.
In the step (2), the high-level pyramid level feature map and the predefined lane line parameter P0 are input into a first detector for processing to obtain a first positioning lane line parameter Refine P0, and the lane line parameter Refine P0 and the middle-level pyramid level feature map are subjected to roiign processing to extract a first lane line parameter P1;
processing the middle pyramid level feature map and the first lane line parameter P1 by a second detector to obtain a second positioning lane line parameter Refine P1, and performing ROIAlign processing on the second positioning lane line parameter Refine P1 and the low pyramid level feature map to extract a second lane line parameter P2;
and processing the low-level pyramid level feature map and the second lane line parameter P2 by a third detector to obtain a third positioning lane line parameter Refine P2, which is used as the output of the lane line detection model.
In the step (1), pyramid level feature maps of different levels are obtained by extracting a feature pyramid network FPN, and L is0,L1，…，LN-1Respectively showing the pyramid level feature maps of N levels from L0To LN-1The resolution of the features is gradually increased, and the semantic information is gradually reduced;
then in step (2), the optimization procedure according to the following formula is defined as follows:
Figure BDA0003488099160000031
Figure BDA0003488099160000032
Pt＝argmin Loss(Pt，G)
wherein, Ft-1Corresponding to the characteristics of the lane line,
Figure BDA0003488099160000033
representing an optimisation function RtThereafterLane line parametersAn estimated value; rtDenotes the optimization procedure in the t-th detector, Lt-1(Pt-1) Denotes ROIAlign, G denotes the true value of the lane line, Koss (P)tAnd G) represents a calculation loss function, and argmin represents a parameter for optimizing the loss function to obtain the optimal value.
In the step (5), the loss function adopted for the lane line classification is Focal loss, and the loss function adopted for the lane line position regression is Smooth-l1 loss.
The input of the convolutional neural network is a picture, the convolutional neural network is a lane line (including category classification and position regression), and the process of generating P1 and P2 is the process of regression and classification.
The method of the invention has the following beneficial effects:
the invention provides a lane line detection method based on cross-layer optimization, which is used for detecting lane lines by combining high-level and low-level network characteristics and improving the detection precision.
The method can be well applied to various lane line detection networks, has great precision improvement on the main stream lane line detection data set CULane, and shows the superiority of the algorithm.
Drawings
Fig. 1 is a frame diagram according to the present invention.
Detailed Description
The invention will be further elucidated and described with reference to the drawings and the detailed description.
As shown in fig. 1, the embodiment case of the present invention includes the following:
(1) inputting a road picture and extracting a pyramid hierarchical feature map in the road picture by using a convolutional neural network;
in the step (1), the convolutional neural network comprises a plurality of convolution modules, and the input road pictures are processed by the plurality of continuous convolution modules and then the results processed by the adjacent different convolution modules are transmitted and superposed to obtain a plurality of pyramid hierarchical characteristic maps.
The convolutional neural network is characterized in that input road pictures are processed by three consecutive convolution modules, a backbone feature map is obtained after each convolution module processing, three backbone feature maps of a high level, a middle level and a low level are sequentially obtained according to the sequence of the three convolution modules, the backbone feature map of the high level is processed by one convolution module to obtain a pyramid hierarchical feature map of the highest level, a result obtained after the backbone feature map of the middle level is processed by one convolution module is added with the pyramid hierarchical feature map of the highest level to obtain a pyramid hierarchical feature map of the middle level, and a result obtained after the backbone feature map of the low level is processed by one convolution module is added with the pyramid hierarchical feature map of the middle level to obtain a pyramid hierarchical feature map of the low level.
The resolution of the three pyramid level feature maps at the high level, the middle level and the low level is gradually increased, and the semantic information is gradually reduced.
(2) Preliminarily detecting and positioning lane line parameters by using a detector by using a higher-level pyramid level feature map, and then continuously detecting the detected and positioned lane line parameters as the input of relatively lower-level features;
in the step (2), the high-level pyramid level feature map and the predefined lane line parameter P0 are input into a first detector for processing to obtain a first positioning lane line parameter Refine P0, and the lane line parameter Refine P0 and the middle-level pyramid level feature map are subjected to roiign processing to extract a first lane line parameter P1;
processing the middle pyramid level feature map and the first lane line parameter P1 by a second detector to obtain a second positioning lane line parameter Refine P1, and performing ROIAlign processing on the second positioning lane line parameter Refine P1 and the low pyramid level feature map to extract a second lane line parameter P2;
and processing the low-level pyramid level feature map and the second lane line parameter P2 by a third detector to obtain a third positioning lane line parameter Refine P2, which is used as the output of the lane line detection model.
The predefined lane line parameter P0 is input to the first detector and the output of the last detector is the output of the lane line detection model.
Pyramid level feature maps of different levels, L, are obtained by extracting feature pyramid network FPN0,L1，…，LN-1Respectively showing the pyramid level feature maps of N levels from L0To LN-1The resolution of the features is gradually increased, and the semantic information is gradually reduced;
then in step (2), each optimization process is set to R1，…，RNThe optimized parameter of the lane line of each layer is P1，…，PNThen the optimization procedure according to the following formula is defined as follows:
Figure BDA0003488099160000041
Figure BDA0003488099160000042
Pt＝argmin Loss(Pt，G)
wherein, Ft-1In correspondence with the characteristics of the lane line,
Figure BDA0003488099160000043
representing an optimisation function RtThereafterLane line parametersAn estimated value; rtDenotes the optimization procedure in the t-th detector, Lt-1(Pt-1) To representROIAlign, G represents the true value of the lane line, Loss (P)tAnd G) represents a calculation loss function, and argmin represents a parameter for optimizing the loss function to obtain the optimal value. N is 3 in the present invention.
P0The parameters representing the predefined lane lines are preset. Preferably, the number of iterations of step (3) is set to 3, i.e. N is 3.
(3) Continuously repeating the optimization process of the step (2) for a preset number of times to obtain a finally predicted lane line;
(4) the lane line detection model formed by the steps (1) to (3) is trained by using the road pictures acquired by known acquisition in advance and the lane line in the road pictures, the steps (1) to (3) are repeated, and the lane line detection model is optimized by using a loss function until the lane line detection model is converged;
(5) and after the training of the lane line detection model is finished, using the trained lane line detection model for a lane line detection task, and obtaining the lane line position in the road picture for the road picture to be detected in real time.
The loss function adopted for lane line classification is Focal loss, and the loss function adopted for lane line position regression is Smooth-l1 loss.
In a specific embodiment, the parameters in the Focal loss function are set to α ═ 0.5 and γ ═ 2.0, respectively. The parameter in the smoothen-l 1 loss function is set to β 1.0.
The invention performed experiments on the CULane dataset and the structure CLRNet proposed by the invention performed experiments. The results are given in the table below.
TABLE 1
Figure BDA0003488099160000051
In the table, Setting represents the configuration of different optimization processes, and mF1, F1@50, F1@75, and F1@90 represent different evaluation indexes respectively.
As can be seen in the table, optimization is done with only high-level features (R)0) Or low level features (R)2) Get similarThe effect is that the effect is improved to a certain extent by carrying out multiple iterative optimization on the same layer, but the improvement effect is limited. The combination of high-level features and low-level features can be improved by nearly 2 points, and the effectiveness of the invention is illustrated.

Claims (8)

1. A lane line detection method based on cross-layer optimization is characterized by comprising the following steps: the method comprises the following steps:
(1) inputting a road picture and extracting a pyramid hierarchical feature map in the road picture by using a convolutional neural network;
(2) preliminarily detecting and positioning lane line parameters by using a detector by using a higher-level pyramid level feature map, and then continuously detecting the detected and positioned lane line parameters as the input of relatively lower-level features;
(3) continuously repeating the optimization process of the step (2) for a preset number of times to obtain a finally predicted lane line;
(4) training a lane line detection model formed by the steps (1) to (3), repeating the steps (1) to (3) and optimizing the lane line detection model by using a loss function until the lane line detection model is converged;
(5) and after the training of the lane line detection model is finished, using the trained lane line detection model for a lane line detection task, and obtaining the lane line position in the road picture for the road picture to be detected in real time.
2. The lane line detection method based on cross-layer optimization according to claim 1, wherein: in the step (1), the convolutional neural network includes a plurality of convolution modules, and the input road image is processed by a plurality of continuous convolution modules and then results processed by different adjacent convolution modules are transmitted and superimposed to obtain a plurality of pyramid hierarchical feature maps.
3. The lane line detection method based on cross-layer optimization according to claim 1 or 2, wherein: in the step (1), the convolutional neural network specifically processes the input road picture through a plurality of consecutive convolutional modules, and obtains the backbone characteristic diagram after each convolutional module processing.
4. The lane line detection method based on cross-layer optimization according to claim 3, wherein: the method comprises three convolution modules, three backbone feature maps of a high level, a middle level and a low level are sequentially obtained according to the sequence of the three convolution modules, the backbone feature map of the high level is processed by the convolution module for one time to obtain a pyramid level feature map of the high level, a result obtained after the backbone feature map of the middle level is processed by the convolution module for one time is added with the pyramid level feature map of the high level to obtain a pyramid level feature map of the middle level, and a result obtained after the backbone feature map of the low level is processed by the convolution module for one time is added with the pyramid level feature map of the middle level to obtain a pyramid level feature map of the low level.
5. The lane line detection method based on cross-layer optimization according to claim 1 or 4, wherein: in the step (2), the three pyramid level feature maps of the high level, the middle level and the low level are transmitted through a plurality of detectors.
6. The lane line detection method based on cross-layer optimization according to claim 5, wherein: in the step (2), the high-level pyramid level feature map and the predefined lane line parameter P0 are input into a first detector for processing to obtain a first positioning lane line parameter Refine P0, and the lane line parameter Refine P0 and the middle-level pyramid level feature map are subjected to roiign processing to extract a first lane line parameter P1;
processing the middle pyramid level feature map and the first lane line parameter P1 by a second detector to obtain a second positioning lane line parameter Refine P1, and performing ROIAlign processing on the second positioning lane line parameter Refine P1 and the low pyramid level feature map to extract a second lane line parameter P2;
and processing the low-level pyramid level feature map and the second lane line parameter P2 by a third detector to obtain a third positioning lane line parameter Refine P2, which is used as the output of the lane line detection model.
7. The lane line detection method based on cross-layer optimization according to claim 1 or 2, wherein: in the step (1), pyramid level feature maps of different levels are obtained by extracting a feature pyramid network FPN, and L is0，L1，…，LN-1Respectively showing the pyramid level feature maps of N levels from L0To LN-1The resolution of the features is gradually increased, and the semantic information is gradually reduced;
then in step (2), the optimization procedure according to the following formula is defined as follows:
Figure FDA0003488099150000021
Figure FDA0003488099150000022
Pt＝argmin Loss(Pt，G)
wherein, Ft-1In correspondence with the characteristics of the lane line,
Figure FDA0003488099150000023
representing an optimisation function RtThereafterLane line parametersAn estimated value; rtDenotes the optimization procedure in the t-th detector, Lt-1(Pt-1) Denotes ROIAlign, G denotes the true value of the lane line, Loss (P)tAnd G) represents a calculation loss function, and argmin represents a parameter for optimizing the loss function to obtain the optimal value.
8. The lane line detection method based on cross-layer optimization according to claim 1, wherein: in the step (5), the loss function adopted for the lane line classification is Focal loss, and the loss function adopted for the lane line position regression is Smooth-l1 loss.
CN202210088294.2A 2022-01-25 2022-01-25 Lane line detection method based on cross-layer optimization Active CN114445801B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210088294.2A CN114445801B (en) 2022-01-25 2022-01-25 Lane line detection method based on cross-layer optimization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210088294.2A CN114445801B (en) 2022-01-25 2022-01-25 Lane line detection method based on cross-layer optimization

Publications (2)

Publication Number Publication Date
CN114445801A true CN114445801A (en) 2022-05-06
CN114445801B CN114445801B (en) 2022-10-21

Family

ID=81369909

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210088294.2A Active CN114445801B (en) 2022-01-25 2022-01-25 Lane line detection method based on cross-layer optimization

Country Status (1)

Country Link
CN (1) CN114445801B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200234072A1 (en) * 2019-01-21 2020-07-23 Boe Technology Group Co., Ltd. Method and Apparatus for Detecting Target Objects in Images
CN111460919A (en) * 2020-03-13 2020-07-28 华南理工大学 Monocular vision road target detection and distance estimation method based on improved YO L Ov3
CN111582201A (en) * 2020-05-12 2020-08-25 重庆理工大学 Lane line detection system based on geometric attention perception
CN111738110A (en) * 2020-06-10 2020-10-02 杭州电子科技大学 Remote sensing image vehicle target detection method based on multi-scale attention mechanism

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200234072A1 (en) * 2019-01-21 2020-07-23 Boe Technology Group Co., Ltd. Method and Apparatus for Detecting Target Objects in Images
CN111460919A (en) * 2020-03-13 2020-07-28 华南理工大学 Monocular vision road target detection and distance estimation method based on improved YO L Ov3
CN111582201A (en) * 2020-05-12 2020-08-25 重庆理工大学 Lane line detection system based on geometric attention perception
CN111738110A (en) * 2020-06-10 2020-10-02 杭州电子科技大学 Remote sensing image vehicle target detection method based on multi-scale attention mechanism

Also Published As

Publication number Publication date
CN114445801B (en) 2022-10-21

Similar Documents

Publication Publication Date Title
CN109902806B (en) Method for determining target bounding box of noise image based on convolutional neural network
CN110210551B (en) Visual target tracking method based on adaptive subject sensitivity
CN108509978B (en) Multi-class target detection method and model based on CNN (CNN) multi-level feature fusion
CN110533084B (en) Multi-scale target detection method based on self-attention mechanism
CN110516536B (en) Weak supervision video behavior detection method based on time sequence class activation graph complementation
CN114067143B (en) Vehicle re-identification method based on double sub-networks
CN109285162A (en) A kind of image, semantic dividing method based on regional area conditional random field models
CN113627228B (en) Lane line detection method based on key point regression and multi-scale feature fusion
CN105512684A (en) Vehicle logo automatic identification method based on principal component analysis convolutional neural network
Pan et al. A generic FCN-based approach for the road-network extraction from VHR remote sensing images–using openstreetmap as benchmarks
CN112990065B (en) Vehicle classification detection method based on optimized YOLOv5 model
CN112489054A (en) Remote sensing image semantic segmentation method based on deep learning
CN110909741A (en) Vehicle re-identification method based on background segmentation
CN117079139B (en) Remote sensing image target detection method and system based on multi-scale semantic features
CN112149526B (en) Lane line detection method and system based on long-distance information fusion
CN112784834A (en) Automatic license plate identification method in natural scene
CN114283162A (en) Real scene image segmentation method based on contrast self-supervision learning
CN111815526B (en) Rain image rainstrip removing method and system based on image filtering and CNN
CN112861840A (en) Complex scene character recognition method and system based on multi-feature fusion convolutional network
CN114037640A (en) Image generation method and device
CN115661777A (en) Semantic-combined foggy road target detection algorithm
CN111881914B (en) License plate character segmentation method and system based on self-learning threshold
CN111612803B (en) Vehicle image semantic segmentation method based on image definition
CN113627481A (en) Multi-model combined unmanned aerial vehicle garbage classification method for smart gardens
CN116994164A (en) Multi-mode aerial image fusion and target detection combined learning method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant