CN114863392A - Lane line detection method, lane line detection device, vehicle, and storage medium - Google Patents

Lane line detection method, lane line detection device, vehicle, and storage medium Download PDF

Info

Publication number
CN114863392A
CN114863392A CN202210444361.XA CN202210444361A CN114863392A CN 114863392 A CN114863392 A CN 114863392A CN 202210444361 A CN202210444361 A CN 202210444361A CN 114863392 A CN114863392 A CN 114863392A
Authority
CN
China
Prior art keywords
lane line
feature map
network
line detection
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210444361.XA
Other languages
Chinese (zh)
Inventor
叶航军
蔡锐
赵婕
祝贺
王斌
周珏嘉
邱叶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Xiaomi Mobile Software Co Ltd
Original Assignee
Beijing Xiaomi Mobile Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Xiaomi Mobile Software Co Ltd filed Critical Beijing Xiaomi Mobile Software Co Ltd
Priority to CN202210444361.XA priority Critical patent/CN114863392A/en
Publication of CN114863392A publication Critical patent/CN114863392A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Image Analysis (AREA)

Abstract

The present disclosure relates to a lane line detection method, apparatus, vehicle, and storage medium, the lane line detection method detecting an image by acquiring a target; the target detection image is input into a preset lane line detection model, so that the preset lane line detection model outputs a target position of a lane line in the target detection image, wherein the preset lane line detection model is obtained through training of a preset initial model comprising a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of the lane line in a lane line detection sample image, the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image, lane line detection is carried out through the preset lane line detection model, and accuracy of a lane line detection result can be effectively improved.

Description

Lane line detection method, lane line detection device, vehicle, and storage medium
Technical Field
The disclosure relates to the technical field of intelligent networked automobiles, in particular to a lane line detection method and device, a vehicle and a storage medium.
Background
With the continuous development of scientific technology, the automatic driving technology gradually moves towards the development climax. Safety is always the focus of attention of the automatic driving technology, the environmental perception capability of the automatic driving system plays a decisive role in safety performance, and lane line detection is a key link in environmental perception.
In the lane line testing process, can often because light is darker, the reflection of light exists and shelters from, either the lane line is washed by the rainwater for a long time, and the lane line is fuzzy in the road image that leads to shooing, and the scene that the lane line is located in addition varies greatly, consequently can lead to the lane line to detect the degree of difficulty greatly, the lower problem of testing result accuracy.
Disclosure of Invention
To overcome the problems in the related art, the present disclosure provides a lane line detection method, apparatus, vehicle, and storage medium.
According to a first aspect of the embodiments of the present disclosure, there is provided a lane line detection method, the method including:
acquiring a target detection image;
inputting the target detection image into a preset lane line detection model so that the preset lane line detection model outputs a target position of a lane line in the target detection image;
the preset lane line detection model is obtained by training in the following mode:
acquiring a plurality of lane line detection sample images, wherein the lane line detection sample images comprise lane line marking positions;
inputting each lane line detection sample image into a preset initial model, wherein the preset initial model comprises a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of a lane line in the lane line detection sample image, and the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image;
and training the preset initial model according to the first predicted position, the second predicted position and the lane line marking position to obtain the preset lane line detection model.
Optionally, the preset initial model further includes a feature extraction network, the feature extraction network is coupled to the classification network and the segmentation network, and the preset initial model is configured to be used for
Acquiring a multi-scale sample feature map corresponding to each lane line detection sample image through the feature extraction network, and determining a first sample feature map comprising image global information according to the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
and determining a second predicted position of the lane line in the lane line detection sample image according to the multi-scale sample feature map through the classification network.
Optionally, the preset lane line detection model is obtained by training,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in a preset initial model, and determining a first sample feature map comprising image global information according to a minimum scale sample feature map in the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
determining a first loss value corresponding to a first loss function according to the first predicted position and the lane marking position;
determining a second predicted position of the lane line in the lane line detection sample image according to the minimum scale sample feature map through the classification network;
determining a second loss value corresponding to a second loss function according to the second predicted position and the lane marking position;
determining a third loss value corresponding to a third loss function according to the first loss value and the second loss value;
and under the condition that the third loss value is greater than or equal to a preset loss value threshold, adjusting model parameters of the preset initial model to obtain an updated target model, and executing the step of obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in the preset initial model again until a third loss value corresponding to a third loss function is determined according to the first loss value and the second loss value, until the segmentation network in the current target model is deleted under the condition that the third loss value is determined to be smaller than the preset loss value threshold, so as to obtain the preset lane line detection model.
Optionally, the feature extraction network comprises a feature pyramid network and a global feature extraction network, the feature pyramid network comprises a bottom-up sub-network and a top-down sub-network, an input of the global feature extraction network is coupled to an output of the bottom-up sub-network, an output of the global feature extraction network is coupled to an input of the top-down sub-network, an output of the global feature extraction network is further coupled to an input of the segmentation network, an output of the top-down sub-network is also coupled to an input of the segmentation network, the preset initial model is used for,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through a bottom-up sub-network in the feature pyramid network, and inputting a minimum-scale sample feature map in the multi-scale sample feature map into the global feature extraction network, so that the global feature extraction network outputs the first sample feature map to the segmentation network and the top-down sub-network;
inputting a second multi-scale sample feature map to the segmentation network through the top-down sub-network, so that the segmentation network determines a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
Optionally, the determining, by the segmentation network, a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map includes:
performing upsampling processing on the first sample characteristic diagram and the second sample characteristic diagram through the segmentation network to obtain a sample characteristic diagram to be segmented;
segmenting the sample feature map to be segmented according to a preset segmentation mode to obtain a plurality of local feature maps;
performing convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map;
determining a first probability that each pixel in the lane line detection sample image belongs to a lane line according to the target sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image from the first probability of each pixel.
Optionally, the method according to the preset segmentation mode includes:
dividing the sample feature map to be segmented into H local feature maps of C W, dividing the sample feature map to be segmented into C local feature maps of H W, and dividing the sample feature map to be segmented into at least one of W local feature maps of H C, wherein H is the number of layers of the sample feature map to be segmented, C is the length of the sample feature map to be segmented, and W is the width of the sample feature map to be segmented.
Optionally, the convolving and stitching the plurality of local feature maps to obtain the target sample feature map includes:
acquiring a current local feature map, and performing convolution operation on the current local feature map to obtain a convolved specified local feature map;
splicing the appointed local feature map and the next local feature map corresponding to the current local feature map to obtain an updated current local feature map;
determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
under the condition that the next local feature map corresponding to the current local feature map is determined to be not the last local feature map in the plurality of local feature maps, obtaining the current local feature map again, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
and under the condition that the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps, acquiring the current local feature map, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and taking the specified local feature map corresponding to the current local feature map as the target sample feature map.
Optionally, the output of the bottom-up sub-network is coupled to the input of the classification network, and the preset initial model is configured to:
inputting a minimum-scale sample feature map to the classification network through the bottom-up sub-network;
and dividing the minimum-scale sample feature map into a plurality of anchor frames through the classification network, determining a second probability that each anchor frame belongs to the lane line, and determining a second predicted position of the lane line in the lane line detection sample image according to the second probability.
Optionally, inputting the target detection image into a preset lane line detection model, so that the preset lane line detection model outputs a target position of a lane line in the target detection image, including:
acquiring a target feature map corresponding to the target detection image through the feature extraction network;
and inputting the target feature map into the classification network to obtain the target position of a lane line in the target detection image output by the classification network.
According to a second aspect of the embodiments of the present disclosure, there is provided a lane line detection apparatus, the apparatus including:
an acquisition module configured to acquire a target detection image;
a determination module configured to input the target detection image into a preset lane line detection model so that the preset lane line detection model outputs a target position of a lane line in the target detection image;
the preset lane line detection model is obtained by training in the following mode:
acquiring a plurality of lane line detection sample images, wherein the lane line detection images comprise lane line marking positions;
inputting each lane line detection sample image into a preset initial model, wherein the preset initial model comprises a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of a lane line in the lane line detection sample image, and the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image;
and training the preset initial model according to the first predicted position, the second predicted position and the lane line marking position to obtain the preset lane line detection model.
Optionally, the preset initial model further includes a feature extraction network, the feature extraction network is coupled to the classification network and the segmentation network, and the preset initial model is configured to be used for
Acquiring a multi-scale sample feature map corresponding to each lane line detection sample image through the feature extraction network, and determining a first sample feature map comprising image global information according to the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
and determining a second predicted position of the lane line in the lane line detection sample image according to the multi-scale sample feature map through the classification network.
Optionally, the preset lane line detection model is obtained by training,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in a preset initial model, and determining a first sample feature map comprising image global information according to a minimum scale sample feature map in the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
determining a first loss value corresponding to a first loss function according to the first predicted position and the lane marking position;
determining a second predicted position of the lane line in the lane line detection sample image according to the minimum scale sample feature map through the classification network;
determining a second loss value corresponding to a second loss function according to the second predicted position and the lane marking position;
determining a third loss value corresponding to a third loss function according to the first loss value and the second loss value;
and under the condition that the third loss value is greater than or equal to a preset loss value threshold, adjusting model parameters of the preset initial model to obtain an updated target model, and executing the step of obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in the preset initial model again until a third loss value corresponding to a third loss function is determined according to the first loss value and the second loss value, until the segmentation network in the current target model is deleted under the condition that the third loss value is determined to be smaller than the preset loss value threshold, so as to obtain the preset lane line detection model.
Optionally, the feature extraction network comprises a feature pyramid network and a global feature extraction network, the feature pyramid network comprises a bottom-up sub-network and a top-down sub-network, an input of the global feature extraction network is coupled to an output of the bottom-up sub-network, an output of the global feature extraction network is coupled to an input of the top-down sub-network, an output of the global feature extraction network is further coupled to an input of the segmentation network, an output of the top-down sub-network is also coupled to an input of the segmentation network, the preset initial model is used for,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through a bottom-up sub-network in the feature pyramid network, and inputting a minimum scale sample feature map in the multi-scale sample feature map into the global feature extraction network, so that the global feature extraction network outputs the first sample feature map to the segmentation network and the top-down sub-network;
inputting a second multi-scale sample feature map to the segmentation network through the top-down sub-network, so that the segmentation network determines a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
Optionally, the determining, by the segmentation network, a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map includes:
performing upsampling processing on the first sample characteristic diagram and the second sample characteristic diagram through the segmentation network to obtain a sample characteristic diagram to be segmented;
segmenting the sample feature map to be segmented according to a preset segmentation mode to obtain a plurality of local feature maps;
performing convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map;
determining a first probability that each pixel in the lane line detection sample image belongs to a lane line according to the target sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image from the first probability of each pixel.
Optionally, the method according to the preset segmentation mode includes:
dividing the sample feature map to be segmented into H local feature maps of C W, dividing the sample feature map to be segmented into C local feature maps of H W, and dividing the sample feature map to be segmented into at least one of W local feature maps of H C, wherein H is the number of layers of the sample feature map to be segmented, C is the length of the sample feature map to be segmented, and W is the width of the sample feature map to be segmented.
Optionally, the convolving and stitching the plurality of local feature maps to obtain the target sample feature map includes:
acquiring a current local feature map, and performing convolution operation on the current local feature map to obtain a convolved specified local feature map;
splicing the appointed local feature map and the next local feature map corresponding to the current local feature map to obtain an updated current local feature map;
determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
under the condition that the next local feature map corresponding to the current local feature map is determined to be not the last local feature map in the plurality of local feature maps, obtaining the current local feature map again, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
and under the condition that the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps, acquiring the current local feature map, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and taking the specified local feature map corresponding to the current local feature map as the target sample feature map.
Optionally, the output of the bottom-up sub-network is coupled to the input of the classification network, and the preset initial model is configured to:
inputting a minimum-scale sample feature map to the classification network through the bottom-up sub-network;
and dividing the minimum-scale sample feature map into a plurality of anchor frames through the classification network, determining a second probability that each anchor frame belongs to the lane line, and determining a second predicted position of the lane line in the lane line detection sample image according to the second probability.
Optionally, the determining module is configured to:
acquiring a target feature map corresponding to the target detection image through the feature extraction network;
and inputting the target feature map into the classification network to obtain the target position of a lane line in the target detection image output by the classification network.
According to a third aspect of the embodiments of the present disclosure, there is provided a vehicle including the lane line detection apparatus described in the above second aspect.
According to a fourth aspect of the embodiments of the present disclosure, there is provided a lane line detection apparatus including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of the first aspect above.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the method of the first aspect of the present disclosure.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: detecting an image by acquiring a target; and inputting the target detection image into a preset lane line detection model so that the preset lane line detection model outputs the target position of a lane line in the target detection image, wherein the preset lane line detection model is obtained by training a preset initial model comprising a segmentation network and a classification network, and lane line detection is performed through the preset lane line detection model, so that the accuracy of a lane line detection result can be effectively improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a flow chart illustrating a lane line detection method according to an exemplary embodiment of the present disclosure;
FIG. 2 is a block diagram illustrating a model structure of a pre-set initial model according to an exemplary embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a split network shown in an exemplary embodiment of the present disclosure;
FIG. 4 is a flow chart illustrating the training of a pre-defined lane-line detection model according to an exemplary embodiment of the present disclosure;
FIG. 5 is a flow chart of a lane line detection method according to the embodiment shown in FIG. 4;
fig. 6 is a block diagram of a lane line detection apparatus shown in an exemplary embodiment of the present disclosure;
fig. 7 is a block diagram illustrating a lane line detection apparatus according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
It should be noted that all actions of acquiring signals, information or data in the present application are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.
Before describing the embodiments of the present disclosure in detail, first, the following description is made on an application scenario of the present disclosure, and the present disclosure may be applied to a lane line detection process, which may be an important link for lane recognition when sensing a driving environment in an automatic driving, unmanned driving, or route navigation process. The method for recognizing the lane at present mainly comprises two types, namely a method based on image characteristics and a method based on a model, wherein the principle of the image characteristic recognition is that the difference of characteristics between the edge of a lane marking line and the road conditions around a road image is utilized, the common characteristic difference comprises the texture, the edge geometric shape, the continuity of the boundary, the gray value and the contrast of an interested area and the like of the image, the image characteristics can be extracted by using a threshold segmentation method and the like, and the algorithm is simple and easy to realize and has the defects of low accuracy of a lane line detection result and poor reliability of the lane recognition result due to the fact that the method is easily interfered by factors such as noise, shadow shielding of trees, illumination change, discontinuity of the marking line and the like. The principle based on model matching is that a corresponding lane line model is established according to the geometric characteristics of a structured road, and the parameters of the lane model are identified so as to identify lane marking lines. Due to the limitation of the image feature method, the image feature method and the model matching method are generally combined in practical research, so that a proper lane line detection model is established, the accuracy of lane line identification can be improved on one hand, and the adaptability of the system can be enhanced on the other hand.
The present disclosure provides a lane line detection method, apparatus, vehicle, and storage medium, the lane line detection method detecting a lane line by acquiring a target detection image; the target detection image is input into a preset lane line detection model so that the preset lane line detection model outputs the target position of a lane line in the target detection image, wherein the preset lane line detection model is trained through a preset initial model comprising a segmentation network and a classification network, lane line detection is carried out through the preset lane line detection model, and the accuracy of lane line detection results in various detection scenes can be effectively improved.
The technical scheme of the disclosure is explained in detail by combining specific embodiments.
Fig. 1 is a flow chart illustrating a lane line detection method according to an exemplary embodiment of the present disclosure; as shown in fig. 1, the lane line detection method may include:
step 101, acquiring a target detection image.
The target detection image may be any image to be detected in various lane line detection scenes (e.g., a detection scene of a straight lane line, a detection scene of a hyperbolic lane line, a detection scene of a B-spline curve lane line, etc.), and the image to be detected may include a lane line to be detected.
Step 102, inputting the target detection image into a preset lane line detection model, so that the preset lane line detection model outputs a target position of a lane line in the target detection image.
In this step, the preset lane line detection model is obtained by training through the following methods shown in S11 to S13:
s11, a plurality of lane line detection sample images are acquired.
The lane line detection image includes a lane line marking position.
S12, inputting each of the lane line detection sample images into a preset initial model, the preset initial model including a segmentation network for determining a first predicted position of a lane line in the lane line detection sample image and a classification network for determining a second predicted position of the lane line in the lane line detection sample image.
The segmentation network is used for dividing the feature map corresponding to the lane line detection sample image into a plurality of local feature maps, then carrying out convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map, and inputting the target sample feature map into the classification layer so that the classification layer outputs a second predicted position of the lane line in the lane line detection sample image.
It should be noted that, the segmentation network performs feature graph division, performs convolution operation on the local feature graphs obtained by the division to effectively obtain more feature graph details, and then performs classification operation on the feature graphs containing more details to effectively improve the accuracy of the classification result, thereby improving the accuracy of the lane line detection result. In addition, the classification layer in the split network may be a full connection layer or other network modules for classification, and there are many network modules with classification functions in the prior art, which is not limited in this disclosure.
S13, training the preset initial model according to the first predicted position, the second predicted position and the lane marking position to obtain the preset lane detection model.
It should be noted that, in the training process, the preset lane line detection model is obtained by training a preset initial model including the classification network and the segmentation network, but in the application process, the preset lane line detection model may include the segmentation network and the classification network, may also include the classification network, does not include the segmentation network, and may also include the segmentation network, does not include the classification network.
According to the technical scheme, the target detection image is obtained; the target detection image is input into a preset lane line detection model, so that the preset lane line detection model outputs the target position of a lane line in the target detection image, wherein the preset lane line detection model is obtained through training of a preset initial model comprising a segmentation network and a classification network, lane line detection is carried out through the preset lane line detection model, and the accuracy of lane line detection results in various detection scenes can be effectively improved.
Fig. 2 is a block diagram illustrating a model structure of a preset initial model according to an exemplary embodiment of the disclosure, where as shown in fig. 2, the preset initial model further includes a feature extraction network, the feature extraction network is coupled to the classification network and the segmentation network, and the preset initial model is configured to:
acquiring a multi-scale sample feature map corresponding to each lane line detection sample image through the feature extraction network, and determining a first sample feature map comprising image global information according to the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
and determining a second predicted position of the lane line in the lane line detection sample image according to the multi-scale sample feature map through the classification network.
Wherein, the feature extraction network may include a feature pyramid network and a global feature extraction network, the feature pyramid network includes a bottom-up sub-network and a top-down sub-network, the global feature extraction network may be an Encoder in a Transformer network, an input of the global feature extraction network is coupled to an output of the bottom-up sub-network, an output of the global feature extraction network is coupled to an input of the top-down sub-network, an output of the global feature extraction network is further coupled to an input of the segmentation network, an output of the top-down sub-network is also coupled to an input of the segmentation network, an output of the bottom-up sub-network is coupled to an input of the classification network, and the preset initial model is used for:
obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through a bottom-up subnetwork in the feature pyramid network (e.g., feature map x in fig. 2) 1 And a characteristic diagram x 2 And a characteristic diagram x 3 And a characteristic diagram x 4 ) And the minimum scale sample feature map (such as feature map x in FIG. 2) in the multi-scale sample feature map 4 ) Inputting the global feature extraction network to make the global feature extraction network output the first sample feature map to the segmentation network and the top-down sub-network (as in FIG. 2, feature map x is output by the global feature extraction network 4 Conversion into a feature map X 1 )；
Inputting a second multi-scale sample feature map (such as feature map X in FIG. 2) into the segmentation network via the top-down sub-network 1 And characteristic diagram X 2 And characteristic diagram X 3 And characteristic diagram X 4 ) So that the segmentation network determines a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
It should be noted that, in the feature pyramid network, the feature map at the lower layer (shallow layer) has rich detail information, and the feature map at the higher layer (deep layer) has rich semantic information, and the implementation process of the segmentation network determining the first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map may include the following steps S21 to S25:
and S21, performing upsampling processing on the first sample feature map and the second sample feature map through the segmentation network to obtain a to-be-segmented sample feature map.
Illustratively, the feature map for the second sample of the multi-scale may comprise a feature map X as in FIG. 2 1 And characteristic diagram X 2 And characteristic diagram X 3 And feature map X 4 For the feature map X 1 Feature map X 2 And the characteristic diagram X 3 Respectively up-sampling to obtain the characteristic diagram X 4 Feature map with same size, and the feature map X 1 Feature map X 2 And the characteristic diagram X 3 Corresponding to the characteristic diagram X 4 The feature map with the same size and the feature map X 4 And performing concat processing to obtain the characteristic diagram of the sample to be segmented.
And S22, segmenting the sample feature map to be segmented according to a preset segmentation mode to obtain a plurality of local feature maps.
Wherein, should include according to presetting cutting apart the mode: dividing the sample feature map to be segmented into H local feature maps of C W, dividing the sample feature map to be segmented into C local feature maps of H W, and dividing the sample feature map to be segmented into at least one of W local feature maps of H C, wherein H is the number of layers of the sample feature map to be segmented, C is the length of the sample feature map to be segmented, and W is the width of the sample feature map to be segmented.
Illustratively, as shown in fig. 3, fig. 3 is a schematic diagram of a split network according to an exemplary embodiment of the present disclosure, which is split from four directions, namely, the to-be-segmented sample feature map of H × C × W is regarded as a cube, the feature map of H × C × W is divided into H local feature maps of C × W (e.g., SCNN _ D in fig. 3) from top to bottom along the height of the cube, the feature map of H × C × W is divided into H local feature maps of C W (e.g., SCNN _ U in fig. 3) from bottom to top along the height of the cube, the feature map of H × C × W is divided into W local feature maps of H × C (e.g., SCNN _ R in fig. 3) from left to right along the width of the cube (e.g., corresponding W), and the feature map of H × C W is divided into W local feature maps of H × C (e.g., SCNN _ R in fig. 3) from right to left along the width of the cube (e.g., corresponding W).
And S23, performing convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map.
One possible implementation manner in this step is:
acquiring a current local feature map, and performing convolution operation on the current local feature map to obtain a convolved specified local feature map; splicing the appointed local feature map and the next local feature map corresponding to the current local feature map to obtain an updated current local feature map; determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps; under the condition that the next local feature map corresponding to the current local feature map is determined to be not the last local feature map in the local feature maps, obtaining the current local feature map again, performing convolution operation on the current local feature map to obtain a specified local feature map after convolution, and determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the local feature maps; and under the condition that the next local feature map corresponding to the current local feature map is the last local feature map in the local feature maps, acquiring the current local feature map, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and taking the specified local feature map corresponding to the current local feature map as the target sample feature map.
By way of example, and still taking the example shown in fig. 3 as an example, after obtaining the top-to-bottom H C × W local feature maps (e.g., C × W-1, C × W-2 to C × W-H), the convolution operation may be performed on the local feature map C × W-1 (with a convolution kernel of 1 × 1), then the convolution result corresponding to the local feature map C W-1 (the designated local feature map corresponding to C × W-1) may be concatenated with the local feature map C × W-2 to obtain the concatenated current local feature map, then the convolution operation may be performed on the current local feature map (with a convolution kernel of 2 and 1 × 1) to obtain the designated local feature map corresponding to C × W-2, and then the designated local feature map corresponding to the local feature map C × W-2 may be concatenated with the local feature map C × W-3, so as to obtain the updated current local feature map, the process is cycled until obtaining the designated local feature map corresponding to C x W-H, wherein the dimension of the designated local feature map corresponding to C x W-H is H x C x W, then, slicing the designated local feature map corresponding to C x W-H (also a feature map of H x C x W) from bottom to top along the high direction to obtain H C x W local feature maps (for example, C x W +1, C x W +2 to C x W + H), performing convolution splicing operation on the C x W +1, C x W +2 to C x W + H to obtain the designated local feature map corresponding to C x W + H (the dimension is H x W), and then performing convolution splicing operation on the C x W + H local feature map corresponding to C x W + H along the left to right direction (the cube is H), dividing the H, C, W feature map into W local feature maps, obtaining left-to-right local feature maps of H, C (such as H, C-1, H, C-2 to H, C-W), performing a predetermined convolution operation on the H, C-1, H, C-2 to H, C-W to obtain a designated local feature map (dimension H, C, W), then dividing the H, C, W feature map into right-to-left local feature maps (such as H, C +1, H, C +2 to H, C + W), and performing a predetermined convolution operation on the H, C + W, H, C +2 to H, C + W, and taking the designated local feature map corresponding to H x C + W as the target sample feature map.
And S24, determining a first probability that each pixel in the lane line detection sample image belongs to the lane line according to the target sample feature map.
In this step, since the target sample feature map includes more detailed information in the lane line detection sample image, the accuracy of the first probability that each pixel in the lane line detection sample image obtained according to the target sample feature map belongs to the lane line is higher.
S25, determining a first predicted position of the lane line in the lane line detection sample image based on the first probability for each pixel.
In this step, the pixel position where the first probability is greater than or equal to the first preset probability threshold may be used as the first predicted position of the lane line in the lane line detection sample image.
Through the above S21 to S25, a target sample feature map with more detailed information can be obtained through the segmentation network, and then the first probability that each pixel in the lane line detection sample image belongs to the lane line is determined through the target sample feature map, and determining the first predicted position according to the first probability can effectively improve the accuracy of the first predicted position.
Optionally, the preset initial model is further configured to:
inputting a minimum-scale sample feature map (such as feature map x in FIG. 2) into the classification network through the bottom-up sub-network 4 )；
And dividing the minimum-scale sample feature map into a plurality of anchor frames through the classification network, determining a second probability that each anchor frame belongs to the lane line, and determining a second predicted position of the lane line in the lane line detection sample image according to the second probability.
In addition, the feature map x 4 The manner of dividing the anchor frames into the anchor frames is more common in the prior art and is easier to obtain, which is not described herein again, and the specific details of determining the second probability that each anchor frame belongs to the lane line through the classification network (which may be a full connection layer) belong to a common calculation process in the art, which is not limited by the present disclosure. In addition, the pixel position corresponding to the anchor frame with the second probability greater than or equal to the second preset probability threshold may be used as the second predicted position.
According to the technical scheme, the second predicted position of the lane line in the lane line detection sample image can be obtained through the classification network, and the second predicted position is obtained through the minimum scale sample Feature map output by the FPN (Feature Pyramid network), and the minimum scale sample Feature contains abundant contextual Feature information, so that the accuracy of the second predicted position can be effectively improved.
FIG. 4 is a flow chart illustrating the training of a pre-defined lane-line detection model according to an exemplary embodiment of the present disclosure; as shown in fig. 4, the preset lane line detection model can be obtained by training through the following steps:
step 401, obtaining a plurality of lane line detection sample images, where the lane line detection sample images include lane line labeling positions.
The lane line detection sample image may be any one of various lane line detection scenes (for example, a detection scene of a straight lane line, a detection scene of a hyperbolic lane line, a detection scene of a B-spline curve lane line, and the like), and the lane line position may be marked in the lane line detection sample image.
Step 402, obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in the preset initial model, and determining a first sample feature map including image global information according to a minimum scale sample feature map in the multi-scale sample feature map.
The feature extraction network may include a feature pyramid network and a global feature extraction network, the feature pyramid network includes a bottom-up sub-network and a top-down sub-network, the global feature extraction network may be an Encoder in a Transformer network, and a multi-scale sample feature map corresponding to the lane line detection sample image is obtained through the bottom-up sub-network in the feature pyramid network (for example, the feature map x in fig. 2) 1 And a characteristic diagram x 2 And a characteristic diagram x 3 And a characteristic diagram x 4 ) And the minimum scale sample feature map (such as feature map x in FIG. 2) in the multi-scale sample feature map 4 ) Inputting the global feature extraction network such that the global feature extraction network outputs the first sample feature map to the segmentation network and the top-down subnetwork.
And step 403, determining a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map through the segmentation network.
In this step, the first sample feature map is input into the top-down sub-network, so that the top-down sub-network inputs a multi-scale second sample feature map (e.g., feature map X in FIG. 2) into the segmentation network 1 And characteristic diagram X 2 And characteristic diagram X 3 And characteristic diagram X 4 ) So that the segmentation network determines a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
For a specific implementation of the segmentation network determining the first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map, the steps shown in S21 to S25 above may be referred to, and details of the present disclosure are not repeated herein.
Step 404, determining a first loss value corresponding to the first loss function according to the first predicted position and the lane marking position.
The first Loss function may be a cross entropy Loss function, a Focal Loss (Focal Loss) Loss function, or other Loss functions in the prior art.
Step 405, determining a second predicted position of the lane line in the lane line detection sample image according to the minimum-scale sample feature map through the classification network.
Step 406, determining a second loss value corresponding to the second loss function according to the second predicted position and the marked position of the lane line.
The second Loss function may also be a cross-entropy Loss function or a Focal local Loss function, or may also be other Loss functions in the prior art.
Step 407, determining a third loss value corresponding to a third loss function according to the first loss value and the second loss value.
Illustratively, the third loss function may be L total ＝αL cls +βL seg Wherein L is total Representing the third loss value, alpha and beta are weighting coefficients. L is cls Is a second loss value, L, corresponding to a second loss function seg Is a first loss value corresponding to the first loss function.
At step 408, it is determined whether the third loss value is greater than or equal to a predetermined loss value threshold.
In this step, if the third loss value is greater than or equal to the preset loss value threshold, step 409 is executed, and if the third loss value is determined to be less than the preset loss value threshold, step 410 is executed.
Step 409, adjusting the model parameters of the preset initial model to obtain an updated target model.
After this step, it is possible to jump to step 402 again to perform step 402 to step 408 again.
Step 410, deleting the segmentation network in the current target model to obtain the preset lane line detection model.
According to the technical scheme, the preset lane line detection model is trained together through the segmentation network and the classification network, the accuracy of a model detection result can be effectively improved, the segmentation network can be deleted in an application stage in order to improve the model processing speed, and only the feature extraction network and the classification network are reserved, so that the accuracy of the lane line detection result can be ensured, and the detection efficiency of the model can be effectively improved.
Alternatively, when the target position of the lane line in the target detection image is determined by the preset lane line detection model obtained in the above step shown in fig. 4, the following step shown in fig. 5 may be included, where fig. 5 is a flowchart of a lane line detection method according to the embodiment shown in fig. 4; as shown in fig. 5, the method includes:
step 501, a target detection image is obtained.
The target detection image may be any image to be detected in various lane line detection scenes (e.g., a detection scene of a straight lane line, a detection scene of a hyperbolic lane line, a detection scene of a B-spline curve lane line, etc.), and the image to be detected may include a lane line to be detected.
And 502, acquiring a target feature map corresponding to the target detection image through the feature extraction network.
Wherein the feature extraction network may include a feature pyramid network including a bottom-up sub-network and a top-down sub-network, and the target feature map may be a minimum scale feature map (e.g., feature map x in fig. 2) input from the bottom-up sub-network 4 )。
Step 503, inputting the target feature map into the classification network to obtain the target position of the lane line in the target detection image output by the classification network.
The target position of the lane line in the target detection image is determined by the method shown in the steps 501 to 503, so that the accuracy of the lane line detection result can be ensured, and the detection efficiency of the model can be effectively improved.
Fig. 6 is a block diagram illustrating a lane line detection apparatus according to an exemplary embodiment of the present disclosure, and as shown in fig. 6, the apparatus may include:
an acquisition module 601 configured to acquire a target detection image;
a determining module 602 configured to input the target detection image into a preset lane line detection model, so that the preset lane line detection model outputs a target position of a lane line in the target detection image;
the preset lane line detection model is obtained by training in the following mode:
acquiring a plurality of lane line detection sample images, wherein the lane line detection images comprise lane line marking positions;
inputting each lane line detection sample image into a preset initial model, wherein the preset initial model comprises a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of a lane line in the lane line detection sample image, and the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image;
and training the preset initial model according to the first predicted position, the second predicted position and the lane line marking position to obtain the preset lane line detection model.
According to the technical scheme, the target detection image is obtained; inputting the target detection image into a preset lane line detection model so that the preset lane line detection model outputs the target position of a lane line in the target detection image, wherein the preset lane line detection model is obtained by training a preset initial model comprising a segmentation network and a classification network, and the accuracy of lane line detection results in various detection scenes can be effectively improved.
Optionally, the preset initial model further comprises a feature extraction network coupled to the classification network and the segmentation network, the preset initial model being configured to
Acquiring a multi-scale sample feature map corresponding to each lane line detection sample image through the feature extraction network, and determining a first sample feature map comprising image global information according to the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
and determining a second predicted position of the lane line in the lane line detection sample image according to the multi-scale sample feature map through the classification network.
Alternatively, the preset lane line detection model is trained in the following manner,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in a preset initial model, and determining a first sample feature map comprising image global information according to a minimum scale sample feature map in the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
determining a first loss value corresponding to a first loss function according to the first predicted position and the lane marking position;
determining a second predicted position of the lane line in the lane line detection sample image according to the minimum scale sample feature map through the classification network;
determining a second loss value corresponding to a second loss function according to the second predicted position and the lane marking position;
determining a third loss value corresponding to a third loss function according to the first loss value and the second loss value;
and under the condition that the third loss value is greater than or equal to a preset loss value threshold, adjusting model parameters of the preset initial model to obtain an updated target model, and executing the step of obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in the preset initial model again until a third loss value corresponding to a third loss function is determined according to the first loss value and the second loss value, until the segmentation network in the current target model is deleted under the condition that the third loss value is determined to be less than the preset loss value threshold, so as to obtain the preset lane line detection model.
Optionally, the feature extraction network comprises a feature pyramid network and a global feature extraction network, the feature pyramid network comprises a bottom-up sub-network and a top-down sub-network, an input of the global feature extraction network is coupled to an output of the bottom-up sub-network, an output of the global feature extraction network is coupled to an input of the top-down sub-network, an output of the global feature extraction network is further coupled to an input of the segmentation network, an output of the top-down sub-network is also coupled to an input of the segmentation network, the preset initial model is used,
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through a bottom-up sub-network in the feature pyramid network, and inputting a minimum scale sample feature map in the multi-scale sample feature map into the global feature extraction network, so that the global feature extraction network outputs the first sample feature map to the segmentation network and the top-down sub-network;
and inputting a second multi-scale sample feature map to the segmentation network through the top-down sub-network so that the segmentation network determines a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
Optionally, the determining, by the segmentation network, a first predicted position of the lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map includes:
performing upsampling processing on the first sample characteristic diagram and the second sample characteristic diagram through the segmentation network to obtain a sample characteristic diagram to be segmented;
segmenting the sample feature map to be segmented according to a preset segmentation mode to obtain a plurality of local feature maps;
performing convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map;
determining a first probability that each pixel in the lane line detection sample image belongs to the lane line according to the target sample feature map;
a first predicted position of the lane line in the lane line detection sample image is determined based on the first probability of each pixel.
Optionally, the method according to the preset segmentation includes:
dividing the sample feature map to be segmented into H local feature maps of C W, dividing the sample feature map to be segmented into C local feature maps of H W, and dividing the sample feature map to be segmented into at least one of W local feature maps of H C, wherein H is the number of layers of the sample feature map to be segmented, C is the length of the sample feature map to be segmented, and W is the width of the sample feature map to be segmented.
Optionally, the convolving and stitching the plurality of local feature maps to obtain the target sample feature map includes:
acquiring a current local feature map, and performing convolution operation on the current local feature map to obtain a convolved specified local feature map;
splicing the appointed local feature map and the next local feature map corresponding to the current local feature map to obtain an updated current local feature map;
determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
under the condition that the next local feature map corresponding to the current local feature map is determined to be not the last local feature map in the local feature maps, obtaining the current local feature map again, performing convolution operation on the current local feature map to obtain a specified local feature map after convolution, and determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the local feature maps;
and under the condition that the next local feature map corresponding to the current local feature map is the last local feature map in the local feature maps, acquiring the current local feature map, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and taking the specified local feature map corresponding to the current local feature map as the target sample feature map.
Optionally, the output of the bottom-up subnetwork is coupled to the input of the classification network, and the preset initial model is configured to:
inputting a minimum-scale sample feature map to the classification network through the bottom-up sub-network;
and dividing the minimum-scale sample feature map into a plurality of anchor frames through the classification network, determining a second probability that each anchor frame belongs to the lane line, and determining a second predicted position of the lane line in the lane line detection sample image according to the second probability.
Optionally, the determining module is configured to:
acquiring a target feature map corresponding to the target detection image through the feature extraction network;
and inputting the target feature map into the classification network to obtain the target position of a lane line in the target detection image output by the classification network.
According to the technical scheme, the preset lane line detection model is trained together through the segmentation network and the classification network, the accuracy of a model detection result can be effectively improved, the segmentation network can be deleted in an application stage in order to improve the model processing speed, and only the feature extraction network and the classification network are reserved, so that the accuracy of the lane line detection result can be ensured, and the detection efficiency of the model can be effectively improved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The present disclosure also provides a vehicle including the lane line detection apparatus described above in fig. 6.
The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the lane line detection method provided by the present disclosure.
Fig. 7 is a block diagram illustrating a lane line detection apparatus according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 7, the apparatus 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.
The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the lane marking detection method described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the apparatus 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Power component 806 provides power to the various components of device 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 800.
The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect an open/closed state of the device 800, the relative positioning of the components, such as a display and keypad of the device 800, the sensor assembly 814 may also detect a change in position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, orientation or acceleration/deceleration of the device 800, and a change in temperature of the device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate communications between the apparatus 800 and other devices in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the lane line detection methods described above.
In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the lane line detection method described above is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the lane line detection method described above when executed by the programmable apparatus.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (13)

1. A lane line detection method, comprising:
acquiring a target detection image;
inputting the target detection image into a preset lane line detection model so that the preset lane line detection model outputs a target position of a lane line in the target detection image;
the preset lane line detection model is obtained by training in the following mode:
acquiring a plurality of lane line detection sample images, wherein the lane line detection sample images comprise lane line marking positions;
inputting each lane line detection sample image into a preset initial model, wherein the preset initial model comprises a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of a lane line in the lane line detection sample image, and the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image;
and training the preset initial model according to the first predicted position, the second predicted position and the lane line marking position to obtain the preset lane line detection model.
2. The method of claim 1, wherein the pre-set initial model further comprises a feature extraction network coupled to the classification network and the segmentation network, the pre-set initial model being configured to:
acquiring a multi-scale sample feature map corresponding to each lane line detection sample image through the feature extraction network, and determining a first sample feature map comprising image global information according to the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
and determining a second predicted position of the lane line in the lane line detection sample image according to the multi-scale sample feature map through the classification network.
3. The method of claim 2, wherein the predetermined lane line detection model is trained by:
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in a preset initial model, and determining a first sample feature map comprising image global information according to a minimum scale sample feature map in the multi-scale sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map through the segmentation network;
determining a first loss value corresponding to a first loss function according to the first predicted position and the lane marking position;
determining a second predicted position of the lane line in the lane line detection sample image according to the minimum scale sample feature map through the classification network;
determining a second loss value corresponding to a second loss function according to the second predicted position and the lane marking position;
determining a third loss value corresponding to a third loss function according to the first loss value and the second loss value;
and under the condition that the third loss value is greater than or equal to a preset loss value threshold, adjusting model parameters of the preset initial model to obtain an updated target model, and executing the step of obtaining a multi-scale sample feature map corresponding to the lane line detection sample image through the feature extraction network in the preset initial model again until a third loss value corresponding to a third loss function is determined according to the first loss value and the second loss value, until the segmentation network in the current target model is deleted under the condition that the third loss value is determined to be smaller than the preset loss value threshold, so as to obtain the preset lane line detection model.
4. The method of claim 2, wherein the feature extraction network comprises a feature pyramid network and a global feature extraction network, the feature pyramid network comprises a bottom-up sub-network and a top-down sub-network, an input of the global feature extraction network is coupled to an output of the bottom-up sub-network, an output of the global feature extraction network is coupled to an input of the top-down sub-network, an output of the global feature extraction network is further coupled to an input of the segmentation network, an output of the top-down sub-network is also coupled to an input of the segmentation network, and the preset initial model is used for:
acquiring a multi-scale sample feature map corresponding to the lane line detection sample image through a bottom-up sub-network in the feature pyramid network, and inputting a minimum scale sample feature map in the multi-scale sample feature map into the global feature extraction network, so that the global feature extraction network outputs the first sample feature map to the segmentation network and the top-down sub-network;
inputting a second multi-scale sample feature map to the segmentation network through the top-down sub-network, so that the segmentation network determines a first predicted position of a lane line in the lane line detection sample image according to the first sample feature map and the second sample feature map.
5. The method of claim 4, wherein the segmentation network determining a first predicted location of a lane line in the lane line detection sample image from the first sample feature map and the second sample feature map comprises:
performing upsampling processing on the first sample characteristic diagram and the second sample characteristic diagram through the segmentation network to obtain a sample characteristic diagram to be segmented;
segmenting the sample feature map to be segmented according to a preset segmentation mode to obtain a plurality of local feature maps;
performing convolution and splicing processing on the plurality of local feature maps to obtain a target sample feature map;
determining a first probability that each pixel in the lane line detection sample image belongs to a lane line according to the target sample feature map;
determining a first predicted position of a lane line in the lane line detection sample image from the first probability of each pixel.
6. The method according to claim 5, wherein the preset partition comprises:
dividing the sample feature map to be segmented into H local feature maps of C W, dividing the sample feature map to be segmented into C local feature maps of H W, and dividing the sample feature map to be segmented into at least one of W local feature maps of H C, wherein H is the number of layers of the sample feature map to be segmented, C is the length of the sample feature map to be segmented, and W is the width of the sample feature map to be segmented.
7. The method of claim 5, wherein the convolving and stitching the plurality of local feature maps to obtain the target sample feature map comprises:
acquiring a current local feature map, and performing convolution operation on the current local feature map to obtain a convolved specified local feature map;
splicing the appointed local feature map and the next local feature map corresponding to the current local feature map to obtain an updated current local feature map;
determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
under the condition that the next local feature map corresponding to the current local feature map is determined to be not the last local feature map in the plurality of local feature maps, obtaining the current local feature map again, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and determining whether the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps;
and under the condition that the next local feature map corresponding to the current local feature map is the last local feature map in the plurality of local feature maps, acquiring the current local feature map, performing convolution operation on the current local feature map to obtain a convolved specified local feature map, and taking the specified local feature map corresponding to the current local feature map as the target sample feature map.
8. The method of claim 4, wherein the outputs of the bottom-up sub-network are coupled to the inputs of the classification network, and wherein the predetermined initial model is used to:
inputting a minimum-scale sample feature map to the classification network through the bottom-up sub-network;
and dividing the minimum-scale sample feature map into a plurality of anchor frames through the classification network, determining a second probability that each anchor frame belongs to the lane line, and determining a second predicted position of the lane line in the lane line detection sample image according to the second probability.
9. The method according to any one of claims 2 to 8, wherein inputting the target detection image into a preset lane line detection model to cause the preset lane line detection model to output a target position of a lane line in the target detection image comprises:
acquiring a target feature map corresponding to the target detection image through the feature extraction network;
and inputting the target feature map into the classification network to obtain the target position of a lane line in the target detection image output by the classification network.
10. A lane line detection apparatus, characterized in that the apparatus comprises:
an acquisition module configured to acquire a target detection image;
a determination module configured to input the target detection image into a preset lane line detection model so that the preset lane line detection model outputs a target position of a lane line in the target detection image;
the preset lane line detection model is obtained by training in the following mode:
acquiring a plurality of lane line detection sample images, wherein the lane line detection images comprise lane line marking positions;
inputting each lane line detection sample image into a preset initial model, wherein the preset initial model comprises a segmentation network and a classification network, the segmentation network is used for determining a first predicted position of a lane line in the lane line detection sample image, and the classification network is used for determining a second predicted position of the lane line in the lane line detection sample image;
and training the preset initial model according to the first predicted position, the second predicted position and the lane line marking position to obtain the preset lane line detection model.
11. A vehicle characterized by comprising the lane line detection apparatus of claim 10 above.
12. A lane line detection apparatus, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 9.
13. A computer-readable storage medium, on which computer program instructions are stored, which program instructions, when executed by a processor, carry out the steps of the method according to any one of claims 1 to 9.
CN202210444361.XA 2022-04-25 2022-04-25 Lane line detection method, lane line detection device, vehicle, and storage medium Pending CN114863392A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210444361.XA CN114863392A (en) 2022-04-25 2022-04-25 Lane line detection method, lane line detection device, vehicle, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210444361.XA CN114863392A (en) 2022-04-25 2022-04-25 Lane line detection method, lane line detection device, vehicle, and storage medium

Publications (1)

Publication Number Publication Date
CN114863392A true CN114863392A (en) 2022-08-05

Family

ID=82634163

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210444361.XA Pending CN114863392A (en) 2022-04-25 2022-04-25 Lane line detection method, lane line detection device, vehicle, and storage medium

Country Status (1)

Country Link
CN (1) CN114863392A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523550A (en) * 2023-11-22 2024-02-06 中化现代农业有限公司 Apple pest detection method, apple pest detection device, electronic equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523550A (en) * 2023-11-22 2024-02-06 中化现代农业有限公司 Apple pest detection method, apple pest detection device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
CN108629354B (en) Target detection method and device
CN106651955B (en) Method and device for positioning target object in picture
CN107480665B (en) Character detection method and device and computer readable storage medium
US11288531B2 (en) Image processing method and apparatus, electronic device, and storage medium
CN111461182B (en) Image processing method, image processing apparatus, and storage medium
CN111104920B (en) Video processing method and device, electronic equipment and storage medium
US20220392202A1 (en) Imaging processing method and apparatus, electronic device, and storage medium
US20210326649A1 (en) Configuration method and apparatus for detector, storage medium
US20220222831A1 (en) Method for processing images and electronic device therefor
CN113194253B (en) Shooting method and device for removing reflection of image and electronic equipment
CN114267041B (en) Method and device for identifying object in scene
CN108717542B (en) Method and device for recognizing character area and computer readable storage medium
CN112200040A (en) Occlusion image detection method, device and medium
CN111931781A (en) Image processing method and device, electronic equipment and storage medium
KR102446688B1 (en) Target tracking method and apparatus, electronic device and storage medium
CN113052874B (en) Target tracking method and device, electronic equipment and storage medium
CN114863392A (en) Lane line detection method, lane line detection device, vehicle, and storage medium
CN113496237B (en) Domain adaptive neural network training and traffic environment image processing method and device
CN113313115A (en) License plate attribute identification method and device, electronic equipment and storage medium
CN112990197A (en) License plate recognition method and device, electronic equipment and storage medium
CN111709993A (en) Object pose information determination method and device, terminal and storage medium
CN116206363A (en) Behavior recognition method, apparatus, device, storage medium, and program product
CN112330721B (en) Three-dimensional coordinate recovery method and device, electronic equipment and storage medium
CN114842457B (en) Model training and feature extraction method and device, electronic equipment and medium
CN115861739B (en) Training method, device, equipment, storage medium and product of image segmentation model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination