CN115063761A - Lane line detection method, device, equipment and storage medium - Google Patents
Lane line detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115063761A CN115063761A CN202210555568.4A CN202210555568A CN115063761A CN 115063761 A CN115063761 A CN 115063761A CN 202210555568 A CN202210555568 A CN 202210555568A CN 115063761 A CN115063761 A CN 115063761A
- Authority
- CN
- China
- Prior art keywords
- lane line
- image
- features
- lane
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 131
- 238000000034 method Methods 0.000 claims abstract description 23
- 230000004044 response Effects 0.000 claims description 23
- 230000006870 function Effects 0.000 claims description 19
- 230000007246 mechanism Effects 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000012805 post-processing Methods 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 230000004927 fusion Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 241000283070 Equus zebra Species 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of automatic driving, and discloses a lane line detection method, a lane line detection device, lane line detection equipment and a storage medium. The method comprises the following steps: acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features; identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point; extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units; and carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected. The invention reduces the calculated amount of lane line detection and improves the accuracy of lane line detection.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to a lane line detection method, a lane line detection device, lane line detection equipment and a storage medium.
Background
With the development of intelligent driving technology, lane line detection has become one of important basic functions of vehicle automatic driving, and accurately detecting and identifying lane lines is an important prerequisite for functions of lane departure early warning, lane keeping, lane changing and the like of an automatic driving vehicle. And the detection of lane lines from the images is a key technology for intelligent driving.
The existing schemes mainly comprise a bottom-up scheme and a top-down scheme. For the bottom-up lane line detection, the image is generally segmented pixel by pixel, and then the vectorized lane line is obtained through post-processing. The method uses the characteristic of each local part to predict whether each pixel is a lane line pixel. The method needs complicated and error-prone rule-based post-processing, is not robust enough for processing the conditions of sheltered and unclear lane lines, and needs a large amount of calculation. For top-down lane line detection, this method generally obtains a feature description of each lane line based on an anchor frame or a global attention mechanism, and the field of view is large, but each local feature is not fully utilized to predict the lane line, so that the lane line far away or with large curvature does not perform well enough. In summary, the conventional lane line detection method has a problem of insufficient accuracy.
Disclosure of Invention
The invention mainly aims to solve the technical problem that the accuracy of the existing lane line detection method is insufficient.
The invention provides a lane line detection method in a first aspect, which comprises the following steps: acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features; identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point; extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units; and carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
Optionally, in a first implementation manner of the first aspect of the present invention, the detecting a second key point of a lane line in the local image feature includes: performing convolution processing on the local image features to obtain a corresponding feature map, and determining a response value of each convolved key point in the feature map; and determining a second key point of the lane line from the feature map according to the response value.
Optionally, in a second implementation manner of the first aspect of the present invention, the identifying, based on the first key point and the second key point, each lane line candidate unit corresponding to the image to be detected includes: adding noise to the first key points to obtain first key points with noise, and selecting preset first number of second key points from all second key points according to the response value from high to low; and taking the first key point with noise and the selected second key point as each lane line candidate unit corresponding to the image to be detected.
Optionally, in a third implementation manner of the first aspect of the present invention, the extracting, by using each lane line candidate unit, an image lane line feature from the image local feature to obtain a lane line global feature corresponding to the lane line candidate unit includes: determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature; and mutually fusing the position characteristic, the content characteristic and the image local characteristic by using a preset attention mechanism to obtain the lane line global characteristic corresponding to each lane line candidate unit.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the extracting, by using each lane line candidate unit, an image lane line feature from the image local feature to obtain a lane line global feature corresponding to the lane line candidate unit includes: determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature; predicting a preset second number of lane line candidate units related to lane lines in the lane line candidate units according to the position features and the content features; screening and predicting image local features corresponding to lane line candidate units from the image local features and taking the image local features as lane line features; and fusing the lane line features to the lane line candidate units obtained by corresponding prediction to obtain the lane line global features of the corresponding lane line candidate units.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the performing lane line track detection on the corresponding lane line candidate unit by using the lane line global feature to obtain the lane line corresponding to the image to be detected includes: stacking the global features of the lane lines and the corresponding local features of the images to obtain a feature tensor corresponding to each lane line candidate unit; and according to the characteristic tensor, carrying out lane line track detection on each lane line candidate unit to obtain a corresponding detection result, and generating a lane line corresponding to the image to be detected according to the detection result.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after performing lane line trajectory detection on each lane line candidate unit according to the feature tensor to obtain a corresponding detection result, the method further includes: calculating a first loss value by using a preset first loss function according to a detection result of the lane line candidate unit corresponding to the first key point; calculating a second loss value by using a preset second loss function according to the detection result of the lane line candidate unit corresponding to the second key point; and adjusting the detection result according to the first loss value and the second loss value to obtain a new detection result.
A second aspect of the present invention provides a lane line detection apparatus, including: the extraction module is used for acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features; the identification module is used for identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point; the extraction module is used for extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units; and the detection module is used for carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
Optionally, in a first implementation manner of the second aspect of the present invention, the extracting module includes: the convolution unit is used for performing convolution processing on the local image characteristics to obtain a corresponding characteristic diagram and determining a response value after convolution of each key point in the characteristic diagram; and the key point determining unit is used for determining a second key point of the lane line from the feature map according to the response value.
Optionally, in a second implementation manner of the second aspect of the present invention, the identification module includes: the selecting unit is used for adding noise to the first key points to obtain the first key points with noise, and selecting preset first number of second key points from all the second key points according to the response value from high to low; and the unit determining unit is used for taking the first key point with noise and the selected second key point as each lane line candidate unit corresponding to the image to be detected.
Optionally, in a third implementation manner of the second aspect of the present invention, the extraction module includes a fusion unit, configured to: determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature; and mutually fusing the position characteristic, the content characteristic and the image local characteristic by using a preset attention mechanism to obtain the lane line global characteristic corresponding to each lane line candidate unit.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the extraction module further includes a prediction unit, configured to: determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature; predicting a preset second number of lane line candidate units related to lane lines in the lane line candidate units according to the position features and the content features; screening and predicting image local features corresponding to lane line candidate units from the image local features and taking the image local features as lane line features; and fusing the lane line features to the lane line candidate units obtained by corresponding prediction to obtain the lane line global features of the corresponding lane line candidate units.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the detecting module includes: the stacking unit is used for stacking the global features of the lane lines and the corresponding local features of the images to obtain a feature tensor corresponding to each lane line candidate unit; and the detection unit is used for carrying out lane line track detection on each lane line candidate unit according to the feature tensor to obtain a corresponding detection result, and the detection result generates the lane line corresponding to the image to be detected.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the detection module further includes a learning unit, configured to: calculating a first loss value by using a preset first loss function according to a detection result of the lane line candidate unit corresponding to the first key point; calculating a second loss value by using a preset second loss function according to the detection result of the lane line candidate unit corresponding to the second key point; and adjusting the detection result according to the first loss value and the second loss value to obtain a new detection result.
A third aspect of the present invention provides a lane line detection apparatus, comprising: a memory and at least one processor, the memory having instructions stored therein; the at least one processor invokes the instructions in the memory to cause the lane line detection apparatus to perform the lane line detection method described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the lane line detection method described above.
According to the technical scheme, a novel neural network architecture is designed, a first key point is obtained by detecting the key points of the lane lines, and a plurality of lane line candidate units are obtained. And simultaneously, inputting a second key point of the key points of the disturbed true lane line into the network as a lane selection candidate unit. And calculating spatial features and content features for each lane line candidate unit, and performing information interaction between the features and image features by using a self-attention mechanism and a cross-attention mechanism to finally obtain global lane line features. We stack the global lane line features further with the local image features, and finally regress the position and category of the whole lane line. Therefore, the lane line detection result can be directly output end to end without post-processing; the problem of local feature missing can be overcome by using global features; the problem that a distance or a curve and the like are difficult to process by global features can be solved by using the local features, so that the accuracy of lane line detection is ensured while the overall calculated amount is reduced.
Drawings
FIG. 1 is a schematic diagram of a lane marking detection method according to a first embodiment of the present invention;
FIG. 2 is a diagram of a lane marking detection method according to a second embodiment of the present invention;
FIG. 3 is a diagram of a lane marking detection method according to a third embodiment of the present invention;
FIG. 4 is a schematic diagram of an embodiment of a lane marking detection apparatus according to an embodiment of the present invention;
fig. 5 is a schematic view of another embodiment of the lane line detecting device according to the embodiment of the present invention;
fig. 6 is a schematic diagram of an embodiment of the lane line detection apparatus according to the embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a lane line detection method, a lane line detection device, lane line detection equipment and a storage medium, wherein an image to be detected corresponding to a driving scene is obtained, image local features and first key points in the image to be detected are extracted, and second key points of a lane line in the local image features are detected; identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point; extracting image lane line features from the image local features by utilizing each lane line candidate unit to obtain lane line global features corresponding to the lane line candidate units; and carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected. The invention reduces the calculated amount of lane line detection and improves the accuracy of lane line detection.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a detailed flow of the embodiment of the present invention is described below, and referring to fig. 1, a first embodiment of the lane line detection method in the embodiment of the present invention includes:
101. acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features;
it is to be understood that the executing subject of the present invention may be a lane line detecting device, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
In the embodiment, in the driving process of the vehicle, a current driving scene is shot through the camera equipment arranged on the vehicle, and a plurality of images to be detected can be obtained; or the image to be detected corresponding to the driving scene obtained by vehicle history shooting; or an image to be detected generated by simulating a driving scene. Wherein, each image to be detected can be an image shot in different directions of the driving scene.
In the embodiment, each image to be detected performs image feature extraction through a convolutional neural network to extract image features related to lane lines in each image to be detected, and each image to be detected independently extracts locally related features in the image, so that image local features are extracted and only image features of a current vehicle in a certain direction of a driving scene are represented.
Specifically, the image local features of the image to be detected can be extracted by adopting a residual neural network, a feature pyramid and the like, so that the image local features with the feature size of N × H × W × C are obtained, wherein N is the batch size, H is the height, W is the width, and C is the number of channels.
In this embodiment, for the target detection image, a true value of each lane line is labeled in advance, where a key point of each labeled region may be directly extracted as a first key point, and each first key point includes information of the lane line true value labeled in advance. And for the extracted image local features, performing further convolution processing with smaller granularity on the image local features through the convolution neural network to obtain second key points related to lane line prediction.
Specifically, the size of the key point may be one pixel point, or may be a combination of a preset number of adjacent pixel points, for example, 4 pixel points 2 × 2 form a first pixel point and a second pixel point of a square.
102. Identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point;
in this embodiment, based on a convolutional neural network, target detection is performed on an image to be detected, a first key point and a second key point related to a lane line are extracted from the image to be detected, and then a plurality of lane line candidate units with higher degree of correlation with the lane line in the image to be detected are further identified from the first key point and the second key point. The identification method may specifically be based on the screening of the response value of the previous convolution, or may further be based on the selection of a deep learning algorithm.
Specifically, the first key point and the second key point include partial information in local features of the image, including position information in the image to be detected, detail information about a lane line, and the like. And at the moment, the first key point and the second key point still contain position information and detail information based on the image to be detected, the first key point and the second key point of different images to be detected are integrated on the same plane image, and then the lane line candidate unit of the image to be detected in the whole situation is identified.
103. Extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units;
in this embodiment, the lane line candidate unit is each unit area with a higher lane line correlation degree, which is detected and identified from the image to be detected, and carries the position of the global plane image of the driving scene where the lane line is located and the lane line information of the unit area, and the lane line feature with a higher lane line correlation degree is extracted from the local image features of the image to be detected, which are extracted in the past and fused into the corresponding lane line candidate unit, so that the information abundance degree of the lane line candidate unit is enriched, and the accuracy of the subsequent lane line prediction is improved.
Specifically, the image lane line features are extracted from the image local features based on the lane line candidate unit and are fused, and the two features can be matched, screened and fused based on the attention mechanism, so that a surface (lane line global features) is generated based on a point (lane line candidate unit) and a surface (image local features); or predicting a preset number of unit areas directly from the lane line candidate units, and then directly fusing the selected unit areas with the image local features at the corresponding positions to generate a surface (lane line global features) based on the points (lane line candidate units).
104. And carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
In the embodiment, the fine detection granularity of the global features of the lane lines is larger than that of all pixels applying the image to be detected, but the global features of the lane lines contain the related information of the lane lines with the same information amount; meanwhile, the granularity is smaller than the fine granularity of a prediction frame of the image to be detected, and the image to be detected contains more lane line related information; and (4) predicting whether the lane line candidate unit is the lane line or not and the category of the lane line by combining the advantages of the lane line candidate unit and the candidate unit, and finally mapping the lane line candidate unit back to the image to be detected to obtain the lane line of the image to be detected.
In this embodiment, after the global features of the lane lines are extracted, each lane line candidate unit is classified by a classifier, which includes: whether the lane line is the lane line or not and the type of the lane line such as a solid line, a broken line, a zebra crossing and the like realize end-to-end lane line prediction without image post-processing.
In the embodiment of the invention, a novel neural network architecture is designed, and a plurality of lane line candidate units are obtained by detecting the lane line key points to obtain the first key points. And simultaneously, inputting a second key point of the key points of the disturbed true lane line into the network as a lane selection candidate unit. And calculating spatial features and content features for each lane line candidate unit, and performing information interaction between the features and image features by using a self-attention mechanism and a cross-attention mechanism to finally obtain global lane line features. We stack the global lane line features further with the local image features, and finally regress the position and category of the whole lane line. Therefore, the lane line detection result can be directly output end to end without post-processing; the problem of local feature missing can be overcome by using global features; the problem that a distance or a curve and the like are difficult to process by global features can be solved by using the local features, so that the accuracy of lane line detection is ensured while the overall calculated amount is reduced.
Referring to fig. 2, a second embodiment of the lane line detection method according to the embodiment of the present invention includes:
201. acquiring an image to be detected corresponding to a driving scene, and extracting image local features and first key points in the image to be detected;
202. performing convolution processing on the local image features to obtain a corresponding feature map, and determining a response value of each convolved key point in the feature map;
203. determining a second key point of the lane line from the feature map according to the response value;
in this embodiment, the local image features in the image to be detected are extracted through a convolutional neural network such as a residual neural network and a feature pyramid network, here, a feature map including a plurality of key points is further extracted from the local image features, each key point in the feature map carries a response value after convolution processing, and the higher the response value is, the higher the probability that the corresponding key point is a lane line is, so that a second key point with the highest degree of correlation with the lane line is screened out from the key points in the feature map based on the response value.
Specifically, when the feature map h (X) is extracted from the local image feature X by using the residual neural network, the feature map h (X) is extracted from the local image feature X by using the residual function X + f (X) ═ h (X) and the trained or trained f (X).
Specifically, if a feature pyramid network is used, C is obtained by down-sampling the local image features with a multilayer convolution layer 1 、C 2 、……、C n Then obtaining the feature M with the same channel number through each convolution layer 1 、M 2 、……、M n Then from M n Down to M 1 In combination with C 1 、C 2 、……、C n And performing upsampling processing, and performing convolution of a preset convolution kernel, such as a convolution kernel of 3 x 3, to obtain a final feature map. Wherein, in each up-sampling, M i ＝M j i+1 +C i And i belongs to n, and j is an upsampling multiple.
In addition, if the feature size of the local image feature is N × H × W × C, the feature map extracted here is N × H × W × 1, the corresponding response value may be the confidence of each keypoint, and a larger response value indicates that the keypoint is more likely to be associated with the lane line.
204. Adding noise to the first key points to obtain first key points with noise, and selecting a preset first number of second key points from all second key points according to the response value from high to low;
205. taking the first key points with noise and the second key points obtained by selection as the candidate units of each lane line corresponding to the image to be detected;
in this embodiment, when the lane line candidate unit in the image to be detected is detected based on the first key point and the second key point, the first key point and the second key point are preprocessed to be used as input of subsequent detection. And then screening out the lane line candidate units at the corresponding positions of the image to be detected according to the preprocessed first key points and the preprocessed second key points.
Specifically, for a first key point which is pre-labeled with a true value, noise is added according to a preset rule or randomly to disturb the true value labeling, so that the false value labeling is used as a lane line candidate unit for subsequent model feature extraction, fusion, stacking and detection. For example, each first keypoint is shifted by a random position, and the position information is changed.
Specifically, for the detected second keypoints, lane line candidate units are screened according to the response values obtained by the previous convolution from high to low, and since the higher the response values are, the higher the degree of correlation between the corresponding second keypoints and the lane lines is, the preset first number of lane line candidate units with the highest degree of correlation with the lane lines is screened here. And preliminarily screening out second key points from the key points of the image to be detected, and limiting the number of the second key points by a second number, thereby further reducing the calculation amount.
Specifically, here, the lane line candidate unit may be extracted based on an autonomous learning manner. Based on a deep learning algorithm, the first key point and the second key point are used as input, the first key point and the second key point which are probably lane lines and the pixel points around the second key point and the second key point are detected, and the lane line candidate unit is formed.
206. Determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
207. mutually fusing the position characteristic, the content characteristic and the image local characteristic by using a preset attention mechanism to obtain lane line global characteristics corresponding to each lane line candidate unit;
in this embodiment, the lane line candidate unit carries a position feature Q _ p and a content feature Q _ c, where the position feature Q _ p represents position information of the lane line candidate unit in the image to be detected, and the content feature Q _ c includes pixel point information of the lane line candidate unit, detailed information of the lane line, and the like. And performing self attention matching and cross attention matching with the image local features through the position features Q _ p and the content features Q _ c to determine the matching relation between the extracted image local features and the lane line candidate units, and fusing the extracted image local features and the lane line candidate units to obtain the lane line global features.
Specifically, here, a surface (lane line global feature) is generated based on a point (lane line candidate unit) and a surface (image local feature) by using an attention mechanism. For example, by using a self-attention and cross-attention mechanism, for each lane line candidate unit, the image local features of the adjacent lane line candidate units on the cross path of the lane line candidate unit can be acquired, through a recursive operation, each lane line candidate unit can finally acquire the remote dependence of the image local features of all the lane line candidate units, and the corresponding position features and content features are respectively fused with the dependent image local features to generate the lane line global features of each lane line candidate unit.
208. And carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
Referring to fig. 3, a third embodiment of the lane line detection method according to the embodiment of the present invention includes:
301. acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features;
302. identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point;
303. determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
304. predicting a preset second number of lane line candidate units related to lane lines in the lane line candidate units according to the position features and the content features;
305. screening and predicting image local features corresponding to lane line candidate units from the image local features and taking the image local features as lane line features;
306. fusing the lane line features to the lane line candidate units obtained by corresponding prediction to obtain the lane line global features of the corresponding lane line candidate units;
in this embodiment, for the lane line candidate units carrying the position feature Q _ p and the content feature Q _ c, a preset second number of lane line candidate units more related to the lane line may be directly screened from the lane line candidate units, and the lane line features of the same pixel point or the adjacent pixel points are matched from the image local features to obtain the lane line global feature through fusion.
Specifically, in this embodiment, in a manner of generating a plane (lane line global feature) based on points (lane line candidate units), the lane line candidate units related to lane lines in the overall situation are further learned according to the position features and content features in the lane line candidate units through deep learning, and then a second number of lane line candidate units are selected according to the confidence level, so as to select lane line features for fusion, thereby obtaining the lane line global feature.
307. Stacking the global features of the lane lines and the corresponding local features of the images to obtain a feature tensor corresponding to each lane line candidate unit;
308. and according to the feature tensor, carrying out lane line track detection on each lane line candidate unit to obtain a corresponding detection result, and generating a lane line corresponding to the image to be detected according to the detection result.
In this embodiment, the lane line global feature and the image local feature are stacked, and the prediction of the lane line is performed from the consideration of the local and global relevant features of the driving scene. After the global lane line features and the local image features are stacked, feature tensors which represent the whole driving scene and are related to the lane lines can be obtained, and the feature tensors are input into a classifier for classification.
Specifically, the feature tensor is an array of data with different dimensions after the global feature of the lane line and the local feature of the image are stacked, and may include data with different dimensions of three orders of the position feature, the content feature and the local feature of the image, so as to form a third-order feature tensor.
In addition, for historical data sets to be detected, supervised learning can be performed on the convolutional neural network in the front so as to update the detection result, and the specific steps are as follows:
1) calculating a first loss value by using a preset first loss function according to a detection result of the lane line candidate unit corresponding to the first key point;
2) calculating a second loss value by using a preset second loss function according to the detection result of the lane line candidate unit corresponding to the second key point;
3) and adjusting the detection result according to the first loss value and the second loss value to obtain a new detection result.
In this embodiment, for the lane line candidate unit obtained by detecting the first keypoint, the optimal binary matching (first loss function) is performed on the lane line formed by the predicted first keypoint and the marked actual lane line, and a first loss value is calculated; for the lane line candidate unit detected from the second keypoint after the noise processing based on the labeled keypoints, since the actual lane line to which the lane line composed of the second keypoints should correspond has already been determined, the second loss value may be calculated by using a direct matching method (second loss function).
Specifically, for a first loss function such as optimal binary matching, a first key point set X is input, actual key points are combined with Y, a bounding box A of a global plane image of a driving scene is defined, and X and Y loss L replacement is unchanged. X, Y is used to construct a binary ranking function σ as follows:
by calculating the matching cost of X and Y, and combining A, the optimal binary matching is searched in the binary sorting function, so that the matching loss of the two is minimum, which is specifically shown as follows:
L match (x i .y σ(j) )＝-∏p i (x i )+∏L box (a i ,a σ(j) )，a i ,a σ(j) ∈A。
here again, the calculated matching loss is taken as the first loss value.
Specifically, for the second loss function, such as the direct matching method, the distance between the first keypoint set X and the actual keypoint set at the corresponding position may be directly calculated, and this is used as the second loss value. And finally, performing model parameter adjustment on the convolutional neural network by combining the first loss value and the second loss value so as to adjust the final inspection result to obtain a new detection result.
In the above description of the lane line detection method in the embodiment of the present invention, the following description of the lane line detection device in the embodiment of the present invention refers to fig. 4, and an embodiment of the lane line detection device in the embodiment of the present invention includes:
the extraction module 401 is configured to acquire an image to be detected corresponding to a driving scene, extract image local features and first key points in the image to be detected, and detect second key points of a lane line in the local image features;
an identifying module 402, configured to identify, based on the first key point and the second key point, each lane line candidate unit corresponding to the image to be detected;
an extraction module 403, configured to extract, by using each lane line candidate unit, an image lane line feature from the image local features to obtain a lane line global feature corresponding to the lane line candidate unit;
and the detection module 404 is configured to perform lane line track detection on the corresponding lane line candidate unit by using the lane line global features to obtain a lane line corresponding to the image to be detected.
In the embodiment of the invention, a novel neural network architecture is designed, and a plurality of lane line candidate units are obtained by detecting the lane line key points to obtain the first key points. And simultaneously, inputting a second key point of the key points of the disturbed true lane line into the network as a lane selection candidate unit. And calculating spatial features and content features for each lane line candidate unit, and performing information interaction between the features and image features by using a self-attention mechanism and a cross-attention mechanism to finally obtain global lane line features. We stack the global lane line features further with the local image features, and finally regress the position and category of the whole lane line. Therefore, the lane line detection result can be directly output end to end without post-processing; the problem of local feature missing can be overcome by using global features; the problem that a distance or a curve and the like are difficult to process by global features can be solved by using the local features, so that the accuracy of lane line detection is ensured while the overall calculated amount is reduced.
Referring to fig. 5, another embodiment of the lane line detection apparatus according to the embodiment of the present invention includes:
the extraction module 401 is configured to acquire an image to be detected corresponding to a driving scene, extract image local features and first key points in the image to be detected, and detect second key points of a lane line in the local image features;
an identifying module 402, configured to identify, based on the first key point and the second key point, each lane line candidate unit corresponding to the image to be detected;
an extraction module 403, configured to extract, by using each lane line candidate unit, an image lane line feature from the image local features to obtain a lane line global feature corresponding to the lane line candidate unit;
and the detection module 404 is configured to perform lane line track detection on the corresponding lane line candidate unit by using the lane line global features to obtain a lane line corresponding to the image to be detected.
Specifically, the extracting module 401 includes:
a convolution unit 4011, configured to perform convolution processing on the local image feature to obtain a corresponding feature map, and determine a response value after convolution of each key point in the feature map;
and a key point determining unit 4012, configured to determine a second key point of the lane line from the feature map according to the response value.
Specifically, the identifying module 402 includes:
a selecting unit 4021, configured to add noise to the first keypoints to obtain noisy first keypoints, and select a preset first number of second keypoints from all second keypoints according to the response value from high to low;
a unit determining unit 4022, configured to use the first keypoints with noise and the selected second keypoints as candidate units of each lane line corresponding to the image to be detected.
Specifically, the extraction module 403 includes a fusion unit 4031, configured to:
determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
and mutually fusing the position characteristic, the content characteristic and the image local characteristic by using a preset attention mechanism to obtain the lane line global characteristic corresponding to each lane line candidate unit.
Specifically, the extraction module 403 further includes a prediction unit 4032, configured to:
determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
predicting a preset second number of lane line candidate units related to lane lines in the lane line candidate units according to the position features and the content features;
screening and predicting image local features corresponding to lane line candidate units from the image local features and taking the image local features as lane line features;
and fusing the lane line features to the lane line candidate units obtained by corresponding prediction to obtain the lane line global features of the corresponding lane line candidate units.
Specifically, the detecting module 404 includes:
a stacking unit 4041, configured to stack the lane line global features and the corresponding image local features to obtain a feature tensor corresponding to each lane line candidate unit;
the detection unit 4042 is configured to perform lane line trajectory detection on each lane line candidate unit according to the feature tensor to obtain a corresponding detection result, and the detection result generates a lane line corresponding to the image to be detected.
Specifically, the detecting module 404 further includes a learning unit 4043, configured to:
calculating a first loss value by using a preset first loss function according to a detection result of the lane line candidate unit corresponding to the first key point;
calculating a second loss value by using a preset second loss function according to the detection result of the lane line candidate unit corresponding to the second key point;
and adjusting the detection result according to the first loss value and the second loss value to obtain a new detection result.
The lane line detection device in the embodiment of the present invention is described in detail in terms of the modular functional entity in fig. 4 and 5, and the lane line detection device in the embodiment of the present invention is described in detail in terms of the hardware processing.
Fig. 6 is a schematic structural diagram of a lane line detection apparatus 600 according to an embodiment of the present invention, where the lane line detection apparatus 600 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing applications 633 or data 632. Memory 620 and storage medium 630 may be, among other things, transient or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations for the lane line detection apparatus 600. Further, the processor 610 may be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the lane line detection apparatus 600.
The lane line detection apparatus 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input-output interfaces 660, and/or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the lane line detection apparatus configuration shown in fig. 6 does not constitute a limitation of the lane line detection apparatus, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The invention further provides lane line detection equipment, which comprises a memory and a processor, wherein computer readable instructions are stored in the memory, and when being executed by the processor, the computer readable instructions cause the processor to execute the steps of the lane line detection method in the embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the lane line detection method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a portable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A lane line detection method is characterized by comprising the following steps:
acquiring an image to be detected corresponding to a driving scene, extracting local image features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features;
identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point;
extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units;
and carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
2. The lane line detection method according to claim 1, wherein the detecting a second key point of a lane line in the local image feature comprises:
performing convolution processing on the local image features to obtain a corresponding feature map, and determining a response value of each convolved key point in the feature map;
and determining a second key point of the lane line from the feature map according to the response value.
3. The lane line detection method according to claim 2, wherein the identifying each lane line candidate unit corresponding to the image to be detected based on the first and second key points comprises:
adding noise to the first key points to obtain first key points with noise, and selecting a preset first number of second key points from all second key points according to the response value from high to low;
and taking the first key point with noise and the selected second key point as each lane line candidate unit corresponding to the image to be detected.
4. The method according to claim 1, wherein the extracting an image lane feature from the image local feature by using each lane line candidate unit to obtain a lane line global feature corresponding to the lane line candidate unit includes:
determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
and mutually fusing the position characteristic, the content characteristic and the image local characteristic by using a preset attention mechanism to obtain the lane line global characteristic corresponding to each lane line candidate unit.
5. The method according to claim 1, wherein the extracting an image lane feature from the image local feature by using each lane line candidate unit to obtain a lane line global feature corresponding to the lane line candidate unit includes:
determining a feature vector in each lane line candidate unit, wherein the feature vector comprises a position feature and a content feature;
predicting a preset second number of lane line candidate units related to lane lines in the lane line candidate units according to the position features and the content features;
screening and predicting image local features corresponding to lane line candidate units from the image local features and taking the image local features as lane line features;
and fusing the lane line features to the lane line candidate units obtained by corresponding prediction to obtain the lane line global features of the corresponding lane line candidate units.
6. The lane line detection method according to any one of claims 1 to 5, wherein the performing lane line trajectory detection on the corresponding lane line candidate unit by using the lane line global feature to obtain the lane line corresponding to the image to be detected comprises:
stacking the global features of the lane lines and the corresponding local features of the images to obtain a feature tensor corresponding to each lane line candidate unit;
and according to the characteristic tensor, carrying out lane line track detection on each lane line candidate unit to obtain a corresponding detection result, and generating a lane line corresponding to the image to be detected according to the detection result.
7. The method according to claim 6, further comprising, after performing lane line trajectory detection on each lane line candidate unit according to the feature tensor to obtain a corresponding detection result:
calculating a first loss value by using a preset first loss function according to a detection result of the lane line candidate unit corresponding to the first key point;
calculating a second loss value by using a preset second loss function according to the detection result of the lane line candidate unit corresponding to the second key point;
and adjusting the detection result according to the first loss value and the second loss value to obtain a new detection result.
8. A lane line detection apparatus, characterized by comprising:
the extraction module is used for acquiring an image to be detected corresponding to a driving scene, extracting image local features and first key points in the image to be detected, and detecting second key points of a lane line in the local image features;
the identification module is used for identifying each lane line candidate unit corresponding to the image to be detected based on the first key point and the second key point;
the extraction module is used for extracting image lane line characteristics from the image local characteristics by using each lane line candidate unit to obtain lane line global characteristics corresponding to the lane line candidate units;
and the detection module is used for carrying out lane line track detection on the corresponding lane line candidate unit by adopting the global features of the lane lines to obtain the lane lines corresponding to the images to be detected.
9. A lane line detection apparatus, characterized by comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the lane line detection apparatus to perform the steps of the lane line detection method of any of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the steps of the lane line detection method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210555568.4A CN115063761B (en) | 2022-05-19 | 2022-05-19 | Lane line detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210555568.4A CN115063761B (en) | 2022-05-19 | 2022-05-19 | Lane line detection method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115063761A true CN115063761A (en) | 2022-09-16 |
| CN115063761B CN115063761B (en) | 2024-06-25 |
Family
ID=83198734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210555568.4A Active CN115063761B (en) | 2022-05-19 | 2022-05-19 | Lane line detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115063761B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019170012A1 (en) * | 2018-03-09 | 2019-09-12 | 腾讯科技（深圳）有限公司 | Traffic lane line data processing method and apparatus, computer device, and storage medium |
| CN111460984A (en) * | 2020-03-30 | 2020-07-28 | 华南理工大学 | Global lane line detection method based on key point and gradient balance loss |
| CN111783666A (en) * | 2020-07-01 | 2020-10-16 | 北京计算机技术及应用研究所 | Rapid lane line detection method based on continuous video frame corner feature matching |
| CN112949493A (en) * | 2021-03-03 | 2021-06-11 | 深圳瑞为智能科技有限公司 | Lane line detection method and system combining semantic segmentation and attention mechanism |
| CN113011338A (en) * | 2021-03-19 | 2021-06-22 | 华南理工大学 | Lane line detection method and system |
| CN113468967A (en) * | 2021-06-02 | 2021-10-01 | 北京邮电大学 | Lane line detection method, device, equipment and medium based on attention mechanism |
-
2022
- 2022-05-19 CN CN202210555568.4A patent/CN115063761B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019170012A1 (en) * | 2018-03-09 | 2019-09-12 | 腾讯科技（深圳）有限公司 | Traffic lane line data processing method and apparatus, computer device, and storage medium |
| CN111460984A (en) * | 2020-03-30 | 2020-07-28 | 华南理工大学 | Global lane line detection method based on key point and gradient balance loss |
| CN111783666A (en) * | 2020-07-01 | 2020-10-16 | 北京计算机技术及应用研究所 | Rapid lane line detection method based on continuous video frame corner feature matching |
| CN112949493A (en) * | 2021-03-03 | 2021-06-11 | 深圳瑞为智能科技有限公司 | Lane line detection method and system combining semantic segmentation and attention mechanism |
| CN113011338A (en) * | 2021-03-19 | 2021-06-22 | 华南理工大学 | Lane line detection method and system |
| CN113468967A (en) * | 2021-06-02 | 2021-10-01 | 北京邮电大学 | Lane line detection method, device, equipment and medium based on attention mechanism |
Non-Patent Citations (2)
| Title |
|---|
| ZHAN QU ET AL.: "Focus on Local: Detecting Lane Marker from Bottom Up via Key Point", 《2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》, 2 November 2021 (2021-11-02), pages 14117 - 14125 * |
| 景辉: "基于卷积神经网络的全局车道线检测算法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, 15 January 2019 (2019-01-15), pages 24 - 44 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115063761B (en) | 2024-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yang et al. | Inceptext: A new inception-text module with deformable psroi pooling for multi-oriented scene text detection | |
| Bevandić et al. | Discriminative out-of-distribution detection for semantic segmentation | |
| Lee et al. | Weakly supervised semantic segmentation using out-of-distribution data | |
| Molina-Cabello et al. | Vehicle type detection by ensembles of convolutional neural networks operating on super resolved images | |
| US20200074178A1 (en) | Method and system for facilitating recognition of vehicle parts based on a neural network | |
| US20180114071A1 (en) | Method for analysing media content | |
| Hoang et al. | Enhanced detection and recognition of road markings based on adaptive region of interest and deep learning | |
| Harish Padmanaban et al. | Optimizing the Identification and Utilization of Open Parking Spaces Through Advanced Machine Learning | |
| CN111274926B (en) | Image data screening method, device, computer equipment and storage medium | |
| CN113903022B (en) | Text detection method and system based on feature pyramid and attention fusion | |
| KR102197930B1 (en) | System and method for recognizing license plates | |
| Vijayakumar et al. | Yolo-based object detection models: A review and its applications | |
| CN115393598A (en) | Weakly supervised semantic segmentation method based on non-salient region object mining | |
| Ribeiro et al. | Brazilian mercosur license plate detection: a deep learning approach relying on synthetic imagery | |
| CN116872961A (en) | Control system for intelligent driving vehicle | |
| Wang et al. | ATG-PVD: ticketing parking violations on a drone | |
| CN115063761A (en) | Lane line detection method, device, equipment and storage medium | |
| CN114972434A (en) | End-to-end multi-target tracking system for cascade detection and matching | |
| Xu et al. | Pushing the envelope of thin crack detection | |
| Wang et al. | Extraction of main urban roads from high resolution satellite images by machine learning | |
| Satti et al. | Recognizing the Indian Cautionary Traffic Signs using GAN, Improved Mask R‐CNN, and Grab Cut | |
| Chang et al. | Re-Attention is all you need: Memory-efficient scene text detection via re-attention on uncertain regions | |
| Panda et al. | RRBCNN: Aircraft detection and classification using Bounding Box Regressor based on Scale Reduction module | |
| Wozniak et al. | Place inference via graph-based decisions on deep embeddings and blur detections | |
| Aluri et al. | Geometric Deep Learning for Enhancing Irregular Scene Text Detection. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |