CN110705388B - Target vehicle lane change identification method for auxiliary driving based on prediction feedback - Google Patents
Target vehicle lane change identification method for auxiliary driving based on prediction feedback Download PDFInfo
- Publication number
- CN110705388B CN110705388B CN201910870854.8A CN201910870854A CN110705388B CN 110705388 B CN110705388 B CN 110705388B CN 201910870854 A CN201910870854 A CN 201910870854A CN 110705388 B CN110705388 B CN 110705388B
- Authority
- CN
- China
- Prior art keywords
- target vehicle
- lane
- motion
- lane change
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008859 change Effects 0.000 title claims abstract description 160
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000001133 acceleration Effects 0.000 claims abstract description 44
- 238000005457 optimization Methods 0.000 claims abstract description 15
- 238000012937 correction Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000012360 testing method Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 239000007983 Tris buffer Substances 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 9
- 230000006399 behavior Effects 0.000 description 7
- 206010039203 Road traffic accident Diseases 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000012706 support-vector machine Methods 0.000 description 2
- 241000918728 Monotes Species 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/584—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of vehicle lights or traffic lights
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Traffic Control Systems (AREA)
- Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
Abstract
The invention provides a target vehicle lane change identification method for auxiliary driving based on prediction feedback, which comprises the steps of firstly, acquiring the motion state information and road structure information of surrounding target vehicles in real time to obtain the motion state of the target vehicles in a self-vehicle coordinate system; then calculating the motion state of the target vehicle under a ground coordinate system, and extracting the feature quantity of the lane changing intention of the target vehicle at the current moment; preliminarily identifying a lane change result of the target vehicle according to the lane change intention characteristic quantity; predicting the motion track of the target vehicle by adopting a polynomial fitting and optimization method for the preliminary recognition result; and taking the motion trail conforming to the uniform acceleration motion constraint as a reference motion trail of the target vehicle, calculating the accumulated distance deviation between the predicted motion trail of the target vehicle and the reference motion trail, and checking the initial recognition result to be used as a final lane change recognition result of the target vehicle. The method improves the accuracy of the target vehicle lane change identification under the observation noise, and enhances the robustness of the target vehicle lane change identification method.
Description
Technical Field
The invention relates to the technical field of environment perception of advanced auxiliary driving systems, in particular to a target vehicle lane change identification method for auxiliary driving based on prediction feedback.
Background
The dynamic changes of other moving vehicles in the road range have a vital influence on the road traffic safety, and in an advanced auxiliary driving system, the selection of the intelligent networked automobile driving strategy needs to consider the movement behaviors of other vehicles. The movement behaviors of vehicles on the structured road include lane changing and lane keeping, wherein the vehicle lane changing behavior poses a great threat to road traffic safety. According to the statistics of European Union, the number of traffic accidents caused by lane change accounts for 4% -10% of the total number of all traffic accidents, and the traffic delay time caused by the lane change traffic accidents accounts for 10% of the total delay time. Studies by the department of transportation in the united states have shown that recognition of a lane change by a vehicle is carried out 1.5 seconds ahead to allow sufficient response time to respond. Therefore, the lane change traffic accident can be effectively avoided by recognizing the lane change behavior of the vehicles on the structured road in advance.
The identification of the lane change behavior of the target vehicle (i.e. the vehicle to be identified) mainly comprises two major steps: firstly, extracting characteristic quantity of a target vehicle lane change intention; and then, recognizing the lane change intention according to the extracted characteristic quantity of the lane change intention of the target vehicle.
The existing research adopts two types of lane change intention characteristic quantities: the target vehicle motion state parameter and the running environment state parameter. The target vehicle motion state parameters include lateral and longitudinal motion speeds of the target vehicle, and a lateral distance of the vehicle from a lane line, and the like. The driving environment state parameters describe the relative motion relationship between the target vehicle and other vehicles on the road, such as the distance and relative speed between the target vehicle and the vehicle ahead on the same lane. The contribution of the lane change intention characteristic quantity in the lane change identification behavior is not equal, Schlechtriemen and the like compare the effects and the correlation of different characteristic quantities in the lane change identification behavior, and the three most effective lane change intention characteristic quantities, namely the longitudinal vehicle following relative speed of a target vehicle and a front vehicle in the same lane, the transverse movement speed of the target vehicle and the transverse distance between the target vehicle and a lane line, are obtained.
For lane change intention recognition algorithms, two categories can be currently distinguished: a rule-based identification method and a machine learning-based identification method. The rule-based identification method generally provides a series of lane change occurrence conditions, and when the target vehicle lane change intention characteristic quantity meets the lane change occurrence conditions, the target vehicle is considered to be changing lanes. For example, Monot and the like obtain the probability value of lane change of the target vehicle according to the transverse movement speed of the target vehicle and the transverse distance between the target vehicle and the lane line, and when the probability value is more than 50%, the target vehicle is considered to be in lane change. The algorithm based on the rules has high calculation efficiency, is suitable for an advanced assistant driving system with higher real-time requirement, but has low recognition accuracy. The machine learning-based method comprises a Support Vector Machine (SVM), a Hidden Markov Model (HMM), a Bayesian decision, a neural network and the like. The lane change identification method based on machine learning can process high-dimensional characteristic quantity input, has high identification accuracy rate, and has high requirement on computing resources.
The existing target vehicle lane change identification method adopts multi-dimensional lane change intention characteristic quantity, uses machine learning identification algorithm more, and shows that the lane change identification accuracy rate is more than 90% on the NGSIM data set (https:// ops.fhwa.dot.gov/traffic requirements/ngsim.htm) of the United states department of transportation. However, the target vehicle motion parameters and the driving environment parameters observed by the vehicle-mounted sensor system have observation noise, which has a great influence on the identification accuracy of the lane change identification algorithm, for example, the target vehicle lane change identification accuracy tested on the observation data processed by the kalman filter by the montot and the like is only 85%, and the recall rate is only 59%. In order to solve the problem, Woo et al predict a target vehicle lane change track and tracks of other vehicles on a road on the basis of a recognition result based on a lane change characteristic quantity, and if the future track of the target vehicle is crossed with the future tracks of the other vehicles, a collision risk exists, which indicates that the predicted track of the target vehicle is unreasonable, and further indicates that the recognition result of the lane change state of the target vehicle is unreasonable, and correction should be performed. Woo et al have improved the accuracy of lane change intent recognition by this method of predictive trajectory feedback correction. However, such feedback correction is suitable for a working condition with a large vehicle flow, and can be used only by other vehicles around the target vehicle, and cannot be performed under a high-speed working condition with a small vehicle flow.
Disclosure of Invention
The invention provides a track prediction feedback-based target vehicle lane change identification method, which aims to solve the problem of insufficient robustness of a target vehicle lane change identification method under the condition that observation noise exists in a vehicle-mounted sensor system.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a target vehicle lane change identification method for auxiliary driving based on prediction feedback, which is characterized by comprising the following steps:
s1, acquiring motion state information and road structure information of surrounding target vehicles in real time through a sensor system carried on the intelligent networked automobile, and obtaining the motion state of the target vehicle and a road and lane line equation of the target vehicle at the current moment in a self-vehicle coordinate system after lane line recognition and target detection tracking algorithm processing;
s2, calculating the motion state of the target vehicle at the current moment and a rectangular envelope frame of the target vehicle at the current moment in the ground coordinate system according to the motion state of the target vehicle at the current moment in the own vehicle coordinate system and a road lane line equation, and extracting the feature quantity of the lane change intention of the target vehicle at the current moment in the ground coordinate system; the lane-changing intention characteristic quantity comprises the transverse speed v of the center of mass of the target vehicle along the direction vertical to the lane line of the roadyAnd the transverse distance d from the rectangular envelope frame of the target vehicle to the lane line of the roadyAnd the longitudinal speed Deltav of the target vehicle relative to the moving vehicle in front of the same lane in the following scene along the tangential direction of the lane line of the roadxAnd a longitudinal distance dx；
S3, according to the extracted feature quantity of the target vehicle lane change intention at the current moment in the ground coordinate system, defining the predicted lane change time and the following danger coefficient of the target vehicle as lane change criterion; counting according to the HighD data set to obtain a lane change threshold value for predicting lane change time and a car following danger coefficient; if the predicted lane changing time or the following danger coefficient is larger than the corresponding lane changing threshold value of the target vehicle, preliminarily recognizing that the target vehicle is changing lanes or is about to change lanes, judging that the lane is changed leftwards or rightwards according to the sign of the predicted lane changing time, and otherwise, preliminarily recognizing that the target vehicle keeps the current lane;
s4, predicting the motion trail of the target vehicle by adopting a polynomial fitting and optimization method according to the preliminary identification result obtained in the step S3: firstly, determining the motion trail constraint of a target vehicle, obtaining a motion trail conforming to the motion trail constraint by using polynomial fitting, and then screening by an optimization method to obtain the most possible motion trail as a predicted motion trail of the target vehicle;
s5, taking the motion trail conforming to the uniform acceleration motion constraint as a reference motion trail of the target vehicle; calculating the accumulated distance deviation between the predicted motion track of the target vehicle and the reference motion track of the target vehicle, if the deviation value exceeds a set track deviation threshold value, judging that the predicted motion track of the target vehicle does not accord with motion track constraint, namely, the initial recognition result does not accord with motion constraint, if the initial recognition result is not lane change, correcting the predicted motion track to lane change, if the initial recognition result is lane change, correcting the predicted motion track to lane change, and after feedback correction, obtaining a final lane change recognition result of the target vehicle; and if the deviation value does not exceed the set track deviation threshold value, keeping the initial recognition result unchanged, and taking the initial recognition result as a final target vehicle lane change recognition result.
The invention has the characteristics and beneficial effects that:
the method comprises the steps of firstly identifying whether a target vehicle changes lanes or not by using lane change intention characteristic quantity, predicting a target vehicle motion track conforming to a lane change intention by using polynomial fitting according to a lane change intention identification result, simultaneously generating a target vehicle reference track conforming to motion constraint by using a uniform acceleration motion model, and correcting a lane change identification result if the predicted lane change track is inconsistent with the reference track in a short time, namely, identifying that the lane change is corrected as the lane change, and identifying that the lane change is corrected as the lane change. Due to the motion inertia of the vehicle, the motion of the vehicle is closer to a uniform acceleration motion model in a short time, so that the motion track generated by the uniform acceleration model is used as a reference track in a short time to check whether the predicted target vehicle track change track meets the motion inertia constraint. In addition, the motion trail curvature generated by polynomial fitting is continuous and accords with the actual motion situation of the vehicle, so that the motion trail of the target vehicle is predicted by polynomial fitting.
A test result on a HighD data set (https:// www.highd-dataset. com /) published by Gem industry university in Germany shows that by adding track prediction feedback to the recognition result based on the characteristic quantity, the method improves the accuracy of lane change recognition of the target vehicle under observation noise and enhances the robustness of the lane change recognition method of the target vehicle.
Drawings
Fig. 1 is an overall flowchart of a lane change recognition method for a driving assistance target vehicle according to an embodiment of the present invention.
FIG. 2 is a statistical probability density distribution diagram of the feature quantity of the predicted lane change time in the embodiment of the present invention.
FIG. 3 is a statistical probability density distribution diagram of the following risk coefficient characteristic quantity in the embodiment of the invention.
Fig. 4 is a schematic diagram of generation of a reference motion trajectory in the embodiment of the present invention.
FIG. 5 is a schematic diagram of fitting a motion trajectory in an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The invention provides a track prediction feedback-based lane change identification method for a target vehicle for driving assistance, which is shown in figure 1 and comprises the following steps:
s1, acquiring motion state information and road structure information of surrounding target vehicles in real time through a sensor system carried on the intelligent networked automobile, and obtaining the motion state of the target vehicle at the current moment and a road and lane line equation under a self-vehicle coordinate system after the information is processed by a conventional lane line identification and target detection tracking algorithm;
s2, according to the motion state of the target vehicle at the current moment in the own vehicle coordinate system and the road lane line equation obtained in the step S1, the motion state of the target vehicle at the current moment in the ground coordinate system and the rectangular envelope frame of the target vehicle are calculated, the lane change intention characteristic quantity of the target vehicle at the current moment in the ground coordinate system is extracted, and the lane change intention characteristic quantity comprises the transverse speed v of the center of mass of the target vehicle in the direction perpendicular to the road lane lineyAnd the transverse distance d from the rectangular envelope frame of the target vehicle to the lane line of the roadyAnd the longitudinal speed Deltav of the target vehicle relative to the moving vehicle in front of the same lane in the following scene along the tangential direction of the lane line of the roadxAnd a longitudinal distance dxAnd the like; the method comprises the following specific steps:
s2.1, converting the motion state of the target vehicle at the current moment in the own vehicle coordinate system and the road lane line equation obtained in the step S1 into the current moment t in the ground coordinate system0Motion state [ x (t) of target vehicle0),y(t0),vx(t0),vy(t0),ax(t0),ay(t0)]And a target vehicle rectangular envelope frame, wherein [ x (t)0),y(t0)]Is the position coordinate of the centroid of the target vehicle at the current moment, vx(t0),vy(t0) Respectively the longitudinal movement speed and the transverse movement speed of the target vehicle at the current moment，ax(t0),ay(t0) Respectively representing the longitudinal acceleration and the transverse acceleration of the target vehicle at the current moment;
s2.2, according to the motion state of the target vehicle and a lane line equation under the ground coordinate system, calculating the transverse velocity v of the mass center of the target vehicle along the direction vertical to the lane line at the current momentyThe vertical distance d from the rectangular envelope frame of the target vehicle to the lane liney；
S2.3, judging whether a moving vehicle exists in the front of the moving vehicle in the same lane of the target vehicle at the current moment, if so, calculating the tangential longitudinal speed delta v of the moving vehicle in the same lane relative to the target vehicle along the lane line of the roadxAnd a longitudinal distance dx；
S3, according to the extracted feature quantity of the target vehicle lane change intention at the current time under the ground coordinate system, defining the predicted lane change time TTLC and the following danger coefficient RP of the target vehicle as lane change criterion; counting according to the HighD data set to obtain a lane change threshold value for predicting lane change time and a car following danger coefficient; if the predicted lane changing time or the following danger coefficient is larger than the corresponding lane changing threshold value of the target vehicle, preliminarily recognizing that the target vehicle is changing lanes or is about to change lanes, judging that the lane is changed leftwards or rightwards according to the sign of the predicted lane changing time, and otherwise, preliminarily recognizing that the target vehicle keeps the current lane; the method specifically comprises the following steps:
s3.1, according to the extracted feature quantity of the target vehicle lane change intention at the current time under the ground coordinate system, defining the predicted lane change time TTLC and the following danger coefficient RP which are used as the lane change criterion of the target vehicle:
the predicted lane change time TTLC is defined as the vertical distance d from the rectangular envelope frame of the target vehicle to the lane line of the road at the current moment in the ground coordinate systemyTransverse velocity v in the direction perpendicular to the lane line of the road with the target vehicle center of massyThe expression is as follows:
the following risk factor RP is defined as follows:
a) if no moving vehicle exists in front of the same lane of the target vehicle at the current moment, setting a following danger coefficient RP to be zero;
b) if a moving vehicle exists in front of the target vehicle in the same lane at the current moment, defining a following danger coefficient RP as a weighted sum of a headway THW and a collision time TTC of the target vehicle, wherein the expression is as follows:
wherein, the headway THW of the target vehicle is dx/vxTime to collision TTC ═ d of target vehiclex/Δvx；vxIs the longitudinal speed of the target vehicle in the tangential direction of the lane line, dxThe longitudinal distance, Deltav, of the target vehicle relative to the moving vehicle in front of the same lane along the tangential direction of the lane line of the roadxAnd adjusting the values of the weight coefficients a and b of the head time interval and the collision time of the target vehicle at the current moment, testing on a HighD data set, and selecting the optimal weight coefficients a and b to enable the difference between the following danger coefficients RP of the lane-changing vehicle and the lane-not-changing vehicle to be maximum.
And S3.2, counting the predicted lane change time TTLC and the following danger coefficient RP of the lane change vehicles and the lane change-free vehicles in the HighD data set, and determining the threshold values of the predicted lane change time TTLC and the following danger coefficient RP as the lane change threshold values alpha and beta of the target vehicle respectively to ensure that the tested lane change identification accuracy on the HighD data set is the highest as shown in figures 2 and 3.
S3.3, dividing the lane changing state of the target vehicle into a left lane changing state, a right lane changing state and a non-lane changing state according to the lane changing threshold value of the target vehicle obtained in the step S3.2; the judgment conditions of changing the lane to the left and changing the lane to the right are respectively as follows:
LCL＝(0＜TTLC＜α)∪(RP＞β) (3)
LCR＝(-α＜TTLC＜0)∪(RP＜-β) (4)
in the formula, LCL represents that the lane change state of the target vehicle is a left lane change, and LCR represents that the lane change state of the target vehicle is a right lane change; the TTLC and the RP are respectively the predicted lane change time and the following danger coefficient of the target vehicle defined in the step S3.1; alpha and beta are respectively the predicted lane changing time of the target vehicle and the threshold value of the following danger coefficient set in the step S3.2; and if the conditions of changing the tracks to the left and changing the tracks to the right are not met, the tracks are not changed.
S4, predicting the motion trail of the target vehicle by adopting a polynomial fitting and optimization method according to the primary recognition result: firstly, determining the motion trail constraint of a target vehicle, obtaining a motion trail conforming to the motion trail constraint by using polynomial fitting, and then screening by an optimization method to obtain the most possible motion trail as a predicted motion trail of the target vehicle; the method specifically comprises the following steps:
s4.1 fitting the target vehicle motion track conforming to the lane change state by adopting a polynomial
S4.11 determining the starting point constraint of the motion trail of the target vehicle, and setting the state of the starting point of the motion trail of the target vehicle as [ x0,y0,vx0,vy0,ax0,ay0]Wherein [ x ]0,y0]Is the coordinate of the starting point of the motion track, and is equal to the position coordinate of the target vehicle at the current moment, namely x0＝x(t0),y0＝y(t0)；[vx0,vy0]As the starting point speed of the motion trajectory, [ a ]x0,ay0]For the acceleration of the starting point of the motion trail, because the observation of the state of the target vehicle has noise, the speed and the acceleration of the starting point of the motion trail are obtained by mean value filtering, namely the speed and the acceleration of the starting point of the motion trail are mean values of the speed and the acceleration of the target vehicle in the past period of time delta t, and the calculation formulas are shown as (5) to (8):
vx0＝mean(vx(t)) (5)
vy0＝mean(vy(t)) (6)
ax0＝mean(ax(t)) (7)
ay0＝mean(ay(t)) (8)
wherein t ∈ [ t ]0-Δt，t0]Delta t is the time length of the historical track; v. ofx(t)、vy(t) a transverse velocity function perpendicular to the lane line and a longitudinal velocity function tangential to the lane line on the historical track of the target vehicle, respectively; a isx(t)、ay(t) a lateral acceleration function perpendicular to the lane line and a longitudinal acceleration function tangential to the lane line on the target vehicle history track, respectively.
S4.12, determining the motion track end point constraint of the target vehicle, if the lane change state is a left lane change, enabling the track end point to be on the central line of the lane on the left side, if the lane change state is a right lane change, enabling the track end point to be on the central line of the lane on the right side, and if the lane change state is a non-lane change, enabling the track end point to be on the central line of the current lane; and making the motion constraint of the target vehicle motion track terminal point as follows:
y1＝y0+Δy (9)
vy1＝0 (10)
ay1＝0 (11)
vx1＝vx0+ax0tpred (12)
ax1＝ax0 (13)
wherein,
equation (9) represents the transverse coordinate y of the center of mass of the target vehicle perpendicular to the lane line at the end point of the constraint trajectory1Equal to the transverse coordinate y of the center of mass of the target vehicle at the starting point of the track0Sum of the target vehicle lateral motion offset Δ y;
equations (10) and (11) represent the target vehicle lateral velocity a perpendicular to the lane line at the end point of the constraint trajectoryy1And the target vehicle lateral acceleration ay1Are all zero;
equation (12) shows that the longitudinal motion of the constrained target vehicle along the lane line corresponds to uniform acceleration, vx0，vx1Respectively the starting point and the end point of the trackTarget vehicle longitudinal speed tangential to lane line, ax0Is the target vehicle longitudinal acceleration, tpredFor predicting the movement duration from the track starting point to the track end point, the track end point time t1＝t0+tpred；
Equation (13) expresses the longitudinal acceleration ax tangential to the lane line at the end of the constrained trajectory1Longitudinal acceleration a tangential to the lane line at the origin of the trajectoryx0Are equal.
S4.13, fitting the transverse motion of the target vehicle perpendicular to the lane line and the longitudinal motion along the tangential direction of the lane line by using a polynomial of 5 th degree and a polynomial of 4 th degree respectively, obtaining a polynomial coefficient by adopting a least square method, and setting the transverse motion trajectory equation of the target vehicle as follows:
y(t)＝cy5t5+cy4t4+cy3t3+cy2t2+cy1t+cy0 (14)
then the constraints of the starting point and the end point of the transverse motion trail of the target vehicle are written as follows:
similarly, let the longitudinal motion trajectory equation of the target vehicle be:
x(t)＝cx4t4+cx3t3+cx2t2+cx1t+cx0 (16)
then the constraint conditions of the starting point and the end point of the longitudinal motion trail of the target vehicle are written as follows:
s4.2 predicting the time t of movementpredIn contrast, the target vehicle lane change trajectory obtained by fitting is not unique, and as shown in fig. 4, the most likely motion trajectory is selected from the K candidate motion trajectories obtained in S4.13 by an optimization method, preferablyThe cost function of the optimization comprises the time and the lateral acceleration of the movement, and the cost function of the optimization is as follows:
C(Ti)＝ti+γmax(ay(t)) (18)
wherein, tiAn ith candidate motion track T of the target vehicleiCorresponding predicted motion time, i ═ 1,2, …, K, ay(t) is a lateral acceleration function in a direction perpendicular to the lane line during movement of the target vehicle from the starting point to the ending point; gamma is a coefficient, and the coefficient in the optimization cost function is adjusted by selecting any vehicle motion track in the HighD data set, so that the selected predicted track and the real motion track of the target vehicle are determined as a close mode;
the finally selected track enables the optimization cost function to be minimum and serves as the predicted motion track T of the target vehicleoptThe expression is as follows:
Topt＝argmin(C(Ti))i＝1，2，...，K (19)
s5, taking the motion trail conforming to the uniform acceleration motion constraint as a reference motion trail of the target vehicle; calculating the accumulated distance deviation between the predicted motion track of the target vehicle and the reference motion track of the target vehicle, if the deviation value exceeds a set track deviation threshold value, judging that the predicted motion track of the target vehicle does not accord with motion track constraint, namely, the initial lane change recognition result does not accord with the motion constraint, if the initial lane change recognition result is not lane change, correcting the predicted motion track into lane change, if the initial lane change recognition result is lane change, correcting the predicted motion track into lane change, and after feedback correction, obtaining a final lane change recognition result of the target vehicle; and if the deviation value does not exceed the set track deviation threshold value, keeping the initial recognition result unchanged, and taking the initial recognition result as a final target vehicle lane change recognition result. The method specifically comprises the following steps:
s5.1, taking the track conforming to the uniform acceleration motion constraint as a reference motion track of the target vehicle
Because the vehicle motion is close to the uniform acceleration in a short time due to the state inertia of the vehicle motion, the track changing track conforming to the uniform acceleration motion constraint is used as the reference motion track of the target vehicle for checking the predicted motion track of the target vehicle. The motion state of the reference motion trail starting point of the target vehicle is made to be [ x ]0,y0,vx0,vy0,ax0,ay0]The target vehicle is kept in uniform acceleration motion and then a period of time Deltat is obtainedrInner reference trajectory:
wherein, t2＝t0+Δtr. As shown in fig. 5, at a short time Δ tr(Δtr1-5 s, in this embodiment, 3s), the even acceleration motion trajectory of the vehicle is closer to the real motion trajectory of the target vehicle. Selecting Δ trIs such that any vehicle in the HighD data set is atrThe uniform acceleration motion track in time is close to the real motion track. x is the number ofr(t),yrAnd (t) respectively representing a transverse motion reference track of the reference motion track of the target vehicle along the tangential direction of the lane line and a longitudinal motion reference track vertical to the lane line in the prediction time range.
S5.2, calculating the accumulated distance deviation between the predicted motion trail of the target vehicle and the reference motion trail of the target vehicle
Defining the deviation of the predicted motion trail of the target vehicle obtained in the step S4 from the reference motion trail of the target vehicle as the sum of the distances between the discrete track points, wherein the calculation formula is as follows:
wherein N is the total discrete number of the predicted time length, g is each discrete point in the predicted time length,δ t is the discrete time step of the predicted duration. x (t)g),y(tg) Respectively predicting motion tracks, x, of the target vehicle in the transverse direction and the longitudinal direction discretized in the prediction time lengthr(tg),yr(tg) Respectively are the target vehicle transverse and longitudinal reference motion tracks discretized in the prediction time length.
And S5.3, if the track deviation is larger than the set track deviation threshold value, the lane change identification state does not accord with the motion constraint, if the initial lane change identification is the lane change, the state of the target vehicle is corrected from the lane change to the lane change-free state, otherwise, the lane change-free state is corrected to the lane change, and the final lane change identification result is obtained after the feedback correction.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (4)
1. A target vehicle lane change recognition method for driving assistance based on prediction feedback is characterized by comprising the following steps:
s1, acquiring motion state information and road structure information of surrounding target vehicles in real time through a sensor system carried on the intelligent networked automobile, and obtaining the motion state of the target vehicle and a road and lane line equation of the target vehicle at the current moment in a self-vehicle coordinate system after lane line recognition and target detection tracking algorithm processing;
s2, calculating the motion state of the target vehicle at the current moment and a rectangular envelope frame of the target vehicle at the current moment in the ground coordinate system according to the motion state of the target vehicle at the current moment in the own vehicle coordinate system and a road lane line equation, and extracting the feature quantity of the lane change intention of the target vehicle at the current moment in the ground coordinate system; the lane-changing intention characteristic quantity comprises the transverse speed v of the center of mass of the target vehicle along the direction vertical to the lane line of the roadyAnd the transverse distance d from the rectangular envelope frame of the target vehicle to the lane line of the roadyAnd the longitudinal direction of the target vehicle relative to the moving vehicle in front of the same lane in the following scene along the tangential direction of the lane line of the roadTo velocity DeltavxAnd a longitudinal distance dx；
S3, according to the extracted feature quantity of the target vehicle lane change intention at the current moment in the ground coordinate system, defining the predicted lane change time and the following danger coefficient of the target vehicle as lane change criterion; counting according to the HighD data set to obtain a lane change threshold value for predicting lane change time and a car following danger coefficient; if the predicted lane changing time or the following danger coefficient is larger than the corresponding lane changing threshold value of the target vehicle, preliminarily recognizing that the target vehicle is changing lanes or is about to change lanes, judging that the lane is changed leftwards or rightwards according to the sign of the predicted lane changing time, and otherwise, preliminarily recognizing that the target vehicle keeps the current lane;
s4, predicting the motion trail of the target vehicle by adopting a polynomial fitting and optimization method according to the preliminary identification result obtained in the step S3: firstly, determining the motion trail constraint of a target vehicle, obtaining a motion trail conforming to the motion trail constraint by using polynomial fitting, and then screening by an optimization method to obtain the most possible motion trail as a predicted motion trail of the target vehicle;
s5, taking the motion trail conforming to the uniform acceleration motion constraint as a reference motion trail of the target vehicle; calculating the accumulated distance deviation between the predicted motion track of the target vehicle and the reference motion track of the target vehicle, if the deviation value exceeds a set track deviation threshold value, judging that the predicted motion track of the target vehicle does not accord with motion track constraint, namely, the initial recognition result does not accord with motion constraint, if the initial recognition result is not lane change, correcting the predicted motion track to lane change, if the initial recognition result is lane change, correcting the predicted motion track to lane change, and after feedback correction, obtaining a final lane change recognition result of the target vehicle; if the deviation value does not exceed the set track deviation threshold value, maintaining the initial recognition result as a final target vehicle lane change recognition result;
the step S3 specifically includes the following steps:
s3.1, according to the extracted feature quantity of the target vehicle lane change intention at the current time under the ground coordinate system, defining the predicted lane change time TTLC and the following danger coefficient RP which are used as the lane change criterion of the target vehicle:
the predicted lane change time TTLC is defined as the current target vehicle in the ground coordinate systemPerpendicular distance d from rectangular envelope frame to road lane lineyTransverse velocity v along the direction perpendicular to the road lane line with the target vehicle center of massyThe expression is as follows:
the following risk factor RP is defined as follows:
a) if no moving vehicle exists in front of the same lane of the target vehicle at the current moment, setting a following danger coefficient RP to be zero;
b) if a moving vehicle exists in front of the target vehicle in the same lane at the current moment, defining a following danger coefficient RP as a weighted sum of a headway THW and a collision time TTC of the target vehicle, wherein the expression is as follows:
wherein,
headway time distance THW ═ d of target vehiclex/vx，dxThe longitudinal distance v of the target vehicle relative to the moving vehicle in front of the same lane at the current moment along the tangential direction of the lane line of the roadxThe tangential longitudinal speed of the target vehicle along the lane line at the current moment;
time to collision TTC ═ d of target vehiclex/Δvx，ΔvxThe tangential longitudinal speed of the target vehicle relative to the moving vehicle in front of the same lane at the current moment along the lane line of the road is obtained;
a, b are respectively the weight coefficients of the headway and the collision time of the target vehicle at the current moment, the values of the weight coefficients a and b of the headway and the collision time are adjusted, the test is carried out on a HighD data set, and the optimal weight coefficients a and b are selected to enable the difference between the following danger coefficients RP of the lane-changing vehicle and the lane-changing-free vehicle to be maximum;
s3.2, the predicted lane change time TTLC and the following danger coefficient RP of the lane change vehicles and the lane change-free vehicles in the HighD data set are counted, and the threshold values of the predicted lane change time TTLC and the following danger coefficient RP are determined to be respectively used as the lane change threshold values alpha and beta of the target vehicle, so that the accuracy of testing lane change identification on the HighD data set is highest;
s3.3, dividing the lane changing state of the target vehicle into a left lane changing state, a right lane changing state and a non-lane changing state according to the lane changing threshold values alpha and beta of the target vehicle; the judgment conditions of changing the lane to the left and changing the lane to the right are respectively as follows:
LCL＝(0＜TTLC＜α)∪(RP＞β) (3)
LCR＝(-α＜TTLC＜0)∪(RP＜-β) (4)
in the formula, the LCL indicates that the lane change state of the target vehicle is a left lane change, the LCR indicates that the lane change state of the target vehicle is a right lane change, and the lane change is not performed if neither the left nor the right lane change condition is satisfied.
2. The method for identifying a lane change of a target vehicle according to claim 1, wherein the step S4 specifically comprises the steps of:
s4.1 fitting the target vehicle motion track conforming to the lane change state by adopting a polynomial
S4.11 determining the starting point constraint of the motion trail of the target vehicle, and setting the state of the starting point of the motion trail of the target vehicle as [ x0，y0，vx0，vy0，ax0，ay0]Wherein [ x ]0，y0]Is the coordinate of the starting point of the motion track and the current time t under the ground coordinate system0The position coordinates of the centroid of the target vehicle are equal; [ v ] ofx0，vy0]As the starting point speed of the motion trajectory, [ a ]x0，ay0]For the acceleration of the starting point of the motion trail, because the observation of the state of the target vehicle has noise, the speed and the acceleration of the starting point of the motion trail are obtained by mean value filtering, namely the speed and the acceleration of the starting point of the motion trail are mean values of the speed and the acceleration of the target vehicle in the past period of time delta t, and the calculation formulas are shown as (5) to (8):
vx0＝mean(vx(t)) (5)
vy0＝mean(vy(t)) (6)
ax0＝mean(ax(t)) (7)
ay0＝mean(ay(t)) (8)
wherein t ∈ [ t ]0-Δt，t0]Delta t is the time length of the historical track; v. ofx(t)、vy(t) a transverse velocity function perpendicular to the lane line and a longitudinal velocity function tangential to the lane line on the historical track of the target vehicle, respectively; a isx(t)、ay(t) a lateral acceleration function perpendicular to the lane line and a longitudinal acceleration function tangential to the lane line on the historical track of the target vehicle, respectively;
s4.12, determining the motion track end point constraint of the target vehicle, if the lane change state is a left lane change, enabling the track end point to be on the central line of the lane on the left side, if the lane change state is a right lane change, enabling the track end point to be on the central line of the lane on the right side, and if the lane change state is a non-lane change, enabling the track end point to be on the central line of the current lane; and making the motion constraint of the target vehicle motion track terminal point as follows:
y1＝y0+Δy (9)
vy1＝0 (10)
ay1＝0 (11)
vx1＝vx0+ax0tpred (12)
ax1＝ax0 (13)
wherein,
equation (9) represents the transverse coordinate y of the center of mass of the target vehicle perpendicular to the lane line at the end point of the constraint trajectory1Equal to the transverse coordinate y of the center of mass of the target vehicle at the starting point of the track0Sum of the target vehicle lateral motion offset Δ y;
the equations (10) and (11) are expressed as respectively restricting the target vehicle transverse velocity v perpendicular to the lane line at the end point of the trajectoryy1And the target vehicle lateral acceleration ay1Are all zero;
equation (12) shows that the longitudinal motion of the constrained target vehicle along the lane line corresponds to uniform acceleration, vx0，vx1Respectively the beginning and the end of the trackTarget vehicle longitudinal speed at a point tangential to the lane line, ax0Is the target vehicle longitudinal acceleration, tpredFor predicting the movement duration from the track starting point to the track end point, the track end point time t1＝t0+tpred；
Equation (13) expresses the longitudinal acceleration a tangential to the lane line at the end of the constrained trajectoryx1Longitudinal acceleration a tangential to the lane line at the origin of the trajectoryx0Equal;
s4.13, fitting a 5 th-order polynomial and a 4 th-order polynomial on the transverse motion of the target vehicle perpendicular to the lane line and the longitudinal motion of the target vehicle along the tangential direction of the lane line respectively, obtaining a polynomial coefficient by adopting a least square method, and setting the transverse motion trajectory equation of the target vehicle as follows:
y(t)＝cy5t5+cy4t4+cy3t3+cy2t2+cy1t+cy0 (14)
then the constraints of the starting point and the end point of the transverse motion trail of the target vehicle are written as follows:
the longitudinal motion trajectory equation of the target vehicle is set as follows:
x(t)＝cx4t4+cx3t3+cx2t2+cx1t+cx0 (16)
then the constraint conditions of the starting point and the end point of the longitudinal motion trail of the target vehicle are written as follows:
s4.2, selecting the most possible motion track from the K candidate motion tracks obtained in the step S4.13 through an optimization method, wherein the optimized cost function comprises the motion time and the lateral acceleration, and the used optimized cost function is as follows:
C(Ti)＝ti+γmax(ay(t)) (18)
wherein, tiAn ith candidate motion track T of the target vehicleiCorresponding predicted motion time, i ═ 1,2, …, K, ay(t) is a lateral acceleration function of the target vehicle in the direction perpendicular to the lane line in the process from the starting point of the predicted movement track to the end point of the predicted movement track; gamma is a coefficient, and the coefficient in the optimization cost function is adjusted by selecting any vehicle motion track in the HighD data set, so that the selected predicted track and the real motion track of the target vehicle are determined as a close mode;
the finally selected track enables the optimization cost function to be minimum and serves as the predicted motion track T of the target vehicleoptThe expression is as follows:
Topt＝argmin(C(Ti))i＝1，2，...，K (19)。
3. the method for identifying a lane change of a target vehicle according to claim 2, wherein the step S5 specifically comprises the steps of:
s5.1, taking the track conforming to the uniform acceleration motion constraint as a reference motion track of the target vehicle
The motion state of the reference motion trail starting point of the target vehicle is made to be [ x ]0，y0，vx0，vy0，ax0，ay0]The target vehicle is kept in uniform acceleration motion for a period of time DeltatrThe inner reference motion trajectory is:
wherein, t2＝t0+Δtr(ii) a Selecting Δ trIs such that in the HighD datasetAt Δ t for any vehiclerThe uniform acceleration motion track in time is closest to the real motion track;
s5.2, calculating the accumulated distance deviation between the predicted motion trail of the target vehicle and the reference motion trail of the target vehicle
Defining the deviation of the predicted motion trail of the target vehicle obtained in the step S4 from the reference motion trail of the target vehicle as the sum of the distances between the discrete track points, wherein the calculation formula is as follows:
wherein N is the total discrete number of the predicted duration, g is each discrete point in the predicted duration, and δ t is the discrete time step length of the predicted duration;
and S5.3, if the track deviation is larger than the set track deviation threshold value, the lane change identification state does not accord with the motion constraint, if the initial lane change identification is the lane change, the state of the target vehicle is corrected from the lane change to the lane change-free state, otherwise, the lane change-free state is corrected to the lane change, and the final lane change identification result is obtained after the feedback correction.
4. The method of claim 3, wherein Δ t is the distance between the target vehicle and the lane change indicatorrTaking the time to be 1-5 s.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910870854.8A CN110705388B (en) | 2019-09-16 | 2019-09-16 | Target vehicle lane change identification method for auxiliary driving based on prediction feedback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910870854.8A CN110705388B (en) | 2019-09-16 | 2019-09-16 | Target vehicle lane change identification method for auxiliary driving based on prediction feedback |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110705388A CN110705388A (en) | 2020-01-17 |
| CN110705388B true CN110705388B (en) | 2022-04-01 |
Family
ID=69195400
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910870854.8A Active CN110705388B (en) | 2019-09-16 | 2019-09-16 | Target vehicle lane change identification method for auxiliary driving based on prediction feedback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110705388B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111354225B (en) * | 2020-03-03 | 2022-05-03 | 中交第一公路勘察设计研究院有限公司 | Expressway vehicle accident risk assessment and early warning intervention method |
| CN111368260B (en) * | 2020-03-18 | 2023-09-05 | 东软睿驰汽车技术（上海）有限公司 | Vehicle following method and device |
| CN113511204B (en) * | 2020-03-27 | 2022-11-11 | 华为技术有限公司 | Vehicle lane changing behavior identification method and related equipment |
| CN111703424B (en) * | 2020-05-26 | 2021-12-24 | 武汉理工大学 | Intelligent networking automobile multi-sensor fusion auxiliary lane changing method and system |
| US12030507B2 (en) | 2020-06-29 | 2024-07-09 | Dr. Ing. H.C. F. Porsche Aktiengesellschaft | Method and system for predicting a trajectory of a target vehicle in an environment of a vehicle |
| CN112133089B (en) * | 2020-07-21 | 2021-11-19 | 西安交通大学 | Vehicle track prediction method, system and device based on surrounding environment and behavior intention |
| CN111967094B (en) * | 2020-09-01 | 2022-08-16 | 吉林大学 | Backward lane line calculating method based on Mobileye lane line equation |
| WO2022099526A1 (en) * | 2020-11-12 | 2022-05-19 | 深圳元戎启行科技有限公司 | Method for training lane change prediction regression model, and lane change predicton method and apparatus |
| CN112347567B (en) * | 2020-11-27 | 2022-04-01 | 青岛莱吉传动***科技有限公司 | Vehicle intention and track prediction method |
| CN112560782A (en) * | 2020-12-26 | 2021-03-26 | 浙江天行健智能科技有限公司 | Vehicle lane changing behavior identification method based on random forest algorithm |
| CN112965489B (en) * | 2021-02-05 | 2022-04-08 | 北京理工大学 | Intelligent vehicle high-speed lane change planning method based on collision detection |
| CN113022570B (en) * | 2021-03-19 | 2022-06-10 | 武汉理工大学 | Method and device for identifying lane changing behavior of vehicle |
| CN113139331A (en) * | 2021-03-29 | 2021-07-20 | 西北工业大学 | Air-to-air missile situation perception and decision method based on Bayesian network |
| CN113353083B (en) * | 2021-08-10 | 2021-10-29 | 所托(杭州)汽车智能设备有限公司 | Vehicle behavior recognition method |
| CN114220262B (en) * | 2021-11-19 | 2023-06-23 | 东南大学 | Vehicle lane change track deviation calculation method based on traffic simulation |
| CN116110216B (en) * | 2022-10-21 | 2024-04-12 | 中国第一汽车股份有限公司 | Vehicle line crossing time determining method and device, storage medium and electronic device |
| CN115731708B (en) * | 2022-11-15 | 2023-10-17 | 东南大学 | Real-time vehicle track lane change point monitoring method based on Bayesian theory |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3922194B2 (en) * | 2003-03-11 | 2007-05-30 | 日産自動車株式会社 | Lane departure warning device |
| US7561032B2 (en) * | 2005-09-26 | 2009-07-14 | Gm Global Technology Operations, Inc. | Selectable lane-departure warning system and method |
| CN103395391B (en) * | 2013-07-03 | 2015-08-05 | 北京航空航天大学 | A kind of vehicle lane-changing alarming device and change state identification method |
| KR101480652B1 (en) * | 2013-12-11 | 2015-01-09 | 현대자동차주식회사 | Lane change control apparatus and control method of the same |
| CN103754221B (en) * | 2014-01-24 | 2017-05-24 | 清华大学 | Vehicle adaptive cruise control system |
| CN104732785A (en) * | 2015-01-09 | 2015-06-24 | 杭州好好开车科技有限公司 | Driving behavior analyzing and reminding method and system |
| CN104960524B (en) * | 2015-07-16 | 2017-06-06 | 北京航空航天大学 | Many cars collaboration lane change control system and its method based on truck traffic |
| KR102362686B1 (en) * | 2017-11-06 | 2022-02-15 | 현대자동차주식회사 | Vehicle and method for controlling the same |
| CN109131326B (en) * | 2018-08-08 | 2020-04-14 | 大连理工大学 | Self-adaptive cruise controller with lane change auxiliary function and working method thereof |
| CN109885066B (en) * | 2019-03-26 | 2021-08-24 | 北京经纬恒润科技股份有限公司 | Motion trail prediction method and device |
| CN110097785B (en) * | 2019-05-30 | 2022-06-07 | 长安大学 | Recognition early warning device and early warning method for front vehicle cut-in or emergency lane change |
-
2019
- 2019-09-16 CN CN201910870854.8A patent/CN110705388B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110705388A (en) | 2020-01-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110705388B (en) | Target vehicle lane change identification method for auxiliary driving based on prediction feedback | |
| CN109885066B (en) | Motion trail prediction method and device | |
| CN110155046B (en) | Automatic emergency braking hierarchical control method and system | |
| CN111104969B (en) | Collision possibility pre-judging method for unmanned vehicle and surrounding vehicles | |
| CN110843789B (en) | Vehicle lane change intention prediction method based on time sequence convolution network | |
| US8947218B2 (en) | Driving support device | |
| US8112225B2 (en) | Method and system for collision avoidance | |
| CN110077398B (en) | Risk handling method for intelligent driving | |
| JP5385056B2 (en) | Driving status estimation device, driving support device | |
| US8233663B2 (en) | Method for object formation | |
| Kumagai et al. | Prediction of driving behavior through probabilistic inference | |
| CN114371707B (en) | Pedestrian track prediction and active collision avoidance method and system considering human-vehicle interaction | |
| JP2007534041A (en) | Lane change driving recognition method and apparatus for vehicles | |
| CN111354225A (en) | Expressway vehicle accident risk assessment and early warning intervention method | |
| CN111038502A (en) | Safe vehicle distance pre-estimation, correction, early warning and driving qualification evaluation method and system | |
| CN109878530B (en) | Method and system for identifying lateral driving condition of vehicle | |
| CN112249008A (en) | Unmanned automobile early warning method aiming at complex dynamic environment | |
| KR20220002789A (en) | Lane change assistance system and lane chagne method using the same | |
| Oh et al. | In-depth understanding of lane changing interactions for in-vehicle driving assistance systems | |
| CN114987539A (en) | Individual collision grading early warning method and system for automatic driving automobile based on risk field model | |
| Yao et al. | Target vehicle selection algorithm for adaptive cruise control based on lane-changing intention of preceding vehicle | |
| CN112026767B (en) | Method and system for processing error identification of guardrail by self-adaptive cruise and vehicle | |
| CN114394091B (en) | Vehicle speed control method for traffic vehicles in adaptive cruise system under parallel road scene | |
| CN114475587B (en) | Risk assessment algorithm for introducing target behaviors and collision probability | |
| CN114464012B (en) | Vehicle and pedestrian collision early warning method and system based on Internet of vehicles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |