EP2293223B1 - Vision system and method for a motor vehicle - Google Patents
Vision system and method for a motor vehicle Download PDFInfo
- Publication number
- EP2293223B1 EP2293223B1 EP09010814.3A EP09010814A EP2293223B1 EP 2293223 B1 EP2293223 B1 EP 2293223B1 EP 09010814 A EP09010814 A EP 09010814A EP 2293223 B1 EP2293223 B1 EP 2293223B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- vision system
- determination means
- image
- convergence point
- sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 6
- 238000003384 imaging method Methods 0.000 claims description 42
- 238000012545 processing Methods 0.000 claims description 27
- 238000012790 confirmation Methods 0.000 claims description 22
- 238000001454 recorded image Methods 0.000 claims description 15
- 230000003139 buffering effect Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000005259 measurement Methods 0.000 description 5
- 238000003708 edge detection Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
Definitions
- the invention relates to a vision system for a motor vehicle, comprising at least one imaging device adapted to record images from a region surrounding the motor vehicle, and an electronic processing means for processing images recorded by said imaging device.
- the invention furthermore relates to a corresponding vision method.
- the orientation of vehicle-mounted cameras is calibrated at the manufacturer which is time consuming, difficult and costly. Furthermore, the orientation of a vehicle-mounted camera may change due to external factors like car load, external forces, thermal effects etc., which may affect the functioning of the vision system.
- EP 1 681 577 A2 discloses a vision system where the optical flow in images recorded by an imaging means is determined and a vanishing point is calculated from the determined optical flow. If the position of the calculated vanishing point deviates from the position of a set point, the image position is adjusted accordingly.
- the determination of optical flow requires a sequence of images and is based on very complex mathematical approximation methods which are impractical to implement in motor vehicles where the processing capabilities are generally limited.
- the invention solves this object with the features of the independent claims.
- the invention is based on the fact that in images taken from a moving motor vehicle usually essentially straight lines are present which relate to essentially parallel lines in the imaged scene. These are in particular lines relating to the course of the road, especially ground lines like road edges or lane markings.
- the lines can be determined using known image processing tools like an edge detection algorithm. This requires much less processing resources than the known determination of the optical flow.
- the convergence point of these lines denotes a central vanishing point which can provide a unambiguous reference point.
- position information of the convergence point provides a measure for the current orientation of the imaging device. Suitable position information of the convergence point may for example be the absolute position, a position relative to a set point, for example a central point or a corner of the detection area or the image, or an offset between the convergence point and a set point.
- the electronic processing means is preferably adapted to account for, in said image processing of said recorded images, said current orientation of said imaging device provided by said determination means.
- the vision system comprises a confirmation means adapted to evaluate, on the basis of an independent source, a convergence point determination reliability.
- the confirmation means takes account of the fact that the convergence point determination may not be reliable under all conditions. For example the convergence point determination may not be reliable during turning left or right, driving along a curved road, driving along a road with steep slopes, etc.
- the confirmation means uses information relating to the vehicle kinematics which preferably is evaluated over a certain period after recording of an image under consideration. Therefore, the independent source used by the confirmation means comprises a sensing arrangement measuring vehicle kinematics, in particular a yaw sensor, a pitch sensor and/or a vehicle speed sensor.
- the vision system preferably comprises a buffer for buffering information relating to each recorded image until the reliability of the corresponding offset determination has been evaluated, because at the time of recording an image only the future vehicle kinematics provide information whether the road ahead of the imaging device is essentially straight or not.
- the confirmation means is preferably adapted to determine whether the road lying ahead of the imaging device was straight at the time of recording the image under consideration. This may preferably be done using signals from a yaw sensor and/or a sensor measuring a steering wheel angle in a certain period after recording of the image under consideration.
- images taken at low or zero vehicle speed may be not reliable since the orientation of the vehicle relative to the road may deviate from straight orientation, for example in case of manoeuvring into a parking space etc.
- the confirmation means may use information about the type of road the vehicle is moving on. This may be determined preferably using signals from a properly equipped satellite navigation receiver which may be combined with information from a vehicle speed sensor. For example the confirmation means may determine the offset determination to be reliable only on certain types of roads like highways or main roads.
- the confirmation means may additionally use information about the quality of the recorded image which may be extracted from the recorded images by image processing. For example the confirmation means may determine the offset determination to be reliable only in case the image quality exceeds a specified image quality threshold.
- the confirmation means may act in different ways.
- the confirmation means comprises a filter adapted to filter out convergence points with insufficient reliability.
- the confirmation means may for example set the straight line determination and/or convergence point determination inoperative in case unfavourable conditions are determined.
- the position of a convergence point may be averaged over time before using it in determining the current orientation of the imaging device.
- a confirmation means is not strictly necessary for the invention because variations of a convergence point around a central convergence point tend to cancel out in the long run.
- the confirmation means can lead to a much faster and more robust convergence point determination.
- a convergence point determined by the determination means may be used in different ways.
- the current orientation of said imaging device is accounted for in the image processing of the recorded images, for example by compensating for an offset between said convergence point and a set point, or providing a software calibration of the imaging device.
- a hardware re-adjustment of the orientation of the imaging means automatically may be performed in order to re-set an offset between said convergence point and a set point essentially to zero.
- the determination means preferably is adapted to disregard image lines essentially lying in an upper part of the image.
- the determination means may use additional information from a vehicle load sensor which may lead to better results for different vehicle load cases.
- the vision system 10 is mounted in a motor vehicle and comprises an imaging means 11 for recording images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle.
- the imaging means 11 comprises one or more optical and/or infrared imaging devices 12a, 12b, in particular cameras, where infrared covers near IR with wavelengths below 5 microns and/or far IR with wavelengths beyond 5 microns.
- the imaging means 11 comprises a plurality imaging devices 12a, 12b in particular forming a stereo imaging means 11; alternatively only one imaging device forming a mono imaging means could be used.
- the imaging devices 12a, 12b are coupled to an electronic processing means 14 where processing of the images taken by the imaging devices 12a, 12b is carried out by means of a corresponding software.
- the processing means 14 comprises an object identifying means 15 adapted to identify and classify possible objects in front of the motor vehicle, such as pedestrians, bicyclists or large animals, a tracking means 16 adapted to track the position of object candidates in the recorded images identified by the object detection means 15 over time, and a decision means 17 adapted to activate or control vehicle safety means 18, 19, 20, ... depending on the result of the processing in the object detection and tracking means 15, 16.
- the vehicle safety means 18, 19, 20, --- may comprise a warning means 18 adapted to provide a collision warning to the driver by suitable optical, acoustical and/or haptical warning signals; display means 19 for displaying information relating to an identified object; one or more restraint systems 20 such as occupant airbags or safety belt tensioners, pedestrian airbags, hood lifters and the like, and/or dynamic vehicle control systems such as brakes.
- a warning means 18 adapted to provide a collision warning to the driver by suitable optical, acoustical and/or haptical warning signals
- display means 19 for displaying information relating to an identified object
- one or more restraint systems 20 such as occupant airbags or safety belt tensioners, pedestrian airbags, hood lifters and the like, and/or dynamic vehicle control systems such as brakes.
- the electronic processing means 14 is preferably programmed or programmable and may comprise a microprocessor or micro-controller.
- the electronic processing means 14 is preferably realised in an on-board electronic control unit (ECU) and may be connected to the imaging devices 12a, 12b via a separate cable or alternatively via a vehicle data bus.
- ECU electronice control unit
- the ECU and one of the imaging devices 12a, 12b can be integrated into a single unit. All steps from imaging over image processing to activation or control of safety means 18, 19, 20, ... are performed automatically and continuously during driving in real time.
- the processing means 14 comprises a determination means 30 which has access to the image data input to the processing means 14.
- the determination means 30 is programmed to determine, using image processing techniques, essentially straight lines in a recorded image 40. Alternatively, the straight line determination may also be performed together with the object detection in the object detection means 15.
- Fig. 2 schematically shows an example of an image taken from a vehicle driving on a road 41.
- the determination means 30 determines, for example using an edge detection algorithm like a Sobel operator algorithm, essentially straight lines 42a, 42b; 43a to 43c; 44a to 44c; ... in the image 40 corresponding the road edges 42a, 42b and objects 43, 44 in the scene such as buildings, bridges or the like.
- the determination means 30 applies a filter to the determined straight lines 42a, 42b; 43a to 43c; 44a to 44c; ... in order to rule out the lines which are not suited for the offset determination.
- This line filter may apply different criteria to the detected lines.
- the lines 43a, 43b, 43c, ... of the building 43 may be disregarded since they are essentially lying in an upper part of the image 40 and are therefore not to be regarded as lines on the ground.
- the essentially horizontal lines 44a, 44b, 44c of the building 44 may be disregarded since essentially horizontal lines are generally not suited for the determination of a vanishing point.
- the short vertical lines of the building 44 may be disregarded because only lines with a length exceeding a certain minimum length are suited for offset determination.
- the remaining straight lines passing the above described line filter here the lines 42a, 42b corresponding to the road edges, can be considered as lines on the ground 45 which are parallel in reality and therefore converge in a convergence point 46 in the projection of the 3-dimensional scene in front of the vehicle onto the 2-dimensional image 40.
- the convergence point 46 is determined as the section point of the lines 42a, 42b.
- the lines 42a, 42b used for determination of the convergence point 46 do not have to be essentially straight over their whole length; only a straight portion 47a, 47b of sufficient length, expediently in the lower portion of the image, is required.
- the upper curved section 48a, 48b of the lines 42a, 42b corresponding to a curved road section lie beyond a certain threshold distance s and may therefore be disregarded.
- the determination means 30 may then apply a filter to the determined convergence point 46 in order to rule out unsuited convergence points. For example a convergence point may be dismissed if it lies outside the image 40, or if the distance to the previously determined convergence point exceeds a certain threshold, taking into account that sudden large changes in the orientation of the imaging device 12a, 12b are not likely. If a horizon 61 is determined in the image 40, the converging point 46 may be required to lie on or close to the horizon 61.
- the determination means 30 preferably confirms whether certain conditions necessary for an accurate convergence point determination are fulfilled.
- the determination means 30 preferably confirms that the road ahead of the vehicle is straight.
- the determination means 30 may evaluate whether an offset 49 of the convergence point 46 from a pre-determined set-point 50 is significant or not. For example the x-/y-distances between the convergence point 46 and the set point 50 in horizontal (x) and/or vertical (y) direction may be calculated and the absolute values of these differences may independently be compared to certain x- and y-thresholds. Alternatively the two-dimensional distance between the convergence point 46 and the set point 50 may be calculated and compared to a pre-determined threshold. The set point 50 may for example be the central point of the recorded image 40. The following steps may then only be carried out if the offset 49 is found to be significant; otherwise the offset 49 may be ignored.
- the confirmation means 30 uses vehicle kinematics information from one or more sensors 52 to 55 independent from the imaging means 11, in particular a vehicle yaw sensor 52, a vehicle pitch sensor 53 and/or a vehicle speed sensor 54 and/or a sensor 55 measuring the steering wheel angle.
- This information is measured over a certain period starting from the point where a particular image 40 under consideration is recorded until a point where the vehicle has reached the threshold distance s, see Fig. 2 .
- the measuring period for each image may be specified in seconds or meters and may for example be at least 50 m, more preferably at least 100 m.
- the reliability of the convergence point determination is confirmed if the signal from the yaw sensor 52 does not exceed a certain value over the whole measurement period, indicating that the road ahead of the vehicle was straight at the time a particular image 40 was taken.
- the signal from the sensor 55 measuring the steering wheel angle does not exceed a certain value over the whole measurement period.
- the reliability of the convergence point determination preferably is only confirmed if the signal from the pitch sensor 53, alternatively to or in combination with the signal from the yaw sensor 52, does not exceed a certain value over the whole measurement period.
- the vehicle since form example the vehicle may not be aligned parallel to the road at low speeds, the reliability of the convergence point determination may only be confirmed if the signal from the speed sensor 54, alternatively to or in combination with the signal from the yaw sensor 52, does exceed a certain value over the whole measurement period.
- the information on the position of the converging point 46 in that image 40 is expediently buffered in a buffer 51 connected to the determination means 30. If the reliability of the convergence point determination is finally confirmed, the determination means can fetch the information on the position of the converging point 46 from the buffer 51 and use it in a suitable manner, for example in order to compensate for the offset 49 of the convergence point 46 from the set point 50.
- the confirmation means 30 may use information indicating on which type of road the vehicle moves in particular from a suitably equipped satellite navigation receiver 60.
- the reliability of the convergence point determination may for example only be confirmed if the vehicle is found to move on high-speed roads like highways.
- the type of road the vehicle is moving on may also be determinable by information from the speed sensor 55.
- the buffer 51 may be dispensed with.
- the determination means 30, possibly after averaging a plurality of convergence point positions over time, preferably provides information relating to the position of the convergence point 46, in particular its absolute or relative x- and y-values or information derived from these, for example the offset 49, to other means enabling them to use it in one way or the other, as shall be explained below.
- a hardware correction in order to compensate for the offset 49 is performed by re-adjusting the imaging device 12a, 12b for which the offset has been determined, i.e. adjusting the orientation of the imaging device 12a, 12b such that the converging point 46 falls together with the set point 50.
- the imaging device 12a, 12b which has recorded the image 40 should be moved to the right and to the top.
- information relating to the position of the convergence point 46 is provided to the object identifying means 15 and/or the object tracking means 16 enabling them to account for the actual orientation of the imaging device 12a, 12b relating to the position of the convergence point 46.
- the object identifying means 15 and/or the object tracking means 16 may perform a software compensation for the offset 49, or provide a software calibration of the imaging device 12a, 12b.
- a combination of the embodiments according to Figures 1 and 3 is also possible.
- information relating to the position of the convergence point 46 may be provided to the object identifying means 15 and/or the object tracking means 16 allowing for software calibration, whereas if the offset 49 becomes too large a hardware re-adjustment of the orientation of the imaging device 12a, 12b may be carried out.
- the determination means 30 may use additional information from a vehicle load sensor 70 which may lead to better results for different vehicle load cases.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Description
- The invention relates to a vision system for a motor vehicle, comprising at least one imaging device adapted to record images from a region surrounding the motor vehicle, and an electronic processing means for processing images recorded by said imaging device. The invention furthermore relates to a corresponding vision method.
- Conventionally the orientation of vehicle-mounted cameras is calibrated at the manufacturer which is time consuming, difficult and costly. Furthermore, the orientation of a vehicle-mounted camera may change due to external factors like car load, external forces, thermal effects etc., which may affect the functioning of the vision system.
-
EP 1 681 577 A2 discloses a vision system where the optical flow in images recorded by an imaging means is determined and a vanishing point is calculated from the determined optical flow. If the position of the calculated vanishing point deviates from the position of a set point, the image position is adjusted accordingly. However, the determination of optical flow requires a sequence of images and is based on very complex mathematical approximation methods which are impractical to implement in motor vehicles where the processing capabilities are generally limited. - Vision systems wherein a vanishing point is calculated from line detection are disclosed for example in
US 5 638 116 , Yu-Ming Liang et al: "Video Stabilization for a Camcorder Mounted on a Moving Vehicle" IEEE Transactions on vehicular technology, vol. 53, no. 6, 1 November 2004, pages 1636-1648,WO 2009/027090 andEP 1 873 716 A1 . - It is the object of the present invention to provide a computational effective vision system and method contributing to an overall cost reduction.
- The invention solves this object with the features of the independent claims. The invention is based on the fact that in images taken from a moving motor vehicle usually essentially straight lines are present which relate to essentially parallel lines in the imaged scene. These are in particular lines relating to the course of the road, especially ground lines like road edges or lane markings. The lines can be determined using known image processing tools like an edge detection algorithm. This requires much less processing resources than the known determination of the optical flow. The convergence point of these lines denotes a central vanishing point which can provide a unambiguous reference point. In particular, position information of the convergence point provides a measure for the current orientation of the imaging device. Suitable position information of the convergence point may for example be the absolute position, a position relative to a set point, for example a central point or a corner of the detection area or the image, or an offset between the convergence point and a set point.
- Knowing the current orientation of the imaging device allows for a calibration during movement of the vehicle in real time, which reduces, or even removes, the need for a calibration of the imaging means at the manufacturer. In this respect, the electronic processing means is preferably adapted to account for, in said image processing of said recorded images, said current orientation of said imaging device provided by said determination means.
- The vision system comprises a confirmation means adapted to evaluate, on the basis of an independent source, a convergence point determination reliability. The confirmation means takes account of the fact that the convergence point determination may not be reliable under all conditions. For example the convergence point determination may not be reliable during turning left or right, driving along a curved road, driving along a road with steep slopes, etc.
- The confirmation means uses information relating to the vehicle kinematics which preferably is evaluated over a certain period after recording of an image under consideration. Therefore, the independent source used by the confirmation means comprises a sensing arrangement measuring vehicle kinematics, in particular a yaw sensor, a pitch sensor and/or a vehicle speed sensor. In this case, the vision system preferably comprises a buffer for buffering information relating to each recorded image until the reliability of the corresponding offset determination has been evaluated, because at the time of recording an image only the future vehicle kinematics provide information whether the road ahead of the imaging device is essentially straight or not.
- The confirmation means is preferably adapted to determine whether the road lying ahead of the imaging device was straight at the time of recording the image under consideration. This may preferably be done using signals from a yaw sensor and/or a sensor measuring a steering wheel angle in a certain period after recording of the image under consideration.
- Other factors may be considered in the confirmation means. For example, images taken at low or zero vehicle speed may be not reliable since the orientation of the vehicle relative to the road may deviate from straight orientation, for example in case of manoeuvring into a parking space etc.
- Alternatively or in addition to evaluating the vehicle kinematics, the confirmation means may use information about the type of road the vehicle is moving on. This may be determined preferably using signals from a properly equipped satellite navigation receiver which may be combined with information from a vehicle speed sensor. For example the confirmation means may determine the offset determination to be reliable only on certain types of roads like highways or main roads.
- The confirmation means may additionally use information about the quality of the recorded image which may be extracted from the recorded images by image processing. For example the confirmation means may determine the offset determination to be reliable only in case the image quality exceeds a specified image quality threshold.
- The confirmation means may act in different ways. In a preferred embodiment the confirmation means comprises a filter adapted to filter out convergence points with insufficient reliability. Alternatively, the confirmation means may for example set the straight line determination and/or convergence point determination inoperative in case unfavourable conditions are determined.
- The position of a convergence point may be averaged over time before using it in determining the current orientation of the imaging device. In this respect it should be noted that a confirmation means is not strictly necessary for the invention because variations of a convergence point around a central convergence point tend to cancel out in the long run. However, the confirmation means can lead to a much faster and more robust convergence point determination.
- A convergence point determined by the determination means may be used in different ways. Preferably the current orientation of said imaging device is accounted for in the image processing of the recorded images, for example by compensating for an offset between said convergence point and a set point, or providing a software calibration of the imaging device. In another embodiment a hardware re-adjustment of the orientation of the imaging means automatically may be performed in order to re-set an offset between said convergence point and a set point essentially to zero.
- Since the straight lines to be detected are preferably lines on the ground like road edges or lane markings, the determination means preferably is adapted to disregard image lines essentially lying in an upper part of the image.
- The determination means may use additional information from a vehicle load sensor which may lead to better results for different vehicle load cases.
- In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, wherein:
- Fig. 1
- shows a schematic drawing of a vision system for a motor vehicle;
- Fig. 2
- shows an exemplary image of a region in front of the motor vehicle; and
- Fig. 3
- shows a schematic drawing of a vision system for a motor vehicle in another embodiment.
- The
vision system 10 is mounted in a motor vehicle and comprises an imaging means 11 for recording images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. Preferably the imaging means 11 comprises one or more optical and/orinfrared imaging devices 12a, 12b, in particular cameras, where infrared covers near IR with wavelengths below 5 microns and/or far IR with wavelengths beyond 5 microns. Preferably the imaging means 11 comprises aplurality imaging devices 12a, 12b in particular forming a stereo imaging means 11; alternatively only one imaging device forming a mono imaging means could be used. - The
imaging devices 12a, 12b are coupled to an electronic processing means 14 where processing of the images taken by theimaging devices 12a, 12b is carried out by means of a corresponding software. In particular, the processing means 14 comprises anobject identifying means 15 adapted to identify and classify possible objects in front of the motor vehicle, such as pedestrians, bicyclists or large animals, a tracking means 16 adapted to track the position of object candidates in the recorded images identified by the object detection means 15 over time, and a decision means 17 adapted to activate or control vehicle safety means 18, 19, 20, ... depending on the result of the processing in the object detection and tracking means 15, 16. The vehicle safety means 18, 19, 20, --- may comprise a warning means 18 adapted to provide a collision warning to the driver by suitable optical, acoustical and/or haptical warning signals; display means 19 for displaying information relating to an identified object; one or morerestraint systems 20 such as occupant airbags or safety belt tensioners, pedestrian airbags, hood lifters and the like, and/or dynamic vehicle control systems such as brakes. - The electronic processing means 14 is preferably programmed or programmable and may comprise a microprocessor or micro-controller. The electronic processing means 14 is preferably realised in an on-board electronic control unit (ECU) and may be connected to the
imaging devices 12a, 12b via a separate cable or alternatively via a vehicle data bus. In another embodiment the ECU and one of theimaging devices 12a, 12b can be integrated into a single unit. All steps from imaging over image processing to activation or control of safety means 18, 19, 20, ... are performed automatically and continuously during driving in real time. - The processing means 14 comprises a determination means 30 which has access to the image data input to the processing means 14. The determination means 30 is programmed to determine, using image processing techniques, essentially straight lines in a recorded
image 40. Alternatively, the straight line determination may also be performed together with the object detection in the object detection means 15. -
Fig. 2 schematically shows an example of an image taken from a vehicle driving on aroad 41. The determination means 30 determines, for example using an edge detection algorithm like a Sobel operator algorithm, essentiallystraight lines image 40 corresponding theroad edges - In the next step the determination means 30 applies a filter to the determined
straight lines building 43 may be disregarded since they are essentially lying in an upper part of theimage 40 and are therefore not to be regarded as lines on the ground. The essentiallyhorizontal lines 44a, 44b, 44c of the building 44 may be disregarded since essentially horizontal lines are generally not suited for the determination of a vanishing point. The short vertical lines of the building 44 may be disregarded because only lines with a length exceeding a certain minimum length are suited for offset determination. - The remaining straight lines passing the above described line filter, here the
lines ground 45 which are parallel in reality and therefore converge in aconvergence point 46 in the projection of the 3-dimensional scene in front of the vehicle onto the 2-dimensional image 40. Theconvergence point 46 is determined as the section point of thelines Fig. 2 , thelines convergence point 46 do not have to be essentially straight over their whole length; only a straight portion 47a, 47b of sufficient length, expediently in the lower portion of the image, is required. The uppercurved section 48a, 48b of thelines - The determination means 30 may then apply a filter to the determined
convergence point 46 in order to rule out unsuited convergence points. For example a convergence point may be dismissed if it lies outside theimage 40, or if the distance to the previously determined convergence point exceeds a certain threshold, taking into account that sudden large changes in the orientation of theimaging device 12a, 12b are not likely. If a horizon 61 is determined in theimage 40, the convergingpoint 46 may be required to lie on or close to the horizon 61. - Furthermore, the determination means 30 preferably confirms whether certain conditions necessary for an accurate convergence point determination are fulfilled. In particular, the determination means 30 preferably confirms that the road ahead of the vehicle is straight.
- In one embodiment, the determination means 30 may evaluate whether an offset 49 of the
convergence point 46 from a pre-determined set-point 50 is significant or not. For example the x-/y-distances between theconvergence point 46 and theset point 50 in horizontal (x) and/or vertical (y) direction may be calculated and the absolute values of these differences may independently be compared to certain x- and y-thresholds. Alternatively the two-dimensional distance between theconvergence point 46 and theset point 50 may be calculated and compared to a pre-determined threshold. Theset point 50 may for example be the central point of the recordedimage 40. The following steps may then only be carried out if the offset 49 is found to be significant; otherwise the offset 49 may be ignored. - In the embodiment shown in
Figs. 1 and3 the confirmation means 30 uses vehicle kinematics information from one ormore sensors 52 to 55 independent from the imaging means 11, in particular avehicle yaw sensor 52, avehicle pitch sensor 53 and/or avehicle speed sensor 54 and/or asensor 55 measuring the steering wheel angle. This information is measured over a certain period starting from the point where aparticular image 40 under consideration is recorded until a point where the vehicle has reached the threshold distance s, seeFig. 2 . The measuring period for each image may be specified in seconds or meters and may for example be at least 50 m, more preferably at least 100 m. - The reliability of the convergence point determination is confirmed if the signal from the
yaw sensor 52 does not exceed a certain value over the whole measurement period, indicating that the road ahead of the vehicle was straight at the time aparticular image 40 was taken. For the same purpose it may be proved, alternatively to or in combination with the signal from theyaw sensor 52, that the signal from thesensor 55 measuring the steering wheel angle does not exceed a certain value over the whole measurement period. - Since steep slopes may negatively affect a reliable offset determination as well, the reliability of the convergence point determination preferably is only confirmed if the signal from the
pitch sensor 53, alternatively to or in combination with the signal from theyaw sensor 52, does not exceed a certain value over the whole measurement period. - Furthermore, since form example the vehicle may not be aligned parallel to the road at low speeds, the reliability of the convergence point determination may only be confirmed if the signal from the
speed sensor 54, alternatively to or in combination with the signal from theyaw sensor 52, does exceed a certain value over the whole measurement period. - In the measurement period for a
particular image 40 under consideration, the information on the position of the convergingpoint 46 in thatimage 40 is expediently buffered in abuffer 51 connected to the determination means 30. If the reliability of the convergence point determination is finally confirmed, the determination means can fetch the information on the position of the convergingpoint 46 from thebuffer 51 and use it in a suitable manner, for example in order to compensate for the offset 49 of theconvergence point 46 from theset point 50. - Alternatively to or in combination with vehicle kinematics information, the confirmation means 30 may use information indicating on which type of road the vehicle moves in particular from a suitably equipped
satellite navigation receiver 60. The reliability of the convergence point determination may for example only be confirmed if the vehicle is found to move on high-speed roads like highways. The type of road the vehicle is moving on may also be determinable by information from thespeed sensor 55. - In an embodiment where the confirmation means 30 does not use vehicle kinematics information, the
buffer 51 may be dispensed with. - If the reliability of the convergence point determination is finally confirmed, the determination means 30, possibly after averaging a plurality of convergence point positions over time, preferably provides information relating to the position of the
convergence point 46, in particular its absolute or relative x- and y-values or information derived from these, for example the offset 49, to other means enabling them to use it in one way or the other, as shall be explained below. - In the embodiment shown in
Fig. 1 a hardware correction in order to compensate for the offset 49 is performed by re-adjusting theimaging device 12a, 12b for which the offset has been determined, i.e. adjusting the orientation of theimaging device 12a, 12b such that the convergingpoint 46 falls together with theset point 50. In the example shown inFig. 2 theimaging device 12a, 12b which has recorded theimage 40 should be moved to the right and to the top. - In the preferred embodiment shown in
Fig. 3 information relating to the position of theconvergence point 46 is provided to theobject identifying means 15 and/or the object tracking means 16 enabling them to account for the actual orientation of theimaging device 12a, 12b relating to the position of theconvergence point 46. For example, theobject identifying means 15 and/or the object tracking means 16 may perform a software compensation for the offset 49, or provide a software calibration of theimaging device 12a, 12b. - A combination of the embodiments according to
Figures 1 and3 is also possible. For example, in case ofsmall offsets 49, information relating to the position of theconvergence point 46 may be provided to theobject identifying means 15 and/or the object tracking means 16 allowing for software calibration, whereas if the offset 49 becomes too large a hardware re-adjustment of the orientation of theimaging device 12a, 12b may be carried out. - The determination means 30 may use additional information from a
vehicle load sensor 70 which may lead to better results for different vehicle load cases.
Claims (15)
- A vision system (10) for a motor vehicle, comprising an imaging means (11) with at least one imaging device (12a, 12b) adapted to record images from a region surrounding the motor vehicle, and an electronic processing means (14) for processing said recorded images, wherein said vision system comprises a determination means (30) adapted to determine during driving, by using image processing performed in said electronic processing means (14), at least two essentially straight lines (42a, 42b) in the recorded image (40), to determine a convergence point (46) from said lines, and to determine a current orientation of said imaging device (12a, 12b) based on position information of said convergence point (46), wherein said vision system (10) comprises a confirmation means adapted to evaluate, on the basis of an independent source (52, 53, 54, 55, 60), a convergence point determination reliability, characterized in that said independent source used by said confirmation means comprises a yaw sensor (52), a pitch sensor (53) and/or a sensor (55) measuring the steering wheel angle.
- The vision system as claimed in claim 1, wherein said electronic processing means (14) is adapted to account for, in said image processing of said recorded images, said current orientation of said imaging device (12a, 12b) provided by said determination means (30).
- The vision system as claimed in claim 1 or 2, wherein said determination means (30) comprises said confirmation means.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) is adapted to determine whether the road lying ahead of the imaging device (12a, 12b) was straight at the time of recording the image.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information from a sensing arrangement (52-54) measuring vehicle kinematics, in particular a yaw sensor (52), a pitch sensor (53) and/or a speed sensor (54).
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information from a sensor (55) measuring a steering wheel angle.
- The vision system as claimed in any one of the preceding claims, comprising a buffer (51) for buffering information relating to each recorded image (40) until a reliability of the corresponding convergence point determination has been confirmed.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information about the type of road the vehicle is moving on.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information from a satellite navigation receiver (60).
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information about the quality of the recorded image.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) comprises a filter adapted to filter out unsuited straight lines or convergence points.
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) is adapted to adjust the orientation of said imaging device (12a, 12b) in order to compensate for an offset (49) between said convergence point (46) and a set point (50).
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) is adapted to disregard image lines essentially lying in an upper part of the image (40).
- The vision system as claimed in any one of the preceding claims, wherein said determination means (30) uses information from a vehicle load sensor (70).
- A vision method for a motor vehicle, comprising the steps of recording images from a region surrounding the motor vehicle by an imaging means (11) comprising at least one imaging device (12a, 12b), processing said recorded images, determining during driving, by using image processing, at least two essentially straight lines (42a, 42b) in the recorded image (40), determining a convergence point (46) from said lines, determining a current orientation of said imaging device (12a, 12b) based on position information of said convergence point (46), and evaluating, on the basis of an independent source (52, 53, 54, 55, 60), a convergence point determination reliability, characterized in that said independent source used by said confirmation means comprises a yaw sensor (52), a pitch sensor (53) and/or a sensor (55) measuring the steering wheel angle.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP09010814.3A EP2293223B1 (en) | 2009-08-24 | 2009-08-24 | Vision system and method for a motor vehicle |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP09010814.3A EP2293223B1 (en) | 2009-08-24 | 2009-08-24 | Vision system and method for a motor vehicle |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2293223A1 EP2293223A1 (en) | 2011-03-09 |
EP2293223B1 true EP2293223B1 (en) | 2016-08-24 |
Family
ID=41203670
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP09010814.3A Active EP2293223B1 (en) | 2009-08-24 | 2009-08-24 | Vision system and method for a motor vehicle |
Country Status (1)
Country | Link |
---|---|
EP (1) | EP2293223B1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3208739A1 (en) | 2016-02-22 | 2017-08-23 | Autoliv Development AB | Driver assistance system and method for a motor vehicle |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5638116A (en) * | 1993-09-08 | 1997-06-10 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
DE102005001429A1 (en) | 2005-01-12 | 2006-07-20 | Robert Bosch Gmbh | Method for image-position correction of a monitor image |
JP4820221B2 (en) * | 2006-06-29 | 2011-11-24 | 日立オートモティブシステムズ株式会社 | Car camera calibration device and program |
US20110115912A1 (en) * | 2007-08-31 | 2011-05-19 | Valeo Schalter Und Sensoren Gmbh | Method and system for online calibration of a video system |
US20090179916A1 (en) * | 2008-01-10 | 2009-07-16 | Williams Steven A | Method and apparatus for calibrating a video display overlay |
-
2009
- 2009-08-24 EP EP09010814.3A patent/EP2293223B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
EP2293223A1 (en) | 2011-03-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11312353B2 (en) | Vehicular control system with vehicle trajectory tracking | |
US20210170947A1 (en) | Vehicular trailering assist system with trailer state estimation | |
CN107797118B (en) | Camera-based trailer identification and blind area adjustment | |
CN106463064B (en) | Object recognition device and vehicle travel control device using same | |
EP3161507B1 (en) | Method for tracking a target vehicle approaching a motor vehicle by means of a camera system of the motor vehicle, camera system and motor vehicle | |
US9135709B2 (en) | Vehicle-to-vehicle distance calculation apparatus and method | |
US7965871B2 (en) | Moving-state determining device | |
JP4420011B2 (en) | Object detection device | |
US9538144B2 (en) | Full speed lane sensing using multiple cameras | |
CN110073429B (en) | Method for monitoring the surroundings of a vehicle combination and monitoring system | |
CN105270408B (en) | Method for determining driving dynamics of a commercial vehicle and driver assistance system | |
JP6649865B2 (en) | Object detection device | |
US11431958B2 (en) | Vision system and method for a motor vehicle | |
JP4832489B2 (en) | Lane judgment device | |
US20110169957A1 (en) | Vehicle Image Processing Method | |
CN107021018A (en) | A kind of commerial vehicle visible system | |
JP2011221983A (en) | Vehicle environment recognition system | |
WO2012045323A1 (en) | Method and driver assistance system for warning a driver of a motor vehicle of the presence of an obstacle in an environment of the motor vehicle | |
EP3087532B1 (en) | Method for determining a width of a target vehicle by means of a camera system of a motor vehicle, camera system and motor vehicle | |
JP5083172B2 (en) | Collision prediction device | |
US9308918B2 (en) | Object collision warning system and method for a motor vehicle | |
JP2009298362A (en) | Lane departure warning device of vehicle | |
CN108885833A (en) | Vehicle checking device | |
EP2293223B1 (en) | Vision system and method for a motor vehicle | |
EP3627448A1 (en) | Vision system and method for a motor vehicle |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA RS |
|
17P | Request for examination filed |
Effective date: 20110823 |
|
17Q | First examination report despatched |
Effective date: 20111122 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20160301 |
|
INTG | Intention to grant announced |
Effective date: 20160314 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 823662 Country of ref document: AT Kind code of ref document: T Effective date: 20160915 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602009040559 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 8 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20160824 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 823662 Country of ref document: AT Kind code of ref document: T Effective date: 20160824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161124 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161226 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160831 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161125 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160831 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160831 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009040559 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161124 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160824 |
|
26N | No opposition filed |
Effective date: 20170526 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20090824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20160831 Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160824 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602009040559 Country of ref document: DE Owner name: ARRIVER SOFTWARE AB, SE Free format text: FORMER OWNER: AUTOLIV DEVELOPMENT AB, VARGARDA, SE Ref country code: DE Ref legal event code: R081 Ref document number: 602009040559 Country of ref document: DE Owner name: VEONEER SWEDEN AB, SE Free format text: FORMER OWNER: AUTOLIV DEVELOPMENT AB, VARGARDA, SE |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: 732E Free format text: REGISTERED BETWEEN 20190808 AND 20190814 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20190829 Year of fee payment: 11 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20190812 Year of fee payment: 11 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20200824 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200831 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200824 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602009040559 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G06K0009000000 Ipc: G06V0010000000 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602009040559 Country of ref document: DE Representative=s name: BARDEHLE PAGENBERG PARTNERSCHAFT MBB PATENTANW, DE |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R081 Ref document number: 602009040559 Country of ref document: DE Owner name: ARRIVER SOFTWARE AB, SE Free format text: FORMER OWNER: VEONEER SWEDEN AB, VARGARDA, SE |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20230711 Year of fee payment: 15 |