EP2732401B1 - Bildgebungsvorrichtung und verfahren zur erzeugimg eines bildes von fahrbahnmarkierungen - Google Patents

Bildgebungsvorrichtung und verfahren zur erzeugimg eines bildes von fahrbahnmarkierungen Download PDF

Info

Publication number
EP2732401B1
EP2732401B1 EP12744085.7A EP12744085A EP2732401B1 EP 2732401 B1 EP2732401 B1 EP 2732401B1 EP 12744085 A EP12744085 A EP 12744085A EP 2732401 B1 EP2732401 B1 EP 2732401B1
Authority
EP
European Patent Office
Prior art keywords
luminance
profiles
image
markings
reflected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Not-in-force
Application number
EP12744085.7A
Other languages
English (en)
French (fr)
Other versions
EP2732401A1 (de
Inventor
Yannick Guillard
Pierre Charbonnier
Philippe FOUCHER
Yazid SEBSADJI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institut Francais des Sciences et Technologirs des Transports de lAmenagement et des Reseaux
Original Assignee
Institut Francais des Sciences et Technologirs des Transports de lAmenagement et des Reseaux
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institut Francais des Sciences et Technologirs des Transports de lAmenagement et des Reseaux filed Critical Institut Francais des Sciences et Technologirs des Transports de lAmenagement et des Reseaux
Publication of EP2732401A1 publication Critical patent/EP2732401A1/de
Application granted granted Critical
Publication of EP2732401B1 publication Critical patent/EP2732401B1/de
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/55—Specular reflectivity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/50—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/804—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for lane monitoring
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/55—Specular reflectivity
    • G01N2021/551—Retroreflectance

Definitions

  • An imaging device for producing an image of pavement road markings and a method for producing such an image.
  • Road markings also known as 'road signs', play a central role in road safety. Therefore, maintaining these markings and monitoring their status are important operations.
  • the main monitoring operation is simply the visual inspection.
  • the inspection campaigns for the markings include the acquisition of photos, or even films, on which the markings appear. These photos or these films make it possible to study the evolution of the state of the markings and to decide on the necessary maintenance and / or renovation operations.
  • the patent application EP1486799 describes an imaging device for determining the condition of road markings of a roadway using photos taken in a vehicle.
  • the system includes a light projector, measuring means and calculating means for producing images of the roadway and calculating the retroreflected light.
  • a first objective of the invention is to propose a device capable of producing information relating to the road markings of a roadway, so that the information produced is sufficient to be able to evaluate the state of the markings. but are provided in a relatively compact format and provide quick access to important information at an affordable cost.
  • an imaging device for producing an image of road markings of a roadway comprising a visible light projector capable of illuminating the roadway in a direction of illumination at an angle of incidence close to 1 , 24 °, to allow the measurement of the luminance of the roadway; means for measuring the illumination thus applied to the roadway by the projector; means for acquiring luminance profiles, capable of acquiring luminance profiles at measurement locations successively illuminated by the projector, the luminance profiles being acquired from the radiation retroreflected by the roadway observed at an angle close to 2, 29 °; and calculation means capable of determining retroreflected luminance profiles from acquired luminance profiles, the retroreflected luminance values of said retroreflected luminance profiles being each equal to the ratio of the luminance of a surface illuminated by the projector to illumination to which the surface is subjected; the calculation means being further capable of producing an image representing said markings from a plurality of luminance profiles acquired by said acquisition means.
  • the retroreflected luminance coefficient of road markings characterizes the ability of these to return the light from the lights of a vehicle to the driver. It is denoted R L and is expressed in mcd.m -2 .lx -1 . As indicated above, the retroreflected luminance coefficient is obtained by dividing the luminance (measured by the acquisition means) by the illumination (measured by the illumination measuring means).
  • the illumination may be the total illumination, or may be the illumination resulting only from the light projected by the projector.
  • 'profile' we denote a sequence (ordinate) of values (an N-tuplet).
  • the luminance profiles therefore contain the luminance values in a sequence of measurement points, generally aligned, measured simultaneously or substantially simultaneously at a measurement location.
  • profiles must be arranged in such a way as to cover the road markings for which an image is to be produced.
  • campaigns are carried out to measure the retroreflected luminance of the markings.
  • Such devices comprise mobile acquisition means, intended to be placed at successive measurement locations, and adapted at each of these locations to acquire a luminance profile comprising N luminance values (N> 1).
  • the successive profiles acquired by these acquisition means can be used to form an image from the luminance values acquired.
  • the number of values contained in each profile is relatively small (It can vary typically from about thirty values measured to 100 or 200 values); and although the retroreflected luminance is not identical to the information and in particular the color information that can be collected by a camera, and in particular is not information as rich as the color information collected by a camera, the image formed from retroreflected luminance profiles, however, provides sufficient information to assess the status of road markings and make decisions about their maintenance.
  • the imaging device provides a usable image for evaluating the state of road markings, and this even in the case where the acquired luminance profiles comprise a restricted number of values, less than 200 or even 150. Luminance profiles with 80 values or less can even be used.
  • the image produced by an imaging device according to the invention contains in practice extremely rich information on the state of road markings, which can advantageously replace the much more basic information used in practice to decide on the maintenance of road markings (the photo campaigns or visual inspections mentioned above are in fact generally used only in a very partial way).
  • This image can also be used as data entry point for a geographic information system (GIS) listing road markings.
  • GIS geographic information system
  • the luminance profiles are measured at successive measuring locations along the path traveled by the vehicle on which the device is mounted. Of course, these profiles all have the same number of elements. These profiles are naturally acquired such that they substantially form an overlap of the mark or markings studied (which means that every point of the marking appears in principle in one or other of the profiles).
  • the luminance profiles are exploited by the calculation means to constitute an image or a set of images which makes it possible to evaluate in a simple and direct manner the state of the markings.
  • the proposed device is therefore particularly economical to use.
  • the image formed using a device according to the invention comprises a number of columns corresponding to the number of values acquired by the acquisition means for each of the luminance profiles. Conversely, it can include a considerable number of lines (which can be split or limited at will), each line corresponding to an acquired profile. Thanks to this, the consultation of all the information obtained can be done by simple scrolling of the image ('scrolling'): Thus, there is no redundancy in the information obtained, thanks to which the consultation of the information obtained is much faster than the examination of the corresponding sequence of images which would have concerned the same markings, according to the previous method.
  • the image obtained by the device according to the invention is thus much lighter (in terms of file size) than a film containing the same information.
  • the device mainly comprises the following components: a projector, illumination measuring means, retro-reflective luminance profile acquisition means, and calculation means.
  • the term 'projector' broadly designates a device for transmitting and projecting light. It can thus comprise a single light source as well as a plurality of light sources. It projects a light having in general a predetermined spectral distribution (in particular according to the standardized illuminant A defined by the ISO10526 standard) and controlled, on the surface or surfaces to be measured.
  • the projected light is a visible light, which may or may not be a coherent light.
  • a device can be operated in a so-called 'passive' mode, projector off, to acquire luminance profiles in ambient light.
  • the acquisition means then provide luminance profiles simply from the radiation emitted by the studied surface under the effect of ambient light or natural light (day or possibly night), without this surface being illuminated by a light. additional light. If the ambient light is insufficient, illumination may be provided to allow the acquisition of the luminance profiles. This additional lighting may come from the imaging device, the headlights of the vehicle on which the device is mounted, or any other projector on board this vehicle.
  • the device then provides images of the markings corresponding to the lighting conditions during the measurement.
  • the device according to the invention can thus, and especially in this passive mode, be used to produce images of all types of markings, including markings having a low luminance coefficient retroreflected, whether because of their wear, or because they are not markings designed to be visible at night.
  • the projector For the measurement of the retroreflected luminance coefficient, especially by day, the projector must be used.
  • the device according to the invention is more particularly advantageous for markings intended to be visible at night, having a high retro-reflective luminance coefficient, since it allows simultaneously for these markings to acquire their retroreflected luminance coefficients, and to constitute an image.
  • the acquisition means are provided for making acquisitions point by point, or line by line, but not in a matrix manner. It is thus well means of acquisition of profiles, and not of matrix acquisition means such as CCD matrices. As a result, the amount of information acquired remains moderate, which limits the calculation requirements by the calculation means.
  • the acquisition means are arranged in such a way that the vertical planes containing the illumination and observation directions are substantially merged.
  • the acquisition means comprise at least one light detector.
  • This detector can be of any type: a simple photodetector, such as a photomultiplier (phototube or photomultiplier tube), a photoconductor (or photoconductive detector), a photodiode; or a multiple photodetector (for simultaneous acquisition of several values of the luminance profile), such as a position sensor device (PSD) or a charge-coupled photodetector (CCD).
  • a simple photodetector such as a photomultiplier (phototube or photomultiplier tube), a photoconductor (or photoconductive detector), a photodiode; or a multiple photodetector (for simultaneous acquisition of several values of the luminance profile), such as a position sensor device (PSD) or a charge-coupled photodetector (CCD).
  • PSD position sensor device
  • CCD charge-coupled photodetector
  • the acquisition means may in particular be able to acquire luminance profiles directly representing the signals supplied by the light detector (s).
  • the luminance values are obtained from the output signals delivered by the light detector (s), with only a shaping treatment of these signals, which may include in particular and by For example, the application of a scale factor, an averaging operation, or the smoothing of a succession of successive values in time and / or from adjacent detectors.
  • This direct processing excludes the calculation of a difference between the signals delivered by the one or more detectors in a first lighting state of the surface and in a second lighting state of the surface different from the first state.
  • the acquisition means comprise a set (or a strip) of detectors for producing the retroreflected luminance profiles, each value of the profile being provided by a detector.
  • the detectors are preferably in sufficiently large numbers; however, a relatively small number of detectors are sufficient to obtain very significant information on the status of road markings.
  • the computing means produce the image (markings) by assigning the luminance profiles to the successive lines of an image. (Of course, it is also possible to transpose the image to produce an image in which the profiles are assigned successively to the different columns.)
  • the image produced is generally in gray levels; false colors can be assigned to facilitate interpretation (ie, assignment to different color pixels of the visible spectrum as a function of the luminance values, according to an arbitrary function).
  • the computing means can produce their images in different ways.
  • the calculation means may constitute the image representing the markings from a plurality of luminance profiles and / or retroreflected luminance profiles.
  • the image produced by the device can thus be constituted by the luminance profiles themselves.
  • the image produced by the device is constituted by the retroreflected luminance profiles, which profiles have more representative values than the luminance values (the division by illumination has the effect of normalizing or homogenizing the values of the luminance values). different profiles).
  • the calculation means are capable of producing an image representing the markings by combining luminance profiles and retroreflected luminance profiles.
  • This "combining" operation refers to any operation that produces the image representing the markings using as input data both luminance profiles and retroreflected luminance profiles.
  • the calculation means may comprise a selection module, adapted according to a predetermined criterion to select the profiles among the retroreflected luminance profiles and the luminance profiles, which will be used to constitute the image representing the markings. The criterion used may be based for example on an evaluation of the luminance variations along the profile.
  • the profiles used to produce the reflectance image are profiles successively acquired by the acquisition means.
  • the calculation means may for example operate only one profile out of two, three, and so on.
  • the image produced by the calculation means may be an image in real colors (in the case of a color acquisition) or monochrome.
  • each luminance profile or each retroreflected luminance profile is obtained from a single acquisition made by the light detectors of the acquisition means.
  • the calculation of the retroreflected luminance coefficients may require a double acquisition of the luminance profiles, namely a projector acquisition turned on, and an off projector acquisition.
  • the acquisition means then calculate the retroreflected luminance profiles on the basis of differences between the luminance profiles acquired when the projector is switched off and the luminance profiles acquired when the projector is switched on.
  • the acquisition means can compute the retroreflected luminance profiles by subtracting the luminance profiles acquired from the off-screen projector with the luminance profiles acquired while the projector is on.
  • a second objective of the invention is to propose a method for producing information on road markings of a roadway which makes it possible to provide information relating to road markings enabling the condition of these road markings to be evaluated, this information being produced in a compact format and allowing quick consultation.
  • a method of producing a road marking image of a roadway, using a retroreflected luminance coefficient of road markings comprising a plurality of acquisitions of luminance profiles, made at a plurality of successive measurement locations, each acquisition consisting of projecting visible light onto surfaces of the roadway at an angle of incidence of approximately 1.24 ° , and to acquire a luminance profile representative of illuminated surfaces; the method comprising in addition, a step of constituting said image from the luminance profiles acquired at the successive measurement locations; the luminance profiles being acquired from the radiation retroreflected by the pavement observed at an angle close to 2.29 °; the measuring device being able to determine retroreflected luminance profile from the acquired luminance profiles; the measuring device comprising means for measuring the illumination applied to said illuminated surfaces; and the retroreflected luminance values of the retroreflected luminance profiles being each equal to the ratio of the luminance of a surface illuminated by the projector to the illumination at which said surface is subjected by the projector.
  • FIG. 1 An imaging device 10 in a first embodiment of the invention will now be described ( figure 1 ).
  • This imaging device 10 comprises a light projector 23, means 25 for measuring the illumination applied to the roadway, means 22,24 for acquiring luminance profiles (P1 ... P20), and calculation means 40.
  • calculation means 40 produce in addition to the images road markings.
  • the device 10 is mounted on board a motor vehicle 12 represented on the figure 1 , circulating on a roadway 14 on which are made road markings 16. These markings are generally in the form of continuous or discontinuous bands, simple or sometimes double; they can be placed on one side or the other of the taxiway. They may also include different signs, such as arrows (see figures 2 or 3 ).
  • the road marking 16 measured consists of a double strip disposed on the left side of the vehicle lane 12.
  • the main hardware components of the device 10 are a central unit 22, a measuring head 24, means 25 for measuring the illumination produced by the projector, a geographical positioning system (GPS) 26, and a projection system 28.
  • GPS geographical positioning system
  • the projector 23 comprises a set of light-emitting diodes placed in the central unit 22, and the projection system 28.
  • the white light produced by the diodes is transmitted by optical fiber to the projection system 28.
  • the system 28 then projects this light onto the pavement, in a lighting direction forming an angle of incidence close to 1.24 ° with the roadway (Neighbor signifying in this document: +/- 0.05 °).
  • the projection system 28 is placed below the measuring head 24, outside the vehicle.
  • the device 10 is powered by the not shown battery of the vehicle 12.
  • the acquisition means luminance profiles are constituted by the measuring head 24 associated with the central unit 22.
  • the acquisition of luminance profiles is as follows:
  • the light-emitting diodes of the projector 23 are powered to produce light.
  • This light is transmitted by the optical fibers to the projection system 28 by which it is projected on the ground in a zone 32 called 'measuring zone'.
  • This zone is a predetermined zone, situated at the front left of the vehicle about 6 meters in front of the measuring head, and whose relative position with respect to the measuring head is defined and standardized in a very precise manner in order to obtain a normalized value for the retroreflected luminance coefficient R L.
  • the vehicle 12 and therefore the device 10 are positioned naturally so that the measurement zone 32 is located on the part of the roadway 14 that is to be studied.
  • the coefficient R L is measured simultaneously in thirty-two points.
  • the light is projected simultaneously in the form of thirty-two light beams 30, of circular section, called "incident beams" (For reasons of readability, only twelve beams are represented on the screen. figure 1 ).
  • the measuring zone 32 consists of the meeting of elliptical-shaped light spots E 1 to E 32 formed on the ground by the light beams 30.
  • the illuminated surface struck by the incident light beams emits a radiation called reflected radiation.
  • This radiation is captured by the acquisition means (more precisely, by the measuring head 24), which comprise for this purpose a set of aligned detectors.
  • the acquisition means comprise thirty-two photodetectors integrated in the measuring head 24, which capture the radiation reflected respectively by each of the light spots E 1 to E 32 .
  • the measuring head 24 is arranged in such a way that the luminance profiles are acquired in one direction at an angle close to 2.29 ° with respect to the surface of the roadway.
  • measuring head 24 and the projection system 28 are shown offset. In reality, they are placed one above the other, so that the vertical planes containing the directions of illumination and observation are substantially merged.
  • the signals produced by the photodetectors are transmitted to the central unit 22, which is a PC-type microcomputer.
  • the central unit 22 constitutes from these signals a luminance profile having thirty-two values.
  • a part of the radiation emitted by the diodes is directed towards the means 25 for measuring the illumination.
  • These means 25 analyze this radiation and determine the value of the illumination to which are subjected the illuminated ellipses E 1 to E 32 of the measurement zone.
  • the calculation means 40 calculate a retroreflected luminance profile, comprising thirty-two values respectively equal to the retroreflected luminance values of each of the ellipses E 1 to E 32 .
  • the retroreflected luminance coefficient is obtained by dividing the luminance measured for the ellipse by the illumination thereof.
  • the acquisition means comprise a set of thirty-two aligned photodetectors.
  • the profiles in the device 10 are simultaneously acquired by a strip of photodetectors, the invention is not limited to this mode of acquisition.
  • the profiles can be acquired by scanning the marking, in particular zig-zag, the important thing is to obtain a set of profiles constituting an overlap at least partial, and preferably complete, the tag studied.
  • the measurement zone can be illuminated only by a single light beam, and the acquisition means may comprise only one photodetector.
  • the profiles be measured in a direction perpendicular to the direction of travel of the vehicle; the angle between the direction of travel and the measurement direction of the profiles may be different from 90 °.
  • the acquisition of luminance can be done not only in white light, but in any wavelength range having a visible part.
  • the light used may be non-coherent, such as white light, or even coherent: the luminance profile acquisition means may comprise a laser source as a means of illuminating the measurement zone.
  • the device comprises means 26 for acquiring position information of the luminance profiles.
  • the computing means 40 are provided to exploit this position information to produce the image or the images representing the markings. It can be understood that by "position of the luminance profiles" is meant the position of the location at which the luminance profile considered has been measured.
  • the device 10 thus comprises a geographical positioning system (GPS) 26 as a position information acquisition means.
  • GPS geographical positioning system
  • the system 26 periodically acquires the position of the vehicle and transmits it to the calculation means 40.
  • the calculation means 40 calculate an estimate the position of each measurement location (by interpolation or any other appropriate method).
  • the position information of the luminance profiles is an absolute position information.
  • the GPS is replaced by an odometer as a means for acquiring position information of the luminance profiles.
  • An approximate position of the position of each measurement location can then be evaluated as a curvilinear abscissa along the track taken.
  • the position of the luminance profiles is (but not necessarily) a relative position, defined with respect to the starting point, and not an absolute position.
  • the image or images produced by the computing means are georeferenced.
  • the position information can also be used to perform a post-processing of the image produced by the imaging device 10. Such a post-processing will be detailed later.
  • a first phase of production of the images consists in making a number of acquisition of luminance profiles at successive measurement locations. These profiles are chosen so as to form at least a partial recovery of the markings studied.
  • the driver of the vehicle 12 drives the latter so that the measurement zone sweeps the markings to be measured during the movement of the vehicle.
  • the device 10 is activated and proceeds to the successive acquisition of luminance profiles. This process is illustrated by the figure 1 .
  • the vehicle 12 is shown in solid lines in a first position P, which corresponds to a first measurement location 32 at which a first luminance profile is measured.
  • a second acquisition is triggered at a second measurement location 32' to provide a second profile.
  • the vehicle then moves to a third position P "to allow a profile to be measured at a third location 32", and so on.
  • the successive measuring locations are distant according to the direction of movement of the vehicle by a distance D. This depends on the frequency of acquisitions by the device 10 and the speed of the vehicle.
  • the distance D can be set so that there is or no overlap between the measurement zones of the device 10 at successive measuring locations; preferably, in accordance with the illustration, it is preferable to make sure that the measurement locations substantially achieve an overlap of the surface of the roadway studied, but without however being superimposed.
  • the image is formed from the measured luminance profiles. This step of constituting the images of road markings by the calculation means 40 is illustrated by the figure 2 .
  • the profiles acquired by the device 10 each comprise thirty-two luminance values.
  • Each profile is therefore a N-tuplet or vector of 32 values R L1 , R L2 , ... R L32 .
  • the retroreflected luminance profiles are obtained.
  • the successive retroreflected luminance profiles are successively assigned to the lines of an image.
  • twenty profiles P1, P2, ... P20, acquired successively during the movement of the vehicle along a drawdown arrow were obtained; the corresponding retroreflected luminance profiles are successively assigned to the twenty lines of an image.
  • the image thus produced has on the sides dark pixels corresponding to the dark gray color of the roadway; white pixels corresponding to the elementary measurement zones E i which have been measured entirely on the marking; and light gray pixels, which correspond to the elementary zones of measurement which have been measured partly on the marking and partly on the pavement.
  • each of the elementary measurement zones E i has a ground surface of the order of 5 ⁇ 40 cm
  • the pixels of the image obtained are not square but rectangular with a ratio of the order of 8 between the height and width.
  • each acquired luminance value can be assigned for example to 8 successive lines, as represented in FIG. Figure 2B .
  • the method according to the invention makes it possible to very easily obtain an image of the marking, this image being, however, of rather low resolution in the embodiment presented.
  • this image being, however, of rather low resolution in the embodiment presented.
  • nothing prevents the use of a measuring device having a higher resolution, ie an increased number of luminance measurement points (ie photodetectors) per unit length.
  • the image was formed from a series of retroreflected luminance profiles.
  • the calculation means 40 are able to establish 'raw' luminance profiles. These raw profiles contain the values directly provided by the photodetectors of the Measuring head 24. Also, the image of road markings can be produced from the luminance profiles themselves rather than from retroreflected luminance profiles. This possibility makes it possible in particular to carry out day measurement campaigns to constitute images of road markings.
  • Different post-treatments can be applied to the image produced by the imaging device 10 in order to make this image more easily exploitable.
  • a first family of post-treatments consists of identification operations of the markings present in the image. This operation is a pattern recognition that can use all known algorithms in this area. For this purpose, the calculation means are able to extract in the image the type, the position and / or the contour of the road markings.
  • the markings thus detected can then be identified in the image, for example by codes ('1', '3', 'CD' in the image on the right of the image).
  • figure 3 which is the image obtained after the various post-treatments).
  • a second family of post-treatments consists of grinding operations:
  • An identified marking can be replaced in the image either by a thumbnail (small image) representing the standard marking; alternatively, the correction may consist only of a color change of the pixels, the marking pixels being for example replaced by white pixels, the pixels of the road being for example replaced by black pixels.
  • the pixels placed at the edge of the marking then remain preferably of intermediate color.
  • a third family of post-processing consists of shifting operations to change the position of one (or more) markings in the image. It is therefore necessary to move in the image a group of pixels representing a previously identified road marking.
  • This offset may be in particular an offset in the direction of the image which corresponds to a direction perpendicular to the displacement of the vehicle (horizontal direction in the case of the figure 2 , the direction corresponding to the direction of displacement being the vertical direction in the image of the figure 2 ).
  • the calculation means have shifted horizontally the pixels corresponding to the drawdown arrow, without there being vertical offset of these pixels.
  • the offset can also be an offset in the direction of the image that corresponds to the direction of vehicle travel (vertical direction in the case of figure 2 ).
  • the calculation means are able to modify in said image the position of road markings in the direction corresponding to the direction of movement of the luminance profile acquisition means, in particular as a function of said position information. Such a modification is then made mainly according to the available position information (georeferenced absolute position, curvilinear abscissa on the path, etc.), in order to increase the accuracy of the final image obtained.
  • the computing means 40 integrated in the central unit 22 are on board the vehicle 12.
  • the computing means may be means in the form of for example a computer in an office. All image forming operations or images representing road markings, as well as post-processing applied to these images, can then be performed on this computer, whether in real time or offline (in batch).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Vascular Medicine (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Claims (13)

  1. Bildgebungsvorrichtung (10) zur Erstellung eines Bilds von Fahrbahnmarkierungen (1, 3, CD) einer Fahrbahn (14) aus einer Vielzahl von Profilen einer rückflektierten Luminanz, die aufeinanderfolgenden Messorten entsprechen, wobei die Vorrichtung umfasst:
    einen Projektor von sichtbarem Licht (23), der geeignet ist, die Fahrbahn in einer Beleuchtungsrichtung unter einem Einfallwinkel nahe 1,24° zu beleuchten, um die Messung der Luminanz der Fahrbahn zu gestatten,
    Mittel zur Messung der Beleuchtung, die so von dem Projektor auf die Fahrbahn ausgeübt wird,
    Mittel (22, 24) zur Erfassung der Luminanzprofile (P1 ... P20), welche geeignet sind, Luminanzprofile an Messorten zu erfassen, die aufeinanderfolgend von dem Projektor beleuchtet werden, wobei die Luminanzprofile aus der von der Fahrbahn rückreflektierten Strahlung, beobachtet unter einem Winkel nahe 2,29°, erfasst werden, und
    Rechenmittel (40), die geeignet sind, die Profile der rückreflektierten Luminanz aus den erfassten Luminanzprofilen zu bestimmen, wobei die Werte der rückreflektierten Luminanz der Profile der rückreflektierten Luminanz jeweils gleich dem Verhältnis der Luminanz einer von dem Projektor beleuchteten Fläche zu der Beleuchtung, der die Fläche ausgesetzt wird, sind,
    wobei die Rechenmittel (40) außerdem geeignet sind, ein Bild, das die Markierungen darstellt, aus einer Vielzahl von Luminanzprofilen zu erstellen, die von den Erfassungsmitteln erfasst werden.
  2. Vorrichtung nach Anspruch 1, wobei die erfassten Luminanzprofile eine begrenzte Anzahl von Werten umfassen, die kleiner ist als 200 und vorzugsweise kleiner als 150.
  3. Vorrichtung nach Anspruch 1 oder 2, wobei die Erfassungsmittel geeignet sind, Erfassungen Punkt-für-Punkt oder Linie-für-Linie, jedoch nicht matrixförmig vorzunehmen.
  4. Vorrichtung nach einem der Ansprüche 1 bis 3, wobei die Erfassungsmittel geeignet sind, ein Profil der rückreflektierten Luminanz auf der Basis von Unterschieden zwischen einem Luminanzprofil, das erfasst wird, wenn der Projektor ausgeschaltet ist, und einem Luminanzprofil, das erfasst wird, wenn der Projektor beleuchtet ist, zu berechnen.
  5. Vorrichtung nach einem der Ansprüche 1 bis 4, wobei die Rechenmittel das Bild erstellen, indem die Profile der Vielzahl von Profilen aufeinanderfolgenden Linien eines Bilds zugeordnet werden.
  6. Vorrichtung nach einem der Ansprüche 1 bis 5, außerdem umfassend Mittel (26) zur Erfassung von Positionsinformationen der Luminanzprofile, und wobei die Rechenmittel (40) geeignet sind, die Positionsinformationen zu nützen, um das mindestens eine Bild zu erstellen.
  7. Vorrichtung nach Anspruch 6, wobei die Rechenmittel geeignet sind, in dem Bild die Position von Fahrbahnmarkierungen in der Richtung zu modifizieren, die der Richtung der Verschiebung der Mittel zur Erfassung der Luminanzprofile entspricht, insbesondere als Funktion der Positionsinformationen.
  8. Vorrichtung nach Anspruch 6 oder 7, wobei die die Rechenmittel geeignet sind, in dem Bild die Position der Fahrbahnmarkierungen in der Richtung zu modifizieren, die einer Richtung rechtwinklig zu der Verschiebungsrichtung der Mittel zur Erfassung der Luminanzprofile entspricht, insbesondere als Funktion der Positionsinformationen, oder auch um die Markierungen in dem Bild zu zentrieren.
  9. Vorrichtung nach einem der Ansprüche 1 bis 8, wobei die Rechenmittel geeignet sind, in dem Bild den Typ, die Position und/oder die Kontur der Fahrbahnmarkierungen zu extrahieren.
  10. Vorrichtung nach einem der Ansprüche 1 bis 9, wobei die Mittel zur Erfassung der Luminanzprofile eine Einheit von ausgerichteten Detektoren umfassen.
  11. Vorrichtung nach einem der Ansprüche 1 bis 10, wobei die Mittel (22, 24) zur Erfassung der Profile mindestens einen Lichtdetektor umfassen, und geeignet sind, die Luminanzprofile zu erfassen, welche direkt die Signale repräsentieren, die von dem mindestens einen Lichtdetektor geliefert werden.
  12. Vorrichtung nach einem der Ansprüche 1 bis 11, wobei die Rechenmittel das Bild, das die Markierungen darstellt, aus einer Vielzahl von Luminanzprofilen und/oder Profilen einer rückreflektierten Luminanz bilden.
  13. Verfahren zur Erstellung eines Bilds von Fahrbahnmarkierungen einer Fahrbahn aus einer Vielzahl von Profilen einer rückflektierten Luminanz, die aufeinanderfolgenden Messorten entsprechen, unter Verwendung einer Vorrichtung zur Messung von Koeffizienten einer rückreflektierten Luminanz der Fahrbahnmarkierungen, wobei das Verfahren eine Vielzahl von Erfassungen von Luminanzprofilen umfasst, die an einer Vielzahl von aufeinanderfolgenden Messorten vorgenommen werden, wobei jede Erfassung aus der Projektion eines sichtbaren Lichts auf Flächen der Fahrbahn unter einem Einfallwinkel nahe 1,24°, und der Erfassung eines Luminanzprofiles, das die beleuchteten Flächen darstellt, besteht,
    wobei das Verfahren außerdem einen Schritt der Erstellung des Bilds aus Luminanzprofilen umfasst, die an aufeinanderfolgenden Messorten erfasst werden,
    wobei die Luminanzprofile aus der von der Fahrbahn rückreflektierter Strahlung, beobachtet unter einem Winkel nahe 2,29°, erfasst werden,
    wobei die Messvorrichtung geeignet ist, Profile der rückreflektierten Luminanz aus den erfassten Luminanzprofilen zu bestimmen,
    wobei die Messvorrichtung Mittel zur Messung der Beleuchtung umfasst, die auf die beleuchteten Flächen ausgeübt wird, und
    wobei die Werte der rückreflektierten Luminanz der Profile der rückreflektierten Luminanz jeweils gleich dem Verhältnis der Luminanz einer von dem Projektor beleuchteten Fläche zu der Beleuchtung, der die Fläche durch den Projektor ausgesetzt wird, sind.
EP12744085.7A 2011-07-12 2012-07-11 Bildgebungsvorrichtung und verfahren zur erzeugimg eines bildes von fahrbahnmarkierungen Not-in-force EP2732401B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1156362A FR2977957B1 (fr) 2011-07-12 2011-07-12 Dispositif et procede d'imagerie pour produire une image de marquages routiers
PCT/FR2012/051647 WO2013007955A1 (fr) 2011-07-12 2012-07-11 Dispositif et procede d'imagerie pour produire une image de marquages routiers

Publications (2)

Publication Number Publication Date
EP2732401A1 EP2732401A1 (de) 2014-05-21
EP2732401B1 true EP2732401B1 (de) 2018-10-17

Family

ID=46639611

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12744085.7A Not-in-force EP2732401B1 (de) 2011-07-12 2012-07-11 Bildgebungsvorrichtung und verfahren zur erzeugimg eines bildes von fahrbahnmarkierungen

Country Status (6)

Country Link
EP (1) EP2732401B1 (de)
DK (1) DK2732401T3 (de)
ES (1) ES2702113T3 (de)
FR (1) FR2977957B1 (de)
PT (1) PT2732401T (de)
WO (1) WO2013007955A1 (de)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9829888B2 (en) * 2015-11-17 2017-11-28 Ford Global Technologies, Llc Distinguishing lane markings for a vehicle to follow
WO2018041407A1 (de) * 2016-08-31 2018-03-08 Sew-Eurodrive Gmbh & Co. Kg Verfahren zur spurführung, system zur spurführung und fahrzeug

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1486799A2 (de) * 2003-06-10 2004-12-15 Tieliikelaitos Verfahren und Anordnung zur Bestimmung des Zustandes von Fahrbahnmarkierungen

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4721389A (en) * 1985-04-18 1988-01-26 Potters Industries, Inc. Method and apparatus for measuring retroreflectivity of a reflective layer on a surface
GB2338300B (en) * 1997-03-05 2001-07-11 Philip Anthony Gallagher Reflectivity measuring apparatus and method
US20040160595A1 (en) * 2003-02-14 2004-08-19 Lafarge Road Marking, Inc. Road marking evaluation and measurement system
JP4793094B2 (ja) * 2006-05-17 2011-10-12 株式会社デンソー 走行環境認識装置
FR2935489B1 (fr) 2008-08-29 2010-10-22 France Etat Ponts Chaussees Dispositif et procede de mesure du coefficient de luminance retroreflechie de marquages routiers
KR101244498B1 (ko) * 2008-10-22 2013-03-18 주식회사 만도 차선을 인식하는 방법 및 장치
AU2009245853B2 (en) * 2009-12-08 2013-12-19 Radar Portal Systems Pty Ltd High speed photometric stereo pavement scanner

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1486799A2 (de) * 2003-06-10 2004-12-15 Tieliikelaitos Verfahren und Anordnung zur Bestimmung des Zustandes von Fahrbahnmarkierungen

Also Published As

Publication number Publication date
FR2977957B1 (fr) 2016-07-01
WO2013007955A1 (fr) 2013-01-17
EP2732401A1 (de) 2014-05-21
FR2977957A1 (fr) 2013-01-18
PT2732401T (pt) 2019-01-24
ES2702113T3 (es) 2019-02-27
DK2732401T3 (en) 2019-01-28

Similar Documents

Publication Publication Date Title
EP2225608B1 (de) Vorrichtung zur beurteilung einer reifenfläche
US7801333B2 (en) Vision system and a method for scanning a traveling surface to detect surface defects thereof
AU2009245853B2 (en) High speed photometric stereo pavement scanner
EP3097407B1 (de) Verfahren und vorrichtung zur erkennung insbesondere von lichtbrechenden defekten
EP1790976B1 (de) Optisches Verfahren und optische Vorrichtung zur Erkennung von Defekten der Oberfläche und der Struktur eines heißen, sich kontinuierlich bewegenden Produkts
CN105452807A (zh) 测距***、以及信号产生装置
FR2938330A1 (fr) Evaluation du relief de la surface d'un pneumatique par stereovision active
EP1868161A1 (de) Verfahren zur Bestimmung der Sichtweite für einen Fahrzeugfahrer
CN102194103A (zh) 用于车辆的主动可视性支持设备和方法
EP2732401B1 (de) Bildgebungsvorrichtung und verfahren zur erzeugimg eines bildes von fahrbahnmarkierungen
Kang et al. High-resolution depth imaging with a small-scale SPAD array based on the temporal-spatial filter and intensity image guidance
JP5229427B2 (ja) 距離測定装置および距離測定方法
EP3926529B1 (de) Systeme und methoden zur bestimmung einer anzahl von personen in einem fahrzeug
FR3055702A1 (fr) Dispositif et procede pour determiner l'etat de la surface d'une chaussee
CA2509076C (en) A vision system and a method for scanning a traveling surface to detect surface defects thereof
CA2941627A1 (fr) Procede de releve de donnees photometriques de precision amelioree
EP2508870A1 (de) Vorrichtung zur Inspektion von durchlaufendem Bandmaterial
EP1791078A1 (de) Verfahren zur Detektion von Hindernissen, das in einem Motorfahrzeug implementiert wird
EP3570730A1 (de) Verfahren und vorrichtung zur messung der an der oberfläche von biologischem gewebe emittierten fluoreszenz
EP4364108A1 (de) Verfahren zur analyse der optischen qualität eines verglasungselements, verfahren zur kalibrierung einer kamera, so analysiertes verglasungselement
EP4145960A1 (de) System zur steuerung einer adaptiven beleuchtungsanlage und entsprechendes verfahren
EP4178732A1 (de) Vorrichtung zur inspektion von objekten in einem sich bewegenden fluss und maschine mit solch einer vorrichtung
FR3101144A1 (fr) Système et procédé de contrôle de la couleur d’un article en déplacement
FR2948478A1 (fr) Procede de releve d'informations relatives a des panneaux de signalisation verticale, dispositif associe et application
FR2741467A1 (fr) Chaine d'acquisition et de traitement d'images

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20131220

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RIN1 Information on inventor provided before grant (corrected)

Inventor name: SEBSADJI, YAZID

Inventor name: CHARBONNIER, PIERRE

Inventor name: FOUCHER, PHILIPPE

Inventor name: GUILLARD, YANNICK

RIN1 Information on inventor provided before grant (corrected)

Inventor name: SEBSADJI, YAZID

Inventor name: FOUCHER, PHILIPPE

Inventor name: CHARBONNIER, PIERRE

Inventor name: GUILLARD, YANNICK

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20170905

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180511

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: FRENCH

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012052321

Country of ref document: DE

Ref country code: AT

Ref legal event code: REF

Ref document number: 1054852

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181115

REG Reference to a national code

Ref country code: CH

Ref legal event code: NV

Representative=s name: MICHELI AND CIE SA, CH

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Ref document number: 2732401

Country of ref document: PT

Date of ref document: 20190124

Kind code of ref document: T

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20190116

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20190123

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20181017

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2702113

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20190227

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1054852

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190117

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190117

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190217

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190118

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012052321

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

26N No opposition filed

Effective date: 20190718

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190711

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PT

Payment date: 20200616

Year of fee payment: 9

Ref country code: IE

Payment date: 20200622

Year of fee payment: 9

Ref country code: DK

Payment date: 20200625

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20200817

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20200720

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20120711

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20210721

Year of fee payment: 10

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602012052321

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06K0009200000

Ipc: G06V0030140000

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20210720

Year of fee payment: 10

Ref country code: CH

Payment date: 20210722

Year of fee payment: 10

Ref country code: BE

Payment date: 20210726

Year of fee payment: 10

Ref country code: DE

Payment date: 20210712

Year of fee payment: 10

REG Reference to a national code

Ref country code: DK

Ref legal event code: EBP

Effective date: 20210731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210712

Ref country code: PT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181017

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210711

Ref country code: DK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210731

REG Reference to a national code

Ref country code: ES

Ref legal event code: FD2A

Effective date: 20220927

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210712

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602012052321

Country of ref document: DE

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602012052321

Country of ref document: DE

Representative=s name: CBDL PATENTANWAELTE GBR, DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20220711

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220711

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230201

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220731