EP3418766A1 - Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet - Google Patents

Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet Download PDF

Info

Publication number
EP3418766A1
EP3418766A1 EP18176901.9A EP18176901A EP3418766A1 EP 3418766 A1 EP3418766 A1 EP 3418766A1 EP 18176901 A EP18176901 A EP 18176901A EP 3418766 A1 EP3418766 A1 EP 3418766A1
Authority
EP
European Patent Office
Prior art keywords
time
light
sensor
parameter
avalanche
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18176901.9A
Other languages
German (de)
English (en)
Other versions
EP3418766B1 (fr
Inventor
Martin Köhl
Stefan Kienzler
Kai Waslowski
Ulrich Zwölfer
Christophe Thil
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sick AG
Original Assignee
Sick AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sick AG filed Critical Sick AG
Publication of EP3418766A1 publication Critical patent/EP3418766A1/fr
Application granted granted Critical
Publication of EP3418766B1 publication Critical patent/EP3418766B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/4861—Circuits for detection, sampling, integration or read-out
    • G01S7/4863—Detector arrays, e.g. charge-transfer gates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
    • G01S17/14—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein a voltage or current pulse is initiated and terminated in accordance with the pulse transmission and echo reception respectively, e.g. using counters
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/4865—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
    • G01S17/18—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves wherein range gates are used
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497—Means for monitoring or calibrating

Definitions

  • the invention relates to an optoelectronic sensor and a method for measuring the distance to an object in a surveillance area according to the preamble of claims 1 and 15, respectively.
  • a distance to the object is determined in addition to the pure object detection.
  • the distance information and three-dimensional images or so-called depth maps are detected when the sensor is spatially resolving.
  • a scanner scans the surveillance area with a light beam, while a 3D camera also determines distance information for each of its pixels instead of or in addition to the brightness information.
  • the pixels may also each have a plurality of photosensitive elements, which together contribute to a distance value.
  • a conventional method for distance measurement is the time of flight measurement.
  • a short light pulse is emitted and the time to receive a remission or reflection of the light pulse measured.
  • To gain a higher robustness against interference events and noise effects for example DE 10 2007 013 714 A1 It is known to send a plurality of individual light pulses one after the other, to collect the received signals subsequently generated in a histogram and then to evaluate them together, for example via a search for a maximum in the histogram, from which the time of reception is derived.
  • Such histogram evaluations require a lot of memory, since the total light propagation time to be expected in the measurement area is subdivided into bins whose width is at least close to the desired measurement resolution. If the distance measurement is to be spatially resolving, as in a 3D camera, then this memory requirement still scales with the The number of pixels, or the acquisition time increases significantly to avoid the increased memory requirements by sequential processing of the pixels. Particularly hindering for the development of cost-effective integrated evaluation modules such as in the form of an ASIC (Application Specific Integrated Circuit).
  • ASIC Application Specific Integrated Circuit
  • avalanche photodiode Avalanche Photo Diode
  • the incident light triggers a controlled avalanche effect.
  • the charge carriers generated by incident photons are multiplied, and a photocurrent is produced which is proportional to the light reception intensity but much larger than that of a simple PIN diode.
  • Geiger mode the avalanche photodiode is biased above the breakdown voltage, so that even a single, released by a single photon charge carrier can trigger an avalanche, which then recruits all available charge carriers due to the high field strength.
  • the avalanche photodiode thus counts as the eponymous Geiger counter individual events.
  • Geiger mode avalanche photodiodes are also referred to as SPAD (Single-Photon Avalanche Diode).
  • the high sensitivity of the SPADs also brings disadvantages, since in the limiting case even a single disturbing photon or internal noise event delivers the same signal as a pronounced useful signal.
  • the SPAD is subsequently not accessible for a certain dead time, this dead time in fact means on the short time scales of a time of flight measurement, that a SPAD is then available again at a repeated measurement.
  • the conventional methods for measuring the time of flight measurement do not take these special features of SPADs into account. Although they can therefore be transferred to SPAD light receivers, there is still room for improvement through SPADs.
  • a light emitter With a light emitter, light signals are emitted, received again after reflection or remission on an object in a light receiver, and the single light transit time is determined.
  • the light receiver has at least one avalanche photodiode or SPAD operated in Geiger mode.
  • a plurality of individual light transit times are measured from the sensor to the object and a common measured value is determined therefrom. Strictly speaking, a single runtime is determined for round trip.
  • the statistics are obtained over time and / or location, namely by repeated measurements with several successive individual light pulses or by the light receiver has several avalanche photodiodes in Geiger mode.
  • the invention is based on the idea of modeling the special statistical behavior of SPADs during measurement and background events.
  • measurement events are caused by photons of the light signal
  • background events are all other triggers of an avalanche, such as dark noise or extraneous light reception.
  • the sought evaluation result is the exact reception time t s , from which then follows with the help of a reference time, in particular the transmission time, the light transit time and the speed of light taking into account the distance. This means the actual reception time t s corresponding to the distance of the object and not that of a single light signal on a single avalanche photodiode, which represents only a single measurement associated with measurement errors or possibly corresponding to a background event.
  • This reception time t s is inventively sought in a predetermined time interval [ t 0 , t 1 ], which is for example a measurement period between two transmitted light signals or from a transmission time to a maximum range corresponding time or a sub-interval of the measurement period.
  • the model from which the determination of the reception time t s is derived is based on the number N (t) of the avalanche photodiodes still available at a particular time t . These are those avalanche photodiodes that are not already in their dead time due to a measurement or background event.
  • the model takes into account the special nature of avalanche photodiodes in Geiger mode or SPADs.
  • the number N ( t ) can also include repeated measures.
  • the maximum initial number of avalanche photodiodes available must by no means be directly determined by the number of avalanche photodiodes physically present.
  • the total pool of available avalanche photodiodes becomes n times n times by n measurements. Even in only one measurement, it is not absolutely necessary for all existing avalanche photodiodes to be available because they may not be included in the measurement due to parameterization, defect or the like.
  • the invention has the advantage that a light transit time and thus the distance to an object can be determined particularly accurately and with very little effort.
  • the model not only takes into account the particular nature of avalanche photodiodes in Geiger mode or SPADs, but even uses their properties constructively to determine an exact time of reception t s . Since the background in particular can be included, the evaluation is also very robust against disturbances such as strong extraneous light or high dark noise and thus particularly suitable for demanding industrial applications.
  • the evaluation unit is preferably designed to determine the reception time t s with the aid of a background parameter ⁇ .
  • the background parameter ⁇ is then the time constant of an exponential function. However, it may also be a quantity which describes the background in an equivalent or approximate way.
  • the evaluation unit is preferably designed to determine the background parameter ⁇ from a measurement of individual light propagation times.
  • a measurement with an inactive light transmitter ie without emitting light signals, so that the light receiver registers only background.
  • methods of order statistics it is also possible to determine the background parameter ⁇ from individual time delays with a measurement event.
  • the background parameter ⁇ can be easily specified, whether as a fixed parameter or as a result of a measurement from another source.
  • the evaluation unit is preferably designed to determine the reception time with the aid of a signal strength parameter p .
  • This is a dimensionless parameter which depends on the intensity of the remitted light signal. The stronger the incidence of light, the more measurement events are triggered, which then results in a strong measurement signal. Accordingly, after a strong measurement signal, the number N (t) also decreases more, which involves the model by the signal strength parameter p .
  • a plurality of avalanches are preferably triggered according to a Dirac pulse ⁇ ( t - t s ) at the time of reception t s .
  • the received signal is thus modeled as an infinitesimally short pulse. This is not completely true, but at least very short transmission pulses are common and possible.
  • a Dirac pulse of triggering avalanche photodiodes leads to a sudden decrease in the number N ( t ) avalanche photodiodes available according to the Heaviside function ⁇ , weighted with the signal strength parameter p.
  • Other functions, such as a Gaussian pulse are conceivable, but require at least one additional parameter for describing the pulse shape, and therefore the evaluation can not be carried out in the same simplicity.
  • the evaluation unit is preferably designed to determine the signal strength parameter p from the number of avalanche photodiodes still available at the lower time limit t 0 and the upper time limit t 1 .
  • Avalanche photodiodes which were still available at the beginning of the time interval and are not anymore, have registered either a measurement or a background event.
  • the background can be detected by the background parameter ⁇ .
  • the remaining difference is therefore the consequence of measurement events and therefore suitable for determining the signal strength parameter p .
  • the signal strength parameter p is determined simply and quickly by a closed expression.
  • the evaluation unit is preferably designed to sequentially first determine a background parameter ⁇ , then with the aid of the background parameter ⁇ a signal strength parameter p and then with the aid of background parameter ⁇ and signal strength parameter p the reception time t s .
  • a sequential determination of all required parameters for the determination of the reception time t s is preferably possible without approximations or iterative procedure. Due to this sequential determination and the absence of iterative procedures, the procedure according to the invention is significantly more robust and at the same time less complicated than a simultaneous estimation or calculation of several parameters.
  • the evaluation unit preferably calculates numerically, not analytically, but the rule for the numerical calculation is an analytical formula. An alternative procedure such as an iterative evaluation is conceivable, but not required.
  • the time t, together with the associated number N (t) of avalanche photodiodes still available, which enters into the calculation can be mathematically shifted with knowledge of the background parameter ⁇ in the time range before the reception time t s and can therefore be freely selected.
  • the total number N (0) of total available avalanche photodiodes known from the outset can be mathematically adjusted at a time t .
  • this number N ( t ) of avalanche photodiodes still available can be counted by counting the total number of events before time t and subtracting the result of N (0) even without knowing the background parameter.
  • a further parameter is required to determine the common measured value t s , because otherwise the model would be underdetermined. In particular, however, no knowledge of N ( t ) inside [ t 0 , t 1 ] is necessary if one chooses a suitable further parameter.
  • the further parameter is preferably an average ⁇ of the individual light propagation times in the predeterminable time interval [ t 0 , t 1 ].
  • the evaluation unit can very easily determine this mean value ⁇ , and indeed "on the fly", without memorizing individual light propagation times. As a result, in contrast to a conventional histogram evaluation, considerable storage or bandwidth resources can be saved for providing the histogram data.
  • the measurement resolution is not limited by the evaluation, for example, as is conventional to a bin width of a histogram. Computing facilities for nonlinear functions through lookup tables or approximations are conceivable.
  • the evaluation unit is preferably designed to first find or refine a specifiable time interval that is shorter than a measurement period.
  • the time interval does not matter. In a real measurement, however, the effects of noise cause that too long a time interval does not necessarily provide the desired accuracy.
  • the time interval can be found by a pre-evaluation, for example by a threshold evaluation, which roughly localizes a maximum of measurement events.
  • this pre-estimate is estimated as how many single-light durations are to be expected due to background events in a time interval, assuming an exponentially decreasing frequency of background events, and then a time interval is determined in which there are significantly more individual time delays than expected.
  • the light receiver preferably has a multiplicity of avalanche photodiodes operated in a Geiger mode and a plurality of individual light transit time measuring units which are assigned individually or in groups to the avalanche photodiodes and in particular have at least one TDC.
  • the invention is also applicable in the case of only one avalanche photodiode and consequently only one single light transit time measuring unit, since the number N ( t ) of available avalanche photodiodes is by no means limited to one by repeated measurement.
  • the number of physically present avalanche photodiodes and measurement duration or number of repeated measurements are interchangeable.
  • the avalanche photodiodes are preferably arranged in line or matrix fashion. There are then different variants of the interconnection.
  • the avalanche photodiodes can deliver a common measured value overall. But it is also possible to obtain a spatial resolution by several avalanche photodiodes as a group each determine a common reading. This then yields a 3D image sensor whose effective local resolution corresponds to the number of groups, the group size allowing an exchange relationship of spatial resolution, accuracy and measurement duration of the range determination.
  • Single light transit times are optionally measured for each avalanche photodiode or only together for several avalanche photodiodes.
  • the individual light transit time measuring units may be assigned permanently or variably to certain avalanche photodiodes.
  • the single light transit time measuring units preferably have a TDC (time-to-digital converter).
  • TDC time-to-digital converter
  • the TDC is preferably started at the time of transmission and stopped at the time of reception by the received individual light pulse.
  • Other modes are conceivable, such as starting the TDCs each with the triggering of an avalanche and then stop at a known time, such as the end of the measurement period.
  • FIG. 1 shows a simplified block diagram of a sensor 10 for determining the distance to an object in a monitoring area 12 by measuring light transit times.
  • the sensor 10 is in FIG. 1 divided into an upper transmission path 14 and a lower reception path 16. This division is not intended to imply technical characteristics.
  • the invention relates primarily to the receive path 16, so that for the transmit path 14 any known implementation is conceivable.
  • the elements of the transmission path 14 may be individual components, but may also be integrated with the elements of the reception path 16 on a common component.
  • light signals are generated, preferably with a pulse generator 18 short individual pulses.
  • the term single pulse refers to a single, as opposed to a total measurement, not the pulse shape. Rather, the pulse shapes, pauses and lengths can be varied, for example for coding or adaptation to ambient conditions. However, for the purposes of the invention, the simpler idea of a uniform sequence of individual pulses, which have sufficient time interval between them, that the measurements do not influence one another is sufficient.
  • a light transmitter 20, for example an LED or laser diode generates from the electronic transmission signal corresponding individual light pulses 22 which are emitted into the monitoring area 12. If the individual light pulses 22 encounter an object there, a corresponding reflected or remitted individual light pulse returns 24 back to the sensor 10 and encounters a light receiver 26 which generates therefrom an electronic received signal.
  • the light receiver 26 has avalanche photodiodes, not shown, in particular a large number of avalanche photodiodes in a line or matrix arrangement. In this case, a spatial resolution can be maintained and so create a 3D image sensor, with a shared spatial analysis of multiple avalanche photodiodes a reduced spatial resolution at more accurate distance measurement can be achieved. In extreme cases, all avalanche photodiode elements are used to determine a common measured value.
  • the avalanche photodiodes are operated as already briefly described in a Geiger mode and also referred to as SPADs.
  • Avalanche photodiodes or APDs are biased above their breakdown voltage and the avalanche current can be triggered by a single photon.
  • SPADs are therefore extremely sensitive, but at the same time susceptible to erroneous measurements, because a light transit time determined by a SPAD can erroneously go back to dark noise or the registration of an extraneous light photon and then be completely uncorrelated with the distance of the object.
  • an avalanche photodiode after an avalanche for a dead time is no longer available.
  • the model approach according to the invention for determining the light transit time is adapted to these properties of the SPADs.
  • a single light transit time measuring unit 28 determines the respective individual light transit time between emitting a single light pulse 22 and receiving the associated remitted individual light pulse 24.
  • individual light transit time measuring units 28 can be provided, which are assigned permanently or dynamically to avalanche photodiodes or groups thereof.
  • a block of TDCs time-to-digital converter
  • the respective TDC is started when a single light pulse is emitted and stopped by a signal generated by the avalanche in an associated avalanche photodiode.
  • the mode of operation of the TDCs is not fixed, for example, another possible mode of operation is the so-called common stop mode, in which the signal of the avalanche photodiode starts the respective TDC and all TDCs are stopped together, for example at the end of a measurement period.
  • FIG. 1 Only rudimentary and are only further below with reference to the FIGS. 3 and 4 explained in more detail.
  • a memory 30 This can already be done in summary, for example in a histogram with a bin width chosen in consideration of the desired resolution and memory requirements, or to avoid the accumulation of a histogram or at least a finely resolved histogram directly successively certain statistical quantities such as the average, the average per bin or the number of individual light durations and the like are formed.
  • the evaluation may be limited to a partial area (ROI) that corresponds to forecasts, other previous knowledge or an assumption of an environment of the distance to be measured.
  • the individual light propagation times and / or the quantities obtained thereon are then evaluated together in a measured value block 32 in order ultimately to obtain the distance to the object.
  • ROI partial area
  • At least the receive path 16 is integrated on an ASIC.
  • own blocks for the light receiver 26 on the one hand and the evaluation circuits 28, 30, 32 on the other hand be provided.
  • at least the respective individual light transit time measuring unit 28 is arranged directly at the light receiver 26 and, in particular with individual avalanche photodiodes or groups thereof, forms intelligent pixels or a pixel-like evaluation.
  • the accumulator 30 and the measured value block 32 can also be integrated into these pixels.
  • a higher-level controller decides whether and how the measurement results of the pixels are used with spatial resolution or averaged again.
  • an FPGA Field Programmable Gate Array
  • a microprocessor is used, on which the accumulator 30 and / or the measured value block 32 and optionally also the single light transit time measuring unit 28 is implemented.
  • FIG. 1 shows only the relevant for the actual measurement components of the optoelectronic sensor 10.
  • the sensor 10 may be a simple button which measures the object distance on an axis and then outputs, for example, a continuous numerical value for the distance, or the sensor 10 acts as a switch whose switching state changes depending on the presence or absence of an object in a predeterminable distance range ,
  • the axis of the button can be rotated by appropriate rotating mirror or as a total rotating measuring head in rotation and then forms a Scanner.
  • Another exemplary embodiment of the sensor 10 is a 3D camera.
  • FIG. 2 shows by way of example an exemplary histogram of a plurality of individual light durations.
  • the bins on the X-axis are time intervals of possible light propagation times, here in arbitrary units and in high resolution, ie with a small bin width.
  • the Y-axis represents the associated number of detected individual light delays.
  • the histogram is a total of a distribution of the measured individual light durations.
  • the histogram shows a clear maximum, which is recognized approximately in the 270th bin with the naked eye and stands out clearly from the background of the single-light transit times caused by dark noise, stray light and other disturbing effects.
  • the maximum could be found with a threshold score and from this the distance of the object can be determined.
  • this requires significant memory for the high-resolution histogram, especially if one imagines that such a histogram would have to be stored in a 3D image sensor for each pixel.
  • the situation is in FIG. 2 very simple in that the maximum stands out very clearly from the background. This is by no means the case with a real measurement, in particular with weakly remittent or distant objects.
  • a special evaluation is performed, which is based on a model of the distribution of individual light durations of measurement and background events, which incorporates the special properties of SPADs.
  • This evaluation is preferably not done on the full, high-resolution histogram, which is shown in the first line for better understanding. Rather, the histogram is preferably detected only with a comparatively poor resolution, or only certain statistical variables are determined at all and the individual light durations themselves are discarded after their influence on these quantities has been evaluated.
  • the evaluation can be carried out with significantly less resources such as memory, computing power and bandwidth for data transmission.
  • FIG. 3 shows an exemplary course of the number of measurement and background events as a function of time. This corresponds in principle to the in FIG. 2 shown histogram. However, the distribution here is continuous and not discrete. Moreover, it is an idealized representation: although there are background events, but no noise in the sense that the background events occur at a constant rate. This constant rate is superimposed on the effect that an avalanche photodiode is in its dead time after the triggering of an avalanche and therefore can not be triggered again. Therefore, the number of events decreases exponentially over time. There are a large number of measurement events around the reception time t s and therefore a measurement peak. As a result, even fewer avalanche photodiodes are suddenly available, and in the remaining measurement period, the exponential decay continues due to background events at a correspondingly lower level.
  • FIG. 4 shows an associated time-dependent course of the number of avalanche photodiodes still available N ( t ), which decreases exponentially due to the background and decreases abruptly at the reception time t s . It is once again pointed out, as already mentioned several times, that the number of initially available avalanche photodiodes N ( t ) does not necessarily coincide with the number of physically present avalanche photodiodes, since there can be repeated measures. One could therefore also speak of a number of possible events or an event pool, but here the reference to still avalanche photodiodes still available is maintained with the concept understanding just explained.
  • the behavior of avalanche photodiodes during a measurement should now be recorded mathematically.
  • a time interval [ t 0 , t 1 ] is selected or specified, which includes the measuring peak and thus the receiving time t s to be determined.
  • Several measurement peaks can occur in one measurement period, for example in the case of semi-transparent objects or edge hits.
  • the time interval [ t 0 , t 1 ] can be determined by preliminary evaluation or a previous measurement be narrowed down. In an idealized observation with a constant background rate, the accuracy of the measurement would not depend on the time interval [ t 0 , t 1 ], the result would be independent of this. In practice, however, in the case of noisy individual time delays, the result becomes more accurate at a finer time interval [ t 0 , t 1 ].
  • the background parameter ⁇ can be determined in advance by a measurement. It can be ensured that only background events and no measurement events occur, be it by no light signals are emitted or the observation period is set so that there are no measurement events therein. With the means of order statistics, the background parameter ⁇ can also be extracted from a measurement with measurement events.
  • a second parameter p detects the strength of the signal pulse, ie how strong in FIG. 3 the measuring peak and thus in FIG. 4 the sudden drop of N ( t ) at the time of reception t s is pronounced.
  • the measurement peak is modeled as a Dirac pulse ⁇ ( t - t s ), weighted with the parameter p .
  • Another pulse shape such as a Gaussian distribution, would also be conceivable, but not only requires more complicated calculations, but also usually the description of the pulse with at least two parameters. This in turn leaves no simple solution in which the parameters are sequential and therefore particularly easy to determine.
  • the first term describes changes by background events
  • the second term changes by measurement events
  • the background parameter ⁇ is known, but the signal strength parameter p is not yet known.
  • N (t) and thus N ( t 0 ) can be calculated with knowledge of background parameters ⁇ and without knowledge of signal strength parameter p from the solution given above for N (t) .
  • a closed expression should also be found analytically for the time of reception t s .
  • another parameter is required because otherwise the model or the solution N (t) of the differential equation would remain under-determined.
  • the mean value ⁇ of the individual light propagation times in the time interval [ t 0 , t 1 ] is used.
  • the mean value ⁇ is practically very easy to estimate: It is checked for the individual light delays, in particular "on the fly", whether they are in the time interval [ t 0 , t 1 ]. If this is the case, the individual runtime is added to a sum and a counter is counted up, the mean ⁇ then being the quotient of the sum and the count.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Optical Radar Systems And Details Thereof (AREA)
EP18176901.9A 2017-06-21 2018-06-11 Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet Active EP3418766B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
DE102017113674.6A DE102017113674B4 (de) 2017-06-21 2017-06-21 Optoelektronischer Sensor und Verfahren zur Messung der Entfernung zu einem Objekt

Publications (2)

Publication Number Publication Date
EP3418766A1 true EP3418766A1 (fr) 2018-12-26
EP3418766B1 EP3418766B1 (fr) 2019-11-20

Family

ID=62597411

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18176901.9A Active EP3418766B1 (fr) 2017-06-21 2018-06-11 Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet

Country Status (4)

Country Link
US (1) US10948575B2 (fr)
EP (1) EP3418766B1 (fr)
CN (1) CN109100737B (fr)
DE (1) DE102017113674B4 (fr)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102017113675B4 (de) * 2017-06-21 2021-11-18 Sick Ag Optoelektronischer Sensor und Verfahren zur Messung der Entfernung zu einem Objekt
JP7043218B2 (ja) * 2017-10-26 2022-03-29 シャープ株式会社 光センサ、距離測定装置、および電子機器
DE102018129246B4 (de) * 2018-11-21 2020-10-15 Infineon Technologies Ag Interferenzdetektierung und -minderung für lidarsysteme
DE102019202459A1 (de) * 2019-02-22 2020-08-27 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Lasermessvorrichtung zum Messen von Entfernungen und Verfahren zum Betreiben einer Lasermessvorrichtung zum Messen von Entfernungen
JP7362265B2 (ja) * 2019-02-28 2023-10-17 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム
DE112020001783T5 (de) * 2019-04-02 2021-12-30 Ams International Ag Flugzeitsensor
JP7193413B2 (ja) * 2019-05-10 2022-12-20 株式会社東芝 電子装置及び距離計測方法
CN111999719B (zh) * 2019-05-10 2024-03-12 中国科学院半导体研究所 用于激光雷达的单光子tof图像传感器
CN110501687B (zh) * 2019-08-26 2021-08-10 哈尔滨工业大学 一种Gm-APD激光雷达目标有效探测的自适应光学口径调控方法
CN111337937A (zh) * 2020-04-22 2020-06-26 深圳市灵明光子科技有限公司 光电传感采集模组、光电传感测距方法以及测距装置
DE102020120858A1 (de) 2020-08-07 2022-02-10 Wenglor sensoric elektronische Geräte GmbH Verfahren sowie Messvorrichtung zur Bestimmung einer Distanz
CN113484870B (zh) * 2021-07-20 2024-05-14 Oppo广东移动通信有限公司 测距方法与装置、终端及非易失性计算机可读存储介质

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102007013714A1 (de) * 2007-03-22 2008-10-02 Sick Ag Optoelektronischer Sensor und Verfahren zur Messung einer Entfernung oder einer Entfernungsänderung
US20160033644A1 (en) * 2014-07-31 2016-02-04 Stmicroelectronics (Research & Development) Ltd. Time of flight determination

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102009029372A1 (de) * 2009-09-11 2011-03-24 Robert Bosch Gmbh Messvorrichtung zur Messung einer Entfernung zwischen der Messvorrichtung und einem Zielobjekt mit Hilfe optischer Messstrahlung
US8848171B2 (en) * 2009-12-22 2014-09-30 Leica Geosystems Ag Highly accurate distance measurement device
DE102010045657A1 (de) * 2010-09-17 2012-03-22 Wabco Gmbh Umfeld-Überwachungssystem für ein Fahrzeug
DE102010061382B4 (de) * 2010-12-21 2019-02-14 Sick Ag Optoelektronischer Sensor und Verfahren zur Erfassung und Abstandsbestimmung von Objekten
EP2817652B1 (fr) * 2012-02-21 2016-11-23 Karlsruher Institut für Technologie Procédé et système pour l'étalonnage de récepteur simultané et localisation pour multilatération
US9176361B2 (en) * 2014-03-11 2015-11-03 Sony Corporation Optical analog to digital converter and method
DE102017207317B4 (de) * 2017-05-02 2022-03-03 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Vorrichtung zur Ermittlung eines Abstands zu einem Objekt sowie entsprechendes Verfahren

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102007013714A1 (de) * 2007-03-22 2008-10-02 Sick Ag Optoelektronischer Sensor und Verfahren zur Messung einer Entfernung oder einer Entfernungsänderung
US20160033644A1 (en) * 2014-07-31 2016-02-04 Stmicroelectronics (Research & Development) Ltd. Time of flight determination

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
BEER MAIK ET AL: "Dead time effects in the indirect time-of-flight measurement with SPADs", 2017 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), IEEE, 28 May 2017 (2017-05-28), pages 1 - 4, XP033155984, DOI: 10.1109/ISCAS.2017.8050357 *
JANE X. LUU ET AL: "Saturation effects in heterodyne detection with Geiger-mode InGaAs avalanche photodiode detector arrays", APPLIED OPTICS, vol. 45, no. 16, 1 June 2006 (2006-06-01), pages 3798 - 3804, XP055505002 *
MILSTEIN A B ET AL: "Acquisition algorithm for direct-detection ladars with Geiger-mode avalanche photodiodes", APPLIED OPTICS, OPTICAL SOCIETY OF AMERICA, WASHINGTON, DC; US, vol. 47, no. 2, 10 January 2008 (2008-01-10), pages 286 - 311, XP001511081, ISSN: 0003-6935, DOI: 10.1364/AO.47.000296 *

Also Published As

Publication number Publication date
DE102017113674B4 (de) 2019-03-28
EP3418766B1 (fr) 2019-11-20
US10948575B2 (en) 2021-03-16
DE102017113674A1 (de) 2018-12-27
CN109100737A (zh) 2018-12-28
US20180372849A1 (en) 2018-12-27
CN109100737B (zh) 2022-07-08

Similar Documents

Publication Publication Date Title
EP3418766B1 (fr) Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet
EP3418767B1 (fr) Capteur optoélectronique et procédé de mesure de la distance par rapport à un objet
EP3185038B1 (fr) Capteur optoelectronique et procede de mesure d'un eloignement
DE102017207317B4 (de) Vorrichtung zur Ermittlung eines Abstands zu einem Objekt sowie entsprechendes Verfahren
EP3611535B1 (fr) Capture de lumière à l'aide d'une pluralité d'éléments de photodiode à avalanche
EP2910969B1 (fr) Capteur optoélectronique et procédé de saisie d'objets dans une zone de surveillance
EP2469296B1 (fr) Capteur optoélectronique et procédé destiné à la détection et la détermination de l'éloignement d'objets
EP1884797B1 (fr) Dispositif et procédé destinés à la détermination de distance
EP2486370B1 (fr) Télémètre optique avec dispositif d'étalonnage
EP3557286B1 (fr) Capteur optoélectronique et procédé de détection et de détermination de distance d'un objet
DE102010003843A1 (de) Entfernungsmessgerät mit homogenisierender Messauswertung
DE202013101039U1 (de) Optoelektronischer Sensor zur Entfernungsmessung
EP3671276B1 (fr) Capteur optoélectronique et procédé de détection d'un objet
EP3839556B1 (fr) Capteur optoélectronique et procédé de détection d'un objet
WO2018172258A1 (fr) Système lidar à base de spad (photodiodes avalanche à photon unique)
DE202014007924U9 (de) Sensor mit Hintergrundausblendung
DE102010064682B3 (de) Optoelektronischer Sensor und Verfahren zur Erfassung und Abstandsbestimmung von Objekten
DE102011010102B4 (de) Verfahren zum Messen einer Entfernung und Vorrichtung zur Durchführung des Verfahrens
DE102018205369A1 (de) Verfahren zur Verarbeitung von Messdaten eines LIDAR Messsystems

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190328

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G01S 17/89 20060101ALI20190528BHEP

Ipc: G01S 17/10 20060101ALI20190528BHEP

Ipc: G01S 7/497 20060101ALI20190528BHEP

Ipc: G01S 7/487 20060101ALI20190528BHEP

Ipc: G01S 7/486 20060101AFI20190528BHEP

INTG Intention to grant announced

Effective date: 20190701

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502018000391

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1204800

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191215

RAP2 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: SICK AG

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20191120

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200220

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200221

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200220

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200320

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200412

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502018000391

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20200821

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200611

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200630

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210630

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191120

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20220611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220611

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230620

Year of fee payment: 6

Ref country code: DE

Payment date: 20230620

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: AT

Payment date: 20230616

Year of fee payment: 6

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20230630

Year of fee payment: 6