EP2251849B1 - Method for improved recognition of conduit-type objects - Google Patents

Method for improved recognition of conduit-type objects Download PDF

Info

Publication number
EP2251849B1
EP2251849B1 EP10002260.7A EP10002260A EP2251849B1 EP 2251849 B1 EP2251849 B1 EP 2251849B1 EP 10002260 A EP10002260 A EP 10002260A EP 2251849 B1 EP2251849 B1 EP 2251849B1
Authority
EP
European Patent Office
Prior art keywords
measurement points
straight line
profiles
matrix
line
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Not-in-force
Application number
EP10002260.7A
Other languages
German (de)
French (fr)
Other versions
EP2251849A2 (en
EP2251849A3 (en
Inventor
Michael Hoyer
Oliver Ruebsamen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hensoldt Sensors GmbH
Original Assignee
Airbus DS Electronics and Border Security GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Airbus DS Electronics and Border Security GmbH filed Critical Airbus DS Electronics and Border Security GmbH
Publication of EP2251849A2 publication Critical patent/EP2251849A2/en
Publication of EP2251849A3 publication Critical patent/EP2251849A3/en
Application granted granted Critical
Publication of EP2251849B1 publication Critical patent/EP2251849B1/en
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G5/00Traffic control systems for aircraft, e.g. air-traffic control [ATC]
    • G08G5/0073Surveillance aids
    • G08G5/0086Surveillance aids for monitoring terrain

Definitions

  • the invention relates to a method for improved detection of line-type objects according to the features of patent claim 1.
  • the present method is suitable for low-flying aircraft to avoid collisions with power lines and other line-type objects such as e.g. Guy wires of masts, cable cars, etc.
  • a method for obstacle warning for low-flying aircraft In this case, distance images of the environment in front of the aircraft are generated by means of a sensor. Obstacle contours are extracted from the distance images by searching for discontinuities ("hops") between adjacent image pixels by means of high-pass filters. With a navigation system, the location of the Hindemisk correction can be superimposed graphically on the natural external view. A recognition of the obstacle in the true sense does not take place.
  • DE 10 2005 047 273 B4 Another method for supporting low-level flights is described, which is based on the examination of jumps in the recorded distance images. In this method, wire-like obstacles are detected in strong environmental conditions, such as clutter or when looking at the sky.
  • US 2008 / 0007708A1 also describes a method for line detection in which a Hough transformation is used.
  • a Hough transformation By means of a rangefinder distance images are taken during the flight and potential images in which lines could occur selected. Subsequently, a projection of the potential line measuring points in a horizontal Level. In a subsequent Hough transformation and subsequent shape transformation, the identification of straight line pieces takes place.
  • the object of the invention is to provide a method for improved line detection, with which lines are detected reliably and in real time.
  • a set of three-dimensional measurement points in a geodesic coordinate system are generated from the distance values of a distance image generated by a distance sensor of this environment, taking into account the position and location of the aircraft. From this set of measurement points, potential line measurement points are extracted using known filter methods.
  • the potential line measurement points are projected into a horizontal plane and identified by means of a polynomial phase transformation and a subsequent spectral analysis of line progressions.
  • the potential line measurement pixels are projected on and at a predeterminable distance from a straight line found in the first step into a vertical plane to the horizontal plane and the respective straight line from the first step, and catenoid or parabolic courses are identified by means of a quadratic shape transformation.
  • a mathematical analogy is established between a line fit and the determination of the spectral parameters of a signal having a plurality of components.
  • an assignment of suitable measuring points i. which are within a defined corridor, i. having a predetermined distance to the estimated in the first step straight line, instead.
  • the evaluation in the second method step according to the invention is generalized to the identification of catenoids.
  • the identification can also be done on parabolas.
  • the result of an evaluation of a distance image (data set) can be compared with the results of the temporally advanced images in addition to the confirmation.
  • the position of the lines relative to one another can be investigated on the basis of the parameterization found by the polynomial phase transformation, and implausible line courses can be eliminated.
  • the attitude relative to the extrapolated flight trajectory of the missile can be examined to generate a warning in critical cases.
  • the pilot can be informed, for example, about the distance, height and direction relative to the helicopter axis or flight vector.
  • the line can be drawn as a symbol, cloud of measurement point or katenoid based on the found parameterization as a video or FLIR overlay or in a digital map.
  • the sensor selected is an imaging front-end which actively scans the scene and whose image information consists of distance values. Distance images are advantageous for the subsequent automatic image evaluation for line detection.
  • the sensors used are a laser scanner, which preferably operates in the infrared range, or an imaging radar in the mm-wave range. Although the radar has certain advantages in terms of bad weather conditions (fog), according to the current state of the art, only one laser scanner has the desired vertical and lateral image resolution. Both sensors are equally suitable as active systems for day and night operation.
  • a suitable laser scanner is designed for long ranges (> 1km) and high image resolution (eg 0.5 horizontal * 0.1 degrees vertical). For each distance image, the spatial direction into which the laser measurement pulse was emitted must be clearly determinable in the sensor-oriented coordinate system, by the design of the scanner and possibly by additional measurements during the scanning process.
  • the navigation system is used to determine the position and position of the aircraft.
  • Each distance value is converted into a scene point using the coordinates of the associated spatial scanning direction and the current position and attitude parameters of the aircraft.
  • a scene point (measuring point) is defined by its three spatial coordinates in a terrestrial coordinate system. From each distance image, a so-called earth-fixed measuring point cloud is formed in this way, from which possibly existing lines can be extracted.
  • Data processing is handled by a mid-power processor. On this, the above-mentioned transformation of distance measurement values into measurement points is first carried out.
  • the resulting data serves as input for an evaluation unit.
  • the evaluation unit determines whether the individual lines are in dangerous proximity to the current position of the aircraft or the intended trajectory.
  • the result of this evaluation is communicated visually and / or acoustically to the pilot.
  • an imaging front-end can be used, which actively scans the surroundings of the aircraft and whose image information consists of distance values. The resulting distance images are used in the subsequent process steps for line detection.
  • Suitable sensors are a laser scanner or an imaging radar in the mm-wave range. In contrast to the radar, which has certain advantages with regard to bad weather operation (fog), a laser scanner has a comparatively better vertical and lateral image resolution. Both sensors are equally suitable as active systems for day and night operation.
  • a suitable laser scanner is designed for long ranges (> 1km) and high image resolution (eg 0.5 horizontal * 0.1 degrees vertical). For each distance image is determined by the design of the scanner and by any additional measurements during the scan, the spatial direction in which the laser measuring pulse was emitted clearly in the sensor-oriented coordinate system.
  • the navigation system is used to determine the position and position of the aircraft.
  • Each distance value is converted to a measuring point using the coordinates of the associated spatial scanning direction and the current position and attitude parameters of the aircraft.
  • a measuring point is defined by its three spatial coordinates in a geodetic coordinate system. From each distance image arises in this way a so-called earth-proof measuring point cloud from which existing lines can be extracted.
  • Data processing is handled by a mid-power processor. On this a transformation of the distance measurement values into measuring points is carried out. The resulting data serves as input for an evaluation unit. The evaluation unit determines whether the individual lines are in dangerous proximity to the current position of the aircraft or the intended trajectory The result of this evaluation is communicated visually and / or acoustically to the pilot.
  • the starting point for the PPT used in the invention is a reduced amount of three-dimensional measuring points.
  • the reduced quantity is due to the fact that only such measured values are subjected to a PPT which has not been eliminated by clutter or interference filters.
  • a first method step of the invention only the horizontal (x, y) component of the measured values is evaluated in order to identify the straight line courses in this point set. Horizontal lines are potential directions of passage of lines.
  • all vertical planes belonging to the straight lines found here are examined for catenoid pieces or parabolic pieces [Katenoide: cosh (x)]. That is, all measurement points lying in or near such a plane are projected into that plane, and the resulting set of two-dimensional measurement points is examined for catenoids using quadratic shape transformation (QFT).
  • QFT quadratic shape transformation
  • Polynomial phase transformation (polynomial phase transform - PPT)
  • the invention is based on the so-called polynomial phase transform.
  • the problem of the line detection according to the invention is converted into a problem of the frequency determination of a sinusoidal signal or the spectral analysis of a frequency mixture in the case of multiple lines.
  • known signal processing methods are available (Fourier Transformation, Matrix Pencil, FFT, Wavelets, Esprit, Pisarenco, MUSIC etc.) where the matrix Pencil has proven to be particularly efficient and will be explained in detail below.
  • the frequencies ⁇ j are determined according to the invention.
  • these include the Fourier transform (FT), the Maximum Likelihood Method (MLM), the least squares method (LSM) and the Matrix Pencil Method (MPM).
  • FT Fourier transform
  • MLM Maximum Likelihood Method
  • LSM least squares method
  • MPM Matrix Pencil Method
  • the matrix pencil method is used.
  • the other methods for frequency determination can be used.
  • T means transposed and L is a chosen parameter, the so-called pencil parameter, with d ⁇ L ⁇ N - d.
  • the continuity of each element of the truncated pseudoinverse is Y 0 + also at the point where the noise equals 0, and thus also the continuity of the z r 's.
  • Y 0 + is identical to X 0 + if and only if the noise is zero.
  • Y 0 + Y 1 If LM has singular values equal to zero that do not contain information about the z r 's, one can reduce the size of the matrix before calculating the singular values.
  • the singular values z r are the M eigenvalues of Z E * which are the same as the M non-zero singular values of Y 0 + Y 1 ,
  • Quadratic Form Transform (QFT)
  • the pixels of the line are transformed into the front view. In other words, you no longer look at the pixels of the straight line from a bird's eye view. Instead, one uses the first found pixel of the straight as the zero point of a new coordinate system, which uses the found straight line as one of the new axes, and the z axis as the second new axis. Thus, one obtains a graph in which the found straight should have catenoid shape if it corresponds to a high voltage line.
  • the transformation PPT is based on a polynomial of degree 1.
  • ⁇ # 0 expediently ⁇ is set to 1.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Processing (AREA)

Description

Die Erfindung betrifft ein Verfahren zur verbesserten Erkennung von leitungsartigen Objekten gemäß den Merkmalen des Patentanspruchs 1.The invention relates to a method for improved detection of line-type objects according to the features of patent claim 1.

Das vorliegende Verfahren ist geeignet für tief fliegende Fluggeräte zur Vermeidung von Kollisionen mit Hochspannungsleitungen und anderen leitungsartigen Objekten wie z.B. Abspanndrähte von Masten, Seilbahnen etc.The present method is suitable for low-flying aircraft to avoid collisions with power lines and other line-type objects such as e.g. Guy wires of masts, cable cars, etc.

In DE 196 05 218 C1 ist ein Verfahren zur Hinderniswarnung für tief fliegende Fluggeräte beschrieben. Dabei werden mittels eines Sensors Entfernungsbilder der vor dem Fluggerät sich befindenden Umgebung erzeugt. Aus den Entfernungsbildern werden Hinderniskonturen extrahiert, indem zwischen benachbarten Bildpixeln mittels Hochpassfiltern nach Diskontinuitäten ("Sprüngen") gesucht wird. Mit einem Navigationssystem kann die Lage der Hindemiskorrektur der natürlichen Außensicht graphisch überlagert werden. Eine Erkennung des Hindernisses im eigentlichen Sinn findet nicht statt. In DE 10 2005 047 273 B4 wird ein weiteres Verfahren zur Unterstützung von Tiefflügen beschrieben, welches auf der Untersuchung von Sprüngen in den aufgenommenen Entfernungsbildern basiert. Bei diesem Verfahren werden drahtähnliche Hindernisse bei starken Umgebungseinflüssen, z.B. Clutter oder beim Blick gegen den Himmel erkannt.In DE 196 05 218 C1 a method is described for obstacle warning for low-flying aircraft. In this case, distance images of the environment in front of the aircraft are generated by means of a sensor. Obstacle contours are extracted from the distance images by searching for discontinuities ("hops") between adjacent image pixels by means of high-pass filters. With a navigation system, the location of the Hindemisk correction can be superimposed graphically on the natural external view. A recognition of the obstacle in the true sense does not take place. In DE 10 2005 047 273 B4 Another method for supporting low-level flights is described, which is based on the examination of jumps in the recorded distance images. In this method, wire-like obstacles are detected in strong environmental conditions, such as clutter or when looking at the sky.

In DE 100 55 572 C1 ist ein Verfahren zur Leitungserkennung beschrieben welches mit Einsatz einer Hough Transformation arbeitet. Die Hough Transformation ist bekanntermaßen ein äußerst rechenintensives Verfahren da im Prinzip sämtliche theoretisch denkbaren Lösungen auch gerechnet werden müssen. Es ergeben sich dabei szenenabhängig drastisch unterschiedliche Verarbeitungszeiten.In DE 100 55 572 C1 is a method for line detection described which works with the use of a Hough transformation. The Hough transformation is known to be a very computationally intensive process because in principle all theoretically conceivable solutions must also be expected. Depending on the scene, drastically different processing times result.

US 2008/0007708A1 beschreibt ebenfalls ein Verfahren zur Leitungserkennung, bei welchem eine Hough-Transformation zum Einsatz kommt. Mittels eines Entfernungsmessers werden während des Fluges Entfernungsbilder aufgenommen und potentielle Bilder, in welchem Leitungen vorkommen könnten selektiert. Anschließend erfolgt eine Projektion der potentiellen Leitungsmesspunkte in eine horizontale Ebene. In einer nachfolgenden Hough-Transformation und anschließenden Formtransformation erfolgt die Identifizierung von Geradenstücken. US 2008 / 0007708A1 also describes a method for line detection in which a Hough transformation is used. By means of a rangefinder distance images are taken during the flight and potential images in which lines could occur selected. Subsequently, a projection of the potential line measuring points in a horizontal Level. In a subsequent Hough transformation and subsequent shape transformation, the identification of straight line pieces takes place.

Abed-Meraim, K.; Beghdadi, A., "Multi-line fitting using polynomial phase transforms and downsampling," in Acoustics, Speech, and Signal Processing, 2001. Proceedings. (ICASSP '01). 2001 IEEE International Conference on , vol.3, no., pp.1701-1704 vol.3, 2001 doi: 10.1109/ICASSP.2001.941266 beschreibt ein Signalverarbeitungsverfahren zur Schätzung von Parametern paralleler Linien in zwei dimensionalen digitalen Bildern mit unterschiedlichem Versatz, wobei die Parameterschätzung über eine polynominale Phasentransformation gelöst wird. Abed-Meraim, K .; Beghdadi, A., "Multi-line fitting using polynomial phase transforms and downsampling," in Acoustics, Speech, and Signal Processing, 2001. Proceedings. (ICASSP '01). 2001 IEEE International Conference on, vol. 3, no., Pp.1701-1704 vol.3, 2001 doi: 10.1109 / ICASSP.2001.941266 describes a signal processing method for estimating parallel line parameters in two-dimensional differential-offset digital images, the parameter estimation being solved by a polynomial phase transformation.

Aufgabe der Erfindung ist es, ein Verfahren zur verbesserten Leitungserkennung anzugeben, mit welchem Leitungen zuverlässig und in Echtzeit erkannt werden.The object of the invention is to provide a method for improved line detection, with which lines are detected reliably and in real time.

Diese Aufgabe wird mit dem Verfahren gemäß den Merkmalen des geltenden Anspruchs 1 gelöst. Vorteilhafte Ausführungen der Erfindung sind Gegenstand von Unteransprüchen.This object is achieved by the method according to the features of the current claim 1. Advantageous embodiments of the invention are the subject of dependent claims.

Zur verbesserten Erkennung von leitungsartigen Objekten in einer vor einem Fluggerät befindlichen Umgebung werden aus den Entfernungswerten eines mittels eines Entfernungssensors erzeugten Entfernungsbildes dieser Umgebung unter Berücksichtigung der Position und Lage des Fluggeräts eine Menge von dreidimensionalen Messpunkten in einem geodätischen Koordinatensystem erzeugt. Aus dieser Menge von Messpunkten werden unter Verwendung bekannter Filtermethoden potentielle Leitungsmesspunkte extrahiert. In einem ersten erfindungsgemäßen Schritt werden die potentiellen Leitungsmesspunkte in eine horizontate Ebene projiziert und mittels einer polynomialen Phasentransformation und einer nachfolgenden spektralen Analyse Geradenverläufe identifiziert. In einem zweiten erfindungsgemäßen Schritt werden die potentiellen Leitungsmesspixel auf und in einem vorgebbaren Abstand zu einer im ersten Schritt gefundenen Geraden in eine vertikale Ebene zu der horizontalen Ebene und der jeweiligen Geraden aus dem ersten Schritt projiziert und mittels einer quadratischen Formtransformation Katenoiden- oder Parabelverläufe identifiziert.For improved detection of line-like objects in an aircraft environment, a set of three-dimensional measurement points in a geodesic coordinate system are generated from the distance values of a distance image generated by a distance sensor of this environment, taking into account the position and location of the aircraft. From this set of measurement points, potential line measurement points are extracted using known filter methods. In a first step according to the invention, the potential line measurement points are projected into a horizontal plane and identified by means of a polynomial phase transformation and a subsequent spectral analysis of line progressions. In a second step according to the invention, the potential line measurement pixels are projected on and at a predeterminable distance from a straight line found in the first step into a vertical plane to the horizontal plane and the respective straight line from the first step, and catenoid or parabolic courses are identified by means of a quadratic shape transformation.

Erfindungsgemäß erfolgt die horizontale Projektion der potentiellen Leitungsmesspunkte im ersten Schritt auf eine NxM-Matrix gemäß R = q 1 1 q M 1 q 1 N q M N mit q i k 0 1 , wobei i = 1 , .. , M und k = 1 , .. , N ist .

Figure imgb0001
According to the invention, the horizontal projection of the potential line measuring points in the first step is carried out in accordance with an NxM matrix R = q 1 1 ... q M 1 q 1 N ... q M N with q i k 0 1 . where i = 1 . .. . M and k = 1 . .. . N is ,
Figure imgb0001

Die Geradenverläufe werden mittels einer polynomialen Phasentransformation und einer nachfolgenden spektralen Analyse identifiziert, wobei in der polynomialen Phasentransformation die potentiellen Leitungsmesspunkte von der Matrix R in ein sinusförmiges Signal z(k) der Form z k = j = 1 d e i μ 1 k tanϕ j + x j

Figure imgb0002
mit

j:
Variable zwischen 1 und d
d:
Zahl der Geraden
µ1:
konstanter Parameter zwischen 0,01 bis 1
xj:
Achsenabschnitt der j.ten Gerade
ϕj
Winkel der j.ten Gerade
transformiert werden und in der spektralen Analyse die Frequenzen fj=µ1*tan ϕj bestimmt werden.The line paths are identified by means of a polynomial phase transformation and a subsequent spectral analysis, wherein in the polynomial phase transformation the potential line measurement points from the matrix R into a sinusoidal signal z (k) of the form z k = Σ j = 1 d e i μ 1 k tanφ j + x j
Figure imgb0002
With
j:
Variable between 1 and d
d:
Number of straight lines
μ 1 :
constant parameter between 0.01 to 1
xj :
Intersection of the jth straight
φ j
Angle of the jth straight
be transformed and in the spectral analysis, the frequencies fj = μ 1 * tan φ j are determined.

Bei den beschriebenen bekannten Filtermethoden handelt es sich um Filter, welche Clutter und Störungen, z.B. durch Sonneneinstrahlung eliminieren. Solche Filter werden z.B. in DE 10 2005 047 273 B4 oder DE 198 28 318 C2 beschrieben. Messpunkte welehe in diesen Filtermethoden als Glutter oder Störung identifiziert werden, werden in dem ersten und zweiten erfindungsgemäßen Verfahrensschritt nicht berücksichtigt. Dadurch wird die Rechenzeit erheblich reduziert.In the known filter methods described are filters that eliminate clutter and interference, for example by sunlight. Such filters are eg in DE 10 2005 047 273 B4 or DE 198 28 318 C2 described. Measurement points which are identified as glutter or interference in these filter methods are not taken into account in the first and second method steps according to the invention. This considerably reduces the computing time.

Die den potentiellen Leitungsmesspunkten entsprechenden Pixel werden in eine horizontale Ebene projiziert in welcher Leitungen als Geradenstücke erscheinen. Es gilt nun mit einem geeigneten Verfahren diese Geradenstücke zu erkennen.The pixels corresponding to the potential line measurement points are projected into a horizontal plane in which lines appear as line segments. It now applies a suitable method to recognize these straight line pieces.

Wegen der bekannten Nachteile kommt hier nicht die Hough Transformation zum Einsatz.Because of the known disadvantages here is not the Hough transformation used.

In dem erfindungsgemäßen Verfahren wird eine mathematische Analogie zwischen einem Linienfit und der Bestimmung der spektralen Parameter eines Signals mit mehreren Komponenten hergestellt. Für die spektrale Analyse existieren effiziente Verfahren in der digitalen Signalverarbeitung.In the method according to the invention, a mathematical analogy is established between a line fit and the determination of the spectral parameters of a signal having a plurality of components. For spectral analysis, there are efficient methods in digital signal processing.

In einem erfindungsgemäßen Verfahrensschritt findet eine Zuordnung geeigneter Messpunkte, d.h. welche sich innerhalb eines definierten Korridors, d.h. einen vorgegebenen Abstand aufweisend, um die im ersten Verfahrensschritt abgeschätzten Gerade befinden, statt.In a method step according to the invention, an assignment of suitable measuring points, i. which are within a defined corridor, i. having a predetermined distance to the estimated in the first step straight line, instead.

Die Auswertung im zweiten erfindungsgemäßen Verfahrensschritt erfolgt verallgemeinert auf die Identifizierung von Katenoiden. Als Näherung zur Rechenzeitersparnis kann die Identifizierung auch auf Parabeln erfolgen.The evaluation in the second method step according to the invention is generalized to the identification of catenoids. As an approximation to the computational time savings, the identification can also be done on parabolas.

Um eine niedrige Falschalarmrate zu erreichen, kann das Ergebnis einer Auswertung eines Entfernungsbildes (Datensatz) zusätzlich zur Bestätigung mit den Ergebnissen der zeitlich voran gegangenen Bilder verglichen werden.In order to achieve a low false alarm rate, the result of an evaluation of a distance image (data set) can be compared with the results of the temporally advanced images in addition to the confirmation.

Es können zur Erhöhung der Verfahrenssicherheit auch mehrere Datensätze zur Auswertung akkumuliert werden.It can be accumulated to increase the process safety and several records for evaluation.

Werden mehrere Leitungen gefunden können anhand der durch die polynomiale Phasentransformation gefundene Parametrisierung die Lage der Leitungen relativ zueinander untersucht werden und unplausible Leitungsverläufe eliminiert werden.If several lines are found, the position of the lines relative to one another can be investigated on the basis of the parameterization found by the polynomial phase transformation, and implausible line courses can be eliminated.

Ist eine Leitung bestätigt, kann die Lage relativ zur extrapolierten Flugtrajektorie des Flugkörpers untersucht werden, um in kritischen Fällen eine Warnung zu erzeugen.If a line is confirmed, the attitude relative to the extrapolated flight trajectory of the missile can be examined to generate a warning in critical cases.

Aufgrund der bekannten Lageinformation kann der Pilot z.B. über Entfernung, Höhe und Richtung relativ zur Hubschrauberachse oder Flugvektor informiert werden.Due to the known position information, the pilot can be informed, for example, about the distance, height and direction relative to the helicopter axis or flight vector.

Die Leitung kann als Symbol, Messpunktwolke oder Katenoide anhand der gefundenen Parametrisierung als Video- oder FLIR Overlay oder in eine digitale Karte eingezeichnet werden.The line can be drawn as a symbol, cloud of measurement point or katenoid based on the found parameterization as a video or FLIR overlay or in a digital map.

Sensorsystemsensor system

Als Sensor wird ein bildgebendes Front-End gewählt, welches die Szene aktiv abtastet und dessen Bildinformation aus Entfernungswerten besteht. Entfernungsbilder sind für die nachfolgende automatische Bildauswertung zur Leitungserkennung vorteilhaft. Als Sensoren kommen ein Laserscanner, der Vorzugsweise im Infrarotbereich arbeitet, oder ein abbildendes Radar im mm-Wellenbereich in Frage. Obwohl das Radar im Hinblick auf Schlechtwetterbetrieb (Nebel) gewisse Vorteile besitzt, hat nach aktuellem Stand der Technik nur ein Laserscanner die gewünschte vertikale und laterale Bildauflösung. Beide Sensoren sind als aktive Systeme gleichermaßen für Tag und Nachtbetrieb geeignet.
Ein geeigneter Laserscanner ist für große Reichweiten (>1km) und hohe Bildauflösung (z.B. 0,5 horizontal* 0,1 Grad vertikal) ausgelegt. Für jedes Entfernungsbild muss die räumliche Richtung, in die der Lasermesspuls ausgesandt wurde, eindeutig im sensorfesten Koordinatensystem bestimmbar sein, durch die Bauart des Scanners sowie evtl. durch zusätzliche Messungen während des Scanvorgangs.
The sensor selected is an imaging front-end which actively scans the scene and whose image information consists of distance values. Distance images are advantageous for the subsequent automatic image evaluation for line detection. The sensors used are a laser scanner, which preferably operates in the infrared range, or an imaging radar in the mm-wave range. Although the radar has certain advantages in terms of bad weather conditions (fog), according to the current state of the art, only one laser scanner has the desired vertical and lateral image resolution. Both sensors are equally suitable as active systems for day and night operation.
A suitable laser scanner is designed for long ranges (> 1km) and high image resolution (eg 0.5 horizontal * 0.1 degrees vertical). For each distance image, the spatial direction into which the laser measurement pulse was emitted must be clearly determinable in the sensor-oriented coordinate system, by the design of the scanner and possibly by additional measurements during the scanning process.

Das Navigationssystem dient zur Bestimmung von Position und Lage des Fluggeräts. Jeder Entfernungswert wird mit Hilfe der Koordinaten der zugehörigen räumlichen Scanrichtung und den aktuellen Positions- und Lageparametern des Fluggeräts in einen Szenenpunkt umgerechnet. Ein Szenenpunkt (Messpunkt) ist durch seine drei Ortskoordinaten in einem erdfesten Koordinatensystem festgelegt. Aus jedem Entfernungsbild entsteht auf diese Weise eine sog. erdfeste Messpunktwolke, aus der evtl. vorhandene Leitungen extrahiert werden können.The navigation system is used to determine the position and position of the aircraft. Each distance value is converted into a scene point using the coordinates of the associated spatial scanning direction and the current position and attitude parameters of the aircraft. A scene point (measuring point) is defined by its three spatial coordinates in a terrestrial coordinate system. From each distance image, a so-called earth-fixed measuring point cloud is formed in this way, from which possibly existing lines can be extracted.

Für die Datenverarbeitung ist ein Prozessor mittlerer Leistung zuständig. Auf diesem wird zunächst die oben genannte Transformation von Entfernungsmesswerten in Messpunkte durchgeführt.Data processing is handled by a mid-power processor. On this, the above-mentioned transformation of distance measurement values into measurement points is first carried out.

Die sich daraus ergebenen Daten dienen als Input für eine Auswerteeinheit. Die Auswerteeinheit stellt fest, ob sich die einzelnen Leitungen in gefährlicher Nähe zur aktuellen Position des Fluggeräts oder zur vorgesehenen Flugbahn befinden. Das Ergebnis dieser Auswertung wird dem Piloten optisch und/oder akustisch mitgeteilt.The resulting data serves as input for an evaluation unit. The evaluation unit determines whether the individual lines are in dangerous proximity to the current position of the aircraft or the intended trajectory. The result of this evaluation is communicated visually and / or acoustically to the pilot.

Als Sensor kann ein bildgebendes Front-End verwendet werden, welches die Umgebung des Fluggeräts aktiv abtastet und dessen Bildinformation aus Entfernungswerten besteht. Die daraus entstehenden Entfernungsbilder werden in den nachfolgenden Verfahrensschritten zur Leitungserkennung verwendet. Als Sensoren kommen ein Laserscanner oder ein abbildendes Radar im mm-Wellenbereich in Frage. Im Gegensatz zum Radar, welches im Hinblick auf Schlechtwetterbetrieb (Nebel) gewisse Vorteile besitzt, hat ein Laserscanner eine vergleichsweise bessere vertikale und laterale Bildauflösung. Beide Sensoren sind als aktive Systeme gleichermaßen für Tag und Nachtbetrieb geeignet.
Ein geeigneter Laserscanner ist für große Reichweiten (>1km) und hohe Bildauflösung (z.B. 0,5 horizontal* 0,1 Grad vertikal) ausgelegt. Für jedes Entfernungsbild ist, durch die Bauart des Scanners sowie durch etwaige zusätzliche Messungen während des Scanvorgangs, die räumliche Richtung in die der Lasermesspuls ausgesandt wurde eindeutig im sensorfesten Koordinatensystem bestimmbar.
As a sensor, an imaging front-end can be used, which actively scans the surroundings of the aircraft and whose image information consists of distance values. The resulting distance images are used in the subsequent process steps for line detection. Suitable sensors are a laser scanner or an imaging radar in the mm-wave range. In contrast to the radar, which has certain advantages with regard to bad weather operation (fog), a laser scanner has a comparatively better vertical and lateral image resolution. Both sensors are equally suitable as active systems for day and night operation.
A suitable laser scanner is designed for long ranges (> 1km) and high image resolution (eg 0.5 horizontal * 0.1 degrees vertical). For each distance image is determined by the design of the scanner and by any additional measurements during the scan, the spatial direction in which the laser measuring pulse was emitted clearly in the sensor-oriented coordinate system.

Das Navigationssystem dient zur Bestimmung von Position und Lage des Fluggeräts. Jeder Entfernungswert wird mit Hilfe der Koordinaten der zugehörigen räumlichen Scanrichtung und den aktuellen Positions- und Lageparametern des Fluggeräts in einen Messpunkt umgerechnet. Ein Messpunkt ist durch seine drei Ortskoordinaten in einem geodätischen Koordinatensystem festgelegt. Aus jedem Entfernungsbild entsteht auf diese Weise eine so genante erdfeste Messpunktwolke, aus der vorhandene Leitungen extrahiert werden können.The navigation system is used to determine the position and position of the aircraft. Each distance value is converted to a measuring point using the coordinates of the associated spatial scanning direction and the current position and attitude parameters of the aircraft. A measuring point is defined by its three spatial coordinates in a geodetic coordinate system. From each distance image arises in this way a so-called earth-proof measuring point cloud from which existing lines can be extracted.

Für die Datenverarbeitung ist ein Prozessor mittlerer Leistung zuständig. Auf diesem wird eine Transformation der Entfernungsmesswerte in Messpunkte durchgeführt. Die sich daraus ergebenen Daten dienen als Input für eine Auswerteeinheit. Die Auswerteeinheit stellt fest, ob sich die einzelnen Leitungen in gefährlicher Nähe zur aktuellen Position des Fluggeräts oder zur vorgesehenen Flugbahn Das Ergebnis dieser Auswertung wird dem Piloten optisch und/oder akustisch mitgeteilt.Data processing is handled by a mid-power processor. On this a transformation of the distance measurement values into measuring points is carried out. The resulting data serves as input for an evaluation unit. The evaluation unit determines whether the individual lines are in dangerous proximity to the current position of the aircraft or the intended trajectory The result of this evaluation is communicated visually and / or acoustically to the pilot.

Ausgangspunkt für die in der Erfindung verwendete PPT ist eine reduzierte Menge von dreidimensionalen Messpunkten. Die reduzierte Menge begründet sich dadurch, dass lediglich solche Messwerte einer PPT unterzogen werden, welche nicht durch Clutter- oder Störungsfilter eliminiert wurden.The starting point for the PPT used in the invention is a reduced amount of three-dimensional measuring points. The reduced quantity is due to the fact that only such measured values are subjected to a PPT which has not been eliminated by clutter or interference filters.

In einem ersten Verfahrensschritt der Erfindung wird nur die horizontale (x,y)-Komponente der Messwerte ausgewertet, um die Geradenverläufe in dieser Punktmenge zu identifizieren. Horizontale Geraden sind potentielle Verlaufsrichtungen von Leitungen. In einem zweiten Verfahrensschritt der Erfindung werden alle vertikalen Ebenen, die zu den hier gefundenen Geraden gehören, auf Katenoidenstücke oder Parabelstücke untersucht [Katenoide: cosh(x)]. Das heißt, es werden alle Messpunkte, die in oder nahe bei einer soichen Ebene liegen, in diese Ebene projiziert, und der resultierende Satz von zweidimensionalen Messpunkten wird mittels Anwendung einer quadratischen Formtransformation (QFT) auf Katenoiden bzw. Parabeln untersucht.In a first method step of the invention, only the horizontal (x, y) component of the measured values is evaluated in order to identify the straight line courses in this point set. Horizontal lines are potential directions of passage of lines. In a second method step of the invention, all vertical planes belonging to the straight lines found here are examined for catenoid pieces or parabolic pieces [Katenoide: cosh (x)]. That is, all measurement points lying in or near such a plane are projected into that plane, and the resulting set of two-dimensional measurement points is examined for catenoids using quadratic shape transformation (QFT).

Polynomiale Phasentransformation (Polynomial Phase Transform - PPT)Polynomial phase transformation (polynomial phase transform - PPT)

Die Erfindung basiert auf der so genannten Polynomial Phase Transform. Darin wird das Problem der Geradenerkennung erfindungsgemäß in ein Problem der Frequenzbestimmung eines sinusförmigen Signals überführt bzw. der spektralen Analyse eines Frequenzgemisches im Falle mehrerer Geraden. Hierfür stehen bekannte Signalverarbeitungsverfahren zur Verfügung (Fourier Transformation, Matrix Pencil, FFT, Wavelets, Esprit, Pisarenco, MUSIC etc.) wobei sich der Matrix Pencil als besonders effizient erwiesen hat und im Weiteren im Detail erläutert wird.The invention is based on the so-called polynomial phase transform. Therein, the problem of the line detection according to the invention is converted into a problem of the frequency determination of a sinusoidal signal or the spectral analysis of a frequency mixture in the case of multiple lines. For this purpose, known signal processing methods are available (Fourier Transformation, Matrix Pencil, FFT, Wavelets, Esprit, Pisarenco, MUSIC etc.) where the matrix Pencil has proven to be particularly efficient and will be explained in detail below.

Im Folgenden wird zunächst kurz die polynomiale Phasentransformation erläutert:

  • Die horizontale Projektion der potentiellen Leitungspixel wird erfindungsgemäß auf eine NxM-Matrix abgebildet. Die Elemente dieser Matrix haben den Wert 1 wenn sie ein potentielles
  • Leitungspixel repräsentieren, ansonsten den Wert 0. Die Matrix entspricht also einem diskretisierten Binärbild mit Draufsicht von oben.
The following briefly explains the polynomial phase transformation:
  • The horizontal projection of the potential line pixels is mapped onto an NxM matrix according to the invention . The elements of this matrix have the value 1 if they are a potential one
  • Line pixels represent otherwise the value 0. The matrix thus corresponds to a discretized binary image with top view from above.

Sei R eine NxM-Matrix mit: R = q 1 1 q M 1 q 1 N q M N

Figure imgb0003
worin die qi(k)∈{0,1} , dann hat die polynomiale Phasentransformation die Form: z k = j = 1 M e i P q j k
Figure imgb0004
Let R be an NxM matrix with: R = q 1 1 ... q M 1 q 1 N ... q M N
Figure imgb0003
where the q i ( k ) ∈ {0,1}, then the polynomial phase transformation has the form: z k = Σ j = 1 M e i P q j k
Figure imgb0004

Sei P(x) ein Polynom von x. Im Folgenden werden nur Polynome vom Grad 1 benutzt. Speziell für die Detektion von Geraden wird folgendes Polynom benutzt: P x = μ 1 x mit x = y tan ϕ + x 0

Figure imgb0005
µ1 ist ein konstanter, geeignet zu wählender Parameter, typischerweise im Bereich 0,01 bis 1. Mit diesem Parameter wird die Samplingfrequenz beeinflusst.Let P ( x ) be a polynomial of x. In the following, only polynomials of degree 1 are used. The following polynomial is used specifically for the detection of straight lines: P x = μ 1 x With x = y tan φ + x 0
Figure imgb0005
μ 1 is a constant parameter that can be selected, typically in the range 0.01 to 1. This parameter affects the sampling frequency.

Im Falle von d Geraden mit Winkeln ϕj und Achsenabschnitten xj mit 1≤ jd und einer Linienbreite, so dass jede Gerade nur einen Eintrag pro Zeile erzeugt, gilt erfindungsgemäß für den k-ten Eintrag von z : z k = j = 1 d e i P k tan ϕ j + x j = j = 1 d e i μ 1 k tan ϕ j + x j

Figure imgb0006
In the case of d lines with angles φ j and intercept sections x j with 1≤ jd and a line width such that each line generates only one entry per line, according to the invention, for the shortest entry of z : z k = Σ j = 1 d e i P k tan φ j + x j = Σ j = 1 d e i μ 1 k tan φ j + x j
Figure imgb0006

Dieses ist ein sinusförmiges Signal, welches man in der Form (5) schreiben kann: z k = e 1 x j e 1 k tan ϕ j = b j e i f j k

Figure imgb0007
This is a sinusoidal signal which can be written in the form (5): z k = Σ e 1 x j e 1 k tan φ j = Σ b j e i f j k
Figure imgb0007

Die Frequenzen fj enthalten nun die gesuchten Informationen, nämlich die Winkel ϕj: f j = µ 1 tan ϕ j ϕ j = arctan f j µ 1

Figure imgb0008
The frequencies f j now contain the searched information, namely the angles φ j : f j = μ 1 tan φ j φ j = arctan f j μ 1
Figure imgb0008

Matrix Pencil MethodeMatrix Pencil Method

Im nächsten Schritt werden erfindungsgemäß die Frequenzen ϕj bestimmt. Hierzu gibt es verschiedene Mögtichkeiten. Dazu zählen die Fourier-Transformation (FT), die Maximum-Likelihood Methode (MLM), die Methode der kleinsten Quadrate (LSM) und die Matrix Pencil Methode (MPM). Im Folgenden wird die Matrix Pencil Methode angewendet. Selbstverständlich lassen sich auch die anderen Methoden zur Frequenzbestimmung heranziehen.In the next step, the frequencies φ j are determined according to the invention. There are various possibilities for this. These include the Fourier transform (FT), the Maximum Likelihood Method (MLM), the least squares method (LSM) and the Matrix Pencil Method (MPM). In the following the matrix pencil method is used. Of course, the other methods for frequency determination can be used.

Zunächst werden die Eingangsdaten in folgender Art und Weise zusammengefasst: x k = x k , , x N L + k 1 T X 0 = x L 1 , , x 0 X 1 = x L , , x 1

Figure imgb0009
First, the input data is summarized in the following way: x k = x k . ... . x N - L + k - 1 T X 0 = x L - 1 . ... . x 0 X 1 = x L . ... . x 1
Figure imgb0009

Worin "T" transponiert bedeutet und L ein gewählter Parameter ist, der sogenannte Pencilparameter, mit dLN - d. Die x(k) sind die gemessenen Daten in der Form: x k = j = 1 d b j z j k mit z j = e if j

Figure imgb0010
Where " T " means transposed and L is a chosen parameter, the so-called pencil parameter, with dLN - d. The x ( k ) are the measured data in the form: x k = Σ j = 1 d b j z j k With z j = e if j
Figure imgb0010

Drei Aussagen sind wichtig, um die MPM nachzuvollziehen:

  • Erstens: X 0 = Z L BZ R X 1 = Z L BZZ R
    Figure imgb0011
    mit Z L = 1 1 z 1 z M z 1 N L 1 z M N L 1 Z R = z 1 L 1 z 1 L 2 1 1 1 z M L 1 z M L 2 1 B = diag b 1 , b 2 , , b M Z = diag z 1 , z 2 , , z M
    Figure imgb0012
  • Zweitens:
    • Jedes der {z t;r =1,...,M} ist eine Rang reduzierende Zahl des Matrix Pencils X 1 - zX0, wenn ML ≤ N - M ist.
    • Keine der {zt ;r =1,...,M}ist eine Rang reduzierende Zahl des Matrix Pencils, wenn L < M oder L > N-M ist.
  • Drittens:
    • Wenn M ≤ L ≤ N - M ist, ergeben sich die Lösungen des Singulär- Eigenwertproblems: X 1 zX 0 q = 0 p t X 1 zX 0 = 0
      Figure imgb0013
      zu: z = z r q = q r = r te Spalte von Z R + = Z R t Z R Z R t 1 p t = p r t = r te Zeile von Z L + = Z L t Z L 1 Z L t
      Figure imgb0014
      • X 0 und X 1 sind nicht quadratisch, deswegen ist es eine Singulärwertzerlegung und kein einfaches Eigenwertproblem.
      • q r und p r , werden die Rechts- bzw. Linkssingulärvektoren zum Singulärwert zr genannt.
      • "+" steht für die Moore-Penrose Inverse oder Pseudoinverse. "t" bedeutet konjugiert und transponiert, "-1" ist die Inverse.
Three statements are important to understand the MPM:
  • First: X 0 = Z L BZ R X 1 = Z L BZZ R
    Figure imgb0011
    With Z L = 1 ... 1 z 1 ... z M z 1 N - L - 1 ... z M N - L - 1 Z R = z 1 L - 1 z 1 L - 2 ... 1 ... 1 ... 1 z M L - 1 z M L - 2 ... 1 B = diag b 1 . b 2 . ... . b M Z = diag z 1 . z 2 . ... . z M
    Figure imgb0012
  • Secondly:
    • Each of { z t ; r = 1 , ..., M } is a rank reducing number of the matrix pencil X 1 - zX 0 when M L N - M.
    • None of the { z t ; r = 1, ..., M } is a rank reducing number of the matrix pencil when L <M or L> NM.
  • Third:
    • If M ≤ L ≤ N - M , the solutions of the singular eigenvalue problem result: X 1 - zX 0 q = 0 p t X 1 - zX 0 = 0
      Figure imgb0013
      to: z = z r q = q r = r - te column of Z R + = Z R t Z R Z R t - 1 p t = p r t = r - te line from Z L + = Z L t Z L - 1 Z L t
      Figure imgb0014
      • X 0 and X 1 are not quadratic, so it is a singular value decomposition and not a simple eigenvalue problem.
      • q r and p r , the right and left singular vectors are called the singular value z r .
      • "+" stands for the Moore-Penrose Inverse or Pseudoinverse. "t" means conjugated and transposed, "-1" is the inverse.

Basierend auf (10) bekommt man die folgenden Ergebnisse zur Berechnung der {zt;r =1,...,M} von X 0 und X 1:

  • Multipliziert man die erste Gleichung von (11) von links mit X 0 + ,
    Figure imgb0015
    erhält man: X 0 + X 1 q r = z r X 0 + X 0 q r = z r q r
    Figure imgb0016
Based on (10) you get the following results for the calculation of {z t ; r = 1, ..., M } of X 0 and X 1 :
  • Multiply the first equation of (11) from the left X 0 + .
    Figure imgb0015
    you get: X 0 + X 1 q r = z r X 0 + X 0 q r = z r q r
    Figure imgb0016

Das impliziert, dass {zt ; r =1,...,M} die M Singulärwerte von X 0 + X 1 ,

Figure imgb0017
sind. Da X 0 + X 1
Figure imgb0018
den Rang M ≤ L hat, existieren auch noch L-M Singulärwerte mit dem Wert 0 für das Matrixprodukt.
Für X 1 X 0 +
Figure imgb0019
ergibt sich auf äquivalente Weise, dass X 1 X 0 +
Figure imgb0020
M Singulärwerte vergleichbar mit den zr 's hat und N-L-M Eigenwerte gleich 0, und X 1 X 0 +
Figure imgb0021
(oder X 0 + X 1
Figure imgb0022
) hat M Singulärwerte vergleichbar mit z r 1 s
Figure imgb0023
und L -M (oder N-L-M) Singulärwerte gleich 0.
Für verrauschte Daten y(k) = x(k)+n(k) worin n(k) das Rauschen ist, sind y(k), Y0 und Y 1 genauso definiert, wie x (k), X 0 und X 1 .
Die Pseudoinversen X 0 +
Figure imgb0024
oder X 1 +
Figure imgb0025
werden ersetzt durch die auf Rang M verkürzte Pseudoinverse Y 0 +
Figure imgb0026
bzw. Y 1 + .
Figure imgb0027
Y 0 +
Figure imgb0028
ist definiert als: Y 0 + = r = 1 M 1 σ 0 r v 0 r u 0 r t = V 0 A 1 U 0 t
Figure imgb0029
wobei {σ0t ;r =1,...,M} die M größten Singulärwerte von Y 0 sind; v 0r und u 0 r sind die zugehörigen Singulärvektoren; V 0 = v 01 , , v 0 M U 0 = u 01 , , u 0 M A = diag σ 01 , , σ 0 M
Figure imgb0030
Y 1 +
Figure imgb0031
ist ähnlich definiert.This implies that { z t ; r = 1 , ..., M } the M singular values of X 0 + X 1 .
Figure imgb0017
are. There X 0 + X 1
Figure imgb0018
has the rank M ≤ L , there are also LM singular values with the value 0 for the matrix product.
For X 1 X 0 +
Figure imgb0019
Equivalent to that X 1 X 0 +
Figure imgb0020
M has singular values comparable to those of z r 's and has NLM eigenvalues equal to 0, and X 1 X 0 +
Figure imgb0021
( or X 0 + X 1
Figure imgb0022
) M has singular values comparable to z r - 1 ' s
Figure imgb0023
and L -M (or NLM ) singular values equal to 0.
For noisy data y ( k ) = x ( k ) + n ( k ) where n ( k ) is the noise, y ( k ) , Y 0 and Y 1 are defined as well x (K), X 0 and X. 1
The pseudo inverses X 0 +
Figure imgb0024
or X 1 +
Figure imgb0025
are replaced by the pseudoinverse shortened to rank M. Y 0 +
Figure imgb0026
respectively. Y 1 + ,
Figure imgb0027
Y 0 +
Figure imgb0028
is defined as: Y 0 + = Σ r = 1 M 1 σ 0 r v 0 r u 0 r t = V 0 A - 1 U 0 t
Figure imgb0029
where {σ 0 t ; r = 1, ..., M } are the M largest singular values of Y 0 ; v 0 r and u 0 r are the associated singular vectors; V 0 = v 01 . ... . v 0 M U 0 = u 01 . ... . u 0 M A = diag σ 01 . ... . σ 0 M
Figure imgb0030
Y 1 +
Figure imgb0031
is similarly defined.

Was die Rauschkomponenten in Y 0 betrifft, ist die Stetigkeit jedes Elements der verkürzten Pseudoinversen Y 0 +

Figure imgb0032
auch an dem Punkt erhalten, wo das Rauschen gleich 0 ist, und somit auch die Stetigkeit der zr 's. Im Gegensatz zu der Tatsache dass die komplette Pseudoinverse von Y 0 unstetig ist an dem Punkt wo das Rauschen null ist, und somit würde es verschiedene Probleme verursachen, die komplette Pseudoinverse zu berechen bei einem niedrigen Rauschpegel. Y 0 +
Figure imgb0033
ist identisch mit X 0 +
Figure imgb0034
dann und nur dann wenn das Rauschen null ist. Da Y 0 + Y 1
Figure imgb0035
L-M Singulärwerte gleich null hat, die keine Informationen über die zr 's beinhalten, kann man die Größe der Matrix reduzieren, bevor man die Singulärwerte berechnet. Ersetzt man X 0 und X 1 in (13) durch Y 0, und Y 1 und substituiert (14) in (13) für Y 0 + ,
Figure imgb0036
erhält man: V 0 A 1 U 0 t Y 1 q r = z r q r
Figure imgb0037
da V 0 t V 0 = I M
Figure imgb0038
und q r = V 0 t V 0 q r
Figure imgb0039
ergibt Multiplizieren der Gleichung (17) von links mit V 0 t :
Figure imgb0040
A 1 U 0 t Y 1 V 0 V 0 t q r = z r V 0 t q r
Figure imgb0041
As for the noise components in Y 0 , the continuity of each element of the truncated pseudoinverse is Y 0 +
Figure imgb0032
also at the point where the noise equals 0, and thus also the continuity of the z r 's. In contrast to the fact that the complete pseudo inverse of Y 0 is discontinuous at the point where the noise is zero, and thus it would cause various problems to compute the complete pseudo inverse at a low noise level. Y 0 +
Figure imgb0033
is identical to X 0 +
Figure imgb0034
if and only if the noise is zero. There Y 0 + Y 1
Figure imgb0035
If LM has singular values equal to zero that do not contain information about the z r 's, one can reduce the size of the matrix before calculating the singular values. Substituting X 0 and X 1 into (13) by Y 0 and Y 1 and substituting (14) in (13) for Y 0 + .
Figure imgb0036
you get: V 0 A - 1 U 0 t Y 1 q r = z r q r
Figure imgb0037
there V 0 t V 0 = I M
Figure imgb0038
and q r = V 0 t V 0 q r
Figure imgb0039
results in multiplying equation (17) from the left with V 0 t :
Figure imgb0040
A - 1 U 0 t Y 1 V 0 V 0 t q r = z r V 0 t q r
Figure imgb0041

Nun kann man erkennen, dass die Schätzung der zr 's gefunden werden können durch Berechnung der Eigenwerte der M × M nicht-symmetrischen Matrix: Z E = A 1 U 0 t Y 1 V 0

Figure imgb0042
Now it can be seen that the estimation of z r 's can be found by calculating the eigenvalues of the M × M non-symmetric matrix: Z e = A - 1 U 0 t Y 1 V 0
Figure imgb0042

Somit sind die Singulärwerte zr die M Eigenwerte von ZE* , welche dieselben sind wie die M von null verschiedenen Singulärwerte von Y 0 + Y 1 .

Figure imgb0043

Somit sind die gesuchten Frequenzen: f j = Im log z r
Figure imgb0044
Thus, the singular values z r are the M eigenvalues of Z E * which are the same as the M non-zero singular values of Y 0 + Y 1 ,
Figure imgb0043

Thus, the frequencies you are looking for are: f j = in the log z r
Figure imgb0044

Quadratic Form Transform (QFT)Quadratic Form Transform (QFT)

Um herauszufinden, ob die gefundene Gerade auch eine Hochspannungsleitung ist, werden die Pixel der Gerade in die Frontansicht transformiert. Das heißt, man betrachtet die Pixel der Gerade nicht mehr aus der Vogelperspektive. Man benutzt stattdessen das erste gefundene Pixel der Gerade als Nullpunkt eines neuen Koordinatensystems, welches die gefundene Gerade als eine der neuen Achsen benutzt, und die z-Achse als die zweite neue Achse. Somit erhält man ein Diagramm, in dem die gefundene Gerade Katenoidenform haben sollte, wenn sie einer Hochspannungsleitung entspricht. Die Transformation PPT basiert auf einem Polynom vom Grad 1. Also ist es damit nicht möglich, die Parameter eines cosh zu bestimmen, denn das grundlegende Problem dabei ist, das man eine Gleichung der Form: f k = A ar cosh k C A + B

Figure imgb0045
zu lösen hätte, legt man die allgemeine Form des cosh zu Grunde: F x = A cosh x B A + C
Figure imgb0046
To find out if the straight line found is also a high voltage line, the pixels of the line are transformed into the front view. In other words, you no longer look at the pixels of the straight line from a bird's eye view. Instead, one uses the first found pixel of the straight as the zero point of a new coordinate system, which uses the found straight line as one of the new axes, and the z axis as the second new axis. Thus, one obtains a graph in which the found straight should have catenoid shape if it corresponds to a high voltage line. The transformation PPT is based on a polynomial of degree 1. So it is not possible to determine the parameters of a cosh, because the basic problem is that you can use an equation of the form: f k = A ar cosh k - C A + B
Figure imgb0045
to solve, it is based on the general form of the cosh: F x = A cosh x - B A + C
Figure imgb0046

Deshalb benutzt man zunächst die ersten beiden Terme der Taylorentwicklung des cosh: f x = cosh x = n df dx n x x 0 n = 1 + 1 2 x 2 +

Figure imgb0047
Therefore, one first uses the first two terms of the Taylor development of the cosh: f x = cosh x = Σ n df dx n x - x 0 n = 1 + 1 2 x 2 + ...
Figure imgb0047

Wenn die gefundene Gerade einer Hochspannungsleitung entspricht, sollte sie also in einer annähernden Parabelform zu sehen sein. Nun kann man mit Hilfe einer weiteren Transformation, der sogenannten Quadratic Form Transform, diese Parabelform überprüfen. In der Frontansicht wird also ein Test auf Parabelform gemacht, allerdings nicht mit der PPT, sondern mit der QFT. Anschließend wird das Signal wieder an die Matrix Pencil Methode (MPM) weitergegeben.If the straight line found corresponds to a high voltage line, it should therefore be seen in an approximate parabolic shape. Now, with the help of another transformation, the so-called Quadratic Form Transform, one can check this parabolic shape. In the front view, therefore, a test is made on parabola, but not with the PPT, but with the QFT. Subsequently, the signal is passed on again to the Matrix Pencil Method (MPM).

Diese Transformation wird benutzt um die Parameter einer Parabel zu bestimmen. Im Weiteren wird die Funktion der Transformation kurz erläutert:

  • Ausgangspunkt ist die allgemeine Form einer Parabel: y = a x x 0 2 + y 0
    Figure imgb0048
This transformation is used to determine the parameters of a parabola. In the following, the function of the transformation is briefly explained:
  • The starting point is the general form of a parabola: y = a x - x 0 2 + y 0
    Figure imgb0048

Dann hat das für die Transformation benutzte Signal ähnlich zu der PPT die Form: z k = e i µ 1 a k x 0 2 + y 0

Figure imgb0049
Then, similar to the PPT, the signal used for the transformation has the form: z k = e i μ 1 a k - x 0 2 + y 0
Figure imgb0049

Die Transformation an sich ist dann: z k + τ = e 1 a k + τ x 0 2 + y 0 z * k τ = e 1 a k τ x 0 2 + y 0

Figure imgb0050
The transformation itself is then: z k + τ = e 1 a k + τ - x 0 2 + y 0 z * k - τ = e - 1 a k - τ - x 0 2 + y 0
Figure imgb0050

Und schließlich führt folgende Rechnung dann zum Resultat: Z k = z k + τ z * k τ = e iaµ 1 2 2 x 0 k 2 x 0 τ 2 + 2 x 0 τ 2 kx 0 = e iaµ 1 4 4 x 0 τ = A 0 e 1 4 k

Figure imgb0051
And finally, the following calculation leads to the result: Z k = z k + τ z * k - τ = e iaμ 1 2 - 2 x 0 k - 2 x 0 τ - - 2 + 2 x 0 τ - 2 kx 0 = e iaμ 1 4 - 4 x 0 τ = A 0 e 1 4 k
Figure imgb0051

Dieses Ergebnis hängt nur linear von der Variable k ab, so dass es nun in der Matrix Penxcil Method benutzt werden kann um den Parameter a zu bestimmen: Z k = A 0 e 1 4 aτk = A 0 e ifk 4 1 τ = f a = 1 4 τµ 1 f

Figure imgb0052
This result depends only linearly on the variable k, so that it can now be used in the Matrix Penxcil Method to determine the parameter a : Z k = A 0 e 1 4 aτk = A 0 e ifk 4 1 τ = f a = 1 4 τμ 1 f
Figure imgb0052

Dabei gilt es zu beachten, τ # 0, zweckmäßig wird τ auf 1 gesetzt.It should be noted, τ # 0, expediently τ is set to 1.

Wäre der Wert gleich null, würde man mit (26) den Betrag der komplexen Zahl z mit 1 festlegen. Damit wäre keine Weiterverarbeitung möglich.If the value were equal to zero, then (26) would set the amount of the complex number z to 1. Thus, no further processing would be possible.

Claims (4)

  1. Method for improved detection of wire-type objects in an area in front of an aircraft, wherein, taking into account the position and attitude of the aircraft, a set of three-dimensional measurement points is produced from the distance values of a range image of this area, which is produced using a distance sensor, in a geodetic coordinate system, from which set potential wire measurement points are extracted using known filter methods for identifying measurement points which are eliminated as clutter or disturbances, wherein in a first step, the potential wire measurement points are projected into a horizontal plane, straight line profiles are identified, and wherein, in a second step, the potential wire measurement points on, and at a specifiable distance from, a straight line found in the first step are projected into a plane vertical to the horizontal plane and the respective straight line from the first step, and catenoid or parabolic profiles are identified using a quadratic form transformation, wherein in the first step, the horizontal projection of the potential wire measurement points is effected onto an N×M matrix, according to R = q 1 1 q M 1 q 1 N q M N
    Figure imgb0055
    with q i k 0 1 ,
    Figure imgb0056
    k=1,...N, wherein i=1,...,M and and wherein the straight line profiles are identified using a polynomial phase transformation and a subsequent spectral analysis, wherein in the polynomial phase transformation, the potential wire measurement points from the matrix R are transformed into a sinusoidal signal z(k) of the form z k = j = 1 d e i µ 1 k tan ϕ j + x j
    Figure imgb0057
    wherein
    j: variable between 1 and d
    d: number of straight lines
    µ1: constant parameter between 0.01 and 1
    xj: axis section of the j-th straight line
    ϕj: angle of the j-th straight line,
    and the frequencies fj=µ1*tanϕj are determined in the spectral analysis.
  2. Method according to Claim 1, characterized in that found catenoid or parabolic profiles are compared to temporally preceding identifications in order to achieve a low false alarm rate.
  3. Method according to Claim 1 or 2, characterized in that a range image accumulated from a plurality of temporally successive range images is analysed.
  4. Method according to one of the preceding claims, characterized in that implausible wire profiles are eliminated in a plurality of wires found in a range image.
EP10002260.7A 2009-05-09 2010-03-05 Method for improved recognition of conduit-type objects Not-in-force EP2251849B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
DE102009020636.1A DE102009020636B4 (en) 2009-05-09 2009-05-09 Method for improved detection of line-type objects

Publications (3)

Publication Number Publication Date
EP2251849A2 EP2251849A2 (en) 2010-11-17
EP2251849A3 EP2251849A3 (en) 2012-08-08
EP2251849B1 true EP2251849B1 (en) 2016-10-12

Family

ID=42289768

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10002260.7A Not-in-force EP2251849B1 (en) 2009-05-09 2010-03-05 Method for improved recognition of conduit-type objects

Country Status (2)

Country Link
EP (1) EP2251849B1 (en)
DE (1) DE102009020636B4 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104849708B (en) * 2015-05-18 2017-03-08 中国民航大学 High speed machine moving target parameter estimation method based on the conversion of frequency domain polynomial-phase

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE19605218C1 (en) * 1996-02-13 1997-04-17 Dornier Gmbh Obstacle warning system for low-flying aircraft
DE19828318C2 (en) * 1998-06-25 2001-02-22 Eurocopter Deutschland Wire highlighting
DE10055572C1 (en) * 2000-11-09 2002-01-24 Astrium Gmbh Real-time overhead line identification method for low-flying aircraft uses evaluation of image data provided by image sensor scanning field of view ahead of aircraft
FR2888944B1 (en) * 2005-07-20 2007-10-12 Eurocopter France METHOD FOR TELEMETRY DETECTION OF SUSPENDED WIRED OBJECTS
DE102005047273B4 (en) * 2005-10-01 2008-01-03 Eads Deutschland Gmbh Method for supporting low-level flights

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ABED-MERAIM K ET AL: "Multi-line fitting using polynomial phase transforms and downsampling", 2001 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING. PROCEEDINGS. (ICASSP). SALT LAKE CITY, UT, MAY 7 - 11, 2001; [IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP)], NEW YORK, NY : IEEE, US, vol. 3, 7 May 2001 (2001-05-07), pages 1701 - 1704, XP010802866, ISBN: 978-0-7803-7041-8, DOI: 10.1109/ICASSP.2001.941266 *

Also Published As

Publication number Publication date
EP2251849A2 (en) 2010-11-17
DE102009020636A1 (en) 2010-11-18
EP2251849A3 (en) 2012-08-08
DE102009020636B4 (en) 2017-09-28

Similar Documents

Publication Publication Date Title
DE102015100134B4 (en) Method for detecting and tracking objects using multiple radar sensors
EP2418510B1 (en) Method for evaluating the suitability of a piece of land for a landing zone or taxi surface for airplanes
DE102018102884A1 (en) LIDAR RADAR RELATIVE POSE CALIBRATION
EP1531343B1 (en) Method for tracking objects
DE10307580B3 (en) Magnetic field sensor position detection and tracking method for controlling image simulation for pilot night-vision device
EP2417476B1 (en) Fmcw radar sensor and method for frequency matching
EP2927844A1 (en) 3d object position and pose estimation
EP1770595B1 (en) Procedure to assist low-flying aircraft in obstacle detection
DE102015107388A1 (en) Collision avoidance with static targets in confined spaces
DE102015115786B4 (en) METHOD AND SYSTEM REALIZED ON A COMPUTER FOR PROCESSING A SEQUENCE OF IMAGES
DE102017128194A1 (en) Accurate self-locating using a synthetic aperture automotive radar
DE102018132805A1 (en) Procedure for improved object detection
EP1205765B1 (en) Method of recognising conducting cables for low flying aircraft
EP1634241B1 (en) Method and device for object recognition in driver aid systems for motor vehicles
DE102019114723A1 (en) REDUCTION OF VIBRATIONS IN A RADAR SYSTEM ON A MOVING PLATFORM
DE102021100351A1 (en) ADAPTIVE EXAMINATION FOR LIDAR-BASED GROUPING
DE102006039104A1 (en) Method for ranging or detection of stationary objects in surroundings of road vehicle using graphic data, involves testing object hypothesis such as stationary object against object hypothesis such as flat road
EP3663881B1 (en) Method for controlling an autonomous vehicle on the basis of estimated movement vectors
EP2251849B1 (en) Method for improved recognition of conduit-type objects
DE102018001533B4 (en) Real-time method for dynamic coordinate and parameter estimation with grid-based non-linear probabilistic filters
DE102018121158A1 (en) Ground sensing point method and driver support system configured to perform such a method
WO2022129266A1 (en) Method for detecting at least one object in surroundings by means of reflection signals of a radar sensor system
DE102019215440B4 (en) Recognition of traffic signs
EP2699933B1 (en) Method and device for determining target parameters
DE102006051091B4 (en) Method for object recognition in vehicles by means of close-range detection

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA ME RS

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA ME RS

RIC1 Information provided on ipc code assigned before grant

Ipc: G08G 5/00 20060101AFI20120629BHEP

17P Request for examination filed

Effective date: 20130201

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: AIRBUS DEFENCE AND SPACE GMBH

17Q First examination report despatched

Effective date: 20151130

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: AIRBUS DS ELECTRONICS AND BORDER SECURITY GMBH

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

INTG Intention to grant announced

Effective date: 20160718

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 837129

Country of ref document: AT

Kind code of ref document: T

Effective date: 20161015

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502010012526

Country of ref document: DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20161012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170113

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170112

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170212

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170213

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502010012526

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 502010012526

Country of ref document: DE

Owner name: HENSOLDT SENSORS GMBH, DE

Free format text: FORMER OWNER: AIRBUS DS ELECTRONICS AND BORDER SECURITY GMBH, 82024 TAUFKIRCHEN, DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170112

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

26N No opposition filed

Effective date: 20170713

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170331

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170305

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170331

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20170331

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 837129

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20100305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161012

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20220324

Year of fee payment: 13

Ref country code: DE

Payment date: 20220322

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20220323

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20220331

Year of fee payment: 13

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 502010012526

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20230305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230305

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230331

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231003

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230305