CN115328181A - Method for positioning key target space in unmanned aerial vehicle power transmission line inspection - Google Patents
Method for positioning key target space in unmanned aerial vehicle power transmission line inspection Download PDFInfo
- Publication number
- CN115328181A CN115328181A CN202210921547.XA CN202210921547A CN115328181A CN 115328181 A CN115328181 A CN 115328181A CN 202210921547 A CN202210921547 A CN 202210921547A CN 115328181 A CN115328181 A CN 115328181A
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- aerial vehicle
- unmanned aerial
- coordinates
- binocular
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000005540 biological transmission Effects 0.000 title claims abstract description 40
- 238000007689 inspection Methods 0.000 title claims abstract description 29
- 238000006243 chemical reaction Methods 0.000 claims abstract description 40
- 238000012937 correction Methods 0.000 claims abstract description 24
- 230000000007 visual effect Effects 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 39
- 238000013519 translation Methods 0.000 claims description 17
- 230000002159 abnormal effect Effects 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 7
- 238000003384 imaging method Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000010438 heat treatment Methods 0.000 claims description 5
- 239000013598 vector Substances 0.000 claims description 4
- 238000007476 Maximum Likelihood Methods 0.000 claims description 2
- 230000001174 ascending effect Effects 0.000 claims description 2
- 230000010354 integration Effects 0.000 claims description 2
- 238000005096 rolling process Methods 0.000 claims description 2
- 238000004804 winding Methods 0.000 claims 2
- 239000004615 ingredient Substances 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 101100221616 Halobacterium salinarum (strain ATCC 29341 / DSM 671 / R1) cosB gene Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/08—Control of attitude, i.e. control of roll, pitch, or yaw
- G05D1/0808—Control of attitude, i.e. control of roll, pitch, or yaw specially adapted for aircraft
- G05D1/0816—Control of attitude, i.e. control of roll, pitch, or yaw specially adapted for aircraft to ensure stability
- G05D1/0825—Control of attitude, i.e. control of roll, pitch, or yaw specially adapted for aircraft to ensure stability using mathematical models
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
- G05D1/106—Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
Landscapes
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Aviation & Aerospace Engineering (AREA)
- Mathematical Analysis (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Physics (AREA)
- Mathematical Optimization (AREA)
- Algebra (AREA)
- Position Fixing By Use Of Radio Waves (AREA)
Abstract
The invention provides a method for positioning a key target space in power transmission line inspection of an unmanned aerial vehicle, which comprises the following steps: 1) Acquiring binocular visual information of a key target to be detected and GPS and height information of an unmanned aerial vehicle; 2) Calculating the three-dimensional coordinates of the target to be measured in a world coordinate system according to the binocular vision model; 3) Realizing binocular calibration, correction and sparse stereo matching based on a binocular vision space positioning principle; 4) Constructing a coordinate system conversion model of the unmanned aerial vehicle and a key target reference world, and calculating the height of the key target; 5) Constructing an unmanned aerial vehicle geographic longitude and latitude, height, azimuth and key target conversion model, and calculating a key target geographic coordinate; 6) And guiding the unmanned aerial vehicle to patrol and fly by utilizing the key target position. The invention provides a method for positioning a key target space in power transmission line inspection of an unmanned aerial vehicle, which can cooperatively process visible light characteristic image information of a detected target and GPS (global positioning system) information of the detected unmanned aerial vehicle, obtain the longitude and latitude and the height of the key target and improve the inspection efficiency of the unmanned aerial vehicle of the power transmission line.
Description
Technical Field
The invention belongs to the technical field of key target space positioning in a power transmission line, and particularly relates to a key target space positioning method in unmanned aerial vehicle power transmission line inspection based on binocular stereo vision and airborne GPS cooperation.
Background
In the unmanned aerial vehicle transmission line inspection process, the current position information of the unmanned aerial vehicle can only be displayed in real time by the existing unmanned aerial vehicle inspection system, then the relative distance between the unmanned aerial vehicle and a target to be detected is obtained through binocular vision or other distance sensors, the position of the target to be detected captured by the unmanned aerial vehicle carrying camera is difficult to obtain, and therefore the spatial position of a key target object cannot be accurately described by a universal coordinate system.
In the actual power transmission line inspection work, the specific spatial position information of the fault defect is often required to be obtained and recorded in time, for example, the position information of the invading foreign matter is recorded so as to carry out the subsequent clearing work, and the position information of the abnormal heating area is recorded so as to carry out the subsequent detection, the maintenance work and the like. The existing solution is that manual measurement is adopted or the position of a detected target is approximated by the geographic position of the unmanned aerial vehicle, the former is usually limited by a special geographic position and cannot be reached by personnel, so that the work of positioning, measuring, overhauling and the like cannot be smoothly carried out, the latter is not accurate enough, and when the error is too large, electric power accidents such as collision of the unmanned aerial vehicle and the power transmission line are easily caused. For example, patent specification CN202111677929.4 discloses a system for detecting offset distance of tower top of power tower by unmanned aerial vehicle, which transmits ranging laser beam to tower top of power tower through laser ranging sensor, analyzes ranging signal, analyzes offset distance between unmanned aerial vehicle and power tower, can only obtain relative position relationship between the two, and has lower practical significance.
In order to solve the problems, the pixel coordinates of the detected key target in the two-dimensional image need to be matched with the geographic coordinates in the real scene, so that the form expression convenient for the power equipment inspector to use is obtained, and the unmanned aerial vehicle is guided to quickly and accurately inspect.
Disclosure of Invention
In order to solve the problems, the invention provides a method for positioning a key target space in unmanned aerial vehicle power transmission line inspection based on binocular stereo vision and airborne GPS cooperation, which can cooperatively process visible light characteristic image information of a detected target and GPS information of the detected unmanned aerial vehicle, realize matching of the visible light characteristic image information and the GPS information of the detected unmanned aerial vehicle, construct a target three-dimensional space positioning coordinate system conversion relation model, obtain longitude and latitude and height of the key target according to flight data and equipment parameters of the unmanned aerial vehicle, and improve the inspection efficiency of the unmanned aerial vehicle of the power transmission line.
The invention particularly relates to a method for positioning a key target space in unmanned aerial vehicle power transmission line inspection based on binocular stereo vision and airborne GPS cooperation, which comprises the following steps:
step (1): acquiring binocular visual information of a key target to be detected by using a binocular camera carried by the unmanned aerial vehicle, and acquiring longitude and latitude information and height information of real-time flight of the unmanned aerial vehicle by using a GPS (global positioning system) of the unmanned aerial vehicle;
step (2): calculating the three-dimensional coordinates of the target to be measured in a world coordinate system according to the binocular vision model, constructing a binocular vision pinhole model, and realizing the conversion among the world coordinates, the camera coordinates, the physical coordinates and the pixel coordinates;
and (3): realizing binocular calibration based on a binocular vision space positioning principle, then carrying out distortion correction and binocular stereo correction on the images according to a calibration result, and finally carrying out sparse stereo matching on corresponding points in the binocular images to obtain a position corresponding relation between the binocular images;
and (4): constructing a target three-dimensional space positioning coordinate system conversion relation model according to the target space information and the real-time flight data of the unmanned aerial vehicle, and deducing the height of a key target;
and (5): constructing a conversion model of the geographical longitude and latitude, the height, the azimuth angle and the key target of the unmanned aerial vehicle through the geographical longitude and latitude, the height and the azimuth angle of the unmanned aerial vehicle and the coordinates of the unmanned aerial vehicle and the key target under a reference world coordinate system, and calculating the geographical coordinates, namely the longitude and latitude, of the key target;
and (6): the longitude and latitude information and the height information of the key target are used for guiding the unmanned aerial vehicle to patrol and fly, so that the unmanned aerial vehicle can directly fly to positions of abnormal heating, damaged parts and key parts in the power transmission line-road patrol and inspection process to perform abnormal detection or damage detection on the power transmission line.
Compared with the prior art, the beneficial effects are: the method for positioning the key target space in the unmanned aerial vehicle power transmission line patrol comprises the steps of constructing a target three-dimensional space positioning derivation model by acquiring binocular stereoscopic vision information and corresponding airborne GPS information of a key target of the power transmission line and combining binocular stereoscopic vision, real-time flight data of the unmanned aerial vehicle and equipment parameters according to a position conversion relation among an image coordinate system, a camera coordinate system, a world coordinate system and a geographic coordinate system, so that longitude and latitude and height information of the key target are directly obtained according to the real-time flight data of the unmanned aerial vehicle and the equipment parameters, pixel coordinates of the key target in a two-dimensional image are matched with the geographic coordinates in a real scene, the pixel coordinates are converted into a universal geodetic coordinate form which is convenient for personnel to understand, and finally, the unmanned aerial vehicle is guided to autonomously fly and patrol according to the geodetic coordinates, so that the unmanned aerial vehicle power transmission line patrol efficiency is greatly improved.
Drawings
FIG. 1 is a block diagram of a method for locating a key target space in power transmission line inspection of an unmanned aerial vehicle according to the present invention;
FIG. 2 is a schematic diagram of an implementation flow of a method for locating a key target space in power transmission line inspection of an unmanned aerial vehicle according to the invention;
FIG. 3 is a schematic diagram showing the position relationship among a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system in binocular vision;
FIG. 4 is a schematic diagram of the relationship conversion of four coordinate systems in binocular vision;
FIG. 5 is a schematic view of a binocular vision system imaging;
fig. 6 is a schematic view of binocular stereo correction;
FIG. 7 is a schematic diagram of sparse stereo matching;
FIG. 8 is a schematic diagram of binocular vision and unmanned aerial vehicle GPS system coordinated key target height calculation;
FIG. 9 is a flowchart of a key target geodetic coordinate algorithm;
fig. 10 is a schematic diagram of the relationship between the geographic coordinate system and the world coordinate system.
Detailed Description
The following describes in detail a specific embodiment of the method for positioning the key target space in the unmanned aerial vehicle power transmission line inspection based on the cooperation of binocular stereo vision and airborne GPS according to the present invention.
As shown in figure 1, the method for locating the space of the key target in the unmanned aerial vehicle power transmission line patrol, which is provided by the invention, comprises a key target binocular vision information acquisition module, an unmanned aerial vehicle longitude and latitude information acquisition module, a binocular camera calibration module, a distortion correction and stereo correction module, a sparse stereo matching module, a key target height calculation module and a key target longitude and latitude solving module. The key target binocular vision information acquisition module acquires abnormal heating, discharging and fault area images in the power transmission line through real-time shooting of a binocular camera carried by the unmanned aerial vehicle to achieve information acquisition, and the unmanned aerial vehicle longitude and latitude information acquisition module acquires real-time longitude and latitude information of the unmanned aerial vehicle through an onboard GPS. The binocular camera calibration module, the distortion correction and stereo correction module and the sparse stereo correction module jointly complete the process from calibration, distortion correction to stereo matching of binocular vision, the key target height and longitude and latitude calculation module realizes the position information solving of key targets such as abnormal targets and objects to be detected in the power transmission line, and the modules have respective connection modes so as to realize the function of key target space positioning of the power transmission line.
As shown in fig. 2, the specific operation flow of the method for positioning the key target space in the unmanned aerial vehicle power transmission line inspection based on the binocular stereo vision and the airborne GPS is as follows:
1) Acquiring binocular visual information of a key target to be detected by using a binocular camera carried by the unmanned aerial vehicle, and acquiring longitude and latitude information and height information of real-time flight of the unmanned aerial vehicle by using a GPS (global positioning system) of the unmanned aerial vehicle;
2) Calculating the three-dimensional coordinates of the target to be measured in the world coordinate system according to the binocular vision model, as shown in fig. 3, the binocular vision model is constructed by a pinhole model, and the conversion among the world coordinates, the camera coordinates, the physical coordinates and the pixel coordinates can be realized, and the specific process is as follows:
21 Firstly, the coordinates of the target to be measured in the world coordinate system are processed by using the rotation matrix R and the translation matrix T to obtain the coordinates in the corresponding camera coordinates, and the three-dimensional coordinates in the world coordinate system are marked as [ X ] w ,Y w ,Z w ]And the obtained coordinate of the camera coordinate system is marked as [ X ] c ,Y c ,Z c ]The conversion formula is shown in the following formula (1), wherein R represents a rotation matrix, T represents a translation matrix, R 11 ～r 33 Is an element in the rotation matrix R, t 1 ～t 3 Are the elements of the translation matrix T.
22 Under an ideal pinhole imaging model, the coordinates in the image coordinate system can be calculated according to the coordinate in the camera coordinate system in the step 21) and the triangle similarity principle, as shown in the following formula (2), wherein f represents the focal length of the binocular camera;
23 In step 22), the image coordinates represent the position information of the three-dimensional object on the two-dimensional image, and the image coordinates are discretized according to the pixel size of the camera without considering the influence of lens distortion factors to obtain the coordinates in a pixel coordinate system, wherein d is x And d y Respectively represent pixelsPhysical dimensions in the x-axis and y-axis directions, i.e., pixel coordinates u and v;
24 In conjunction with the transformation relationship between the respective coordinate systems of the binocular vision model, as shown in fig. 4, a direct transformation relationship between the world coordinate system and the pixel coordinate system is found, where u 0 And v 0 Representing the coordinates of the principal point, M 1 Is a camera internal reference matrix, M 2 The method comprises the following steps of (1) directly calculating a three-dimensional world coordinate corresponding to a pixel coordinate of a target to be measured according to the pixel coordinate by taking a camera external reference matrix as P, wherein P is the product of an internal reference matrix and the external reference matrix;
3) As shown in fig. 5, the binocular calibration is realized based on the binocular vision spatial positioning principle, then the image is subjected to distortion correction and binocular stereo correction according to the calibration result, and finally the corresponding points in the binocular image are subjected to sparse stereo matching to obtain the position corresponding relation between the binocular images, and the specific process is as follows:
31 Camera parameters include internal parameters and external parameters, the calibration of the binocular camera is the process of analyzing the internal and external parameters, a plurality of groups of clear calibration plate images containing different positions are shot according to a target image to be detected shot by the binocular camera carried by the unmanned aerial vehicle, and required parameters are solved according to the relation between extracted angular points, and the calibration method mainly comprises the following steps:
a. solving a homography matrix H between the checkerboard calibration board plane and the image plane, and deducing internal reference and external reference initial values of a left eye camera and a right eye camera in the binocular camera according to the homography matrix, wherein the left eye internal and external references are respectively marked as P1 and K1, and the right eye internal and external references are marked as P2 and K2;
b. optimizing the initial value by using a maximum likelihood estimation algorithm to obtain more accurate internal reference and external reference, wherein the left eye internal and external reference and the right eye internal and external reference are respectively marked as P1', K1', P2', K2';
c. solving distortion parameters of the monocular camera, and recording the distortion parameters as S;
d. and deriving extrinsic parameters between the two cameras by using the extrinsic parameters of the left eye camera and the right eye camera respectively, namely a rotation matrix R1 and a translation matrix T1 of the right eye camera relative to the left eye camera.
32 As shown in fig. 6), according to the calibration result of the binocular camera in step 31), the binocular images captured by the binocular camera carried by the unmanned aerial vehicle are corrected, including image distortion correction and binocular stereo correction, and the specific steps are as follows:
a. dividing the rotation matrix into two parts, namely R3 and R4 respectively, according to the rotation matrix R1 and the translation matrix T1 of the left eye camera to the right eye camera obtained in the step 31) d, and respectively rotating the left eye image and the right eye image according to the new rotation matrix to make the imaging planes of the left eye image and the right eye image coplanar;
b. by three orthogonal vectors e 1 ,e 2 ,e 3 Constructing a rotation vector from a base line to an epipolar line, rotating an x axis of an imaging plane along an optical axis which is adjusted to be parallel to a component of a translation matrix T1 in the x axis to be parallel to the component of the translation matrix T1 in the x axis, realizing alignment of lines of binocular images, further realizing stereo correction of the binocular images, and recording the corrected binocular images as I 1 。
33 For corrected binocular images I) as shown in fig. 7 1 Sparse stereo matching based on the improved SURF algorithm is carried out, and the method specifically comprises the following steps:
a. first, feature point sets of the left eye image and the right eye image are denoted as Pos _ left and Pos _ right, where m and n respectively denote the number of feature points in the left eye image and the right eye image, and the feature point sets are expressed by the following expression (x' i ,y’ i ) And (x) i ,y i ) Respectively representing the coordinates of the characteristic points in the left and right eyes:
b. coarse matching is carried out on all feature points by adopting an Euclidean distance algorithm, and each feature point (x 'in the left eye image is calculated respectively' i ,y’ i ) With all feature points (x) in the right eye image i ,y i ) The Euclidean distance L;
c. sorting all rough matching point pairs according to the ascending order of the distance, deleting many-to-one or one-to-many abnormal matching point pairs, and respectively recording the characteristic point pairs reserved from the left-eye image and the right-eye image as Pos _ left 'and Pos _ right';
d. selecting the first K point pairs with the smallest distance from the coarse matching point pairs, and recording the point pairs as:
Pos_K＝{{(x’ 1 ,y’ 1 ),(x 1 ,y 1 )},{(x' 2 ,y' 2 ),(x 2 ,y 2 )},...,{(x' K ,y' K ),(x K ,y K )}} (6)；
e. connecting the matching points in the binocular image, calculating the slope of all point pair connecting lines in Pos _ K, counting the slope of a straight line with higher occurrence probability, counting the abnormal slope with lower frequency, removing the corresponding matched point pairs, and updating the matching point pair set as shown in the following formula (7):
Pos_K new ＝{{(x l1 ,y l1 ),(x r1 ,y r1 )},{(x l2 ,y l2 ),(x r2 ,y r2 )},...,{(x ln ,y ln ),(x rn ,y rn )}} (7)；
f. and finishing sparse stereo matching according to the updated matching point pairs.
4) As shown in fig. 8, a target three-dimensional space positioning coordinate system transformation relation model is constructed according to target space information and real-time flight data of the unmanned aerial vehicle, and the height of a key target is deduced in the following specific process:
41 Using an industrial personal computer to receive flight data of the unmanned aerial vehicle in real time and extract information required by key target positioning, including longitude and latitude (L) of the unmanned aerial vehicle UAV ,B UAV ) Height h UAV Pitch angle alpha, azimuth angle beta, roll angle gamma and the pitch angle theta of the camera pan-tilt;
42 With the optical center of the left eye camera as the origin and the optical axis as Z w Axis, establishing a world coordinate system O w -X w Y w Z w O is provided with w -X' w Y’ w Z' w Is the coordinate system of the eye camera when the pitch angle of the camera is 0, then the coordinate system O w -X w Y w Z w Is O w -X' w Y’ w Z' w The coordinate system is X' w The axis is a coordinate system after the rotating shaft rotates anticlockwise by theta angle, and the transformation relation is shown as the following formula (8):
43 Measuring coordinates (x ') of the drone in a reference world coordinate system from a GPS location module of the drone' uav ,y’ uav ,z’ uav ) Then, calculating the coordinates (x) of the unmanned aerial vehicle in the real world coordinate system based on the formula in the step 42) uav ,y uav ,z uav )；
44 Coordinate system O with the pitching angle alpha of the unmanned plane, the rolling angle gamma and the pitching angle theta of the camera all being 0 w -X″ w Y″ w Z″ w And recording the coordinate system as a reference world coordinate system, recording the coordinate system with the pitch angle theta in the step 42) as an actual world coordinate system, and constructing a conversion relation between the reference world coordinate system and the actual world coordinate system, wherein the conversion relation is shown as the following formula (9):
45 Let (x) ob ,y ob ,z ob ) Representing the coordinates of the object in the real world coordinate system from the coordinates (x) in the real world coordinate system calculated in step 43) uav ,y uav ,z uav ) And the coordinates (x) of the target in the real world coordinate system ob ,y ob ,z ob ) Obtaining the coordinates (x ″) of the key target and the unmanned aerial vehicle in the reference world coordinate system by using the coordinate system conversion formula provided in the step 44) ob ,y″ ob ,z″ ob ) And (x ″) UAV ,y″ UAV ,z″ UAV )；
46 According to the principle that the Y' axis of the reference world coordinate system points to the geocentric vertically, the height difference between the unmanned aerial vehicle and the key target is calculated, and the calculation formula is shown as the following formula (10), wherein Y ″, the in Represents the coordinate of the key target on the Y' axis in the reference coordinate system:
Δh＝y' UAV -y″ in (10)；
47 The height of the key target to be measured from the ground is calculated according to the formula of the height difference in the step 46) and the height of the unmanned aerial vehicle in the actual coordinate system, and the calculation formula is shown as the following formula (11), wherein h is UAV Height to ground representative of drone:
h ob ＝h UAV +Δh (11)；
5) As shown in fig. 9, a conversion model of the geographical longitude and latitude, the altitude, the azimuth and the key target of the unmanned aerial vehicle is constructed through the geographical longitude and latitude, the altitude, the azimuth and the coordinates of the unmanned aerial vehicle and the key target under the reference world coordinate system, and the geographical coordinates, namely the longitude and latitude, of the key target are calculated, and the specific process is as follows:
51 Firstly, a geodetic coordinate system is constructed by taking a binocular camera carried by the unmanned aerial vehicle as an origin, and points in the coordinate system are described by longitude (L), latitude (B) and height (H);
52 As shown in fig. 10, the geodetic coordinates are converted into the spherical center coordinates according to the geometric relationship between the geodetic coordinate system and the spherical center coordinate system, and the specific steps are as follows:
a. let geodetic coordinates of the survey station P be (L, B, H), and coordinates thereof in the geodetic coordinate system be (X) e ,Y e ,Z e ) And the coordinates in the sphere center coordinate system are (x, y). Converting coordinates in the geodetic coordinate system into a spherical center coordinate system by using a triangular relation, wherein the triangular relation is as follows (12):
b. according to the ellipse equation and the first eccentricity of the meridian and the major radius and the minor radius of the ellipsoid corresponding to the geodetic coordinate systemRadius, and calculating to obtain primary conversion results x and y, wherein the ellipse equation of the meridian is
A first eccentricity of
The conversion formula is shown in the following formula (13), wherein
Expressing the curvature radius of the unitary mortise ring at the point, and the lengths of the semi-long axis and the semi-short axis of the ellipse a and b:
x＝(N+H)cosB
y＝(N(1-e 2 )+H)sinB (13)；
c. integrating the step a and the step b to realize the conversion from the geodetic coordinate system to the spherical center rectangular coordinate system, wherein the conversion formula after integration is as follows (14):
53 According to the geometric relationship between the sphere center coordinate system and the station-centered horizon direct coordinate system, the sphere center coordinate (X) e ,Y e ,Z e ) Converted into a horizontal rectangular coordinate (X) of the center of the station p ,Y p ,Z p ) The method comprises the following specific steps:
a. firstly, performing translation operation on a sphere center rectangular coordinate to enable the original points of two coordinate systems to be overlapped;
b. the sphere center coordinate system is wound around Z e Shaft rotation (90 ° + L 0 ) Degree, around X e Shaft rotation (90-beta) 0 ) Make two rotations in sequence, L 0 And beta 0 The longitude and latitude of the origin of the center-of-station ground flat rectangular coordinate system are respectively converted as follows, wherein (X) 0 ,Y 0 ,Z 0 ) Is the coordinate of the origin of the rectangular coordinates of the station center under the rectangular coordinates of the sphere center, R x And R z Represents X e Axis and Z e The rotation transformation of (1):
54 The method comprises the following steps of) constructing a station center straight angular coordinate system by taking the current position of the unmanned aerial vehicle as an origin, and constructing a conversion relation between a reference world coordinate system and the station center ground straight angular coordinate system, wherein the specific process is as follows:
a. firstly, a reference world coordinate system O is set w -X″ w Y″ w Z″ w Performing translation to make the coordinate origin O w And O p Overlapping;
b. then, the reference world coordinate system is wound around Y ″) w Rotation of axis by-beta degree, and rewinding by X w The axis is rotated by 90 degrees, the conversion relation is as follows, wherein beta is the azimuth angle of the unmanned aerial vehicle, (x ″) UAV ,y″ UAV ,z″ UAV ) Representing unmanned plane, namely origin O of station center ground flat rectangular coordinate system p In a reference world coordinate system O w -X″ w Y″ w Z″ w Coordinates (c) of (a):
55 Based on the coordinate conversion results of steps 52), 53), 54), the coordinate (x') of the key target to be measured in the reference world coordinate system is determined ob ,y″ ob ,z″ ob ) Converting into a horizontal coordinate system of the station center to obtain a corresponding horizontal coordinate (x) of the station center p ,y p ,z p ) Then, the horizontal coordinate of the station center is converted into a corresponding rectangular coordinate system of the sphere center to obtain a rectangular coordinate (x) of the sphere center e ,y e ,z e ) And finally, converting the spherical center rectangular coordinate into a geodetic coordinate system, and calculating to obtain the longitude and latitude of the key target to be measured, wherein a longitude and latitude calculation formula is shown as the following formula (17):
solving the final longitude and latitude of the key target by using an iteration method, and iterating initiallyThe values are as follows, wherein (x) e ,y e ,z e ) Represents the sphere center rectangular coordinates, and a, b represent the ellipse semi-major and semi-minor axis lengths:
the iteration formula is as follows, and when the dimension B reaches the required precision, the iteration is terminated:
6) The longitude and latitude information and the height information of the key target are used for guiding the unmanned aerial vehicle to patrol and fly, so that the unmanned aerial vehicle can directly fly to positions of abnormal heating, damaged parts and key parts in the power transmission line-road patrol and inspection process to perform abnormal detection or damage detection on the power transmission line.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and not for limiting the same. It will be understood by those skilled in the art that various modifications and equivalents may be made to the embodiments of the invention as described herein, and such modifications and variations are intended to be within the scope of the claims appended hereto.
Claims (10)
1. A method for locating a key target space in unmanned aerial vehicle power transmission line inspection is characterized by comprising the following steps:
step (1): acquiring binocular visual information of a key target to be detected by using a binocular camera carried by the unmanned aerial vehicle, and acquiring longitude and latitude information and height information of real-time flight of the unmanned aerial vehicle by using a GPS (global positioning system) of the unmanned aerial vehicle;
step (2): calculating the three-dimensional coordinates of the target to be measured in a world coordinate system according to the binocular vision model, constructing a binocular vision pinhole model, and realizing the conversion among the world coordinates, the camera coordinates, the physical coordinates and the pixel coordinates;
and (3): realizing binocular calibration based on a binocular vision space positioning principle, then carrying out distortion correction and binocular stereo correction on the images according to a calibration result, and finally carrying out sparse stereo matching on corresponding points in the binocular images to obtain a position corresponding relation between the binocular images;
and (4): constructing a target three-dimensional space positioning coordinate system conversion relation model according to the target space information and the real-time flight data of the unmanned aerial vehicle, and deducing the height of a key target;
and (5): constructing a conversion model of the geographical longitude and latitude, the height, the azimuth angle and the key target of the unmanned aerial vehicle through the geographical longitude and latitude, the height and the azimuth angle of the unmanned aerial vehicle and the coordinates of the unmanned aerial vehicle and the key target under a reference world coordinate system, and calculating the geographical coordinates, namely the longitude and latitude, of the key target;
and (6): the longitude and latitude information and the height information of the key target are used for guiding the unmanned aerial vehicle to patrol and fly, so that the unmanned aerial vehicle can directly fly to positions of abnormal heating, damaged parts and key parts in the power transmission line-road patrol and inspection process to perform abnormal detection or damage detection on the power transmission line.
2. The method for locating the space of the key target in the unmanned aerial vehicle power transmission line inspection according to claim 1, wherein in the step (2), the specific step of calculating the three-dimensional coordinate of the target to be measured in the world coordinate system by using the binocular vision model comprises the following steps:
21 Firstly, the coordinates of the target to be measured in the world coordinate system are processed by using the rotation matrix R and the translation matrix T to obtain the coordinates in the corresponding camera coordinates, and the three-dimensional coordinates in the world coordinate system are marked as [ X ] w ,Y w ,Z w ]And the obtained coordinate of the camera coordinate system is marked as [ X ] c ,Y c ,Z c ]Conversion formula is
Wherein R represents a rotation matrix, T represents a translation matrix, R 11 ～r 33 Is an element in the rotation matrix R, t 1 ～t 3 Is an element of the translation matrix T;
22 At least one of the ingredients ofUnder the ideal pinhole imaging model, the coordinates in the image coordinate system can be calculated according to the principle of similarity between the coordinates in the camera coordinate system and the triangle in the step 21):
wherein f represents the focal length of the binocular camera;
23 22) represents the position information of the three-dimensional object on the two-dimensional image, and discretizes the image coordinate according to the pixel size of the camera under the condition of not considering the influence of lens distortion factors to obtain the coordinate in the pixel coordinate system
Wherein d is x And d y Represents the physical dimensions of the pixel in the x-axis and y-axis directions, i.e., pixel coordinates u and v, respectively;
24 To find the direct conversion relation between the world coordinate system and the pixel coordinate system in combination with the conversion relation between the binocular vision model and each coordinate system
Wherein u 0 And v 0 Representing the coordinates of the principal point, M 1 Is a camera internal reference matrix, M 2 The method is characterized in that the method is a camera external parameter matrix, and P is the product of an internal parameter matrix and the external parameter matrix, so that the three-dimensional world coordinate corresponding to the pixel coordinate of the target to be measured is directly calculated according to the pixel coordinate of the target to be measured.
3. The method for locating the space of the key target in the unmanned aerial vehicle transmission line inspection according to claim 1, wherein in the step (3), the specific steps of realizing binocular calibration, three-dimensional correction and sparse three-dimensional matching are as follows:
31 According to an image of a target to be detected shot by an unmanned aerial vehicle carrying binocular camera, shooting a plurality of groups of clear calibration plate images containing different positions, and solving required parameters according to the relation between the extracted angular points;
32 Correcting binocular images shot by the binocular camera carried by the unmanned aerial vehicle according to the calibration result of the binocular camera in the step 31), wherein the correction comprises image distortion correction and binocular stereo correction;
33 To corrected binocular image I 1 And performing sparse stereo matching based on the improved SURF algorithm.
4. The method for positioning the space of the key target in the unmanned aerial vehicle power transmission line inspection according to claim 3, wherein in the step 31), the specific step of solving the required parameters according to the extracted relationship between the angular points comprises the following steps:
a. solving a homography matrix H between the checkerboard calibration board plane and the image plane, and deducing internal reference and external reference initial values of a left eye camera and a right eye camera in the binocular camera according to the homography matrix, wherein the left eye internal and external references are respectively marked as P1 and K1, and the right eye internal and external references are marked as P2 and K2;
b. optimizing the initial value by using a maximum likelihood estimation algorithm to obtain more accurate internal reference and external reference, wherein the left eye internal and external reference and the right eye internal and external reference are respectively marked as P1', K1', P2', K2';
c. solving distortion parameters of the monocular camera, and recording the distortion parameters as S;
d. and deriving extrinsic parameters between the two cameras by using the respective extrinsic parameters of the left eye camera and the right eye camera, namely a rotation matrix R1 and a translation matrix T1 of the right eye camera relative to the left eye camera.
5. The method for positioning the key target space in the power transmission line inspection of the unmanned aerial vehicle according to claim 3, wherein in the step 32), binocular images shot by a binocular camera carried by the unmanned aerial vehicle are corrected, and the method comprises the specific steps of image distortion correction and binocular stereo correction:
a. according to the rotation matrix R1 and the translation matrix T1 of the left eye camera to the right eye camera obtained in the step d) of the step 31), dividing the rotation matrix into two parts, respectively recording as R3 and R4, and respectively rotating the left eye image and the right eye image according to the new rotation matrix to enable the imaging planes of the left eye image and the right eye image to be coplanar;
b. by three orthogonal vectors e 1 ,e 2 ,e 3 From baseline to poleRotating the x axis of the imaging plane to be parallel to the component of the translation matrix T1 on the x axis along the optical axis which is adjusted to be parallel to the rotation vector of the line, realizing the alignment of the rows of the binocular image, further realizing the stereo correction of the binocular image, and recording the corrected binocular image as I 1 。
6. The method for positioning the space of the key target in the unmanned aerial vehicle power transmission line inspection according to claim 3, wherein in the step 33), the specific steps of improving the sparse stereo matching of the SURF algorithm are as follows:
a. first, feature point sets of the left eye image and the right eye image are denoted as Pos _ left and Pos _ right, where m and n respectively denote the number of feature points in the left eye image and the right eye image, and the feature point sets are expressed by the following expression (x' i ,y’ i ) And (x) i ,y i ) Respectively representing the coordinates of the characteristic points in the left and right eyes:
b. coarse matching is carried out on all feature points by adopting an Euclidean distance algorithm, and each feature point (x 'in the left eye image is calculated respectively' i ,y’ i ) And all the feature points (x) in the right eye image i ,y i ) The Euclidean distance L;
c. sorting all rough matching point pairs according to the ascending order of the distance, deleting many-to-one or one-to-many abnormal matching point pairs, and respectively recording the characteristic point pairs reserved from the left-eye image and the right-eye image as Pos _ left 'and Pos _ right';
d. the first K point pairs with the smallest distance among the coarse matching point pairs are selected and recorded as:
Pos_K＝{{(x’ 1 ,y’ 1 ),(x 1 ,y 1 )},{(x' 2 ,y' 2 ),(x 2 ,y 2 )},...,{(x' K ,y' K ),(x K ,y K )}}；
e. connecting the matching points in the binocular image, calculating the slope of all point pair connecting lines in Pos _ K, counting the slope of a straight line with higher occurrence probability, counting the abnormal slope with lower frequency, rejecting the corresponding matched point pairs, and updating the matching point pair set as shown in the following formula:
Pos_K new ＝{{(x l1 ,y l1 ),(x r1 ,y r1 )},{(x l2 ,y l2 ),(x r2 ,y r2 )},...,{(x ln ,y ln ),(x rn ,y rn )}}
f. and finishing sparse stereo matching according to the updated matching point pairs.
7. The method for spatially locating the key target in the unmanned aerial vehicle power transmission line inspection according to claim 1, wherein in the step (4), a target three-dimensional spatial location coordinate system conversion relationship model is constructed, and the specific step of deriving the height of the key target is as follows:
41 Using an industrial personal computer to receive flight data of the unmanned aerial vehicle in real time and extract information required by key target positioning, including longitude and latitude (L) of the unmanned aerial vehicle UAV ,B UAV ) Height h UAV Pitch angle alpha, azimuth angle beta, roll angle gamma and pitching angle theta of a camera holder;
42 With the optical center of the left eye camera as the origin and the optical axis as Z w Axis, establishing a world coordinate system O w -X w Y w Z w Is provided with O w -X' w Y' w Z' w Is the coordinate system of the eye camera when the pitch angle of the camera is 0, then the coordinate system O w -X w Y w Z w Is O w -X' w Y' w Z' w The coordinate system is X' w The axis is a coordinate system after the rotating shaft rotates anticlockwise by theta degrees, and the transformation relation is shown as the following formula:
43 Measuring coordinates (x ') of the drone in a reference world coordinate system from a GPS positioning module of the drone' uav ,y’ uav ,z’ uav ) Then, calculating the coordinates (x) of the unmanned aerial vehicle in the real world coordinate system based on the formula in the step 42) uav ,y uav ,z uav )；
44 Coordinate system O with the pitching angle alpha of the unmanned plane, the rolling angle gamma and the pitching angle theta of the camera all being 0 w -X” w Y” w Z” w And recording the coordinate system with the pitch angle theta in the step 42) as a reference world coordinate system, and recording the coordinate system with the pitch angle theta in the step 42) as an actual world coordinate system, and constructing a conversion relation between the reference world coordinate system and the actual world coordinate system, wherein the conversion relation is shown as the following formula:
45 Let (x) ob ,y ob ,z ob ) Representing the coordinates of the object in the real world coordinate system from the coordinates (x) in the real world coordinate system calculated in step 43) uav ,y uav ,z uav ) And the coordinates (x) of the target in the real world coordinate system ob ,y ob ,z ob ) Obtaining the coordinates (x) of the key target and the unmanned aerial vehicle in the reference world coordinate system by using the coordinate system conversion formula provided in the step 44) " ob ,y” ob ,z” ob ) And (x) " UAV ,y” UAV ,z” UAV )；
46 According to the principle that the Y 'axis of the reference world coordinate system points to the geocentric vertically, the height difference between the unmanned aerial vehicle and the key target is calculated, and the calculation formula is shown as the following formula, wherein Y' in Representing the coordinates of the key target on the Y "axis in the reference coordinate system:
Δh＝y' UAV -y” in ；
47 According to the formula of the height difference in the step 46) and the height of the unmanned aerial vehicle in the actual coordinate system, calculating the height of the key target to be measured from the ground, wherein the calculation formula is shown as the following formula, h UAV Height to ground representative of drone:
h ob ＝h UAV +Δh。
8. the method for spatially locating the key target in the unmanned aerial vehicle power transmission line inspection according to claim 1, wherein in the step (5), a conversion model of the geographical longitude and latitude, the height, the azimuth and the key target of the unmanned aerial vehicle is constructed, and the specific step of calculating the geographical coordinates, namely the longitude and latitude, of the key target is as follows:
51 Firstly, a geodetic coordinate system is constructed by taking a binocular camera carried by the unmanned aerial vehicle as an origin, and points in the coordinate system are described by longitude (L), latitude (B) and height (H);
52 Converting the geodetic coordinates into spherical coordinates according to the geometric relationship between the geodetic coordinate system and the spherical coordinate system;
53 According to the geometric relationship between the sphere center coordinate system and the station-centered horizon direct coordinate system, the sphere center coordinate (X) e ,Y e ,Z e ) Converting into a horizontal rectangular coordinate (X) of the station center p ,Y p ,Z p )；
54 Constructing a station center straight angular coordinate system by taking the current position of the unmanned aerial vehicle as an origin, and constructing a conversion relation between a reference world coordinate system and a station center ground straight angular coordinate system;
55 According to the coordinate conversion results of the steps 52), 53) and 54), the coordinate (x) of the key target to be measured in the reference world coordinate system is determined " ob ,y” ob ,z” ob ) Converting into a horizontal coordinate system of the station center to obtain a corresponding horizontal coordinate (x) of the station center p ,y p ,z p ) Then, the horizontal coordinate of the station center is converted into a corresponding rectangular coordinate system of the sphere center to obtain a rectangular coordinate (x) of the sphere center e ,y e ,z e ) And finally, converting the spherical center rectangular coordinate into a geodetic coordinate system, and calculating to obtain the longitude and latitude of the key target to be measured, wherein a longitude and latitude calculation formula is shown as the following formula:
solving the final longitude and latitude of the key target by using an iteration method, wherein the iteration initial value is as follows, (x) e ,y e ,z e ) Representing the rectangular coordinates of the center of sphere, a, b representing the semi-major and semi-minor axes of the ellipseLength of shaft:
the iteration formula is as follows, and when the dimension B reaches the required precision, the iteration is terminated:
9. the method for positioning the space of the key target in the unmanned aerial vehicle power transmission line inspection according to claim 8, wherein in the step 52), the specific step of converting the geodetic coordinates into the spherical center coordinates comprises the following steps:
a. let geodetic coordinates of the survey station P be (L, B, H), and coordinates thereof in the geodetic coordinate system be (X) e ,Y e ,Z e ) The coordinates in the sphere center coordinate system are (x, y); converting coordinates in the geodetic coordinate system into a spherical center coordinate system by using a trigonometric relationship as follows:
b. according to the elliptical equation of the meridian, the first eccentricity and the ellipsoid long radius and the ellipsoid short radius corresponding to the geodetic coordinate system, calculating to obtain a primary conversion result x and y, wherein the elliptical equation of the meridian is
A first eccentricity of
The conversion formula is shown in the following formula, wherein
Expressing the curvature radius of the prime and unitary ring of the point, the lengths of the semi-long axis and the semi-short axis of the ellipse a and b:
c. integrating the step a and the step b to realize the conversion from the geodetic coordinate system to the spherical center rectangular coordinate system, wherein the conversion formula after the integration is as follows:
10. the method for locating the space of the key target in the electric transmission line inspection of the unmanned aerial vehicle according to claim 8,
in step 53), the center coordinates (X) of the sphere are determined e ,Y e ,Z e ) Converted into a horizontal rectangular coordinate (X) of the center of the station p ,Y p ,Z p ) The method comprises the following specific steps:
a. firstly, performing translation operation on a sphere center rectangular coordinate to enable the original points of two coordinate systems to be overlapped;
b. winding the sphere center coordinate system around Z e Axial rotation (90 ° + L 0 ) Degree, around X e Shaft rotation (90-beta) 0 ) Rotate twice in sequence, L 0 And beta 0 The longitude and latitude of the origin of the station center ground flat rectangular coordinate system are respectively, and the conversion relationship is as follows:
wherein (X) 0 ,Y 0 ,Z 0 ) Is the coordinate of the origin of the rectangular coordinates of the station center under the rectangular coordinates of the sphere center, R x And R z Represents X e Axis and Z e The rotation transformation of (1):
in step 54), the center coordinates (X) of the sphere are determined e ,Y e ,Z e ) Converted into a horizontal rectangular coordinate (X) of the center of the station p ,Y p ,Z p ) The method comprises the following specific steps:
a. firstly, a reference world coordinate system O is set w -X” w Y” w Z” w Performing translation to make the coordinate origin O w And O p Overlapping;
b. however, the device is not suitable for use in a kitchenThen winding the reference world coordinate system around Y " w Rotation of the axis by-beta degrees, rewinding by X " w The shaft rotates 90 DEG, and the conversion relation is
Wherein beta is the azimuth angle of the unmanned plane, (x " UAV ,y” UAV ,z” UAV ) Representing unmanned plane, namely origin O of station center ground flat rectangular coordinate system p In a reference world coordinate system O w -X” w Y” w Z” w Coordinates of (2).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210921547.XA CN115328181A (en) | 2022-08-02 | 2022-08-02 | Method for positioning key target space in unmanned aerial vehicle power transmission line inspection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210921547.XA CN115328181A (en) | 2022-08-02 | 2022-08-02 | Method for positioning key target space in unmanned aerial vehicle power transmission line inspection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115328181A true CN115328181A (en) | 2022-11-11 |
Family
ID=83918736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210921547.XA Pending CN115328181A (en) | 2022-08-02 | 2022-08-02 | Method for positioning key target space in unmanned aerial vehicle power transmission line inspection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115328181A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117437563A (en) * | 2023-12-13 | 2024-01-23 | 黑龙江惠达科技股份有限公司 | Plant protection unmanned aerial vehicle dotting method, device and equipment based on binocular vision |
-
2022
- 2022-08-02 CN CN202210921547.XA patent/CN115328181A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117437563A (en) * | 2023-12-13 | 2024-01-23 | 黑龙江惠达科技股份有限公司 | Plant protection unmanned aerial vehicle dotting method, device and equipment based on binocular vision |
| CN117437563B (en) * | 2023-12-13 | 2024-03-15 | 黑龙江惠达科技股份有限公司 | Plant protection unmanned aerial vehicle dotting method, device and equipment based on binocular vision |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022170878A1 (en) | System and method for measuring distance between transmission line and image by unmanned aerial vehicle | |
| CN111473739B (en) | Video monitoring-based surrounding rock deformation real-time monitoring method for tunnel collapse area | |
| CN110044300B (en) | Amphibious three-dimensional vision detection device and detection method based on laser | |
| EP1242966B1 (en) | Spherical rectification of image pairs | |
| CN110319772B (en) | Visual large-span distance measurement method based on unmanned aerial vehicle | |
| CN111369630A (en) | Method for calibrating multi-line laser radar and camera | |
| CN106645205A (en) | Unmanned aerial vehicle bridge bottom surface crack detection method and system | |
| CN113850126A (en) | Target detection and three-dimensional positioning method and system based on unmanned aerial vehicle | |
| CN108665499B (en) | Near distance airplane pose measuring method based on parallax method | |
| CN105424006A (en) | Unmanned aerial vehicle hovering precision measurement method based on binocular vision | |
| CN106767720A (en) | Single-lens oblique photograph measuring method, device and system based on unmanned plane | |
| CN113516708B (en) | Power transmission line inspection unmanned aerial vehicle accurate positioning system and method based on image recognition and UWB positioning fusion | |
| CN103278138A (en) | Method for measuring three-dimensional position and posture of thin component with complex structure | |
| CN106019264A (en) | Binocular vision based UAV (Unmanned Aerial Vehicle) danger vehicle distance identifying system and method | |
| CN110706273B (en) | Real-time collapse area measurement method based on unmanned aerial vehicle | |
| Mader et al. | UAV-based acquisition of 3D point cloud–a comparison of a low-cost laser scanner and SFM-tools | |
| CN114743021A (en) | Fusion method and system of power transmission line image and point cloud data | |
| CN114526710A (en) | Sea surface measuring system, sea surface measuring method, and storage medium | |
| He et al. | Automated relative orientation of UAV-based imagery in the presence of prior information for the flight trajectory | |
| CN111244822A (en) | Fixed-wing unmanned aerial vehicle line patrol method, system and device in complex geographic environment | |
| CN115328181A (en) | Method for positioning key target space in unmanned aerial vehicle power transmission line inspection | |
| CN116385504A (en) | Inspection and ranging method based on unmanned aerial vehicle acquisition point cloud and image registration | |
| CN110030979B (en) | Spatial non-cooperative target relative pose measurement method based on sequence images | |
| Bertram et al. | Generation the 3D model building by using the quadcopter | |
| CN113340272B (en) | Ground target real-time positioning method based on micro-group of unmanned aerial vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |