CA2356618C - Sensing head and apparatus for determining the position and orientation of a target object - Google Patents
Sensing head and apparatus for determining the position and orientation of a target object Download PDFInfo
- Publication number
- CA2356618C CA2356618C CA002356618A CA2356618A CA2356618C CA 2356618 C CA2356618 C CA 2356618C CA 002356618 A CA002356618 A CA 002356618A CA 2356618 A CA2356618 A CA 2356618A CA 2356618 C CA2356618 C CA 2356618C
- Authority
- CA
- Canada
- Prior art keywords
- dimensional
- tri
- target object
- light source
- dimensional data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
- G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
- G01S5/163—Determination of attitude
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/46—Indirect determination of position data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Electromagnetism (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
A sensing head for providing bi-dimensional and tri-dimensional data determinative of a position and an orientation of a target object is provided. The sensing head combines both a bi-dimensional and a tri-dimensional sensors. General illumination is associated to the bi-dimensional sensor to allow the acquisition of bi-dimensional data representative of the target object. A stripe--shaped laser light output is projected and diffused on the target object and is sensed by the tri-dimensional sensor. Tri-dimensional data is thus acquired. By proper processing of the bi-dimensional and tri-dimensional data, the position and orientation of the target object may be determined without moving the sensor head and without knowing its position and orientation.
Description
SENSING HEAD AND APPARATUS FOR DETERMINING THE POSITION AND
ORIENTATION OF A TARGET OBJECT
FIELD OF THE INVENTION
The present invention relates to the field of optical sensors and more particularly concerns a more efficient sensor for determining the position and orientation of a target object.
BACKGROUND OF THE INVENTION
Optical sensors are useful tools for position and orientation measurements of objects of various types and sizes.
In particular, in the automobile industry where an automobile body must be inspected for defects during the manufacturing process, such sensors are used to collect measurements of the position and orientation of selected objects on the car body. The resulting data is stored in a database for analysis. This type of inspection is carried out using an articulated robot with a laser-camera mounted on its wrist, the laser-camera providing the measurements. The robot is moved over the car body and the camera takes measurements of selected objects. The robot can move in any direction and orientation over a selected feature of the car body.
2o However, in order to measure position and orientation, the robot has to go to the selected feature area and rotate, tilt and turn the camera in several directions in order to capture all the measurements needed for the database. Even with these robot movements, the camera will not be able to measure all the angles needed for a proper characterization of the object.
The above explained inspection process has two important drawbacks.
Firstly, for a given cycle, the inspection time is lengthen by the many movements of the camera necessary in order to capture the angles needed to provide the required data. Secondly, in order to take orientation measurements, the camera must be moved in several angles. These movements are sometimes impossible 3o because other objects can be in the way of the robot and camera. The geometry of the car body becomes a limiting factor.
ORIENTATION OF A TARGET OBJECT
FIELD OF THE INVENTION
The present invention relates to the field of optical sensors and more particularly concerns a more efficient sensor for determining the position and orientation of a target object.
BACKGROUND OF THE INVENTION
Optical sensors are useful tools for position and orientation measurements of objects of various types and sizes.
In particular, in the automobile industry where an automobile body must be inspected for defects during the manufacturing process, such sensors are used to collect measurements of the position and orientation of selected objects on the car body. The resulting data is stored in a database for analysis. This type of inspection is carried out using an articulated robot with a laser-camera mounted on its wrist, the laser-camera providing the measurements. The robot is moved over the car body and the camera takes measurements of selected objects. The robot can move in any direction and orientation over a selected feature of the car body.
2o However, in order to measure position and orientation, the robot has to go to the selected feature area and rotate, tilt and turn the camera in several directions in order to capture all the measurements needed for the database. Even with these robot movements, the camera will not be able to measure all the angles needed for a proper characterization of the object.
The above explained inspection process has two important drawbacks.
Firstly, for a given cycle, the inspection time is lengthen by the many movements of the camera necessary in order to capture the angles needed to provide the required data. Secondly, in order to take orientation measurements, the camera must be moved in several angles. These movements are sometimes impossible 3o because other objects can be in the way of the robot and camera. The geometry of the car body becomes a limiting factor.
2 There is therefore a need for an optical sensor providing the data necessary for describing the position and orientation of an object without moving the camera around during the measurement process and without knowing the orientation of the camera and the robot wrist, which is not the case of prior art devices.
SUMMARY OF THE INVENTION
In accordance with a first aspect of the present invention, there is provided a sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a 1o sensing area, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines;
a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and 3o having an output for transmitting the same after a single 2D acquisition and a single 3D scan; and
SUMMARY OF THE INVENTION
In accordance with a first aspect of the present invention, there is provided a sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a 1o sensing area, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines;
a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and 3o having an output for transmitting the same after a single 2D acquisition and a single 3D scan; and
3 a casing enclosing said laser light source, optical arrangement, bi-dimensional and tri-dimensional sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the general illumination light source being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
In accordance with another aspect of the present invention, there is also provided an apparatus for determining a position, dimensions and an orientation of 1o a target object extending in a sensing area, comprising:
a sensing head for providing bi-dimensional and tri-dimensional data determinative of said position, dimensions and orientation of the target object, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines; and 3a a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmifting the same after a single 2D acquisition and a single 3D scan;
control means for controlling the light sources and the sensors; and processing means for processing said bi-dimensional and tri-dimensional data received from the collector circuit to determine therefrom the position, dimensions and orientation of the target object, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
In accordance with another aspect of the present invention, there is also provided a sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a sensing area, comprising:
an arrangement of LEDs directed towards the sensing area for general illumination of the target object;
a first CMOS sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said 2o bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along a line extending in said plane, the optical arrangement having a prism through which the image of the target object acquired by the first multi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a second CMOS sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said line determinable by triangulation;
3b a collector circuit connected to the first and second CMOS sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D
scan;
and a casing enclosing said arrangement of LEDs, optical arrangement, first and second CMOS sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the arrangement of LEDs being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-1o dimensional data without any displacement of the sensing head.
Further advantages and features of the present invention will be better understood upon reading of preferred embodiments thereof with reference to the appended drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic representation of a sensing head according to a preferred embodiment of the invention.
FIG. 2 is a perspective view of the sensing head of FIG. 1 showing the fields of view of the respective sensors.
FIG. 3 is a bloc diagram of an apparatus according to a preferred embodiment of the invention.
In accordance with another aspect of the present invention, there is also provided an apparatus for determining a position, dimensions and an orientation of 1o a target object extending in a sensing area, comprising:
a sensing head for providing bi-dimensional and tri-dimensional data determinative of said position, dimensions and orientation of the target object, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines; and 3a a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmifting the same after a single 2D acquisition and a single 3D scan;
control means for controlling the light sources and the sensors; and processing means for processing said bi-dimensional and tri-dimensional data received from the collector circuit to determine therefrom the position, dimensions and orientation of the target object, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
In accordance with another aspect of the present invention, there is also provided a sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a sensing area, comprising:
an arrangement of LEDs directed towards the sensing area for general illumination of the target object;
a first CMOS sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said 2o bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along a line extending in said plane, the optical arrangement having a prism through which the image of the target object acquired by the first multi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a second CMOS sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said line determinable by triangulation;
3b a collector circuit connected to the first and second CMOS sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D
scan;
and a casing enclosing said arrangement of LEDs, optical arrangement, first and second CMOS sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the arrangement of LEDs being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-1o dimensional data without any displacement of the sensing head.
Further advantages and features of the present invention will be better understood upon reading of preferred embodiments thereof with reference to the appended drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic representation of a sensing head according to a preferred embodiment of the invention.
FIG. 2 is a perspective view of the sensing head of FIG. 1 showing the fields of view of the respective sensors.
FIG. 3 is a bloc diagram of an apparatus according to a preferred embodiment of the invention.
4 DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION
With reference to FIGs. 1 and 2, there is shown a sensing head 10 for providing data determinative of a position and an orientation of a target object 30 extending in a sensing area 32. The data is provided in two sets: bi-dimensional data representative of an image of the target object 30 along a given plane 34, and tri-dimensional data representative of tri-dimensional coordinates of the object 30 along sample lines extending in the plane 34. Upon proper processing, this data may be used to determine the required position and orientation of the target object 30. The plane 34 corresponds to the field of view of the bi-dimensional sensor, as 1o better shown in FIG. 2.
The sensing head 10 therefore has two sensors, a bi-dimensional sensor 13 and a tri-dimensional sensor 14. The bi-dimensional sensor 13 is directed towards the sensing area and cooperates with a general illumination light source 20 also directed towards the sensing area. Upon illumination of the target object 30 by the general illumination light source 20, a bi-dimensional image of the object along a plane perpendicular to the line of view of the bi-dimensional sensor 13 is acquired, defining the bi-dimensional data. The general illumination light source preferably includes an arrangement of LEDs facing the sensing area. In the preferred embodiment red colored LEDs are used to better compensate for ambient light.
The tri-dimensional data is acquired through the use of a laser light source 16 generating a light output. An optical arrangement is optically coupled to the laser light source for controllable projection of the light output along distinct lines extending in the plane defined above. Preferably, the laser source 16 is a laser stripe diode assembly so that the light output it generates is stripe-shaped.
In the illustrated embodiment, the optical arrangement includes a rotating mirror 19 positioned across the path of the light output, controllable between distinct angular positions. Each of these angular positions allows the projection of the light output along one of the distinct lines for which the tri-dimensional data is acquired.
Minimally, data should be acquired along at least two such lines. In the preferred 3o embodiment, three lines are used. A galvanometer 17 coupled to the rotating mirror 19 through a rotating shaft 18 controls the angular position of the mirror 19.
Also in the preferred embodiment, a prism 15 is provided in the path of the light output after the mirror 19 to redirect the output towards the sensing area.
The laser source 16 may be of a flying spot type provided in such a case with a suitable spot moving mechanism (not shown in the Figure).
With reference to FIGs. 1 and 2, there is shown a sensing head 10 for providing data determinative of a position and an orientation of a target object 30 extending in a sensing area 32. The data is provided in two sets: bi-dimensional data representative of an image of the target object 30 along a given plane 34, and tri-dimensional data representative of tri-dimensional coordinates of the object 30 along sample lines extending in the plane 34. Upon proper processing, this data may be used to determine the required position and orientation of the target object 30. The plane 34 corresponds to the field of view of the bi-dimensional sensor, as 1o better shown in FIG. 2.
The sensing head 10 therefore has two sensors, a bi-dimensional sensor 13 and a tri-dimensional sensor 14. The bi-dimensional sensor 13 is directed towards the sensing area and cooperates with a general illumination light source 20 also directed towards the sensing area. Upon illumination of the target object 30 by the general illumination light source 20, a bi-dimensional image of the object along a plane perpendicular to the line of view of the bi-dimensional sensor 13 is acquired, defining the bi-dimensional data. The general illumination light source preferably includes an arrangement of LEDs facing the sensing area. In the preferred embodiment red colored LEDs are used to better compensate for ambient light.
The tri-dimensional data is acquired through the use of a laser light source 16 generating a light output. An optical arrangement is optically coupled to the laser light source for controllable projection of the light output along distinct lines extending in the plane defined above. Preferably, the laser source 16 is a laser stripe diode assembly so that the light output it generates is stripe-shaped.
In the illustrated embodiment, the optical arrangement includes a rotating mirror 19 positioned across the path of the light output, controllable between distinct angular positions. Each of these angular positions allows the projection of the light output along one of the distinct lines for which the tri-dimensional data is acquired.
Minimally, data should be acquired along at least two such lines. In the preferred 3o embodiment, three lines are used. A galvanometer 17 coupled to the rotating mirror 19 through a rotating shaft 18 controls the angular position of the mirror 19.
Also in the preferred embodiment, a prism 15 is provided in the path of the light output after the mirror 19 to redirect the output towards the sensing area.
The laser source 16 may be of a flying spot type provided in such a case with a suitable spot moving mechanism (not shown in the Figure).
5 The tri-dimensional sensor 14 is directed towards the sensing area 32, and sees a volumetric field of view 35 (shown in FIG. 2). The tri-dimensional data is acquired by sensing diffusion of the light output on the target object 30.
Standard optical triangulation techniques allow the range of the object along each line to be determined from this data. Any appropriate optical components such as lenses 1o (FIG. 2) may additionally be used to properly focus incoming signals on the sensors 13 and 14.
Preferably, both the bi-dimensional and tri-dimensional sensors are CMOS
optical sensors, but any other appropriate type of sensor such as a CCD camera may also be used.
A collector circuit 21 is connected to the bi-dimensional and tri-dimensional sensors 13 and 14, and receives the bi-dimensional and tri-dimensional data therefrom. The collector circuit has an output 24 for transmitting this data.
In the preferred embodiment, a casing 23 encloses the laser light source 16, the optical arrangement, the sensors 13 and 14 and the collector circuit 21.
2o The casing 23 has a bottom opening 25 through which the light output and the view line of each sensor pass. Also preferably, the casing 23 is provided with a fastening member for attachment to a robot arm, for moving the sensing head from one target object to another between inspection cycles.
Advantageously, the sensing head according to the present invention allows the acquisition of all the data necessary for inspection in a short time, without having to move the sensing head around during the inspection cycle and without the need to know the camera orientation.
In accordance with another aspect of the present invention, there is provided an apparatus 36 for determining the position and orientation of a target object. FIG. 3 shows a bloc diagram for such an apparatus 36.
Standard optical triangulation techniques allow the range of the object along each line to be determined from this data. Any appropriate optical components such as lenses 1o (FIG. 2) may additionally be used to properly focus incoming signals on the sensors 13 and 14.
Preferably, both the bi-dimensional and tri-dimensional sensors are CMOS
optical sensors, but any other appropriate type of sensor such as a CCD camera may also be used.
A collector circuit 21 is connected to the bi-dimensional and tri-dimensional sensors 13 and 14, and receives the bi-dimensional and tri-dimensional data therefrom. The collector circuit has an output 24 for transmitting this data.
In the preferred embodiment, a casing 23 encloses the laser light source 16, the optical arrangement, the sensors 13 and 14 and the collector circuit 21.
2o The casing 23 has a bottom opening 25 through which the light output and the view line of each sensor pass. Also preferably, the casing 23 is provided with a fastening member for attachment to a robot arm, for moving the sensing head from one target object to another between inspection cycles.
Advantageously, the sensing head according to the present invention allows the acquisition of all the data necessary for inspection in a short time, without having to move the sensing head around during the inspection cycle and without the need to know the camera orientation.
In accordance with another aspect of the present invention, there is provided an apparatus 36 for determining the position and orientation of a target object. FIG. 3 shows a bloc diagram for such an apparatus 36.
6 The apparatus 36 includes a sensing head 10 as described above. Control and processing circuits are respectively provided for controlling the light sources 16 and 20 and the sensors 13 and 14, and processing the bi-dimensional and tri-dimensional data received from the collector circuit 21 to determine therefrom the position and orientation of the target object.
In the preferred embodiment, the control and the processor circuits are combined into a control unit 40. The control unit 40 has a frame grabber 42, a processor board 46, a scanner and illumination board 54, a processor 62, and a bus structure 64. The frame grabber 42 has an input 44 receiving the bi-1o dimensional data from the bi-dimensional sensor. The 3D processor board 46 has control outputs 48 and 50 respectively connected to the bi-dimensional and tri-dimensional sensors 13 and 14 and an input 52 receiving the tri-dimensional data from the tri-dimensional sensor 14. The scanner and illumination control board has control outputs 56 and 58 respectively connected to the light sources 20 and 16 and an input 60 receiving positional information from the optical arrangement.
The bus structure 64 interconnects the processor 62 with the frame grabber 42, 3D processor board 46 and illumination control board 54.
Preferably, the scanner and illumination control board 54 and the processor 62 have interfaces 66 for robot communication.
The frame grabber, the 3D processing, the scanner and illumination control and the processor function could be all embodied in a single board.
In the preferred embodiment of the invention, the data processing for determining the position and orientation of the object from the collected bi-dimensional and tri-dimensional data is as follows.
All computations made internally to the sensing head are made in a fixed point of reference with respect to the casing of the sensing head, called mechanical reference. The use of a same reference point for both sensors and for all positions in the plane allows a thorough combining of all the data acquired by the sensing head and determines therefrom the position and orientation of the target object.
In the preferred embodiment, the control and the processor circuits are combined into a control unit 40. The control unit 40 has a frame grabber 42, a processor board 46, a scanner and illumination board 54, a processor 62, and a bus structure 64. The frame grabber 42 has an input 44 receiving the bi-1o dimensional data from the bi-dimensional sensor. The 3D processor board 46 has control outputs 48 and 50 respectively connected to the bi-dimensional and tri-dimensional sensors 13 and 14 and an input 52 receiving the tri-dimensional data from the tri-dimensional sensor 14. The scanner and illumination control board has control outputs 56 and 58 respectively connected to the light sources 20 and 16 and an input 60 receiving positional information from the optical arrangement.
The bus structure 64 interconnects the processor 62 with the frame grabber 42, 3D processor board 46 and illumination control board 54.
Preferably, the scanner and illumination control board 54 and the processor 62 have interfaces 66 for robot communication.
The frame grabber, the 3D processing, the scanner and illumination control and the processor function could be all embodied in a single board.
In the preferred embodiment of the invention, the data processing for determining the position and orientation of the object from the collected bi-dimensional and tri-dimensional data is as follows.
All computations made internally to the sensing head are made in a fixed point of reference with respect to the casing of the sensing head, called mechanical reference. The use of a same reference point for both sensors and for all positions in the plane allows a thorough combining of all the data acquired by the sensing head and determines therefrom the position and orientation of the target object.
7 Each pixel of the bi-dimensional sensor is represented by a curve in space corresponding to the field of view of this pixel. Depending on the required precision and the quality of the optical components used, this curve may be a straight line, in its simplest form, or of a more complex nature such as quadratic, spline, etc.
This curve for each pixel may be represented mathematically by the following:
Xmechanical - f(Zmechanicad ymechanical - f(Zmechanicad In the case of a straight line, each pixel may be represented by the following equations:
Xmechanical - mxZmechanical +bx Ymechanical - myZmechanical + by There are therefore four matrices to consider, one for each coefficient.
The bi-dimensional image data processing determines the pixels of interest, such as, for example, the pixels holding the center point and the extremities of the target object. The row and column numbers i and j of these pixels are then used to obtain the proper matrix coefficients. In the case of data processing with sub-pixel resolution, the row and column numbers i and j take fractionai values and an interpolation of the corresponding rows and columns is done in the matrix.
In the tri-dimensional case, the tri-dimensional sensor is first calibrated for each position of a laser line, itself determined by the position of the rotating mirror.
The calibration process involves two steps. Firstly, two tables are used to transform coordinates from the (i,j) reference system of the sensor to (yoPtical, zoPt;cal) coordinates in a "laser plane" which is parallel to the light output. Secondly, since the laser plane is mobile, further tables or equations need to be used to further transform the coordinates from the optical reference system to the mechanical reference system. This transformation takes into account the rotation
This curve for each pixel may be represented mathematically by the following:
Xmechanical - f(Zmechanicad ymechanical - f(Zmechanicad In the case of a straight line, each pixel may be represented by the following equations:
Xmechanical - mxZmechanical +bx Ymechanical - myZmechanical + by There are therefore four matrices to consider, one for each coefficient.
The bi-dimensional image data processing determines the pixels of interest, such as, for example, the pixels holding the center point and the extremities of the target object. The row and column numbers i and j of these pixels are then used to obtain the proper matrix coefficients. In the case of data processing with sub-pixel resolution, the row and column numbers i and j take fractionai values and an interpolation of the corresponding rows and columns is done in the matrix.
In the tri-dimensional case, the tri-dimensional sensor is first calibrated for each position of a laser line, itself determined by the position of the rotating mirror.
The calibration process involves two steps. Firstly, two tables are used to transform coordinates from the (i,j) reference system of the sensor to (yoPtical, zoPt;cal) coordinates in a "laser plane" which is parallel to the light output. Secondly, since the laser plane is mobile, further tables or equations need to be used to further transform the coordinates from the optical reference system to the mechanical reference system. This transformation takes into account the rotation
8 and translation of the origin of the mechanical reference system with respect to the various optical reference systems.
The tri-dimensional data processing preferably processes the signal before conversion from the optical reference system to the mechanical reference system.
Only the points of interest then need to be converted in the manner described above.
The bi-dimensional and tri-dimensional data then need to be properly combined in order to extract the desired information.
In the preferred embodiment, the tri-dimensional data includes 1o measurements along at least two laser lines. For each of these lines, at least two points belonging to the object's surface are obtained from the image processing.
The coordinates of these points are transformed from the optical reference system to the mechanical reference system. The surface of the object is then mathematically represented in this new reference system. By way of example, let the object be an opening in a plane surface as would be the case for the inspection of automobile parts. The equation of the best plane passing through all the obtained points is computed and represented by an equation in the form of:
Zmechanical - mxxmechanical + myymechanical + b From the bi-dimensional image, the equations for the pixels of interest are obtained as described above. The intersection of both bi-dimensional and tri-dimensional fields of view is then obtained by resolving a system of three equations and three unknown formed by the two straight lines and the above plane. This processing is done for all points of interest. The center pixel yields the coordinates (xmechanical, ymechanical, Zmechanical) of the target object, corresponding to point P in FIG. 1. The extremity pixels are used to determine the object's size, compensate for its orientation and for the distance between the object and the bi-dimensional sensor. For the plane equation above, two of three degrees of freedom are determined with regards to the orientation of the object by computing the directing cosines of the normal to the plane. The third degree of freedom is
The tri-dimensional data processing preferably processes the signal before conversion from the optical reference system to the mechanical reference system.
Only the points of interest then need to be converted in the manner described above.
The bi-dimensional and tri-dimensional data then need to be properly combined in order to extract the desired information.
In the preferred embodiment, the tri-dimensional data includes 1o measurements along at least two laser lines. For each of these lines, at least two points belonging to the object's surface are obtained from the image processing.
The coordinates of these points are transformed from the optical reference system to the mechanical reference system. The surface of the object is then mathematically represented in this new reference system. By way of example, let the object be an opening in a plane surface as would be the case for the inspection of automobile parts. The equation of the best plane passing through all the obtained points is computed and represented by an equation in the form of:
Zmechanical - mxxmechanical + myymechanical + b From the bi-dimensional image, the equations for the pixels of interest are obtained as described above. The intersection of both bi-dimensional and tri-dimensional fields of view is then obtained by resolving a system of three equations and three unknown formed by the two straight lines and the above plane. This processing is done for all points of interest. The center pixel yields the coordinates (xmechanical, ymechanical, Zmechanical) of the target object, corresponding to point P in FIG. 1. The extremity pixels are used to determine the object's size, compensate for its orientation and for the distance between the object and the bi-dimensional sensor. For the plane equation above, two of three degrees of freedom are determined with regards to the orientation of the object by computing the directing cosines of the normal to the plane. The third degree of freedom is
9 fixed by the bi-dimensional imagery and corresponds to the orientation of the object in the plane. All the variables shown in FIG. 1 (R, Dl, D2, (X1, a2, a3) may therefore be obtained from a minimal amount of data.
In summary, the present invention provides a sensing head that can be simply positioned by a robotic arm over a target object, for example on a car body, at any angle and still be able to collect accurate measurements of its X,Y,Z
position and orientation without having to move the sensing head any further and without the need of knowing the orientation of the camera. The robot still has to move the sensing head to the feature to be measured, but it does not have to 1o move during measurement. A five-dimensional picture is taken from a known position and six coordinate values are then computed by the system using special mathematics based on a novel optical geometry design.
In the preferred embodiment, the sensing head is mounted on the articulated robot wrist. The articulated robot positions the sensing head over a target object. The sensing head takes a bi-dimensional picture and performs a tri-dimensional scan. The bi-dimensional picture may then be correlated with the tri-dimensional scan and the result is the generation of six coordinate values.
These six computed values are sent to the database.
The robot then moves on to another selected object and the above process is performed again. The data collected is cumulated in a database for analysis purposes.
It should be understood that instead of moving the camera, the camera could be set in fixed position and the robot could position the feature to be measured in front of the camera.
The present invention in its preferred embodiment provides the following advantageous features. The combination of the bi-dimensional image and the tri-dimensional scanned surface by a moving laser stripe in the same camera enables the measurement of the X,Y,Z position and orientation in space, relative to a known coordinate reference system. The sensing performs all the measurements without knowing the position or angle of the articulated robot.
All the angles and absolute positions are measured and calculated by the sensing head without having to move the robot or know it's spatial orientation and position.
The tri-dimensional images are obtained by a second sensor, thus separating the optical triangulation from the bi-dimensional optics and bi-dimensional sensor. In other systems similar to this one, the tri-dimensional images are obtained by doing 5 the triangulation on the bi-dimensional sensor and therefore the bi-dimensional and tri-dimensional images are generated by the same sensor. The advantage of having a separate sensor for the bi-dimensional and tri-dimensional images is that the processing for each one does not have to cope with the artifacts of the other one.
In summary, the present invention provides a sensing head that can be simply positioned by a robotic arm over a target object, for example on a car body, at any angle and still be able to collect accurate measurements of its X,Y,Z
position and orientation without having to move the sensing head any further and without the need of knowing the orientation of the camera. The robot still has to move the sensing head to the feature to be measured, but it does not have to 1o move during measurement. A five-dimensional picture is taken from a known position and six coordinate values are then computed by the system using special mathematics based on a novel optical geometry design.
In the preferred embodiment, the sensing head is mounted on the articulated robot wrist. The articulated robot positions the sensing head over a target object. The sensing head takes a bi-dimensional picture and performs a tri-dimensional scan. The bi-dimensional picture may then be correlated with the tri-dimensional scan and the result is the generation of six coordinate values.
These six computed values are sent to the database.
The robot then moves on to another selected object and the above process is performed again. The data collected is cumulated in a database for analysis purposes.
It should be understood that instead of moving the camera, the camera could be set in fixed position and the robot could position the feature to be measured in front of the camera.
The present invention in its preferred embodiment provides the following advantageous features. The combination of the bi-dimensional image and the tri-dimensional scanned surface by a moving laser stripe in the same camera enables the measurement of the X,Y,Z position and orientation in space, relative to a known coordinate reference system. The sensing performs all the measurements without knowing the position or angle of the articulated robot.
All the angles and absolute positions are measured and calculated by the sensing head without having to move the robot or know it's spatial orientation and position.
The tri-dimensional images are obtained by a second sensor, thus separating the optical triangulation from the bi-dimensional optics and bi-dimensional sensor. In other systems similar to this one, the tri-dimensional images are obtained by doing 5 the triangulation on the bi-dimensional sensor and therefore the bi-dimensional and tri-dimensional images are generated by the same sensor. The advantage of having a separate sensor for the bi-dimensional and tri-dimensional images is that the processing for each one does not have to cope with the artifacts of the other one.
10 While embodiments of this invention have been illustrated in the accompanying drawings and described above, it will be evident to those skilled in the art that changes and modifications may be made therein without departing from the essence of this invention. All such modifications or variations are believed to be within the scope of the invention as defined by the claims appended hereto.
Claims (15)
1. A sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a sensing area, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines;
a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D scan; and a casing enclosing said laser light source, optical arrangement, bi-dimensional and tri-dimensional sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the general illumination light source being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines;
a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D scan; and a casing enclosing said laser light source, optical arrangement, bi-dimensional and tri-dimensional sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the general illumination light source being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
2. The sensing head according to claim 1, wherein the general illumination light source comprises an arrangement of LEDs facing the sensing area.
3. The sensing head according to claim 2, wherein said LEDs generate red colored general illumination.
4. The sensing head according to claim 1, wherein said bi-dimensional sensor comprises a CMOS optical sensor.
5. The sensing head according to claim 1, wherein said laser light source comprises a laser stripe diode assembly, the light output generated by said laser light source being stripe-shaped.
6. The sensing head according to claim 5, wherein the optical arrangement comprises a controllable rotating mirror across a path of the light output, said mirror having distinct angular positions, each of which corresponding to a projection of the light output along one of the distinct lines extending in the plane.
7. The sensing head according to claim 6, wherein the optical arrangement further comprises a galvanometer connected to the collector circuit and coupled to the rotating mirror for setting said rotating mirror in each of said angular positions.
8. The sensing head according to claim 1, wherein said tri-dimensional sensor comprises a CMOS optical sensor.
9. The sensing head according to claim 1, wherein the optical element comprises a prism extending between the bi-dimensional sensor and the sensing area.
10. The sensing head according to claim 1, wherein the casing is provided with a fastening member for attachment to a robot arm.
11. An apparatus for determining a position, dimensions and an orientation of a target object extending in a sensing area, comprising:
a sensing head for providing bi-dimensional and tri-dimensional data determinative of said position, dimensions and orientation of the target object, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines; and a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D scan;
control means for controlling the light sources and the sensors; and processing means for processing said bi-dimensional and tri-dimensional data received from the collector circuit to determine therefrom the position, dimensions and orientation of the target object, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
a sensing head for providing bi-dimensional and tri-dimensional data determinative of said position, dimensions and orientation of the target object, comprising:
a general illumination light source directed towards the sensing area;
a bi-dimensional sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along distinct lines extending in said plane, the optical arrangement having an optical element through which the image of the target object acquired by the bi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a tri-dimensional sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said lines; and a collector circuit connected to the bi-dimensional and tri-dimensional sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D scan;
control means for controlling the light sources and the sensors; and processing means for processing said bi-dimensional and tri-dimensional data received from the collector circuit to determine therefrom the position, dimensions and orientation of the target object, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
12. The apparatus according to claim 11, wherein the control means and the processing means are combined into a control unit comprising a frame grabber having an input receiving the bi-dimensional data from the bi-dimensional sensor, a 3D processor board having control outputs connected to the bi-dimensional and tri-dimensional sensors and an input receiving the tri-dimensional data from the tri-dimensional sensor, a scanner and illumination control board having control outputs connected to the light sources and an input receiving positional information from the optical arrangement, a processor, and a bus structure interconnecting the processor with the frame grabber, 3D processor board and illumination control board.
13. The apparatus according to claim 12, wherein the scanner and illumination control board and the processor have interfaces for robot communication.
14. A sensing head for providing bi-dimensional and tri-dimensional data determinative of a position, dimensions and an orientation of a target object extending in a sensing area, comprising:
an arrangement of LEDs directed towards the sensing area for general illumination of the target object;
a first CMOS sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
an arrangement of LEDs directed towards the sensing area for general illumination of the target object;
a first CMOS sensor directed towards said sensing area and cooperating with the general illumination light source for acquiring the bi-dimensional data upon illumination of the target object by said general illumination light source, said bi-dimensional data being representative of an image of the target object along a plane;
a laser light source;
15 an optical arrangement optically coupled to the laser light source for controllable projection of a light output generated by the laser light source along a line extending in said plane, the optical arrangement having a prism through which the image of the target object acquired by the first multi-dimensional sensor and the light output from the laser light source pass in a same reference axis;
a second CMOS sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said line determinable by triangulation;
a collector circuit connected to the first and second CMOS sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D
scan;
and a casing enclosing said arrangement of LEDs, optical arrangement, first and second CMOS sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the arrangement of LEDs being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
a second CMOS sensor directed towards said sensing area for acquiring the tri-dimensional data by sensing diffusion of the light output on the target object, said tri-dimensional data being representative of tri-dimensional coordinates of said target object along said line determinable by triangulation;
a collector circuit connected to the first and second CMOS sensors for receiving the bi-dimensional and tri-dimensional data therefrom and having an output for transmitting the same after a single 2D acquisition and a single 3D
scan;
and a casing enclosing said arrangement of LEDs, optical arrangement, first and second CMOS sensors and collector circuit, the casing having a bottom opening through which the light output and a view line of each sensor pass, the arrangement of LEDs being disposed next to the bottom opening, whereby the sensing head forms a unitary unit adapted to provide all the bi-dimensional and tri-dimensional data without any displacement of the sensing head.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA002356618A CA2356618C (en) | 2001-09-04 | 2001-09-04 | Sensing head and apparatus for determining the position and orientation of a target object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA002356618A CA2356618C (en) | 2001-09-04 | 2001-09-04 | Sensing head and apparatus for determining the position and orientation of a target object |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2356618A1 CA2356618A1 (en) | 2003-03-04 |
| CA2356618C true CA2356618C (en) | 2007-07-10 |
Family
ID=4169895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002356618A Expired - Lifetime CA2356618C (en) | 2001-09-04 | 2001-09-04 | Sensing head and apparatus for determining the position and orientation of a target object |
Country Status (1)
| Country | Link |
|---|---|
| CA (1) | CA2356618C (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114879210B (en) * | 2022-07-12 | 2022-09-20 | 吉光半导体科技有限公司 | Target object motion monitoring method and device and computer equipment |
-
2001
- 2001-09-04 CA CA002356618A patent/CA2356618C/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| CA2356618A1 (en) | 2003-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11408728B2 (en) | Registration of three-dimensional coordinates measured on interior and exterior portions of an object | |
| US5461478A (en) | Method and apparatus for measuring three-dimensional position and orientation of an object using light projection | |
| CN109115126B (en) | Method for calibrating a triangulation sensor, control and processing unit and storage medium | |
| US6730926B2 (en) | Sensing head and apparatus for determining the position and orientation of a target object | |
| US7903261B2 (en) | Controlling a projected pattern | |
| EP0181553B1 (en) | 3-d active vision sensor | |
| JP3070953B2 (en) | Method and system for point-by-point measurement of spatial coordinates | |
| US6480287B2 (en) | Three dimensional scanning system | |
| Boochs et al. | Increasing the accuracy of untaught robot positions by means of a multi-camera system | |
| GB2544181A (en) | Three-dimensional imager that includes a dichroic camera | |
| US5991437A (en) | Modular digital audio system having individualized functional modules | |
| EP2104365A1 (en) | Method and apparatus for rapid three-dimensional restoration | |
| US20050068523A1 (en) | Calibration block and method for 3D scanner | |
| US6055054A (en) | Three dimensional inspection system | |
| CA2751878A1 (en) | Measurement of positional information for a robot arm | |
| JP2859946B2 (en) | Non-contact measuring device | |
| CA2356618C (en) | Sensing head and apparatus for determining the position and orientation of a target object | |
| JPH05248819A (en) | Calibrating method of data of calibration object for measuring three dimensional position of object by camera and measuring method of three dimensional position | |
| Clark et al. | Measuring range using a triangulation sensor with variable geometry | |
| US20220124253A1 (en) | Compensation of three-dimensional measuring instrument having an autofocus camera | |
| Araki et al. | High speed and continuous rangefinding system | |
| CN111028298B (en) | Convergent binocular system for rigid coordinate system space transformation calibration | |
| JP5079779B2 (en) | Camera calibration method and camera calibration apparatus | |
| JPH09145320A (en) | Three-dimensional input camera | |
| JP2003503684A (en) | Measuring device for measuring the position and orientation of the first part to be worked, inspected or moved |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKEX | Expiry |
Effective date: 20210907 |