WO2001071662A2

WO2001071662A2 - System and method for globally aligning individual views based on non-accurate repeatable positioning devices

Info

Publication number: WO2001071662A2
Application number: PCT/IB2001/000654
Authority: WO
Inventors: Meny Benady
Original assignee: Cognitens, Ltd.
Priority date: 2000-03-20
Filing date: 2001-03-16
Publication date: 2001-09-27
Also published as: WO2001071662A3; EP1330783A2

Abstract

A system is described for aligning a reconstructed three-dimensional object local coordinate system in a scene to a global coordinate system, comprising an image recording module and an image processor module (11 and 15). The image recording module is configured to (i) during a mapping phase, record at least one set of images from a respective location relative to a trial scene along with at least one marker, the set of images sufficient to allow a reconstruction of the at least one marker in a local coordinate system; and (ii) during an operational phase, record at least one set of images of a working scene from the respective location, the set of image being sufficient to allow a reconstruction of at least one portion of at one working object (11). The image processor module is configured to (i) during the mapping phase, reconstruct from the set of images recorded during the mapping phase the at least one marker in the local coordinate system and determine a relationship between the local coordinate system and the global coordinate system using information identifying the coordinate of the at least one marker in the global coordinate system; and (ii) during the operational phase, reconstruct from the set of images recorded during the mapping phase the at least one portion of the at least one working object in the local coordinate system determined during the operational phase, align the reconstructed at least one portion to the global coordinate system (15).

Description

SYSTEM AND METHOD FOR GLOBALLY ALIGNING INDIVIDUAL VIEWS BASED ON NON- ACCURATE REPEAT ABLE POSITIONING DEVICES

FIELD OF THE INVENTION

This invention relates generally to the field of reconstructing and/or manipulating surface geometries of one or more three dimensional objects in a scene, from a plurality of two dimensional images of the scene, and more particularly to a system and method for relating local coordinate systems associated with reconstructions of at least a portion of the scene to, for example, a global coordinate system.

BACKGROUND OF THE INVENTION

Reconstruction and/or manipulation (generally, "reconstruction") of surface features of three- dimensional object(s) in a scene, from a plurality of two-dimensional images of the object(s), is useful in a number of applications. Illustrative applications include, for example, inspection to ensure that the object(s) were properly manufactured. U. S. Patent No. 6,167,151, issued December 26, 2000, in the names of Dan Albeck, et al., and entitled "Apparatus And Method For 3- Dimensional Surface Geometry Reconstruction" (hereinafter referred to as the "Albeck patent") describes an apparatus for performing such reconstruction using a rig including an optical head comprising three cameras, using a tensor arrangement described in U. S. Patent No. 5,821,943, issued October 13, 1998, in the name of Amnon Shashua, and entitled "Apparatus And Method For Recreating And Manipulating A 3D Object Based On A 2D Projection Thereof (hereinafter referred to as the "Shashua patent") to generate information regarding reconstruction for the features of the object(s) from three images generated by the cameras. In the arrangement described in the Shashua application, the surface features that are reconstructed are defined by points that have coordinates in a coordinate system relative to one of the cameras in the rig. A problem arises in reconstruction if the surface features that are to be reconstructed cannot all be recorded by all of the cameras with the rig in one position. The apparatus described in the Albeck patent provides a mechanism for moving the rig so as to allow the cameras to record sets of images of various portions of the surface(s) of the object(s). However, when the rig moves from one location, in which the cameras record a set of images of one portion of the object(s), to another location, in which the cameras record another set of images of another portion of the object(s), the coordinate system for the points defining the surface features of the various portions of the object(s) also changes. Each location is associated with a "local" coordinate system. In order to utilize the reconstruction information generated in the two local coordinate systems in a unitary manner in connection with the object(s) in the scene, it is necessary to relate the local coordinate systems to a common global coordinate system, which will allow all of the points of the various portions of the reconstructed object(s) to be related to the global coordinate system, effectively "stitching" the reconstructions together. The global coordinate system can conveniently be one of the two local coordinate systems that were used in the reconstruction, or it can be a third coordinate system, but in any case all of the points for the various portions of the reconstructed object need to be related to the global coordinate system. When a rig, such as the rig described in the Albeck patent is moved from one position to another to facilitate recording of sets of images of different portions of the object(s) for use in generating respective reconstructions in respective local coordinate systems, the movement comprises one or both of a translational component and a rotational component, both in three dimensions. If the translational and rotational components are known, and, during a movement, are controlled sufficiently precisely, the translational and rotational components can be used to relate the reconstruction in the local coordinate system after the movement to the local coordinate system prior to the movement, or vice versa. However, this may be difficult for a number of reasons. If, for example, the mass of the rig is sufficiently large, in a number of applications the movement of the rig cannot readily be controlled sufficiently precisely to relate the two sets of local coordinate systems. In addition, the accuracy of movement tends to be relatively sensitive to environmental conditions, which can change. Furthermore, the calibration of monitoring arrangements that monitor movement may change over time, and may differ for different positions or directions of motion. On the other hand, in systems in which movement can be controlled highly accurately, the motion tends to be relatively slow, in which case reconstructions from several positions may require a relatively long time to perform.

As an alternative to relying on accuracy of motion to facilitate relating two local coordinate systems, it may be possible to determine the relationship between local coordinate systems by using markers that are provided on, for example, the object(s) in the scene. In that case, it will be necessary to ensure that the markers are visible in views recorded in both coordinate systems. In generating the reconstruction, the coordinates of the markers in both local coordinate systems are determined. Since the markers actually comprise the same points in both local coordinate systems, the translational and rotational components relating the two local coordinate systems can be determined, which, in turn, can be used to relate the reconstructions in the two local coordinate systems. Generally, for a movement that is not constrained as to translation and rotation, reconstructions of the same three markers will need to be provided in both local coordinate systems in order to determine the translational and rotational components required to relate the two local coordinate systems. A problem arises, however, in that, in many applications it is not always possible to provide the object(s) in the scene with sufficient markers to facilitate use of this methodology in relating the local coordinate systems for respective reconstructions.

SUMMARY OF THE INVENTION

The invention provides a new and improved system and method for relating local coordinate systems associated with reconstructions of at least a portion of the scene to, for example, a global coordinate system.

In brief summary, the invention provides a system is described for aligning a reconstructed three-dimensional object local coordinate system in a scene to a global coordinate system, comprising an image recording module and an image processor module. The image recording module is configured to (i) during a mapping phase, record at least one set of images from a respective location relative to a trial scene along with at least one marker, the set of images sufficient to allow a reconstruction of the at least one marker in a local coordinate system; and (ii) during an operational phase, record at least one set of images of a working scene from said respective location, the set of image being sufficient to allow a reconstruction of at least one portion of at one working object. The image processor module is configured to (i) during the mapping phase, reconstruct from the set of images recorded during the mapping phase the at least one marker in the local coordinate system and determine a relationship between the local coordinate system and the global coordinate system using information identifying the coordinates of the at least one marker in the global coordinate system; and (ii) during the operational phase, reconstruct from the set of images recorded during the mapping phase the at least one portion of the at least one working object in the local coordinate system and, using the relationship between the local coordinate system and the global coordinate system determined during the operational phase, align the reconstructed at least one portion to the global coordinate system.

BRIEF DESCRIPTION OF THE DRAWINGS

This invention is pointed out with particularity in the appended claims. The above and further advantages of this invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which: FIG. 1 schematically depicts one embodiment of a system 10, including a non-accurate, repeatable positioning device, for aligning a locally-reconstructed three-dimensional object to a global coordinate system, constructed in accordance with the invention;

FIG. 2 is a flow chart depicting operations performed by the system 10 depicted in FIG. 1 in accordance with the invention.

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT

FIG. 1 schematically depicts one embodiment of a system 10, including a non-accurate, repeatable positioning device, for aligning a locally-reconstructed three-dimensional object to a global coordinate system, constructed in accordance with the invention. With reference to FIG. 1 , system 10 includes a rig 11 that supports an optical head 12 that comprises a plurality of cameras 12(0) through 12(C) (generally referred to by reference numeral 12(c)) mounted on a common support 13. The cameras 12(c) are preferably focused on generally the same portion of a scene, including the same portion(s) of the surface(s) of one or more objects generally identified by reference numeral 14 in the scene, thereby to facilitate recording of a set of two-dimensional images of the respective portion(s) by the respective cameras 12(c). The cameras 12(c) provide the set of recorded images to an image processor 15 which generates, from the images provided thereto by the cameras 12(c), information relating to a three-dimensional reconstruction of surface features of the respective object(s) 14 in the scene, for the respective portion(s) to which the cameras 12(c) are focused. In one embodiment, the rig 11 and optical head are similar to the respective elements of the apparatus described in the aforementioned Albeck patent. In that embodiment, the cameras 12(c) preferably include CCD ("charge-coupled device") image recording devices, which provide image information in digital form to the image processor 15 for processing. In addition, in that embodiment, the image processor 15 includes suitably-programmed computing devices (such as a suitably-programmed general purpose computer) that can generate the three-dimensional surface information from the set of two-dimensional images in a manner similar to that described in, for example, the aforementioned Shashua patent.

The system 10 is further provided with a motor 16 which, under control of a control processor 17, can move the rig 11 to facilitate direction of the cameras 12(c) to another portion of the scene 14, including respective other portion(s) of the surface(s) of respective object(s) in the scene 14 from which position the cameras 12(c) can record a second set of images. The rig 11 before one such illustrative movement, indicated in FIG. 1 by the arrow associated with reference numeral 18, is depicted in solid lines in FIG. 1 , and rig 11 after the illustrative movement 18 depicted in dashed lines in FIG. 1. Generally, the control processor 17 can control the motor 16 in a "position repeatable" manner; that is, the control processor 17 can control the motor 16 so as to position the rig 11 in a location that it has visited, such as the respective positions before and after the illustrative movement 18, but they may not be able to identify the specific location and angular orientation of the rig 11 relative to, for example, a selected coordinate system which may, but need not, comprise the global coordinate system. In addition, the set of images recorded by the respective cameras 12(c) prior to the illustrative movement 18 of rig 11 is represented by images 20(A)(0) through 20(A)(C) (generally identified by reference numeral 20( A)(c)), and the set of images recorded by the respective cameras 12(c) after the illustrative movement 18 of rig 11 is represented by images 20(B)(0) through 20(B)(C) (generally identified by reference numeral 20(B)(c)). The respective sets of images 20(A)(c) and 20(B)(c) are preferably not of coincident portions of the surface(s) of the object(s) 14 in the scene. Instead, the sets of images 20(A)(c) and 20(B)(c) will preferably be of different portions of the respective surface(s). The portions of the respective surface(s) represented by the sets of images 20(A)(c) and 20(B)(c) may, but need not, overlap.

The image processor 15 processes the images 20(A)(c) provided by the cameras 12(c) prior to the illustrative movement 18 to generate the three-dimensional surface information relating to the portion(s) of the object(s) in the scene 14 depicted by the images 20(A)(c), effectively generating reconstructions the portions of the object(s) depicted in those images 20(A)(c). In addition, the image processor 15 processes the images 20(B)(c) provided by the cameras 12(c) after the illustrative movement 18 to generate the three-dimensional surface information relating to the portion(s) of the object(s) depicted by images 20(B)(c), also effectively generating reconstructions of portions of the object(s) that are depicted in those image 20(B)(c). In one embodiment, operations performed by the image processor 15 in generating the three-dimensional surface information, in connection with the images recorded both before and after the illustrative movement 18, may correspond to those operations described in, for example, the aforementioned Shashua patent, and the information so generated may represent a three-dimensional reconstruction of the portion(s) of the object(s) depicted in the respective set of images. In each case, the respective reconstruction is in the form of a set of points whose positions are specified in the respective the local coordinate systems. To be able to utilize both sets of points in a unitary fashion, it is desirable to associate both of them to a common global coordinate system. The particular global coordinate system that is selected is not important. For simplicity and without loss of generality, the global coordinate system can be selected to be the local coordinate system before the illustrative movement 18 or after the illustrative movement 18. On the other hand, the global coordinate system may comprise a coordinate system separate and apart from any of the local coordinate systems.

Generally, the relation between the global coordinate system and the local coordinate systems consists of one or both of a translational component and/or a rotational component, both in three dimensional space. The translational component relates the position of the origin of one coordinate system to the origin of another coordinate system. Similarly, the rotational component relates the angular positions of the axes of the one coordinate system to the angular positions of the axes of the other coordinate system. If the one coordinate system is one of the local coordinate systems, and the other coordinate system is the global coordinate system, the translational component essentially comprises the coordinates of the origin of the local coordinate system in the global coordinate system. Similarly, the rotational component identifies the angular orientations of the axes defining the local coordinate system to the respective axes defining the global coordinate system. It will be appreciated that it will normally be sufficient to specify the angular orientations of two of the axes in the local coordinate system relative to the axes of the global coordinate system, since that will generally also serve to specify the angular orientations of the third axis in the respective coordinate systems, particularly if the axes comprising the respective coordinate systems are orthogonal to each other. Accordingly, nine values, three associated with the translational component and six associated with the rotational component, will suffice to specify the relationship between the two coordinate systems.

It will be appreciated that, if the translational position and angular orientation of the rig 11 before and after the illustrative movement 18 can be controlled or determined (by, for example, mechanical or electronic sensors) to a sufficient degree of accuracy they (that is, the position and angular orientation) can be used to define the translational and rotational components to relate the coordinate system after the illustrative movement 18 to the coordinate system before the illustrative movement 18. The invention provides an arrangement by which system 10 can provide that reconstructions in respective local coordinate systems can be related to a global coordinate system, even if the system 10 includes an arrangement for positioning the optical head 12 that is relatively inaccurate, but whose positioning is repeatable. According to the invention, the system 10 operates in essentially two phases, including a mapping phase and an operational phase. During the mapping phase, the scene is provided with one or more trial objects whose surfaces are generally similar to the surfaces of the objects that will comprise the scene 14 during the operational phase. In addition, during the mapping phase, the surfaces of the trial objects are provided with accessories, such as reflective markers, that can assist in determining the relationships between the local coordinate systems associated with the respective positions, to the global coordinate system. The coordinates of the markers in the global coordinate system will also be determined. Generally, during the mapping phase, the system 10 positions the optical head 12 in a plurality of locations relative to the scene 14, to facilitate recording of images from which reconstructions are generated in the respective local coordinate systems. Since the system 10 may be capable of positioning the optical head 12 with a limited accuracy, the exact position of the head may not be achieved. However, the reconstructions include the markers and the system 10 can use them to determine the translational and rotational components relating the respective local coordinate systems to the global coordinate system. The system 10 records the positions at which the images were recorded for the respective local coordinate systems, and, in addition, the translational and rotational components relating the respective local coordinate systems to the global coordinate system for use during the operational phase.

During the operational phase, the trial objects are replaced by objects, which will be referred to herein as "working objects," for which reconstructions are to be generated during the operational phase. The reconstructions generated during the operational phase may be used for any of a number of purposed, including, for example, inspection to ensure that the working objects were properly manufactured. The working objects used during the operational phase will not be provided with markers. Instead, the system 10 will place the optical head 12 at the positions at which the optical head 12 was placed during the mapping phase to facilitate recording of images from which reconstructions in the respective local coordinate systems can be generated. After the system 10 has generated the reconstructions in the respective local coordinate systems, it can use the translational and rotational components associating the respective local coordinate systems to the global coordinate system, as determined during the mapping phase, to align the reconstructions to the global coordinate system.

The system 10 may perform these operations through a series of iterations with a series of scenes comprising various working objects. To determine whether the positioning of the optical head 12 is repeatable, the scene 14, during both the mapping phase and the operational phase, may also be provided with surfaces, other than those on the trial objects or the working objects, that have markers mounted thereon. The surfaces may, for example, be in the foreground and/or background of the objects comprising the scene 14, and the markers may be used to verify the repeatability of the positioning of the optical head. With this background, operations performed by the system 10 in accordance with the invention will be described in connection with the flow chart depicted in FIG. 2. With reference to FIG. 2, after one or more trial objects along with the markers are provided to comprise the scene 14 during the mapping phase (step 100), and the coordinates of the markers determined in the global coordinate system (step 101), the system 10, in particular the motor 16 under control of the control processor 17, positions the optical head 12 in a location relative to the scene 14 (step 102). In that location, the optical head 12 will record a set of images from which a reconstruction can be generated in the respective local coordinate system associated with the location (step 103). The image processor 15 receives the set of images (step 104), generates a reconstruction of at least the markers (step 105), and determines the relationship between the local coordinate system associated with the location and the global coordinate system (step 106) for the location at which the set of images was recorded in step 103.

The control processor 17 then determines whether to move the optical head 12 to another location from which a set of images can be recorded for an additional reconstruction during the mapping phase (step 107). If the control processor 17 makes a positive determination in step 107 it will return to step 102 to position the optical head in another location and facilitate recording of the set of images in step 103. In addition, the image processor 15 can perform steps 104 through 106 in connection with the recorded images to determine the relationship between the local coordinate system for the new reconstruction and the global coordinate system. The system will perform steps 101 through 107 through one or more iterations, until it is determined that, for example, reconstructions have been generated for sufficient numbers of locations to suffice during the operational phase. At that point, the system can terminate the mapping phase and sequence to the operational phase. It will be appreciated that the control processor 17 will retain the positions of each of the locations that were visited in step 102 for the respective iterations.

Following the mapping phase, the system 10 will enter the operational phase. In that phase, one or more working objects are provided to comprise the scene 14 during the operational phase (step 110), and the system 10, in particular the motor 16 under control of the control processor 17, positions the optical head 12 in a location relative to the scene 14, the location being one of the locations that were visited during the mapping phase (step 111). In that location, the optical head 12 will record a set of images from which a reconstruction can be generated in the respective local coordinate system associated with the location (step 112). The image processor 15 will receive the set of images (step 113), generate a reconstruction of the surfaces of the test object(s) in the local coordinate system associated with the location (step 114), and, using the previously-determined relationship between the local coordinate system associated with the location and the global coordinate system, determine the coordinates of the reconstruction in the global coordinate system (step 115).

The control processor 17 then determines whether to move the optical head 12 to another location from which a set of images can be recorded for an additional reconstruction during the operational phase (step 116). If the control processor 17 makes a positive determination, it will return to step 111 to move the optical head 12 to another location from among the locations that were visited during the mapping phase. The system 19 can repeat operations 111 through 115 through a plurality of iterations to generate a reconstructions, in the global coordinate system, of the working objects.

It will be appreciated that the system 10 can repeat the operational phase through a plurality of iterations each associated with, for example, one or more object(s) comprising scene 14.

The invention provides a number of advantages. In particular, the invention provides an arrangement by which system 10 can provide that reconstructions in respective local coordinate systems can be related to a global coordinate system, even if the system 10 includes an arrangement for positioning the optical head 12 that is relatively inaccurate, but whose positioning is repeatable.

It will be appreciated that numerous changes and modifications may be made to the system 10 as described above. For example, As described above, when the system 10 re-positions the optical head 12, if it does so using a general movement, that is, a movement in which any of the translational and rotational components of the position of the optical head 12 can change, three markers will be necessary to determine the components of the movement and, thus, the position of the optical head 12 in the global coordinate system after the movement. On the other hand, if the movement is such that the optical head 12 is constrained to move along, for example, one axis in the local coordinate system, one marker will suffice, providing the orientation of the axes is known. Thus, for example, if the system 10 determines the positions and angular orientations of the optical head 12 at a series of locations P„ P₂,.-._> P_K (generally P,) for a series of movements, and, for a particular location P, performs a movement to a location P,' in which it constrains motion along a particular axis, then one marker visible in images recorded at both locations P, and P,' will suffice to determine the movement along the particular axis. It will be appreciated that, in that case, the movement will be a translation along that axis, and using the position and angular orientation determined for location P, in global coordinates during the mapping phase, the system can readily determine the position and angular orientation for location P,' in global coordinates.

During the operational phase, the system 10 can, as described above, using its positioning repeatability capability, position the optical head 12 at any of the locations P„ P₂,...P_K, as well as location P,', and with the associated angular orientation relative to the scene. On the other hand, the system 10 can position the optical head 12 at location P, using its positioning repeatability capability, and thereafter move the optical head 12 to location P,¹ by translating it along the particular axis by an amount determined during the mapping phase. Similarly, the system 10 can position the optical head 12 at location P,' using its positioning repeatability capability, and thereafter move the optical head to location P, by translating it along the particular axis by the amount determined during the mapping phase, but in the opposite direction. Generally, the positioning of the optical head 12 in this manner will be more accurate than for a general movement that can include translation along and rotation around all of the axes in the global coordinate system.

As a further modification, when the system moves the optical head 12 between locations P, and P,' as described immediately above, the movement can be constrained to any number "K" of axes, such that "K" is less than "N," the total number of axes in the local coordinate system.

In addition, as described above, the system 10 can move the optical head 12 to enable images of the scene 14 to be recorded from a plurality of positions. Alternatively, the system 10 can maintain the optical head 12 in a particular position and move the scene 14 using, for example, a robot manipulator. As a further alternative, the system 10 can move the optical head 12 and scene 14 relative to each other.

It will be appreciated that a system in accordance with the invention can be constructed in whole or in part from special purpose hardware or a general purpose computer system, or any combination thereof, any portion of which may be controlled by a suitable program. Any program may in whole or in part comprise part of or be stored on the system in a conventional manner, or it may in whole or in part be provided in to the system over a network or other mechanism for transferring information in a conventional manner. In addition, it will be appreciated that the system may be operated and/or otherwise controlled by means of information provided by an operator using operator input elements (not shown) which may be connected directly to the system or which may transfer the information to the system over a network or other mechanism for transferring information in a conventional manner. The foregoing description has been limited to a specific embodiment of this invention. It will be apparent, however, that various variations and modifications may be made to the invention, with the attainment of some or all of the advantages of the invention. It is the object of the appended claims to cover these and such other variations and modifications as come within the true spirit and scope of the invention.

What is claimed as new and desired to be secured by Letters Patent of the United States is:

Claims

1. A system for aligning a reconstructed three-dimensional object local coordinate system in a scene to a global coordinate system comprising:

A. an image recording module configured to

(i) during a mapping phase, record at least one set of images from a respective location relative to a trial scene along with at least one marker, the set of images sufficient to allow a reconstruction of the at least one marker in a local coordinate system; and

(ii) during an operational phase, record at least one set of images of a working scene from said respective location, the set of image being sufficient to allow a reconstruction of at least one portion of at one working object; and

B. an image processor module configured to

(i) during the mapping phase, reconstruct from the set of images recorded during the mapping phase the at least one marker in the local coordinate system and determine a relationship between the local coordinate system and the global coordinate system using information identifying the coordinates of the at least one marker in the global coordinate system; and

(ii) during the operational phase, reconstruct from the set of images recorded during the mapping phase the at least one portion of the at least one working object in the local coordinate system and, using the relationship between the local coordinate system and the global coordinate system determined during the operational phase, align the reconstructed at least one portion to the global coordinate system.

2. A system as defined in claim 1 in which the image recording module includes an optical head including at least one image recording device and a rig configured to position the optical head at at least one location.

3. A system as defined in claim 2 in which the rig is configured to position the optical head at a plurality of locations relative to said scene.

4. A system as defined in claim 3 in which the optical head is configured to record at least one image at each of said plurality of locations relative to said scene.

5. A system as defined in claim 4 in which the optical head is configured to record a plurality of images at each of said plurality of locations relative to said scene, said plurality of image corresponding to a set of images.

6. A system as defined in claim 3 in which the image recording module is configured to move the optical head to said scene.

7. A system as defined in claim 3 in which the image recording module is configured to move the scene relative to the optical head.

8. A system as defined in claim 3 in which the image recording module is configured to move both the scene and the optical head relative to each other.

9. A system as defined in claim 3 in which, when the rig is configured to, when moving the optical head from at least one location to a second location, the movement is constrained to be along one axis of a coordinate system, the translational component of movement along the one axis being determined during the mapping phase, the rig using the translational component in moving the optical head between the at least one location and the second location during the operational phase.

10. A system as defined in claim 9 in which the rig uses the translational component in moving the optical head from the at least one location to the second location.

11. A system as defined in claim 10 in which the rig uses the translational component in moving the optical head from the second location to the at least one location.