US20240004080A1 - Capturing environmental scans using landmarks based on semantic features - Google Patents
Capturing environmental scans using landmarks based on semantic features Download PDFInfo
- Publication number
- US20240004080A1 US20240004080A1 US18/469,258 US202318469258A US2024004080A1 US 20240004080 A1 US20240004080 A1 US 20240004080A1 US 202318469258 A US202318469258 A US 202318469258A US 2024004080 A1 US2024004080 A1 US 2024004080A1
- Authority
- US
- United States
- Prior art keywords
- scanner
- scan
- data
- slam
- environment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007613 environmental effect Effects 0.000 title 1
- 238000005259 measurement Methods 0.000 claims abstract description 76
- 238000000034 method Methods 0.000 claims abstract description 67
- 238000004422 calculation algorithm Methods 0.000 description 26
- 238000004891 communication Methods 0.000 description 26
- 230000008569 process Effects 0.000 description 17
- 238000013507 mapping Methods 0.000 description 15
- 238000003860 storage Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 13
- 238000012545 processing Methods 0.000 description 13
- 238000004590 computer program Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 238000006073 displacement reaction Methods 0.000 description 9
- 239000013598 vector Substances 0.000 description 9
- 230000000007 visual effect Effects 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 8
- 230000008901 benefit Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 6
- 238000012937 correction Methods 0.000 description 6
- 230000001413 cellular effect Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 238000013459 approach Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- NJPPVKZQTLUDBO-UHFFFAOYSA-N novaluron Chemical compound C1=C(Cl)C(OC(F)(F)C(OC(F)(F)F)F)=CC=C1NC(=O)NC(=O)C1=C(F)C=CC=C1F NJPPVKZQTLUDBO-UHFFFAOYSA-N 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 241000350052 Daniellia ogea Species 0.000 description 3
- 238000003491 array Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 241000196324 Embryophyta Species 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000005755 formation reaction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000000523 sample Substances 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000012897 Levenberg–Marquardt algorithm Methods 0.000 description 1
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 206010039203 Road traffic accident Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000010363 phase shift Effects 0.000 description 1
- 239000011435 rock Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
- G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3804—Creation or updating of map data
- G01C21/3807—Creation or updating of map data characterised by the type of data
- G01C21/383—Indoor data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
- G01C21/3804—Creation or updating of map data
- G01C21/3833—Creation or updating of map data characterised by the source of data
- G01C21/3848—Data obtained from both position sensors and additional sensors
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/04—Systems determining the presence of a target
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4808—Evaluating distance, position or velocity data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/497—Means for monitoring or calibrating
- G01S7/4972—Alignment of sensor
Definitions
- the present application is directed to a system that optically scans an environment, such as a building, and in particular to a mobile scanning system that generates two-dimensional (2D) and three-dimensional (3D) scans of the environment.
- a TOF laser scanner is a scanner in which the distance to a target point is determined based on the speed of light in air between the scanner and a target point.
- Laser scanners are typically used for scanning closed or open spaces such as interior areas of buildings, industrial installations and tunnels. They may be used, for example, in industrial applications and accident reconstruction applications.
- a laser scanner optically scans and measures objects in a volume around the scanner through the acquisition of data points representing object surfaces within the volume.
- Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value.
- This raw scan data is collected, stored and sent to a processor or processors to generate an image, 2D/3D, representing the scanned area or object.
- Generating an image requires at least three values for each data point. These three values may include the distance and two angles, or may be transformed values, such as the x, y, z coordinates.
- an image is also based on a fourth gray-scale value, which is a value related to irradiance of scattered light returning to the scanner.
- the beam steering mechanism includes a first motor that steers the beam of light about a first axis by a first angle that is measured by a first angular encoder (or other angle transducer).
- the beam steering mechanism also includes a second motor that steers the beam of light about a second axis by a second angle that is measured by a second angular encoder (or other angle transducer).
- Many contemporary laser scanners include a camera mounted on the laser scanner for gathering camera digital images of the environment and for presenting the camera digital images to an operator of the laser scanner. By viewing the camera images, the operator of the scanner can determine the field of view of the measured volume and adjust settings on the laser scanner to measure over a larger or smaller region of space.
- the digital images from the camera may be transmitted to a processor to add color to the scanner image.
- To generate a color scanner image at least three positional coordinates (such as x, y, z) and three color values (such as red, green, blue “RGB”) are collected for each data point.
- a triangulation system such as a scanner, projects either a line of light (e.g., from a laser line probe) or a pattern of light (e.g., from a structured light) onto the surface.
- a camera is coupled to a projector in a fixed mechanical relationship.
- the light/pattern emitted from the projector is reflected off of the surface and detected by the camera. Since the camera and projector are arranged in a fixed relationship, the distance to the object may be determined from captured images using trigonometric principles.
- Triangulation systems provide advantages in quickly acquiring coordinate data over large areas.
- the scanner acquires, at different times, a series of images of the patterns of light formed on the object surface. These multiple images are then registered relative to each other so that the position and orientation of each image relative to the other images are known.
- various techniques have been used to register the images.
- One common technique uses features in the images to match overlapping areas of adjacent image frames. This technique works well when the object being measured has many features relative to the field of view of the scanner. However, if the object contains a relatively large flat or curved surface, the images may not properly register relative to each other.
- a 3D image of a scene may require multiple scans from different registration positions.
- the overlapping scans are registered in a joint coordinate system, for example, as described in U.S. Published Patent Application No. 2012/0069352 ('352), the contents of which are incorporated herein by reference.
- Such registration is performed by matching targets in overlapping regions of the multiple scans.
- the targets may be artificial targets such as spheres or checkerboards or they may be natural features such as corners or edges of walls.
- Some registration procedures involve relatively time-consuming manual procedures such as identifying by a user each target and matching the targets obtained by the scanner in each of the different registration positions.
- Some registration procedures also require establishing an external “control network” of registration targets measured by an external device such as a total station.
- an object e.g. a wall, a column, or a desk
- the TOF scanner is moved to different locations and separate scans are performed.
- the 3D coordinate data i.e. the point cloud
- Some existing measurement systems have been mounted to a movable structure, such as a cart, and moved on a continuous basis through the building to generate a digital representation of the building.
- a movable structure such as a cart
- these provide generally lower data quality than stationary scans.
- These systems tend to be more complex and require specialized personnel to perform the scan.
- the scanning equipment including the movable structure may be bulky, which could further delay the scanning process in time sensitive situations, such as a crime or accident scene investigation.
- the cart is stopped at scan locations so that the measurements can be performed. This further increases the time to scan an environment.
- a system includes a scanner device configured to capture scan-data of surrounding environment.
- the system also includes one or more processors operably coupled to the scanner, wherein the one or more processors are operable to perform simultaneous locating and mapping of the scanner device in the surrounding environment.
- Performing the simultaneous locating and mapping (SLAM) includes capture the scan-data of a portion of a map of the surrounding environment, wherein the scan-data comprises a point cloud.
- performing SLAM includes detecting a set of lines in the point cloud, and identifying a semantic feature based at least in part on the set of lines.
- Performing SLAM further includes assigning a first scan position of the scanner device in the surrounding environment at the present time t 1 as a landmark, and linking the landmark with the portion of the map. Further, performing SLAM includes determining that the scanner device has moved, at time t 2 , to the scan position that was marked as the landmark based on identifying said semantic feature in another scan-data from the scanner device. In response, a second scan position of the scanner device at time t 2 is determined. Also, a displacement vector is determined for the map based on a difference between the first scan position and the second scan position. Further, a revised second scan position is computed based on the second scan position and the displacement vector, and the scan-data is registered using the revised scan position.
- a method for performing a simultaneous location and mapping of a scanner device in a surrounding environment includes capturing the scan-data of a portion of a map of the surrounding environment, wherein the scan-data comprises a point cloud.
- the method also includes detecting a set of lines in the point cloud, and identifying a semantic feature based at least in part on the set of lines.
- the method further includes assigning a first scan position of the scanner device in the surrounding environment at the present time t 1 as a landmark, and linking the landmark with the portion of the map.
- the method further includes determining that the scanner device has moved, at time t 2 , to the scan position that was marked as the landmark based on identifying said semantic feature in another scan-data from the scanner device.
- a second scan position of the scanner device at time t 2 is determined. Also, a displacement vector is determined for the map based on a difference between the first scan position and the second scan position. Subsequently, a revised second scan position is computed based on the second scan position and the displacement vector, and the scan-data is registered using the revised scan position.
- a non-transitory computer-readable medium has program instructions embodied therewith, the program instructions readable by a processor to cause the processor to perform a method performing a simultaneous location and mapping of a scanner device in a surrounding environment.
- FIG. 1 depicts a system for scanning an environment according to one or more embodiments of the present disclosure
- FIG. 2 depicts a flowchart of a method for using sensor data for simultaneously locating and mapping a scanner in an environment according to one or more embodiments of the present disclosure
- FIG. 3 depicts a graphical representation of an example SLAM implementation
- FIG. 4 depicts example semantic features detected according to one or more embodiments of the present disclosure
- FIG. 5 depicts an example of improving loop closure using semantic features according to one or more embodiments of the present disclosure
- FIG. 6 depicts a result of an alignment using semantic features that are detected according to one or more embodiments of the present disclosure
- FIG. 7 depicts a flowchart of a method to determine semantic features and use such features as landmarks during implementation of SLAM according to one or more embodiments
- FIG. 8 depicts an example scenario where semantic features are detected according to one or more embodiments of the present disclosure
- FIG. 9 schematically illustrates an example scenario in which an offset is continuously introduced into the scan data
- FIG. 10 illustrates the accumulation of errors and inefficiencies in an example scenario
- FIG. 11 depicts an example method for correcting scan positions using landmarks from semantic features while performing a scan according to one or more embodiments
- FIG. 12 is a view of a user interface of a scanner in accordance with an embodiment of the invention.
- FIG. 13 depicts a resulting correction performed using landmarks from semantic features according to one or more embodiments of the present disclosure
- FIG. 14 depicts a block diagram of matching landmarks from one frame with another according to one or more embodiments of the present disclosure
- FIG. 15 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure
- FIG. 16 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure
- FIG. 17 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure
- FIG. 18 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure
- FIG. 19 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure.
- FIG. 20 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure.
- FIG. 21 depicts a computing system that can implement one or more embodiments of the present disclosure.
- Embodiments of the present disclosure provide technical solutions to technical challenges in existing environment scanning systems.
- the scanning systems can capture two-dimensional or three-dimensional (3D) scans.
- Such scans can include 2D maps, 3D point clouds, or a combination thereof.
- the scans can include additional components, such as annotations, images, textures, measurements, and other details.
- Embodiments of the present disclosure facilitate a mobile scanning platform that allows for simultaneous scanning, mapping, and trajectory generation of an environment while the platform is moving.
- Embodiments of the present disclosure provide a hand-held scanning platform that is sized and weighted to be carried by a single person.
- Embodiments of the present disclosure provide for a mobile scanning platform that may be used to scan an environment in an autonomous or semi-autonomous manner.
- SLAM simultaneous localization and mapping
- a scanner such as the FARO® SCANPLAN®, FARO® SWIFT®, FARO® FREESTYLE®, or any other scanning system incrementally builds the scan of the environment, while the scanner is moving through the environment, and simultaneously the scanner tries to localize itself on this scan that is being generated.
- An example of a handheld scanner is described in U.S. patent application Ser. No. 15/713,931, the contents of which is incorporated by reference herein in its entirety.
- This type of scanner may also be combined with a another scanner, such as a time of flight scanner as is described in commonly owned U.S. patent application Ser. No. 16/567,575, the contents of which are incorporated by reference herein in its entirety.
- the scanners listed above are just examples and that the type of scanner used in one or more embodiments does not limit the features of the technical solutions described herein.
- FIG. 1 depicts a system for scanning an environment according to one or more embodiments of the present disclosure.
- the system 100 includes a computing system 110 coupled with a scanner 120 .
- the coupling facilitates wired and/or wireless communication between the computing system 110 and the scanner 120 .
- the scanner 120 can include a 2D scanner, a 3D scanner, or a combination of both.
- the scanner 120 is capturing measurements of the surroundings of the scanner 120 .
- the measurements are transmitted to the computing system 110 to generate a map 130 of the environment in which the scanner is being moved.
- the map 130 can be generated by combining several submaps. Each submap is generated using SLAM.
- FIG. 2 depicts a high level operational flow for implementing SLAM according to one or more embodiments of the present disclosure.
- Implementing SLAM 210 includes generating one or more submaps corresponding to one or more portions of the environment. The submaps are generated using the one or more sets of measurements from the sets of sensors 122 . Generating the submaps may be referred to as “local SLAM” ( 212 ). The submaps are further combined by the SLAM algorithm to generate the map 130 . Combining the submpas process may be referred to as “global SLAM” ( 214 ). Together, generating the submaps and the final map of the environment is referred to herein as implementing SLAM, unless specifically indicated otherwise.
- SLAM 210 can include operations such as filtering, sampling, and others, which are not depicted.
- the local SLAM 212 facilitates inserting a new set of measurement data captured by the scanner 120 into a submap construction. This operation is sometimes referred to as “scan matching.”
- a set of measurements can include one or more point clouds, distance of each point in the point cloud(s) from the scanner 120 , color information at each point, radiance information at each point, and other such sensor data captured by a set of sensors 122 that is equipped on the scanner 120 .
- the sensors 122 can include a LIDAR 122 A, a depth camera 122 B, a camera 122 C, etc.
- the scanner 120 can also include an inertial measurement unit (IMU) 126 to keep track of a 3D orientation of the scanner 120 .
- IMU inertial measurement unit
- the captured measurement data is inserted into the submap using an estimated pose of the scanner 120 .
- the pose can be extrapolated by using the sensor data from sensors such as IMU 126 , (sensors besides the range finders) to predict where the scanned measurement data is to be inserted into the submap.
- sensors such as IMU 126 , (sensors besides the range finders) to predict where the scanned measurement data is to be inserted into the submap.
- Various techniques are available for scan matching. For example, a point to inset the measured data can be determined by interpolating the submap and sub-pixel aligning the scan. Alternatively, the measured data is matched against the submap to determine the point of insertion.
- a submap is considered as complete when the local SLAM 212 has received at least a predetermined amount of measurement data. Local SLAM 212 drifts over time, and global SLAM 214 is used to fix this drift.
- a submap is a representation of a portion of the environment and that the map 130 of the environment includes several such submaps “stitched” together. Stitching the maps together includes determining one or more landmarks on each submap that is captured and aligning and registering the submaps with each other to generate the map 130 . Further, generating each submap includes combining or stitching one or more sets of measurements. Combining two sets of measurements requires matching, or registering one or more landmarks in the sets of measurements being combined.
- generating each submap and further combining the submaps includes registering a set of measurements with another set of measurements during the local SLAM ( 212 ), and further, generating the map 130 includes registering a submap with another submap during the global SLAM ( 214 ). In both cases, the registration is done using one or more landmarks.
- a “landmark” is a feature that can be detected in the captured measurements and be used to register a point from the first set of measurements with a point from the second set of measurements.
- the landmark can facilitate registering a 3D point cloud with another 3D point cloud or to register an image with another image.
- the registration can be done by detecting the same landmark in the two images (or point clouds) that are to be registered with each other.
- a landmark can include, but is not limited to features such as a doorknob, a door, a lamp, a fire extinguisher, or any other such identification mark that is not moved during the scanning of the environment.
- landmarks can also include stairs, windows, decorative items (e.g., plant, picture-frame, etc.), furniture, or any other such structural or stationary objects.
- landmarks can also include “artificial” landmarks that are added by the operator of the scanner 120 .
- Such artificial landmarks can include identification marks that can be reliably captured and used by the scanner 120 .
- Examples of artificial landmarks can include predetermined markers, such as labels of known dimensions and patterns, e.g., a checkerboard pattern, a target sign, or other such preconfigured markers (e.g. spherical markers).
- the global SLAM ( 214 ) can be described as a pose graph optimization problem.
- the SLAM algorithm is used to provide concurrent construction of a model of the environment (the scan), and an estimation of the state of the scanner 120 moving within the environment.
- SLAM provides a way to track the location of a robot in the world in real-time and identify the locations of landmarks such as buildings, trees, rocks, walls, doors, windows, paintings, décor, furniture, and other world features.
- SLAM also generates or builds up a model of the environment to locate objects including the landmarks that surround the scanner 120 and so that the scan data can be used to ensure that the scanner 120 is on the right path as the scanner 120 moves through the world, i.e., environment. So, the technical challenge with the implementation of SLAM is that while building the scan, the scanner 120 itself might lose track of where it is by virtue of its motion uncertainty because there is no presence of an existing map of the environment because the map is being generated simultaneously.
- the basis for SLAM is to gather information from the set of sensors 120 and motions over time and then use information about measurements and motion to reconstruct a map of the environment.
- the SLAM algorithm defines the probabilities of the scanner 120 being at a certain location in the environment, i.e., at certain coordinates, using a sequence of constraints. For example, consider that the scanner 120 moves in some environment, the SLAM algorithm is input the initial location of the scanner 120 , say (0,0) initially, which is also called as Initial Constraints. The SLAM algorithm is then inputted several relative constraints that relate each pose of the scanner 120 to a previous pose of the scanner 120 . Such constraints are also referred to as relative motion constraints.
- SLAM The technical challenge of SLAM can also be described as follows.
- the “perceptions” include the captured data and the mapped detected planes 410 .
- Solving the full SLAM problem now includes estimating the posterior probability of the trajectory of the scanner 120 x 1:T and the map M of the environment given all the measurements plus an initial position x 0 : P(x 1:T , M
- the initial position x 0 defines the position in the map and can be chosen arbitrarily.
- SLAM There are several known approaches to implement SLAM, for example, graph SLAM, multi-level relaxation SLAM, sparse matrix-based SLAM, hierarchical SLAM, etc. The technical solutions described herein are applicable regardless of which technique is used to implement SLAM.
- FIG. 3 depicts a graphical representation of an example SLAM implementation.
- every node 310 corresponds to a pose of the scanner 120 .
- Nearby poses are connected by edges 320 that model spatial constraints between poses of the scanner 120 arising from measurements.
- Edges e t-1, t between consecutive poses model odometry measurements, while the other edges represent spatial constraints arising from multiple observations of the same part of the environment.
- a graph-based SLAM approach constructs a simplified estimation problem by abstracting the raw sensor measurements. These raw measurements are replaced by the edges 320 in graph 300 , which can then be seen as “virtual measurements.”
- An edge 320 between two nodes 310 are labeled with a probability distribution over the relative locations of the two poses, conditioned to their mutual measurements.
- x t , M t ) is multi-modal, and therefore the Gaussian assumption does not hold. This means that a single observation z t might result in multiple potential edges connecting different poses in the graph, and the graph connectivity needs itself to be described as a probability distribution. Directly dealing with this multi-modality in the estimation process would lead to a large combinatorial increase of complexity. As a result of that, most practical approaches restrict the estimate to the most likely topology. Hence, a constraint resulting from observation has to be determined.
- the goal of a graph-based mapping algorithm is to compute a Gaussian approximation of the posterior over the trajectory of the scanner 120 . This involves computing the mean of this Gaussian as the configuration of the nodes 310 that maximizes the likelihood of the observations. Once the mean is known, the information matrix of the Gaussian can be obtained in a straightforward fashion, as is known in the art. In the following the task of finding is characterized with this maximum as a constraint optimization problem.
- z ij and ⁇ ij be respectively the mean and the information matrix of a virtual measurement between the node i and the node j.
- This virtual measurement is a transformation that makes the observations acquired from i maximally overlap with the observation acquired from j.
- ⁇ circumflex over (z) ⁇ ij (x i , x j ) be the prediction of a virtual measurement given a configuration of the nodes x i , and x j .
- this prediction is the relative transformation between the two nodes.
- e(x i , x j , z ij ) be a function that computes a difference between the expected observation ⁇ circumflex over (z) ⁇ ij and the real observation z ij captured by the scanner 120 .
- implementing global SLAM 214 includes determining constraints ( 222 ) between nodes 320 , i.e., submaps, objects, landmarks, or any other elements that are matched.
- Non-global constraints also known as inter submaps constraints
- Global constraints are built automatically between nodes 310 that are closely following each other on a trajectory of the scanner 120 in the environment.
- Global constraints also referred to as loop closure constraints or intra submaps contraints
- “close enough” is based on predetermined thresholds, for example, distance between the same landmark from two submaps being within a predetermined threshold.
- SLAM uses measurements, such as LIDAR data, from the set of sensors 122 , to aggregate the measurements to generate the submaps and eventually the map 130 .
- measurements such as LIDAR data
- a technical challenge with such implementations is that the matching of the sets of measurements is inaccurate due to ambiguities or missing data. This may lead to misaligned sets of measurements and/or submaps, which in turn, cause an erroneous submap and/or map 130 .
- “loop closure” 224 is used to prevent such errors by compensating for accumulated erros.
- loop closure cannot be used in the cases where the same landmarks are not identified in two sets of measurements or submaps that are being combined or stitched.
- One of the causes of such technical challenges is that the existing techniques only rely on the captured data from the sensors 122 without any semantic or geometric analysis of the measurement data.
- semantic features in the map 130 can include, for example, a corner with a 90-degree angle, walls with a particular length, pillars with a defined shape that can be defined using geometry primitives like a circle.
- a “semantic feature” is detected in the captured measurement data by analyzing the captured data and identifying particular landmarks based on the captured data having a specific user-identifiable arrangement that may not be readily discernable from only the data itself. For example, a 90 degree corner may not be discernable until captured measurement data is converted to a line map of the environment.
- FIG. 4 depicts example semantic features detected according to one or more embodiments of the present disclosure.
- the semantic features include a corner 410 , a wall 420 of a measureable length, line crossings above a threshold (not shown).
- the detection of these features is performed on a single scan line generated from the 2D data captured by the scanner 120 . This can be done by searching for specific formations in the measurement data. Detecting predetermined shapes, or formations in measurement data can be performed using known techniques. For example, in one or more embodiments of the present disclosure, image processing on submaps and remap the extracted features to scanlines. For example, line detection can be done with a specific convolution filters and/or a Hough transformation.
- a “scan” can be a set of measurement data, where multiple such scans are combined to form a submap, and where several such submaps are combined to form the map 130 .
- the semantic feature is used as a constraint for matching of the multiple scans during the local SLAM.
- the semantic features are also used for initialization to generate submaps consisting of multiple matched data. This matching maybe implemented as nonlinear optimization with a cost function.
- the cost function can include equations for the distance of the semantic features from the scanner 120 to improve the accuracy and robustness of the SLAM algorithm.
- the semantic features can be used in some situations to improve the robustness and speed of optimization during initialization of the SLAM algorithm.
- the semantic features can be reused as indicator for loop closure in the case where the feature which can be identified globally e.g. line/wall segments through their length. If multiple such landmarks are identified between two submaps the loop closure can be evaluated using the timestamp and the semantic features for the alignment of the multiple submaps.
- FIG. 5 depicts an example of improving loop closure using semantic features according to one or more embodiments of the present disclosure.
- the same semantic features, the corner 410 , and the measurable wall 420 are detected in two submaps.
- the semantic features can be used for aligning the two submaps when generating the map 130 .
- the semantic features can be used to create a constraint for such alignment.
- FIG. 6 depicts a result of an alignment using semantic features that are detected according to one or more embodiments of the present disclosure.
- the submaps and the semantic features for example, the corner 410 , and the measurable wall 420 , are aligned better (e.g. more accurately) compared to the misalignment in FIG. 5 .
- semantic features can also be used in the global SLAM optimization as constraints for the connection between the submaps and the orientation of the scanner 120 .
- the global SLAM 214 is completed by registering 226 the submaps and stitching the submaps to generate the map 130 .
- SLAM 210 is performed iteratively as newer measurements are acquired by the scanner 120 .
- the computing system 110 can be a desktop computer, a laptop computer, a tablet computer, a phone, or any other type of computing device that can communicate with the scanner 120 .
- the computing system 110 is a portable computing device that is carried by the operator during scanning operations (e.g. clipped to a belt or carried in a pouch or pocket).
- One or more operations for implementing SLAM can be performed by the computing system 110 .
- one or more of the operations can be performed by a processor 124 that is equipped on the scanner 120 .
- the processor 124 and the computing system 110 can implement SLAM in a distributed manner.
- the processor 124 can include one or more processing units.
- the processor 124 controls the measurements performed using the set of sensors 122 . In one or more examples, the measurements are performed based on one or more instructions received from the computing system 110 .
- the computing device 110 and/or a display (not shown) of the scanner 120 provides a live view of the map 130 of the environment being scanned by the scanner 120 using the set of sensors 122 .
- Map 130 can be a 2D or 3D representation of the environment seen through the different sensors. Map 130 can be represented internally as a grid map.
- a grid map is a 2D or 3D arranged collection of cells (2D) or voxels (3D), representing an area of the environment.
- the grid map stores for every cell/voxel, a probability indicating if the cell area is occupied or not.
- Other representations of the map 130 can be used in one or more embodiments of the present disclosure.
- the scanner 120 along with capturing the map 130 , is also locating itself within the environment.
- the scanner 120 uses odometry, which includes using data from motion or visual sensors to estimate the change in position of the scanner 120 over time. Odometry is used to estimate the position of the scanner 120 relative to a starting location. This method is sensitive to errors due to the integration of velocity measurements over time to give position estimates.
- embodiments of the present disclosure provide accurate data that can be used by such an implementation to provide an improved result of the SLAM implementation.
- the embodiments of the present disclosure accordingly, facilitate improvement to computing technology.
- embodiments of the present disclosure provide a practical application that facilitates generating the map 130 of an environment.
- FIG. 7 depicts a flowchart of a method to determine semantic features and use such features as landmarks during implementation of SLAM according to one or more embodiments.
- SLAM is implemented in batches in a continuous manner. For example, after the scanner 120 has moved a predetermined distance, for example, 5 meters, 10 meters, 2 meters, or any other such predetermined distance, SLAM is performed to combine the measurement data captured over that traveled distance, i.e., since the last iteration of SLAM. Such a periodic batch performance is used to reduce time spent on the SLAM computations.
- SLAM can be executed in a continuous manner at the expense of a larger number of computations.
- the accuracy of the measurements starts drifting, i.e., decreasing as the scanner is moved shorter distances than the predetermined distance used for performing SLAM, for example, 10-15 centimeter. Accordingly, even with measures like loop closure, compensating for such inaccuracies over the span of a relatively large area such as a building, which can be 100 meters, 250 meters, etc., i.e., substantially larger than the distance at which errors start accumulating, is a technical challenge.
- One or more embodiments of the present disclosure address such a challenge and improves the workflow by having the operator of the scanner 120 identify a landmark once, and then identifying the landmark in subsequent scans/measurement data based on line drawings of the captured data.
- the method 700 depicted in FIG. 7 includes capturing measurement from the scanner 120 and detecting lines in the captured data, at block 702 .
- the lines are detected at runtime, as the measurements are being captured by the scanner 120 .
- the lines are detected every time a predetermined number of scans are completed by the scanner 120 .
- the predetermined number of scans can be one, five, ten, or any other value.
- the line detection can be performed on demand in response to a command from the operator.
- the line detection can be performed upon completion of a submap, in which case, the semantic features detected thereafter are used as constraints for the global SLAM only.
- Detecting the lines in the measured data can be performed using any known line detection/generation algorithms. For example, see “Line-segmentation Of 2d Laser Scanner Point Clouds For Indoor Slam Based On A Range Of Residuals,” by Peter et al. The contents of which are incorporated herein in its entirety.
- the method 700 further includes evaluating, automatically, length of each line-segment in the lines detected in the captured scan, at block 704 .
- Each line-segment corresponds to a wall or any other such feature in the environment being scanned.
- the length of each line-segment corresponds to an actual length of the corresponding feature.
- line-segments corresponding to a feature of at least a predetermined length, for example 50 centimeter are identified, at block 706 .
- the line-segments can be identified based on the length of the line-segments themselves, which are scaled versions of the length of the corresponding features, the scaling factor being a preconfigured setting for the scanner 120 . Further, orientations of the line-segments with at least the predetermined length are determined, at block 708 .
- the method includes detecting one or more semantic features based on the identified line-segments and their orientations, at block 710 .
- the semantic features can include walls of at least a second predetermined length, where the second predetermined length is more than the predetermined length used to identify the line-segments.
- the semantic features can include corners, which are identified at junctions of two line-segments that cross each other at least at a predetermined angle value, for example, 90 degrees and more. The angle value considered is the angle formed towards the scanner 120 , in one or more embodiments of the present disclosure. Junctions that form substantially a right angle, i.e., 90 degrees are also identified as semantic features.
- Each detected semantic feature is assigned a unique identifier, at block 712 .
- FIG. 8 depicts an example scenario where semantic features are detected according to one or more embodiments of the present disclosure.
- the scanner position 810 and a timestamp is recorded along with the captured measurement data.
- the line-segments 820 of at least the predetermined length are identified from the scan 805 .
- the semantic features 830 are determined from the identified line-segments 820 .
- the semantic features 830 are automatically detected in one or more embodiments of the present disclosure.
- the operator can manually select a subset of the semantic features 830 that are automatically detected, the selected semantic features beign used as the landmarks. In this case, only the selected semantic features are assigned unique identifiers.
- the operations of detecting the semantic features 830 is repeated in every new line that is identified as the scanner 120 records new scans 805 .
- the submap that is generated is also annotated with the information of the landmark.
- the landmarks are evaluated for occurrence and stability, at block 714 .
- the landmarks that are not repeatably identifiable are removed from the list of landmarks to be used for map generation, in one or more embodiments of the present disclosure.
- the evaluation can be performed manually in one or more embodiments of the present disclosure.
- FIG. 9 schematically illustrates an example scenario in which an offset (referred to as “drift”) is continuously introduced into the scan data.
- drift an offset
- the scanner 120 is moving from a starting position 1510 (real pose). After some movements the scanner 120 is designated to return to an already mapped region, such as the starting position 1510 , however the measured position due to sensor variation and the subsequent measurement error is a different position 1520 (estimated pose).
- the loop closure algorithm(s) that are typically used detect the loop closure correction 1530 and corrects the pose and the maps that have been acquired so far by the scanner 120 .
- FIG. 10 illustrates the accumulation of errors and inefficiencies.
- the scanner 120 starts moving from the start position 1510 . After some movement the scanner 120 takes a scan as described herein from one of a plurality of scan positions 1610 . When the scanner 120 arrives back in the start position 1510 the measurement error due to sensor data variance causes the estimated pose 1520 to differ from the start position 1510 . After loop closure the recorded scan positions still have the same coordinates including the error while the map was corrected by the loop closure algorithm. Consequently the estimated positions of the scans have a deviation. As described earlier, when the loop closure is applied all positions in the map change. But as the scan positions 1610 have been recorded before, the scan positions are not automatically adapted.
- the loop closure can be performed using the landmarks that are detected using the semantic features 830 as described herein.
- the relative observation of a landmark from the scanner 120 delivers an accurate position information and can correct the position of the scanner 120 in the absolute world and remove absolute inaccuracies accumulated from the mapping process.
- the more landmarks (observed with a good accuracy) the better the position accuracy of the scanner 120 and consequently the absolute accuracy of the maps scanned by the scanner 120 .
- “absolute accuracy” is the accuracy of measurements of a map that is scanned compared to a ground truth. For example, a side wall of a building has a real length of 100 m. The side wall when measured by the scanner 120 is 101.1 m. In this case, there is an absolute error of 1.1 m and an absolute accuracy of >1.1 for distance >100 m. Such kind of errors in the scan positions are mitigated using loop closure.
- FIG. 11 depicts an example method 1100 for correcting scan positions using landmarks from semantic features while performing a scan according to one or more embodiments.
- the operator stops and starts to record a scan 805 with the scanner 120 at a scan position 810 from the scan positions 1610 ( FIG. 10 ).
- the scanner 120 automatically stops and starts to record the scan 805 at the scan position 1610 .
- Acquiring the scan 805 includes determining with processor system, in cooperation with the scanner 120 , 3D coordinates of a first collection of points on an object surface while the scanner is located at a first registration position (e.g. position 1510 ) while the moveable platform moves through the environment.
- acquiring the scan 805 includes obtaining by the scanner 120 in cooperation with the processor system a plurality of 2D scan sets.
- Each of the plurality of 2D scan sets is a set of 2D coordinates of points on the object surface collected as the scanner 120 moves from the first registration position ( 1510 ) to a second registration position (e.g. position 1610 ).
- Each of the plurality of 2D scan sets is collected by the scanner 120 at a different position relative to the first registration position 1510 .
- the plurality of the 2D scan sets are together referred to as a 2D map and each of the scan sets is a part of the map 130 .
- one or more landmarks are recorded using the semantic features 830 when the scanner is at a first scan position 1510 , at block 1102 .
- the landmarks are detected, automatically, semi-automatically, or manually.
- the operator can select “add landmark” 1212 ( FIG. 12 ) via a user interface 1210 of the scanner 120 .
- the operator captures the scan 805 from the present scan position 1510 .
- the scan 805 is automatically captured by the scanner 120 in response to an instruction executed by the controller.
- the scan 805 is saved as complementary data with the scan position 1510 and the scan position 1510 is marked to indicate that it is a virtual landmark.
- the virtual landmark is represented as a flag (or any other marker) in the map 130 .
- the present scan position 1510 of the scanner is recorded as part of the virtual landmark.
- the scan 805 is captured to include specific semantic features 830 in the environment.
- the scan 805 is captured to include specific objects 1222 in the environment that are relatively immovable, and that can be used as reference objects at a future time when the moveable platform returns to the scan position 1510 .
- the scan 805 includes objects 1222 that are identifiable and that facilitate the operator to recognize that the platform is at substantially the same position ( 1510 ) when the platform returns after mapping.
- the identifiable objects 1222 can include construction items such as windows, doors, corners, pillars, beams, and the like.
- the identifiable objects 1222 can include items that have a relatively immovable position, such as a fire extinguisher (which may be required at particular position because of regulations/code), a defibrillator, a statue, a water fountain, a plant, and the like.
- a fire extinguisher which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a defibrillator which may be required at particular position because of regulations/code
- a statue which may be required at particular position because of regulations/code
- a water fountain which may be required at particular position because of
- the scanner 120 detects the identifiable objects 1222 and semantic features 830 in the environment as the mapping is being performed.
- the scanner 120 is preconfigured with a list of identifiable objects 1222 and semantic features 830 that can be present in the environment.
- the scanner 120 performs an object and feature recognition in the field of view as the platform is being moved. If any one or more of the objects and features from the list is detected in the field of view, the scanner 120 notifies the operator that the present location can be used as a landmark.
- a visual notification 1250 is shown on the user interface 1210 .
- the visual notification can be a popup, a dropdown, a flashing light, or any other type of visual notification.
- the notification can include an audio notification, a haptic notification, or any other type of notification to suggest the operator that the present position 1510 can be used as a virtual landmark.
- the scanner 120 continues to capture scans 805 at multiple other scan positions 1610 and returning to the first scan position that was marked as a virtual landmark, at block 1104 . Capturing the present position procedure is repeated for every scan 805 captured by the scanner 120 . For example, if the scanner 120 captures n scans a data structure holds n positions with n links to the corresponding measurement data of the portion scanned. In one or more examples, the scanner 120 saves the present position in a data structure such as a list of positions. Every position in the data structure is directly linked to the data structure that is used to store the measurements of the corresponding portion of the environment.
- the scanner 120 computes the measurement error 1530 that is input into the SLAM algorithms to correct the error/drift accumulated from walking or moving the scanner around the scanned portion of the environment, at block 1106 .
- computing the measurement error 1530 includes moving the scanner 120 to an estimated position 1520 .
- the estimated position is an estimate of the first scan position 1510 where the virtual landmark was recorded.
- the operator selects a “measure landmark” instruction 1214 at the estimated position 1520 . In one or more examples, (see FIG.
- the user interface 3610 depicts to the operator the original view (e.g., photo) 1220 that was captured by the scanner 120 when the virtual landmark was recorded.
- a present view 1230 from the mobile device 443 is overlaid on the original image 1220 in a transparent manner. The operator can move the scanner 120 until the present view 1230 substantially matches the original image 1220 in the overlay. Once the operator indicates that the overlays match, the system records a relative observation of the present scan position 1520 of the scanner 120 . The difference 1530 between the recorded position 1510 of the virtual landmark and the present position 1520 is used as the error correction to update and correct the mapping positions.
- the difference is computed as a difference in the original image 1220 and the present view 1230 when the scanner 120 is at the virtual landmark.
- the difference between the images is computed based on the landmarks in the image 1220 and the present view 1230 .
- the method 1100 further includes using the measurement error 1530 to correct the coordinates captured by the scanner 120 , at block 1108 .
- the portion of the map 130 that is scanned and stored since capturing the virtual landmark is updated using the measurement error 1530 , in one or more examples.
- a loop closure operation is executed on the map 120 , and parts of the map are corrected in order to match the real pose, which is the starting position 1510 , with the estimated pose, which is the different position 1520 .
- the loop closure algorithm calculates a displacement for each part of the map 130 that is shifted by the algorithm.
- the scanner 120 determines the scan positions 1610 linked to each portion of the map 130 , at block 1110 .
- a lookup is performed over the data structure that saves the list of positions. The lookup costs a single processor operation, such as an array lookup.
- the scanner 120 applies the displacement vector for a portion of the map 130 to the corresponding scan positions saved in the data structure and saves the resulting displaced (or revised) scan positions back into the data structure, at block 1111 .
- the scanner 120 computes displaced scan positions for each of the saved scan positions 1610 in the data structure. The procedure can be repeated every time the loop closure algorithm is applied.
- the displaced scan positions represent corrected scan positions of the scans 805 that can be used directly without applying further computational expensive point cloud registration algorithms.
- the accuracy of the scan positions 1610 depends on the sensor accuracy of the scanner 120 .
- the displacement vectors 1810 for the portions of the map 130 are determined based on the loop closure operation.
- the scanner 120 applies the displacement vectors 1810 to the scan positions 1610 linked to the portions of the map by the data structure as described herein.
- the resulting displaced scan positions 1910 are accordingly calculated by applying the displacement vectors 1810 to the scan positions 1610 .
- the displaced scan positions 1910 are now correctly located.
- FIG. 13 depicts a resulting correction performed using landmarks from semantic features according to one or more embodiments of the present disclosure.
- Two scans are shown, a first scan 805 captured from a first scan position 1510 , and a second scan 805 ′ captured from a second scan position 1520 .
- the second scan position 1520 is “wrong position” because the operator was trying to be substantially at the first position 1510 .
- the “wrong position” is result of the drift.
- the views 805 and 805 ′ are not aligned as a result of the drift.
- the semantic features 830 are used to determine the offsets 1810 as described herein.
- the offsets 1810 are further used to determine the loop closure correction 1530 , which is then applied to align the views 805 and 805 ′.
- the method 700 includes determining a constraint for SLAM implementation based on the landmarks that are determined using the semantic features 830 , at block 718 .
- Determining the constrain includes generating a relationship by matching the semantic features 830 that are detected from a first position 1610 with corresponding (same) landmarks that are detected in an earlier frame.
- each frame is captured at a particular pose of the scanner 120 .
- the pose can include a position (i.e., coordinates in the environment), and orientation of the scanner 120 .
- FIG. 14 depicts a block diagram of matching landmarks from one frame with another according to one or more embodiments of the present disclosure.
- C t-1 depicts a first position of the scanner 120 at time t ⁇ 1
- C t depicts a second position of the scanner 120 as it is moved to scan the environment. From each of these positions, the scanner 120 can observe and capture measurements of a plane P.
- the scanner 120 captures a scan Ia at the first position and a scan Ib at the second position.
- Each of these scans includes the plane P, which has a normal depicted by n.
- the scanner 120 moves from the first position to the second position at a predetermined speed, for example, R meters per second.
- the scanner 120 is configured to capture successive frames at a predetermined frequency, for example, 10 Hz, 15 Hz, etc.
- the computing system 110 processes the captured frames at a second predetermined frequency, for example, 30 Hz, 45 Hz, etc. Further, consider that the plane P has a landmark a i that is captured at a position x in the scan Ia from the first position; and further, that the landmark a i is at a position y in the second scan Ib that is captured from the second position.
- Matching the landmarks detected from the first position 1610 and those detected from the second position 1620 includes determining the common semantic features, and common landmarks from the two positions.
- the landmark a i includes at least one of such common semantic features.
- the matching further includes determining a relationship between the positions of the same landmark from the separate scan positions.
- the relationship can be a mapping between the point x that represents the landmark from the first position, and the point y that represents the same landmark from the second position.
- the computing system 110 determines a mapping such as Hx ⁇ y.
- H can be a matrix that translates and rotates x, where x and y can be 2D or 3D coordinates.
- x and y can be matrices that represent more than one point.
- H is a relative measurement constraint that can be used by the scanner when implementing the SLAM algorithm.
- the method 700 further includes executing the SLAM algorithm, at block 720 .
- the SLAM algorithm can be implemented by the computing system 110 , and/or the processor 124 . Such use of the constraints generated from the landmarks and/or the semantic features 830 improves the final result of the SLAM algorithm (see FIG. 13 ).
- SLAM can use those constraints in order to find the most likely configuration of the scanner path along with the location of landmarks.
- Embodiments of the present disclosure facilitate improvements to results of SLAM as well as loop closure, and consequently an improved scanning system for generating maps of an environment.
- FIG. 15 , FIG. 16 , and FIG. 17 depict an embodiment of a system 30 having a housing 32 that includes a body portion 34 and a handle portion 36 .
- the system 30 can be used as the scanner 120 .
- the handle 36 may include an actuator 38 that allows the operator to interact with the system 30 .
- the body 34 includes a generally rectangular center portion 35 with a slot 40 formed in an end 42 .
- the slot 40 is at least partially defined by a pair walls 44 that are angled towards a second end 48 .
- a portion of a two-dimensional scanner 50 is arranged between the walls 44 .
- the walls 44 are angled to allow the scanner 50 to operate by emitting a light over a large angular area without interference from the walls 44 .
- the end 42 may further include a three-dimensional camera or RGBD camera 60 .
- the mobile device holder 41 is configured to securely couple a mobile device 43 to the housing 32 .
- the holder 41 may include one or more fastening elements, such as a magnetic or mechanical latching element for example, that couples the mobile device 43 to the housing 32 .
- the mobile device 43 is coupled to communicate with a controller 68 .
- the communication between the controller 68 and the mobile device 43 may be via any suitable communications medium, such as wired, wireless or optical communication mediums for example.
- the holder 41 is pivotally coupled to the housing 32 , such that it may be selectively rotated into a closed position within a recess 46 .
- the recess 46 is sized and shaped to receive the holder 41 with the mobile device 43 disposed therein.
- the second end 48 includes a plurality of exhaust vent openings 56 .
- the exhaust vent openings 56 are fluidly coupled to intake vent openings 58 arranged on a bottom surface 62 of center portion 35 .
- the intake vent openings 58 allow external air to enter a conduit 64 having an opposite opening 66 in fluid communication with the hollow interior 67 of the body 34 .
- the opening 66 is arranged adjacent to a controller 68 which has one or more processors that is operable to perform the methods described herein.
- the external air flows from the opening 66 over or around the controller 68 and out the exhaust vent openings 56 .
- the controller 68 is coupled to a wall 70 of body 34 .
- the wall 70 is coupled to or integral with the handle 36 .
- the controller 68 is electrically coupled to the 2D scanner 50 , the 3D camera 60 , a power source 72 , an inertial measurement unit (IMU) 74 , a laser line projector 76 , and a haptic feedback device 77 .
- IMU inertial measurement unit
- Controller 68 is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results.
- the controller 68 includes one or more processing elements 78 .
- the processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions.
- the one or more processors 78 have access to memory 80 for storing information.
- Controller 68 can convert the analog voltage or current level provided by 2D scanner 50 , camera 60 and IMU 74 into a digital signal to determine a distance from the system 30 to an object in the environment.
- the camera 60 is a 3D or RGBD type camera.
- Controller 68 uses the digital signals that act as input to various processes for controlling the system 30 .
- the digital signals represent one or more system 30 data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation and roll orientation.
- the digital signals may be from components internal to the housing 32 or from sensors and devices located in the mobile device 43 .
- controller 68 accepts data from 2D scanner 50 and IMU 74 and is given certain instructions for the purpose of generating a two-dimensional map of a scanned environment. Controller 68 provides operating signals to the 2D scanner 50 , the camera 60 , laser line projector 76 and haptic feedback device 77 . Controller 68 also accepts data from IMU 74 , indicating, for example, whether the operator is operating in the system in the desired orientation. The controller 68 compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, generates a signal that activates the haptic feedback device 77 .
- predetermined variances e.g. yaw, pitch or roll thresholds
- the data received by the controller 68 may be displayed on a user interface coupled to controller 68 .
- the user interface may be one or more LEDs (light-emitting diodes) 82 , an LCD (liquid-crystal diode) display, a CRT (cathode ray tube) display, or the like.
- a keypad may also be coupled to the user interface for providing data input to controller 68 .
- the user interface is arranged or executed on the mobile device 43 .
- the controller 68 may also be coupled to external computer networks such as a local area network (LAN) and the Internet.
- a LAN interconnects one or more remote computers, which are configured to communicate with controller 68 using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), RS-232, ModBus, and the like.
- Additional systems 30 may also be connected to LAN with the controllers 68 in each of these systems 30 being configured to send and receive data to and from remote computers and other systems 30 .
- the LAN may be connected to the Internet. This connection allows controller 68 to communicate with one or more remote computers connected to the Internet.
- the processors 78 are coupled to memory 80 .
- the memory 80 may include random access memory (RAM) device 84 , a non-volatile memory (NVM) device 86 , a read-only memory (ROM) device 88 .
- the processors 78 may be connected to one or more input/output (I/O) controllers 90 and a communications circuit 92 .
- the communications circuit 92 provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN discussed above.
- Controller 68 includes operation control methods described herein, which can be embodied in application code. These methods are embodied in computer instructions written to be executed by processors 78 , typically in the form of software.
- the software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (Hypertext Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing.
- assembly language VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-
- the 2D scanner 50 measures 2D coordinates in a plane. In the exemplary embodiment, the scanning is performed by steering light within a plane to illuminate object points in the environment. The 2D scanner 50 collects the reflected (scattered) light from the object points to determine 2D coordinates of the object points in the 2D plane. In an embodiment, the 2D scanner 50 scans a spot of light over an angle while at the same time measuring an angle value and corresponding distance value to each of the illuminated object points.
- Examples of 2D scanners 50 include but are not limited to Model LMS103 scanners manufactured by Sick, Inc of Minneapolis, MN and scanner Models URG-04LX-UG01 and UTM-30LX manufactured by Hokuyo Automatic Co., Ltd of Osaka, Japan.
- the scanners in the Sick LMS103 family measure angles over a 270-degree range and over distances up to 20 meters.
- the Hoyuko model URG-04LX-UG01 is a low-cost 2D scanner that measures angles over a 240-degree range and distances up to 4 meters.
- the Hoyuko model UTM-30LX is a 2D scanner that measures angles over a 270-degree range and to distances up to 30 meters. It should be appreciated that the above 2D scanners are exemplary and other types of 2D scanners are also available.
- the 2D scanner 50 is oriented so as to scan a beam of light over a range of angles in a generally horizontal plane (relative to the floor of the environment being scanned). At instants in time the 2D scanner 50 returns an angle reading and a corresponding distance reading to provide 2D coordinates of object points in the horizontal plane. In completing one scan over the full range of angles, the 2D scanner returns a collection of paired angle and distance readings. As the system 30 is moved from place to place, the 2D scanner 50 continues to return 2D coordinate values. These 2D coordinate values are used to locate the position of the system 30 thereby enabling the generation of a two-dimensional map or floorplan of the environment.
- the IMU 74 is a position/orientation sensor that may include accelerometers 94 (inclinometers), gyroscopes 96 , a magnetometer or compass 98 , and altimeters.
- the IMU 74 includes multiple accelerometers 94 and gyroscopes 96 .
- the compass 98 indicates a heading based on changes in magnetic field direction relative to the earth's magnetic north.
- the IMU 74 may further have an altimeter that indicates altitude (height).
- An example of a widely used altimeter is a pressure sensor.
- the IMU 74 determines the pose or orientation of the system 30 about three-axis to allow a determination of a yaw, roll and pitch parameter.
- the system 30 further includes a camera 60 that is a 3D or RGB-D camera.
- the term 3D camera refers to a device that produces a two-dimensional image that includes distances to a point in the environment from the location of system 30 .
- the 3D camera 30 may be a range camera or a stereo camera.
- the 3D camera 30 includes an RGB-D sensor that combines color information with a per-pixel depth information.
- the 3D camera 30 may include an infrared laser projector 31 , a left infrared camera 33 , a right infrared camera 39 , and a color camera 37 .
- the 3D camera 60 is a RealSenseTM camera model R200 manufactured by Intel Corporation.
- the 3D camera 30 is a RealSenseTM LIDAR camera model L515 manufactured by Intel Corporation.
- the mobile device 43 when the mobile device 43 is coupled to the housing 32 , the mobile device 43 becomes an integral part of the system 30 .
- the mobile device 43 is a cellular phone, a tablet computer or a personal digital assistant (PDA).
- the mobile device 43 may be coupled for communication via a wired connection, such as ports 103 , 102 .
- the port 103 is coupled for communication to the processor 78 , such as via I/O controller 90 for example.
- the ports 103 , 102 may be any suitable port, such as but not limited to USB, USB-A, USB-B, USB-C, IEEE 1394 (Firewire), or LightningTM connectors.
- the mobile device 43 is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results.
- the mobile device 43 includes one or more processing elements 104 .
- the processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions.
- the one or more processors 104 have access to memory 106 for storing information.
- the mobile device 43 can convert the analog voltage or current level provided by sensors 108 and processor 78 .
- Mobile device 43 uses the digital signals that act as input to various processes for controlling the system 30 .
- the digital signals represent one or more system 30 data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation, roll orientation, global position, ambient light levels, and altitude for example.
- mobile device 43 accepts data from sensors 108 and is given certain instructions for the purpose of generating or assisting the processor 78 in the generation of a two-dimensional map or three-dimensional map of a scanned environment.
- Mobile device 43 provides operating signals to the processor 78 , the sensors 108 and a display 110 .
- Mobile device 43 also accepts data from sensors 108 , indicating, for example, to track the position of the mobile device 43 in the environment or measure coordinates of points on surfaces in the environment.
- the mobile device 43 compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, may generate a signal.
- the data received by the mobile device 43 may be displayed on display 110 .
- the display 110 is a touch screen device that allows the operator to input data or control the operation of the system 30 .
- the controller 68 may also be coupled to external networks such as a local area network (LAN), a cellular network and the Internet.
- a LAN interconnects one or more remote computers, which are configured to communicate with controller 68 using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), RS-232, ModBus, and the like.
- Additional systems 30 may also be connected to LAN with the controllers 68 in each of these systems 30 being configured to send and receive data to and from remote computers and other systems 30 .
- the LAN may be connected to the Internet. This connection allows controller 68 to communicate with one or more remote computers connected to the Internet.
- the processors 104 are coupled to memory 106 .
- the memory 106 may include random access memory (RAM) device, a non-volatile memory (NVM) device, and a read-only memory (ROM) device.
- the processors 104 may be connected to one or more input/output (I/O) controllers 112 and a communications circuit 114 .
- the communications circuit 114 provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN or the cellular network discussed above.
- Processor 104 includes operation control methods described herein, which can be embodied in application code. These methods are embodied in computer instructions written to be executed by processors 78 , 104 , typically in the form of software.
- the software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (Hypertext Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing.
- assembly language VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC
- the sensors 108 may include but are not limited to: a microphone 116 ; a speaker 118 ; a front or rear facing camera 160 ; accelerometers 162 (inclinometers), gyroscopes 164 , a magnetometers or compass 126 ; a global positioning satellite (GPS) module 168 ; a barometer 170 ; a proximity sensor 132 ; and an ambient light sensor 134 .
- a fusion algorithm that may include a Kalman filter, relatively accurate position and orientation measurements can be obtained.
- the sensors 60 , 74 integrated into the scanner 30 may have different characteristics than the sensors 108 of mobile device 43 .
- the resolution of the cameras 60 , 160 may be different, or the accelerometers 94 , 162 may have different dynamic ranges, frequency response, sensitivity (mV/g) or temperature parameters (sensitivity or range).
- the gyroscopes 96 , 164 or compass/magnetometer may have different characteristics. It is anticipated that in some embodiments, one or more sensors 108 in the mobile device 43 may be of higher accuracy than the corresponding sensors 74 in the system 30 .
- the processor 78 determines the characteristics of each of the sensors 108 and compares them with the corresponding sensors in the system 30 when the mobile device. The processor 78 then selects which sensors 74 , 108 are used during operation.
- the mobile device 43 may have additional sensors (e.g. microphone 116 , camera 160 ) that may be used to enhance operation compared to operation of the system 30 without the mobile device 43 .
- the system 30 does not include the IMU 74 and the processor 78 uses the sensors 108 for tracking the position and orientation/pose of the system 30 .
- the addition of the mobile device 43 allows the system 30 to utilize the camera 160 to perform three-dimensional (3D) measurements either directly (using an RGB-D camera) or using photogrammetry techniques to generate 3D maps.
- the processor 78 uses the communications circuit (e.g. a cellular 4G internet connection) to transmit and receive data from remote computers or devices.
- the system 30 is a handheld portable device that is sized and weighted to be carried by a single person during operation. Therefore, the plane 136 in which the 2D scanner 50 projects a light beam may not be horizontal relative to the floor or may continuously change as the computer moves during the scanning process.
- the signals generated by the accelerometers 94 , gyroscopes 96 and compass 98 (or the corresponding sensors 108 ) may be used to determine the pose (yaw, roll, tilt) of the system 30 and determine the orientation of the plane 51 .
- a haptic feedback device 77 is disposed within the housing 32 , such as in the handle 36 .
- the haptic feedback device 77 is a device that creates a force, vibration or motion that is felt or heard by the operator.
- the haptic feedback device 77 may be, but is not limited to: an eccentric rotating mass vibration motor or a linear resonant actuator for example.
- the haptic feedback device is used to alert the operator that the orientation of the light beam from 2D scanner 50 is equal to or beyond a predetermined threshold.
- the controller 68 transmits a signal to a motor controller 138 that activates a vibration motor 140 . Since the vibration originates in the handle 36 , the operator will be notified of the deviation in the orientation of the system 30 . The vibration continues until the system 30 is oriented within the predetermined threshold or the operator releases the actuator 38 . In an embodiment, it is desired for the plane 136 to be within 10-15 degrees of horizontal (relative to the ground) about the yaw, roll and pitch axes.
- the mobile scanning platform 1800 can be used as the scanner 120 .
- the mobile scanning platform 1800 includes a base unit 1802 having a plurality of wheels 1804 .
- the wheels 1804 are rotated by motors 1805 .
- an adapter plate 1807 is coupled to the base unit 1802 to allow components and modules to be coupled to the base unit 1802 .
- the mobile scanning platform 1800 further includes a 2D scanner 1808 and a 3D scanner 1810 .
- each scanner 1808 , 1810 is removably coupled to the adapter plate 1806 .
- the 2D scanner 1808 may be the scanner illustrated and described herein.
- the 2D scanner 1808 is removable from the adapter plate 1806 and is used to generate a map of the environment, plan a path for the mobile scanning platform to follow, and define 3D scanning locations.
- the 2D scanner 1808 is slidably coupled to a bracket 1811 that couples the 2D scanner 1808 to the adapter plate 1807 .
- the 3D scanner 1810 is a time-of-flight (TOF) laser scanner such as that shown and described herein.
- the scanner 1810 may be that described in commonly owned U.S. Pat. No. 8,705,012, which is incorporated by reference herein.
- the 3D scanner 1810 mounted on a pedestal or post 1809 that elevates the 3D scanner 1810 above (e.g. further from the floor than) the other components in the mobile scanning platform 1800 so that the emission and receipt of the light beam is not interfered with.
- the pedestal 1809 is coupled to the adapter plate 1807 by a u-shaped frame 1814 .
- the mobile scanning platform 1800 further includes a controller 1816 .
- the controller 1816 is a computing device having one or more processors and memory.
- the one or more processors are responsive to non-transitory executable computer instructions for performing operational methods such as those described herein.
- the processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions.
- the one or more processors have access to memory for storing information.
- Coupled for communication to the controller 1816 is a communications circuit 1818 and an input/output hub 1820 .
- the communications circuit 1818 is configured to transmit and receive data via a wireless radio-frequency communications medium, such as WIFI or Bluetooth for example.
- the 2D scanner 1808 communicates with the controller 1816 via the communications circuit 1818
- the mobile scanning platform 1800 further includes a motor controller 1822 that is operably coupled to the control the motors 1805 .
- the motor controller 1822 is mounted to an external surface of the base unit 1802 .
- the motor controller 1822 is arranged internally within the base unit 1802 .
- the mobile scanning platform 1800 further includes a power supply 1824 that controls the flow of electrical power from a power source, such as batteries 1826 for example.
- the batteries 1826 may be disposed within the interior of the base unit 1802 .
- the base unit 1802 includes a port (not shown) for coupling the power supply to an external power source for recharging the batteries 1826 .
- the batteries 1826 are removable or replaceable.
- the automated or motorized mobile scanning platform 1800 is illustrated and described for exemplary purposes and the claims should not be so limited.
- the mobile scanning platform may be a wheeled mobile assembly that is pushed by an operator.
- the mobile platform may be mounted on the operator, such as a backpack configuration.
- the 3D coordinate data or point cloud may be generated by any type of 3D measurement device, such as but not limited to a pulsed TOF laser scanner, frequency modulated continuous wave (FMCW) scanner, triangulation scanner, an area scanner, a structured light scanner, a laser line probe, a laser tracker, or a combination of the foregoing.
- a pulsed TOF laser scanner frequency modulated continuous wave (FMCW) scanner
- FMCW frequency modulated continuous wave
- triangulation scanner an area scanner
- a structured light scanner a laser line probe
- laser tracker or a combination of the foregoing.
- the examples described herein show top views of scan data; however, side views can also be used for registration, and such registration can also be improved as described herein.
- 3D coordinate data may be used for training
- the methods described herein for verifying the registration of landmarks may be used with either two-dimensional or three-dimensional data sets.
- the computer system 2100 can be an electronic, computer framework comprising and/or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein.
- the computer system 2100 can be easily scalable, extensible, and modular, with the ability to change to different services or reconfigure some features independently of others.
- the computer system 2100 may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone.
- computer system 2100 may be a cloud computing node.
- Computer system 2100 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system.
- program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types.
- Computer system 2100 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network.
- program modules may be located in both local and remote computer system storage media including memory storage devices.
- the computer system 2100 has one or more central processing units (CPU(s)) 2101 a , 2101 b , 2101 c , etc. (collectively or generically referred to as processor(s) 2101 ).
- the processors 2101 can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations.
- the processors 2101 also referred to as processing circuits, are coupled via a system bus 2102 to a system memory 2103 and various other components.
- the system memory 2103 can include a read only memory (ROM) 2104 and a random access memory (RAM) 2105 .
- ROM read only memory
- RAM random access memory
- the ROM 2104 is coupled to the system bus 2102 and may include a basic input/output system (BIOS), which controls certain basic functions of the computer system 2100 .
- BIOS basic input/output system
- the RAM is read-write memory coupled to the system bus 2102 for use by the processors 2101 .
- the system memory 2103 provides temporary memory space for operations of said instructions during operation.
- the system memory 2103 can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.
- the computer system 2100 comprises an input/output (I/O) adapter 2106 and a communications adapter 2107 coupled to the system bus 2102 .
- the I/O adapter 2106 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 2108 and/or any other similar component.
- SCSI small computer system interface
- the I/O adapter 2106 and the hard disk 2108 are collectively referred to herein as a mass storage 2110 .
- the mass storage 2110 is an example of a tangible storage medium readable by the processors 2101 , where the software 2111 is stored as instructions for execution by the processors 2101 to cause the computer system 2100 to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail.
- the communications adapter 2107 interconnects the system bus 2102 with a network 2112 , which may be an outside network, enabling the computer system 2100 to communicate with other such systems.
- a portion of the system memory 2103 and the mass storage 2110 collectively store an operating system, which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown in FIG. 21 .
- an operating system which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown in FIG. 21 .
- Additional input/output devices are shown as connected to the system bus 2102 via a display adapter 2115 and an interface adapter 2116 and.
- the adapters 2106 , 2107 , 2115 , and 2116 may be connected to one or more I/O buses that are connected to the system bus 2102 via an intermediate bus bridge (not shown).
- a display 2119 (e.g., a screen or a display monitor) is connected to the system bus 2102 by a display adapter 2115 , which may include a graphics controller to improve the performance of graphics intensive applications and a video controller.
- the computer system 2100 includes processing capability in the form of the processors 2101 , and, storage capability including the system memory 2103 and the mass storage 2110 , input means such as the keyboard 2121 and the mouse 2122 , and output capability including the speaker 2123 and the display 2119 .
- the interface adapter 2116 may include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit.
- Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI).
- PCI Peripheral Component Interconnect
- the computer system 2100 includes processing capability in the form of the processors 2101 , and, storage capability including the system memory 2103 and the mass storage 2110 , input means such as the keyboard 2121 and the mouse 2122 , and output capability including the speaker 2123 and the display 2119 .
- the communications adapter 2107 can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others.
- the network 2112 may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others.
- An external computing device may connect to the computer system 2100 through the network 2112 .
- an external computing device may be an external webserver or a cloud computing node.
- FIG. 21 the block diagram of FIG. 21 is not intended to indicate that the computer system 2100 is to include all of the components shown in FIG. 21 . Rather, the computer system 2100 can include any appropriate fewer or additional components not illustrated in FIG. 21 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the embodiments described herein with respect to computer system 2100 may be implemented with any appropriate logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware, in various embodiments.
- suitable hardware e.g., a processor, an embedded controller, or an application specific integrated circuit, among others
- software e.g., an application, among others
- firmware e.g., any suitable combination of hardware, software, and firmware, in various embodiments.
- aspects of the present disclosure may be embodied as a system, method, or computer program product and may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.), or a combination thereof. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.
- the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
- a computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.
- the computer-readable storage medium may be a tangible medium containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.
- a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.
- a computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium, and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
- the computer-readable medium may contain program code embodied thereon, which may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
- computer program code for carrying out operations for implementing aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
- the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
- the computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- processor controller, computer, DSP, FPGA are understood in this document to mean a computing device that may be located within an instrument, distributed in multiple elements throughout an instrument, or placed external to an instrument.
Landscapes
- Engineering & Computer Science (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- Automation & Control Theory (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
A method is provided that includes recording a landmark at a first scan position of a scanner, the landmark based at least in part on a semantic feature of scan data captured by the scanner. The semantic feature is identified using line-segments of the scan data. The method further includes capturing, by the scanner while moving through the environment, additional scan data at a second scan position. The method further includes, responsive to the scanner returning to the first scan position associated with the landmark, computing a measurement error. The method further includes correcting, using the measurement error, at least a portion of the scan data or the additional scan data.
Description
- This application is a continuation of U.S. application Ser. No. 17/314,102, filed May 7, 2021, which claims the benefit of U.S. Provisional Application Ser. No. 63/031,926 filed May 29, 2020, the entire disclosures of which are incorporated herein by reference in their entirety.
- The present application is directed to a system that optically scans an environment, such as a building, and in particular to a mobile scanning system that generates two-dimensional (2D) and three-dimensional (3D) scans of the environment.
- Automated scanning of an environment is desirable as a number of scans may be performed in order to obtain a complete scan of the area. Various techniques may be used, such as time-of-flight (TOF) or triangulation methods, for example. A TOF laser scanner is a scanner in which the distance to a target point is determined based on the speed of light in air between the scanner and a target point. Laser scanners are typically used for scanning closed or open spaces such as interior areas of buildings, industrial installations and tunnels. They may be used, for example, in industrial applications and accident reconstruction applications. A laser scanner optically scans and measures objects in a volume around the scanner through the acquisition of data points representing object surfaces within the volume. Such data points are obtained by transmitting a beam of light onto the objects and collecting the reflected or scattered light to determine the distance, two-angles (i.e., an azimuth and a zenith angle), and optionally a gray-scale value. This raw scan data is collected, stored and sent to a processor or processors to generate an image, 2D/3D, representing the scanned area or object. Generating an image requires at least three values for each data point. These three values may include the distance and two angles, or may be transformed values, such as the x, y, z coordinates. In an embodiment, an image is also based on a fourth gray-scale value, which is a value related to irradiance of scattered light returning to the scanner.
- Most TOF scanners direct the beam of light within the measurement volume by steering the light with a beam steering mechanism. The beam steering mechanism includes a first motor that steers the beam of light about a first axis by a first angle that is measured by a first angular encoder (or other angle transducer). The beam steering mechanism also includes a second motor that steers the beam of light about a second axis by a second angle that is measured by a second angular encoder (or other angle transducer).
- Many contemporary laser scanners include a camera mounted on the laser scanner for gathering camera digital images of the environment and for presenting the camera digital images to an operator of the laser scanner. By viewing the camera images, the operator of the scanner can determine the field of view of the measured volume and adjust settings on the laser scanner to measure over a larger or smaller region of space. In addition, the digital images from the camera may be transmitted to a processor to add color to the scanner image. To generate a color scanner image, at least three positional coordinates (such as x, y, z) and three color values (such as red, green, blue “RGB”) are collected for each data point.
- In contrast, a triangulation system, such as a scanner, projects either a line of light (e.g., from a laser line probe) or a pattern of light (e.g., from a structured light) onto the surface. In this system, a camera is coupled to a projector in a fixed mechanical relationship. The light/pattern emitted from the projector is reflected off of the surface and detected by the camera. Since the camera and projector are arranged in a fixed relationship, the distance to the object may be determined from captured images using trigonometric principles. Triangulation systems provide advantages in quickly acquiring coordinate data over large areas.
- In some systems, during the scanning process, the scanner acquires, at different times, a series of images of the patterns of light formed on the object surface. These multiple images are then registered relative to each other so that the position and orientation of each image relative to the other images are known. Where the scanner is handheld, various techniques have been used to register the images. One common technique uses features in the images to match overlapping areas of adjacent image frames. This technique works well when the object being measured has many features relative to the field of view of the scanner. However, if the object contains a relatively large flat or curved surface, the images may not properly register relative to each other.
- A 3D image of a scene may require multiple scans from different registration positions. The overlapping scans are registered in a joint coordinate system, for example, as described in U.S. Published Patent Application No. 2012/0069352 ('352), the contents of which are incorporated herein by reference. Such registration is performed by matching targets in overlapping regions of the multiple scans. The targets may be artificial targets such as spheres or checkerboards or they may be natural features such as corners or edges of walls. Some registration procedures involve relatively time-consuming manual procedures such as identifying by a user each target and matching the targets obtained by the scanner in each of the different registration positions. Some registration procedures also require establishing an external “control network” of registration targets measured by an external device such as a total station.
- However, even with these improvements, it is today difficult to remove the need for a user to carry out the manual registration steps as described above. In a typical case, only 30% of 3D scans can be automatically registered to scans taken from other registration positions. Today such registration is seldom carried out at the site of the 3D measurement but instead in a remote location following the scanning procedure. In a typical case, a project requiring a week of scanning requires two to five days to manually register the multiple scans. This adds to the cost of the scanning project. Furthermore, the manual registration process sometimes reveals that the overlap between adjacent scans was insufficient to provide proper registration. In other cases, the manual registration process may reveal that certain sections of the scanning environment have been omitted. When such problems occur, the operator must return to the site to obtain additional scans. In some cases, it is not possible to return to a site. A building that was available for scanning at one time may be impossible to access at a later time for example. Further, a forensics scene of an automobile accident or a homicide is often not available for taking of scans for more than a short time after the incident.
- It should be appreciated that where an object (e.g. a wall, a column, or a desk) blocks the beam of light, that object will be measured but any objects or surfaces on the opposite side will not be scanned since they are in the shadow of the object relative to the scanner. Therefore, to obtain a more complete scan of the environment, the TOF scanner is moved to different locations and separate scans are performed. Subsequent to the performing of the scans, the 3D coordinate data (i.e. the point cloud) from each of the individual scans are registered to each other and combined to form a 3D image or model of the environment.
- Some existing measurement systems have been mounted to a movable structure, such as a cart, and moved on a continuous basis through the building to generate a digital representation of the building. However, these provide generally lower data quality than stationary scans. These systems tend to be more complex and require specialized personnel to perform the scan. Further, the scanning equipment including the movable structure may be bulky, which could further delay the scanning process in time sensitive situations, such as a crime or accident scene investigation.
- Further, even though the measurement system is mounted to a movable cart, the cart is stopped at scan locations so that the measurements can be performed. This further increases the time to scan an environment.
- Accordingly, while existing scanners are suitable for their intended purposes, what is needed is a scanner having certain features of embodiments of the present disclosure.
- According to one or more embodiments, a system includes a scanner device configured to capture scan-data of surrounding environment. The system also includes one or more processors operably coupled to the scanner, wherein the one or more processors are operable to perform simultaneous locating and mapping of the scanner device in the surrounding environment. Performing the simultaneous locating and mapping (SLAM) includes capture the scan-data of a portion of a map of the surrounding environment, wherein the scan-data comprises a point cloud. Further, performing SLAM includes detecting a set of lines in the point cloud, and identifying a semantic feature based at least in part on the set of lines. Performing SLAM further includes assigning a first scan position of the scanner device in the surrounding environment at the present time t1 as a landmark, and linking the landmark with the portion of the map. Further, performing SLAM includes determining that the scanner device has moved, at time t2, to the scan position that was marked as the landmark based on identifying said semantic feature in another scan-data from the scanner device. In response, a second scan position of the scanner device at time t2 is determined. Also, a displacement vector is determined for the map based on a difference between the first scan position and the second scan position. Further, a revised second scan position is computed based on the second scan position and the displacement vector, and the scan-data is registered using the revised scan position.
- According to one or more embodiments, a method for performing a simultaneous location and mapping of a scanner device in a surrounding environment includes capturing the scan-data of a portion of a map of the surrounding environment, wherein the scan-data comprises a point cloud. The method also includes detecting a set of lines in the point cloud, and identifying a semantic feature based at least in part on the set of lines. The method further includes assigning a first scan position of the scanner device in the surrounding environment at the present time t1 as a landmark, and linking the landmark with the portion of the map. The method further includes determining that the scanner device has moved, at time t2, to the scan position that was marked as the landmark based on identifying said semantic feature in another scan-data from the scanner device. In response, as part of the method, a second scan position of the scanner device at time t2 is determined. Also, a displacement vector is determined for the map based on a difference between the first scan position and the second scan position. Subsequently, a revised second scan position is computed based on the second scan position and the displacement vector, and the scan-data is registered using the revised scan position.
- According to one or more embodiments, a non-transitory computer-readable medium has program instructions embodied therewith, the program instructions readable by a processor to cause the processor to perform a method performing a simultaneous location and mapping of a scanner device in a surrounding environment.
- These and other advantages and features will become more apparent from the following description taken in conjunction with the drawings.
- The subject matter, which is regarded as the invention, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
-
FIG. 1 depicts a system for scanning an environment according to one or more embodiments of the present disclosure; -
FIG. 2 depicts a flowchart of a method for using sensor data for simultaneously locating and mapping a scanner in an environment according to one or more embodiments of the present disclosure; -
FIG. 3 depicts a graphical representation of an example SLAM implementation; -
FIG. 4 depicts example semantic features detected according to one or more embodiments of the present disclosure; -
FIG. 5 depicts an example of improving loop closure using semantic features according to one or more embodiments of the present disclosure; -
FIG. 6 depicts a result of an alignment using semantic features that are detected according to one or more embodiments of the present disclosure; -
FIG. 7 depicts a flowchart of a method to determine semantic features and use such features as landmarks during implementation of SLAM according to one or more embodiments; -
FIG. 8 depicts an example scenario where semantic features are detected according to one or more embodiments of the present disclosure; -
FIG. 9 schematically illustrates an example scenario in which an offset is continuously introduced into the scan data; -
FIG. 10 illustrates the accumulation of errors and inefficiencies in an example scenario; -
FIG. 11 depicts an example method for correcting scan positions using landmarks from semantic features while performing a scan according to one or more embodiments; -
FIG. 12 is a view of a user interface of a scanner in accordance with an embodiment of the invention; -
FIG. 13 depicts a resulting correction performed using landmarks from semantic features according to one or more embodiments of the present disclosure; -
FIG. 14 depicts a block diagram of matching landmarks from one frame with another according to one or more embodiments of the present disclosure; -
FIG. 15 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; -
FIG. 16 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; -
FIG. 17 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; -
FIG. 18 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; -
FIG. 19 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; -
FIG. 20 depicts an embodiment of a scanner according to one or more embodiments of the present disclosure; and -
FIG. 21 depicts a computing system that can implement one or more embodiments of the present disclosure. - The detailed description explains embodiments of the invention, together with advantages and features, by way of example, with reference to the drawings.
- Embodiments of the present disclosure provide technical solutions to technical challenges in existing environment scanning systems. The scanning systems can capture two-dimensional or three-dimensional (3D) scans. Such scans can include 2D maps, 3D point clouds, or a combination thereof. The scans can include additional components, such as annotations, images, textures, measurements, and other details.
- Embodiments of the present disclosure facilitate a mobile scanning platform that allows for simultaneous scanning, mapping, and trajectory generation of an environment while the platform is moving. Embodiments of the present disclosure provide a hand-held scanning platform that is sized and weighted to be carried by a single person. Embodiments of the present disclosure provide for a mobile scanning platform that may be used to scan an environment in an autonomous or semi-autonomous manner.
- Typically, when capturing a scan of an environment, a version of the simultaneous localization and mapping (SLAM) algorithm is used. For completing such scans a scanner, such as the FARO® SCANPLAN®, FARO® SWIFT®, FARO® FREESTYLE®, or any other scanning system incrementally builds the scan of the environment, while the scanner is moving through the environment, and simultaneously the scanner tries to localize itself on this scan that is being generated. An example of a handheld scanner is described in U.S. patent application Ser. No. 15/713,931, the contents of which is incorporated by reference herein in its entirety. This type of scanner may also be combined with a another scanner, such as a time of flight scanner as is described in commonly owned U.S. patent application Ser. No. 16/567,575, the contents of which are incorporated by reference herein in its entirety. It should be noted that the scanners listed above are just examples and that the type of scanner used in one or more embodiments does not limit the features of the technical solutions described herein.
-
FIG. 1 depicts a system for scanning an environment according to one or more embodiments of the present disclosure. Thesystem 100 includes acomputing system 110 coupled with ascanner 120. The coupling facilitates wired and/or wireless communication between thecomputing system 110 and thescanner 120. Thescanner 120 can include a 2D scanner, a 3D scanner, or a combination of both. Thescanner 120 is capturing measurements of the surroundings of thescanner 120. The measurements are transmitted to thecomputing system 110 to generate amap 130 of the environment in which the scanner is being moved. Themap 130 can be generated by combining several submaps. Each submap is generated using SLAM. -
FIG. 2 depicts a high level operational flow for implementing SLAM according to one or more embodiments of the present disclosure. ImplementingSLAM 210 includes generating one or more submaps corresponding to one or more portions of the environment. The submaps are generated using the one or more sets of measurements from the sets ofsensors 122. Generating the submaps may be referred to as “local SLAM” (212). The submaps are further combined by the SLAM algorithm to generate themap 130. Combining the submpas process may be referred to as “global SLAM” (214). Together, generating the submaps and the final map of the environment is referred to herein as implementing SLAM, unless specifically indicated otherwise. - It should be noted that the operations shown in
FIG. 2 are at high level, and that typical implementations ofSLAM 210 can include operations such as filtering, sampling, and others, which are not depicted. - The
local SLAM 212 facilitates inserting a new set of measurement data captured by thescanner 120 into a submap construction. This operation is sometimes referred to as “scan matching.” A set of measurements can include one or more point clouds, distance of each point in the point cloud(s) from thescanner 120, color information at each point, radiance information at each point, and other such sensor data captured by a set ofsensors 122 that is equipped on thescanner 120. For example, thesensors 122 can include aLIDAR 122A, adepth camera 122B, acamera 122C, etc. Thescanner 120 can also include an inertial measurement unit (IMU) 126 to keep track of a 3D orientation of thescanner 120. - The captured measurement data is inserted into the submap using an estimated pose of the
scanner 120. The pose can be extrapolated by using the sensor data from sensors such asIMU 126, (sensors besides the range finders) to predict where the scanned measurement data is to be inserted into the submap. Various techniques are available for scan matching. For example, a point to inset the measured data can be determined by interpolating the submap and sub-pixel aligning the scan. Alternatively, the measured data is matched against the submap to determine the point of insertion. A submap is considered as complete when thelocal SLAM 212 has received at least a predetermined amount of measurement data.Local SLAM 212 drifts over time, andglobal SLAM 214 is used to fix this drift. - It should be noted that a submap is a representation of a portion of the environment and that the
map 130 of the environment includes several such submaps “stitched” together. Stitching the maps together includes determining one or more landmarks on each submap that is captured and aligning and registering the submaps with each other to generate themap 130. Further, generating each submap includes combining or stitching one or more sets of measurements. Combining two sets of measurements requires matching, or registering one or more landmarks in the sets of measurements being combined. - Accordingly, generating each submap and further combining the submaps includes registering a set of measurements with another set of measurements during the local SLAM (212), and further, generating the
map 130 includes registering a submap with another submap during the global SLAM (214). In both cases, the registration is done using one or more landmarks. - Here, a “landmark” is a feature that can be detected in the captured measurements and be used to register a point from the first set of measurements with a point from the second set of measurements. For example, the landmark can facilitate registering a 3D point cloud with another 3D point cloud or to register an image with another image. Here, the registration can be done by detecting the same landmark in the two images (or point clouds) that are to be registered with each other. A landmark can include, but is not limited to features such as a doorknob, a door, a lamp, a fire extinguisher, or any other such identification mark that is not moved during the scanning of the environment. The landmarks can also include stairs, windows, decorative items (e.g., plant, picture-frame, etc.), furniture, or any other such structural or stationary objects. In addition to such “naturally” occurring features, i.e., features that are already present in the environment beign scanned, landmarks can also include “artificial” landmarks that are added by the operator of the
scanner 120. Such artificial landmarks can include identification marks that can be reliably captured and used by thescanner 120. Examples of artificial landmarks can include predetermined markers, such as labels of known dimensions and patterns, e.g., a checkerboard pattern, a target sign, or other such preconfigured markers (e.g. spherical markers). - The global SLAM (214) can be described as a pose graph optimization problem. As noted earlier, the SLAM algorithm is used to provide concurrent construction of a model of the environment (the scan), and an estimation of the state of the
scanner 120 moving within the environment. In other words, SLAM provides a way to track the location of a robot in the world in real-time and identify the locations of landmarks such as buildings, trees, rocks, walls, doors, windows, paintings, décor, furniture, and other world features. In addition to localization, SLAM also generates or builds up a model of the environment to locate objects including the landmarks that surround thescanner 120 and so that the scan data can be used to ensure that thescanner 120 is on the right path as thescanner 120 moves through the world, i.e., environment. So, the technical challenge with the implementation of SLAM is that while building the scan, thescanner 120 itself might lose track of where it is by virtue of its motion uncertainty because there is no presence of an existing map of the environment because the map is being generated simultaneously. - The basis for SLAM is to gather information from the set of
sensors 120 and motions over time and then use information about measurements and motion to reconstruct a map of the environment. The SLAM algorithm defines the probabilities of thescanner 120 being at a certain location in the environment, i.e., at certain coordinates, using a sequence of constraints. For example, consider that thescanner 120 moves in some environment, the SLAM algorithm is input the initial location of thescanner 120, say (0,0) initially, which is also called as Initial Constraints. The SLAM algorithm is then inputted several relative constraints that relate each pose of thescanner 120 to a previous pose of thescanner 120. Such constraints are also referred to as relative motion constraints. - The technical challenge of SLAM can also be described as follows. Consider that the scanner is moving in an unknown environment, along a trajectory described by the sequence of random variables x1:T={x1, . . . xT}. While moving, the scanner acquires a sequence of odometry measurements u1:T={u1, . . . , uT} and perceptions of the environment z1:T={z1, . . . , zT}. The “perceptions” include the captured data and the mapped detected
planes 410. Solving the full SLAM problem now includes estimating the posterior probability of the trajectory of the scanner 120 x1:T and the map M of the environment given all the measurements plus an initial position x0: P(x1:T, M|z1:T, u1:T, x0). The initial position x0 defines the position in the map and can be chosen arbitrarily. There are several known approaches to implement SLAM, for example, graph SLAM, multi-level relaxation SLAM, sparse matrix-based SLAM, hierarchical SLAM, etc. The technical solutions described herein are applicable regardless of which technique is used to implement SLAM. -
FIG. 3 depicts a graphical representation of an example SLAM implementation. In the depicted representation of the SLAM as agraph 300, everynode 310 corresponds to a pose of thescanner 120. Nearby poses are connected byedges 320 that model spatial constraints between poses of thescanner 120 arising from measurements. Edges et-1, t between consecutive poses model odometry measurements, while the other edges represent spatial constraints arising from multiple observations of the same part of the environment. - A graph-based SLAM approach constructs a simplified estimation problem by abstracting the raw sensor measurements. These raw measurements are replaced by the
edges 320 ingraph 300, which can then be seen as “virtual measurements.” Anedge 320 between twonodes 310 are labeled with a probability distribution over the relative locations of the two poses, conditioned to their mutual measurements. In general, the observation model P(zt|xt, Mt) is multi-modal, and therefore the Gaussian assumption does not hold. This means that a single observation zt might result in multiple potential edges connecting different poses in the graph, and the graph connectivity needs itself to be described as a probability distribution. Directly dealing with this multi-modality in the estimation process would lead to a large combinatorial increase of complexity. As a result of that, most practical approaches restrict the estimate to the most likely topology. Hence, a constraint resulting from observation has to be determined. - If the observations are affected by (locally) Gaussian noise and the data association is known, the goal of a graph-based mapping algorithm is to compute a Gaussian approximation of the posterior over the trajectory of the
scanner 120. This involves computing the mean of this Gaussian as the configuration of thenodes 310 that maximizes the likelihood of the observations. Once the mean is known, the information matrix of the Gaussian can be obtained in a straightforward fashion, as is known in the art. In the following the task of finding is characterized with this maximum as a constraint optimization problem. - Let x=(x1, . . . , xT)T be a vector of parameters, where xi describes the pose of node i. Let zij and Ωij be respectively the mean and the information matrix of a virtual measurement between the node i and the node j. This virtual measurement is a transformation that makes the observations acquired from i maximally overlap with the observation acquired from j. Further, let {circumflex over (z)}ij(xi, xj) be the prediction of a virtual measurement given a configuration of the nodes xi, and xj. Usually, this prediction is the relative transformation between the two nodes. Let e(xi, xj, zij) be a function that computes a difference between the expected observation {circumflex over (z)}ij and the real observation zij captured by the
scanner 120. For simplicity of notation, the indices of the measurement are encoded in the indices of the error function: eij(xi, xj)=zij−{circumflex over (z)}ij j(xi, xj). - If C is the set of pairs of indices for which a constraint (observation) z exists, the goal of a maximum likelihood approach is to find the configuration of the nodes x* that minimizes the negative log-likelihood F(x) of all the observations: F(x)=Σ(i,j)∈CFij, where Fij=eij TΩijeij. Accordingly, implementing SLAM includes solving the following equation and computing a Gaussian approximation of the posterior over the trajectory of the scanner 120: x*=argminxF(x).
- Several techniques are known for solving the above equations, for example, using Gauss-Newton or the Levenberg-Marquardt algorithms. The technical solutions provided by one or more embodiments of the present disclosure can be used regardless of how the SLAM algorithm is implemented, i.e., regardless of how the above equations are solved. The technical solutions described herein provide the set of constraints C that is used for implementing the SLAM algorithm, using whichever technique is to be used.
- Accordingly, implementing
global SLAM 214 includes determining constraints (222) betweennodes 320, i.e., submaps, objects, landmarks, or any other elements that are matched. Non-global constraints (also known as inter submaps constraints) are built automatically betweennodes 310 that are closely following each other on a trajectory of thescanner 120 in the environment. Global constraints (also referred to as loop closure constraints or intra submaps contraints) are constraints between a new submap andprevious nodes 310 that are considered “close enough” in space and a strong fit, i.e., a good match when running scan matching. Here, “close enough” is based on predetermined thresholds, for example, distance between the same landmark from two submaps being within a predetermined threshold. - For example, existing implementations of SLAM use measurements, such as LIDAR data, from the set of
sensors 122, to aggregate the measurements to generate the submaps and eventually themap 130. A technical challenge with such implementations is that the matching of the sets of measurements is inaccurate due to ambiguities or missing data. This may lead to misaligned sets of measurements and/or submaps, which in turn, cause an erroneous submap and/ormap 130. Typically, “loop closure” 224 is used to prevent such errors by compensating for accumulated erros. However, loop closure cannot be used in the cases where the same landmarks are not identified in two sets of measurements or submaps that are being combined or stitched. One of the causes of such technical challenges is that the existing techniques only rely on the captured data from thesensors 122 without any semantic or geometric analysis of the measurement data. - To address the technical challenges with the loop closure, such as mis-matching landmarks, and missing loop closure, embodiments of the present disclosure add semantic features in the map 130 (and submaps), such that the semantic features can be used as static landmarks. The semantic features can include, for example, a corner with a 90-degree angle, walls with a particular length, pillars with a defined shape that can be defined using geometry primitives like a circle. Here, a “semantic feature” is detected in the captured measurement data by analyzing the captured data and identifying particular landmarks based on the captured data having a specific user-identifiable arrangement that may not be readily discernable from only the data itself. For example, a 90 degree corner may not be discernable until captured measurement data is converted to a line map of the environment.
-
FIG. 4 depicts example semantic features detected according to one or more embodiments of the present disclosure. The semantic features include acorner 410, awall 420 of a measureable length, line crossings above a threshold (not shown). The detection of these features is performed on a single scan line generated from the 2D data captured by thescanner 120. This can be done by searching for specific formations in the measurement data. Detecting predetermined shapes, or formations in measurement data can be performed using known techniques. For example, in one or more embodiments of the present disclosure, image processing on submaps and remap the extracted features to scanlines. For example, line detection can be done with a specific convolution filters and/or a Hough transformation. Here, a “scan” can be a set of measurement data, where multiple such scans are combined to form a submap, and where several such submaps are combined to form themap 130. - Further, if the same semantic features are detected in multiple scans, the semantic feature is used as a constraint for matching of the multiple scans during the local SLAM. The semantic features are also used for initialization to generate submaps consisting of multiple matched data. This matching maybe implemented as nonlinear optimization with a cost function. In one or more embodiments of the present disclosure, the cost function can include equations for the distance of the semantic features from the
scanner 120 to improve the accuracy and robustness of the SLAM algorithm. Further, in one or more embodiments of the present disclosure, the semantic features can be used in some situations to improve the robustness and speed of optimization during initialization of the SLAM algorithm. - Additionally, the semantic features can be reused as indicator for loop closure in the case where the feature which can be identified globally e.g. line/wall segments through their length. If multiple such landmarks are identified between two submaps the loop closure can be evaluated using the timestamp and the semantic features for the alignment of the multiple submaps.
-
FIG. 5 depicts an example of improving loop closure using semantic features according to one or more embodiments of the present disclosure. As shown, the same semantic features, thecorner 410, and themeasurable wall 420, are detected in two submaps. The semantic features can be used for aligning the two submaps when generating themap 130. The semantic features can be used to create a constraint for such alignment. -
FIG. 6 depicts a result of an alignment using semantic features that are detected according to one or more embodiments of the present disclosure. As can be seen, the submaps and the semantic features, for example, thecorner 410, and themeasurable wall 420, are aligned better (e.g. more accurately) compared to the misalignment inFIG. 5 . - The semantic features can also be used in the global SLAM optimization as constraints for the connection between the submaps and the orientation of the
scanner 120. - Once the loop closure is completed, the
global SLAM 214 is completed by registering 226 the submaps and stitching the submaps to generate themap 130. In one or more embodiments of the present disclosure,SLAM 210 is performed iteratively as newer measurements are acquired by thescanner 120. - Referring to
FIG. 1 again, thecomputing system 110 can be a desktop computer, a laptop computer, a tablet computer, a phone, or any other type of computing device that can communicate with thescanner 120. In an embodiment, thecomputing system 110 is a portable computing device that is carried by the operator during scanning operations (e.g. clipped to a belt or carried in a pouch or pocket). - One or more operations for implementing SLAM can be performed by the
computing system 110. Alternatively, or in addition, one or more of the operations can be performed by aprocessor 124 that is equipped on thescanner 120. In one or more embodiments of the present disclosure, theprocessor 124 and thecomputing system 110 can implement SLAM in a distributed manner. Theprocessor 124 can include one or more processing units. Theprocessor 124 controls the measurements performed using the set ofsensors 122. In one or more examples, the measurements are performed based on one or more instructions received from thecomputing system 110. - In one or more embodiments of the present disclosure, the
computing device 110 and/or a display (not shown) of thescanner 120 provides a live view of themap 130 of the environment being scanned by thescanner 120 using the set ofsensors 122.Map 130 can be a 2D or 3D representation of the environment seen through the different sensors.Map 130 can be represented internally as a grid map. A grid map is a 2D or 3D arranged collection of cells (2D) or voxels (3D), representing an area of the environment. In one or more embodiments of the present disclosure, the grid map stores for every cell/voxel, a probability indicating if the cell area is occupied or not. Other representations of themap 130 can be used in one or more embodiments of the present disclosure. - As noted earlier, the
scanner 120, along with capturing themap 130, is also locating itself within the environment. Thescanner 120 uses odometry, which includes using data from motion or visual sensors to estimate the change in position of thescanner 120 over time. Odometry is used to estimate the position of thescanner 120 relative to a starting location. This method is sensitive to errors due to the integration of velocity measurements over time to give position estimates. - It should be noted that the implementation of the SLAM, local and global, itself is known, and also that embodiments of the present disclosure provide accurate data that can be used by such an implementation to provide an improved result of the SLAM implementation. The embodiments of the present disclosure, accordingly, facilitate improvement to computing technology. Further, embodiments of the present disclosure provide a practical application that facilitates generating the
map 130 of an environment. -
FIG. 7 depicts a flowchart of a method to determine semantic features and use such features as landmarks during implementation of SLAM according to one or more embodiments. Typically, SLAM is implemented in batches in a continuous manner. For example, after thescanner 120 has moved a predetermined distance, for example, 5 meters, 10 meters, 2 meters, or any other such predetermined distance, SLAM is performed to combine the measurement data captured over that traveled distance, i.e., since the last iteration of SLAM. Such a periodic batch performance is used to reduce time spent on the SLAM computations. In one or more embodiments of the present disclosure, SLAM can be executed in a continuous manner at the expense of a larger number of computations. It should be noted that in existing systems, the accuracy of the measurements starts drifting, i.e., decreasing as the scanner is moved shorter distances than the predetermined distance used for performing SLAM, for example, 10-15 centimeter. Accordingly, even with measures like loop closure, compensating for such inaccuracies over the span of a relatively large area such as a building, which can be 100 meters, 250 meters, etc., i.e., substantially larger than the distance at which errors start accumulating, is a technical challenge. - To improve the accuracy, existing systems have transitioned from using a manually defined position, for example, using a marker that is mounted at a particular location in the environment, to using the
camera 122C to identifying and using a natural feature in the environment as the landmark. However, a manual step is required for identifying the landmarks in each iteration, which can add to the error. One or more embodiments of the present disclosure address such a challenge and improves the workflow by having the operator of thescanner 120 identify a landmark once, and then identifying the landmark in subsequent scans/measurement data based on line drawings of the captured data. - The
method 700 depicted inFIG. 7 includes capturing measurement from thescanner 120 and detecting lines in the captured data, atblock 702. The lines are detected at runtime, as the measurements are being captured by thescanner 120. In one or more embodiments of the present disclosure, the lines are detected every time a predetermined number of scans are completed by thescanner 120. The predetermined number of scans can be one, five, ten, or any other value. Alternatively, or in addition, the line detection can be performed on demand in response to a command from the operator. Alternatively, or in addition, the line detection can be performed upon completion of a submap, in which case, the semantic features detected thereafter are used as constraints for the global SLAM only. Detecting the lines in the measured data can be performed using any known line detection/generation algorithms. For example, see “Line-segmentation Of 2d Laser Scanner Point Clouds For Indoor Slam Based On A Range Of Residuals,” by Peter et al. The contents of which are incorporated herein in its entirety. - The
method 700 further includes evaluating, automatically, length of each line-segment in the lines detected in the captured scan, atblock 704. Each line-segment corresponds to a wall or any other such feature in the environment being scanned. The length of each line-segment corresponds to an actual length of the corresponding feature. In one or more embodiments of the present disclosure, line-segments corresponding to a feature of at least a predetermined length, for example 50 centimeter, are identified, atblock 706. The line-segments can be identified based on the length of the line-segments themselves, which are scaled versions of the length of the corresponding features, the scaling factor being a preconfigured setting for thescanner 120. Further, orientations of the line-segments with at least the predetermined length are determined, atblock 708. - Further, the method includes detecting one or more semantic features based on the identified line-segments and their orientations, at
block 710. The semantic features can include walls of at least a second predetermined length, where the second predetermined length is more than the predetermined length used to identify the line-segments. Further, the semantic features can include corners, which are identified at junctions of two line-segments that cross each other at least at a predetermined angle value, for example, 90 degrees and more. The angle value considered is the angle formed towards thescanner 120, in one or more embodiments of the present disclosure. Junctions that form substantially a right angle, i.e., 90 degrees are also identified as semantic features. Each detected semantic feature is assigned a unique identifier, atblock 712. -
FIG. 8 depicts an example scenario where semantic features are detected according to one or more embodiments of the present disclosure. For eachscan 805, thescanner position 810 and a timestamp is recorded along with the captured measurement data. The line-segments 820 of at least the predetermined length are identified from thescan 805. Thesemantic features 830 are determined from the identified line-segments 820. Thesemantic features 830 are automatically detected in one or more embodiments of the present disclosure. In one or more embodiments of the present disclosure, the operator can manually select a subset of thesemantic features 830 that are automatically detected, the selected semantic features beign used as the landmarks. In this case, only the selected semantic features are assigned unique identifiers. - Referring to the flowchart in
FIG. 7 , the operations of detecting thesemantic features 830 is repeated in every new line that is identified as thescanner 120 recordsnew scans 805. In one or more embodiments of the present disclosure, along with assigning the unique identifier to the landmark, the submap that is generated is also annotated with the information of the landmark. - Further, the landmarks are evaluated for occurrence and stability, at
block 714. The landmarks that are not repeatably identifiable are removed from the list of landmarks to be used for map generation, in one or more embodiments of the present disclosure. The evaluation can be performed manually in one or more embodiments of the present disclosure. - The landmarks are used to perform a loop closure algorithm, at
block 716.FIG. 9 schematically illustrates an example scenario in which an offset (referred to as “drift”) is continuously introduced into the scan data. Consider that thescanner 120 is moving from a starting position 1510 (real pose). After some movements thescanner 120 is designated to return to an already mapped region, such as thestarting position 1510, however the measured position due to sensor variation and the subsequent measurement error is a different position 1520 (estimated pose). The loop closure algorithm(s) that are typically used detect theloop closure correction 1530 and corrects the pose and the maps that have been acquired so far by thescanner 120. As a consequence all positions in themap 130, including the scan positions 810, the registration points, and the points scanned and stored in the scans, change their coordinates based on theloop closure correction 1530. In a pure mapping application this may not introduce inefficiencies or other issues, however for thescanner 120 that uses scans from different scan positions, such a change in map coordinates leads to errors/inefficiencies because the scan positions are recorded before they are not automatically adapted in this manner. - For example,
FIG. 10 illustrates the accumulation of errors and inefficiencies. Thescanner 120 starts moving from thestart position 1510. After some movement thescanner 120 takes a scan as described herein from one of a plurality of scan positions 1610. When thescanner 120 arrives back in thestart position 1510 the measurement error due to sensor data variance causes the estimated pose 1520 to differ from thestart position 1510. After loop closure the recorded scan positions still have the same coordinates including the error while the map was corrected by the loop closure algorithm. Consequently the estimated positions of the scans have a deviation. As described earlier, when the loop closure is applied all positions in the map change. But as thescan positions 1610 have been recorded before, the scan positions are not automatically adapted. As a consequence there are offsets between thescan positions 1610 and the map acquired by thescanner 120. The error in a scan position and consequently the error of distance in the generated maps accumulates as thescanner 120 is used. Accordingly, the longer the path taken by thescanner 120, the larger the error(s) in the scan positions. By using a registration process (such as Cloud2Cloud registration for example) for the scans the errors in thescan positions 1610 can be corrected in the measurement data. However, such registration process requires additional processing power and time, which may be inefficient. - Accordingly, the loop closure can be performed using the landmarks that are detected using the
semantic features 830 as described herein. The relative observation of a landmark from thescanner 120 delivers an accurate position information and can correct the position of thescanner 120 in the absolute world and remove absolute inaccuracies accumulated from the mapping process. The more landmarks (observed with a good accuracy) the better the position accuracy of thescanner 120 and consequently the absolute accuracy of the maps scanned by thescanner 120. It should be noted that as used herein, “absolute accuracy” is the accuracy of measurements of a map that is scanned compared to a ground truth. For example, a side wall of a building has a real length of 100 m. The side wall when measured by thescanner 120 is 101.1 m. In this case, there is an absolute error of 1.1 m and an absolute accuracy of >1.1 for distance >100 m. Such kind of errors in the scan positions are mitigated using loop closure. -
FIG. 11 depicts anexample method 1100 for correcting scan positions using landmarks from semantic features while performing a scan according to one or more embodiments. In one or more examples, the operator stops and starts to record ascan 805 with thescanner 120 at ascan position 810 from the scan positions 1610 (FIG. 10 ). In another embodiment, thescanner 120 automatically stops and starts to record thescan 805 at thescan position 1610. Acquiring thescan 805 includes determining with processor system, in cooperation with thescanner scan 805 includes obtaining by thescanner 120 in cooperation with the processor system a plurality of 2D scan sets. Each of the plurality of 2D scan sets is a set of 2D coordinates of points on the object surface collected as thescanner 120 moves from the first registration position (1510) to a second registration position (e.g. position 1610). Each of the plurality of 2D scan sets is collected by thescanner 120 at a different position relative to thefirst registration position 1510. The plurality of the 2D scan sets are together referred to as a 2D map and each of the scan sets is a part of themap 130. - Referring now to the flowchart in
FIG. 11 , one or more landmarks are recorded using thesemantic features 830 when the scanner is at afirst scan position 1510, atblock 1102. The landmarks are detected, automatically, semi-automatically, or manually. For example, the operator can select “add landmark” 1212 (FIG. 12 ) via auser interface 1210 of thescanner 120. The operator captures thescan 805 from thepresent scan position 1510. Alternatively, or in addition, thescan 805 is automatically captured by thescanner 120 in response to an instruction executed by the controller. Thescan 805 is saved as complementary data with thescan position 1510 and thescan position 1510 is marked to indicate that it is a virtual landmark. In one or more examples, the virtual landmark is represented as a flag (or any other marker) in themap 130. Thepresent scan position 1510 of the scanner is recorded as part of the virtual landmark. - In one or more examples, the
scan 805 is captured to include specificsemantic features 830 in the environment. In addition, thescan 805 is captured to includespecific objects 1222 in the environment that are relatively immovable, and that can be used as reference objects at a future time when the moveable platform returns to thescan position 1510. For example, thescan 805 includesobjects 1222 that are identifiable and that facilitate the operator to recognize that the platform is at substantially the same position (1510) when the platform returns after mapping. For example, theidentifiable objects 1222 can include construction items such as windows, doors, corners, pillars, beams, and the like. Alternatively, or in addition, theidentifiable objects 1222 can include items that have a relatively immovable position, such as a fire extinguisher (which may be required at particular position because of regulations/code), a defibrillator, a statue, a water fountain, a plant, and the like. - In one or more examples, the
scanner 120 detects theidentifiable objects 1222 andsemantic features 830 in the environment as the mapping is being performed. For example, thescanner 120 is preconfigured with a list ofidentifiable objects 1222 andsemantic features 830 that can be present in the environment. Thescanner 120 performs an object and feature recognition in the field of view as the platform is being moved. If any one or more of the objects and features from the list is detected in the field of view, thescanner 120 notifies the operator that the present location can be used as a landmark. For example, avisual notification 1250 is shown on theuser interface 1210. The visual notification can be a popup, a dropdown, a flashing light, or any other type of visual notification. Alternatively, or in addition, the notification can include an audio notification, a haptic notification, or any other type of notification to suggest the operator that thepresent position 1510 can be used as a virtual landmark. - The
scanner 120 continues to capturescans 805 at multipleother scan positions 1610 and returning to the first scan position that was marked as a virtual landmark, atblock 1104. Capturing the present position procedure is repeated for everyscan 805 captured by thescanner 120. For example, if thescanner 120 captures n scans a data structure holds n positions with n links to the corresponding measurement data of the portion scanned. In one or more examples, thescanner 120 saves the present position in a data structure such as a list of positions. Every position in the data structure is directly linked to the data structure that is used to store the measurements of the corresponding portion of the environment. - At the
position 1510 of the virtual landmark that was added before, thescanner 120 computes themeasurement error 1530 that is input into the SLAM algorithms to correct the error/drift accumulated from walking or moving the scanner around the scanned portion of the environment, atblock 1106. In one or more embodiments of the present disclosure, computing themeasurement error 1530 includes moving thescanner 120 to an estimatedposition 1520. The estimated position is an estimate of thefirst scan position 1510 where the virtual landmark was recorded. In one or more embodiments of the present disclosure, the operator selects a “measure landmark”instruction 1214 at the estimatedposition 1520. In one or more examples, (seeFIG. 12 ) the user interface 3610 depicts to the operator the original view (e.g., photo) 1220 that was captured by thescanner 120 when the virtual landmark was recorded. Apresent view 1230 from the mobile device 443 is overlaid on theoriginal image 1220 in a transparent manner. The operator can move thescanner 120 until thepresent view 1230 substantially matches theoriginal image 1220 in the overlay. Once the operator indicates that the overlays match, the system records a relative observation of thepresent scan position 1520 of thescanner 120. Thedifference 1530 between the recordedposition 1510 of the virtual landmark and thepresent position 1520 is used as the error correction to update and correct the mapping positions. - In one or more examples, the difference is computed as a difference in the
original image 1220 and thepresent view 1230 when thescanner 120 is at the virtual landmark. For example, the difference between the images is computed based on the landmarks in theimage 1220 and thepresent view 1230. - The
method 1100 further includes using themeasurement error 1530 to correct the coordinates captured by thescanner 120, atblock 1108. The portion of themap 130 that is scanned and stored since capturing the virtual landmark is updated using themeasurement error 1530, in one or more examples. In one or more examples, a loop closure operation is executed on themap 120, and parts of the map are corrected in order to match the real pose, which is thestarting position 1510, with the estimated pose, which is thedifferent position 1520. The loop closure algorithm calculates a displacement for each part of themap 130 that is shifted by the algorithm. - In one or more examples, the
scanner 120 determines thescan positions 1610 linked to each portion of themap 130, atblock 1110. In one or more examples, a lookup is performed over the data structure that saves the list of positions. The lookup costs a single processor operation, such as an array lookup. Thescanner 120 applies the displacement vector for a portion of themap 130 to the corresponding scan positions saved in the data structure and saves the resulting displaced (or revised) scan positions back into the data structure, atblock 1111. Thescanner 120 computes displaced scan positions for each of the savedscan positions 1610 in the data structure. The procedure can be repeated every time the loop closure algorithm is applied. - The displaced scan positions represent corrected scan positions of the
scans 805 that can be used directly without applying further computational expensive point cloud registration algorithms. The accuracy of thescan positions 1610 depends on the sensor accuracy of thescanner 120. As shown inFIG. 10 , thedisplacement vectors 1810 for the portions of themap 130 are determined based on the loop closure operation. Thescanner 120 applies thedisplacement vectors 1810 to thescan positions 1610 linked to the portions of the map by the data structure as described herein. The resulting displacedscan positions 1910 are accordingly calculated by applying thedisplacement vectors 1810 to the scan positions 1610. The displacedscan positions 1910 are now correctly located. -
FIG. 13 depicts a resulting correction performed using landmarks from semantic features according to one or more embodiments of the present disclosure. Two scans are shown, afirst scan 805 captured from afirst scan position 1510, and asecond scan 805′ captured from asecond scan position 1520. Here, thesecond scan position 1520 is “wrong position” because the operator was trying to be substantially at thefirst position 1510. The “wrong position” is result of the drift. As can be seen, theviews semantic features 830 are used to determine theoffsets 1810 as described herein. Theoffsets 1810 are further used to determine theloop closure correction 1530, which is then applied to align theviews - Referring again to the flowchart in
FIG. 7 , themethod 700 includes determining a constraint for SLAM implementation based on the landmarks that are determined using thesemantic features 830, atblock 718. Determining the constrain includes generating a relationship by matching thesemantic features 830 that are detected from afirst position 1610 with corresponding (same) landmarks that are detected in an earlier frame. Here, each frame is captured at a particular pose of thescanner 120. The pose can include a position (i.e., coordinates in the environment), and orientation of thescanner 120.FIG. 14 depicts a block diagram of matching landmarks from one frame with another according to one or more embodiments of the present disclosure. Ct-1 depicts a first position of thescanner 120 at time t−1, and Ct depicts a second position of thescanner 120 as it is moved to scan the environment. From each of these positions, thescanner 120 can observe and capture measurements of a plane P. Thescanner 120 captures a scan Ia at the first position and a scan Ib at the second position. Each of these scans includes the plane P, which has a normal depicted by n. - The
scanner 120 moves from the first position to the second position at a predetermined speed, for example, R meters per second. In addition, thescanner 120 is configured to capture successive frames at a predetermined frequency, for example, 10 Hz, 15 Hz, etc. In one or more embodiments of the present disclosure, thecomputing system 110 processes the captured frames at a second predetermined frequency, for example, 30 Hz, 45 Hz, etc. Further, consider that the plane P has a landmark ai that is captured at a position x in the scan Ia from the first position; and further, that the landmark ai is at a position y in the second scan Ib that is captured from the second position. - Matching the landmarks detected from the
first position 1610 and those detected from the second position 1620 includes determining the common semantic features, and common landmarks from the two positions. In this example scenario ofFIG. 14 , the landmark ai includes at least one of such common semantic features. - Once ai is determined, the matching further includes determining a relationship between the positions of the same landmark from the separate scan positions. The relationship can be a mapping between the point x that represents the landmark from the first position, and the point y that represents the same landmark from the second position. For example, the
computing system 110 determines a mapping such as Hx≈y. Here, H can be a matrix that translates and rotates x, where x and y can be 2D or 3D coordinates. In one or more embodiments of the present disclosure, x and y can be matrices that represent more than one point. H is a relative measurement constraint that can be used by the scanner when implementing the SLAM algorithm. - The
method 700 further includes executing the SLAM algorithm, atblock 720. The SLAM algorithm can be implemented by thecomputing system 110, and/or theprocessor 124. Such use of the constraints generated from the landmarks and/or thesemantic features 830 improves the final result of the SLAM algorithm (seeFIG. 13 ). - As an example, consider the use of a landmark that can be seen by the
scanner 120 from various locations, which would be relative measurement constraints every time the scanner sees a landmark. So, SLAM can use those constraints in order to find the most likely configuration of the scanner path along with the location of landmarks. - Embodiments of the present disclosure facilitate improvements to results of SLAM as well as loop closure, and consequently an improved scanning system for generating maps of an environment.
-
FIG. 15 ,FIG. 16 , andFIG. 17 depict an embodiment of asystem 30 having ahousing 32 that includes abody portion 34 and ahandle portion 36. Thesystem 30 can be used as thescanner 120. In an embodiment, thehandle 36 may include an actuator 38 that allows the operator to interact with thesystem 30. In the exemplary embodiment, thebody 34 includes a generallyrectangular center portion 35 with aslot 40 formed in anend 42. Theslot 40 is at least partially defined by apair walls 44 that are angled towards asecond end 48. As will be discussed in more detail herein, a portion of a two-dimensional scanner 50 is arranged between thewalls 44. Thewalls 44 are angled to allow thescanner 50 to operate by emitting a light over a large angular area without interference from thewalls 44. As will be discussed in more detail herein, theend 42 may further include a three-dimensional camera orRGBD camera 60. - Extending from the
center portion 35 is amobile device holder 41. Themobile device holder 41 is configured to securely couple amobile device 43 to thehousing 32. Theholder 41 may include one or more fastening elements, such as a magnetic or mechanical latching element for example, that couples themobile device 43 to thehousing 32. In an embodiment, themobile device 43 is coupled to communicate with acontroller 68. The communication between thecontroller 68 and themobile device 43 may be via any suitable communications medium, such as wired, wireless or optical communication mediums for example. - In the illustrated embodiment, the
holder 41 is pivotally coupled to thehousing 32, such that it may be selectively rotated into a closed position within a recess 46. In an embodiment, the recess 46 is sized and shaped to receive theholder 41 with themobile device 43 disposed therein. - In the exemplary embodiment, the
second end 48 includes a plurality ofexhaust vent openings 56. In an embodiment theexhaust vent openings 56 are fluidly coupled to intake vent openings 58 arranged on a bottom surface 62 ofcenter portion 35. The intake vent openings 58 allow external air to enter a conduit 64 having an opposite opening 66 in fluid communication with the hollow interior 67 of thebody 34. In an embodiment, the opening 66 is arranged adjacent to acontroller 68 which has one or more processors that is operable to perform the methods described herein. In an embodiment, the external air flows from the opening 66 over or around thecontroller 68 and out theexhaust vent openings 56. - The
controller 68 is coupled to a wall 70 ofbody 34. In an embodiment, the wall 70 is coupled to or integral with thehandle 36. Thecontroller 68 is electrically coupled to the2D scanner 50, the3D camera 60, apower source 72, an inertial measurement unit (IMU) 74, alaser line projector 76, and ahaptic feedback device 77. - Elements are shown of the
system 30 with themobile device 43 installed or coupled to thehousing 32.Controller 68 is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. Thecontroller 68 includes one ormore processing elements 78. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one ormore processors 78 have access tomemory 80 for storing information. -
Controller 68 can convert the analog voltage or current level provided by2D scanner 50,camera 60 andIMU 74 into a digital signal to determine a distance from thesystem 30 to an object in the environment. In an embodiment, thecamera 60 is a 3D or RGBD type camera.Controller 68 uses the digital signals that act as input to various processes for controlling thesystem 30. The digital signals represent one ormore system 30 data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation and roll orientation. As will be discussed in more detail, the digital signals may be from components internal to thehousing 32 or from sensors and devices located in themobile device 43. - In general, when the
mobile device 43 is not installed,controller 68 accepts data from2D scanner 50 andIMU 74 and is given certain instructions for the purpose of generating a two-dimensional map of a scanned environment.Controller 68 provides operating signals to the2D scanner 50, thecamera 60,laser line projector 76 andhaptic feedback device 77.Controller 68 also accepts data fromIMU 74, indicating, for example, whether the operator is operating in the system in the desired orientation. Thecontroller 68 compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, generates a signal that activates thehaptic feedback device 77. The data received by thecontroller 68 may be displayed on a user interface coupled tocontroller 68. The user interface may be one or more LEDs (light-emitting diodes) 82, an LCD (liquid-crystal diode) display, a CRT (cathode ray tube) display, or the like. A keypad may also be coupled to the user interface for providing data input tocontroller 68. In one embodiment, the user interface is arranged or executed on themobile device 43. - The
controller 68 may also be coupled to external computer networks such as a local area network (LAN) and the Internet. A LAN interconnects one or more remote computers, which are configured to communicate withcontroller 68 using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), RS-232, ModBus, and the like.Additional systems 30 may also be connected to LAN with thecontrollers 68 in each of thesesystems 30 being configured to send and receive data to and from remote computers andother systems 30. The LAN may be connected to the Internet. This connection allowscontroller 68 to communicate with one or more remote computers connected to the Internet. - The
processors 78 are coupled tomemory 80. Thememory 80 may include random access memory (RAM)device 84, a non-volatile memory (NVM)device 86, a read-only memory (ROM)device 88. In addition, theprocessors 78 may be connected to one or more input/output (I/O)controllers 90 and acommunications circuit 92. In an embodiment, thecommunications circuit 92 provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN discussed above. -
Controller 68 includes operation control methods described herein, which can be embodied in application code. These methods are embodied in computer instructions written to be executed byprocessors 78, typically in the form of software. The software can be encoded in any language, including, but not limited to, assembly language, VHDL (Verilog Hardware Description Language), VHSIC HDL (Very High Speed IC Hardware Description Language), Fortran (formula translation), C, C++, C #, Objective-C, Visual C++, Java, ALGOL (algorithmic language), BASIC (beginners all-purpose symbolic instruction code), visual BASIC, ActiveX, HTML (Hypertext Markup Language), Python, Ruby and any combination or derivative of at least one of the foregoing. - Coupled to the
controller 68 is the2D scanner 50. The2D scanner 50measures 2D coordinates in a plane. In the exemplary embodiment, the scanning is performed by steering light within a plane to illuminate object points in the environment. The2D scanner 50 collects the reflected (scattered) light from the object points to determine 2D coordinates of the object points in the 2D plane. In an embodiment, the2D scanner 50 scans a spot of light over an angle while at the same time measuring an angle value and corresponding distance value to each of the illuminated object points. - Examples of
2D scanners 50 include but are not limited to Model LMS103 scanners manufactured by Sick, Inc of Minneapolis, MN and scanner Models URG-04LX-UG01 and UTM-30LX manufactured by Hokuyo Automatic Co., Ltd of Osaka, Japan. The scanners in the Sick LMS103 family measure angles over a 270-degree range and over distances up to 20 meters. The Hoyuko model URG-04LX-UG01 is a low-cost 2D scanner that measures angles over a 240-degree range and distances up to 4 meters. The Hoyuko model UTM-30LX is a 2D scanner that measures angles over a 270-degree range and to distances up to 30 meters. It should be appreciated that the above 2D scanners are exemplary and other types of 2D scanners are also available. - In an embodiment, the
2D scanner 50 is oriented so as to scan a beam of light over a range of angles in a generally horizontal plane (relative to the floor of the environment being scanned). At instants in time the2D scanner 50 returns an angle reading and a corresponding distance reading to provide 2D coordinates of object points in the horizontal plane. In completing one scan over the full range of angles, the 2D scanner returns a collection of paired angle and distance readings. As thesystem 30 is moved from place to place, the2D scanner 50 continues to return 2D coordinate values. These 2D coordinate values are used to locate the position of thesystem 30 thereby enabling the generation of a two-dimensional map or floorplan of the environment. - Also coupled to the
controller 68 is theIMU 74. TheIMU 74 is a position/orientation sensor that may include accelerometers 94 (inclinometers),gyroscopes 96, a magnetometer orcompass 98, and altimeters. In the exemplary embodiment, theIMU 74 includesmultiple accelerometers 94 andgyroscopes 96. Thecompass 98 indicates a heading based on changes in magnetic field direction relative to the earth's magnetic north. TheIMU 74 may further have an altimeter that indicates altitude (height). An example of a widely used altimeter is a pressure sensor. By combining readings from a combination of position/orientation sensors with a fusion algorithm that may include a Kalman filter, relatively accurate position and orientation measurements can be obtained using relatively low-cost sensor devices. In the exemplary embodiment, theIMU 74 determines the pose or orientation of thesystem 30 about three-axis to allow a determination of a yaw, roll and pitch parameter. - The
system 30 further includes acamera 60 that is a 3D or RGB-D camera. As used herein, theterm 3D camera refers to a device that produces a two-dimensional image that includes distances to a point in the environment from the location ofsystem 30. The3D camera 30 may be a range camera or a stereo camera. In an embodiment, the3D camera 30 includes an RGB-D sensor that combines color information with a per-pixel depth information. In an embodiment, the3D camera 30 may include an infrared laser projector 31, a left infrared camera 33, a right infrared camera 39, and a color camera 37. In an embodiment, the3D camera 60 is a RealSense™ camera model R200 manufactured by Intel Corporation. In still another embodiment, the3D camera 30 is a RealSense™ LIDAR camera model L515 manufactured by Intel Corporation. - In an embodiment, when the
mobile device 43 is coupled to thehousing 32, themobile device 43 becomes an integral part of thesystem 30. In an embodiment, themobile device 43 is a cellular phone, a tablet computer or a personal digital assistant (PDA). Themobile device 43 may be coupled for communication via a wired connection, such asports port 103 is coupled for communication to theprocessor 78, such as via I/O controller 90 for example. Theports - The
mobile device 43 is a suitable electronic device capable of accepting data and instructions, executing the instructions to process the data, and presenting the results. Themobile device 43 includes one ormore processing elements 104. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one ormore processors 104 have access tomemory 106 for storing information. - The
mobile device 43 can convert the analog voltage or current level provided bysensors 108 andprocessor 78.Mobile device 43 uses the digital signals that act as input to various processes for controlling thesystem 30. The digital signals represent one ormore system 30 data including but not limited to distance to an object, images of the environment, acceleration, pitch orientation, yaw orientation, roll orientation, global position, ambient light levels, and altitude for example. - In general,
mobile device 43 accepts data fromsensors 108 and is given certain instructions for the purpose of generating or assisting theprocessor 78 in the generation of a two-dimensional map or three-dimensional map of a scanned environment.Mobile device 43 provides operating signals to theprocessor 78, thesensors 108 and adisplay 110.Mobile device 43 also accepts data fromsensors 108, indicating, for example, to track the position of themobile device 43 in the environment or measure coordinates of points on surfaces in the environment. Themobile device 43 compares the operational parameters to predetermined variances (e.g. yaw, pitch or roll thresholds) and if the predetermined variance is exceeded, may generate a signal. The data received by themobile device 43 may be displayed ondisplay 110. In an embodiment, thedisplay 110 is a touch screen device that allows the operator to input data or control the operation of thesystem 30. - The
controller 68 may also be coupled to external networks such as a local area network (LAN), a cellular network and the Internet. A LAN interconnects one or more remote computers, which are configured to communicate withcontroller 68 using a well-known computer communications protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), RS-232, ModBus, and the like.Additional systems 30 may also be connected to LAN with thecontrollers 68 in each of thesesystems 30 being configured to send and receive data to and from remote computers andother systems 30. The LAN may be connected to the Internet. This connection allowscontroller 68 to communicate with one or more remote computers connected to the Internet. - The
processors 104 are coupled tomemory 106. Thememory 106 may include random access memory (RAM) device, a non-volatile memory (NVM) device, and a read-only memory (ROM) device. In addition, theprocessors 104 may be connected to one or more input/output (I/O)controllers 112 and acommunications circuit 114. In an embodiment, thecommunications circuit 114 provides an interface that allows wireless or wired communication with one or more external devices or networks, such as the LAN or the cellular network discussed above. -
Processor 104 includes operation control methods described herein, which can be embodied in application code. These methods are embodied in computer instructions written to be executed byprocessors - Also coupled to the
processor 104 are thesensors 108. Thesensors 108 may include but are not limited to: amicrophone 116; aspeaker 118; a front or rear facingcamera 160; accelerometers 162 (inclinometers),gyroscopes 164, a magnetometers orcompass 126; a global positioning satellite (GPS)module 168; abarometer 170; aproximity sensor 132; and an ambientlight sensor 134. By combining readings from a combination ofsensors 108 with a fusion algorithm that may include a Kalman filter, relatively accurate position and orientation measurements can be obtained. - It should be appreciated that the
sensors scanner 30 may have different characteristics than thesensors 108 ofmobile device 43. For example, the resolution of thecameras accelerometers gyroscopes more sensors 108 in themobile device 43 may be of higher accuracy than the correspondingsensors 74 in thesystem 30. As described in more detail herein, in some embodiments theprocessor 78 determines the characteristics of each of thesensors 108 and compares them with the corresponding sensors in thesystem 30 when the mobile device. Theprocessor 78 then selects whichsensors mobile device 43 may have additional sensors (e.g. microphone 116, camera 160) that may be used to enhance operation compared to operation of thesystem 30 without themobile device 43. In still further embodiments, thesystem 30 does not include theIMU 74 and theprocessor 78 uses thesensors 108 for tracking the position and orientation/pose of thesystem 30. In still further embodiments, the addition of themobile device 43 allows thesystem 30 to utilize thecamera 160 to perform three-dimensional (3D) measurements either directly (using an RGB-D camera) or using photogrammetry techniques to generate 3D maps. In an embodiment, theprocessor 78 uses the communications circuit (e.g. a cellular 4G internet connection) to transmit and receive data from remote computers or devices. - In the exemplary embodiment, the
system 30 is a handheld portable device that is sized and weighted to be carried by a single person during operation. Therefore, the plane 136 in which the2D scanner 50 projects a light beam may not be horizontal relative to the floor or may continuously change as the computer moves during the scanning process. Thus, the signals generated by theaccelerometers 94,gyroscopes 96 and compass 98 (or the corresponding sensors 108) may be used to determine the pose (yaw, roll, tilt) of thesystem 30 and determine the orientation of theplane 51. - In an embodiment, it may be desired to maintain the pose of the system 30 (and thus the plane 136) within predetermined thresholds relative to the yaw, roll and pitch orientations of the
system 30. In an embodiment, ahaptic feedback device 77 is disposed within thehousing 32, such as in thehandle 36. Thehaptic feedback device 77 is a device that creates a force, vibration or motion that is felt or heard by the operator. Thehaptic feedback device 77 may be, but is not limited to: an eccentric rotating mass vibration motor or a linear resonant actuator for example. The haptic feedback device is used to alert the operator that the orientation of the light beam from2D scanner 50 is equal to or beyond a predetermined threshold. In operation, when theIMU 74 measures an angle (yaw, roll, pitch or a combination thereof), thecontroller 68 transmits a signal to amotor controller 138 that activates avibration motor 140. Since the vibration originates in thehandle 36, the operator will be notified of the deviation in the orientation of thesystem 30. The vibration continues until thesystem 30 is oriented within the predetermined threshold or the operator releases the actuator 38. In an embodiment, it is desired for the plane 136 to be within 10-15 degrees of horizontal (relative to the ground) about the yaw, roll and pitch axes. - Referring now to
FIG. 18 ,FIG. 19 , andFIG. 20 , an embodiment is shown of amobile scanning platform 1800. Themobile scanning platform 1800 can be used as thescanner 120. Themobile scanning platform 1800 includes abase unit 1802 having a plurality ofwheels 1804. Thewheels 1804 are rotated bymotors 1805. In an embodiment, anadapter plate 1807 is coupled to thebase unit 1802 to allow components and modules to be coupled to thebase unit 1802. Themobile scanning platform 1800 further includes a2D scanner 1808 and a3D scanner 1810. In the illustrated embodiment, eachscanner 2D scanner 1808 may be the scanner illustrated and described herein. As will be described in more detail herein, in some embodiments the2D scanner 1808 is removable from the adapter plate 1806 and is used to generate a map of the environment, plan a path for the mobile scanning platform to follow, and define 3D scanning locations. In the illustrated embodiment, the2D scanner 1808 is slidably coupled to abracket 1811 that couples the2D scanner 1808 to theadapter plate 1807. - In an embodiment, the
3D scanner 1810 is a time-of-flight (TOF) laser scanner such as that shown and described herein. Thescanner 1810 may be that described in commonly owned U.S. Pat. No. 8,705,012, which is incorporated by reference herein. In an embodiment, the3D scanner 1810 mounted on a pedestal orpost 1809 that elevates the3D scanner 1810 above (e.g. further from the floor than) the other components in themobile scanning platform 1800 so that the emission and receipt of the light beam is not interfered with. In the illustrated embodiment, thepedestal 1809 is coupled to theadapter plate 1807 by au-shaped frame 1814. - In an embodiment, the
mobile scanning platform 1800 further includes acontroller 1816. Thecontroller 1816 is a computing device having one or more processors and memory. The one or more processors are responsive to non-transitory executable computer instructions for performing operational methods such as those described herein. The processors may be microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and generally any device capable of performing computing functions. The one or more processors have access to memory for storing information. - Coupled for communication to the
controller 1816 is acommunications circuit 1818 and an input/output hub 1820. In the illustrated embodiment, thecommunications circuit 1818 is configured to transmit and receive data via a wireless radio-frequency communications medium, such as WIFI or Bluetooth for example. In an embodiment, the2D scanner 1808 communicates with thecontroller 1816 via thecommunications circuit 1818 - In an embodiment, the
mobile scanning platform 1800 further includes amotor controller 1822 that is operably coupled to the control themotors 1805. In an embodiment, themotor controller 1822 is mounted to an external surface of thebase unit 1802. In another embodiment, themotor controller 1822 is arranged internally within thebase unit 1802. Themobile scanning platform 1800 further includes apower supply 1824 that controls the flow of electrical power from a power source, such asbatteries 1826 for example. Thebatteries 1826 may be disposed within the interior of thebase unit 1802. In an embodiment, thebase unit 1802 includes a port (not shown) for coupling the power supply to an external power source for recharging thebatteries 1826. In another embodiment, thebatteries 1826 are removable or replaceable. - It should be appreciated that the automated or motorized
mobile scanning platform 1800 is illustrated and described for exemplary purposes and the claims should not be so limited. In other embodiments, the mobile scanning platform may be a wheeled mobile assembly that is pushed by an operator. In still further embodiments, the mobile platform may be mounted on the operator, such as a backpack configuration. - It should be appreciated that while embodiments herein describe supporting the registration of landmarks in a 3D point cloud generated by a phase-shift TOF laser scanner, this is, for example, purposes and the claims should not be so limited. In other embodiments, the 3D coordinate data or point cloud may be generated by any type of 3D measurement device, such as but not limited to a pulsed TOF laser scanner, frequency modulated continuous wave (FMCW) scanner, triangulation scanner, an area scanner, a structured light scanner, a laser line probe, a laser tracker, or a combination of the foregoing. Further, it should be appreciated that the examples described herein show top views of scan data; however, side views can also be used for registration, and such registration can also be improved as described herein.
- It should be appreciated that while 3D coordinate data may be used for training, the methods described herein for verifying the registration of landmarks may be used with either two-dimensional or three-dimensional data sets.
- Technical effects and benefits of the disclosed embodiments include, but are not limited to, increasing scan quality and a visual appearance of scans acquired by the 3D coordinate measurement device.
- Turning now to
FIG. 21 , a computer system 2100 is generally shown in accordance with an embodiment. The computer system 2100 can be an electronic, computer framework comprising and/or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein. The computer system 2100 can be easily scalable, extensible, and modular, with the ability to change to different services or reconfigure some features independently of others. The computer system 2100 may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system 2100 may be a cloud computing node. Computer system 2100 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system 2100 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices. - As shown in
FIG. 21 , the computer system 2100 has one or more central processing units (CPU(s)) 2101 a, 2101 b, 2101 c, etc. (collectively or generically referred to as processor(s) 2101). The processors 2101 can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations. The processors 2101, also referred to as processing circuits, are coupled via a system bus 2102 to a system memory 2103 and various other components. The system memory 2103 can include a read only memory (ROM) 2104 and a random access memory (RAM) 2105. The ROM 2104 is coupled to the system bus 2102 and may include a basic input/output system (BIOS), which controls certain basic functions of the computer system 2100. The RAM is read-write memory coupled to the system bus 2102 for use by the processors 2101. The system memory 2103 provides temporary memory space for operations of said instructions during operation. The system memory 2103 can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems. - The computer system 2100 comprises an input/output (I/O) adapter 2106 and a communications adapter 2107 coupled to the system bus 2102. The I/O adapter 2106 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 2108 and/or any other similar component. The I/O adapter 2106 and the hard disk 2108 are collectively referred to herein as a mass storage 2110.
- Software 2111 for execution on the computer system 2100 may be stored in the mass storage 2110. The mass storage 2110 is an example of a tangible storage medium readable by the processors 2101, where the software 2111 is stored as instructions for execution by the processors 2101 to cause the computer system 2100 to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail. The communications adapter 2107 interconnects the system bus 2102 with a network 2112, which may be an outside network, enabling the computer system 2100 to communicate with other such systems. In one embodiment, a portion of the system memory 2103 and the mass storage 2110 collectively store an operating system, which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown in
FIG. 21 . - Additional input/output devices are shown as connected to the system bus 2102 via a display adapter 2115 and an interface adapter 2116 and. In one embodiment, the adapters 2106, 2107, 2115, and 2116 may be connected to one or more I/O buses that are connected to the system bus 2102 via an intermediate bus bridge (not shown). A display 2119 (e.g., a screen or a display monitor) is connected to the system bus 2102 by a display adapter 2115, which may include a graphics controller to improve the performance of graphics intensive applications and a video controller. A keyboard 2121, a mouse 2122, a speaker 2123, etc. can be interconnected to the system bus 2102 via the interface adapter 2116, which may include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit. Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI). Thus, as configured in
FIG. 21 , the computer system 2100 includes processing capability in the form of the processors 2101, and, storage capability including the system memory 2103 and the mass storage 2110, input means such as the keyboard 2121 and the mouse 2122, and output capability including the speaker 2123 and the display 2119. - In some embodiments, the communications adapter 2107 can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network 2112 may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device may connect to the computer system 2100 through the network 2112. In some examples, an external computing device may be an external webserver or a cloud computing node.
- It is to be understood that the block diagram of
FIG. 21 is not intended to indicate that the computer system 2100 is to include all of the components shown inFIG. 21 . Rather, the computer system 2100 can include any appropriate fewer or additional components not illustrated inFIG. 21 (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the embodiments described herein with respect to computer system 2100 may be implemented with any appropriate logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware, in various embodiments. - It will be appreciated that aspects of the present disclosure may be embodied as a system, method, or computer program product and may take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.), or a combination thereof. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.
- One or more computer-readable medium(s) may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In one aspect, the computer-readable storage medium may be a tangible medium containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.
- A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium, and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
- The computer-readable medium may contain program code embodied thereon, which may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. In addition, computer program code for carrying out operations for implementing aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
- It will be appreciated that aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block or step of the flowchart illustrations and/or block diagrams, and combinations of blocks or steps in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
- Terms such as processor, controller, computer, DSP, FPGA are understood in this document to mean a computing device that may be located within an instrument, distributed in multiple elements throughout an instrument, or placed external to an instrument.
- While the invention has been described in detail in connection with only a limited number of embodiments, it should be readily understood that the invention is not limited to such disclosed embodiments. Rather, the invention can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the invention. Additionally, while various embodiments of the invention have been described, it is to be understood that aspects of the invention may include only some of the described embodiments. Accordingly, the invention is not to be seen as limited by the foregoing description but is only limited by the scope of the appended claims.
Claims (1)
1. A method comprising:
recording a landmark at a first scan position of a scanner, the landmark based at least in part on a semantic feature of scan data captured by the scanner, the semantic feature identified using line-segments of the scan data;
capturing, by the scanner while moving through the environment, additional scan data at a second scan position;
responsive to the scanner returning to the first scan position associated with the landmark, computing a measurement error; and
correcting, using the measurement error, at least a portion of the scan data or the additional scan data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/469,258 US20240004080A1 (en) | 2020-05-29 | 2023-09-18 | Capturing environmental scans using landmarks based on semantic features |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202063031926P | 2020-05-29 | 2020-05-29 | |
US17/314,102 US11867818B2 (en) | 2020-05-29 | 2021-05-07 | Capturing environmental scans using landmarks based on semantic features |
US18/469,258 US20240004080A1 (en) | 2020-05-29 | 2023-09-18 | Capturing environmental scans using landmarks based on semantic features |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/314,102 Continuation US11867818B2 (en) | 2020-05-29 | 2021-05-07 | Capturing environmental scans using landmarks based on semantic features |
Publications (1)
Publication Number | Publication Date |
---|---|
US20240004080A1 true US20240004080A1 (en) | 2024-01-04 |
Family
ID=76098793
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/314,102 Active 2041-10-13 US11867818B2 (en) | 2020-05-29 | 2021-05-07 | Capturing environmental scans using landmarks based on semantic features |
US18/469,258 Pending US20240004080A1 (en) | 2020-05-29 | 2023-09-18 | Capturing environmental scans using landmarks based on semantic features |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/314,102 Active 2041-10-13 US11867818B2 (en) | 2020-05-29 | 2021-05-07 | Capturing environmental scans using landmarks based on semantic features |
Country Status (2)
Country | Link |
---|---|
US (2) | US11867818B2 (en) |
EP (1) | EP3916422A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220277479A1 (en) * | 2021-02-26 | 2022-09-01 | Samsung Electronics Co., Ltd. | Method and apparatus for simultaneous localization and mapping, and computer-readable storage medium |
CN114862956B (en) | 2022-07-07 | 2022-09-09 | 中国长江三峡集团有限公司 | Scanning equipment mobile positioning method and system suitable for tunnel construction |
CN115752476B (en) * | 2022-11-29 | 2024-06-18 | 重庆长安汽车股份有限公司 | Vehicle ground library repositioning method, device, equipment and medium based on semantic information |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102009015922B4 (en) | 2009-03-25 | 2016-12-15 | Faro Technologies, Inc. | Method for optically scanning and measuring a scene |
DE102010032726B3 (en) | 2010-07-26 | 2011-11-24 | Faro Technologies, Inc. | Device for optically scanning and measuring an environment |
WO2013049597A1 (en) * | 2011-09-29 | 2013-04-04 | Allpoint Systems, Llc | Method and system for three dimensional mapping of an environment |
SG10201700299QA (en) | 2017-01-13 | 2018-08-30 | Otsaw Digital Pte Ltd | Three-dimensional mapping of an environment |
US10824773B2 (en) | 2017-03-28 | 2020-11-03 | Faro Technologies, Inc. | System and method of scanning an environment and generating two dimensional images of the environment |
EP3474230B1 (en) * | 2017-10-18 | 2020-07-22 | Tata Consultancy Services Limited | Systems and methods for edge points based monocular visual slam |
US11145116B2 (en) | 2017-11-21 | 2021-10-12 | Faro Technologies, Inc. | System and method of scanning an environment and generating two dimensional images of the environment |
US10989532B2 (en) | 2018-10-08 | 2021-04-27 | Faro Technologies, Inc. | System and method of defining a path and scanning an environment |
TWI678682B (en) * | 2018-10-11 | 2019-12-01 | 宏碁股份有限公司 | Positioning method and positioning system |
US11226937B2 (en) | 2019-03-21 | 2022-01-18 | Faro Technologies, Inc. | Distributed measurement system for scanning projects |
US11463680B2 (en) | 2019-11-01 | 2022-10-04 | Faro Technologies, Inc. | Using virtual landmarks during environment scanning |
US11727582B2 (en) | 2019-12-16 | 2023-08-15 | Faro Technologies, Inc. | Correction of current scan data using pre-existing data |
US20220057518A1 (en) * | 2020-08-19 | 2022-02-24 | Faro Technologies, Inc. | Capturing environmental scans using sensor fusion |
US20220365217A1 (en) * | 2021-05-12 | 2022-11-17 | Faro Technologies, Inc. | Generating environmental map by aligning captured scans |
-
2021
- 2021-05-07 US US17/314,102 patent/US11867818B2/en active Active
- 2021-05-24 EP EP21175494.0A patent/EP3916422A1/en not_active Withdrawn
-
2023
- 2023-09-18 US US18/469,258 patent/US20240004080A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US11867818B2 (en) | 2024-01-09 |
US20210373165A1 (en) | 2021-12-02 |
EP3916422A1 (en) | 2021-12-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10914569B2 (en) | System and method of defining a path and scanning an environment | |
Li et al. | NRLI-UAV: Non-rigid registration of sequential raw laser scans and images for low-cost UAV LiDAR point cloud quality improvement | |
US11867818B2 (en) | Capturing environmental scans using landmarks based on semantic features | |
US20200191555A1 (en) | System and method of defining a path and scanning an environment | |
US20220057518A1 (en) | Capturing environmental scans using sensor fusion | |
EP3955020A1 (en) | Laser scanner with ultrawide-angle lens camera for registration | |
EP4109137A1 (en) | Capturing environmental scans using automated transporter robot | |
US10748318B2 (en) | System and method of scanning and editing two dimensional floorplans | |
US11463680B2 (en) | Using virtual landmarks during environment scanning | |
US20220137223A1 (en) | Simultaneous localization and mapping algorithms using three-dimensional registration | |
US11847741B2 (en) | System and method of scanning an environment and generating two dimensional images of the environment | |
US11461526B2 (en) | System and method of automatic re-localization and automatic alignment of existing non-digital floor plans | |
US10546419B2 (en) | System and method of on-site documentation enhancement through augmented reality | |
US20220365217A1 (en) | Generating environmental map by aligning captured scans | |
EP4332631A1 (en) | Global optimization methods for mobile coordinate scanners | |
US10447991B1 (en) | System and method of mapping elements inside walls | |
US20220406005A1 (en) | Targetless tracking of measurement device during capture of surrounding data | |
US11936843B2 (en) | Generating textured three-dimensional meshes using two-dimensional scanner and panoramic camera | |
Fang et al. | A real-time and low-cost 3D SLAM system based on a continuously rotating 2D laser scanner | |
US20230400330A1 (en) | On-site compensation of measurement devices | |
Duan et al. | A low-cost and portable indoor 3D mapping approach using biaxial line laser scanners and a one-dimension laser range finder integrated with microelectromechanical systems | |
US20220207759A1 (en) | Automatic registration of multiple measurement devices | |
US11486701B2 (en) | System and method for performing a real-time wall detection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FARO TECHNOLOGIES, INC., FLORIDA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BRENNER, MARK;FRANK, ALEKSEJ;RAMADNEH, AHMAD;AND OTHERS;SIGNING DATES FROM 20210514 TO 20210517;REEL/FRAME:064940/0454 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |