US20230372968A1 - System for autonomously applying paint to a target surface - Google Patents
System for autonomously applying paint to a target surface Download PDFInfo
- Publication number
- US20230372968A1 US20230372968A1 US18/137,376 US202318137376A US2023372968A1 US 20230372968 A1 US20230372968 A1 US 20230372968A1 US 202318137376 A US202318137376 A US 202318137376A US 2023372968 A1 US2023372968 A1 US 2023372968A1
- Authority
- US
- United States
- Prior art keywords
- paint
- segment
- target surface
- defect
- map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000003973 paint Substances 0.000 title claims abstract description 920
- 239000007921 spray Substances 0.000 claims abstract description 427
- 238000000034 method Methods 0.000 claims abstract description 159
- 230000007547 defect Effects 0.000 claims description 210
- 230000004044 response Effects 0.000 claims description 72
- 230000003287 optical effect Effects 0.000 claims description 43
- 238000007689 inspection Methods 0.000 claims description 39
- 239000002131 composite material Substances 0.000 claims description 22
- 230000008439 repair process Effects 0.000 claims description 15
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 claims description 14
- 239000012080 ambient air Substances 0.000 claims description 8
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 238000002310 reflectometry Methods 0.000 claims description 4
- 238000012790 confirmation Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims 1
- 230000004931 aggregating effect Effects 0.000 abstract description 2
- 230000000873 masking effect Effects 0.000 description 40
- 239000012636 effector Substances 0.000 description 36
- 238000012552 review Methods 0.000 description 17
- 230000000007 visual effect Effects 0.000 description 10
- 238000001514 detection method Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 230000001747 exhibiting effect Effects 0.000 description 8
- 230000002093 peripheral effect Effects 0.000 description 7
- 238000012795 verification Methods 0.000 description 7
- 239000003086 colorant Substances 0.000 description 6
- 230000004913 activation Effects 0.000 description 3
- 239000003570 air Substances 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 230000009849 deactivation Effects 0.000 description 3
- 238000003708 edge detection Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 239000011248 coating agent Substances 0.000 description 2
- 238000000576 coating method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005336 cracking Methods 0.000 description 1
- 238000001035 drying Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012876 topography Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B12/00—Arrangements for controlling delivery; Arrangements for controlling the spray area
- B05B12/08—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means
- B05B12/12—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to conditions of ambient medium or target, e.g. humidity, temperature position or movement of the target relative to the spray apparatus
- B05B12/122—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to conditions of ambient medium or target, e.g. humidity, temperature position or movement of the target relative to the spray apparatus responsive to presence or shape of target
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D—PROCESSES FOR APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D7/00—Processes, other than flocking, specially adapted for applying liquids or other fluent materials to particular surfaces or for applying particular liquids or other fluent materials
- B05D7/50—Multilayers
- B05D7/52—Two layers
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D—PROCESSES FOR APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D1/00—Processes for applying liquids or other fluent materials
- B05D1/36—Successively applying liquids or other fluent materials, e.g. without intermediate treatment
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B12/00—Arrangements for controlling delivery; Arrangements for controlling the spray area
- B05B12/08—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means
- B05B12/084—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to condition of liquid or other fluent material already sprayed on the target, e.g. coating thickness, weight or pattern
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B12/00—Arrangements for controlling delivery; Arrangements for controlling the spray area
- B05B12/08—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means
- B05B12/12—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to conditions of ambient medium or target, e.g. humidity, temperature position or movement of the target relative to the spray apparatus
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B12/00—Arrangements for controlling delivery; Arrangements for controlling the spray area
- B05B12/08—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means
- B05B12/12—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to conditions of ambient medium or target, e.g. humidity, temperature position or movement of the target relative to the spray apparatus
- B05B12/124—Arrangements for controlling delivery; Arrangements for controlling the spray area responsive to condition of liquid or other fluent material to be discharged, of ambient medium or of target ; responsive to condition of spray devices or of supply means, e.g. pipes, pumps or their drive means responsive to conditions of ambient medium or target, e.g. humidity, temperature position or movement of the target relative to the spray apparatus responsive to distance between spray apparatus and target
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B13/00—Machines or plants for applying liquids or other fluent materials to surfaces of objects or other work by spraying, not covered by groups B05B1/00 - B05B11/00
- B05B13/005—Machines or plants for applying liquids or other fluent materials to surfaces of objects or other work by spraying, not covered by groups B05B1/00 - B05B11/00 mounted on vehicles or designed to apply a liquid on a very large surface, e.g. on the road, on the surface of large containers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0008—Industrial image inspection checking presence/absence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05B—SPRAYING APPARATUS; ATOMISING APPARATUS; NOZZLES
- B05B13/00—Machines or plants for applying liquids or other fluent materials to surfaces of objects or other work by spraying, not covered by groups B05B1/00 - B05B11/00
- B05B13/02—Means for supporting work; Arrangement or mounting of spray heads; Adaptation or arrangement of means for feeding work
- B05B13/04—Means for supporting work; Arrangement or mounting of spray heads; Adaptation or arrangement of means for feeding work the spray heads being moved during spraying operation
- B05B13/0405—Means for supporting work; Arrangement or mounting of spray heads; Adaptation or arrangement of means for feeding work the spray heads being moved during spraying operation with reciprocating or oscillating spray heads
- B05B13/041—Means for supporting work; Arrangement or mounting of spray heads; Adaptation or arrangement of means for feeding work the spray heads being moved during spraying operation with reciprocating or oscillating spray heads with spray heads reciprocating along a straight line
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B05—SPRAYING OR ATOMISING IN GENERAL; APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D—PROCESSES FOR APPLYING FLUENT MATERIALS TO SURFACES, IN GENERAL
- B05D1/00—Processes for applying liquids or other fluent materials
- B05D1/02—Processes for applying liquids or other fluent materials performed by spraying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- This invention relates generally to the field of coating applications and more specifically to a new and useful system for autonomously applying paint to a target surface in the field of coating applications.
- FIGS. 1 A and 1 B is a flowchart representation of a method
- FIG. 2 is a flowchart representation of one variation of the method
- FIG. 3 is a flowchart representation of one variation of the method.
- FIG. 4 is a flowchart representation of one variation of the method.
- a method S 100 for applying paint to a target surface includes, during a scan cycle: at a first optical sensor defining a field of view intersecting a spray field of a set of spray nozzles, capturing a first image depicting a target surface of the building in Block Silo; extracting a first set of features from the first image in Block S 114 ; and, based on the first set of features, defining a first target zone on a first segment of the target surface and defining a second target zone on a second segment above the first segment of the target surface.
- the method S 100 further includes, during a first paint application cycle: navigating the set of spray nozzles across the first segment of the target surface in Block S 130 ; in response to detecting the set of spray nozzles facing the first target zone, applying paint within the first target zone onto the first segment of the target surface via the set of spray nozzles in Block S 140 ; and, at a second optical sensor adjacent the set of spray nozzles, capturing a second image depicting paint applied within the first target zone on the first segment of the target surface in Block S 150 .
- the method S 100 also includes, during a second paint application cycle: navigating the set of spray nozzles across the second segment of the target surface in Block S 130 ; in response to detecting the set of spray nozzles facing the second target zone, applying paint from the set of spray nozzles within the second target zone onto the second segment of the target surface via the set of spray nozzles in Block S 140 ; and, at the second optical sensor, capturing a third image depicting paint applied within the second target zone on the second segment of the target surface in Block S 150 .
- the method S 100 further includes combining the second image and the third image into a composite image representing application of paint onto the first segment and the second segment of the target surface in Block S 160 .
- one variation of the method S 100 includes: at an optical sensor mounted adjacent a set of spray nozzles, capturing a first image depicting a target surface of the building in Block Silo; extracting a set of features from the first image in Block S 114 ; based on the set of features, detecting a first masking tape color in a first segment of the target surface in Block S 116 and detecting a second masking tape color in a second segment of the target surface in Block S 116 ; and identifying a target zone on the first segment of the target surface based on the first masking tape color in Block S 118 ; and identifying a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second masking tape color in Block S 118 .
- This variation of the method S 100 also includes, during a first paint application cycle: navigating the set of spray nozzles across the first segment of the target surface in Block S 130 ; and, in response to detecting the set of spray nozzles facing the target zone on the first segment of the target surface, activating the set of spray nozzles, to apply paint within the first target zone on the first segment of the target surface in Block S 140 .
- This variation of the method S 100 also includes, during a second paint application cycle: navigating the set of spray nozzles across the second segment of the target surface; and, in response to detecting a subset of spray nozzles, in the set of spray nozzles, facing the keepout zone on the second segment of the target surface, deactivating the subset of spray nozzles, in the set of spray nozzles, to prevent application of paint within the keepout zone on the second segment of the target surface.
- One variation of the method S 100 includes: at an optical sensor mounted to a spray system, capturing a first sequence of images depicting a target surface of the building in Block Silo; and accessing a toolpath including a set of keypoints for execution by the spray system in Block S 120 .
- This variation of the method S 100 further includes, during a first paint application cycle: autonomously traversing the set of spray nozzles across the first segment of the target surface according to the toolpath in Block S 130 ; in response to the set of spray nozzles facing a first keypoint, in the set of keypoints, applying paint onto the first segment of the target surface in Block S 140 ; and, at the optical sensor, capturing a second sequence of images depicting paint applied onto the first segment of the target surface in Block S 150 .
- This variation of the method S 100 also includes, during a second paint application cycle: autonomously traversing the set of spray nozzles across a second segment above the first segment of the target surface according to the toolpath in Block S 130 ; in response to the set of spray nozzles facing a second keypoint, in the set of keypoints, applying paint onto the second segment of the target surface in Block S 140 ; and, at the optical sensor, capturing a third sequence of images depicting paint applied onto the second segment of the target surface in Block S 150 .
- This variation of the method S 100 further includes aggregating the first sequence of images, the second sequence of images, and the third sequence of images, into a paint map representing application of paint onto the first segment and the second segment of the target surface in Block S 160 .
- One variation of the method S 100 includes, during a first paint application cycle: autonomously navigating a spray system along a toolpath to apply paint across a first segment of the target surface via a first spray nozzle in Block S 140 ; and capturing a first sequence of images depicting paint applied on the first segment of the target surface via an optical sensor adjacent the first spray nozzle in Block S 150 .
- This variation of the method S 100 further includes, during a second paint application cycle: autonomously navigating a spray system along the toolpath to apply paint across a second segment of the target surface via the first spray nozzle in Block S 140 ; and capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle in Block S 150 .
- This variation of the method S 100 also includes: assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S 160 ; extracting a first set of features corresponding to the first segment of the target surface from the paint map in Block Silo; detecting presence of the first defect on the first segment of the target surface based on the first set of features in Block S 172 ; annotating the paint map to indicate the first defect in Block S 180 ; and serving the paint map to an operator for inspection of the first defect in Block S 190 .
- One variation of the method S 100 includes: capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent a spray nozzle in Block Silo; and extracting an initial color value of the target surface from the initial sequence of images in Block S 112 .
- This variation of the method S 100 further includes, during a first paint application cycle: navigating the spray nozzle along a toolpath to apply paint on the first segment of the target surface in Block S 130 ; and capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor adjacent the spray nozzle in Block S 150 .
- This variation of the method S 100 also includes, during a second paint application cycle, autonomously navigating the spray system along the toolpath to apply paint across a second segment of the target surface via the spray nozzle in Block S 130 ; and capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle in Block S 150 .
- This variation of the method S 100 further includes: assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S 160 ; accessing a paint application specification defining a target color value for application of paint on the target surface in Block S 112 ; extracting a first color value of paint applied onto the first segment of the target surface from the paint map in Block Silo; detecting an incomplete paint coverage defect on the first segment of the target surface based on the first color value deviating from the target color value toward the initial color value in Block S 172 ; and annotating the paint map to indicate the incomplete coverage defect in Block S 180 .
- One variation of the method S 100 includes: accessing a spray pattern for a spray nozzle mounted to a spray system in Block S 112 ; and capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent a spray nozzle in Block Silo.
- This variation of the method S 100 further includes, during a first paint application cycle: autonomously navigating the spray system along a toolpath to apply paint across the first segment of the target surface via the spray nozzle in Block S 140 ; capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor in Block S 150 ; and predicting a first paint thickness of paint applied on the first segment of the target surface based on the spray pattern of the spray nozzle in Block S 152 .
- This variation of the method S 100 also includes: assembling the initial sequence of images and the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S 160 ; annotating the paint map with a paint thickness gradient based on the first paint thickness; extracting a first surface discontinuity corresponding to the first segment of the target surface from the paint map in Block S 170 ; detecting presence of the first defect on the first segment of the target surface based on the first surface discontinuity in Block S 172 ; and annotating the paint map to indicate the first defect on the first segment of the target surface in Block S 180 .
- the system and the computer system cooperate: to autonomously apply (e.g., spray, dispense) paint onto exterior walls (e.g., target surfaces) of a building during a paint application procedure; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the building during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the building based on these data; and to compile these data into a paint map of the building representing paint application data and results of the paint application procedure.
- autonomously apply e.g., spray, dispense
- exterior walls e.g., target surfaces
- the system can execute a paint application procedure to autonomously apply paint (e.g., a volume of paint, a coat of paint)—over target surfaces of a building—according to target conditions defined in a paint application specification for the paint loaded into the system, such as: a target paint thickness; a target ambient air temperature; a target surface temperature; a target paint color; a target paint temperature; and/or a target spray overlap; etc.
- paint e.g., a volume of paint, a coat of paint
- the computer system can: compile data captured by the system during the paint application procedure into a paint map that contains visual and non-visual data representing paint application conditions and results; detect regions in the paint map that indicate paint application conditions or results (e.g., surface discontinuities) outside of the paint application specification (e.g., deviating from target conditions); identify these regions in the paint map as a paint defect (e.g., incomplete coverage, sag, run, flaking, blistering, curtain, rust); and flag corresponding segments of target surfaces of the building for paint defect inspection and/or for targeted onsite paint defect inspection by an operator and/or a building associate.
- a paint defect e.g., incomplete coverage, sag, run, flaking, blistering, curtain, rust
- the computer system can also interface with the system to serve the annotated paint map and prompts to the operator via an operator portal and/or a computing device (e.g., a mobile device, a tablet) to selectively inspect these flagged paint defects.
- a computing device e.g., a mobile device, a tablet
- the computer system and the system cooperate to enable the operator and/or the building associate to inspect paint defects remotely and rapidly on the building via the operator portal, and thereby reduce the operator's and/or building associate's inspection duration of the building.
- the computer system can: access a template defect image database containing defect images labeled with a defect type, a defect score, and a paint application procedure; detect images from the post-paint application layer of the paint map that indicate paint application conditions corresponding to the defect images; identify segments of target surfaces of the building—depicted in the images from the pos-paint application layer—within the paint map as a paint defect (e.g., incomplete coverage, sag, run, flaking, blistering, curtain); and compile these paint defects into a list ranked by defect score and serve this list to the operator and/or the building associate for paint defect inspection and/or for targeted onsite paint defect inspection.
- the computer system and system cooperate to enable the operator and/or the building associate to target the most significant paint defects (e.g., severe paint defects) from a paint application procedure according to the list of paint defects.
- the computer system can compile these data captured by the system during the paint application procedure into a live progress report; and serve this progress report to the operator and/or the building associate.
- the computer system and/or the system can: detect nearby objects that may be sensitive to overspray; generate targeted prompts to relocate these objects as the system autonomously navigates along the building; and timely serve these prompts to the operator and/or the building associate to enable the system to continuous apply paint to the building with no or minimal risk of overspray on such objects.
- the computer system and the system can cooperate: to autonomously apply paint to exterior walls of a building according to a paint application specification during a paint application procedure; to compile data captured during the paint application procedure into a paint map of the building that defines a virtual representation of the paint application procedure and results; and to detect and annotate predicted paint defects within the paint map.
- the computer system and/or the system can additionally or alternatively cooperate to apply paint to interior walls, exterior walls, and/or mutable surfaces (e.g., curved surfaces) of a building, a ship (e.g., a ship hull) aircraft, manufacturing equipment, port equipment, and/or shipping containers, etc.
- interior walls, exterior walls, and/or mutable surfaces e.g., curved surfaces
- ship e.g., a ship hull
- manufacturing equipment, port equipment, and/or shipping containers etc.
- a system for autonomously applying paint to a building includes: a chassis including a drive system; a work platform configured to raise and lower on the chassis; a spray system including a set of spray nozzles and a paint supply subsystem configured to selectively supply wet paint from a paint reservoir to the set of spray nozzles; an end effector; an optical sensor; a depth sensor; and a controller.
- the end effector is mounted to the work platform and is configured to support the set of spray nozzles on the work platform and to direct the set of spray nozzles over a range of positions.
- the optical sensor is arranged on the work platform and is configured to capture images of a target surface on the building.
- the depth sensor is configured to output distance values representative of distances between the set of spray nozzles and the target surface.
- the controller is configured to autonomously: detect target zones and keepout zones, on the target surface, proximal the set of spray nozzles based on features detected in images captured by the optical sensor; actuate the drive system to navigate the chassis along lateral segments of the target surface; actuate the work platform to navigate the set of spray nozzles vertically within lateral segments of the target surface; actuate the end effector to direct paint, exiting the set of spray nozzles, across lateral segments of the target surface; selectively actuate the spray system to dispense paint from the set of spray nozzles in response to detecting target zones proximal the set of spray nozzles; and selectively deactivate the spray system in response to detecting keepout zones proximal the set of spray nozzles.
- the controller is further configured to, for each segment of the target surface: estimate a paint thickness of paint applied to the segment of the target surface based on paint flow rate through the spray system and actuation speed of the end effector during application of paint onto the segment of the target surface; and store the paint thickness in association with a location of the segment on the target surface.
- the controller is also configured to store an image, captured by the camera and depicting the segment of the target surface, in association with the segment of the target surface.
- the system can interface with a computer system that assembles locations of segments on the target surface, corresponding paint thicknesses, and corresponding images captured by the system into a paint map that represent paint application across the target surface of the building.
- the end effector includes a linear stage configured to cycle the set of spray nozzles laterally across a target surface of the building.
- the controller can actuate the single linear stage to cyclically drive the set of spray nozzles laterally across a segment (e.g., a four-foot-wide segment) of the target surface while intermittently lowering (or raising) the work platform, thereby rastering the set of spray nozzles across the segment of the target surface and maintaining the set of spray nozzles substantially normal to the target surface.
- the controller can access a paint application specification of paint loaded into the spray system defining: a first width of the first segment of the target surface (e.g., a four-foot-wide segment); and a second width of the second segment of the target surface (e.g., a four-foot-wide segment).
- the controller can then actuate the single linear stage to: raster the set of nozzles across the first width to apply paint laterally across the first segment of the target surface; and raster the set of nozzles across the second width to apply paint laterally across the second segment of the target surface.
- the end effector includes a set of linear stages configured to cycle the set of spray nozzles laterally and vertically across the target surface.
- the controller can actuate the set of linear stages to drive the set of spray nozzles vertically and laterally across a segment (e.g., a four-foot-wide, four-foot-tall segment) of the target surface, thereby rastering the set of spray nozzles across this segment of the target surface and maintaining the set of spray nozzles substantially normal to the target surface.
- the computer system can trigger the work platform to lower (or rise) to locate the set of spray nozzles in a next segment of the target surface.
- the controller can actuate a first linear stage to drive the set of spray nozzles vertically across the height of the first segment (e.g., four-foot-tall) of the target surface.
- the controller can then actuate the second linear stage to: raster the set of spray nozzles laterally across the width of the first segment (e.g., four-foot-wide) to apply paint laterally across the first segment of the target surface.
- the controller can actuate the first linear stage to drive the set of spray nozzles vertically across the height of the second segment (e.g., four-foot-tall) of the target surface.
- the controller can then actuate the second linear stage to: raster the set of spray nozzles laterally across the width of the second segment (e.g., four-foot-wide) to apply paint laterally across the second segment of the target surface.
- the controller can actuate the set of linear stages to drive the set of spray nozzles vertically across segments (e.g., lower, rise) of the target surface and maintain the set of spray nozzles substantially normal to the target surface.
- the controller can actuate a first linear stage to cyclically drive the set of spray nozzles vertically across a first segment (e.g., a four-foot-wide segment) of the target surface while intermittently lowering (or raising) the work platform. Then, following completion of application of paint onto the first segment of the target surface, the controller can actuate a second linear stage to drive the set of spray nozzles vertically across a second segment (e.g., a four-foot-tall segment) of the target surface.
- a first linear stage to cyclically drive the set of spray nozzles vertically across a first segment (e.g., a four-foot-wide segment) of the target surface while intermittently lowering (or raising) the work platform.
- the controller can actuate a second linear stage to drive the set of spray nozzles vertically across a second segment (e.g., a four-foot-tall segment) of the target surface.
- the end effector can further include a depth stage configured to drive the set of spray nozzles—and coupled depth sensors— longitudinal substantially normal to the target surface.
- the controller can: monitor a distance between the set of spray nozzles and the target surface via the depth sensor; and implement closed-loop controls to maintain a target distance between the set of spray nozzles and the target surface by selectively actuating the depth stage of the end effector.
- the end effector can include a set of rotary stages configured to adjust pitch and/or yaw orientations of the set of spray nozzles and coupled depth sensors.
- the controller can: monitor distances between the set of spray nozzles and a set of (e.g., three or more) laterally- and vertically offset points on an adjacent region of the target surface via the depth sensor; and implement closed-loop controls to adjust the pitch and/or yaw orientations of the end effector and equalize distances to these three points on the adjacent region of the target surface, thereby maintaining the set of spray nozzles normal to the region of the target surface in its spray field.
- the end effector can include a multi-axis gantry.
- the end effector can include a robotic arm including multiple segments and joints.
- the system includes a near-field 2D color (e.g., RGB, multispectral) camera: arranged proximal the set of spray nozzles and defining a focal axis approximately parallel to a spray axis of the set of spray nozzles such that the field of view of the color camera intersects a spray field of the set of spray nozzles; and configured to capture 2D color images of the target surface before, during, and/or after the set of spray nozzles applies (i.e., sprays) paint on the adjacent target surface.
- a near-field 2D color e.g., RGB, multispectral
- the system can include: a left color camera arranged to the left of the set of spray nozzles and configured to capture images of the target surface ahead of the set of spray nozzles as the end effector moves the set of spray nozzles leftward; and a right color camera arranged to the right of the set of spray nozzles and configured to capture images of the target surface ahead of the set of spray nozzles as the end effector moves the set of spray nozzles rightward.
- the system can include a near-field 3D stereoscopic camera arranged proximal the set of spray nozzles and configured to capture 3D color images of the target surface.
- the system can similarly include a near-field multi-point depth sensor (e.g., a set of single-point depth sensors, a 3D LIDAR sensor): arranged proximal the set of spray nozzles and defining a focal axis approximately parallel to the spray axis of the set of spray nozzles; and configured to capture depth values or depth maps representing distances from the set of spray nozzles to points on the target surface in the spray field of the set of spray nozzles.
- a near-field multi-point depth sensor e.g., a set of single-point depth sensors, a 3D LIDAR sensor
- the depth sensor can be configured to detect: peripheral distances to at least three discrete points—on a nearby surface—offset about a 20-inch-diameter circle centered on the axis of the set of spray nozzles at a distance of twelve-inches from the set of spray nozzles; and a center distance to a discrete point—on the nearby surface—approximately centered on the axis of the set of spray nozzles at a distance of twelve-inches from the set of spray nozzles.
- the system can also include a housing with an actuatable aperture arranged adjacent the set of spray nozzles, such as mounted on or near the end effector.
- the near-field color, the near-field stereoscopic camera, and/or the depth sensor can be arranged in the housing.
- the controller can: trigger the aperture to close when the spray system is actuated in order to shield these sensors from overspray; and selectively trigger the aperture to open and these sensors to capture images, depth values, and/or depth maps when the spray system is inactive.
- the system can include a far-field 2D color camera, a far-field 3D stereoscopic camera, and/or a far-field depth sensor, such as: arranged on the work platform; and offset behind the set of spray nozzles (e.g., where these sensors are less subject to overspray).
- the near-field sensors can be arranged on the end effector and thus move with the set of spray nozzles.
- these near-field sensors can be fixedly mounted to the work platform, and the controller can: derive a position (i.e., offset distance and orientation) of the work platform relative to the target surface based on data captured by these sensors; track a position of the set of spray nozzles relative to the work platform (e.g., by reading position sensors or encoders arranged on actuators or stages in the end effector); and fuse these platform-to-target-surface data positions and platform-to-spray nozzle positions to track the position of the set of spray nozzles relative to the nearby target surface.
- the far-field optical and depth sensors can be similarly coupled to or decoupled from the end effector and the set of spray nozzles, and the controller can similarly track relative positions of the target surface, these far-field sensors, and the set of spray nozzles.
- the system further includes: an ambient temperature sensor; an infrared thermometer configured to read a surface temperature of the target surface; a windspeed sensor; and/or a humidity sensor; etc., such as mounted to the work platform proximal the set of spray nozzles.
- the computer system can access data representing the building and generate a set of waypoints defining navigation of the system around the building during a subsequent paint application procedure.
- the computer system accesses an aerial or satellite image—such as from an aerial mapping service—of a work site occupied by a building designated for paint application.
- the computer system then interfaces with an operator to select a set of walls of the building for paint application, such as: by enabling the operator to select walls directly from the aerial image; by recording points dropped on the image by the operator and interpolating walls between these points.
- an operator deploys an unmanned aerial vehicle to the work site prior to or with the system.
- the aerial vehicle then executes an autonomous or manually-controlled flight path to navigate around the building while capturing color and/or depth images of the building.
- the aerial vehicle and/or the computer system then compiles these color and/or depth images into a three-dimensional model of the building and surrounding topography.
- the computer system then: presents the model of the building to the operator; and interfaces with the operator to select a set of walls of the building for paint application, such as by enabling the operator to select surfaces or define bounded areas (e.g., boxes) on walls represented in the model of the building.
- bounded areas e.g., boxes
- the computer system can access a virtual (e.g., CAD) three-dimensional georeferenced model of the building, such as generated by an engineer or behalf of a site manager.
- a virtual (e.g., CAD) three-dimensional georeferenced model of the building can similarly access a virtual (e.g., CAD) three-dimensional georeferenced model of the building.
- the computer system then: presents this virtual model of the building to the operator; and interfaces with the operator to select a set of walls of the building for paint application, such as by enabling the operator to select surfaces or define bounded areas (e.g., boxes) on walls represented in the model of the building.
- the computer system then: scans the model of the building—including surrounding topographical features—for obstacles near the selected walls of the building, such as shrubs, utility boxes, signs, and benches; and defines a system path along the walls selected by the operator, such as nominally offset from these walls by a predefined target system offset distance (e.g., six feet) and that maintains a minimum distance (e.g., two feet) from detected obstacles.
- the computer system then constructs an ordered sequence of geospatially-referenced keypoints along this system path, such as offset along the system path by a target building segment width (e.g., six feet); and uploads these keypoints to the system.
- the computer system can: extract a height of a selected wall of the building at each building segment; and annotate each keypoint with a height of the corresponding building segment.
- the system can then navigate the work platform and the set of spray nozzles to the height specified in a keypoint when applying paint to the corresponding building segment on the building, as described below.
- the system can autonomously generate the system path and sequence of keypoints in real-time based on data captured by the system while applying paint to the building.
- the computer system can define a system path (or “toolpath”) along the walls selected by the operator and then construct an ordered sequence of keypoints—representing positions of the set of spray nozzles relative to the target surface—along this system path and upload these keypoints to the system.
- the controller can then execute a paint application cycle and navigate the system along these keypoints of the toolpath to apply paint over a target surface.
- the computer system can scan the model of the building and detect features (e.g., a set of doors, a set of windows, a set of segments) on a target surface of the building.
- the computer system can then derive target zones for application of paint onto the target surface and keepout zones to prevent application of paint onto these features (e.g., a set of doors, a set of windows, a set of segments) of the target surface.
- the computer system can: construct an ordered sequence of keypoints including geospatial locations, orientations, and commands for activation and/or deactivation of the set of spray nozzles upon entry into a target zone and/or a keepout zone during a paint application cycle.
- the computer system can access a first image (e.g., color image)—captured by the optical sensor—depicting a target surface of the building and extract a set of features from the first image. Then, based on these set of features the computer system can derive a set of keypoints, each keypoint in the set of keypoints including: a geospatial location to locate the set of spray nozzles facing the target surface; an orientation to locate a spray axis of the set of spray nozzles normal to the target surface; a first command to activate the set of spray nozzles for application of paint onto the target surface; and a second command to deactivate the set of spray nozzles to prevent application of paint onto the target surface.
- the computer system can then: assemble the set of keypoints into a toolpath for execution by the system and upload these keypoints to the system.
- the controller can then autonomously traverse the set of spray nozzles across segments of the target surface according to the toolpath.
- the computer system can receive dimensions of the building from an operator and leverage these dimensions to derive a boustrophedonic raster pattern. The computer system can then upload this boustrophedonic raster pattern to the system to execute during a paint application cycle.
- the computer system can: receive dimensions of the building from an operator and based on the dimensions of the building: derive a first width of the first segment of the target surface; derive a first height of the first segment of the target surface; derive a second width of a second segment of the target surface; and derive a second height of the second segment of the target surface.
- the computer system can then: define a first set of lateral raster legs (e.g., horizontal raster legs) based on the first width of the first segment and the second width of the second segment of the target surface; define a second set of vertical raster legs and connection the first set of raster legs based on the first height of the first segment and the second height of the second segment of the target surface; and compile the first set of lateral raster legs and the second set of vertical raster legs into a boustrophedonic raster pattern for execution by the system.
- a first set of lateral raster legs e.g., horizontal raster legs
- the computer system then uploads this boustrophedonic raster pattern to the system and the controller can: access the boustrophedonic raster pattern for the target surface; apply paint from the set of spray nozzles across the first segment of the target surface according to the boustrophedonic raster pattern; and apply paint from the set of spray nozzles across the second segment of the target surface according to the boustrophedonic raster pattern.
- the controller can implement these methods and techniques for each other segment of the target surface and for each other target surface to complete application of paint across the building.
- the operator can manually deploy the system onto the work site, such as by navigating the system near a first keypoint along the system path or otherwise near a first corner of the building to face a first target zone on the first segment of the building and then verifies the target zone prior to executing a paint application cycle.
- the system can implement autonomous navigation techniques to autonomously navigate the set of spray nozzles toward the first keypoint along the system path and execute a paint application cycle, as further described below.
- the computer system can generate a prompt for the operator to verify the position of the set of spray nozzles. Then, in response to the operator's verification of the position, the controller can execute a paint application cycle for the first segment. Alternatively, in response to the operator's disapproval of the position, the controller can navigate the set of spray nozzles to a second keypoint and execute a paint application cycle for the first segment.
- the work platform includes a light projection subsystem
- the controller triggers the light projection subsystem to project reference features (e.g., a lighted box perimeter)—visible to the operator—onto the current building segment to fully bound the target zone within this segment of the building.
- the controller can: scan images captured by the color camera(s) for the reference feature; and implement methods and techniques described above to define target and keepout zones—fully bounded by the reference feature—within the segment of the building segment.
- the controller can: project a lighted box perimeter—visible to an operator affiliated with the building—onto a first segment of the target surface; and generate a first prompt for the operator to verify that the lighted box perimeter corresponds to a first target zone on the first segment of the target surface. Then, in response to receiving verification of the lighted box perimeter from the operator and in response to detecting the set of spray nozzles facing the lighted box perimeter on the first segment, the controller can apply paint within the lighted box perimeter (e.g., the first target zone) onto the first segment of the target surface via the set of spray nozzles.
- the controller can then: initiate a second paint application cycle; traverse the lighted box perimeter vertically across the second segment of the target surface; project the lighted box perimeter onto the second segment of the target surface; and generate a second prompt for the operator verify that the lighted box perimeter corresponds to a second target zone on the second segment of the target surface. Then, in response to receiving verification of the lighted box perimeter from the operator and in response to detecting the set of spray nozzles facing the lighted box perimeter on the second segment, the controller can apply paint within the lighted box perimeter (e.g., the second target zone) onto the second segment of the target surface via the set of spray nozzles.
- the controller can apply paint within the lighted box perimeter (e.g., the second target zone) onto the second segment of the target surface via the set of spray nozzles.
- the controller can define paint parameters for the system prior to application of paint onto the target surface (e.g., in real time). For example, in response to receiving verification of the lighted box perimeter from the operator, the controller can: define a first standoff distance between the set of spray nozzles and the first segment of the target surface; define a first flow rate of paint through the set of spray nozzles; and define a first speed of the set of spray nozzles for application of paint onto the first segment of the target surface. The controller can then apply paint within the lighted box perimeter onto the first segment of the target surface based on the first standoff distance between the set of spray nozzles and the first segment of the target surface, the first flow rate, and the first speed.
- the operator can manually deploy the system onto the work site and responsive to the operator's verification and/or disapproval of the lighted box perimeter, the controller can define paint parameters for the system and autonomously activate the set of spray nozzles to apply paint to segments of the target surface during a paint application cycle.
- the controller can autonomously drive the work platform upwardly to locate the top edge of the target surface within the first building segment within the spray field of the set of spray nozzles and within the field of view of the optical sensor.
- the system drives the work platform upwardly to locate the set of spray nozzles at a known peak height of the building, such as a known peak height of the building within the first building segment.
- the controller can: autonomously navigate the set of spray nozzles along the toolpath to a first target position relative a first segment of the target surface based on a first keypoint; and activate the set of spray nozzles to apply paint over the first segment of the target surface according to the toolpath.
- the system can implement methods and techniques described below to detect and track the top edge of this first building segment in a color and/or depth image.
- the computer system retrieves a paint application specification for paint loaded into the spray system, a current configuration of the system, and/or paint application parameters for the building, such as including: a building segment width (e.g., six feet); a building segment height (e.g., four feet or full building height); a geometry (e.g., vertical and lateral spray angles) of the set of spray nozzles currently installed in the spray system; a target nozzle standoff distance range from the target surface (e.g., between eleven inches and thirteen inches); a minimal paint spray overlap (e.g., 50%); a target paint thickness; and/or a surface temperature application range for the paint; etc.
- a building segment width e.g., six feet
- a building segment height e.g., four feet or full building height
- a geometry e.g., vertical and lateral spray angles
- a target nozzle standoff distance range from the target surface e.g., between eleven inches and thirteen inches
- a minimal paint spray overlap e.g
- the controller can also: calculate a vertical raster pitch distance (e.g., six inches) based on the nozzle spray geometry and the minimal paint spray overlap; set or calculate a target paint flow rate; and/or calculate a raster speed (e.g., two feet per second) based on the target paint thickness, the vertical raster pitch distance, the nozzle spray geometry, and the target paint flow rate.
- the controller can retrieve a predefined target raster speed and/or paint flow rate associated with the set of spray nozzles.
- controller (or the computer system) can retrieve or calculate paint parameters for the building in any other way.
- the controller can execute a paint application cycle to navigate the set of spray nozzles across the first segment of the target surface and apply paint from the set of spray nozzles onto this first segment of the target surface.
- the controller can: trigger the end effector to locate the set of spray nozzles at a first position (e.g., a top-right position within the first segment); raster the end effector across the face of the target surface within the first segment based on a set of spray parameters; capture depth images of the target surface (e.g., via the near-field depth sensor); implement closed-loop controls to maintain a pitch and yaw orientation of the set of spray nozzles and a linear standoff distance of the set of spray nozzles from the target surface within a target standoff distance range; capture color images of the target surface (e.g., via the near-field color camera); detect target and keepout zones within the first segment based on features extracted from these color images; selectively activate the spray system when the set of spray nozzles is facing target zones; and selectively deactivate the spray system when the set of spray nozzles is facing keepout zones.
- a first position e.g., a top-right position within the first segment
- the controller then manipulates the work platform and the end effector to locate the set of spray nozzles to face a target zone within the first segment of the target surface such as three inches to the left of the left edge of the segment of the building segment with the axis of the set of spray nozzles aligned with (or slightly below, such as two inches below) the top edge of the segment of the building segment.
- the controller further executes a sequence of scan and paint application cycles during the paint application procedure: to raster the set of spray nozzles across the segment of the target surface; to detect and align the set of spray nozzles to reference features—defined by the projected lighted box perimeter—within a segment of the target surface and/or to a target zone within the segment of the target surface; and to selectively apply paint to these features.
- the controller can: retrieve a current color image representing a segment of the target surface currently facing the set of spray nozzles from the nearfield color camera; detect and extract visual features (e.g., edges, masking tape) from this color image; characterize positions and geometries of target and keepout zones within this segment of the target surface currently facing the set of spray nozzles based on these visual features that represent the target surface within and around the current spray field of the set of spray nozzles.
- visual features e.g., edges, masking tape
- the controller can: retrieve a current depth map representing this segment of the target surface currently facing the set of spray nozzles from the depth sensor; detect and extract spatial features (e.g., edges, planes) from this depth map; characterize positions and geometries of target and keepout zones within this segment of the target surface currently facing the set of spray nozzles based on these spatial features that represent the target surface within and around the current spray field of the set of spray nozzles.
- spatial features e.g., edges, planes
- the system can sample near-field a depth sensor on the work platform to read: a set of peripheral distances to points on a nearby surface, such as distances from the depth sensor to three or more vertically- and laterally-offset points on the nearby surface; and a center distance to a point on the nearby surface approximately centered on the axis of the set of spray nozzles.
- the system then implements closed-loop controls to modulate stages in the end effector to equalize the peripheral distances and to drive the center distance to the standoff distance (e.g., twelve inches).
- the controller can: receive a set of peripheral distances to points on a first segment of the target surface from the depth sensor; detect a first peripheral distance between the set of spray nozzles and the first segment of the target surface; and, in response to the first distance falling within a target standoff distance range between the set of spray nozzles and the first segment of the target surface (e.g., between eleven inches and thirteen inches), apply paint from the set of spray nozzles over the first segment of the target surface according to the toolpath.
- a target standoff distance range between the set of spray nozzles and the first segment of the target surface e.g., between eleven inches and thirteen inches
- the controller can: access a near-field color image from the left near-field color camera; and scan the color image for features indicating the target surface, target zones, and keepout zones ahead of motion of the set of spray nozzles.
- the system can: retrieve a color palette of the target surface, such as from the three-dimensional model of the building or aerial images of the building; isolate a first region of the color image containing color values within and near the color palette of the target surface; and identify and label the first region of the color images as representing the target surface.
- the system can also retrieve a color palette of a volume of paint loaded into the spray system from the, such as from the three-dimensional model of the building or aerial images of the building isolate a second region of the color image containing color values different from the color palette of the target surface; and identify and label the second region of the color images as representing a surface or volume other than the target surface.
- the controller can: access a near-field color image from the left near-field color camera; and scan the color image for features indicating a first volume of paint.
- the controller can: access the paint application specification defining a color palette of the first volume of paint; scan a first image within the paint map for a set of color values analogous to the color palette of the first volume of paint; and label the first image, containing the set of color values as the first volume of paint; and annotate the first image within the paint map with the first paint thickness—predicted during the paint application cycle—corresponding to the first volume of paint.
- the system can: implement edge detection to detect edges of the building (e.g., vertical wall edges, window and door edges, horizontal soffit edges) depicted in the color image; label the first region of the color image—bounded by these edges of the building and containing color values within and near the color palette of the target surface—as the target surface; and label remaining regions of the image as other than the target surface.
- edges of the building e.g., vertical wall edges, window and door edges, horizontal soffit edges
- the system can: read a concurrent depth image from the depth sensor; detect depth discontinuities in the concurrent depth image; identify a first region of the depth image containing depth values with a nominal depth range (e.g., less than six feet) as corresponding to the target surface of the building; project a first boundary of the first region, labeled as the target surface, onto the color image; and label a region of the color image outside of the first boundary as other than the target surface.
- a nominal depth range e.g., less than six feet
- the controller can: access a near-field color image from the right near-field color camera; and implement similar methods and techniques to detect and annotate features near and ahead of motion of the set of spray nozzles.
- the system can sample the near-field depth sensor on the work platform to read: a center distance between the set of spray nozzles and points on a first segment of the target surface.
- the system then implements closed-loop controls to modulate stages in the end effector to maintain the center distance within a standoff distance range (e.g., between eleven inches and thirteen inches).
- the controller can: receive a set of distances between the set of spray nozzles and points on a first segment of the target surface from the depth sensor; detect a center distance (e.g., twelve inches) between the set of spray nozzles and the segment of the target surface; and in response to the center distance falling within a target standoff distance range between the set of spray nozzles and the segment of the target surface (e.g., between 11 inches and 13 inches), apply paint from the set of spray nozzles over the first segment of the target surface according to the toolpath.
- a center distance e.g., twelve inches
- a target standoff distance range between the set of spray nozzles and the segment of the target surface e.g., between 11 inches and 13 inches
- the controller can: navigate the set of spray nozzles along the toolpath to a target position relative a first segment of the target surface based on a keypoint; and detect the center distance (e.g., ten inches) between the set of spray nozzles and the first segment of the target surface at the depth sensor. Then, in response to the center distance falling outside of the target standoff distance range between the set of spray nozzles and the first segment of the target surface (e.g., between 11 inches and 13 inches), the controller can: generate a new target position relative the first segment of the target surface to maintain the center distance within the target standoff distance range; and store this new target position in the keypoint to update the toolpath. The controller can then navigate the set of spray nozzles to this new target position relative the first segment of the target surface based on the keypoint and complete a next paint application cycle.
- the center distance e.g., ten inches
- the system can sample the depth sensor to access a set of depth values and interpret a set of peripheral distances between the set of spray nozzles and the first segment of the target surface.
- the system then implements closed-loop controls to modulate stages in the end effector to maintain the set of spray nozzles normal to the first segment of the target surface during a paint application cycle.
- the system can detect a first set of depth values of the set of spray nozzles at the depth sensor mounted to the spray system and adjacent the optical sensor.
- the controller can then maintain the set of spray nozzles normal to the first segment of the target surface to apply paint laterally across the first segment of the target surface based on the first set of depth values.
- the system can detect a second set of depth values of the set of spray nozzles at the depth sensor.
- the controller can then maintain the set of spray nozzles normal to the second segment of the target surface to apply paint laterally across the second segment of the target surface based on the second set of depth values.
- the system can leverage depth values from the depth sensor to maintain a center distance between the set of spray nozzles and a segment of the target surface within a standoff distance range (e.g., between eleven inches and thirteen inches) and thereby, maintain the set of spray nozzles normal to the segment of the target surface during a paint application cycle.
- a standoff distance range e.g., between eleven inches and thirteen inches
- an operator applies masking tape to regions of the target surface to define keepout zones.
- the controller can: implement edge detection and color matching techniques to identify masking tape in the current pre-paint application color image; and define a boundary between a target zone and keepout zone on the target surface along the masking tape.
- the system can: scan the current pre-paint application color image for pixels containing color values near a known masking tape color (e.g., blue, green); implement edge detection to detect a near-side edge of the masking tape; define the target zone extending up to the nearside edge of the masking tape; and define the keepout zone extending behind the nearside edge of the masking tape.
- a known masking tape color e.g., blue, green
- the system is paired with a masking tape that includes: a first color (e.g., blue) extending along a first edge configured to face a target zone; and a first color (e.g., green) extending along a second edge configured to face a keepout zone.
- the controller can: detect the masking tape in the current pre-paint application color image; detect the first and second colors of the masking tape; identify a target zone on the target surface facing the first color of the masking tape; identify a keepout zone on the target surface facing the second color of the masking tape; and label or confirm these target and keepout zones of the target surface in the color image accordingly.
- the controller can detect the first and second colors of the masking tape within a current pre-paint application color image depicting first and second segments of the target surface.
- the controller can then identify a target zone on the first segment of the target surface based on the first color (e.g., blue) and identify a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second color (e.g., green).
- the controller can: access a first masking tape color associated with target zones (e.g., blue) and indicating activation of the set of spray nozzles to apply paint to the target surface; and access a second masking tape color associated with keepout zones (e.g., green) and indicating deactivation of the set of spray nozzles.
- the controller can extract a set of features from a current pre-paint application color image depicting a first segment of the target surface and based on the set of features: detect the first masking tape color (e.g., blue) in the first segment of the target surface; and identify a first target zone on the first segment of the target surface facing the first masking tape color.
- the controller can activate the set of spray nozzles to apply paint within the first target zone on the first segment of the target surface.
- the controller can: detect the second masking tape color (e.g., green) in a second segment above the first segment of the target surface; and identify a first keepout zone on the second segment of the target surface based on the second masking tape color (e.g., green).
- the controller can: navigate the set of spray nozzles across second segment; and deactivate the set of spray nozzles to prevent paint application on the first keepout zone on the second segment of the target surface.
- the controller can access a masking tape color database containing colors of masking tape annotated with a target zone or a keepout zone for the building.
- the controller can detect first and second colors (e.g., orange, red) of masking tape within a current pre-paint application color image based on the masking tape color database.
- the controller can then identify a target zone on a first segment of the target surface corresponding to the first color (e.g., purple) and identify a keepout zone on a second segment adjacent the first segment of the target surface corresponding to the second color (e.g., red).
- the controller can: access a masking tape color database; extract a set of features from a current pre-paint application color image depicting a first segment of the target surface. Then, based on the set of features, the controller can: detect the first masking tape color (e.g., purple), from the masking tape database, in a first segment of the target surface; and identify a target zone on the first segment of the target surface based on the first masking tape color (e.g., purple).
- the first masking tape color e.g., purple
- the controller can: navigate the set of spray nozzles across the first segment of the target surface along the toolpath; and in response to detecting a subset of spray nozzles facing the target zone, activate the subset of spray nozzles to apply paint within the target zone on the first segment of the target surface. Then, based on the set of features, the controller can: detect a second masking tape color (e.g., red) from the masking tape color database in a second segment of the target surface; and identify a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second masking tape color (e.g., red).
- a second masking tape color e.g., red
- the controller can: navigate the set of spray nozzles across the second segment of the target along the toolpath; and in response to detecting a spray nozzle facing the keepout zone, deactivate the spray nozzle, to prevent paint application within the keepout zone on the second segment of the target surface.
- the controller can leverage a masking tape color database to identify target and keepout zones corresponding to colors of masking tape. Accordingly, the controller can selectively activate and/or deactivate spray nozzles facing a target and/or a keepout zone.
- the system can: define a target surface, a target zone, and/or a keepout zone in the depth image (e.g., rather than the concurrent color image).
- the system can then: activate the spray system (e.g., the paint supply subsystem) in response to detecting a target zone facing the set of spray nozzles (i.e., in response to detecting a target zone within the spray field of the set of spray nozzles); and similarly deactivate the spray system in response to detecting a keepout zone facing the set of spray nozzles (i.e., in response to detecting a keepout zone within the spray field of the set of spray nozzles).
- the spray system e.g., the paint supply subsystem
- the system characterizes the current spray field of the set of spray nozzles (i.e., the geometry of the fan of paint output by the set of spray nozzles and intersecting the target surface under current spray conditions) based on: a current standoff distance of the set of spray nozzles from the target surface; a current orientation (e.g., angular pitch and yaw offset from an adjacent segment of the target surface) of the set of spray nozzles relative to the target surface; a geometry of the set of spray nozzles; and a current paint flow rate through the set of spray nozzles (or a spray system pressure).
- a current standoff distance of the set of spray nozzles from the target surface a current orientation (e.g., angular pitch and yaw offset from an adjacent segment of the target surface) of the set of spray nozzles relative to the target surface; a geometry of the set of spray nozzles; and a current paint flow rate through the set of spray nozzles (or a spray system pressure).
- the system can then project the current spray field onto the current annotated color image based on a known offset between the color camera and the set of spray nozzles; and/or project the current spray field onto the current annotated depth map based on a known offset between the depth sensor and the set of spray nozzles.
- the controller can activate the spray system to dispense paint onto the target surface.
- the controller can deactivate the spray system to dispense paint onto the target surface.
- the set of spray nozzles is oriented on the work platform such that the spray field of the set of spray nozzles is approximately elliptical with a long elliptical axis extending vertically and a short elliptical axis extending horizontally; and paint parameters for the building specify 50% overlap between spray application paths and a set of spray nozzles with spray field height of 12 inches at the target standoff distance from the target surface.
- the controller can: activate the spray system if more than 40% of the height of the vertical span of the spray field intersects a target zone on the target surface and/or if more than 10% of the width of the vertical span of the spray field intersects a target zone on the target surface; and otherwise deactivate the spray system.
- the computer system can repeat this process to: detect target and keepout regions ahead of the set of spray nozzles in color images captured by the near-field color cameras; maintain the set of spray nozzles normal to the surface of the building based on depth values read from the near-field depth sensor; and selectively activate the spray system while controlling the end effector and/or work platform to raster the set of spray nozzles across the segment of the building segment.
- the computer system implements similar methods and techniques to: detect target and keepout regions ahead of the set of spray nozzles in color images captured by the far-field color camera (which may be less subject to overspray than the near-field color cameras); maintain the set of spray nozzles normal to the surface of the building based on depth values read from the far-field depth sensor (which may be less subject to overspray than the near-field depth sensor); and selectively activate the spray system while controlling the end effector and/or work platform to raster the set of spray nozzles across the segment of the building segment.
- the controller can repeat the foregoing process in real-time while rastering the set of spray nozzles across the first segment of the target surface of the building.
- the controller can: execute one scan cycle to generate one annotated color (and/or depth) image—labeled with identified target and keepout zones — per lateral distance interval (e.g., 0.500′′) traversed by the set of spray nozzles; calculate and project the current spray field of the set of spray nozzles onto the current annotated color image; selectively activate or deactivate the spray system based on intersection of the projected spray field with target and keepout zones identified in the current annotated color image; maintain the current active or inactive state of the spray system as the end effector traverses the set of spray nozzles laterally across this segment of the building segment; and repeat this process once the end effector again traverses the lateral distance interval in order to achieve a horizontal paint application resolution of +/ ⁇ 0.25′′.
- the controller can: execute one scan cycle to generate one annotated color (and/or depth image)—labeled with identified target and keepout zones—per lateral imaging distance interval (e.g., 1.0′′) traversed by the set of spray nozzles; implement dead reckoning techniques to interpolate the position of the current spray field of the set of spray nozzles relative to target and keepout zones identified in the current annotated color image based on motion of the end effector; selectively activate or deactivate the spray system based on this interpolated intersection of the projected spray field with target and keepout zones in order to achieve a greater horizontal paint application resolution (e.g., +/ ⁇ 0.05′′); and reset and repeat this process once the end effector again traverses the lateral distance interval (e.g., 1.0′′).
- a greater horizontal paint application resolution e.g., +/ ⁇ 0.05′′
- the controller prior to initiating paint application within the segment of the building segment, the controller: triggers the far-field color camera to capture one pre-paint color image of the entire segment of the building segment; triggers the far-field color camera to capture a set of color images of the entire segment of the building segment and stitches these color images into a composite pre-paint color image of the segment of the building segment; triggers the nearfield color camera to capture a sequence of images while the work platform rasters the color camera across the segment of the building segment; and then compiles these color images into a composite color pre-paint image of the segment of the building segment.
- the controller then: implements methods and techniques described above to detect and label target and keepout zones within the segment of the building segment; defines a raster pattern (or “toolpath”) for the segment of the building segment based on the width of the building segment, the height of the building segment, and the vertical raster pitch distance; projects the raster pattern on the annotated color image of the segment of the building segment; calculates and projects planned spray fields of the set of spray nozzles—when subsequently traversing the raster pattern—onto the annotated color image of the segment of the building segment; detects intersections between these projected spray fields, target zones, and keepout zones identified in the color image; and inserts keypoints containing spray system activation and deactivation commands along the raster pattern based on these intersections.
- toolspath for the segment of the building segment based on the width of the building segment, the height of the building segment, and the vertical raster pitch distance
- the controller then: triggers the aperture(s) in the housing described above to enclose the color camera (and depth sensor, etc.) to shield the color camera from overspray; interfaces with the work platform to navigate the set of spray nozzles along the raster pattern; implements dead reckoning techniques to track the position of the set of spray nozzles along the raster pattern; and selectively activates and deactivates the spray system based on commands contained in keypoints along the raster pattern as the estimated position of the set of spray nozzles reaches these keypoints.
- the work platform includes a light projection subsystem
- the controller triggers the light projection subsystem to project reference features (e.g., a lighted box perimeter)—visible to the operator—onto the current building segment; then triggers the optical sensor to capture an image of the building segment; detects the reference features in a particular region of the image of the building segment; and then implements the foregoing methods and techniques to detect target and keepout zones within the particular region of the image and to define a raster pattern and keypoints based on features detected within this region of the image.
- reference features e.g., a lighted box perimeter
- the controller can manipulate the work platform, the end effector, and/or the chassis to relocate the set of spray nozzles to a target position of a next segment of the building segment, such as below the preceding segment of the building segment or above and to the left of the current segment of the building segment.
- the system can: move the work platform downward to face a next segment within the current building segment; complete application of paint—from top to bottom—within the current building segment; and then autonomously navigate the chassis leftward to a geospatial keypoint that locates the chassis facing a next building segment and raise the work platform and the end effector to locate the set of spray nozzles near the top of this next building segment.
- the system can then autonomously navigate the chassis leftward to a geospatial keypoint that locates the chassis facing a next building segment and raise the work platform to locate the set of spray nozzles near the top of this next building segment.
- the controller implements geospatial triangulation and closed-loop controls to autonomously navigate the chassis from the current geospatial keypoint to the next geospatial keypoint.
- the operator places wireless transmitters on or near the building prior to the paint application procedure.
- the controller queries these wireless transmitters; implements triangulation techniques to track the position of the system relative to these wireless transmitters based on signals received from these wireless transmitters; and implements closed-loop controls to autonomously navigate the chassis from its current position facing the first building segment to a next position facing the second building segment based on the position of the system relative to these wireless transmitters.
- the system includes a set of optical detectors, and the operator places optical emitters (e.g., laser references) at opposing ends of a wall of the building prior to the paint application procedure, such as offset from the wall of the building by a target offset distance (e.g., six feet).
- optical emitters e.g., laser references
- the controller samples the optical detectors to detect light from the optical emitters; tracks the position of the system relative to these optical emitters at the ends of this wall based on light detected by the optical detects; implements closed-loop controls to autonomously navigate the chassis from its current position facing the first building segment to a next position facing the second building segment based on the position of the system relative to these optical emitters; and implements closed-loop controls to autonomously maintain the centerline (or other axis) of the chassis centered along light beams output by the optical emitters wile navigating the chassis between building segments on the building.
- the controller can: track an offset distance between the chassis and the wall of the building based on depth maps read from depth sensors mounted to the chassis, the work platform, and/or the end effector; and implement closed-loop controls to maintain this offset distance between the chassis and the wall while navigating the system between building segments based on distance values contained in depth maps read from these depth sensors.
- the controller can also: capture color images and/or depth maps of a field around the system; and implement object avoidance techniques to avoid collision with objects and surfaces detected in these color images and/or depth maps.
- the controller can: implement object recognition, perception, artificial intelligence, and/or other computer vision techniques to identify types of objects in the field around the system and distance between the system and these objects based on features detected in the color images and/or depth maps; and generate alerts or prompts to redistribute these objects within the work site, such as to reduce risk of overspray on these objects, as shown in FIG. 4 .
- the controller (or the computer system) can set a threshold overspray distance, such as: proportional to current detected or peak local windspeed; or proportional to the maximum height of the building or a remaining segment of the building to be painted.
- the controller can also: retrieve a list of sensitive objects, such as cars, trucks, motorcycles, bicycles, trailers, signs, and people; and implement object recognition techniques to detect presence of such sensitive objects around the system and distances between these objects and the system.
- the controller in response to detecting a sensitive object within the threshold overspray distance, can: pause the paint application procedure; generate an alarm and notification to relocate the sensitive object away from the system; transmit the notification to the operator and/or to a building associate affiliated with the building or work site; and then resume the paint process once the sensitive object is no longer within the threshold overspray distance of the system or once the operator or building associate manually clears the alarm.
- the controller can: set a threshold overspray distance—proportional to the first height of a first segment of the target surface (e.g., a four-foot-tall segment)—between the set of spray nozzles and the target surface; and retrieve a list of sensitive objects associated with the building, such as such as cars, trucks, motorcycles, bicycles, trailers, signs, and people.
- a threshold overspray distance proportional to the first height of a first segment of the target surface (e.g., a four-foot-tall segment)—between the set of spray nozzles and the target surface
- retrieve a list of sensitive objects associated with the building such as such as cars, trucks, motorcycles, bicycles, trailers, signs, and people.
- the controller can: detect a sensitive object (e.g., a car), from the list of sensitive objects, within the threshold overspray distance of the first segment of the target surface based on the a color image captured by the optical sensor; and, in response to detecting the sensitive object within the threshold overspray distance, generate an alarm notifying the operator to remove the sensitive object (e.g., a car). Then, in response to the operator manually clearing the alarm, the controller can navigate the set of spray nozzles along the toolpath to a next target position relative a next segment of the target surface based on a second keypoint contained in the toolpath and execute a paint application cycle for this next segment of the target surface.
- a sensitive object e.g., a car
- the controller can scan the field around the system for sensitive objects not currently within the threshold overspray distance of the system, but that will fall within the threshold overspray distance once the system moves to a next building segment.
- the controller can: predict a target time at which the system will complete application of paint on the current building segment, such as based on the width and height of the building segment, the vertical raster pitch distance, and the current raster speed of the set of spray nozzles; generate a notification to relocate the sensitive object away from the system before the target time in order to avoid disruption to the paint application procedure; transmit the notification to the operator and/or to the building associate; and initiate paint application on the next building segment only upon confirming absence of sensitive objects within the threshold overspray distance of the system or only once the operator or building associate manually confirms paint application on the next building segment.
- the controller can implement similar methods and techniques to: detect other sensitive objects that will fall within the threshold overspray distance of the system when the system navigates to future building segments; predict target times at which the system will complete application of paint on preceding building segments and navigate to these next building segments; and prompt the operator and/or the building associate to relocate the sensitive objects before their corresponding target times in order to avoid disruption to the paint application procedure.
- the controller or the computer system
- controller and the computer system cooperate to capture visual and non-visual paint application data across segments of the target surface of the building and to compile these data into a paint map.
- the controller can: capture an initial sequence of images depicting a target surface of the building; and access a spray pattern of a spray nozzle coupled to the spray system. Then, during a paint application cycle: the controller can detect a first set of ambient conditions of the target surface; navigate the spray system along a toolpath to apply a first volume of paint from the spray nozzle over the target surface; predict a first paint thickness of the first volume of paint of the target surface based on the spray pattern; and capture a first sequence of images depicting the first volume of paint of the target surface. The computer system and/or the controller can then aggregate the initial sequence of images, the first sequence of images, the first set of ambient conditions, and the first paint thickness into a paint map of the building.
- the computer system can: initialize a paint map of the building; and assemble color and/or depth images captured by the system during the paint application procedure into visual pre-paint application, in-process, and/or post-paint application layers within the paint map.
- the controller can capture pre-paint application images, such as: far-field images of segment of the building segment prior to applying paint across the segment of the building segment; near-field images of segment of the building segment while the work platform rasters the near-field camera across the segment of the building segment prior to applying paint to the surface; and/or near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment during application of paint onto the target surface.
- pre-paint application images such as: far-field images of segment of the building segment prior to applying paint across the segment of the building segment; near-field images of segment of the building segment while the work platform rasters the near-field camera across the segment of the building segment prior to applying paint to the surface; and/or near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles leftward across
- the controller can capture in-process images including: near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment during application of paint onto the target surface.
- the controller can capture post-paint application images including: far-field images of the segment of the building segment after applying paint across the segment of the building segment; near-field images of the segment of the building segment while the work platform rasters the near-field camera across the segment of the building segment after completion application of paint across the segment of the building segment; and/or near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment during application of paint onto the target surface.
- the computer system can: stitch these pre-paint application images into a composite (e.g., panoramic) pre-paint application image of the building and store this composite pre-paint application image in a pre-paint application layer of the paint map; stitch these in-process images into a composite (e.g., panoramic) in-process image of the building and store this composite in-process image in an in-process layer of the paint map; and/or stitch the post-paint application images into a composite (e.g., panoramic) post-paint application image of the building and store this composite post-paint application image in a pre-paint application layer of the paint map.
- a composite e.g., panoramic
- the computer system can further assemble these depth maps into a three-dimensional model of the building.
- the computer system can: overlay the foregoing pre-paint application, in-process, and post-paint application layers onto the building model; and enable the operator to virtually navigate between regions of the building model and selectively activate these layers to view pre-paint application, in-process, and post-paint application imagery of these building regions.
- the computer system and/or controller can: access the three-dimensional model of the building; compile the initial sequence of images into a pre-paint application composite image of the target surface; compile the first sequence of images into a post-paint application composite image of the target surface; project the pre-paint application composite image of the target surface onto the three-dimensional model of the building; project the post-paint composite image of the target surface onto the three-dimensional model of the building; and annotate the pre-paint application composite image and the post-paint composite image with the first set of ambient conditions and the first paint thickness within the three-dimensional model of the building.
- the computer system can: link individual pixels in the building model to sets of post-paint application color images that depict areas on the building corresponding to these individual pixels; store this annotated building model as the paint map; and enable the operator to virtually navigate between regions of the building model and select individual pixels within the building model to access pre-paint application, in-process, and/or post-paint application images of corresponding areas of the building.
- the controller can sample and record surface temperatures of the target surface, ambient air temperature, ambient humidity, and/or wind speed, such as at a rate of 2 Hz. Accordingly, the computer system and/or the controller can annotate the paint map with these ambient and surface data.
- the controller implements dead reckoning, structure from motion, visual odometry, and/or other techniques to track the field of view of the infrared thermometer across the target surface; reads surface temperatures across the target surface—such as just before application of paint—from the infrared thermometer; and stores surface temperature readings with concurrent positions on the target surface.
- the computer system (or the controller) can then annotate discrete positions within the paint map with their corresponding surface temperatures just before application of paint on the target surface, such as in real-time.
- the controller can: detect ambient conditions of each segment of the target surface and then aggregate these ambient conditions into the paint map.
- the controller can: detect a first surface temperature of the first segment of the target surface; and record a first humidity condition of air proximal the first segment of the target surface.
- the controller can: detect a second surface temperature of the second segment of the target surface; and record a second humidity condition of air relative the second segment of the target surface. Then, the controller can aggregate the first surface temperature, the second surface temperature, the first humidity condition, and the second humidity condition into the paint map and annotate each segment within the paint map with these ambient conditions before application of paint on the target surface (e.g., in real-time).
- the controller and the computer system can similarly cooperate to annotate discrete positions within the paint map with ambient (e.g., air temperature, humidity, wind speed) data read from these sensors when the spray field of the set of spray nozzles intersected corresponding positions on the target surface.
- ambient e.g., air temperature, humidity, wind speed
- the computer system can annotate each discrete region of the paint map that represents discrete six-inch-tall, twelve-inch wide zones on the target surface with corresponding ambient and surface values.
- controller and the computer system can cooperate to capture and store ambient and surface data in any other resolution within the paint map.
- the controller can: record paint application data; estimate applied (wet) paint thickness on the target surface based on these paint application data; and label regions of the paint map with applied (wet) paint thickness estimated for corresponding zones of the target surface.
- the controller can: read a paint flow rate from the spray system; implement dead reckoning, visual odometry, geospatial positioning, and/or other techniques to calculate a position of the set of spray nozzles relative to the building (e.g., within a coordinate system of the building); and store the current paint flow rate and the corresponding position of the set of spray nozzles relative to the building.
- the computer system can compile these paint flow rate and position data into predicted paint thicknesses across the target surface of the building.
- the computer system can: calculate a spray field geometry of the set of spray nozzles based on this paint flow rate and the set of spray nozzles geometry; calculate a paint area on the building visible to the set of spray nozzles when occupying this spray nozzle position based on this spray field geometry; and estimate an instantaneous thickness of paint applied to this paint area on the building based on this paint flow rate.
- the controller can: record a first paint flow rate of paint through the set of spray nozzles; detect a first distance between the set of spray nozzles and the first segment of the target surface; transform the first paint flow rate and the first distance into a width of a spray field of the set of spray nozzles at the first segment of the target surface; detect a first speed of the set of spray nozzles applying paint across the first segment of the target surface; convert the first speed of the set of spray nozzles and the width of the spray field into a paint application rate representing paint applied within the first target zone on the first segment of the target surface; and generate a first paint thickness of paint applied on the first segment of the target surface based on a combination of the paint application rate and the first speed of the first spray nozzle (e.g., dividing the paint application rate by the first speed of the first spray nozzle).
- the computer system can also: interpolate between these corresponding paint areas to derive interpolated paint areas; and interpolate between these instantaneous thicknesses of applied paint to derive interpolated instantaneous thicknesses of paint applied within these interpolated paint areas.
- the computer system can then: combine (e.g., sum, integrate) these estimated and interpolated instantaneous thicknesses of applied paint within overlapping paint areas across the target surface of the building to estimate total thickness of paint applied to each discrete (e.g., one-inch-square) area of the target surface of the building; and store these estimated total thicknesses of applied paint onto the paint map, such as a paint thickness layer aligned with the target surface depicted in the post-paint application layer of the paint map.
- the computer system further corrects estimated instantaneous thicknesses of applied paint based on local concurrent wind speeds detected by the system.
- high wind speeds around the set of spray nozzles may decrease a ratio of paint exiting the set of spray nozzles to paint accumulating on the target surface.
- the computer system can reduce an estimated instantaneous thickness of paint applied within a paint area on the building proportional to wind speed detected by the system when the set of spray nozzles face this paint area.
- the computer system can implement a non-uniform spray nozzle spray pattern to estimate instantaneous thicknesses of paint applied within a paint area.
- the set of spray nozzles may project paint in a non-uniform pattern with greatest paint thickness applied coaxially within the set of spray nozzles (e.g., near a center of the spray field) and least paint thickness near a periphery of the spray field.
- the computer system can: retrieve a predefined spray pattern for the set of spray nozzles; and adjust, correct, or calculate an instantaneous thickness of paint applied within a paint area based on this spray pattern.
- the computer system further: characterizes differences (e.g., reflectivity, color intensity) between color (or multispectral) values of pixels in the post-paint application layer of the paint map and corresponding pixels (i.e., pixels that represent the same physical areas on the building) in the pre-paint application layer of the paint map; flags a subset of post-paint application pixels that exhibit less than a threshold difference from corresponding pre-paint application pixels; and flags regions across the building corresponding to these pixels as likely exhibiting insufficient paint application.
- differences e.g., reflectivity, color intensity
- the computer system can: identify a first subset of pixels—contained within target regions represented in the paint map—that exhibit less than a threshold difference between pre- and post-paint application values; predict insufficient paint coverage in the corresponding areas of the building (e.g., due to insufficient change in the color and reflectively of the building after application of paint); flag corresponding regions of the paint map for further inspection for incomplete paint coverage; and serve the paint map with prompts to review these flagged regions to the operator.
- the computer system can: extract an initial set of color values from an initial image—depicting a target surface of the building—from the pre-paint application layer of the paint map; extract a first set of color values from a first image—depicting a coat of paint on the target surface—from the post-paint application layer of the paint map; and characterize a difference between the initial set of color values and the first set of color values. Then, in response to the difference falling below a threshold color difference, the computer system can: flag each segment of the target surface—depicted in the initial image and the first image—within the paint map for incomplete paint coverage; and serve the paint map with prompts to review these flagged segments of the target surface to the operator and/or to the building associate.
- the controller and/or the computer system can also execute the process in real-time to: scan a current pre-paint application image captured by the system during the paint application procedure for a first region that exhibits more than the threshold color (e.g., reflectivity, color intensity) difference from a second, known uncoated region of the building; identify the first region as containing sufficient paint thickness; define a keepout zone around the first region in order to prevent application of excess paint that may result in paint sags on this area of the building segment; define a target zone intersection the second region; and implement methods and techniques described above to selectively apply paint within an area of the building segment corresponding to the second region.
- the threshold color e.g., reflectivity, color intensity
- the system executes the foregoing methods and techniques to capture paint application, ambient, and surface data for the building while autonomously applying paint to the building; and offloads (e.g., streams) these data to the computer system, such as during or following completion of this paint application procedure.
- the computer system then executes the foregoing methods and techniques to generate the paint map following completion of the paint application procedure.
- the controller and/or the computer system can execute this process in real-time to generate and develop the paint map during this paint application procedure.
- the controller and/or the computer system can selectively pause the paint application procedure, generate an inspection prompt, and serve this inspection prompt to the operator in near real-time, such as in response to: estimated applied paint thicknesses deviating from the target paint thickness; surface temperatures on the building falling outside of a surface temperature application range specified for the paint; and/or local wind speeds exceeding a threshold wind speed.
- controller and/or the computer system can serve a prompt to the operator in near real-time in response to such conditions and only resume the paint application procedure once confirmed by the operator.
- the computer system and the controller implement methods and techniques described above to estimate real-time paint thickness of paint applied to the target surface of the building; and implement closed-loop controls to adjust paint application parameters (e.g., paint flow rate through the spray system, spray nozzle raster speed) to reduce a difference between the estimated and target paint thicknesses.
- paint application parameters e.g., paint flow rate through the spray system, spray nozzle raster speed
- the controller can: decrease traversal speed of the set of spray nozzles and/or increase flow rate through the spray system in response to a last estimated paint thickness of paint applied to the building falling below the target paint thickness; and vice versa.
- the computer system can: access the paint application specification defining a target paint thickness for segments of the target surface; extract a set of paint conditions including a paint thickness of a first segment of the target surface from the paint map and a first paint flow rate of the first volume of paint exiting from the set of spray nozzles; and calculate a difference between the paint thickness and the target paint thickness. Then, in response to the difference between the paint thickness and the target paint thickness falling below a threshold difference, the controller can: increase a target flow rate for a second volume of paint exiting the set of spray nozzles of the spray system; initiate a second paint application cycle at the target flow rate; and navigate the system along the toolpath to apply the second volume of paint from the set of spray nozzles over the target surface at the target flow rate.
- the computer system can access the paint application specification defining a set of target conditions of paint for the target surface and a spray pattern for the set of spray nozzles.
- the computer system can then: detect surface discontinuities on segments of the target surface based on features extracted from the paint map; leverage secondary data (e.g., paint application conditions, ambient and surface conditions, paint thicknesses) to identify these surface discontinuities as paint defects; annotate these segments within the paint map for defect inspection; and serve the annotated paint map to the operator—such as through an operator portal or a computing device (e.g., a mobile device, a tablet)—for defect inspection of these segments virtually through the paint map and/or in-person at the work site.
- secondary data e.g., paint application conditions, ambient and surface conditions, paint thicknesses
- the computer system can: extract a first set of features corresponding to a first segment of the target surface from the paint map; detect presence of a defect on the first segment of the target surface based on these features; and annotate the pant map to indicate presence of the defect.
- the computer system can then repeat these methods and techniques for a second segment of the target surface; and in response to a second set of features corresponding to the second segment of the target surface, falling within a threshold deviation range of the first set of features, detect absence of the defect on the second segment of the target surface.
- the computer system can then annotate the paint map to indicate absence of the defect on the second segment of the target surface.
- the computer system can serve the annotated paint map and/or paint application layers to the operator via a computing device (e.g., a tablet, a mobile device) thereby enabling the operator to immediately review color image data of all flagged segments for defect inspection on the building.
- a computing device e.g., a tablet, a mobile device
- the computer system flags regions in the paint map annotated with estimated (wet) paint thicknesses significantly less than the target paint thickness for targeted incomplete coverage inspections; and flags regions in the paint map annotated with estimated (wet) paint thicknesses significantly greater than the target paint thickness for paint as paint sag defect inspections.
- the computer system divides the paint map into segments representing one-foot-square areas on the target surface of the building. For a first segment of the paint map, the computer system: calculates a maximum absolute deviation of estimated (wet) paint thicknesses within the first segment from the target paint thickness for the building; flags (e.g., annotates, highlights) the first segment in the paint map for incomplete coverage inspection if this maximum absolute deviation falls below the target paint thickness for the building by more than a threshold coverage deviation (e.g., 25%); and flags the first segment in the paint map for sag inspection if this maximum absolute deviation exceeds the target paint thickness for the building by more than the threshold sag deviation (e.g., 15%).
- the computer system repeats this process for each other segment of the paint map and then presents this annotated paint map to an operator with a prompt to selectively inspect each area of the building flagged in the paint map.
- the computer system can: flag segments in the post-paint application layer of the paint map for incomplete paint coverage inspection; and serve this post-paint application layer to the operator via a mobile device (e.g., a tablet), thereby enabling the operator to immediately review color image data of all flagged incomplete paint coverage defects on the building.
- the computer system can: access the paint application specification defining a target paint thickness for paint on the target surface; and extract a first paint thickness exhibited by a first segment of the target surface from the paint map.
- the computer system can: identify the first segment within the paint map as an incomplete coverage defect; flag the first segment within the paint map for incomplete coverage inspection; generate a prompt for incomplete coverage inspection of first segment of the target surface; and serve the paint map and the prompt to an operator.
- a threshold coverage deviation e.g. 25%
- the computer system can: flag segments in the post-paint application layer of the paint map for paint sag inspection; and serve this post-paint application layer to the operator via the mobile device (e.g., a tablet, a computing device) thereby enabling the operator to immediately review color image data of all flagged paint sag defects on the building.
- the computer system can similarly access the target paint thickness and extract a first paint thickness exhibited by the first segment of the target surface.
- the computer system can: identify the first segment within the paint map as a paint sag defect; flag the first segment within the paint map for paint sag inspection; generate a prompt for paint sag inspection of the first segment of the target surface of the building; and serve the paint map and the prompt to an operator (e.g., via the operator portal).
- a threshold coverage deviation e.g. 25%
- the computer system can: retrieve a post-paint application color image—captured by the system—corresponding to each flagged segment in the paint map; label these post-paint application color images with their locations on the building and/or whether these images depict predicted incomplete coverage or paint sags; and serve these post-paint application color images to the operator for remote inspection of the building.
- the computer system divides the paint map into segments representing one-foot-square areas on the target surface of the building; calculates a maximum absolute deviation of estimated (wet) paint thicknesses within each segment from the target paint thickness for the building; generates an inspection heatmap of the target surface of the building (e.g., a heatmap layer of the paint map) based on the maximum absolute deviation calculated for each segment of the paint map; and then presents this heatmap to the operator with a prompt to prioritize inspection regions of the building based on intensity in corresponding segments of the heatmap.
- an inspection heatmap of the target surface of the building e.g., a heatmap layer of the paint map
- the computer system can: retrieve a post-paint application color image—captured by the system—corresponding to each segment in the paint map; sort these post-paint application color images by absolute deviation from the target paint thickness for the building; and serve these sorted post-paint application color images to the operator for remote manual inspection of the building.
- the computer system can: access a spray pattern of the first spray nozzle. Then, during a first paint application cycle, the computer system can predict a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle. The computer system can then: access a paint application specification defining a target paint thickness for application of paint on the target surface; detect a first surface discontinuity on the first segment of the target surface within the paint map; and, in response to detecting the first surface discontinuity and in response to the first paint thickness exceeding the target paint thickness, detect presence of a sag defect on the first segment of the target surface. The computer system can then: annotate the paint map to indicate the sag defect; generate a prompt to repair the sag defect on the first segment of the target surface; and serve the paint map with the prompt to the operator for inspection of the sag defect.
- the computer system can: isolate a first edge extending toward a lower bound of the first segment of the target surface within the paint map; isolate a second edge coupled to the first edge and extending toward the lower bound of the first segment of the target surface; detect a first linear surface discontinuity on the first segment of the target surface within the paint map based on the first edge and the second edge; and, in response to detecting the first linear surface discontinuity and in response to the first paint thickness exceeding the target paint thickness, detect presence of a sag defect on the first segment of the target surface.
- the computer system can enable the operator to: review and inspect the building remotely; and/or review flagged regions for incomplete paint coverage defects and/or paint sag defects across the building remotely prior to in-person manual review of the most problematic or indiscernible regions.
- the computer system can: detect ambient conditions during a paint application cycle; predict a paint quality of paint applied onto the target surface based on these ambient conditions and a paint thickness of the paint; flag regions of the paint map corresponding to segments of the building exhibiting a paint quality differing from a target paint quality for the target surface; and generate a prompt for the operator to review these segments of the paint map and/or inspect corresponding segments of the building for incomplete coverage defect inspection.
- the computer system can: access a spray pattern of the first spray nozzle; detect a first ambient air temperature proximal the first segment of the target surface; detect a first surface temperature of the first segment of the target surface; predict a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle; and predict a first paint quality of paint applied onto the first segment of the target surface based on the first paint thickness, the first ambient air temperature, and the first surface temperature.
- the computer system can then: access a paint application specification defining a target paint quality for the target surface; detect a first surface discontinuity on the first segment of the target surface within the paint map; in response to detecting the first surface discontinuity and in response to the first paint quality falling below the target paint quality, detect presence of an incomplete coverage defect on the first segment of the target surface; and annotate the paint map to indicate the incomplete coverage defect.
- the computer system can leverage ambient conditions and paint thickness to predict a quality of paint applied onto the target surface and flag regions of the paint map exhibiting a paint quality differing from a target paint quality.
- the computer system can also enable an operator to review these flagged regions for incomplete paint coverage defects.
- the computer system similarly: flags or highlights regions of the paint map corresponding to out-of-range surface and ambient conditions; and prompts the operator to selectively review corresponding areas on the building, such as virtually through the paint map or in-person at the work site.
- the computer system can: flag regions of the paint map corresponding to areas of the building facing the set of spray nozzles when local wind speed—detected by the system—exceeded a threshold windspeed; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete coverage.
- the computer system can: flag regions of the paint map corresponding to areas of the building exhibiting surface temperatures exceeding a manufacturer temperature specification for the paint when facing the set of spray nozzles; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete paint adhesion or blistering.
- the computer system can: detect a first surface temperature of paint applied on the first segment of the target surface; and access a paint application specification defining a target surface temperature for application of paint; and detect a first circular surface discontinuity on the first segment of the target surface within the paint map. Then, in response to detecting the first circular surface discontinuity and in response to the first surface temperature exceeding the target surface temperature, the computer system can detect presence of a paint blister defect on the first segment of the target surface.
- the computer system can then: annotate the paint map to indicate the paint blister defect; generate a prompt to repair the paint blister defect on the first segment of the target surface; and serve the paint map and the prompt to the operator for repair of the paint blister defect prior to a scheduled inspection period for the first paint application cycle. Then, in response to receiving confirmation of removal of the paint blister defect from on the first segment of the target surface from the operator, the controller can initiate a second paint application cycle for the first segment of the target surface.
- the computer system can: flag regions of the paint map corresponding to areas of the building exhibiting surface temperatures below a manufacturer temperature specification for the paint when facing the set of spray nozzles; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete drying and cracking.
- the computer system can: flag regions of the paint map corresponding to segments of the building exhibiting a color of paint deviating from a target color of paint; and generate a prompt for the operator to review these segments of the paint map and/or inspect corresponding segments of the building for paint rust inspection.
- the computer system can: isolate an image (e.g., a post-paint application image) depicting a first segment of the target surface within the paint map; and based on features extracted from the image, detect a color of paint (e.g., beige) on the segment; and access the paint application specification defining a target color of paint (e.g., white) for the first segment of the target surface. Then, in response to the color of paint (e.g., beige) deviating from the target color of paint (e.g., white) on the first segment of the target surface, the computer system can: identify the first segment within the paint map as a rust paint defect; and flag the paint map with this rust defect.
- an image e.g., a post-paint application image
- the computer system can: access a paint application specification defining a target color value for application of paint on the target surface; detect a first color value of paint applied to the first segment of the target surface within the paint map; retrieve a set of paint defect images from a template defect image database, each paint defect image in the set of defect images labeled with a defect type, a defect score, and a paint application procedure; and extract a second set of features from a first paint defect image, in the set of paint defect images, labeled with a rust defect type, a rust repair procedure, and a first defect score.
- the computer system can detect presence of a rust defect on the first segment of the target surface and annotate the paint map to indicate the rust defect.
- the computer system can access a template defect image database containing defect images (e.g., images depicting paint defects) labeled with a defect type, a defect score, and an action (e.g., a paint application procedure).
- defect images e.g., images depicting paint defects
- an action e.g., a paint application procedure
- the computer system can then match features of images from the post-paint application layer of the paint map to analogous features of defect images from the template defect image database to identify paint defects on segments of the target surface.
- the computer system can compile a list of paint defects on the target surface, ranked by defect score, and serve this list to the operator and thereby, enable the operator to selectively review paint defects in a sequential order according to the list.
- the computer system can retrieve a set of paint defect images from the template defect image database, each paint defect image in the set of defect images labeled with a defect type (e.g., incomplete coverage defect, sag paint defect, rust paint defect, flaking paint defect, paint blister defect, run paint defect, curtain paint defect) a defect score, and a paint application procedure (e.g., series of actions to remove the paint defect).
- a defect type e.g., incomplete coverage defect, sag paint defect, rust paint defect, flaking paint defect, paint blister defect, run paint defect, curtain paint defect
- a paint application procedure e.g., series of actions to remove the paint defect.
- the computer system can: annotate the sequence of images with the corresponding defect type, defect score, and paint application procedure within the paint map; compile a list of paint defects of the target surface, ranked by defect score; and serve the paint map and the list of paint defects of the target surface to an operator (e.g., via the operator portal).
- the computer system can retrieve a first paint defect image—labeled with a paint sag defect type and a paint sag application procedure—from the template defect image database. Then, in response to a first image from the post-paint application layer of the paint map corresponding to the first paint defect image, the computer system can: annotate the first image with the paint sag defect type and the paint sag application procedure within the paint map; and initiate a second paint application cycle to remove the paint sag defect from the target surface based on the paint sag application procedure.
- the computer system and system cooperate to enable the operator and/or the building associate to target the most significant paint defects (e.g., severe paint defects) from a paint application procedure according to the list of paint defects.
- most significant paint defects e.g., severe paint defects
- the computer system implements methods and techniques described above to annotate the paint map (e.g., the building model) with regions of the building completed by the system based on color images, depth maps, and/or spray nozzle position data captured by the system; highlights regions of the building completed by the system; and presents the paint map—with highlighted regions—to the operator and/or the building associate.
- the paint map e.g., the building model
- the computer system can: calculate a proportion (or ratio) of paint application on the building completed by the system; and present this proportion to the operator and/or the building associate.
- the computer system can: predict a time to completion at which the system will complete application of paint on the whole building, such as based on the remaining unpainted area of the building and the current raster speed of the set of spray nozzles; and present this time to completion to the operator and/or the building associate.
- the computer system can present the paint map—such as the pre- and post-paint application layers of the paint map—to the building associate, thereby enabling the building associate to immediately and remotely review and inspect the condition of the building prior to application of paint and after application of paint.
- the paint map such as the pre- and post-paint application layers of the paint map
- the computer system and system cooperate to autonomously apply (e.g., spray, dispense) paint onto exterior and/or interior walls (e.g., target surfaces) of a building during a paint application procedure; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the building during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the building based on these data; compile these data into a paint map of the building representing paint application data and results of the paint application procedure; and flag regions of the paint map exhibiting paint conditions deviating from the paint application specification.
- autonomously apply e.g., spray, dispense
- exterior and/or interior walls e.g., target surfaces
- the system and computer system can cooperate to autonomously apply (e.g., spray, dispense) paint onto exterior and/or interior walls of a ship aircraft (e.g., target mobile surface, target curved surfaces) during a paint application procedure; to maintain peripheral and center distances between the set of spray nozzles and walls of the ship aircraft within the target standoff distance range during a paint application cycle; to maintain the set of spray nozzles substantially normal to each segment of the wall of the ship aircraft; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the ship aircraft during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the ship aircraft based on these data; compile these data into a paint map of the ship aircraft representing paint application data and results of the paint
- the systems and methods described herein can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions.
- the instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof.
- Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions.
- the instructions can be executed by computer-executable components integrated by computer-executable components integrated with apparatuses and networks of the type described above.
- the computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device.
- the computer-executable component can be a processor but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Wood Science & Technology (AREA)
- Spray Control Apparatus (AREA)
- Coating Apparatus (AREA)
- Application Of Or Painting With Fluid Materials (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
One variation of a method includes: capturing a first image depicting a target surface; accessing a toolpath including a set of keypoints for execution by the spray system; navigating the set of spray nozzles across a first segment of the target surface; applying paint onto the first segment of the target surface; capturing a second image depicting paint applied onto the first segment of the target surface; navigating the set of spray nozzles across a second segment of the target surface; applying paint onto the second segment of the target surface; capturing a third image depicting paint applied onto the second segment of the target surface; and aggregating the first image, the second image, and the third image into a paint map representing application of paint onto the first segment and the second segment of the target surface.
Description
- This application claims the benefit of U.S. Provisional Application No. 63/333,093, filed on 20 Apr. 2022, and 63/333,100, filed on 20 Apr. 2022, each of which is incorporated in its entirety by this reference.
- This invention relates generally to the field of coating applications and more specifically to a new and useful system for autonomously applying paint to a target surface in the field of coating applications.
-
FIGS. 1A and 1B is a flowchart representation of a method; -
FIG. 2 is a flowchart representation of one variation of the method; -
FIG. 3 is a flowchart representation of one variation of the method; and -
FIG. 4 is a flowchart representation of one variation of the method. - The following description of embodiments of the invention is not intended to limit the invention to these embodiments but rather to enable a person skilled in the art to make and use this invention. Variations, configurations, implementations, example implementations, and examples described herein are optional and are not exclusive to the variations, configurations, implementations, example implementations, and examples they describe. The invention described herein can include any and all permutations of these variations, configurations, implementations, example implementations, and examples.
- As shown in
FIGS. 1-4 , a method S100 for applying paint to a target surface includes, during a scan cycle: at a first optical sensor defining a field of view intersecting a spray field of a set of spray nozzles, capturing a first image depicting a target surface of the building in Block Silo; extracting a first set of features from the first image in Block S114; and, based on the first set of features, defining a first target zone on a first segment of the target surface and defining a second target zone on a second segment above the first segment of the target surface. - The method S100 further includes, during a first paint application cycle: navigating the set of spray nozzles across the first segment of the target surface in Block S130; in response to detecting the set of spray nozzles facing the first target zone, applying paint within the first target zone onto the first segment of the target surface via the set of spray nozzles in Block S140; and, at a second optical sensor adjacent the set of spray nozzles, capturing a second image depicting paint applied within the first target zone on the first segment of the target surface in Block S150.
- The method S100 also includes, during a second paint application cycle: navigating the set of spray nozzles across the second segment of the target surface in Block S130; in response to detecting the set of spray nozzles facing the second target zone, applying paint from the set of spray nozzles within the second target zone onto the second segment of the target surface via the set of spray nozzles in Block S140; and, at the second optical sensor, capturing a third image depicting paint applied within the second target zone on the second segment of the target surface in Block S150. The method S100 further includes combining the second image and the third image into a composite image representing application of paint onto the first segment and the second segment of the target surface in Block S160.
- As shown in
FIG. 1A , one variation of the method S100 includes: at an optical sensor mounted adjacent a set of spray nozzles, capturing a first image depicting a target surface of the building in Block Silo; extracting a set of features from the first image in Block S114; based on the set of features, detecting a first masking tape color in a first segment of the target surface in Block S116 and detecting a second masking tape color in a second segment of the target surface in Block S116; and identifying a target zone on the first segment of the target surface based on the first masking tape color in Block S118; and identifying a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second masking tape color in Block S118. - This variation of the method S100 also includes, during a first paint application cycle: navigating the set of spray nozzles across the first segment of the target surface in Block S130; and, in response to detecting the set of spray nozzles facing the target zone on the first segment of the target surface, activating the set of spray nozzles, to apply paint within the first target zone on the first segment of the target surface in Block S140. This variation of the method S100 also includes, during a second paint application cycle: navigating the set of spray nozzles across the second segment of the target surface; and, in response to detecting a subset of spray nozzles, in the set of spray nozzles, facing the keepout zone on the second segment of the target surface, deactivating the subset of spray nozzles, in the set of spray nozzles, to prevent application of paint within the keepout zone on the second segment of the target surface.
- One variation of the method S100 includes: at an optical sensor mounted to a spray system, capturing a first sequence of images depicting a target surface of the building in Block Silo; and accessing a toolpath including a set of keypoints for execution by the spray system in Block S120.
- This variation of the method S100 further includes, during a first paint application cycle: autonomously traversing the set of spray nozzles across the first segment of the target surface according to the toolpath in Block S130; in response to the set of spray nozzles facing a first keypoint, in the set of keypoints, applying paint onto the first segment of the target surface in Block S140; and, at the optical sensor, capturing a second sequence of images depicting paint applied onto the first segment of the target surface in Block S150.
- This variation of the method S100 also includes, during a second paint application cycle: autonomously traversing the set of spray nozzles across a second segment above the first segment of the target surface according to the toolpath in Block S130; in response to the set of spray nozzles facing a second keypoint, in the set of keypoints, applying paint onto the second segment of the target surface in Block S140; and, at the optical sensor, capturing a third sequence of images depicting paint applied onto the second segment of the target surface in Block S150. This variation of the method S100 further includes aggregating the first sequence of images, the second sequence of images, and the third sequence of images, into a paint map representing application of paint onto the first segment and the second segment of the target surface in Block S160.
- One variation of the method S100 includes, during a first paint application cycle: autonomously navigating a spray system along a toolpath to apply paint across a first segment of the target surface via a first spray nozzle in Block S140; and capturing a first sequence of images depicting paint applied on the first segment of the target surface via an optical sensor adjacent the first spray nozzle in Block S150.
- This variation of the method S100 further includes, during a second paint application cycle: autonomously navigating a spray system along the toolpath to apply paint across a second segment of the target surface via the first spray nozzle in Block S140; and capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle in Block S150.
- This variation of the method S100 also includes: assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S160; extracting a first set of features corresponding to the first segment of the target surface from the paint map in Block Silo; detecting presence of the first defect on the first segment of the target surface based on the first set of features in Block S172; annotating the paint map to indicate the first defect in Block S180; and serving the paint map to an operator for inspection of the first defect in Block S190.
- One variation of the method S100 includes: capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent a spray nozzle in Block Silo; and extracting an initial color value of the target surface from the initial sequence of images in Block S112. This variation of the method S100 further includes, during a first paint application cycle: navigating the spray nozzle along a toolpath to apply paint on the first segment of the target surface in Block S130; and capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor adjacent the spray nozzle in Block S150.
- This variation of the method S100 also includes, during a second paint application cycle, autonomously navigating the spray system along the toolpath to apply paint across a second segment of the target surface via the spray nozzle in Block S130; and capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle in Block S150. This variation of the method S100 further includes: assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S160; accessing a paint application specification defining a target color value for application of paint on the target surface in Block S112; extracting a first color value of paint applied onto the first segment of the target surface from the paint map in Block Silo; detecting an incomplete paint coverage defect on the first segment of the target surface based on the first color value deviating from the target color value toward the initial color value in Block S172; and annotating the paint map to indicate the incomplete coverage defect in Block S180.
- One variation of the method S100 includes: accessing a spray pattern for a spray nozzle mounted to a spray system in Block S112; and capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent a spray nozzle in Block Silo.
- This variation of the method S100 further includes, during a first paint application cycle: autonomously navigating the spray system along a toolpath to apply paint across the first segment of the target surface via the spray nozzle in Block S140; capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor in Block S150; and predicting a first paint thickness of paint applied on the first segment of the target surface based on the spray pattern of the spray nozzle in Block S152.
- This variation of the method S100 also includes: assembling the initial sequence of images and the first sequence of images into a paint map representing application of paint on the first segment of the target surface in Block S160; annotating the paint map with a paint thickness gradient based on the first paint thickness; extracting a first surface discontinuity corresponding to the first segment of the target surface from the paint map in Block S170; detecting presence of the first defect on the first segment of the target surface based on the first surface discontinuity in Block S172; and annotating the paint map to indicate the first defect on the first segment of the target surface in Block S180.
- Generally, the system and the computer system cooperate: to autonomously apply (e.g., spray, dispense) paint onto exterior walls (e.g., target surfaces) of a building during a paint application procedure; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the building during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the building based on these data; and to compile these data into a paint map of the building representing paint application data and results of the paint application procedure.
- In particular, the system can execute a paint application procedure to autonomously apply paint (e.g., a volume of paint, a coat of paint)—over target surfaces of a building—according to target conditions defined in a paint application specification for the paint loaded into the system, such as: a target paint thickness; a target ambient air temperature; a target surface temperature; a target paint color; a target paint temperature; and/or a target spray overlap; etc.
- Furthermore, the computer system can: compile data captured by the system during the paint application procedure into a paint map that contains visual and non-visual data representing paint application conditions and results; detect regions in the paint map that indicate paint application conditions or results (e.g., surface discontinuities) outside of the paint application specification (e.g., deviating from target conditions); identify these regions in the paint map as a paint defect (e.g., incomplete coverage, sag, run, flaking, blistering, curtain, rust); and flag corresponding segments of target surfaces of the building for paint defect inspection and/or for targeted onsite paint defect inspection by an operator and/or a building associate. The computer system can also interface with the system to serve the annotated paint map and prompts to the operator via an operator portal and/or a computing device (e.g., a mobile device, a tablet) to selectively inspect these flagged paint defects. Thus, the computer system and the system cooperate to enable the operator and/or the building associate to inspect paint defects remotely and rapidly on the building via the operator portal, and thereby reduce the operator's and/or building associate's inspection duration of the building.
- Additionally, the computer system can: access a template defect image database containing defect images labeled with a defect type, a defect score, and a paint application procedure; detect images from the post-paint application layer of the paint map that indicate paint application conditions corresponding to the defect images; identify segments of target surfaces of the building—depicted in the images from the pos-paint application layer—within the paint map as a paint defect (e.g., incomplete coverage, sag, run, flaking, blistering, curtain); and compile these paint defects into a list ranked by defect score and serve this list to the operator and/or the building associate for paint defect inspection and/or for targeted onsite paint defect inspection. Thus, the computer system and system cooperate to enable the operator and/or the building associate to target the most significant paint defects (e.g., severe paint defects) from a paint application procedure according to the list of paint defects.
- Furthermore, the computer system can compile these data captured by the system during the paint application procedure into a live progress report; and serve this progress report to the operator and/or the building associate. Similarly, the computer system and/or the system can: detect nearby objects that may be sensitive to overspray; generate targeted prompts to relocate these objects as the system autonomously navigates along the building; and timely serve these prompts to the operator and/or the building associate to enable the system to continuous apply paint to the building with no or minimal risk of overspray on such objects.
- Therefore, the computer system and the system can cooperate: to autonomously apply paint to exterior walls of a building according to a paint application specification during a paint application procedure; to compile data captured during the paint application procedure into a paint map of the building that defines a virtual representation of the paint application procedure and results; and to detect and annotate predicted paint defects within the paint map.
- However, the computer system and/or the system can additionally or alternatively cooperate to apply paint to interior walls, exterior walls, and/or mutable surfaces (e.g., curved surfaces) of a building, a ship (e.g., a ship hull) aircraft, manufacturing equipment, port equipment, and/or shipping containers, etc.
- A system for autonomously applying paint to a building includes: a chassis including a drive system; a work platform configured to raise and lower on the chassis; a spray system including a set of spray nozzles and a paint supply subsystem configured to selectively supply wet paint from a paint reservoir to the set of spray nozzles; an end effector; an optical sensor; a depth sensor; and a controller. The end effector is mounted to the work platform and is configured to support the set of spray nozzles on the work platform and to direct the set of spray nozzles over a range of positions. The optical sensor is arranged on the work platform and is configured to capture images of a target surface on the building. The depth sensor is configured to output distance values representative of distances between the set of spray nozzles and the target surface.
- The controller is configured to autonomously: detect target zones and keepout zones, on the target surface, proximal the set of spray nozzles based on features detected in images captured by the optical sensor; actuate the drive system to navigate the chassis along lateral segments of the target surface; actuate the work platform to navigate the set of spray nozzles vertically within lateral segments of the target surface; actuate the end effector to direct paint, exiting the set of spray nozzles, across lateral segments of the target surface; selectively actuate the spray system to dispense paint from the set of spray nozzles in response to detecting target zones proximal the set of spray nozzles; and selectively deactivate the spray system in response to detecting keepout zones proximal the set of spray nozzles. The controller is further configured to, for each segment of the target surface: estimate a paint thickness of paint applied to the segment of the target surface based on paint flow rate through the spray system and actuation speed of the end effector during application of paint onto the segment of the target surface; and store the paint thickness in association with a location of the segment on the target surface. The controller is also configured to store an image, captured by the camera and depicting the segment of the target surface, in association with the segment of the target surface.
- The system can interface with a computer system that assembles locations of segments on the target surface, corresponding paint thicknesses, and corresponding images captured by the system into a paint map that represent paint application across the target surface of the building.
- In one implementation, the end effector includes a linear stage configured to cycle the set of spray nozzles laterally across a target surface of the building. In this implementation, the controller can actuate the single linear stage to cyclically drive the set of spray nozzles laterally across a segment (e.g., a four-foot-wide segment) of the target surface while intermittently lowering (or raising) the work platform, thereby rastering the set of spray nozzles across the segment of the target surface and maintaining the set of spray nozzles substantially normal to the target surface.
- Additionally or alternatively, the controller can access a paint application specification of paint loaded into the spray system defining: a first width of the first segment of the target surface (e.g., a four-foot-wide segment); and a second width of the second segment of the target surface (e.g., a four-foot-wide segment). The controller can then actuate the single linear stage to: raster the set of nozzles across the first width to apply paint laterally across the first segment of the target surface; and raster the set of nozzles across the second width to apply paint laterally across the second segment of the target surface.
- In another implementation, the end effector includes a set of linear stages configured to cycle the set of spray nozzles laterally and vertically across the target surface. In this implementation, the controller can actuate the set of linear stages to drive the set of spray nozzles vertically and laterally across a segment (e.g., a four-foot-wide, four-foot-tall segment) of the target surface, thereby rastering the set of spray nozzles across this segment of the target surface and maintaining the set of spray nozzles substantially normal to the target surface. In this implementation, following completion of application of paint onto this segment of the target surface, the computer system can trigger the work platform to lower (or rise) to locate the set of spray nozzles in a next segment of the target surface.
- In one example, the controller can actuate a first linear stage to drive the set of spray nozzles vertically across the height of the first segment (e.g., four-foot-tall) of the target surface. The controller can then actuate the second linear stage to: raster the set of spray nozzles laterally across the width of the first segment (e.g., four-foot-wide) to apply paint laterally across the first segment of the target surface. Then, following completion of application of paint onto the first segment of the target surface, the controller can actuate the first linear stage to drive the set of spray nozzles vertically across the height of the second segment (e.g., four-foot-tall) of the target surface. The controller can then actuate the second linear stage to: raster the set of spray nozzles laterally across the width of the second segment (e.g., four-foot-wide) to apply paint laterally across the second segment of the target surface. Thus, the controller can actuate the set of linear stages to drive the set of spray nozzles vertically across segments (e.g., lower, rise) of the target surface and maintain the set of spray nozzles substantially normal to the target surface.
- In another example, the controller can actuate a first linear stage to cyclically drive the set of spray nozzles vertically across a first segment (e.g., a four-foot-wide segment) of the target surface while intermittently lowering (or raising) the work platform. Then, following completion of application of paint onto the first segment of the target surface, the controller can actuate a second linear stage to drive the set of spray nozzles vertically across a second segment (e.g., a four-foot-tall segment) of the target surface.
- In the foregoing implementations, the end effector can further include a depth stage configured to drive the set of spray nozzles—and coupled depth sensors— longitudinal substantially normal to the target surface. Accordingly, the controller can: monitor a distance between the set of spray nozzles and the target surface via the depth sensor; and implement closed-loop controls to maintain a target distance between the set of spray nozzles and the target surface by selectively actuating the depth stage of the end effector.
- Additionally or alternatively, the end effector can include a set of rotary stages configured to adjust pitch and/or yaw orientations of the set of spray nozzles and coupled depth sensors. In this implementation, the controller can: monitor distances between the set of spray nozzles and a set of (e.g., three or more) laterally- and vertically offset points on an adjacent region of the target surface via the depth sensor; and implement closed-loop controls to adjust the pitch and/or yaw orientations of the end effector and equalize distances to these three points on the adjacent region of the target surface, thereby maintaining the set of spray nozzles normal to the region of the target surface in its spray field.
- For example, the end effector can include a multi-axis gantry. Alternatively, the end effector can include a robotic arm including multiple segments and joints.
- In one implementation, the system includes a near-field 2D color (e.g., RGB, multispectral) camera: arranged proximal the set of spray nozzles and defining a focal axis approximately parallel to a spray axis of the set of spray nozzles such that the field of view of the color camera intersects a spray field of the set of spray nozzles; and configured to capture 2D color images of the target surface before, during, and/or after the set of spray nozzles applies (i.e., sprays) paint on the adjacent target surface. For example, the system can include: a left color camera arranged to the left of the set of spray nozzles and configured to capture images of the target surface ahead of the set of spray nozzles as the end effector moves the set of spray nozzles leftward; and a right color camera arranged to the right of the set of spray nozzles and configured to capture images of the target surface ahead of the set of spray nozzles as the end effector moves the set of spray nozzles rightward.
- Additionally or alternatively, the system can include a near-field 3D stereoscopic camera arranged proximal the set of spray nozzles and configured to capture 3D color images of the target surface. The system can similarly include a near-field multi-point depth sensor (e.g., a set of single-point depth sensors, a 3D LIDAR sensor): arranged proximal the set of spray nozzles and defining a focal axis approximately parallel to the spray axis of the set of spray nozzles; and configured to capture depth values or depth maps representing distances from the set of spray nozzles to points on the target surface in the spray field of the set of spray nozzles.
- For example, the depth sensor can be configured to detect: peripheral distances to at least three discrete points—on a nearby surface—offset about a 20-inch-diameter circle centered on the axis of the set of spray nozzles at a distance of twelve-inches from the set of spray nozzles; and a center distance to a discrete point—on the nearby surface—approximately centered on the axis of the set of spray nozzles at a distance of twelve-inches from the set of spray nozzles.
- In the foregoing implementations, the system can also include a housing with an actuatable aperture arranged adjacent the set of spray nozzles, such as mounted on or near the end effector. In this implementation, the near-field color, the near-field stereoscopic camera, and/or the depth sensor can be arranged in the housing. Accordingly, the controller can: trigger the aperture to close when the spray system is actuated in order to shield these sensors from overspray; and selectively trigger the aperture to open and these sensors to capture images, depth values, and/or depth maps when the spray system is inactive.
- Additionally or alternatively, the system can include a far-field 2D color camera, a far-field 3D stereoscopic camera, and/or a far-field depth sensor, such as: arranged on the work platform; and offset behind the set of spray nozzles (e.g., where these sensors are less subject to overspray).
- In the foregoing implementations, the near-field sensors can be arranged on the end effector and thus move with the set of spray nozzles. Alternatively, these near-field sensors can be fixedly mounted to the work platform, and the controller can: derive a position (i.e., offset distance and orientation) of the work platform relative to the target surface based on data captured by these sensors; track a position of the set of spray nozzles relative to the work platform (e.g., by reading position sensors or encoders arranged on actuators or stages in the end effector); and fuse these platform-to-target-surface data positions and platform-to-spray nozzle positions to track the position of the set of spray nozzles relative to the nearby target surface.
- The far-field optical and depth sensors can be similarly coupled to or decoupled from the end effector and the set of spray nozzles, and the controller can similarly track relative positions of the target surface, these far-field sensors, and the set of spray nozzles.
- In one variation, the system further includes: an ambient temperature sensor; an infrared thermometer configured to read a surface temperature of the target surface; a windspeed sensor; and/or a humidity sensor; etc., such as mounted to the work platform proximal the set of spray nozzles.
- During or prior to deployment of the system to a work site, the computer system can access data representing the building and generate a set of waypoints defining navigation of the system around the building during a subsequent paint application procedure.
- In one implementation, the computer system accesses an aerial or satellite image—such as from an aerial mapping service—of a work site occupied by a building designated for paint application. In this implementation, the computer system then interfaces with an operator to select a set of walls of the building for paint application, such as: by enabling the operator to select walls directly from the aerial image; by recording points dropped on the image by the operator and interpolating walls between these points.
- In another implementation, an operator deploys an unmanned aerial vehicle to the work site prior to or with the system. The aerial vehicle then executes an autonomous or manually-controlled flight path to navigate around the building while capturing color and/or depth images of the building. The aerial vehicle and/or the computer system then compiles these color and/or depth images into a three-dimensional model of the building and surrounding topography. The computer system then: presents the model of the building to the operator; and interfaces with the operator to select a set of walls of the building for paint application, such as by enabling the operator to select surfaces or define bounded areas (e.g., boxes) on walls represented in the model of the building.
- In yet another implementation, for a newly-constructed and completed building erected at the work site, the computer system can access a virtual (e.g., CAD) three-dimensional georeferenced model of the building, such as generated by an engineer or behalf of a site manager. Alternatively, for an in-process building currently under construction at the work site, the computer system can similarly access a virtual (e.g., CAD) three-dimensional georeferenced model of the building. The computer system then: presents this virtual model of the building to the operator; and interfaces with the operator to select a set of walls of the building for paint application, such as by enabling the operator to select surfaces or define bounded areas (e.g., boxes) on walls represented in the model of the building.
- In the foregoing implementations, the computer system then: scans the model of the building—including surrounding topographical features—for obstacles near the selected walls of the building, such as shrubs, utility boxes, signs, and benches; and defines a system path along the walls selected by the operator, such as nominally offset from these walls by a predefined target system offset distance (e.g., six feet) and that maintains a minimum distance (e.g., two feet) from detected obstacles. The computer system then constructs an ordered sequence of geospatially-referenced keypoints along this system path, such as offset along the system path by a target building segment width (e.g., six feet); and uploads these keypoints to the system.
- Furthermore, the computer system can: extract a height of a selected wall of the building at each building segment; and annotate each keypoint with a height of the corresponding building segment. The system can then navigate the work platform and the set of spray nozzles to the height specified in a keypoint when applying paint to the corresponding building segment on the building, as described below.
- Alternatively, the system can autonomously generate the system path and sequence of keypoints in real-time based on data captured by the system while applying paint to the building.
- In one variation, the computer system can define a system path (or “toolpath”) along the walls selected by the operator and then construct an ordered sequence of keypoints—representing positions of the set of spray nozzles relative to the target surface—along this system path and upload these keypoints to the system. The controller can then execute a paint application cycle and navigate the system along these keypoints of the toolpath to apply paint over a target surface.
- Furthermore, the computer system can scan the model of the building and detect features (e.g., a set of doors, a set of windows, a set of segments) on a target surface of the building. The computer system can then derive target zones for application of paint onto the target surface and keepout zones to prevent application of paint onto these features (e.g., a set of doors, a set of windows, a set of segments) of the target surface. The computer system can: construct an ordered sequence of keypoints including geospatial locations, orientations, and commands for activation and/or deactivation of the set of spray nozzles upon entry into a target zone and/or a keepout zone during a paint application cycle.
- For example, the computer system can access a first image (e.g., color image)—captured by the optical sensor—depicting a target surface of the building and extract a set of features from the first image. Then, based on these set of features the computer system can derive a set of keypoints, each keypoint in the set of keypoints including: a geospatial location to locate the set of spray nozzles facing the target surface; an orientation to locate a spray axis of the set of spray nozzles normal to the target surface; a first command to activate the set of spray nozzles for application of paint onto the target surface; and a second command to deactivate the set of spray nozzles to prevent application of paint onto the target surface. The computer system can then: assemble the set of keypoints into a toolpath for execution by the system and upload these keypoints to the system. The controller can then autonomously traverse the set of spray nozzles across segments of the target surface according to the toolpath.
- In another variation, the computer system can receive dimensions of the building from an operator and leverage these dimensions to derive a boustrophedonic raster pattern. The computer system can then upload this boustrophedonic raster pattern to the system to execute during a paint application cycle.
- For example, the computer system can: receive dimensions of the building from an operator and based on the dimensions of the building: derive a first width of the first segment of the target surface; derive a first height of the first segment of the target surface; derive a second width of a second segment of the target surface; and derive a second height of the second segment of the target surface. The computer system can then: define a first set of lateral raster legs (e.g., horizontal raster legs) based on the first width of the first segment and the second width of the second segment of the target surface; define a second set of vertical raster legs and connection the first set of raster legs based on the first height of the first segment and the second height of the second segment of the target surface; and compile the first set of lateral raster legs and the second set of vertical raster legs into a boustrophedonic raster pattern for execution by the system.
- The computer system then uploads this boustrophedonic raster pattern to the system and the controller can: access the boustrophedonic raster pattern for the target surface; apply paint from the set of spray nozzles across the first segment of the target surface according to the boustrophedonic raster pattern; and apply paint from the set of spray nozzles across the second segment of the target surface according to the boustrophedonic raster pattern.
- The controller can implement these methods and techniques for each other segment of the target surface and for each other target surface to complete application of paint across the building.
- The operator can manually deploy the system onto the work site, such as by navigating the system near a first keypoint along the system path or otherwise near a first corner of the building to face a first target zone on the first segment of the building and then verifies the target zone prior to executing a paint application cycle. Alternatively, once deployed on the work site, the system can implement autonomous navigation techniques to autonomously navigate the set of spray nozzles toward the first keypoint along the system path and execute a paint application cycle, as further described below.
- In one implementation, once the operator manually deploys the system onto the work site such that the set of spray nozzles face the first target zone on the target surface and/or the first keypoint of the toolpath, the computer system can generate a prompt for the operator to verify the position of the set of spray nozzles. Then, in response to the operator's verification of the position, the controller can execute a paint application cycle for the first segment. Alternatively, in response to the operator's disapproval of the position, the controller can navigate the set of spray nozzles to a second keypoint and execute a paint application cycle for the first segment.
- In one variation, the work platform includes a light projection subsystem, and the controller triggers the light projection subsystem to project reference features (e.g., a lighted box perimeter)—visible to the operator—onto the current building segment to fully bound the target zone within this segment of the building. In this variation, the controller can: scan images captured by the color camera(s) for the reference feature; and implement methods and techniques described above to define target and keepout zones—fully bounded by the reference feature—within the segment of the building segment.
- For example, during a first paint cycle, the controller can: project a lighted box perimeter—visible to an operator affiliated with the building—onto a first segment of the target surface; and generate a first prompt for the operator to verify that the lighted box perimeter corresponds to a first target zone on the first segment of the target surface. Then, in response to receiving verification of the lighted box perimeter from the operator and in response to detecting the set of spray nozzles facing the lighted box perimeter on the first segment, the controller can apply paint within the lighted box perimeter (e.g., the first target zone) onto the first segment of the target surface via the set of spray nozzles. The controller can then: initiate a second paint application cycle; traverse the lighted box perimeter vertically across the second segment of the target surface; project the lighted box perimeter onto the second segment of the target surface; and generate a second prompt for the operator verify that the lighted box perimeter corresponds to a second target zone on the second segment of the target surface. Then, in response to receiving verification of the lighted box perimeter from the operator and in response to detecting the set of spray nozzles facing the lighted box perimeter on the second segment, the controller can apply paint within the lighted box perimeter (e.g., the second target zone) onto the second segment of the target surface via the set of spray nozzles.
- Additionally or alternatively, in response to receiving verification of the lighted box perimeter from the operator, the controller can define paint parameters for the system prior to application of paint onto the target surface (e.g., in real time). For example, in response to receiving verification of the lighted box perimeter from the operator, the controller can: define a first standoff distance between the set of spray nozzles and the first segment of the target surface; define a first flow rate of paint through the set of spray nozzles; and define a first speed of the set of spray nozzles for application of paint onto the first segment of the target surface. The controller can then apply paint within the lighted box perimeter onto the first segment of the target surface based on the first standoff distance between the set of spray nozzles and the first segment of the target surface, the first flow rate, and the first speed.
- Therefore, the operator can manually deploy the system onto the work site and responsive to the operator's verification and/or disapproval of the lighted box perimeter, the controller can define paint parameters for the system and autonomously activate the set of spray nozzles to apply paint to segments of the target surface during a paint application cycle.
- In one implementation, once the system occupies a first keypoint or otherwise faces the first building segment, the controller can autonomously drive the work platform upwardly to locate the top edge of the target surface within the first building segment within the spray field of the set of spray nozzles and within the field of view of the optical sensor. In one implementation, the system drives the work platform upwardly to locate the set of spray nozzles at a known peak height of the building, such as a known peak height of the building within the first building segment.
- For example, during a first paint application cycle, the controller can: autonomously navigate the set of spray nozzles along the toolpath to a first target position relative a first segment of the target surface based on a first keypoint; and activate the set of spray nozzles to apply paint over the first segment of the target surface according to the toolpath.
- Alternatively, the system can implement methods and techniques described below to detect and track the top edge of this first building segment in a color and/or depth image.
- In one implementation, the computer system (or the system) retrieves a paint application specification for paint loaded into the spray system, a current configuration of the system, and/or paint application parameters for the building, such as including: a building segment width (e.g., six feet); a building segment height (e.g., four feet or full building height); a geometry (e.g., vertical and lateral spray angles) of the set of spray nozzles currently installed in the spray system; a target nozzle standoff distance range from the target surface (e.g., between eleven inches and thirteen inches); a minimal paint spray overlap (e.g., 50%); a target paint thickness; and/or a surface temperature application range for the paint; etc.
- The controller (or the computer system) can also: calculate a vertical raster pitch distance (e.g., six inches) based on the nozzle spray geometry and the minimal paint spray overlap; set or calculate a target paint flow rate; and/or calculate a raster speed (e.g., two feet per second) based on the target paint thickness, the vertical raster pitch distance, the nozzle spray geometry, and the target paint flow rate. Alternatively, the controller (or the computer system) can retrieve a predefined target raster speed and/or paint flow rate associated with the set of spray nozzles.
- However, the controller (or the computer system) can retrieve or calculate paint parameters for the building in any other way.
- Generally, with the set of spray nozzles located within a first segment of the target surface at the top of the first building segment, the controller can execute a paint application cycle to navigate the set of spray nozzles across the first segment of the target surface and apply paint from the set of spray nozzles onto this first segment of the target surface.
- More specifically, during a paint application cycle, the controller can: trigger the end effector to locate the set of spray nozzles at a first position (e.g., a top-right position within the first segment); raster the end effector across the face of the target surface within the first segment based on a set of spray parameters; capture depth images of the target surface (e.g., via the near-field depth sensor); implement closed-loop controls to maintain a pitch and yaw orientation of the set of spray nozzles and a linear standoff distance of the set of spray nozzles from the target surface within a target standoff distance range; capture color images of the target surface (e.g., via the near-field color camera); detect target and keepout zones within the first segment based on features extracted from these color images; selectively activate the spray system when the set of spray nozzles is facing target zones; and selectively deactivate the spray system when the set of spray nozzles is facing keepout zones.
- The controller then manipulates the work platform and the end effector to locate the set of spray nozzles to face a target zone within the first segment of the target surface such as three inches to the left of the left edge of the segment of the building segment with the axis of the set of spray nozzles aligned with (or slightly below, such as two inches below) the top edge of the segment of the building segment.
- The controller further executes a sequence of scan and paint application cycles during the paint application procedure: to raster the set of spray nozzles across the segment of the target surface; to detect and align the set of spray nozzles to reference features—defined by the projected lighted box perimeter—within a segment of the target surface and/or to a target zone within the segment of the target surface; and to selectively apply paint to these features.
- As shown in
FIGS. 1A and 1B , during a scan cycle, the controller can: retrieve a current color image representing a segment of the target surface currently facing the set of spray nozzles from the nearfield color camera; detect and extract visual features (e.g., edges, masking tape) from this color image; characterize positions and geometries of target and keepout zones within this segment of the target surface currently facing the set of spray nozzles based on these visual features that represent the target surface within and around the current spray field of the set of spray nozzles. - Additionally or alternatively, during a scan cycle, the controller can: retrieve a current depth map representing this segment of the target surface currently facing the set of spray nozzles from the depth sensor; detect and extract spatial features (e.g., edges, planes) from this depth map; characterize positions and geometries of target and keepout zones within this segment of the target surface currently facing the set of spray nozzles based on these spatial features that represent the target surface within and around the current spray field of the set of spray nozzles.
- In one implementation, during a scan cycle, the system can sample near-field a depth sensor on the work platform to read: a set of peripheral distances to points on a nearby surface, such as distances from the depth sensor to three or more vertically- and laterally-offset points on the nearby surface; and a center distance to a point on the nearby surface approximately centered on the axis of the set of spray nozzles. The system then implements closed-loop controls to modulate stages in the end effector to equalize the peripheral distances and to drive the center distance to the standoff distance (e.g., twelve inches). For example, during the scan cycle, at a depth sensor mounted to the spray system and adjacent the optical sensor, the controller can: receive a set of peripheral distances to points on a first segment of the target surface from the depth sensor; detect a first peripheral distance between the set of spray nozzles and the first segment of the target surface; and, in response to the first distance falling within a target standoff distance range between the set of spray nozzles and the first segment of the target surface (e.g., between eleven inches and thirteen inches), apply paint from the set of spray nozzles over the first segment of the target surface according to the toolpath.
- Then or concurrently—while driving the set of spray nozzles leftward across the segment of the building segment—the controller can: access a near-field color image from the left near-field color camera; and scan the color image for features indicating the target surface, target zones, and keepout zones ahead of motion of the set of spray nozzles. For example, the system can: retrieve a color palette of the target surface, such as from the three-dimensional model of the building or aerial images of the building; isolate a first region of the color image containing color values within and near the color palette of the target surface; and identify and label the first region of the color images as representing the target surface. In this example, the system can also retrieve a color palette of a volume of paint loaded into the spray system from the, such as from the three-dimensional model of the building or aerial images of the building isolate a second region of the color image containing color values different from the color palette of the target surface; and identify and label the second region of the color images as representing a surface or volume other than the target surface.
- Alternatively, while driving the set of spray nozzles leftward across the segment of the building segment—the controller can: access a near-field color image from the left near-field color camera; and scan the color image for features indicating a first volume of paint. For example, the controller can: access the paint application specification defining a color palette of the first volume of paint; scan a first image within the paint map for a set of color values analogous to the color palette of the first volume of paint; and label the first image, containing the set of color values as the first volume of paint; and annotate the first image within the paint map with the first paint thickness—predicted during the paint application cycle—corresponding to the first volume of paint.
- Additionally, the system can: implement edge detection to detect edges of the building (e.g., vertical wall edges, window and door edges, horizontal soffit edges) depicted in the color image; label the first region of the color image—bounded by these edges of the building and containing color values within and near the color palette of the target surface—as the target surface; and label remaining regions of the image as other than the target surface.
- Additionally or alternatively, the system can: read a concurrent depth image from the depth sensor; detect depth discontinuities in the concurrent depth image; identify a first region of the depth image containing depth values with a nominal depth range (e.g., less than six feet) as corresponding to the target surface of the building; project a first boundary of the first region, labeled as the target surface, onto the color image; and label a region of the color image outside of the first boundary as other than the target surface.
- Similarly, while driving the set of spray nozzles rightward across the segment of the building segment, the controller can: access a near-field color image from the right near-field color camera; and implement similar methods and techniques to detect and annotate features near and ahead of motion of the set of spray nozzles.
- In one variation, during a scan cycle, the system can sample the near-field depth sensor on the work platform to read: a center distance between the set of spray nozzles and points on a first segment of the target surface. The system then implements closed-loop controls to modulate stages in the end effector to maintain the center distance within a standoff distance range (e.g., between eleven inches and thirteen inches).
- For example, during the scan cycle, at a depth sensor mounted to the spray system and adjacent the optical sensor, the controller can: receive a set of distances between the set of spray nozzles and points on a first segment of the target surface from the depth sensor; detect a center distance (e.g., twelve inches) between the set of spray nozzles and the segment of the target surface; and in response to the center distance falling within a target standoff distance range between the set of spray nozzles and the segment of the target surface (e.g., between 11 inches and 13 inches), apply paint from the set of spray nozzles over the first segment of the target surface according to the toolpath.
- Alternatively, during a paint application cycle, the controller can: navigate the set of spray nozzles along the toolpath to a target position relative a first segment of the target surface based on a keypoint; and detect the center distance (e.g., ten inches) between the set of spray nozzles and the first segment of the target surface at the depth sensor. Then, in response to the center distance falling outside of the target standoff distance range between the set of spray nozzles and the first segment of the target surface (e.g., between 11 inches and 13 inches), the controller can: generate a new target position relative the first segment of the target surface to maintain the center distance within the target standoff distance range; and store this new target position in the keypoint to update the toolpath. The controller can then navigate the set of spray nozzles to this new target position relative the first segment of the target surface based on the keypoint and complete a next paint application cycle.
- In another variation, the system can sample the depth sensor to access a set of depth values and interpret a set of peripheral distances between the set of spray nozzles and the first segment of the target surface. The system then implements closed-loop controls to modulate stages in the end effector to maintain the set of spray nozzles normal to the first segment of the target surface during a paint application cycle. For example, during a first paint application cycle, the system can detect a first set of depth values of the set of spray nozzles at the depth sensor mounted to the spray system and adjacent the optical sensor. The controller can then maintain the set of spray nozzles normal to the first segment of the target surface to apply paint laterally across the first segment of the target surface based on the first set of depth values. During a second paint application cycle, the system can detect a second set of depth values of the set of spray nozzles at the depth sensor. The controller can then maintain the set of spray nozzles normal to the second segment of the target surface to apply paint laterally across the second segment of the target surface based on the second set of depth values.
- Therefore, the system can leverage depth values from the depth sensor to maintain a center distance between the set of spray nozzles and a segment of the target surface within a standoff distance range (e.g., between eleven inches and thirteen inches) and thereby, maintain the set of spray nozzles normal to the segment of the target surface during a paint application cycle.
- In one variation, an operator applies masking tape to regions of the target surface to define keepout zones. Accordingly, the controller can: implement edge detection and color matching techniques to identify masking tape in the current pre-paint application color image; and define a boundary between a target zone and keepout zone on the target surface along the masking tape.
- For example, the system can: scan the current pre-paint application color image for pixels containing color values near a known masking tape color (e.g., blue, green); implement edge detection to detect a near-side edge of the masking tape; define the target zone extending up to the nearside edge of the masking tape; and define the keepout zone extending behind the nearside edge of the masking tape.
- In one variation, the system is paired with a masking tape that includes: a first color (e.g., blue) extending along a first edge configured to face a target zone; and a first color (e.g., green) extending along a second edge configured to face a keepout zone. In this variation, the controller can: detect the masking tape in the current pre-paint application color image; detect the first and second colors of the masking tape; identify a target zone on the target surface facing the first color of the masking tape; identify a keepout zone on the target surface facing the second color of the masking tape; and label or confirm these target and keepout zones of the target surface in the color image accordingly.
- For example, the controller can detect the first and second colors of the masking tape within a current pre-paint application color image depicting first and second segments of the target surface. The controller can then identify a target zone on the first segment of the target surface based on the first color (e.g., blue) and identify a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second color (e.g., green). In this example, the controller can: access a first masking tape color associated with target zones (e.g., blue) and indicating activation of the set of spray nozzles to apply paint to the target surface; and access a second masking tape color associated with keepout zones (e.g., green) and indicating deactivation of the set of spray nozzles. Then, the controller can extract a set of features from a current pre-paint application color image depicting a first segment of the target surface and based on the set of features: detect the first masking tape color (e.g., blue) in the first segment of the target surface; and identify a first target zone on the first segment of the target surface facing the first masking tape color. During a first paint application cycle, the controller can activate the set of spray nozzles to apply paint within the first target zone on the first segment of the target surface. Then, based on the set of features, the controller can: detect the second masking tape color (e.g., green) in a second segment above the first segment of the target surface; and identify a first keepout zone on the second segment of the target surface based on the second masking tape color (e.g., green). During a next paint application cycle, the controller can: navigate the set of spray nozzles across second segment; and deactivate the set of spray nozzles to prevent paint application on the first keepout zone on the second segment of the target surface.
- In one variation, the controller can access a masking tape color database containing colors of masking tape annotated with a target zone or a keepout zone for the building. In particular, the controller can detect first and second colors (e.g., orange, red) of masking tape within a current pre-paint application color image based on the masking tape color database. The controller can then identify a target zone on a first segment of the target surface corresponding to the first color (e.g., purple) and identify a keepout zone on a second segment adjacent the first segment of the target surface corresponding to the second color (e.g., red).
- For example, the controller can: access a masking tape color database; extract a set of features from a current pre-paint application color image depicting a first segment of the target surface. Then, based on the set of features, the controller can: detect the first masking tape color (e.g., purple), from the masking tape database, in a first segment of the target surface; and identify a target zone on the first segment of the target surface based on the first masking tape color (e.g., purple). During a first paint application cycle, the controller can: navigate the set of spray nozzles across the first segment of the target surface along the toolpath; and in response to detecting a subset of spray nozzles facing the target zone, activate the subset of spray nozzles to apply paint within the target zone on the first segment of the target surface. Then, based on the set of features, the controller can: detect a second masking tape color (e.g., red) from the masking tape color database in a second segment of the target surface; and identify a keepout zone, adjacent the target zone, on the second segment of the target surface based on the second masking tape color (e.g., red). During a second paint application cycle, the controller can: navigate the set of spray nozzles across the second segment of the target along the toolpath; and in response to detecting a spray nozzle facing the keepout zone, deactivate the spray nozzle, to prevent paint application within the keepout zone on the second segment of the target surface.
- Therefore, the controller can leverage a masking tape color database to identify target and keepout zones corresponding to colors of masking tape. Accordingly, the controller can selectively activate and/or deactivate spray nozzles facing a target and/or a keepout zone.
- Additionally or alternatively, the system can: define a target surface, a target zone, and/or a keepout zone in the depth image (e.g., rather than the concurrent color image).
- 7.5 Real-Time Paint Application Controls within Segment
- The system can then: activate the spray system (e.g., the paint supply subsystem) in response to detecting a target zone facing the set of spray nozzles (i.e., in response to detecting a target zone within the spray field of the set of spray nozzles); and similarly deactivate the spray system in response to detecting a keepout zone facing the set of spray nozzles (i.e., in response to detecting a keepout zone within the spray field of the set of spray nozzles).
- In one implementation, the system characterizes the current spray field of the set of spray nozzles (i.e., the geometry of the fan of paint output by the set of spray nozzles and intersecting the target surface under current spray conditions) based on: a current standoff distance of the set of spray nozzles from the target surface; a current orientation (e.g., angular pitch and yaw offset from an adjacent segment of the target surface) of the set of spray nozzles relative to the target surface; a geometry of the set of spray nozzles; and a current paint flow rate through the set of spray nozzles (or a spray system pressure). The system can then project the current spray field onto the current annotated color image based on a known offset between the color camera and the set of spray nozzles; and/or project the current spray field onto the current annotated depth map based on a known offset between the depth sensor and the set of spray nozzles.
- Then, in response to more than a threshold proportion of the spray field—projected onto the current annotated color image (or depth map)—intersecting a target zone indicated in the color image and in response to less than a threshold proportion of the spray field intersecting a keepout zone indicated in the color image, the controller can activate the spray system to dispense paint onto the target surface. Conversely, in response to less than a threshold proportion of the spray field—projected onto the current annotated color image (or depth map)—intersecting a target zone indicated in the color image and in response to more than a threshold proportion of the spray field intersecting a keepout zone indicated in the color image, the controller can deactivate the spray system to dispense paint onto the target surface.
- In one example, the set of spray nozzles is oriented on the work platform such that the spray field of the set of spray nozzles is approximately elliptical with a long elliptical axis extending vertically and a short elliptical axis extending horizontally; and paint parameters for the building specify 50% overlap between spray application paths and a set of spray nozzles with spray field height of 12 inches at the target standoff distance from the target surface. In this example, the controller can: activate the spray system if more than 40% of the height of the vertical span of the spray field intersects a target zone on the target surface and/or if more than 10% of the width of the vertical span of the spray field intersects a target zone on the target surface; and otherwise deactivate the spray system.
- The computer system can repeat this process to: detect target and keepout regions ahead of the set of spray nozzles in color images captured by the near-field color cameras; maintain the set of spray nozzles normal to the surface of the building based on depth values read from the near-field depth sensor; and selectively activate the spray system while controlling the end effector and/or work platform to raster the set of spray nozzles across the segment of the building segment.
- Additionally or alternatively, the computer system implements similar methods and techniques to: detect target and keepout regions ahead of the set of spray nozzles in color images captured by the far-field color camera (which may be less subject to overspray than the near-field color cameras); maintain the set of spray nozzles normal to the surface of the building based on depth values read from the far-field depth sensor (which may be less subject to overspray than the near-field depth sensor); and selectively activate the spray system while controlling the end effector and/or work platform to raster the set of spray nozzles across the segment of the building segment.
- The controller can repeat the foregoing process in real-time while rastering the set of spray nozzles across the first segment of the target surface of the building.
- For example, the controller can: execute one scan cycle to generate one annotated color (and/or depth) image—labeled with identified target and keepout zones — per lateral distance interval (e.g., 0.500″) traversed by the set of spray nozzles; calculate and project the current spray field of the set of spray nozzles onto the current annotated color image; selectively activate or deactivate the spray system based on intersection of the projected spray field with target and keepout zones identified in the current annotated color image; maintain the current active or inactive state of the spray system as the end effector traverses the set of spray nozzles laterally across this segment of the building segment; and repeat this process once the end effector again traverses the lateral distance interval in order to achieve a horizontal paint application resolution of +/−0.25″.
- In another example, the controller can: execute one scan cycle to generate one annotated color (and/or depth image)—labeled with identified target and keepout zones—per lateral imaging distance interval (e.g., 1.0″) traversed by the set of spray nozzles; implement dead reckoning techniques to interpolate the position of the current spray field of the set of spray nozzles relative to target and keepout zones identified in the current annotated color image based on motion of the end effector; selectively activate or deactivate the spray system based on this interpolated intersection of the projected spray field with target and keepout zones in order to achieve a greater horizontal paint application resolution (e.g., +/−0.05″); and reset and repeat this process once the end effector again traverses the lateral distance interval (e.g., 1.0″).
- In one variation, prior to initiating paint application within the segment of the building segment, the controller: triggers the far-field color camera to capture one pre-paint color image of the entire segment of the building segment; triggers the far-field color camera to capture a set of color images of the entire segment of the building segment and stitches these color images into a composite pre-paint color image of the segment of the building segment; triggers the nearfield color camera to capture a sequence of images while the work platform rasters the color camera across the segment of the building segment; and then compiles these color images into a composite color pre-paint image of the segment of the building segment.
- In this variation, the controller then: implements methods and techniques described above to detect and label target and keepout zones within the segment of the building segment; defines a raster pattern (or “toolpath”) for the segment of the building segment based on the width of the building segment, the height of the building segment, and the vertical raster pitch distance; projects the raster pattern on the annotated color image of the segment of the building segment; calculates and projects planned spray fields of the set of spray nozzles—when subsequently traversing the raster pattern—onto the annotated color image of the segment of the building segment; detects intersections between these projected spray fields, target zones, and keepout zones identified in the color image; and inserts keypoints containing spray system activation and deactivation commands along the raster pattern based on these intersections.
- The controller then: triggers the aperture(s) in the housing described above to enclose the color camera (and depth sensor, etc.) to shield the color camera from overspray; interfaces with the work platform to navigate the set of spray nozzles along the raster pattern; implements dead reckoning techniques to track the position of the set of spray nozzles along the raster pattern; and selectively activates and deactivates the spray system based on commands contained in keypoints along the raster pattern as the estimated position of the set of spray nozzles reaches these keypoints.
- In one variation, the work platform includes a light projection subsystem, and the controller: triggers the light projection subsystem to project reference features (e.g., a lighted box perimeter)—visible to the operator—onto the current building segment; then triggers the optical sensor to capture an image of the building segment; detects the reference features in a particular region of the image of the building segment; and then implements the foregoing methods and techniques to detect target and keepout zones within the particular region of the image and to define a raster pattern and keypoints based on features detected within this region of the image.
- Upon reaching the terminus of a raster pattern within the current segment of the building segment, the controller can manipulate the work platform, the end effector, and/or the chassis to relocate the set of spray nozzles to a target position of a next segment of the building segment, such as below the preceding segment of the building segment or above and to the left of the current segment of the building segment.
- In one implementation, for the end effector that includes a vertical stage configured to move the set of spray nozzles vertically downward to traverse a segment of a building segment while the work platform remains in a fixed vertical position, the system can: move the work platform downward to face a next segment within the current building segment; complete application of paint—from top to bottom—within the current building segment; and then autonomously navigate the chassis leftward to a geospatial keypoint that locates the chassis facing a next building segment and raise the work platform and the end effector to locate the set of spray nozzles near the top of this next building segment.
- Alternatively for the end effector that excludes a vertical stage and in which the work platform is configured to lower continuously from the maximum height of the building segment to the base of the building segment, the system can then autonomously navigate the chassis leftward to a geospatial keypoint that locates the chassis facing a next building segment and raise the work platform to locate the set of spray nozzles near the top of this next building segment.
- In one implementation, the controller implements geospatial triangulation and closed-loop controls to autonomously navigate the chassis from the current geospatial keypoint to the next geospatial keypoint.
- In another implementation, the operator places wireless transmitters on or near the building prior to the paint application procedure. In this implementation, the controller: queries these wireless transmitters; implements triangulation techniques to track the position of the system relative to these wireless transmitters based on signals received from these wireless transmitters; and implements closed-loop controls to autonomously navigate the chassis from its current position facing the first building segment to a next position facing the second building segment based on the position of the system relative to these wireless transmitters.
- In a similar implementation, the system includes a set of optical detectors, and the operator places optical emitters (e.g., laser references) at opposing ends of a wall of the building prior to the paint application procedure, such as offset from the wall of the building by a target offset distance (e.g., six feet). In this implementation, the controller: samples the optical detectors to detect light from the optical emitters; tracks the position of the system relative to these optical emitters at the ends of this wall based on light detected by the optical detects; implements closed-loop controls to autonomously navigate the chassis from its current position facing the first building segment to a next position facing the second building segment based on the position of the system relative to these optical emitters; and implements closed-loop controls to autonomously maintain the centerline (or other axis) of the chassis centered along light beams output by the optical emitters wile navigating the chassis between building segments on the building.
- Additionally or alternatively, the controller can: track an offset distance between the chassis and the wall of the building based on depth maps read from depth sensors mounted to the chassis, the work platform, and/or the end effector; and implement closed-loop controls to maintain this offset distance between the chassis and the wall while navigating the system between building segments based on distance values contained in depth maps read from these depth sensors.
- In the foregoing implementation, the controller can also: capture color images and/or depth maps of a field around the system; and implement object avoidance techniques to avoid collision with objects and surfaces detected in these color images and/or depth maps.
- Furthermore, the controller can: implement object recognition, perception, artificial intelligence, and/or other computer vision techniques to identify types of objects in the field around the system and distance between the system and these objects based on features detected in the color images and/or depth maps; and generate alerts or prompts to redistribute these objects within the work site, such as to reduce risk of overspray on these objects, as shown in
FIG. 4 . - In one implementation, the controller (or the computer system) can set a threshold overspray distance, such as: proportional to current detected or peak local windspeed; or proportional to the maximum height of the building or a remaining segment of the building to be painted. The controller can also: retrieve a list of sensitive objects, such as cars, trucks, motorcycles, bicycles, trailers, signs, and people; and implement object recognition techniques to detect presence of such sensitive objects around the system and distances between these objects and the system.
- In this implementation, in response to detecting a sensitive object within the threshold overspray distance, the controller can: pause the paint application procedure; generate an alarm and notification to relocate the sensitive object away from the system; transmit the notification to the operator and/or to a building associate affiliated with the building or work site; and then resume the paint process once the sensitive object is no longer within the threshold overspray distance of the system or once the operator or building associate manually clears the alarm.
- For example, the controller can: set a threshold overspray distance—proportional to the first height of a first segment of the target surface (e.g., a four-foot-tall segment)—between the set of spray nozzles and the target surface; and retrieve a list of sensitive objects associated with the building, such as such as cars, trucks, motorcycles, bicycles, trailers, signs, and people. Then, during a paint application cycle for the first segment of the target surface, the controller can: detect a sensitive object (e.g., a car), from the list of sensitive objects, within the threshold overspray distance of the first segment of the target surface based on the a color image captured by the optical sensor; and, in response to detecting the sensitive object within the threshold overspray distance, generate an alarm notifying the operator to remove the sensitive object (e.g., a car). Then, in response to the operator manually clearing the alarm, the controller can navigate the set of spray nozzles along the toolpath to a next target position relative a next segment of the target surface based on a second keypoint contained in the toolpath and execute a paint application cycle for this next segment of the target surface.
- Additionally or alternatively, in this implementation, while rastering the set of spray nozzles across a first segment of the building, the controller can scan the field around the system for sensitive objects not currently within the threshold overspray distance of the system, but that will fall within the threshold overspray distance once the system moves to a next building segment. Then, in response to detecting a sensitive object that will fall within the threshold overspray distance once the system moves to a next building segment, the controller can: predict a target time at which the system will complete application of paint on the current building segment, such as based on the width and height of the building segment, the vertical raster pitch distance, and the current raster speed of the set of spray nozzles; generate a notification to relocate the sensitive object away from the system before the target time in order to avoid disruption to the paint application procedure; transmit the notification to the operator and/or to the building associate; and initiate paint application on the next building segment only upon confirming absence of sensitive objects within the threshold overspray distance of the system or only once the operator or building associate manually confirms paint application on the next building segment.
- In this implementation, the controller can implement similar methods and techniques to: detect other sensitive objects that will fall within the threshold overspray distance of the system when the system navigates to future building segments; predict target times at which the system will complete application of paint on preceding building segments and navigate to these next building segments; and prompt the operator and/or the building associate to relocate the sensitive objects before their corresponding target times in order to avoid disruption to the paint application procedure. For example, the controller (or the computer system) can retrieve or generate a map of the work site (e.g., generated by the aerial vehicle deployed on the site, generated from optical data captured by the system) or color images captured by the system; and annotate the map and/or these color images with locations of sensitive objects and their target times for relocation to avoid disruption to the paint application procedure.
- Generally, the controller and the computer system cooperate to capture visual and non-visual paint application data across segments of the target surface of the building and to compile these data into a paint map.
- More specifically, during a scan cycle, the controller can: capture an initial sequence of images depicting a target surface of the building; and access a spray pattern of a spray nozzle coupled to the spray system. Then, during a paint application cycle: the controller can detect a first set of ambient conditions of the target surface; navigate the spray system along a toolpath to apply a first volume of paint from the spray nozzle over the target surface; predict a first paint thickness of the first volume of paint of the target surface based on the spray pattern; and capture a first sequence of images depicting the first volume of paint of the target surface. The computer system and/or the controller can then aggregate the initial sequence of images, the first sequence of images, the first set of ambient conditions, and the first paint thickness into a paint map of the building.
- In one implementation, during or after the paint application cycle, the computer system (or the controller) can: initialize a paint map of the building; and assemble color and/or depth images captured by the system during the paint application procedure into visual pre-paint application, in-process, and/or post-paint application layers within the paint map.
- More specifically, during the paint application procedure the controller can capture pre-paint application images, such as: far-field images of segment of the building segment prior to applying paint across the segment of the building segment; near-field images of segment of the building segment while the work platform rasters the near-field camera across the segment of the building segment prior to applying paint to the surface; and/or near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment during application of paint onto the target surface. Similarly, the controller can capture in-process images including: near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment during application of paint onto the target surface. Furthermore, the controller can capture post-paint application images including: far-field images of the segment of the building segment after applying paint across the segment of the building segment; near-field images of the segment of the building segment while the work platform rasters the near-field camera across the segment of the building segment after completion application of paint across the segment of the building segment; and/or near-field images captured by the left near-field camera while the work platform moves the set of spray nozzles rightward across the segment of the building segment and near-field images captured by the right near-field camera while the work platform moves the set of spray nozzles leftward across the segment of the building segment during application of paint onto the target surface.
- Accordingly, the computer system (or the controller) can: stitch these pre-paint application images into a composite (e.g., panoramic) pre-paint application image of the building and store this composite pre-paint application image in a pre-paint application layer of the paint map; stitch these in-process images into a composite (e.g., panoramic) in-process image of the building and store this composite in-process image in an in-process layer of the paint map; and/or stitch the post-paint application images into a composite (e.g., panoramic) post-paint application image of the building and store this composite post-paint application image in a pre-paint application layer of the paint map.
- In the variation described above, in which the system captures depth maps of the building via the depth sensor, the computer system can further assemble these depth maps into a three-dimensional model of the building.
- Alternatively, in the variation described in which the computer system accesses a three-dimensional model of the building, such as based on data captured by an aerial vehicle deployed to the work site, the computer system can: overlay the foregoing pre-paint application, in-process, and post-paint application layers onto the building model; and enable the operator to virtually navigate between regions of the building model and selectively activate these layers to view pre-paint application, in-process, and post-paint application imagery of these building regions. For example, the computer system and/or controller can: access the three-dimensional model of the building; compile the initial sequence of images into a pre-paint application composite image of the target surface; compile the first sequence of images into a post-paint application composite image of the target surface; project the pre-paint application composite image of the target surface onto the three-dimensional model of the building; project the post-paint composite image of the target surface onto the three-dimensional model of the building; and annotate the pre-paint application composite image and the post-paint composite image with the first set of ambient conditions and the first paint thickness within the three-dimensional model of the building.
- Additionally or alternatively, in this variation, the computer system can: link individual pixels in the building model to sets of post-paint application color images that depict areas on the building corresponding to these individual pixels; store this annotated building model as the paint map; and enable the operator to virtually navigate between regions of the building model and select individual pixels within the building model to access pre-paint application, in-process, and/or post-paint application images of corresponding areas of the building.
- Generally, during application of paint onto the target surface of the building, the controller can sample and record surface temperatures of the target surface, ambient air temperature, ambient humidity, and/or wind speed, such as at a rate of 2 Hz. Accordingly, the computer system and/or the controller can annotate the paint map with these ambient and surface data.
- In one implementation, the controller: implements dead reckoning, structure from motion, visual odometry, and/or other techniques to track the field of view of the infrared thermometer across the target surface; reads surface temperatures across the target surface—such as just before application of paint—from the infrared thermometer; and stores surface temperature readings with concurrent positions on the target surface. The computer system (or the controller) can then annotate discrete positions within the paint map with their corresponding surface temperatures just before application of paint on the target surface, such as in real-time.
- For example, the controller can: detect ambient conditions of each segment of the target surface and then aggregate these ambient conditions into the paint map. During a first paint application cycle, the controller can: detect a first surface temperature of the first segment of the target surface; and record a first humidity condition of air proximal the first segment of the target surface. During a second paint application cycle, the controller can: detect a second surface temperature of the second segment of the target surface; and record a second humidity condition of air relative the second segment of the target surface. Then, the controller can aggregate the first surface temperature, the second surface temperature, the first humidity condition, and the second humidity condition into the paint map and annotate each segment within the paint map with these ambient conditions before application of paint on the target surface (e.g., in real-time).
- In another implementation, the controller and the computer system can similarly cooperate to annotate discrete positions within the paint map with ambient (e.g., air temperature, humidity, wind speed) data read from these sensors when the spray field of the set of spray nozzles intersected corresponding positions on the target surface. For example, when configured to raster the set of spray nozzles—defining a one-inch-wide and twelve-inch-tall spray field—across the target surface at a rate of two feet per second with 50% overlap between horizontal raster legs and when the configured to sample these ambient and surface sensors at 2 Hz, the computer system (or the controller) can annotate each discrete region of the paint map that represents discrete six-inch-tall, twelve-inch wide zones on the target surface with corresponding ambient and surface values.
- However, the controller and the computer system can cooperate to capture and store ambient and surface data in any other resolution within the paint map.
- Similarly, during application of paint onto the target surface of the building segment, the controller can: record paint application data; estimate applied (wet) paint thickness on the target surface based on these paint application data; and label regions of the paint map with applied (wet) paint thickness estimated for corresponding zones of the target surface.
- In one implementation, during each sampling interval during application of paint onto the target surface, the controller can: read a paint flow rate from the spray system; implement dead reckoning, visual odometry, geospatial positioning, and/or other techniques to calculate a position of the set of spray nozzles relative to the building (e.g., within a coordinate system of the building); and store the current paint flow rate and the corresponding position of the set of spray nozzles relative to the building. During or upon completion of paint application on the building, the computer system can compile these paint flow rate and position data into predicted paint thicknesses across the target surface of the building.
- In particular, for each paint flow rate and concurrent spray nozzle position pair recorded by the controller, the computer system can: calculate a spray field geometry of the set of spray nozzles based on this paint flow rate and the set of spray nozzles geometry; calculate a paint area on the building visible to the set of spray nozzles when occupying this spray nozzle position based on this spray field geometry; and estimate an instantaneous thickness of paint applied to this paint area on the building based on this paint flow rate.
- For example, the controller can: record a first paint flow rate of paint through the set of spray nozzles; detect a first distance between the set of spray nozzles and the first segment of the target surface; transform the first paint flow rate and the first distance into a width of a spray field of the set of spray nozzles at the first segment of the target surface; detect a first speed of the set of spray nozzles applying paint across the first segment of the target surface; convert the first speed of the set of spray nozzles and the width of the spray field into a paint application rate representing paint applied within the first target zone on the first segment of the target surface; and generate a first paint thickness of paint applied on the first segment of the target surface based on a combination of the paint application rate and the first speed of the first spray nozzle (e.g., dividing the paint application rate by the first speed of the first spray nozzle).
- In this implementation, the computer system can also: interpolate between these corresponding paint areas to derive interpolated paint areas; and interpolate between these instantaneous thicknesses of applied paint to derive interpolated instantaneous thicknesses of paint applied within these interpolated paint areas. The computer system can then: combine (e.g., sum, integrate) these estimated and interpolated instantaneous thicknesses of applied paint within overlapping paint areas across the target surface of the building to estimate total thickness of paint applied to each discrete (e.g., one-inch-square) area of the target surface of the building; and store these estimated total thicknesses of applied paint onto the paint map, such as a paint thickness layer aligned with the target surface depicted in the post-paint application layer of the paint map.
- In one variation, the computer system further corrects estimated instantaneous thicknesses of applied paint based on local concurrent wind speeds detected by the system. In particular, high wind speeds around the set of spray nozzles may decrease a ratio of paint exiting the set of spray nozzles to paint accumulating on the target surface. Accordingly, the computer system can reduce an estimated instantaneous thickness of paint applied within a paint area on the building proportional to wind speed detected by the system when the set of spray nozzles face this paint area.
- Additionally or alternatively, the computer system can implement a non-uniform spray nozzle spray pattern to estimate instantaneous thicknesses of paint applied within a paint area. In particular, the set of spray nozzles may project paint in a non-uniform pattern with greatest paint thickness applied coaxially within the set of spray nozzles (e.g., near a center of the spray field) and least paint thickness near a periphery of the spray field. Accordingly, the computer system can: retrieve a predefined spray pattern for the set of spray nozzles; and adjust, correct, or calculate an instantaneous thickness of paint applied within a paint area based on this spray pattern.
- In one variation, the computer system (or the controller) further: characterizes differences (e.g., reflectivity, color intensity) between color (or multispectral) values of pixels in the post-paint application layer of the paint map and corresponding pixels (i.e., pixels that represent the same physical areas on the building) in the pre-paint application layer of the paint map; flags a subset of post-paint application pixels that exhibit less than a threshold difference from corresponding pre-paint application pixels; and flags regions across the building corresponding to these pixels as likely exhibiting insufficient paint application.
- Furthermore, the computer system can: identify a first subset of pixels—contained within target regions represented in the paint map—that exhibit less than a threshold difference between pre- and post-paint application values; predict insufficient paint coverage in the corresponding areas of the building (e.g., due to insufficient change in the color and reflectively of the building after application of paint); flag corresponding regions of the paint map for further inspection for incomplete paint coverage; and serve the paint map with prompts to review these flagged regions to the operator.
- For example, the computer system can: extract an initial set of color values from an initial image—depicting a target surface of the building—from the pre-paint application layer of the paint map; extract a first set of color values from a first image—depicting a coat of paint on the target surface—from the post-paint application layer of the paint map; and characterize a difference between the initial set of color values and the first set of color values. Then, in response to the difference falling below a threshold color difference, the computer system can: flag each segment of the target surface—depicted in the initial image and the first image—within the paint map for incomplete paint coverage; and serve the paint map with prompts to review these flagged segments of the target surface to the operator and/or to the building associate.
- In this variation, the controller and/or the computer system can also execute the process in real-time to: scan a current pre-paint application image captured by the system during the paint application procedure for a first region that exhibits more than the threshold color (e.g., reflectivity, color intensity) difference from a second, known uncoated region of the building; identify the first region as containing sufficient paint thickness; define a keepout zone around the first region in order to prevent application of excess paint that may result in paint sags on this area of the building segment; define a target zone intersection the second region; and implement methods and techniques described above to selectively apply paint within an area of the building segment corresponding to the second region.
- 9.5 Real-Time v. Post-Hoc Paint Map
- In one implementation, the system: executes the foregoing methods and techniques to capture paint application, ambient, and surface data for the building while autonomously applying paint to the building; and offloads (e.g., streams) these data to the computer system, such as during or following completion of this paint application procedure. The computer system then executes the foregoing methods and techniques to generate the paint map following completion of the paint application procedure.
- Conversely, the controller and/or the computer system can execute this process in real-time to generate and develop the paint map during this paint application procedure. In this variation, the controller and/or the computer system can selectively pause the paint application procedure, generate an inspection prompt, and serve this inspection prompt to the operator in near real-time, such as in response to: estimated applied paint thicknesses deviating from the target paint thickness; surface temperatures on the building falling outside of a surface temperature application range specified for the paint; and/or local wind speeds exceeding a threshold wind speed.
- For example, the controller and/or the computer system can serve a prompt to the operator in near real-time in response to such conditions and only resume the paint application procedure once confirmed by the operator.
- In one variation, the computer system and the controller implement methods and techniques described above to estimate real-time paint thickness of paint applied to the target surface of the building; and implement closed-loop controls to adjust paint application parameters (e.g., paint flow rate through the spray system, spray nozzle raster speed) to reduce a difference between the estimated and target paint thicknesses. For example, the controller can: decrease traversal speed of the set of spray nozzles and/or increase flow rate through the spray system in response to a last estimated paint thickness of paint applied to the building falling below the target paint thickness; and vice versa.
- For example, the computer system can: access the paint application specification defining a target paint thickness for segments of the target surface; extract a set of paint conditions including a paint thickness of a first segment of the target surface from the paint map and a first paint flow rate of the first volume of paint exiting from the set of spray nozzles; and calculate a difference between the paint thickness and the target paint thickness. Then, in response to the difference between the paint thickness and the target paint thickness falling below a threshold difference, the controller can: increase a target flow rate for a second volume of paint exiting the set of spray nozzles of the spray system; initiate a second paint application cycle at the target flow rate; and navigate the system along the toolpath to apply the second volume of paint from the set of spray nozzles over the target surface at the target flow rate.
- In one implementation, the computer system can access the paint application specification defining a set of target conditions of paint for the target surface and a spray pattern for the set of spray nozzles. The computer system can then: detect surface discontinuities on segments of the target surface based on features extracted from the paint map; leverage secondary data (e.g., paint application conditions, ambient and surface conditions, paint thicknesses) to identify these surface discontinuities as paint defects; annotate these segments within the paint map for defect inspection; and serve the annotated paint map to the operator—such as through an operator portal or a computing device (e.g., a mobile device, a tablet)—for defect inspection of these segments virtually through the paint map and/or in-person at the work site.
- In one variation, the computer system can: extract a first set of features corresponding to a first segment of the target surface from the paint map; detect presence of a defect on the first segment of the target surface based on these features; and annotate the pant map to indicate presence of the defect. The computer system can then repeat these methods and techniques for a second segment of the target surface; and in response to a second set of features corresponding to the second segment of the target surface, falling within a threshold deviation range of the first set of features, detect absence of the defect on the second segment of the target surface. The computer system can then annotate the paint map to indicate absence of the defect on the second segment of the target surface.
- Additionally or alternatively, the computer system can serve the annotated paint map and/or paint application layers to the operator via a computing device (e.g., a tablet, a mobile device) thereby enabling the operator to immediately review color image data of all flagged segments for defect inspection on the building.
- In one variation shown in
FIG. 1B , the computer system: flags regions in the paint map annotated with estimated (wet) paint thicknesses significantly less than the target paint thickness for targeted incomplete coverage inspections; and flags regions in the paint map annotated with estimated (wet) paint thicknesses significantly greater than the target paint thickness for paint as paint sag defect inspections. - In one implementation, the computer system divides the paint map into segments representing one-foot-square areas on the target surface of the building. For a first segment of the paint map, the computer system: calculates a maximum absolute deviation of estimated (wet) paint thicknesses within the first segment from the target paint thickness for the building; flags (e.g., annotates, highlights) the first segment in the paint map for incomplete coverage inspection if this maximum absolute deviation falls below the target paint thickness for the building by more than a threshold coverage deviation (e.g., 25%); and flags the first segment in the paint map for sag inspection if this maximum absolute deviation exceeds the target paint thickness for the building by more than the threshold sag deviation (e.g., 15%). The computer system repeats this process for each other segment of the paint map and then presents this annotated paint map to an operator with a prompt to selectively inspect each area of the building flagged in the paint map.
- For example, the computer system can: flag segments in the post-paint application layer of the paint map for incomplete paint coverage inspection; and serve this post-paint application layer to the operator via a mobile device (e.g., a tablet), thereby enabling the operator to immediately review color image data of all flagged incomplete paint coverage defects on the building. In this example, the computer system can: access the paint application specification defining a target paint thickness for paint on the target surface; and extract a first paint thickness exhibited by a first segment of the target surface from the paint map. Then, in response to the first paint thickness falling below a threshold coverage deviation (e.g., 25%) of the target paint thickness, the computer system can: identify the first segment within the paint map as an incomplete coverage defect; flag the first segment within the paint map for incomplete coverage inspection; generate a prompt for incomplete coverage inspection of first segment of the target surface; and serve the paint map and the prompt to an operator.
- Alternatively, the computer system can: flag segments in the post-paint application layer of the paint map for paint sag inspection; and serve this post-paint application layer to the operator via the mobile device (e.g., a tablet, a computing device) thereby enabling the operator to immediately review color image data of all flagged paint sag defects on the building. In this example, the computer system can similarly access the target paint thickness and extract a first paint thickness exhibited by the first segment of the target surface. Then, in response to the first paint thickness exceeding a threshold coverage deviation (e.g., 25%) of the target paint thickness, the computer system can: identify the first segment within the paint map as a paint sag defect; flag the first segment within the paint map for paint sag inspection; generate a prompt for paint sag inspection of the first segment of the target surface of the building; and serve the paint map and the prompt to an operator (e.g., via the operator portal).
- Additionally, the computer system can: retrieve a post-paint application color image—captured by the system—corresponding to each flagged segment in the paint map; label these post-paint application color images with their locations on the building and/or whether these images depict predicted incomplete coverage or paint sags; and serve these post-paint application color images to the operator for remote inspection of the building.
- In another implementation, the computer system: divides the paint map into segments representing one-foot-square areas on the target surface of the building; calculates a maximum absolute deviation of estimated (wet) paint thicknesses within each segment from the target paint thickness for the building; generates an inspection heatmap of the target surface of the building (e.g., a heatmap layer of the paint map) based on the maximum absolute deviation calculated for each segment of the paint map; and then presents this heatmap to the operator with a prompt to prioritize inspection regions of the building based on intensity in corresponding segments of the heatmap. Additionally or alternatively, in this implementation, the computer system can: retrieve a post-paint application color image—captured by the system—corresponding to each segment in the paint map; sort these post-paint application color images by absolute deviation from the target paint thickness for the building; and serve these sorted post-paint application color images to the operator for remote manual inspection of the building.
- In yet another implementation, the computer system can: access a spray pattern of the first spray nozzle. Then, during a first paint application cycle, the computer system can predict a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle. The computer system can then: access a paint application specification defining a target paint thickness for application of paint on the target surface; detect a first surface discontinuity on the first segment of the target surface within the paint map; and, in response to detecting the first surface discontinuity and in response to the first paint thickness exceeding the target paint thickness, detect presence of a sag defect on the first segment of the target surface. The computer system can then: annotate the paint map to indicate the sag defect; generate a prompt to repair the sag defect on the first segment of the target surface; and serve the paint map with the prompt to the operator for inspection of the sag defect.
- Additionally or alternatively, the computer system can: isolate a first edge extending toward a lower bound of the first segment of the target surface within the paint map; isolate a second edge coupled to the first edge and extending toward the lower bound of the first segment of the target surface; detect a first linear surface discontinuity on the first segment of the target surface within the paint map based on the first edge and the second edge; and, in response to detecting the first linear surface discontinuity and in response to the first paint thickness exceeding the target paint thickness, detect presence of a sag defect on the first segment of the target surface.
- Therefore, the computer system can enable the operator to: review and inspect the building remotely; and/or review flagged regions for incomplete paint coverage defects and/or paint sag defects across the building remotely prior to in-person manual review of the most problematic or indiscernible regions.
- In one variation, the computer system can: detect ambient conditions during a paint application cycle; predict a paint quality of paint applied onto the target surface based on these ambient conditions and a paint thickness of the paint; flag regions of the paint map corresponding to segments of the building exhibiting a paint quality differing from a target paint quality for the target surface; and generate a prompt for the operator to review these segments of the paint map and/or inspect corresponding segments of the building for incomplete coverage defect inspection.
- For example, the computer system can: access a spray pattern of the first spray nozzle; detect a first ambient air temperature proximal the first segment of the target surface; detect a first surface temperature of the first segment of the target surface; predict a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle; and predict a first paint quality of paint applied onto the first segment of the target surface based on the first paint thickness, the first ambient air temperature, and the first surface temperature. The computer system can then: access a paint application specification defining a target paint quality for the target surface; detect a first surface discontinuity on the first segment of the target surface within the paint map; in response to detecting the first surface discontinuity and in response to the first paint quality falling below the target paint quality, detect presence of an incomplete coverage defect on the first segment of the target surface; and annotate the paint map to indicate the incomplete coverage defect.
- Therefore, the computer system can leverage ambient conditions and paint thickness to predict a quality of paint applied onto the target surface and flag regions of the paint map exhibiting a paint quality differing from a target paint quality. The computer system can also enable an operator to review these flagged regions for incomplete paint coverage defects.
- In one variation, the computer system (or the controller) similarly: flags or highlights regions of the paint map corresponding to out-of-range surface and ambient conditions; and prompts the operator to selectively review corresponding areas on the building, such as virtually through the paint map or in-person at the work site.
- For example, the computer system can: flag regions of the paint map corresponding to areas of the building facing the set of spray nozzles when local wind speed—detected by the system—exceeded a threshold windspeed; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete coverage.
- In another variation, the computer system can: flag regions of the paint map corresponding to areas of the building exhibiting surface temperatures exceeding a manufacturer temperature specification for the paint when facing the set of spray nozzles; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete paint adhesion or blistering.
- For example, during a first paint application cycle the computer system can: detect a first surface temperature of paint applied on the first segment of the target surface; and access a paint application specification defining a target surface temperature for application of paint; and detect a first circular surface discontinuity on the first segment of the target surface within the paint map. Then, in response to detecting the first circular surface discontinuity and in response to the first surface temperature exceeding the target surface temperature, the computer system can detect presence of a paint blister defect on the first segment of the target surface. The computer system can then: annotate the paint map to indicate the paint blister defect; generate a prompt to repair the paint blister defect on the first segment of the target surface; and serve the paint map and the prompt to the operator for repair of the paint blister defect prior to a scheduled inspection period for the first paint application cycle. Then, in response to receiving confirmation of removal of the paint blister defect from on the first segment of the target surface from the operator, the controller can initiate a second paint application cycle for the first segment of the target surface.
- In yet another variation, the computer system can: flag regions of the paint map corresponding to areas of the building exhibiting surface temperatures below a manufacturer temperature specification for the paint when facing the set of spray nozzles; and generate a prompt for the operator to review these regions of the paint map and/or inspect corresponding areas of the building for incomplete drying and cracking.
- In one variation, the computer system can: flag regions of the paint map corresponding to segments of the building exhibiting a color of paint deviating from a target color of paint; and generate a prompt for the operator to review these segments of the paint map and/or inspect corresponding segments of the building for paint rust inspection.
- For example, the computer system can: isolate an image (e.g., a post-paint application image) depicting a first segment of the target surface within the paint map; and based on features extracted from the image, detect a color of paint (e.g., beige) on the segment; and access the paint application specification defining a target color of paint (e.g., white) for the first segment of the target surface. Then, in response to the color of paint (e.g., beige) deviating from the target color of paint (e.g., white) on the first segment of the target surface, the computer system can: identify the first segment within the paint map as a rust paint defect; and flag the paint map with this rust defect.
- In another example, the computer system can: access a paint application specification defining a target color value for application of paint on the target surface; detect a first color value of paint applied to the first segment of the target surface within the paint map; retrieve a set of paint defect images from a template defect image database, each paint defect image in the set of defect images labeled with a defect type, a defect score, and a paint application procedure; and extract a second set of features from a first paint defect image, in the set of paint defect images, labeled with a rust defect type, a rust repair procedure, and a first defect score. Then, in response to identifying the first set of features analogous to the second set of features and in response to the first color value falling within a threshold color deviation of the target color value, the computer system can detect presence of a rust defect on the first segment of the target surface and annotate the paint map to indicate the rust defect.
- In one implementation shown in
FIG. 3 , the computer system can access a template defect image database containing defect images (e.g., images depicting paint defects) labeled with a defect type, a defect score, and an action (e.g., a paint application procedure). The computer system can then match features of images from the post-paint application layer of the paint map to analogous features of defect images from the template defect image database to identify paint defects on segments of the target surface. Furthermore, the computer system can compile a list of paint defects on the target surface, ranked by defect score, and serve this list to the operator and thereby, enable the operator to selectively review paint defects in a sequential order according to the list. - In one variation, the computer system can retrieve a set of paint defect images from the template defect image database, each paint defect image in the set of defect images labeled with a defect type (e.g., incomplete coverage defect, sag paint defect, rust paint defect, flaking paint defect, paint blister defect, run paint defect, curtain paint defect) a defect score, and a paint application procedure (e.g., series of actions to remove the paint defect). Then, in response to the sequence of images from the post-paint application layer of the paint map corresponding to the set of paint defect images, the computer system can: annotate the sequence of images with the corresponding defect type, defect score, and paint application procedure within the paint map; compile a list of paint defects of the target surface, ranked by defect score; and serve the paint map and the list of paint defects of the target surface to an operator (e.g., via the operator portal).
- For example, the computer system can retrieve a first paint defect image—labeled with a paint sag defect type and a paint sag application procedure—from the template defect image database. Then, in response to a first image from the post-paint application layer of the paint map corresponding to the first paint defect image, the computer system can: annotate the first image with the paint sag defect type and the paint sag application procedure within the paint map; and initiate a second paint application cycle to remove the paint sag defect from the target surface based on the paint sag application procedure.
- Therefore, the computer system and system cooperate to enable the operator and/or the building associate to target the most significant paint defects (e.g., severe paint defects) from a paint application procedure according to the list of paint defects.
- In one variation shown in
FIG. 1B , the computer system: implements methods and techniques described above to annotate the paint map (e.g., the building model) with regions of the building completed by the system based on color images, depth maps, and/or spray nozzle position data captured by the system; highlights regions of the building completed by the system; and presents the paint map—with highlighted regions—to the operator and/or the building associate. - Additionally or alternatively, the computer system can: calculate a proportion (or ratio) of paint application on the building completed by the system; and present this proportion to the operator and/or the building associate.
- Additionally or alternatively, the computer system can: predict a time to completion at which the system will complete application of paint on the whole building, such as based on the remaining unpainted area of the building and the current raster speed of the set of spray nozzles; and present this time to completion to the operator and/or the building associate.
- Furthermore, the computer system can present the paint map—such as the pre- and post-paint application layers of the paint map—to the building associate, thereby enabling the building associate to immediately and remotely review and inspect the condition of the building prior to application of paint and after application of paint.
- Generally, the computer system and system cooperate to autonomously apply (e.g., spray, dispense) paint onto exterior and/or interior walls (e.g., target surfaces) of a building during a paint application procedure; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the building during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the building based on these data; compile these data into a paint map of the building representing paint application data and results of the paint application procedure; and flag regions of the paint map exhibiting paint conditions deviating from the paint application specification.
- However, the system and computer system can cooperate to autonomously apply (e.g., spray, dispense) paint onto exterior and/or interior walls of a ship aircraft (e.g., target mobile surface, target curved surfaces) during a paint application procedure; to maintain peripheral and center distances between the set of spray nozzles and walls of the ship aircraft within the target standoff distance range during a paint application cycle; to maintain the set of spray nozzles substantially normal to each segment of the wall of the ship aircraft; to capture pre-paint application, in-process paint application, and/or post-paint application images of these walls of the ship aircraft during the paint application procedure; capture surface condition, ambient condition, paint application condition, spray system operation, and spray nozzle position data during the paint application procedure; estimate thicknesses of paint (e.g., volumes of paint, coats of paint) applied within discrete (e.g., one-foot-square) segments across these walls of the ship aircraft based on these data; compile these data into a paint map of the ship aircraft representing paint application data and results of the paint application procedure; and flag regions of the paint map exhibiting paint conditions deviating from the paint application specification.
- The systems and methods described herein can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof. Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated by computer-executable components integrated with apparatuses and networks of the type described above. The computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component can be a processor but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.
- As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the embodiments of the invention without departing from the scope of this invention as defined in the following claims.
Claims (20)
1. A method for applying paint to a target surface comprising:
during a first paint application cycle:
autonomously navigating a spray system along a toolpath to apply paint across a first segment of the target surface via a first spray nozzle; and
capturing a first sequence of images depicting paint applied on the first segment the target surface via an optical sensor adjacent the first spray nozzle;
during a second paint application cycle:
autonomously navigating the spray system along the toolpath to apply paint across a second segment of the target surface via the first spray nozzle; and
capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle;
assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface;
extracting a first set of features corresponding to the first segment of the target surface from the paint map;
detecting presence of the first defect on the first segment of the target surface based on the first set of features;
annotating the paint map to indicate the first defect; and
serving the paint map to an operator for inspection of the first defect.
2. The method of claim 1 :
further comprising:
assembling the second sequence of images into the paint map representing application of paint on the first segment and the second segment of the target surface;
extracting a second set of features corresponding to the second segment of the target surface from the paint map; and
in response to the second set of features falling within a threshold deviation range of the first set of features, detecting absence of the first defect on the second segment of the target surface; and
wherein annotating the paint map to indicate the first defect comprises:
annotating the paint map to indicate presence of the first defect on the first segment of the target surface; and
annotating the paint map to indicate absence of the first defect on the second segment of the target surface.
3. The method of claim 1 :
further comprising, during the first paint application cycle:
recording a first paint flow rate of paint through the spray nozzle;
detecting a first distance between the first spray nozzle and the first segment of the target surface;
transforming the first paint flow rate and the first distance into a width of a spray field of the first spray nozzle at the first segment of the target surface;
detecting a first speed of the first spray nozzle applying paint across the first segment of the target surface;
converting the first speed of the first spray nozzle and the width of the spray field into a paint application rate representing paint applied to areas within the first segment of the target surface; and
generating a first paint thickness of paint applied on the first segment of the target surface based on a combination of the paint application rate and the first speed of the first spray nozzle; and
further comprising, annotating the paint map with the first paint thickness.
4. The method of claim 1 :
further comprising:
during a scan cycle preceding the first paint application cycle, accessing a spray pattern of the first spray nozzle;
during the first paint application cycle, predicting a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle; and
accessing a paint application specification defining a target paint thickness for application of paint on the target surface;
wherein extracting the first set of features comprises detecting a first surface discontinuity on the first segment of the target surface within the paint map;
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising a sag defect on the first segment of the target surface:
in response to detecting the first surface discontinuity; and
in response to the first paint thickness exceeding the target paint thickness;
wherein annotating the paint map to indicate the first defect comprises:
annotating the paint map to indicate the sag defect; and
generating a prompt to repair the sag defect on the first segment of the target surface; and
wherein serving the paint map to the operator comprises serving the paint map and the prompt to the operator for inspection of the sag defect.
5. The method of claim 4 :
wherein extracting the first set of features comprises:
isolating a first edge extending toward a lower bound of the first segment of the target surface within the paint map;
isolating a second edge coupled to the first edge and extending toward the lower bound of the first segment of the target surface; and
detecting a first linear surface discontinuity on the first segment of the target surface within the paint map based on the first edge and the second edge; and
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising the sag defect on the first segment of the target surface:
in response to detecting the first linear surface discontinuity; and
in response to the first paint thickness exceeding the target paint thickness.
6. The method of claim 1 :
further comprising:
during the first paint application cycle, detecting a first ambient air temperature proximal the first segment of the target surface; and
accessing a paint application specification defining a target ambient temperature for application of paint;
wherein extracting the first set of features corresponding to the first segment of the target surface from the paint map comprises:
detecting a first reflectivity value of paint applied on the first segment of the target surface within the paint map; and
detecting a first surface discontinuity on the first segment of the target surface based on the first reflectivity value;
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising a dry coverage defect on the first segment of the target surface:
in response to detecting the first surface discontinuity on the first segment of the target surface; and
in response to the first ambient air temperature exceeding the target ambient temperature; and
wherein annotating the paint map to indicate the first defect comprises annotating the paint map to indicate the dry coverage defect.
7. The method of claim 1 :
further comprising during a scan cycle preceding the first paint application cycle:
capturing an initial sequence of images depicting the first segment of the target surface prior to application of paint;
extracting an initial color value of the first segment of the target surface from the initial sequence of images; and
accessing a paint application specification defining a target color value for application of paint on the target surface;
wherein extracting features from the paint map comprises extracting a first color value of paint applied onto the first segment of the target surface from the paint map;
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting an incomplete paint coverage defect on the first segment of the target surface based on the first color value deviating from the target color value toward the initial color value; and
wherein annotating the paint map to indicate the first defect comprises annotating the paint map to indicate the incomplete coverage defect.
8. The method of claim 1 :
further comprising:
during the first paint application cycle, detecting a first surface temperature of paint applied on the first segment of the target surface; and
accessing a paint application specification defining a target surface temperature for application of paint;
wherein extracting the first set of features corresponding to the first segment of the target surface from the paint map comprises detecting a first circular surface discontinuity on the first segment of the target surface within the paint map; and
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising a paint blister defect on the first segment of the target surface:
in response to detecting the first circular surface discontinuity; and
in response to the first surface temperature exceeding the target surface temperature.
9. The method of claim 8 :
wherein annotating the paint map to indicate the first defect comprises:
annotating the paint map to indicate the paint blister defect; and
generating a prompt to repair the paint blister defect on the first segment of the target surface;
wherein serving the paint map to the operator comprises serving the paint map and the prompt to the operator for repair of the paint blister defect prior to a scheduled inspection period for the first paint application cycle; and
further comprising in response to receiving confirmation of repair of the paint blister defect on the first segment of the target surface from the operator, initiating a third paint application cycle for the first segment of the target surface.
10. The method of claim 1 :
wherein extracting the first set of features corresponding to the first segment of the target surface from the paint map comprises detecting a first surface discontinuity on the first segment of the target surface within the paint map;
further comprising:
retrieving a set of paint defect images from a template defect image database, each paint defect image in the set of defect images labeled with a defect type, a defect score, and a paint application procedure; and
extracting a second set of features from a first paint defect image, in the set of paint defect images, labeled with a run defect type, a run repair procedure, and a first defect score; and
wherein detecting presence of the first defect on the first segment of the target surface comprises in response to identifying the first set of features analogous to the second set of features, detecting presence of the first defect comprising a run defect on the first segment of the target surface.
11. The method of claim 10 :
wherein annotating the paint map to indicate the first defect comprises:
annotating the paint map with the run defect type, the run repair procedure, and the first defect score to indicate the run defect; and
generating a prompt to repair the run defect on the first segment of the target surface, according to the run repair procedure;
wherein serving the paint map to the operator for inspection of the first defect comprises serving the paint map and the prompt to the operator to repair the run defect prior to a scheduled inspection period for the first paint application cycle; and
further comprising in response to receiving confirmation of repair of the run defect on the first segment of the target surface from the operator, initiating a third paint application cycle for the first segment of the target surface.
12. The method of claim 1 :
further comprising, accessing a paint application specification defining a target color value for application of paint on the target surface;
wherein extracting the first set of features corresponding to the first segment of the target surface from the paint map comprises detecting a first color value of paint applied to the first segment of the target surface within the paint map;
further comprising:
retrieving a set of paint defect images from a template defect image database, each paint defect image in the set of defect images labeled with a defect type, a defect score, and a paint application procedure; and
extracting a second set of features from a first paint defect image, in the set of paint defect images, labeled with a rust defect type, a rust repair procedure, and a first defect score;
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising a rust defect on the first segment of the target surface:
in response to identifying the first set of features analogous to the second set of features; and
in response to the first color value falling within a threshold color deviation of the target color value; and
wherein annotating the paint map to indicate the first defect comprises annotating the paint map to indicate the rust defect.
13. The method of claim 1 :
further comprising, during a scan cycle preceding the first paint application cycle, accessing a spray pattern of the first spray nozzle;
further comprising, during the first paint application cycle:
detecting a first ambient air temperature proximal the first segment of the target surface;
detecting a first surface temperature of the first segment of the target surface;
predicting a first paint thickness of paint applied to the first segment of the target surface based on the spray pattern of the first spray nozzle; and
predicting a first paint quality of paint applied onto the first segment of the target surface based on the first paint thickness, the first ambient air temperature, and the first surface temperature;
further comprising accessing a paint application specification defining a target paint quality for the target surface;
wherein extracting the first set of features corresponding to the first segment of the target surface from the paint map comprises detecting a first surface discontinuity on the first segment of the target surface within the paint map;
wherein detecting presence of the first defect on the first segment of the target surface comprises detecting presence of the first defect comprising an incomplete coverage defect on the first segment of the target surface:
in response to detecting the first surface discontinuity; and
in response to the first paint quality falling below the target paint quality; and
wherein annotating the paint map to indicate the first defect comprises annotating the paint map to indicate the incomplete coverage defect.
14. The method of claim 1 :
further comprising during a scan cycle preceding the first paint application cycle:
accessing a paint application specification defining a target standoff distance between a column of spray nozzles comprising the first nozzle and the target surface for application of paint on the target surface; and
defining a target flow rate of paint through the column of nozzles based on the target standoff distance;
wherein autonomously navigating the spray system along the toolpath to apply paint across the first segment of the target surface comprises navigating the spray system along the toolpath to apply paint across the first segment of the target surface via the column of spray nozzles comprising the first spray nozzle;
further comprising, during the first paint application cycle, accessing the toolpath comprising a boustrophedonic raster pattern; and
wherein navigating the spray system along the toolpath comprises navigating the spray system along the boustrophedonic raster pattern to apply paint from the set of spray nozzles across the target surface.
15. The method of claim 1 :
further comprising:
accessing a three-dimensional model of a building comprising the target surface;
compiling the first sequence of images into a first post-paint application composite image representing paint applied to the first segment of the target surface; and
compiling the second sequence of images into a second post-paint application composite image representing paint applied to the second segment of the target surface;
wherein assembling the first sequence of images into the paint map comprises:
projecting the pre-paint application composite image of the target surface onto the three-dimensional model of the building to generate a first post-paint application layer of paint applied to the first segment of the target surface; and
projecting the post-paint composite image of the target surface onto the three-dimensional model of the building to generate a second post-paint application layer of paint applied to the second segment of the target surface;
wherein annotating the paint map to indicate the first defect comprises flagging the first post-paint application layer of paint within the three-dimensional model of the building as the first defect; and
wherein serving the paint map to the operator for inspection of the first defect comprises serving the three-dimensional model of the building to the operator for virtual inspection of the first defect.
16. A method for applying paint to a target surface comprising:
capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent a spray nozzle; and
extracting an initial color value of the target surface from the initial sequence of images;
during a first paint application cycle:
navigating the spray nozzle along a toolpath to apply paint on the first segment of the target surface; and
capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor adjacent the spray nozzle;
during a second paint application cycle:
autonomously navigating the spray system along the toolpath to apply paint across a second segment of the target surface via the spray nozzle; and
capturing a second sequence of images depicting paint applied on the second segment of the target surface via the optical sensor adjacent the first spray nozzle;
assembling the first sequence of images into a paint map representing application of paint on the first segment of the target surface;
accessing a paint application specification defining a target color value for application of paint on the target surface;
extracting a first color value of paint applied onto the first segment of the target surface from the paint map;
detecting an incomplete paint coverage defect on the first segment of the target surface based on the first color value deviating from the target color value toward the initial color value; and
annotating the paint map to indicate the incomplete coverage defect.
17. The method of claim 16 :
further comprising:
assembling the second sequence of images into the paint map representing application of paint on the first segment and the second segment of the target surface; and
extracting a second color value of paint applied onto the second segment of the target surface from the paint map;
in response to the second color value corresponding to the target color value, detecting absence of the incomplete paint coverage defect on the second segment of the target surface; and
wherein annotating the paint map to indicate the incomplete coverage defect comprises:
annotating the paint map to indicate presence of the incomplete coverage defect on the first segment of the target surface; and
annotating the paint map to indicate absence of the incomplete coverage defect on the second segment of the target surface.
18. The method of claim 16 :
wherein annotating the paint map to indicate the incomplete coverage defect comprises:
flagging the first segment of the target surface within the paint map; and
generating a first prompt to inspect the incomplete coverage defect on the first segment of the target surface; and
further comprising serving the first prompt and the paint map to an operator for inspection of the incomplete coverage defect.
19. A method for applying paint to a target surface comprising:
accessing a spray pattern for a spray nozzle mounted to a spray system;
capturing an initial sequence of images depicting a first segment of the target surface prior to application of paint via an optical sensor adjacent the spray nozzle;
during a first paint application cycle:
autonomously navigating the spray system along a toolpath to apply paint across the first segment of the target surface via the spray nozzle;
capturing a first sequence of images depicting paint applied on the first segment of the target surface via the optical sensor; and
predicting a first paint thickness of paint applied on the first segment of the target surface based on the spray pattern of the spray nozzle;
assembling the initial sequence of images and the first sequence of images into a paint map representing application of paint on the first segment of the target surface;
annotating the paint map with a paint thickness gradient based on the first paint thickness;
extracting a first surface discontinuity corresponding to the first segment of the target surface from the paint map;
detecting presence of the first defect on the first segment of the target surface based on the first surface discontinuity; and
annotating the paint map to indicate the first defect on the first segment of the target surface.
20. The method of claim 19 :
wherein assembling the initial sequence of images and the first sequence of images into the paint map comprises:
compiling the initial sequence of images into a pre-paint application composite image within the paint map; and
compiling the first sequence of images into a post-paint application composite image within the paint map;
wherein extracting the first surface discontinuity corresponding to the first segment of the target surface from the paint map comprises extracting the first surface discontinuity corresponding to the first segment of the target surface from the post-paint application composite image of the paint map; and
further comprising:
generating a prompt to inspect the first defect on the first segment of the target surface; and
serving the paint map and the prompt to an operator for inspection of the first defect.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/137,376 US20230372968A1 (en) | 2022-04-20 | 2023-04-20 | System for autonomously applying paint to a target surface |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263333100P | 2022-04-20 | 2022-04-20 | |
US202263333093P | 2022-04-20 | 2022-04-20 | |
US18/137,376 US20230372968A1 (en) | 2022-04-20 | 2023-04-20 | System for autonomously applying paint to a target surface |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230372968A1 true US20230372968A1 (en) | 2023-11-23 |
Family
ID=88420543
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/137,374 Active US12036577B2 (en) | 2022-04-20 | 2023-04-20 | System for autonomously applying paint to a target surface |
US18/137,376 Pending US20230372968A1 (en) | 2022-04-20 | 2023-04-20 | System for autonomously applying paint to a target surface |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/137,374 Active US12036577B2 (en) | 2022-04-20 | 2023-04-20 | System for autonomously applying paint to a target surface |
Country Status (2)
Country | Link |
---|---|
US (2) | US12036577B2 (en) |
WO (1) | WO2023205359A2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20240181636A1 (en) * | 2022-05-27 | 2024-06-06 | GrayMatter Robotics Inc. | System and method for autonomous application of a coating to a workpiece |
Family Cites Families (53)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3248020A (en) | 1964-10-09 | 1966-04-26 | Fischer & Co H G | High pressure source for viscous liquids, especially those containing suspended abrasive particles |
US4334495A (en) | 1978-07-11 | 1982-06-15 | Trw Inc. | Method and apparatus for use in making an object |
US5240745A (en) | 1986-04-01 | 1993-08-31 | Honda Giken Kogyo Kabushiki Kaisha | Method for uniformly painting an object with moving spray guns spaced a constant distance from the surface of the object |
DE3861471D1 (en) | 1987-02-10 | 1991-02-14 | Mazda Motor | COATING PROCESS. |
CA1334565C (en) | 1988-05-26 | 1995-02-28 | Ichirou Ishibashi | Method and apparatus for coating surfaces of a workpiece |
US5368232A (en) | 1992-07-13 | 1994-11-29 | Graco, Inc. | Striping apparatus for vehicle travel surfaces |
EP1070546B1 (en) | 1993-05-25 | 2007-10-31 | Nordson Corporation | Powder coating system |
US5419922A (en) | 1994-03-15 | 1995-05-30 | Bmi, Inc. | Method and apparatus for repairing the refractory lining of a refractory vessel |
PT776253E (en) | 1994-10-04 | 2001-04-30 | Procter & Gamble | ELECTROSTATIC SPRAYING OF PARTICLE MATERIAL |
US5578128A (en) | 1995-02-08 | 1996-11-26 | Sealright, Co., Inc. | Apparatus for applying hot melt glue to round and non-round containers |
EP0880998B1 (en) | 1997-05-30 | 2005-08-03 | Hino Jidosha Kogyo Kabushiki Kaisha | Multi-color small amount painting system |
US6315648B1 (en) | 1998-03-13 | 2001-11-13 | Dana L. Neer | Apparatus for pressure treating a surface |
US6252237B1 (en) * | 1998-07-15 | 2001-06-26 | 3M Innovation Properties Company | Low cost thickness measurement method and apparatus for thin coatings |
US6036123A (en) | 1999-01-29 | 2000-03-14 | West; Richard A. | Apparatus for applying foam material to a substrate |
JP3648134B2 (en) | 1999-07-13 | 2005-05-18 | Abb株式会社 | Automatic painting equipment |
WO2002000357A1 (en) | 2000-06-26 | 2002-01-03 | Abb K.K. | Two-tone coating method |
FR2812566B1 (en) | 2000-08-02 | 2003-02-21 | Sames Sa | DEVICE FOR SUPPLYING POWDER COATING PRODUCT TO A PROJECTOR AND PROJECTION INSTALLATION COMPRISING SUCH A DEVICE |
US6557815B1 (en) | 2000-12-05 | 2003-05-06 | University Of Northern Iowa Research Foundation | Universal mounting bracket for laser targeting and feedback system |
JP2002373843A (en) | 2001-06-14 | 2002-12-26 | Nec Corp | Coating system and method for controlling thickness of coating film |
DE10202553A1 (en) | 2002-01-24 | 2003-08-07 | Burkhard Buestgens | Method of applying paints or varnishes |
US6896192B2 (en) | 2002-01-25 | 2005-05-24 | Laser Touch And Technologies Inc. | Spray can targeting and positioning system |
US6699529B2 (en) | 2002-05-20 | 2004-03-02 | Engelhard Corporation | Method for coating vehicular radiators with ozone depleting slurry catalyst |
DE102004033640B4 (en) | 2003-07-18 | 2012-03-29 | Abb As | Inking system |
US7934467B2 (en) | 2004-02-02 | 2011-05-03 | John Stephen Morton | Cost effective automated preparation and coating methodology for large surfaces |
ATE551045T1 (en) | 2005-04-12 | 2012-04-15 | Hara Technologies Inc O | CONTINUOUS FEED TABLET COATING SYSTEM |
US7909265B2 (en) | 2006-08-24 | 2011-03-22 | Randy L Adams | Herbicide and pesticide carrier |
US20080047591A1 (en) | 2006-08-25 | 2008-02-28 | Seitz David M | Bell cup cleaning system and method |
US20090050712A1 (en) | 2007-08-24 | 2009-02-26 | Innovation Direct Llc | Sprayer connector for a paint container |
TWI503180B (en) | 2008-12-30 | 2015-10-11 | Graco Minnesota Inc | Paint sprayer with hose reel |
US8177144B2 (en) | 2009-03-06 | 2012-05-15 | Wagner Spray Tech Corporation | Container holder in a fluid delivery system |
US8347899B2 (en) | 2009-10-20 | 2013-01-08 | Jensen dustin | Wind turbine fluid application apparatus |
US9095865B2 (en) | 2011-11-16 | 2015-08-04 | Csl Silicones Inc. | Mobile coating system for elastomeric materials |
US8524312B2 (en) | 2011-11-16 | 2013-09-03 | Csl Silicones Inc. | Applicator for spraying elastomeric materials |
US8726833B2 (en) | 2012-03-07 | 2014-05-20 | Adam G. Logan | Painting system having a vehicle with lift structure, table actuator, and spray head |
FR2995227B1 (en) | 2012-09-10 | 2014-10-03 | Sames Technologies | PROJECTION INSTALLATION OF COATING PRODUCT |
US20140220249A1 (en) | 2013-02-07 | 2014-08-07 | Abb Technology Ag | Interlaced painting |
US10130962B2 (en) | 2013-10-10 | 2018-11-20 | Briggs & Stratton Corporation | Wirelessly controlled trigger start and chemical tank change-over for pressure washers |
US10124359B2 (en) | 2015-06-17 | 2018-11-13 | Integrated Construction Enterprises, Inc. | Autonomous painting systems and related methods |
US20200016619A1 (en) * | 2015-06-17 | 2020-01-16 | Revolutionice Inc. | Autonomous drywall installation systems and related methods |
CN108289540B (en) | 2015-11-12 | 2019-06-07 | 花王株式会社 | Membrane formation device |
US20180021800A1 (en) | 2016-07-22 | 2018-01-25 | Briggs & Stratton Corporation | Pressure washer accessory tray |
CN110191762B (en) | 2017-01-17 | 2022-08-16 | 固瑞克明尼苏达有限公司 | System for automatic mobile painting of structures |
WO2018183953A1 (en) * | 2017-03-31 | 2018-10-04 | Canvas Construction, Inc. | Automated drywall cutting and hanging system and method |
US10357793B2 (en) * | 2017-05-05 | 2019-07-23 | John M. Harvison | Autonomous painting robot |
KR101940563B1 (en) * | 2017-05-12 | 2019-01-22 | 주식회사 로보프린트 | Method for processing image, method for automatically printing image and nozzle for automatic printing apparatus |
JP7027773B2 (en) | 2017-09-29 | 2022-03-02 | ブラザー工業株式会社 | Pretreatment device |
US10549303B2 (en) | 2017-11-22 | 2020-02-04 | Cnh Industrial America Llc | Wire harness T-joint mount for an agricultural product sprayer |
US10183542B1 (en) | 2017-12-11 | 2019-01-22 | Cnh Industrial America Llc | Suspension control system providing orientation control for an agricultural machine |
FR3083723B1 (en) | 2018-07-13 | 2021-02-26 | Exel Ind | TURBINE, FLUID SPRAYING DEVICE, ASSOCIATED INSTALLATION AND MANUFACTURING PROCESS |
US11110475B2 (en) * | 2018-12-19 | 2021-09-07 | Foreman Technologies Inc. | Modular paint spraying system |
WO2020198322A1 (en) | 2019-03-25 | 2020-10-01 | Carlisle Fluid Technologies, Inc. | Electrostatic coating system and method |
JP7384115B2 (en) | 2020-06-01 | 2023-11-21 | トヨタ自動車株式会社 | Painting system and method |
US11679898B2 (en) | 2020-06-15 | 2023-06-20 | General Electric Company | Inspection and repair tool |
-
2023
- 2023-04-20 US US18/137,374 patent/US12036577B2/en active Active
- 2023-04-20 WO PCT/US2023/019306 patent/WO2023205359A2/en unknown
- 2023-04-20 US US18/137,376 patent/US20230372968A1/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20240181636A1 (en) * | 2022-05-27 | 2024-06-06 | GrayMatter Robotics Inc. | System and method for autonomous application of a coating to a workpiece |
Also Published As
Publication number | Publication date |
---|---|
US12036577B2 (en) | 2024-07-16 |
US20230372966A1 (en) | 2023-11-23 |
WO2023205359A2 (en) | 2023-10-26 |
WO2023205359A3 (en) | 2023-11-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11525270B2 (en) | Automated drywall planning system and method | |
US10625427B2 (en) | Method for controlling location of end effector of robot using location alignment feedback | |
EP3612906B1 (en) | Method and system for environment map generation and alignment | |
CN100387361C (en) | Method for applying paints and varnishes | |
US11369983B2 (en) | Automaton for treating a surface | |
US12036577B2 (en) | System for autonomously applying paint to a target surface | |
KR20160014585A (en) | A supervised autonomous robotic system for complex surface inspection and processing | |
KR101703177B1 (en) | Apparatus and method for recognizing position of vehicle | |
JP4980606B2 (en) | Mobile automatic monitoring device | |
US9823634B2 (en) | System and method for providing three-dimensional paths | |
US11865852B2 (en) | Robotic livery printing system | |
Chen et al. | Seam tracking of large pipe structures for an agile robotic welding system mounted on scaffold structures | |
CN115829963B (en) | Plasma spraying image data identification analysis method and system | |
Xiao et al. | An automatic calibration algorithm for laser vision sensor in robotic autonomous welding system | |
CN102566552A (en) | Road tunnel intelligent overhaul robot facing Internet of things and 3D GIS | |
US20240202897A1 (en) | System for detecting degradation of paint autonomously applied to a building | |
KR101010781B1 (en) | Noncontact environment measuring apparatus, method, and recording medium having computer program recorded | |
Yu et al. | Multiseam tracking with a portable robotic welding system in unstructured environments | |
JP2006320825A (en) | Self-regulated painting vehicle | |
US20240033920A1 (en) | Apparatus and a Method for Removing Coatings by using Laser | |
US20240161329A1 (en) | Localization systems and methods | |
França et al. | Autonomous Robotic System for Visual Inspection in Electrical Substations and Cable Galleries | |
US11857987B1 (en) | Systems and method for painting using drones | |
Teh et al. | Vision Based Indoor Surveillance Patrol Robot Using Extended Dijkstra Algorithm in Path Planning: Manuscript Received: 18 October 2021, Accepted: 4 November 2021, Published: 15 December 2021 | |
KR20230173337A (en) | Unmanned mobile robot for crack inspection of buildings and crack inspection method using the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: FOREMAN TECHNOLOGIES INC., DBA PAINTJET, TENNESSEE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HEGEMAN, NICK;CHACKO, SONIA;JOSE, TOM;AND OTHERS;SIGNING DATES FROM 20231103 TO 20231106;REEL/FRAME:065470/0290 |