GB2615304A

GB2615304A - Continuous smooth surface segmentation

Info

Publication number: GB2615304A
Application number: GB2201118.3A
Authority: GB
Inventors: Adriano Mar Hernandez Tanis
Original assignee: Dyson Technology Ltd
Current assignee: Dyson Technology Ltd
Priority date: 2022-01-28
Filing date: 2022-01-28
Publication date: 2023-08-09
Also published as: WO2023144535A1; GB202201118D0

Abstract

A method of detecting continuous surfaces of an object to be cleaned comprising receiving point cloud data, applying a surface segmentation algorithm to the point cloud, and identifying surface segments of the object with a curvature below a predetermined threshold. The method may comprise a region-growing algorithm wherein a point is selected as a segment seed, and neighbouring points are added to the segment when they are below predetermined smoothness and curvature thresholds. The process may iterate with each neighbouring point being chosen as the seed. The smoothness threshold may be the angle between the normals of the seed point and the chosen neighbouring point. The curvature threshold may be the local curvature of the point. The method may comprise a RANSAC algorithm wherein points are chosen at random as parameters for a function, and wherein it is determined how many points of the point cloud fit within a predetermined tolerance of the function. The process may be repeated until the function that fits the most points is found. A toolpath may be generated using the detected surface segment.

Description

CONTINUOUS SMOOTH SURFACE SEGMENTATION

TECHNICAL FIELD OF THE INVENTION

The present invention relates to a computer-implemented method of identifying one or more continuous smooth surface segments of an object to be cleaned by a 3D cleaning tool. The invention also covers a related method and an input generation system for a control system for a 3D cleaning tool.

BACKGROUND TO THE INVENTION

3D cleaning tools are devices such as robotic devices which may be placed adjacent to an object to be cleaned, such as an item of furniture, or a fixture of a house such as a staircase or floor, and set to clean the item automatically. In order to clean the object effectively, the 3D cleaning tool must first be able to build an electronic picture of the object to be cleaned, so that it can subsequently generate instructions to control the operation of the device.

In order to ensure that the 3D cleaning tool operates effectively, it is necessary to ensure that it is able to build an accurate, useful electronic picture or digital representation of the object to be cleaned, and then that it is able to generate accurate instructions which control the operation of the device.

The majority of the surfaces of furniture such as tables, chairs, and sofas are in the form of continuous, smooth, surfaces. These may be cleaned e.g. by brushing, sweeping, vacuuming, or wiping. In order to plan such tasks, it is first necessary accurately to identify those surfaces. The present invention relates to processes (and other related aspects) for identifying these surfaces.

SUMMARY OF THE INVENTION

At a high level, the present invention relates to methods for identifying one or more surfaces of an object to be cleaned by applying a surface segmentation algorithm to an electronic representation of an outer surface of the object. According to the invention, the segments are identified using a predetermined curvature threshold.

Specifically, a first aspect of the present invention provides a computer-implemented method of identifying one or more continuous smooth surface segments of an object to be cleaned by a 3D cleaning tool, the computer-implemented method comprising: receiving a 3D electronic representation of an outer surface S of the object to be cleaned, the representation comprising a plurality of points PL, each point representative of a point on the outer surface of the object to be cleaned; applying a surface segmentation algorithm to the electronic representation of the surface 5, thereby identifying one or more continuous surface segments Si of the surface S, wherein: each of the continuous surface segments Si comprises a subset of the plurality of points, and the curvature of each of the continuous surface segments Si is below a predetermined curvature threshold.

The use of a surface segmentation algorithm ensures that an accurate picture of the surfaces of the object to be cleaned can be build, which subsequently enables the 3D cleaning tool to clean the object thoroughly and accurately. Examples of surface segmentation algorithms which may be used in implementations of the present invention are set out and explained later on in this application.

The term "3D cleaning tool" as used in this application should be understood to cover any device which is adapted to clean an outer surface of a 3D object. The 3D cleaning tool preferably comprises an end-effector which is configured to contact the outer surface of the object to be cleaned in order to perform the cleaning task. The end-effector may comprise, for example, a vacuum cleaner head, a wipe, a brush, and the like.

A 3D cleaning tool preferably also comprises a motion system which is configured to position the end-effector in the correct position and/or orientation. The motion system may comprise one or more robotic arms which are connected at an elbow. The robotic arms may be rotatable about one or more axes relative to each other. The end-effector may be rotatable about one or more axes relative to a distal robotic arm. A proximal robotic arm may be rotatable about one or more axes relative to a base portion of the 3D cleaning tool.

The 3D cleaning tool may further comprise a control system, optionally located in the base portion, which may be configured to generate and/or transmit instructions to the motion system, the instructions configured to cause the motion system, and optionally the end-effector to move relative to each other in order to position the end-effector in a desired position and/or orientation. More details relating to the control system will be set out later in this application. It should be stressed that computer-implemented methods according to the first aspect of the invention may be used in combination with other 3D cleaning tools which do not take the form described in this paragraph, without departing in any way from the scope of the invention.

The term "outer surface" as used in this application refers to any exposed surface which an external 3D cleaning tool is able to access. The outer surface may comprise a plurality of constituent surfaces, some of which may be flat, and some of which may be curved. The outer surface S may comprise a plurality of constituent smooth continuous surfaces, to be identified using the computer-implemented invention of the present aspect of the invention. The object to be cleaned may comprise an item of furniture (e.g. a chair, a table, a sofa), or a fixture of a house (e.g. a floor, a staircase), but the invention is not limited to furniture or fixtures.

The 3D electronic representation may be received from an external image capture device, such as a 2D camera or a 3D camera. Or, the 3D electronic representation may be received from an external computing device which has generated the representation in response to receiving the raw data from the 2D camera or the 3D camera. Alternatively, the 3D electronic representation may be a computer-aided design (CAD) representation, which has already been prepared. Such a CAD representation may be a design based on which the object was manufactured. In preferred implementations, the 3D electronic representation may comprise a 3D point cloud representation. Herein, a "3D point cloud representation" comprises a plurality of points in 3D space, each having a coordinate indicating its position in 3D space. The points correspond to locations on the outer surface of the object to be cleaned.

The invention requires that one or more continuous surface segments S, are identified using the surface segmentation algorithm, but it should be noted that not necessarily all of the points on the outer surface of the object to be cleaned fall within one of the continuous surface segments SI. For example, areas of high curvature such as the crevices between sofa cushions, or the sharp edge of a square cross-section table leg might not be considered to fall within a continuous surface segment S. The treatment of these points on the outer surface S of the object to be cleaned is explained in detail elsewhere in this application. In order to be classified as a continuous surface segment Si, there may be a threshold number of points falling within that surface, for example 100 to 10,000 points, preferably around 3,000. In this way, small surfaces which may either be too small for an end effector to clean, or surfaces which result from anomalous points, will not be included in the continuous surface segments identified using the computer-implemented method of the first aspect of the invention.

The invention also requires that the curvature of each of the continuous surface segments Si is below a predetermined curvature threshold. This may be understood to mean that the local curvature (i.e. the curvature in the neighbourhood of a give point) at each point (e.g. from the point cloud representation) in a given continuous surface segment St is below the predetermined curvature threshold. When discussing the algorithms in more detail, it will become apparent how this may be determined. The term "continuous" within the meaning of the present application should be understood to mean that there are no breaks in the surface segment, neither are there any sharp changes in curvature. Mathematical definitions of "continuous" will be set out later in the application.

One algorithm which may be used to identify the continuous surface segments Si is a region growing algorithm. At a high level, the region growing algorithm works by identifying a seed point, and then adding neighbouring points to the region if they meet smoothness and curvature criteria. Specifically, the region growing algorithm may comprise: identifying a first seed point PI and selecting a set of neighbouring points to the first seed point Pu such that the resulting surface sub-segment, comprising the first seed point PI and its neighbouring points has a curvature below the predetermined curvature threshold; and iteratively repeating the process, with each neighbouring point in the resulting surface sub-segment becoming a seed point. More specifically, the region growing algorithm may include the following steps: (a) selecting a point PI having the lowest curvature CI as a first initial seed point, the point PI having one or more neighbouring points; (b) identifying the subset of neighbouring points which meet: a predetermined smoothness criterion and a predetermined curvature criterion; (c) adding the identified subset of neighbouring points to a cluster of points forming a first continuous surface segment Si; (d) repeating steps (b) and (c) using each of the identified subset of neighbouring points as new seed points, until no identified neighbouring points meet both the smoothness and curvature criteria.

Here, the smoothness and curvature criteria represent different things. The curvature concerns the local curvature at a point itself, i.e. in the neighbourhood of the point. A neighbouring point may have very low curvature, but if the point is e.g. around the corner of a square table leg, the space between the two points clearly has a sharp change of direction in it. The space between the two points therefore cannot be considered smooth. For the purposes of the present invention it is highly desirable that both smoothness and curvature criteria are met.

Before step (a), the computer-implemented method (preferably the region growing algorithm itself) may include a step of obtaining the curvature of all of the points in the 3D electronic representation of the outer surface S of the object to be cleaned. Obtaining the curvature of a given point may comprise estimating or calculating the normal to the surface at the point. There are many well-established ways to compute normals, including for example those listed on the Point Cloud Library'. The curvature may then be calculated based on at least the estimated or calculated normal to the point. In some cases, it may be possible to calculate the curvature by using the residuals from the eigenvalue local plane fitting problem. This is advantageous since it is already computed when estimating or calculating the normals to the surface.

It should be noted that in step (b) mentioned above, the identified subset may include all of the neighbouring points.

By repeating steps (b) and (c) until no identified neighbouring points meet both the smoothness and curvature criteria, the algorithm identifies all of the points which neighbour each other, and which meet the smoothness and curvature criteria, i.e. one of the continuous surface segments Si. The neighbouring points which do not meet the criteria effectively represent the points immediately surrounding the continuous surface segment SL around its edge. In step (d), it should be understood that only the identified neighbouring points from step (b) are treated as the new seed points: the original seed point is discounted after its neighbours have been identified. The algorithm may further include (preferably on each iteration) a step of adding the identified neighbours which meet the smoothness and curvature criteria to a list, cluster, or set of points which are included in the continuous surface segment SL in question.

https://pcl.readthedocsicilen/latest/normal_estimation.html In a slightly alternative process, after the point with the minimum curvature value has been selected as the seed point, the process may further comprise determining the angles between the normal of the seed point and the respective normals of each of its neighbouring points. Then, the subset of those points which meet the predetermined smoothness criterion may be added to a current region (i.e. a set of points which form part of the continuous surface segment being considered). Then, the subset of neighbouring points whose curvature meets the predetermined curvature threshold may be determined, and the points of the subset may be added to the set of seed points. Then, the current seed may be removed from the set of seeds. If the set of seeds Is empty, then the algorithm has then found the limits of the current region, and the process returns to the step of determining the point with the lowest curvature (of the remaining points). If not, then, for a next seed point of the set of seed points, the process continues from the step of identifying the subset of neighbouring points of that seed point. The process is repeated until no seed points remain in the set of seed points, at which point the process returns to the step of identifying determining the point with the lowest curvature.

The smoothness criterion may be that the seed point and the neighbouring point are not separated by an edge. An edge may be concave (in which case it may be referred to as a "crevice"), or convex. Essentially, an edge is a sharp line between two surfaces. In preferred cases, an edge may be defined based on the relative angle between the normals of two respective points. If there is relative angle between the normals of two points, it will be appreciated that there is a sharp change in angle of the surface containing those two points. This sharp change in angle indicates that the surface between those two points is not smooth, within the meaning of the present invention. Identifying the neighbouring points which meet the smoothness criterion may comprise: determining the angle 9, between the normal Ni of the first seed point and each of its neighbouring points Pn; selecting the set of neighbouring points for which the angle 6, is below a predetermined threshold 6:1.E. In radians, the angle G:m is preferably between 0.0 and 1.0, preferably around 0.05 radians.

In some cases, the predetermined smoothness threshold is selected based on one or more of: the geometry and/or kinematics of the 3D cleaning tool, or surface-contacting end-effector thereof; the nature of the cleaning task to be performed; and the nature of the outer surface to be cleaned. Herein, "kinematics" refers to the range of movement of the various components of the 3D cleaning tool.

In addition to the smoothness criterion, the curvature criterion should also be met. The curvature criterion may be that the local curvature at the neighbouring point in question is below the predetermined curvature threshold. Herein, the term "local curvature" should be understood to mean the curvature in the neighbourhood of the point. The "neighbourhood" may be defined as the number of points within a predetermined radium, or the N nearest neighbours, where N is a predetermined number. Identifying the points which meet the curvature criterion may comprise: obtaining the local curvature of the identified neighbours and selecting the set of neighbouring points for which the local curvature is below the predetermined curvature threshold.

By "curvature", we refer to any parameter which may be used to parametrize the extent to which the outer surface S is curved a given point. Curvature may be represented as the inverse of the radius of curvature, which is the radius of a sphere which best approximates the shape of the surface S at the point.

In some cases, the predetermined curvature threshold is selected based on one or more of: the geometry and/or kinematics of the 3D cleaning tool, or surface-contacting end-effector thereof; the nature of the cleaning task to be performed; and the nature of the outer surface to be cleaned.

The curvature threshold may be between 0.0 and 2.0, preferably around 0.2.

The step of identifying the neighbouring points which meet the smoothness and curvature criteria is effectively a two-part step and may comprise: identifying the subset of neighbouring points which meet the smoothness criterion; and identifying the subset of that subset of neighbouring points which also meet the curvature criterion. Alternatively, the step may comprise: identifying the subset of neighbouring points which meet the curvature criterion; and identifying the subset of that subset of neighbouring points which also meet the smoothness criterion. The identification in each of these cases may proceed as outlined above. Preferably, the subset which meet the smoothness criterion is identified first, since this requires the calculation of the normals to the neighbouring points, from which the local curvatures can then be derived, in order to identify the points meeting the curvature criterion, however it is equally valid to perform the tasks in either order, since the normals may be calculated upon retrieval of the full model, so will be calculated anyway, and also because the comparison for each point is a commutative check.

The processes explained above relate only to the identification of a first continuous smooth surface Si using the region growing algorithm. Of course, for the majority of objects to be cleaned, there will be a plurality of surfaces which require attention. With that in mind, when it is determined that no neighbouring points of a set of new seed points meet both the predetermined smoothness criterion and the predetermined curvature criterion, the computer-implemented method further comprises: of the remaining points of the plurality points which have not yet been added to a cluster representing a continuous smooth surface Si, selecting a point P2 having the lowest curvature C2 as a second initial seed point; and repeating steps (b) to (d) for the second initial seed point P2, thereby generating a cluster of points forming a second continuous surface segment S2. In other words, after a first cluster of points has been identified as belonging to a first continuous surface segment Si, the remaining point with the lowest curvature is selected, and the region growing process is applied again to detect a second continuous surface segment S9. This process continues until all continuous surface segments have been identified. To generalize, when it is determined that no neighbouring points of a set of most recently identified seed points meet both the predetermined smoothness and curvature criteria, the computer-implemented method may further comprise: of the remaining points of the plurality points which have not yet been added to a cluster representing a continuous smooth surface 5, selecting a point P, having the lowest curvature CI as an ith initial seed point; and repeating steps (b) to (d) for the ith initial seed point P, thereby generating a cluster of points forming a ith continuous surface segment C. The output of the region growing algorithm is thus a plurality of clusters of points, each cluster comprising points (more specifically, comprising data identifying points) which are located within a respective continuous surface segment Si_ As we will explain later in this application, these clusters of points may be used to generate a tool path for the end-effector of the 3D cleaning tool to follow in order to clean the object as required.

The above description relates to the use of a region growing algorithm for identifying the various continuous surface segments S. Alternatively, or additionally, a random sample consensus method algorithm (herein, "RANSAC algorithm") may be used. Generally speaking, a RANSAC algorithm approximates a surface using e.g. parametric model, and then aims to identify all points in the 3D electronic representation which fall within a threshold distance of a surface represented by that parametric model. This process is then repeated until all points (within reason) are accounted for, or preferably performed until a model is identified that fits a threshold number of threshold of points within the point cloud. RANSAC algorithms are advantageous because they are accurate and quick to compute for objects such as tables and chairs which include easily parameterizable surfaces (e.g. planar surfaces). RANSAC is not applicable only to planar surfaces, it may be applied to any easily parameterizable surface, such as a parabolic, cylindrical or spherical surface too, or a portion thereof.

The RANSAC algorithm may be configured to identify a first plurality of parameters of a model which represents a first continuous surface segment Si_ and the computer-implemented method may further comprise identifying a first subset of points which are located within the first continuous segment Si and adding them to a first cluster of points which form the first continuous surface segment Si. In order to do so, the RANSAC algorithm may include the steps of: (a)selecting N points from the plurality of points, wherein N is at least the number of points required to estimate the parameters of a parametric model representing the continuous surface segment 5; (b) determining the values of the parameters for a continuous surface segment S, including the N selected points; (c) determining how many points of the plurality of points fit the model within a predetermined tolerance level; and (d) repeat steps (a) to (c) a predetermined number of times, and identify the set of parameters which maximize the value determined in step (c). The computer-implemented method (preferably the RANSAC algorithm itself) may further include the step of identifying a parametric model which is appropriate to the object to be cleaned. Examples of parametric models which may be used are a plane, which may be represented by the equation: ax + by + cz + d = 0 In which a, b, c, and d are parameters to be estimated. Of course, alternative equations may also be used to represent a plane. As discussed previously, other parametric models which may be used include cylindrical surfaces, spherical surfaces, parabolic surface, paraboloidal surfaces, hyperbolic surfaces, hyperboloidal surfaces, and conical surfaces, the defining equations of which are well-known.

The N points are selected randomly or pseudo-randomly.

Herein, "tolerance" may refer to maximum (e.g. a maximum perpendicular) distance from the surface on which the parametric model is based, e.g. the maximum perpendicular distance from a plane or parametric surface. The tolerance may be between 0.1 and 5 cm, preferably around 0.4cm.

Step (c) defined above requires that the number of points of the plurality of points fit within the predetermined tolerance be determined. This may include the steps of: identifying all of the points which are within the predetermined tolerance level of the surface defined by the parametric model; and counting the number of identified points.

After the set of parameters which maximizes the number of points within a predetermined tolerance of the surface defined by the parametric model has been identified, additional steps may be carried out in order further to improve the identification of the surface of the object to be cleaned. For example, the RANSAC algorithm may further comprise: re-estimating the model parameters using all of the points which are located within the continuous surface segment Si defined by the model having the set of parameters identified in step (d) above, to generate a final set of model parameters. The final set of model parameters so generated are likely to give a more accurate representation of the surface in question, because a larger number of initial points N are used.

After the RANSAC algorithm has been used to identify a first continuous surface segment SI, the computer-implemented method may further comprise: applying the RANSAC algorithm to the remaining points which have not been added to the first cluster of points, to identify a second set of parameters of a model which represents a second continuous surface.52. The identification of the second continuous surface 52 may take place in the same manner as the identification of the first continuous surface segment Si. For conciseness, those steps are not repeated here. In a general case, the computer-implemented method may further comprise: applying the RANSAC algorithm to the remaining points which have not yet been added to a cluster of points, to identify an ith set of parameters of a model which represents an ith continuous surface segment S. This general identification step may proceed in the same manner as before.

The steps set out above relate to a regular RANSAC algorithm. However, in some cases, a connected RANSAC algorithm may be used. This is beneficial when the object (or objects) to be cleaned include two or more non-contiguous surfaces which nevertheless happen to conform to different regions of the same parametric model. The most common example of this would be when the object to be cleaned includes a plurality of noncontiguous coplanar regions. In this case, the RANSAC algorithm might determine that all of those non-contiguous coplanar regions to be part of the same continuous surface segment.5%, whereas in reality, there are actually a plurality of separate continuous surface segments Consider an example in which the object to be cleaned includes a table and four chairs, where each of the four chairs has a generally planar seat. Operating normally, the RANSAC algorithm would most likely conclude that all four of the seat surfaces are part of the same surface (since they likely all fall roughly within the same plane, to within a given tolerance). It may be desirable to avoid this, which can be achieved using a connected RANSAC algorithm. Essentially, a connected RANSAC algorithm identifies only regions within the parametric model in which all of the points are connected. For example, a point will only be considered to form part of the identified region if it is within a predetermined distance of at least one other point in the region.

In implementations employing the connected RANSAC algorithm, the computer-implemented method may include identifying a first subset of points which are located within a predetermined distance (i.e. tolerance) of the surface defined by the parametric model; and identifying one or more connected regions of points within the first subset of points. Identifying one or more connected regions of points within the first subset of points may comprise using a region growing algorithm (independently of a region growing algorithm which may be used to identify the continuous surface segments Si, as described earlier in this application). Specifically, the computer-implemented method may further comprise: selecting a first seed point Pi from the first subset of points; identifying all neighbouring points which are within a predetermined threshold distance of the first seed point; repeating this process with the identified neighbouring points becoming new first seed points, until there are no neighbouring points within the predetermined threshold distance of the most recent set of neighbouring points. At that point, the first seed point and all identified neighbouring points are considered to represent a first continuous surface segment Si, and added to a cluster of points. This process may then be repeated by selecting, from the remaining points in the first subset of points, a second seed point f2, and repeating the above process to identify a second continuous surface segment.52. A third seed point may then be selected, and so on, until all of the points from the first subset of points have been assigned to a cluster representing a respective continuous surface segment Si.

After one or more continuous surface segments SI have been identified from the first subset of points, the computer-implemented method may further comprise: fitting a new parametric model from the remaining points, and determining a second subset of points which are located within a predetermined distance (i.e. tolerance) of the surface defined by the parametric model; and identifying one or more connected regions of points within the second subset of points.

Identifying the one or more connected regions within the second subset of points may proceed as outlined for the first subset of points. Subsequently, or generally, the computer-implemented invention may comprise: identifying, from the remaining points, fitting a new parametric model, determining an ith subset of points which are located within a predetermined distance of a surface defined by the new parametric model; and identifying one or more connected regions of points within the ith subset of points. This process may be repeated until all points in the 3D electronic representation of the object to be cleaned have been assigned to a cluster of points representing a respective continuous surface segment S-In some cases, if the outer surface of the object to be cleaned includes some regions which may be easily represented by a surface defined by a parametric model, and other surfaces for which this is not the case, the computer-implemented method may comprise the application of both a RANSAC algorithm and a region growing algorithm. In some cases, the application of a RANSAC algorithm may comprise the application of both a regular RANSAC algorithm and a connected RANSAC algorithm.

The output of the computer-implemented method is preferably one of more clusters of points, each cluster comprising a respective plurality of points from the 3D electronic representation of the outer surface S of the object to be cleaned. The points in a given cluster correspond to one of the identified continuous surface segments A second aspect of the invention provides a method of identifying one or more continuous surface segments of an object to be cleaned by a 3D cleaning tool, the method comprising: generating an electronic representation of an outer surface S of the object to be cleaned, the representation comprising a plurality of points Pt, each point representative of a point on the outer surface of the object to be cleaned; transmitting the generated electronic representation to a computer or other data processing unit; and performing the computer-implemented method of the first aspect of the invention. In embodiments of the second aspect of the invention, generating the electronic representation may comprise obtaining a CAD representation of the object to be cleaned; or obtaining a 3D representation from a 3D camera or a 2D camera. It goes without saying that the optional features set out previously in respect of the first aspect of the invention apply equally well to the second aspect of the invention, unless they are clearly incompatible.

A third aspect of the present invention provides an input generation system for a control system for a 3D cleaning tool, the system comprising: an input generation module configured to generate a representation of an outer surface S of an object to be cleaned, the representation comprising a plurality of points Pi, each point representative of a point on the outer surface of the object to be cleaned; a transmitter; and a computer or other data processing unit, wherein: the transmitter is configured to transmit the generated electronic representation to a computer or other data processing unit; and the computer or other data processing unit is configured to perform the computer-implemented method of the first aspect of the invention. Again, it goes without saying that the optional features set out previously in respect of the first aspect of the invention apply equally well to the third aspect of the invention, unless they are clearly incompatible.

The computer-implemented method of the first aspect of the invention may further comprise generating a 3D cleaning tool path based on the one or more identified continuous smooth surface segments. Generating a 3D cleaning tool path based on the one or more identified continuous smooth segments may be implemented in the form of a computer-implemented method of generating a 3D tool path defining how a 3D cleaning tool moves across one or more continuous surface segments Si during a cleaning operation, the computer-implemented method comprising: for a first continuous surface segment Si, receiving continuous surface segment data comprising an indication of a first plurality of points in 3D space which are part of the first continuous surface segment SI; generating a representation of the first continuous surface segment Si; and determining a first 3D cleaning tool path which covers the representation of the first continuous surface segment Su the first 3D cleaning tool path comprising a first plurality of path points to be traversed, in operation, by the 3D cleaning tool.

Herein, the term "continuous" surface may mean that all points of the surface are located within a predetermined threshold distance of at least one other point in the surface. All points in a continuous surface may also meet a predetermined smoothness criterion which ensures that there are no sharp edges between two points within a surface. This is explained in more detail elsewhere in this application. All points on a continuous surface may also meet a predetermined curvature criterion which stipulates that the local curvature at a given point must be below a predetermined curvature threshold. Again, this is explained in more detail elsewhere in this application.

The step of determining a first 3D cleaning tool path may comprise or refer to generating a first 3D generating tool path. In some cases, the computer-implemented method may further comprise a step of generating instructions to send to a 3D cleaning tool. Such instructions, when executed by the 3D cleaning tool, preferably cause the 3D cleaning tool, specifically the end effector thereof, to follow the generated or determined first (or other) 3D cleaning tool path across the surface of the object to be cleaned. The 3D cleaning tool path "covers" the representation of the first continuous surface segment. Here, the term "covers" should be understood to mean that all or substantially all of the representation of the continuous surface segment SI, when taking into account the dimensions of the end-effector which is intended to perform the cleaning task. Overlap between strokes of the end-effector may also be borne in mind, as described in more detail later.

The 3D cleaning tool path generated may comprise one or more (most likely a plurality) of strokes, wherein a stroke refers to one movement of the end-effector. In some cases, the path may comprise a plurality of strokes which are straight lines (or substantially straight lines), which may be parallel to each other. Between each of the strokes, the path may comprise a turn, in which the end-effector changes direction in order to place it in the correct position for the subsequent stroke.

The computer-implemented method includes a step of generating a representation of the first continuous surface segment Si. In some cases, the representation of the first continuous surface segment S, may be an isoparametric representation.

Herein, an "isoparametric representation" refers to a representation in which a parametric surface is generated, wherein each point on the parametric surface corresponds to a point on the "real" continuous surface segment Si. In preferred cases, the isoparametric representation is an 2D isoparametric surface representation. This means that every point on the continuous surface segment S, can be defined based on two variables u and v. So, where a point may be represented as e.g. P(x,y,z) in real, or Euclidean space, in the parametric representation, it may be represented in the form S(u,v). Ways in which the isoparametric representation may be generated are explained in more detail later in the application. Herein, "corresponds" should be understood to mean that every point in real, Euclidean space, is mapped onto a point in the parametric space by means of an appropriately selected transformation. By employing a 2D parametric surface representation of the continuous surface segment Si, the task of generating a tool path is simplified.

After generating the isoparametric representation, the computer-implemented method preferably further comprises: determining a path which covers the isoparametric representation of the first continuous surface segment Si in the parametric space; and transforming the path in the isoparametric representation of the first continuous surface segment S-into a path in the original (i.e. "real") first continuous surface segment Si. In the parametric space of the isoparametric representation, the 3D tool cleaning path may include, or be in one of the following forms: a plurality of parallel lines traversing the isoparametric representation of the first continuous surface segment S, from one side to the opposite side, either in the same direction or in a serpentine fashion; a plurality of concentric circles; a plurality of concentric_ shapes, the perimeter of each of which is parallel to the perimeter of the isoparametric representation of the first continuous surface segment S,; and a spiral. Herein, "a serpentine fashion" refers to a path which passes from one side to the other of the isoparametric representation, then changes direction and returns to the other side in the opposite direction. In this way, the cleaning may take place throughout the whole path, without the surface-contacting end-effector having to separate from the surface to return to the other side. In the present context, parallel may also cover substantially parallel. In some cases, the path may comprise a plurality of the above geometries, i.e. a combination of two or more of the geometries outlined above, for example to perform more than one cleaning task, or to account for obstacles in the surface.

The selection of the path geometry is liable to vary, for example based on one or more of the following: the nature of the cleaning task to be performed; the nature of the surface to be cleaned; and the geometry/kinematics of the 3D cleaning tool, or the surface-contacting end-effector thereof. For example, if the object to be cleaned is a table or a sofa, strokes which are straight, parallel, and/or continuous may be used, perhaps going from one extreme of the surface to the other (the isoplanar method described below is useful there).

In other cases, a wiping task might require a circular concentric strokes or corner concentric motions (the isoparametric approach is useful in these cases). Different kinds of cleaning tools might also be chosen for each task.

There are various other parameters which are associated with the tool path which may also be controlled. A non-exhaustive list of these is provided below: - In cases in which a concentric stroke pattern is used, the number of concentric strokes.

- A height offset relative to the surface.

- The number of points on a polynomial spline which is used to fit the path.

- A minimum path length, wherein paths which are shorter than the minimum path length are discarded.

- An extrapolation limit, which defines quantitatively whether a stroke finishes earlier or later than its calculated endpoint.

In other cases, another representation may be used, namely an isoplanar representation, in which the first continuous surface segment St is represented by a mesh comprising a plurality of contiguous geometric shapes; and determining the path which covers the first continuous surface segment Si comprises determining a path which covers the isoplanar representation of the first continuous surface segmentation Si. Preferably, the mesh comprises a plurality of triangles. In contrast to the isoparametric approach, when taking the isoplanar approach, everything takes place in real, Euclidean space -there is no need to transform any coordinates into any parametric shape. This means, for example, that lines which are parallel in the isoplanar representation of the continuous surface segment St will be parallel in the actual continuous surface segment St. More generally, any path geometry which is established based on the isoplanar representation will be reflected accurately in the actual continuous surface segment S, since no transformation of coordinates is required.

Determining the path which covers the isoplanar representation of the first continuous surface segment may comprise: intersecting the isoplanar representation of the first continuous surface segment Si with a plurality of cutting planes, the intersections of the cutting planes and the isoplanar representation of the first continuous surface segment Si forming the path to be traversed by the 3D cleaning tool. Preferably, the path is serpentine, and preferably the cutting planes are parallel to each other. Of course, other geometries such as those described earlier for the isoparametric representation may also be used for the isoplanar representation, with the difference that, in the isoplanar case, there is no need to transform the paths back into the real, Euclidean space.

Regardless of whether the isoparametric, isoplanar, or another representation entirely is used, when determining a path across the representation it is necessary to ensure that all portions of the representation of the continuous surface segment Si and therefore the continuous surface segment Si itself will be covered, or traversed by an active area of the surface-contacting end-effector of the 3D cleaning tool. Herein, the "active area" is the part of the end-effector which actually performs the cleaning function when in contact with the surface, e.g. the aperture of a vacuuming tool, or the wet area of a wiping tool. This may be achieved by a consideration of the overlap between adjacent strokes of the end-effector tool when the 3D cleaning tool path is actually executed by the 3D cleaning tool. In the present context, a "stroke" refers to a path of the end-effector tool, in use.

Each stroke has a given width (i.e. an extent in a direction which is parallel or substantially parallel to the direction of movement of the end-effector), which is predetermined based on the geometry of the end-effector (specifically the active area thereof). The computer-implemented method may further comprise receiving: a predetermined overlap threshold defining a minimum amount of overlap between adjacent parallel strokes of the 3D cleaning tool path to be generated; or a predetermined stroke spacing defining a maximum spacing between adjacent parallel strokes of the 3D cleaning tool path to be generated; and the 3D cleaning tool path is determined such that either: the overlap between adjacent parallel strokes of the 3D cleaning tool path is greater than the predetermined overlap threshold; or a maximum path between adjacent parallel strokes of the 3D cleaning tool is less than the predetermined stroke spacing threshold. More generally, the path is preferably determined such that every point in the first continuous surface segment 5', (or, equivalently, the representation thereof) is covered by the active area of the end-effector of the 3D cleaning tool when the path is executed. There may be some exceptions, in scenarios in which it is physically impossible for the end effector to reach various regions of the continuous surface segment Si. This is discussed below in more detail.

Preferably, the 3D cleaning tool is a robotic tool which has a predetermined kinematic profile. Herein, the term "kinematic profile" is a profile which contains data indicative of the allowable range of movement of the various components of the 3D cleaning tool. It is possible to determine, based on the kinematic profile of the 3D cleaning tool, whether the end-effector is able to adopt a particular position, in a particular orientation. Such a kinematic profile can be used as a filter on data representing points in space, identifying which points can be reached by the end-effector, and which points can't. In this application, we refer to this as a "kinematic filter". This filtering step may be performed either at the outset, before the step of generating the 3D cleaning tool path, or after the path has been generated. Specifically: * The computer-implemented method may further comprise initially filtering the continuous surface segment wherein filtering comprises: determining, indicating the geometry and/or kinematics cleaning tool, whether the 3D cleaning tool or a surface-contacting end-effector thereof, is able to based on of the 3D data, data access each point in the plurality of points; and removing the points which cannot be accessed from the continuous surface segment data, to generate filtered continuous surface segment data; and generating the 3D representation is based on the filtered continuous surface segment data.

* Alternatively, or addiTionally, the computer-implemented method may further comprise filtering the first plurality of path poi=s, wherein filtering comprises: determining, based on data indicating the geometry and kinematics of the 3D cleaning tool, whether the 3D cleaning tool or a surface-contacting end-effector thereof, is able to access each point in the first plurality of path poi=s; and removing the points which cannot be accessed from the continuous surface segment data, to generate a filtered first plurality of path points. In some cases, this may be preferable to the previous approach since fewer points may be considered.

In the above, the data indicating the geometry and/or kinematics of the 3D cleaning tool may be in unified robot description format (URDF).

The above description relates only to the process of determining a first 3D cleaning tool path. However, in reality, a given object to be cleaned may comprise a plurality of continuous surface segments Accordingly, the computer-implemented method may further comprise: for a second continuous surface segment 59, receiving continuous surface segment data comprising an indication of a second plurality of points in 3D space which are part of the second continuous surface segment S2; generating a 3D representation of the second continuous surface segment S?; and determining a second 3D cleaning tool path which covers the 3D representation of the second continuous surface segment 52, the second 3D cleaning tool path comprising a second plurality of path points to be traversed, in operation, by the 3D cleaning tool.

All of the optional features set out above in respect of the determination of the first 3D cleaning tool path apply equally well to the determination of the second 3D cleaning tool path. More generally, the computer-implemented method may comprise: for an ith surface segment Si, receiving continuous surface segment data comprising an indication of an ich-plurality of points in 3D space which are part of the ith continuous surface segment St; generating a 3D representation of the ith continuous surface segment S,; and determining an i 3D cleaning tool path which covers the 3D representation of the ith continuous surface segment Si, the id' 3D cleaning tool path comprising a ith plurality of path points to be traversed, in operation, by the 3D cleaning tool. As before, the optional features set out previously in respect of the determination of the first 3D cleaning tool path apply equally well to the determination of the ith 3D cleaning tool path.

In these cases, in which more than one 3D cleaning tool path is generated, the computer-implemented method may comprise a step of receiving continuous surface segment data which comprises a respective plurality of points in 3D space for each of the continuous surface segments S. The computer-implemented method may further comprise: filtering the second plurality of path points, wherein filtering comprises: determining, based on data indicating the geometry and kinematics of the 3D cleaning tool, whether the 3D cleaning tool or a surface-contacting end-effector thereof, is able to access each point in the second plurality of path points; and removing the points which cannot be accessed from the continuous surface segment data, to generate a filtered second plurality of path points. More generally, the computer-implemented invention may comprise filtering the plurality of path points, wherein filtering comprises: determining, based on data indicating the geometry and kinematics of the 3D cleaning tool, whether the 3D cleaning tool or a surface-contacting end-effector thereof, is able to access each point in the ith plurality of path points; and removing the points which cannot be accessed from the continuous surface segment data, to generate a filtered ith plurality of path points. All of the optional features set out above in respect of filtering the first plurality of path points apply equally well to filtering the second or iui pluralities of path points.

In some cases, after the 3D cleaning tool path has been generated, the invention further comprises generating a set of instructions to be sent to the 3D cleaning tool, the set of instructions based on one or more of the following: the first plurality of path pints, the filtered first plurality of path points, the second plurality of path points, the filtered second plurality of path points, the l plurality of path points, and the filtered ith plurality of path points. The instructions are preferably such that when they are received by the 3D cleaning tool, or, for example, a data processing unit or module thereof, they cause the 3D cleaning tool (specifically, the end-effector thereof) to execute the generated 3D cleaning tool path.

In addition to generating the 3D cleaning tool path, and instructions for the 3D cleaning tool, the invention may further comprise a transmission step, whereby the instructions are sent to the 3D cleaning tool, or a component thereof.

Accordingly, the invention may further include a method of controlling a 3D cleaning tool to clean, using a surface-contacting end-effector, one or more continuous surface segments Si of an object to be cleaned, the end-effector adapted to clean the one or more continuous surface segments Si, the method comprising: performing the computer-implemented method of the first aspect of the invention; generating a set of instructions to be sent to the 3D cleaning tool, the set of instructions based on one or more of the following: the first plurality of path points, the filtered first plurality of path points, the second plurality of path points, the filtered second plurality of path points, the ith plurality of path points, and the filtered ith plurality of path points; and transmitting the generated set of instructions to the 3D cleaning tool, the set of instructions operable to cause the 3D cleaning tool to clean the continuous surface segments Si using the end-effector.

In addition to the input generation system, systems of some aspects of the present invention may further comprise a control system for a 3D cleaning tool, the control system configured to control a surface-contacting end-effector of the 3D cleaning tool to clean one or more continuous surface segments Si of an object to be cleaned, the control system comprising: a computer or other data processing unit configured to perform the computer-implemented method of the first aspect of the invention, and to generate a set of instructions to be sent to the 3D cleaning tool, the set of instructions based on one or more of the following: the first plurality of path points, the filtered first plurality of path points, the second plurality of path points, the filtered second plurality of path points, the ith plurality of path points, and the filtered it' plurality of path points; and a transmitter configured to send the generated set of instructions to the 3D cleaning tool, the set of instructions operable to cause the 3D cleaning tool to clean the one or more continuous surface segments using the end-effector. Generation of the set of instructions may further comprise fitting a spline to the path. A further aspect of the invention may provide a 3D cleaning tool comprising the input generation system and the control system.

In some cases, in addition to identifying continuous surface segments of the outer surface of the object to be cleaned, the computer-implemented method of the first aspect of the invention may further comprise identifying one or more crevices of the object to be cleaned by the 3D cleaning tool.

Specifically, this may be achieved by further including a computer-implemented method of identifying one or more crevices of an object to be cleaned by a 3D cleaning tool: receiving a 3D electronic representation of an outer surface of the object to be cleaned, the representation comprising a plurality of points P1, each point representative of a point on the outer surface of the object; and identifying the subset of points within the plurality of points having concave local curvature which exceeds a predetermined local curvature threshold. The 3D electronic representation may be in the form of a point cloud.

Herein, the term "local curvature" should be understood to mean the curvature in the neighbourhood of the point. The "neighbourhood" may be the region which is delineated by an imaginary line connecting all of the closest neighbours (e.g. neighbours which are located within a given distance, or a predetermined number of nearest neighbours).

The identified subset of points are those where the concave local curvature "exceeds" a predetermined local curvature threshold. In the context of the present application, the term "exceeds" may be interpreted to mean either "is greater than", or "is greater than or equal to". It may be necessary to differentiate between a crevice and an "edge". In the context of the present application, the term "edge" is used to refer to a sharp convex region, such as a ridge between two smooth surfaces. Effectively, an edge is the convex equivalent of a (concave) crevice.

Identifying the subset of points may comprise, for each point within the plurality of points: determining the respective local curvature CL of each point within the plurality of points; comparing the respective local curvature with the predetermined local curvature threshold; and selecting those points for which the respective local curvature exceeds the predetermined local curvature threshold.

By "curvature", we refer to any parameter which may be used to parametrize the extent to which the outer surface S is curved a given point.

In some cases, curvature may be represented as the inverse of the radius of curvature, which is the radius of a sphere which best approximates the shape of the surface S at the point. It should be stressed that if the term "curvature" is interpreted as "radius of curvature", then in order to identify the crevices it is necessary to identify those points for which the radius of curvature is below a threshold value (since a more curved surface has a smaller radius of curvature).

Preferred implementations of the computer-implemented method of the first aspect of the present invention do not use the radius of curvature, however, as we discuss in detail below.

Determining the respective curvature of any given point may comprise estimating or calculating the normal to the surface at the point. The curvature may then be calculated based on at least the estimated or calculated normal to the point. In preferred cases, it may be possible to calculate a measure of the curvature by using the residuals from the eigenvalue local plane fitting problem, which gives a dimensionless curvature value. More detail about this calculation process is set out later in this application. Calculating curvature in this way is advantageous since it may be done at practically no additional computational cost when calculating the normals. The measure of curvature is unsigned when calculated in this manner, and it may therefore be necessary to perform additional steps in order to determine e.g. whether a given point is ridge or a crevice.

Regions having a high curvature may be either crevices or edges. In order to identify crevices, it is necessary to identify only the concave points. Accordingly, the step of identifying may comprise: for each point within the plurality of points, determining whether the respective local curvature is concave. Determining whether the respective local curvature is concave may comprise for each of the k nearest neighbours of the point: determining the average distance between that nearest neighbour, and the tangent plane defined by the normal n, to the point; and wherein a positive determined average distance value indicates that local curvature is a concave local curvature. This is explained diagrammatically later in this application. Put alternatively, the computer-implemented method may comprise: determining whether the average difference between that nearest neighbour and the tangent plane defined by the normal nn to the point is positive or negative, and if the average difference is positive, determining that the local curvature is concave, and if the difference is negative, determining that the local curvature is convex. When it is determined that the local curvature is concave, the computer-implemented method may further comprise: adding the point to the plurality of points which are determined to be in a crevice. When it is determined that the local curvature is convex, the computer-implemented method may comprise discarding the point.

In some cases, the predetermined local curvature threshold is selected based on one or more of: the geometry and/or kinematics of the 3D cleaning tool, or surface-contacting end-effector thereof; the nature of the cleaning task to be performed; and the nature of the outer surface to be cleaned.

Example values for the curvature threshold may range from 0.0 to 0.5 m, preferably around 0.01 m. Herein, distance units are used, as they correspond to an average distance from the tangent plane to a point.

Points having a curvature exceeding the curvature threshold are considered part of a crevice. It should be stressed here that this means that the signed value of the curvature should be higher than this value, for an area of high curvature to be considered a crevice. As discussed, in order to form part of a crevice, the local curvature must be concave, i.e. the signed value of the curvature should be no more than zero in order to meet the requirements of a crevice.

By varying the threshold depending on the factors set out above, it is possible to ensure that the optimal points are selected for e.g. a given end-effector or a given cleaning task.

Aspects of the present invention may further comprise a method of identifying one or more crevices of an object to be cleaned by a 3D cleaning tool, the method comprising: generating an electronic representation of an outer surface 5 of the object to be cleaned, the representation comprising a plurality of points Pi, each point representative of a point on the outer surface of the object to be cleaned; transmitting the generated electronic representation to a computer or other data processing device; and performing the computer-implemented method discussed above.

Some aspects of the present invention provide an input generation system. However, this input generation system may be modified in order to provide an input relating to crevices too. Specifically, the input generation system for a control system for a 3D cleaning tool, may further comprise: an input generation module configured to generate a representation of an outer surface S of an object to be cleaned, the representation comprising a plurality of points 5,, each point representative of a point on the outer surface of the object to be cleaned; a transmitter; and a computer or other data processing unit, wherein: the transmitter is configured to transmit the generated electronic representation to a computer or other data processing unit; and the computer or other data processing unit is configured to perform the computer-implemented method of the first aspect of the invention.

In those cases where the method further comprises the identification of crevices, the generation of a 3D cleaning tool path will also need to take account of the crevices.

Accordingly, in such cases the computer-implemented method of the first aspect of the invention may further comprise determining which points of the outer surface correspond to crevices that may need to be cleaned. This may be achieved by incorporating a computer-implemented method of generating a 3D tool path defining how a 3D cleaning tool moves along one or more crevices during a cleaning operation, the computer-implemented invention comprising: receiving data comprising an indication of a plurality of crevice points, which are points in 3D space which are located within a crevice; determining a path which traverses the one or more crevices by converting the points located in a crevice into a respective plurality of nodes, the nodes defining the 3D cleaning tool path.

The overall output of the method of such a computer-implemented method is a path comprising a plurality of waypoints. Herein, the term "waypoint" should be understood to refer to a point along the path which the end-effector (preferably an active area thereof) passes through, in use. A "node" here refers to a point in 3D space, and may interpreted similarly to, or identically to, "waypoint".

Determining a path may use graphical techniques. For example, in some cases, determining a path may comprise generating a graph based on the plurality of crevice points. Herein, a "graph" is a visualizable representation of the relationship between the locations of points in 3D space. In some cases, an actual visual representation may be generated, and e.g. displayed to a user. Alternatively, the visual representation may not actually be generated. The step of generating a graph may comprise generating a convenient coordinate system for representing the plurality of crevice points. Generating the graph may comprise downsampling the data comprising the indication of the plurality of crevice points at a predetermined downsampling resolution, to generate a plurality of nodes, each node having an associated surface normal and position. In the present context, the "surface normal" represents a vector which is perpendicular to a tangent plane to the surface at the node. Collectively, the normal and position may be referred to as "node attributes". Herein, "downsampling" refers to a process of sampling a subset of the plurality of crevice nodes at a resolution which is less than the original resolution. This necessarily results in fewer nodes. More specifically, Pointcloud downsampling refers in general to the process of reducing the number of points in a pointcloud by selecting or creating a subset of the original set of points while keeping as much information as possible.

The method which may be applied here is often referred to voxel grid downsampling, whereby a 3D mesh grid is created. For each cubic volume in the grid (a voxel), all the enclosed points are averaged, and the output is the resulting average point (also called centroid) . The downsampling resolution represents the size of the edge of each of the voxels, or in other words, the size step at which the grid is created. As in a picture, the larger the voxel size (downsampling resolution) is, the coarser the result will be and the more information will be lost.

One way in which the downsampling process may take place is to superimpose a 3D grid, or lattice onto the plurality of crevice points. Then, for each cell defined by the grid or lattice, which may contain one or more crevice points therewithin, generating a point which represents the average location of all of the crevice points within that cell. In such cases, the size of the cell may be referred to as the downsampling resolution. This may also be referred to as a "voxel size". In preferred cases, the voxel size may be around lcm.

There is a trade-off when considering the downsampling resolution: if the resolution is too coarse, details on the path may be missed and displacement of the downsampling grid may heavily influence the results. However, if the resolution is too fine, the number of nodes and connections may create parallel paths along a single crevice edge, which means that It can take a long time to compute a complete traversal which covers all branches.

Once the plurality of nodes has been generated by downsampling, generating the graph may further comprise generating a plurality of edces between neighbouring nodes. Herein, "neighbouring nodes" may be defined as notes which are located within a first predetermined distance threshold from each other. The first predetermined distance is preferably approximately 1.5 times as large as the downsampling resolution. This ratio has been shown to give rise to a balanced amount of edges between points.

Each edge preferably has a normal difference, and a relative displacement associated therewith. Herein, the "relative displacement" represents the displacement (i.e. the distance and direction) between the two nodes in Euclidean space, and may be represented by the vector resulting from subtracting the position of one node from the position of the other node. In addition to the relative displacement, each edge may have a normal difference associated therewith. This preferably represents or comprises the difference in direction between the normals, which may be expressed as an angle (bearing in mind that u*u= 1/1111/1 cos°, where 0 is the angle between the vectors u and V). Collectively, the relative displacement and the normal difference may be referred to as the "edge attributes" of each edge.

After downsampling, the resulting nodes (or the original crevice points) may be located on the surfaces either side of the true crevice, rather than on the crevice itself. In this case, the normals of those points may not represent the true desired direction of the 3D cleaning tool at that point. In order to address this, the computer-implemented method of the first aspect of the invention may further comprise a step of graph reduction, in which connected nodes are iteratively averaged out, merging those which end up nearby each other. The output of the graph reduction process is a graph skeleton which uniquely represents each of the crevices in the outer surface of the object to be cleaned, and any branches thereof. The aim of graph reduction is to eliminate any small branches of crevices which may result from overly fine sampling, and which do not represent the true geometry of the object to be cleaned.

There are many ways in which graph reduction may be effected, but in one case, graph reduction may comprise: (a) for each node, computing an average position and average normal based on the attributes of that node and all nodes connected to it, and generating a respective average node with those attributes, the generated plurality of average nodes replacing the initial plurality of nodes (here, the averaging step does not remove the node, it simply replaces its attributes with the average values of the connected nodes; this is in contrast to the merging step which follows, in which the number of nodes is reduced); (b) merging pairs of average nodes which are located within a second predetermined distance threshold from each other, to generate a modified plurality of average nodes (each node of the modified plurality of nodes having the average of the attributes of the two nodes which merged to form it; the threshold may be, for example about 1 cm, or preferably, equal to the voxel size or downsampling resolution); and (c) repeating, a predetermined number of times (for example, around 5 times), steps (a) and (b) using the modified plurality of average nodes as a starting point; (d) after the process has been repeated the predetermined number of times, generating a plurality of edges between neighbouring pairs of nodes of the modified plurality of average nodes which are located within a third predetermined distance threshold from each other, each newly-generated edge having associated therewith a normal difference, and relative displacement, thereby generating a reduced graph. In preferred cases, the second predetermined distance threshold may be approximately the same as the downsampling resolution, in order to give rise to an overall output resolution which is the same as the initial one.

By performing this graph reduction process, from a graph which is initially obtained by using a very fine downsampling resolution, we obtain a graph skeleton which represents every crevice branch uniquely, and accurately.

At this point, a reduced graph representing the locations of the crevices in 3D space has been generated. Having generated this graph, it is necessary to determine how the end-effector should traverse this reduced graph. Alternatively, in cases in which neither the graph generation nor graph reduction take place, it TS necessary to determine how the end-effector should traverse the plurality of crevice points, or the plurality of nodes. In other cases, where graph generation takes place, but no graph reduction, it is necessary to determine how the end-effector should traverse the generated graph. Accordingly, determining a path may further comprise applying a complete traversal algorithm to an input, where the input is either: the plurality of crevice points; the plurality of nodes; the generated graph; or the reduced graph.

The output of a complete traversal algorithm is a path which traverses all of the points or nodes of the input, preferably no more than once (though in some cases, this may not be possible). The complete traversal algorithm may comprise identifying and separating the input into one or more connected components. Herein, a "connected component" is a plurality of points, all of which are connected, in e.g. a chain or branching structure. In order to be "connected" to another point or node, there should be an edge between the two, which reflects the fact that the two are separated by no more than a threshold distance, e.g. the third predetermined threshold distance.

The complete traversal algorithm may comprise, for the input, or for each connected component of the input: (a)identifying the nodes which are the furthest away from each other and identifying the shortest path between those nodes, and saving the identified path; and (b) removing the nodes on the identified shortest path; (c) for the remaining nodes after the nodes have been removed in step (b), repeating steps (a) and (b) until all nodes in the input, or connected component of the input have been assigned to a shortest path. Herein, where we refer to "removing" the nodes, this should be understood to mean that the nodes are being removed from consideration. They are not removed from the path or the graph. This is evident to the skilled person.

The steps outlined above relate to a computer-implemented method in which graph generation and reduction are used to obtain a graph skeleton which accurately reflects the branching structure of the crevices of the objected to be cleaned. In an alternative case, the graph method may not be used, and instead, determining a path may comprise: identifying one or more branches within the plurality of crevice points, wherein identifying the one or more branches comprises: (a) selecting a crevice point of the plurality of crevice points, the selected crevice point forming a seed point; (b) identifying all of the crevice points of the plurality of crevice points which neighbour the seed point within a predetermined threshold distance, and selecting these points as new seed points; (c) repeating step (b) for the newly-selected seed points, until there are no crevice points which neighbour newly-generated seed points, at which point the branch is complete; and (d) repeating steps (a) to (c) until all crevice points of the plurality of crevice points have been assigned to a branch; (e) determining a path which traverses all of the identified branches. Determining the path may comprise the use of a complete traversal algorithm as before.

The output of the complete traversal algorithms or other path determination processes is a list of paths, where each path corresponds to an ordered list of paths, such that all of the branches in the crevice area are visited. In some cases, there may be a threshold number of points which a path must comprise in order to be considered a path. Preferably, this threshold number is 5 (i.e. paths with fewer than 5 points are not stored as a path). Since the nodes in the graph correspond to points in a point cloud, i.e. the crevice points, the attributes of those points can be used to find waypoints that can be sent to the 3D cleaning tool to move the crevice tool along those waypoints. Accordingly, the computer-implemented method may further comprise: generating, based on the determined path, a set of instructions to be sent to the 3D cleaning tool. Generating the set of instructions may comprise fitting a spline along the points on the determined path. This will be discussed in more detail later in the application.

In those cases where the input generation system of the present invention is configured also to detect crevices, the invention may further comprise a control system for a 3D cleaning tool, the control system configured to control a surface-contacting end-effector of the 3D cleaning tool to clean one or more crevices of an object to be cleaned, the control system comprising: a computer or other data processing unit configured to perform the computer-implemented method of the first aspect of the invention, and to generate a set of instructions based on the determined path; and a transmitter configured to send the generated set of instructions to the 3D cleaning tool, the set of instructions operable to cause the 3D cleaning tool to clean the one or more crevices using the end-effector according to the determined path.

The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described with reference to the accompanying drawings, in which: - Fig. 1 is a high-level system diagram showing various components associated with the present invention.

- Fig. 2 is a schematic of an input generation system and a control system for a 3D cleaning tool.

-Figs. 3A and 3B are flowcharts illustrating processes which may be performed by the system of Fig. 1.

- Fig. 4 is an illustration of a representation of a chair showing continuous surface segments and crevices therebetween. The representation is shown on a computer screen.

- Fig. 5 is a flowchart showing high-level steps involved in the identification of continuous surface segments.

- Fig. 6 is a flowchart illustrating the steps of an exemplary RANSA0 algorithm.

- Fig. 7A shows an example of 3D electronic representation of the outer surface of an object to be cleaned, in this case a table and chairs.

- Figs. 75 and 70 illustrate the outcomes of, respectively, a normal RANSAC algorithm and a connected RANSAC algorithm on the representation of Fig. 7A - Fig. 8 is a flowchart illustrating the steps of an exemplary region growing algorithm.

- Fig. 9 is a flowchart illustrating the high-level steps involved in generating a 3D cleaning tool path based on the detected continuous surface segments.

- Fig. 10 is a flowchart illustrating a series of steps which may be used to generate a 3D cleaning tool path based on an isoparametric representation of continuous surface segments of an object to be cleaned.

- Fig. 11 is a schematic illustrating an SOM approach to generating a grid representation.

-Fig. 12 is a schematic illustrating various vectors which may be considered when generating isoparametric representations of continuous surface segments of an object to be cleaned.

- Figs. 13A to 130 show examples of 3D cleaning tool paths which may be generated based on an isoparametric representation of continuous surface segments of an object to be cleaned.

- Fig. 14A is a flowchart illustrating a series of steps which may be used to generate a 3D cleaning tool path based on an isoplanar representation of continuous surface segments of an object to be cleaned.

- Fig. 14B is a schematic illustrating the use of intersecting cutting planes of an isoplanar representation of a continuous surface segment of an object to be cleaned, in order to generate a 3D cleaning tool path.

- Fig. 15A is a flowchart illustrating the high-level steps involved in detecting crevices in a 3D electronic representation of an outer surface of an object to be cleaned.

-Fig. 15B shows an example of an output of the flowchart of Fig. 15A.

- Fig. 16 is a flowchart illustrating the high-level steps involved in generating a 3D cleaning tool path based on the detected crevices.

-Fig. 17 is a flowchart illustrating an exemplary graph generation process.

-Fig. 18 is a flowchart illustrating am exemplary graph reduction process.

- Figs. 19A to 19F show the outputs of various iterations of the graph reduction process of Fig. 18.

- Fig. 20 is a flowchart illustrating an exemplary complete traversal algorithm.

- Figs. 21A to 210 show the outputs of various steps in the flowchart of Fig. 20.

-Fig. 22 is a flowchart illustrating an exemplary instruction generation process.

DETAILED DESCRIPTION OF THE DRAWINGS

Aspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this text are incorporated herein by reference.

At a high level, the present disclosure relates to control of a 3D cleaning tool 102. Specifically, the disclosure relates to methods (including computer-implemented methods) for identifying various specified regions of an object to be cleaned (namely continuous surface segments and crevices), and processing data related to the identified regions in order to generate paths to be followed by a surface-contacting end-effector 110 of the 3D cleaning tool 102, in order to perform some cleaning task, such as vacuuming, brushing, wiping, or the like. Fig. 1 is a system diagram showing an overall system 100 which may be used to perform various aspects of the present invention, as well as other related methods which may or may not fall within the scope of the claims.

System 100 may include a 3D cleaning tool 102, a control system 104, an input generation system 103, a 3D camera 106, and an external computing device 108. The input generation system 105 may be configured to receive inputs from the 3D camera 106 and the external computing device 108, either via a direct (e.g. wired) connection, or, for example, via a network (not shown), which may be a wired or wireless network such as a Wi-Fi network or a cellular network. The 3D cleaning tool comprises an end-effector 110, and may be configured to receive inputs from the control system 104, again, either directly or via a network (also not shown). The network may be the same network as or a different network from the network via which the input generation system 105 is configured to receive input from the 3D camera 106 and/or the external computing device 108. In some cases, the input generation system 105 and the control system 104 may be part of the same component. Alternatively, the input generation system 105 may be part of the control system 104. Alternatively, the two may be connected either directly, or via a network. The connection may take the same form as the connections previously described, and may be the same or different connections and/or networks. All arrangements are covered by the invention, but throughout this application, the input generation system 105 and the control system 104 are shown as separate modules. This should not be regarded as limiting.

For the purposes of this application, the input generation system 105 may contain the modules which are configured to receive the 3D representation of the object to be cleaned from external sources such as the 3D camera 106 and the external computing device 108, and to identify the continuous surface segments and/or the crevices in the outer surface of an object to be cleaned. And, the control system 104 may contain the modules which are configured to generate an appropriate tool path, and/or instructions for the 3D cleaning tool 102, and to transmit those instructions to the 3D cleaning tool 102.

Fig. 2 is a schematic diagram of an input generation system 105 and control system 104. To reiterate, even though these are shown as separate components, they may both be included in a single component, or one may contain the other. The structure of these systems is described first, before discussing the function of these systems in more detail with reference to the other drawings. Input generation system 105 may comprise an interface module 1050 which may be configured to receive inputs from the 3D camera 106 and/or the external computing device 108. Input generation system 105 may further comprise a processor 1052 and a memory 1054, which may include a buffer 1056. Processor 1052 may comprise a surface segmentation module 1058 and a crevice detection module 1060.

The surface segmentation module 1058 may comprise a RANSAC module 1062 and a region growing algorithm (RGA) module 1064. The input generation module may further include a control system interface module 1066. Control system 104 may comprise an input generation system interface module 1068 via which it may be able to communicate with the input generation system 105. Control system 104, like the input generation system 105, may also comprise a processor 1070 and a memory 1072, which may include a buffer 1674. Processor 1070 may comprise a tool path generation (continuous surface segment, CSS) module 1076 and a tool path generation (crevice) module 1078. The tool path generation module (CSS) 1076 may comprise a representation generation module 1080, which may comprise an isoplanar representation generation module 1082 and an isoparametric representation generation module 1084, and a filtering module 1086. The tool path generation module (crevice) 1078 may include a graph generation module 1088, a graph reduction module 1090, and a traversal algorithm module 1092. The control system 104 may further comprise an instruction generation module 1094, and a 3D cleaning tool interface module 1096, via which it may communicate with the 3D cleaning tool 102. It should be noted that in some cases (not shown), the input generation system 105 and/or the control system 104 may form part of the 3D cleaning tool. Alternatively, they may be separate components, such as computing devices e.g. a laptop, desktop PC, tablet, or smartphone.

Figs. 3A and 3B show flowcharts which illustrate high-level methods which may be performed by various aspects of this disclosure. In Fig. 3A, step Sl, performed by the surface segmentation module 1058 of the input generation module 105 is the identification of one or more continuous surface segments, and step 52, performed by the tool path generation module (CBS) 1076 is the generation of a 3D cleaning tool path based on the identified continuous surface segments. Analogously, in Fig. 3B, step 53, performed by the crevice detection module 1060 of the input generation system 105 is the identification of one or more crevices, and step 54, performed by the tool path generation module (crevice) 1078 is the generation of a 3D cleaning tool path based on the identified crevices. Of course, an object to be cleaned, e.g. an item of furniture may well include both continuous surface segments and crevices.

It may be desirable to clean both of these components in the same cleaning operation. Accordingly, various aspects of the disclosure may include the methods of both Fig. 3A and Fig. 3B. In some cases, the methods of each of Figs. 3A and Fig. 3B may take place simultaneously, and in parallel with each other (e.g. independently of each other). Then, in some cases, a further step may be included in which the resulting 3D cleaning tool paths are combined. Alternatively, steps 51 and 53 may be performed first (in either order), and then the outputs of those steps may be sent simultaneously to the control system 104, and then steps 52 and 54 may take place afterwards (again, in either order). In alternative cases, steps 51 and S2 may take place, then steps S3 and 34 may follow. Or, steps 53 and 54 may take place first, followed by steps Si and S2. Essentially, steps Si to 34 may be performed in any order as long as step 51 takes place before 52, and step 53 takes place step 54.

Now, the processes of each of steps Si to 54 will be described in detail with reference to the other drawings.

Firstly, the surface segmentation step Si is described. Fig. 4 Illustrates exactly what is meant by the surface segmentation step. A chair (or, more accurately, a representation of a chair) is shown, having four surfaces, labelled (1), (2), (3) and (4). On receiving a 3D electronic representation of the chair (preferably in the form of a 3D point cloud which sets out the location of various points on the outer surface of such a chair), the purpose of the surface segmentation process shown in Fig. 5 in the form of a flowchart is to identify the four separate continuous smooth surface segments (1) , (2) , (3) , and (4) , and preferably to indicate which of the plurality of points in the 3D electronic representation belong to each respective surface segment. In step 510 of Fig. 5, the 3D electronic representation of the outer surface of the object to be cleaned may be received by the input generation system 105 (e.g. from the 3D camera 106 or the external computing device 108, via the interface module 1050). Implicitly, it will be appreciated that prior to the method of Fig. 5, the 3D camera 106 may obtain a 3D representation of the object to be cleaned. Alternatively, the 3D representation may exist in the form of a CAD representation on the external computing device 108, which simply transmits that representation to the input generation system 105. In step 312, a surface segmentation algorithm may be applied to the electronic representation of the outer surface S of the object to be cleaned, preferably by the surface segmentation module 1058 of processor 1052 of the input generation system 105. The output of the surface segmentation algorithm is preferably an indication of one or more continuous surface segments Si. Two exemplary surface segmentation algorithms are now described.

Fig. 6 is a flowchart illustrating a series of steps of a random sample consensus method (RANSAC) algorithm which may be performed by NSAC module 1062 of the processor 1052, of the input generation system 105. In a first step 5120, N points are selected from the plurality of points, where N is at least the number of points required to estimate the parameters of a parametric model representing the continuous surface segment S. We consider the example where the parametric model is a plane which may be represented by the equation: ax + by + cz + d = 0 In the case of a plane, it is possible to define a plane in terms of three points, so N may be chosen to be 3. In step 5121, the model is fit to the N selected points, in order to find the values of a, b, c, and d. Then, in step 5122, the number of points (inliers) from the set of all of the input points (i.e. the points in the 3D representation) fit that model within a predetermined tolerance is determined. In other words, the number of points which are within a predetermined distance of the plane defined by the model is determined. This process is then repeated X times, where X is predetermined value. This is illustrated by decision step 5123. After steps 5120 to 5122 have been repeated N times, in step 5124, the model parameters which gave rise to the largest result in step 5122 are selected. In step 5125, the model parameters are re-determined based on all of the inliers of the model identified in step 5124, to generate a final continuous surface segment Si. In step S126, it is determined whether there are any other points which are not accounted for. If not, the process ends, otherwise, the inliers are removed from consideration in step 5127, and the process returns to step 5120. This continues until all points in the 3D electronic representation are accounted for, at which point the algorithm is terminated.

In some cases, a connected RANSAC algorithm may be used. This differs from the "normal" RANSAC algorithm in that it includes a further connected components step is performed on all the planar inliers found by RANSAC, and only the biggest connected cluster is retained. This process is explained in more detail earlier in this application.

Figs. 7A to 7C illustrate the effects of the RANSAC and connected RANSAC algorithms. Fig. 7A shows an illustration of a 3D point cloud of a table and chairs. Fig. 7B shows a schematic diagram of a surface segmentation which may be obtained using a RANSAC algorithm, where different types of shading illustrate different identified planar surfaces. From this, it will be noted that the planar surfaces identified from the tabletop and base of the tabletop extend to the middle of the backs of the chairs. Furthermore, the RANSAC algorithm identifies all of the seats of the chairs as belonging to the same surface, as well as a layer in the table legs. Furthermore, the RANSAC algorithm identifies a portion of the bottom right-hand table leg as belonging to the same surfaces as the backs of the chairs on the right-hand side.

Fig. 7C shows the same setup when a connected RANSAC algorithm is used, and shows that each of the seats of the chairs are identified as separate surfaces, and the plane of the table does not intersect the chair backs. Of course, in other cases, the RANSAC algorithm works perfectly well, this example is only illustrative of a situation in which a connected RANSAC algorithm is preferable, i.e. when there are a plurality of non-contiguous but coplanar smooth surfaces within on object or objects to be cleaned.

Fig. 8 is a flowchart illustrating a series of steps of a region growing algorithm (RGA) which may be performed by RGA module 1064 of the processor 1052, of the input generation system 105. A region growing algorithm is effective when segmenting surfaces of items where a planar (or other easily parameterizable) model cannot be used. So, while a RANSAC algorithm would be useful for e.g. tables, counters, shelves, stairs, it is preferable to use an RCA for furniture types such as sofas, chairs, beds, and bathtubs. These lists are, of course, not exhaustive. At a high-level, an RCA works iteratively by adding contiguous points to a surface as long as they are (a) not edges, and (b) have a low relative curvature. The output of the algorithm is preferably a set of clusters, where each cluster contains a set of points that are considered to belong to the same smooth surface. As with the RANSAC algorithm shown in Tic. 6, before the RCA is performed, in step 310 (see Fig. 5) the 3D camera 106 may obtain a 3D representation of the object to be cleaned. Alternatively, the 3D representation may exist in the form of a CAD representation on the external computing device 108, which simply transmits that representation to the input generation system 105.

More specifically, in a first step 3130 the RCA may comprise computing the curvature value Ci of all points in the point cloud. The fastest way to estimate the curvature is to use the residuals from the eigenvalue local plane fitting problem, which is already estimated when estimating the normals of the surface. Herein, curvature may be defined in 3D as the degree to which a surface deviates from a plane. Mathematically, it is measured as the reciprocal of the radius of curvature at the point where it is measured. In the present context, calculating the exact reciprocal of the radius of curvature is unnecessary, and computationally expensive, since the considered surfaces have varying curvatures at each point.

So, instead, the curvature at any point P1 may be computed via proxies that measure the deviation of the points surrounding Pi from the plane given by the normal at that point ni. Normals may be estimated by estimating the normal of a plane tangent to the surface, which takes the form of a least-square plane fitting estimation problem.

On a 3D surface represented by points, the 3D covariance matrix at each point will naturally have the largest eigenvectors in the directions tangent to the surface at that point (since it is where the points are) . therefore, the normal at that point will be the third eigenvector, which is perpendicular (normal) to the previous two. The more curved the surface is, the more distance there will actually be between the neighbouring points considered in the equation and the plane defined by the two first eigenvectors. This is why this normal computation method automatically provides a good proxy for the curvature, but not for its sign.

The nommals and curvatures are therefore computed via the eigenvectors and eigenvalues of a covariance matrix C created from the k nearest neighbours of the query point PI as follows: C =L(Pi FO *(pt_P)T i=1 C-vi = -vi for j E 0,1,21 Here, /3 represents the 30 centroid of the nearest neighbours around the point in question, Aj is the jul eigenvalue and 771 is the jth eigenvector. Based on these results, an approximation to the curvature at each point can easily be obtained as the relationship between the eigenvalues of the covariance matrix, as presented above: curvature, C, = Ao + A1 + A2 This method of calculating the curvature yields a smooth approximation of the curvature with practically no extra computation after normal estimation. This is sufficient for smooth surface estimation, as the smoothness does not depend on whether the surface is concave or convex. We will see later that additional steps are required in order to determine whether the surface is convex or concave at the point, in the context of crevice detection. After the curvature has been estimated for each of the points in the 3D point cloud, in step S132, the points are sorted by their curvature value, and the point with the minimum curvature value is selected as the seed point for the first region. Then, in step 3134, a smoothness check is performed, in which the angle between the normal of the seed point and that of each of its neighbouring points is determined. The neighbouring points for which the angle is below a predetermined smoothness threshold eth are Ao then added to the region. This step ensures that the output surface is smooth, i.e. without sudden, large, abrupt variations in the normals of its points. Then, in step 5136, the neighbouring points whose curvature C is below a predetermined curvature threshold are selected. This step removes edges (i.e. regions with high local curvature) from the set of seeds, preventing the surface from growing past them. In step 5138, the current (i.e. initial) seed is removed from the set of seeds. It is then determined whether the seed set is empty, in decision step S140. If the seed set is empty, this means that the algorithm has found the limits of the current region, so the method returns to step 5132. If not, then the method returns to step 5134 for each seed point, using the new seeds as a starting point. The process continues until all points in the point cloud are accounted for.

It will be noted that in both the RANSAC algorithm and region growing algorithm, the points which are included in a given region, i.e. continuous surface segment S, are selected based on various thresholds and tolerances. These thresholds and tolerances may be varied to suit a particular item of furniture, cleaning task, 3D cleaning tool 102 (or end-effector 110) geometry. The thresholds may be variable, and can be changed by a user, e.g. via a user interface of the input generation system 105 (not shown), or of an external computing device such as external computing device 108 as shown in Fig. 1.

Returning to Fig. 5, at this point, now that step 512 is completed, the process moves to step 514, in which data representing the identified continuous surface segments Si may be transmitted to the control system 104, preferably the tool path generation module (CSS) 1076 thereof. In implementations in which the input generation module 105 and the control system 104 are the same component, or part of the same component, this step 514 may not take place.

At this point, the method moves to step 52 of Fig. 3A, in which a 3D cleaning tool path is generated by the tool path generation module (CSS) 1076 of the processor 1070 of the control system 104. A high-level flowchart illustrating the tool path generation process is set out in Fig. 9. For the avoidance of doubt, it should be understood that the tool path generation process starts after the continuous surface segment data has been received by the control system 104, specifically the tool path generation (CSS) module 1076 thereof. In a first step 520, a representation of a continuous surface segment Si is generated. And, after the representation has been generated in step 522, a 3D cleaning tool path is generated, which covers the representation of the continuous surface segment S,. The generated path may comprise a plurality of path points which are to be traversed by the 3D cleaning tool path, in operation.

We now explain these steps in more detail. The explanation refers to one continuous surface segment Si, but it will be appreciated that the same process is performed for each of the identified continuous surface segments. In step 520, a representation of the surface is generated by the representation generation module 1080. The data which is generated by e.g. the surface segmentation module 1058 of the processor 1052 of the input generation system 105 essentially includes several clusters of points (or data identifying those points), each cluster representing a different, respective continuous surface segment Si. The purpose of generating a representation of the continuous surface segments Si is to generate a model of the surface in 3D space, such that a tool path may be plotted geometrically on that model, rather than the continuous surface segment S-being represented only by a somewhat abstract list of points. This meaning will become clearer when we consider examples of representations which may be generated by the representation generation module 1080. It should be stressed that the representations described below do not represent an exhaustive list, and other processes for generating a model representing the continuous surface segments Si in 3D space such that a 3D cleaning tool path can be generated and/or plotted thereon are also envisaged.

In a first example, the isoparametric representation generation module 1084 generates an isoparametric representation of the continuous surface segment SI. In an isoparametric representation, the continuous surface segment S, is described in a parametric space. In an exemplary case, the isoparametric representation may be in the form of a polynomial parametric surface, such as a non-uniform rational B-spline, or NURBS, which is a well-defined representation of 3D geometry which can accurately describe any shape. Fig. 10 shows an example process by which a representation in the form of a NURBS may be generated by the isoparametric representation generation module 1084. In a first step 5202, the topology of the continuous surface segment Sn may be learned, e.g. using a self-organizing map (SOM) approach. A self-organized map is an unsupervised machine learning technique which may be used to produce a low-dimensional (typically two-dimensional) representation of a higher-dimensional data set while preserving the topological structure of the data. More specifically, a SOM is a network of weights: 1117 = two, Wntl, "itC Rn The weights are topologically laid out as nodes in a lattice grid as shown in Fig. 11. Each weight represents the position of the node in the input (i.e. real, Euclidean) space. The purpose of the SOM is to find the underlying topologies of an input space (in this case, the plurality of points making up the continuous surface segment S) with a reduced set of in nodes. Intuitively (i.e. as a result of the unsupervised learning) the lattice will assume the underlying shape of the continuous surface segment S, while preserving the property that neighbouring points in the continuous surface segment Sn will remain close in the lattice. The result is thus a discrete grid which assumes the shape of the underlying surface segment in an orderly fashion (i.e. it remains a grid). The process via which the SOM is generated may be summarized as follows, with more detail in Kohonen (1990)2: 1. Initialization: random values are selected for each of the initial weight vectors wi C fie.

2. Sampling: draw a sample training input vector x-D = fx0,...,xDI, xi E IV from the input space.

3. Matching: Find the winning neuron (BMU, best matching ii2 unit) WO = arg min llx Wi II that has the weight vector isism closest to the input vector.

4. Updating: apply the update equation: = Igt)G0c; 1(x), -wj) in which y(0 is the learning rate at time step t, and H100-1 G (x; 1(x) , o-) = exp [ is Gaussian neighbourhood function 20-2 centred at the current BMU.

5. Continuation: increment t t 1 and go to step 2 until t is greater than a threshold or the weights stcp changing.

After the grid representation has been generated in step 3202, in step 3204, a NURBS is fit to the grid representation, preferably by a maximum likelihood method, as described in the NURBS book?.

The output of this process is preferably a surface given by: S(u, v; 0) 0 = arg mjn 'Mu, 12) -w112 uym T. Kohonen. 'The seif-organizing map," in Proceedings of the IEEE, vol. 78. no. 9, pp. 14641480, Sept. 1990, dui: 0.1 09/5.58325.

3 The NURBS Book (2' edition) by Les Piegl & Wayne Tiller -pages 410 to 413 on "Least Squares Curve Approximation" The surface S(17,v) is a parametric surface representation of the continuous surface segment, and one may trace paths in that space 0 d u,v 1. In other words, once the (in this case, but not necessarily) two-dimensional parametric representation has been generated, a 3D cleaning tool path may be generated in that parametric space, in step 3206.

Examples of 3D cleaning tool paths which may be generated from isoparametric representations which may be generated in the manner outlined above are shown in Figs. 13R to 135, but before that, we discuss briefly how an arbitrary path in real space may be converted into coordinates in the parametric space with reference to Fig. 12. Specifically, it is necessary to consider how a displacement Al in real space may be mapped to a displacement Av in the parametric space. In this example, it will be noted that the displacement Al only corresponds to a shift in the v direction in the parametric space, but it will be appreciated that the steps below may straightforwardly be generalized to calculate the shift in the U direction too, if necessary.

Various vectors are shown in Fig. 12: * B represents the direcTion of the displacement Al in the real space. as as * N au Eip as as represents the normal to the plane at a point au Eh) (.4., V) . * E=-AV, representing the differential displacement vector in the metric space as At' is changed.

* T=-ds/ 11111 representing the unit vector in metric space au au ' representing the direcTion in which one would travel as U is changed by an amount AU.

Also, we may define R= 11/ though this is not shown in av Fig. 12. Here R represents the unit vector in metric space representing the direction in which one would travel as 12 is changed by an amount Av.

Using differential geometry and linear algebra, it is therefore possible to obtain a linear approximation to convert Al to the corresponding displacement in parametric space, Av. This way, it is possible to map the desired displacement in metric space Al which is the spacing between strokes (dependent on tool geometry, etc.) to At' corresponding to this displacement, since we are tracing the path in metric space. Based on these, it is possible to calculate At' as follows: Al AI Av -as (N x nT * R x Here, the nglbrackets in the denominator represent an inner product operation.

Fig. 13A shows an example in which the surface is to be cleaned using parallel strokes in parametric space. This may be achieved as follows: * Input: NURBS, or any other, parametric surface representing the continuous surface segment S1 to be cleaned.

* Let Au be a chosen constant increment in parametric space for the parameter u in parametric space (e.g. Au -0.1).

* Let it = 0, and V -0.

* Let Al be a chosen constant spacing between the parallel strokes in real space, e.g. 0.03m. As discussed elsewhere in the application, this parameter may be dependent on cleaning tool geometry and the desired degree of overlap between adjacent cleaning strokes.

* Let C = fro, ... , initially empty, be a list of cleaning strokes, where each stroke T = [pp, ...13m) is given by a list of waypoints.

While u < 1, do: 1. Initialise T-00 to be an empty list of waypoints (empty stroke).

2. While V < 1, do: a. Query waypoint at (u,v) position p*-S(u,v) b. Append waypoint to stroke T C. Move forward on the stroke: I. Compute the normal of S(u,v) which is the normalised cross product of the partial derivatives with respect to U and v (see above).

II. Define T and R as above.

iii. Calculate Av as outlined above.

iv. Update parametric coordinate v as vv-Pav.

3. Append generated stroke 7-to the list of cleaning strokes C. 4. Update parametric coordinate u as u<-u-Ham..

Performing this process until u = 1 will yield a list of parallel strokes, approximately spaced by Al.

Fig. 13B shows an alternative example, in which circular strokes are generated. Below, we set out an equivalent method for generating a path, in which the coordinates u and v are mapped to polar coordinates a and r.

* Input: NURBS, or any other, parametric surface representing the continuous surface segment S, to be cleaned.

r cos a)+1 (-' sin a)+1) * Let f(a,r) - 2 2 define the polar mapping from (a,r) -> (u,v), for a point (u,v) lying on a circle with radius r in parametric space at cc radians.

* Let C = fro, ..., TO, initially empty, be a list of cleaning strokes, where each stroke T = [pc, ..., pm) is given by a list of waypoints.

* Initialize a = 0, r = r,nin, where rutin may be e.g. 0.05 (i.e. the radius of the circle must be at least 5cm.

* Given a choice of updates Aa,Ar, the algorithm then proceeds as set out below.

While r 1 do: 1. Initialise T 0 to be an empty list of waypoints (empty stroke).

2. While a < 27r, do: a. Let (u, = f (a, r) b. Query waypoint at (u,v) position p e-S(u, v) c. Append waypoint to stroke T d. Move forward on circle: i. Update angular coordinate a <-a + Aa 3. Append generated stroke T to the list of cleaning strokes C. 4. Update parametric coordinate u as r<-r-Ar.

Performing this process until r= 1 will yield a list of concentric circular strokes, approximately spaced by Ar.

Fig. 13C shows a third example in which a series of strokes which are parallel to the edce of the surface are generated.

* Input: NURBS, or any other, parametric surface representing the continuous surface segment Si to be cleaned.

* Let g(t,r) = (rop(t-1.5),r(p(t-2.5)), where 0 =Inax(0Juin(14.5 -IxD define a mapping from (t,r)-(u,v), for a point (u,v) lying on a square with side length r in parametric space at t.

* Let C={2-0,...,/"), initially empty, be a list of cleaning strokes, where each stroke T=trlo,...,pm) is given by a list of waypoints.

* Initializc t = 0, r=rmtn, whcrc tint,. may bc c.g. 0.05 (i.c.

the side length of the square must be at least 5cm.

* Given a choice of updates At,Ar, the algorithm then proceeds as set out below.

While r<1 do: 5. Initialise 2--0 0 to be an empty list of waypoints (empty stroke).

6. While t.S; 4, do: a. Let (u) = g(t,r) b. Query waypoint at (u,v) position p S(u,v) c. Append waypoint tc stroke T d. Move forward on the square: a. Update angular coordinate t<-t+At 7. Append generated stroke T to the list of cleaning strokes C. 8. Update parametric coordinate u as r*r-Ar.

Performing this process until r= 1 will yield a list of corner concentric strokes, approximately spaced by Ar.

After the 3D cleaning tool path has been generated, for example using the above methods, in step 5206 of Fig. 10, the generated 3D leaning tool path (preferably in the form of a list of waypoints) may optionally then be transmitted to the filtering module 1086, step 8208. The operation of the filtering module 1086 will be described later, after an explanation of the generation of a 3D tool cleaning path using the isoplanar representation.

We now describe the processes involved in generating an isoplanar representation of a continuous surface segment Si.

At a high level, the isoplanar method generates 3D cleaning tool paths by intersecting a plane with a given mesh model generated from an input continuous surface segment Si. This process may be performed by the isoplanar representation generation module 1082 of the representation generation module 1080 of the tool path generation module (CSS) 1076 of the processor 1070 of the control system 104. The steps are illustrated in Fig. 14A. In a first step 5302, a triangular mesh is generated, and in step 8304, a 3D tool path is generated based on the triangular mesh.

There are many well-known methods of generating a triangular mesh representation of a surface from a plurality of points. One example method is the use of a ball-pivoting algorithm (BPA). The principle of the EPA is very simple: Three points form a triangle if a ball of a user-specified radius p touches them without containing any other point. Starting with a seed triangle, the ball pivots around an edge (i.e., it revolves around the edge while keeping in contact with the edge's endpoints) until it touches another point, forming another triangle. The process continues until all reachable edges have been tried, and then starts from another seed triangle, until all points have been considered. The process can then be repeated with a ball of larger radius to handle uneven sampling densities. A detailed explanation of the ball-pivoting algorithm can be found in Bernardini et al. (1999)4.

In step S304, the 3D cleaning tool path is generated. An example algorithm by which the path may generated is as follows. A schematic diagram showing the plurality of cutting planes is provided in Fig. 14B * Input: triangular mesh model representing the surface to be cleaned.

* Let xmin,xm" be the maximum and minimum limits of the surface.

* Let PESE(3) be the position and orientation of the 3D cleaning tool 102 (specifically, the end-effector 110 thereof) with respect to a fixed coordinate frame (the continuous surface segment Si to be cleaned), representing by a 3 x 4 matrix, P= [nx,ny,nz,ti where all elements are column vectors of three dimensions.

* Based on the current orientation of the robot contained in P, the equation of a cutting plane is defined as h(x,p,n)=(p-x)Tn, where a basic choice for n can be either of the horizontal vectors given by the 3D cleaning tool's pose nx,ny. This choice may be dependent on cleaning task, and desired direction of cleaning strokes as well as the choice of reference frame. Setting x(-xmin, defines the plane at the starting edge of the surface, which can then gradually be moved towards xm".

F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva and G. Taubin, "The ball-pivoting algorithm for surface reconstruction," in IEEE Transactions on Visualization and Computer Graphics, vol. 5, no. 4, Pp. 349-359, Oct-Dec. 1999, doi: 10.1 I 09/2945.8 I 7351.

* Let C= fro, ..., TO, initially empty, be a list of cleaning strokes, where each stroke T = fp°, ...,pm) is given by a list of waypoints.

* Let Al be a chosen constant spacing between parallel strokes, e.g. 0.05m. This parameter, as discussed elsewhere, may be dependent on e.g. the geometry of the 3D cleaning tool, and the desired degree of overlap between adjacent cleaning strokes.

The algorithm may then proceed as follows: 1. Let)cx"ii," 2. While x x,n", do: a. Let I plane Intersect(ne sh, h(x, p, n)) be a stroke generated from intersecting the mesh model with the plane equation given by h(x,p,n).

b. Move the plane forward: i. Update the plane position x(-x+,6./ c. Append generated stroke I to the list C. The final output of the algorithm may then be a list of parallel strokes C="-0,...,1-0 which are spaced by approximately Al.

After the 3D cleaning tool path has been generated, for example using the above methods, in step 5304 of Fig. 10, the generated 3D cleaning tool path may optionally then be transmitted to the filtering module 1086, in step 5306.

The filtering tool may be configured to determine based on data indicating the geometry and kinematics of the 3D cleaning tool 102, e.g. in the form of unified robot description format data, whether the end-effector 110 of the 3D cleaning tool 102 is able to access all of the generated waypoints of each 3D cleaning tool path. If not, the inaccessible points are removed from the plurality of waypoints. In some cases, if it is determined that too many of the waypoints are inaccessible, e.g. by comparison to a threshold, then the 3D cleaning tool path may be rejected entirely. The threshold may take the form of a simple number of waypoints, or may be in terms of an accessible area of the continuous surface segment Si in question. The output of the filtering module 1086 may be a filtered plurality of points forming the waypoints of the generated 3D cleaning tool path, and may be transmitted from the control system 104 to the 3D cleaning tool 102, for execution. In some cases, the instruction generating module 1094 may generate some executable instructions based on the filtered plurality of points, and these instructions may be sent to the 3D cleaning tool 102 for execution.

The above detailed description focuses on the detection of continuous surface segments, and the subsequent generation of appropriate 3D cleaning tool paths. While most furniture cleaning tasks are directed towards the surfaces of items of furniture, some tasks require knowledge of the intersections between those surfaces, referred to in this patent application as "crevices". Crevice detection may be regarded as the complementary problem to the detection of smooth continuous surfaces. While smooth surfaces are sets of contiguous points with low local curvature, crevices can be characterized as the sets of contiguous points with a high, concave local curvature.

We therefore now consider the detection of crevices, and the subsequent generation of 3D cleaning tool paths for cleaning those crevices. Crevice detection, i.e. step 53 of Fig. 3B, may be performed by the crevice detection module 1060 of the processor 1052 of the input generation system 105. Then, in step S4 of Fig. 3E, tool path generation based on the detected crevices, may be performed by the tool path generation module (crevice) 1078 of the processor 1070 of the control system 104. We will consider steps 53 and 34 in more detail, in turn.

Referring once again to Fig. 4, the crevices are the narrow trench regions between e.g. surfaces (1) and (2), (1) and (3), (1) and (4), (2) and (3), and (2) and (4). On receiving a 3D electronic representation of the chair (preferably in the form of a 3D point cloud which sets out the location of various points on the outer surface of such a chair), the purpose of the crevice detection process shown in Fig. 15 in the form of a flowchart is to identify the crevices described above, and preferably to indicate which of the plurality of points in the 3D electronic representation are located in each respective crevice. In step 530 of Fig. 15, the 3D electronic representation of the outer surface of the object to be cleaned may be received by the input generation system 105 (e.g. from the 3D camera 106 or the external computing device 108, via the interface module 1050). Implicitly, it will be appreciated that prior to the method of Fig. 5, the 3D camera 106 may obtain a 3D representation of the object to be cleaned. Alternatively, the 3D representation may exist in the form of a CAD representation on the external computing device 108, which simply transmits that representation to the input generation system 105.

After the 3D electronic representation has been received in step 530, the curvature of each of the points in the 3D representation is determined in step 532. This may be done by considering the eigenvalues of the covariance matrix, as explained earlier in this application. For conciseness, the explanation will not be repeated here. Next, in step 534, those points having a sufficiently high curvature are identified, e.g. by comparison with a threshold. Then, in step 536, the concave points are identified. In some cases, the order of steps 534 and 536 may be reversed. The eigenvalue method of estimating curvature does not yield any information as to whether the surface is concave or convex, and accordingly, an additional step may be required. In order to determine whether the point is concave or convex, at each point P, with normal n,, the average distance from its k neighbours to the tangent plane perpendicular to nl at the point is obtained. If this distance is positive, it means that the neighbouring points are above the plane, and thus the curvature at the point P, is concave. And, if the distance is negative, it means that the neighbouring points are below the plane, and thus the curvature at Pi is convex. In some cases, based on the result of the determination, a sign can be added to the curvature values in order to designate concavity or convexity, e.g. + for concave, or -for convex (or vice versa). Then, from that list of signed curvatures, the concave points which have a curvature greater than the predetermined local curvature threshold may be selected.

Now that steps 534 and 336 have been completed, the process moves to step 338, in which data representing the identified crevices may be transmitted to the control system 104, preferably the tool path generation module (crevice) 1078 thereof. In implementations in which the input generation module 105 and the control system 104 are the same component, or part of the same component, this step 538 may not take place. An example of an output of the crevice detection process is shown in Fig. 153 in which the light regions represent the detected crevices of an armchair.

At this point, the method moves to step 84 of Fig. 3B, namely the generation of a 3D cleaning tool path based on the detected crevices. This step is performed by the tool path generation module (crevice) 1078, which may comprise: a graph generation module 1088, a graph reduction module 1090, and a traversal algorithm module 1092. As is indicated from the names of the modules, and shown in Fig. 16, the tool path generation process may include, in step 340 generation of a graph by the graph generation module 1088, in step 542 reduction of the graph by the graph reduction module 1090, and in step 344 applying a complete traversal algorithm to the reduced graph by the traversal algorithm module 1092. The output of this process is, as with the 3D cleaning tool paths generated in respect of the continuous surface segments, a path in the form of e.g. a series of nodes. In a final step 346, the instruction generation module 1094 may, based on the path, generate instructions, perhaps in the form of a plurality of waypoints to be visited by the end-effector 110 of the 3D cleaning tool 102, in use.

The path may include a series of segments, each in turn made up from a six-dimensional vector specifying the position and orientation of the end-effector 110 of the 3D cleaning tool 102, akin to the paths generated in respect of the continuous surface segments, explained in detail above. There are, however, a few differences between crevices and continuous surface segments which affect how the path should be computed. Firstly, crevices include elongated stretches which are long in one dimension (i.e. along the crevice), but short in the other two dimensions (i.e. the boundary surfaces). Secondly, the points in the crevice have, by definition, high concave curvature. And, thirdly, a single detected crevice may comprise a plurality of branches. This means that the techniques which are used to generate the 3D cleaning tool paths for continuous surface segments may not be suitable for generating 3D tool cleaning paths along crevices. At a high-level, the techniques used to generate a 3D tool path covering all of the crevices involves finding a path along the centre of a ridge, converting it into a series of connected nodes, and applying a complete traversal algorithm to generate a multipath message.

Fig. 17 shows a graph generation process, in which a graph is generated from the 3D electronic representation of the outer surface of the object to be cleaned, which may be in the form of a point cloud. Nodes in the graph correspond to points and normals, and the position and normals of the points are stored as node attributes. The edges between the nodes store information on the relationship between the points which they connect. In a first step 5402, the point cloud may be downsampled at a predetermined downsampling resolution, which may be expressed as a voxel size. Every resulting point from the downsampling may then be considered a node of the graph. Then, in step 3404, edges are generated between pairs of nodes which are located within a threshold distance of each other.

Finally, in step 5406, edge attributes are calculated, namely a Euclidean distance between the two nodes, and a normal difference. These terms are defined earlier in this application. The output of the process of Fig. 17 is thus a graph comprising a plurality of nodes (each having associated attributes) and edges connecting various pairs of nodes (each edge also having associated attributes).

When generating the graph, there is a trade-off to be considered when selecting the downsampling resolution: too coarse and details on the path may be missed, and displacements of the voxel grid may heavily influence the results. Too fine, and the number of nodes and connections sometimes creates parallel paths along a single crevice edge, and complete traversal may take too long. Moreover, some of the original points may be located on the limiting surfaces, rather than on the crevice itself, and their normals may therefore not correspond with the desired direction of the crevice cleaning tool. The graph reduction process of step 542 aims to address some of these issues. The process is shown in more detail in Fig. 18. In step 5502, for each pair of calculated nodes, an average node is calculated, which is located halfway along the edge connecting those nodes. Then, in step 3504, pairs of average nodes which are within a predetermined threshold distance (preferably the same as the voxel size or downsampling resolution) of each other are merged with each other. This process acts to reduce the number of nodes. The process shown in Fig. 18 may be repeated a predetermined number of times, in order to obtain a reduced graph. In this way, it is possible to start with a graph generated using a very fine downsampling resolution, and end up with a reduced graph which represents every crevice branch uniquely. Figs. 19A to 197 show an example of a graph being reduced over five iterations. It will be noted that in the final image, the graph has been reduced to a single path covering each crevice branch, with the exception of a few outliers. It has been shown that, with a higher number of iterations, the path is smoother, however, the branches are also shorter, as a consequence of averaging the extreme points in step 5502. There is therefore a trade-off, which may be influenced by many factors. The preferred number of iterations may be 2, 3, 4, 6, 7, 8, 9, or 10.

After the graph reduction, in step 344 of Fig. 16 takes, place: the application of a complete traversal algorithm to the reduced graph generated in step 542. Given the reduced graph (or the original graph, in cases in which no graph reduction step is required, or performed for another reason), the purpose of the complete traversal algorithm is to identify one or more paths along the various branches of the detected crevices so that all of the determined points on the graph are covered or traversed by the end-effector 110 of the 3D cleaning tool 102. It should be stressed that various forms of complete traversal algorithm may be used, and the algorithm discussed now with reference to Fig. 20 is only one preferable example. As discussed, the input to the algorithm is either the generated graph, or the reduced graph. In some cases, the input may be the raw data in the form of the 3D point cloud, or other electronic representation of the points which are located within the crevices of the object to be cleaned. In the following discussion, we refer to it simply as the "Input". Broadly speaking, the algorithm works by identifying the longest shortest path (i.e. the shortest path spanning two extremes of the graph), storing (e.g. to a buffer 1074) that path, and removing those nodes from consideration. The process then repeats itself until all nodes have been covered, or the remaining paths are too short (i.e. below a predetermined length threshold, or number of nodes).

In a first step 5440, a minimum spanning tree is computed, in order to remove cycles from the graph. Intuitively, this just refers to cutting any loops that may exist in the original graph. A cycle in a graph refers to a situation where, starting from node A, and following the edges, it is possible to return to the same node without going twice over any edge.

Removing the cycles means that now, departing from node A, it is only possible to return to node A again by backtracking, and hence going at least twice over some of the edges.

Then, in order to remove small side branches, or other small branches, in step 5442, the tree is pruned. This may include removing all points from the graph which are located in branches which are below a threshold length (either in terms of actual length, or number of nodes). If Fig. 21A shows the original graph, the output of steps 5440 and 5442 are shown in Fig. 215, which shows the pruned minimum spanning tree. The crevice areas on a piece of furniture (or other object to be cleaned) may not be all be connected in one large tree -there may be several disconnected trees. In order to address this, connected components are identified in step 3444, and the identified connected components are separated. In step 5445, one of the connected components may be selected (e.g. the largest, or selection may be random). The next step may be to identify the longest shortest path between all nodes in the selected connected component. In order to do so, in step 5446 the two nodes which are located the furthest away from each other may be identified, and in step 3448, the shortest path (i.e. through the nodes of the graph) between those two points may be identified, a result of which may be shown in Fig. 213.

Then, in step 5450, this path may be added to e.g. a list of paths which may be stored in the buffer 1074, or otherwise stored, at least temporarily. In related step 3452, the nodes on the graph from the identified path may be removed from consideration. In step 3454, it may be determined whether all of the nodes in the connected component (with the possible exception of outliers) have been included in a path. If not, the method may return to step 5446. If so, the method may proceed to step 3436 in which it may be determined whether all of the connected components have been processed. If not, the process may return to step 3445, and a set of paths may be generated for a next connected component. If the result of the determination in step 3456 is positive, a path has been generated which traverses all points in the graph (or raw data), and the algorithm ends.

The result of the algorithm is a list of paths, where each path corresponds to an ordered list of nodes, such that all of the branches corresponding to the crevices of the object to be cleaned are visited. Nodes in the graph correspond (at least approximately) to points in the point cloud, so their attributes (i.e. the node attributes) can be used to find, in step 346 of Fig. 16, waypoints that can be sent to the 3D cleaning tool 102, in order to cause the end-effector 110 thereof (in this case preferably in the form of a crevice cleaning tool) to traverse those waypoints. The following process of generating waypoints may also be applied in the case of the path generated to cover the continuous surface segments, as explained previously.

The steps may be performed by the instruction generation module 1094 of the control system 104, and are illustrated in Fig. 22. In a first step, 3480, the instruction generation module 1094 may receive the paths generated by the tool path generation module (CSS) 1076 or tool path generation module (crevice) 1078. Then, for each path generated in the previous step, the following steps may be performed: in step 3482, a spline may be fitted along the path, using standard techniques. Then, for each point on the path, a frame (i.e. a frame of reference or a coordinate system, or set of axes) may be fitted in step 3484.

x, y, and z may be used to refer to the XYZ vectors which define a frame of reference. A frame of reference defines a 6 degree of freedom "pose" (position and orientation) with respect to any other frame of reference. The end-effector of the 3D cleaning tool has a frame of reference that describes its pose with respect to the reference frame in the base of the robot. The XYZ vectors defined here create a list of frames of reference (one per waypoint) which are the poses the end-effector aims to accomplish in order to complete the cleaning task.

At this point, for a given object to be cleaned, such as the chair in Fig. 4, the continuous surface segments and/or the crevices therebetween may be identified. Then, instructions for the 3D cleaning tool 102 may be generated, in order to enable the end-effector 110 thereof to perform the desired cleaning task.

The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the invention in diverse forms thereof.

While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the invention.

For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations.

Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described.

Throughout this specification, including the claims which follow, unless the context requires otherwise, the word "comprise" and "include", and variations such as "comprises", "comprising", and "including" will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

It must be noted that, as used in the specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value, and/or to "about" another particular value.

When such a range is expressed, another embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent "about," it will be understood that the particular value forms another embodiment. The term "about" in relation to a numerical value is optional and means

for example +/-10%.

Claims

CLAIMS1. A computer-implemented method of identifying one or more continuous smooth surface segments of an object to be cleaned by a 3D cleaning tool, the computer-implemented method comprising: receiving a 3D electronic representation of an outer surface S of the object to be cleaned, the representation comprising a plurality of points Pi, each point representative of a point on the outer surface of the object to be cleaned; applying a surface segmentation algorithm to the electronic representation of the surface S, thereby identifying one or more continuous surface segments Si of the surface S, wherein: each of the continuous surface segments Si comprises a subset of the plurality of points, and the curvature of each of the continuous surface segments Si is below a predetermined curvature threshold.
2. A computer-implemented method according to claim 1, wherein: the surface segmentation algorithm is a region growing algorithm configured to identify the one or more continuous surface segments Si by: identifying a first seed point PI and selecting a set of neighbouring points to the first seed point Pi, such that the resulting surface sub-segment, comprising the first seed point PI and its neighbouring points has a curvature below the predetermined curvature threshold; and iteratively repeating the process, with each neighbouring point in the resulting surface sub-segment becoming a seed point.
3. A computer-implemented method according to claim 2, wherein: the region growing algorithm comprises the steps of: (a) selecting a point P1 having the lowest curvature CI as a first initial seed point, the point PI having one or more neighbouring points; (b) identifying the subset of neighbouring points which meet: a predetermined smoothness criterion and a predetermined curvature criterion; (c) adding the identified subset of neighbouring points to a cluster of points forming a first continuous surface segment (d) repeating steps (b) and (c) using each of the identified subset of neighbouring points as new seed points, until no identified neighbouring points meet both the smoothness and curvature criteria.
4. A computer-implemented method according to claim 3, wherein: the smoothness criterion is that the seed point and the neighbouring point are not separated by an edge.
5. A computer-implemented method according to claim 4, wherein: identifying the neighbouring points which meet the predetermined smoothness criterion comprises: determining the angle en between the normal NI of the first seed point and each of its neighbouring points Pn; selecting the set of neighbouring points for which the angle Or is below a predetermined threshold eaT.
6. A computer-implemented method according to any one of claims 3 to 5, wherein: the curvature criterion is that the local curvature of the neighbouring point is below the predetermined curvature threshold.
7. A computer-implemented method according to any one of claims 3 to 6, wherein: when it is determined that no neighbouring points of a set of new seed points meet both the predetermined smoothness criterion and the predetermined curvature criterion, the computer-implemented method further comprises: of the remaining points of the plurality points which have not yet been added to a cluster representing a continuous smooth surface Sl, selecting a point F2 having the lowest curvature 02 as a second initial seed point; and repeating steps (h) to (d) for the second initial seed point P2, thereby generating a cluster of points forming a second continuous surface segment
8. A computer-implemented method according to any one of claims 3 to 7, wherein: the predetermined smoothness threshold is selected based on one or more of: the geometry and/or kinematics of the 3D cleaning tool, or surface-contacting end effector thereof; the nature of cleaning task to be performed; and the nature of the outer surface to be cleaned.
9. A computer-implemented method according to any one of claims 3 to 8, wherein: the predetermined curvature threshold is selected based on one or more of: the geometry and/or kinematics of the 3D cleaning tool, or a surface-contacting end effector thereof; the nature of cleaning task to be performed; and the nature of the outer surface to be cleaned.
10. A computer-implemented method according to claim 1, wherein: the surface segmentation algorithm is a random sample consensus method (RANSAC) algorithm.
11. A computer-implemented method according to claim 10, wherein: the RANSAC algorithm is configured to identify a first set of parameters of a model which represents a first continuous surface segment Su and either: (a) the computer-implemented method further comprises identifying a first subset of points which are located within the first continuous surface segment Si and adding them to a first cluster of points which form the first continuous surface segment Su or (b) the computer implemented method further comprises: identifying a first subset of points which are located within the first continuous surface segment Si; identifying one or more connected sets of points within the first subset of points, and adding them to a first cluster of points which form the first continuous surface segment SI.
12. A computer-implemented method according to claim 11, wherein: the computer-implemented method further comprises: applying the RANSAC algorithm to the remaining points which have not been added to the first cluster of points, to identify a second set of parameters of a model which represents a second continuous surface segment S9; and either: (a) identifying a second subset of points which are located within the second continuous surface segment S2 and adding them to a second cluster of points which form the second continuous surface segment 52; or (b) identifying a second subset of points which are located within the second continuous surface segment 53; identifying the largest connected set of points within the second subset of points, and adding them to a second cluster of points which form the second continuous surface segment 52.
13. A computer-implemented method according to any one of claims 10 to 12, wherein: the RANSAC algorithm comprises the steps of: (a) selecting N points from the plurality of points, wherein N is at least the number of points required to estimate the parameters of a parametric model representing the continuous surface segment Si; (b) determining the values of the parameters for a continuous surface segment Si including the N selected points; (c) determining how many points of the plurality of points fit the model within a predetermined tolerance level; (d) repeat steps (a) to (c) a predetermined number of times, and identify the set of parameters which maximize the value determined in step (c).
14. A computer-implemented method according to claim 13, wherein: the RANSAC algorithm further comprises re-estimating the model parameters using all of the points which are located within the continuous surface segment Si defined by the model having the set of parameters identified in step (d) to generate a final set of model parameters.
15. A computer-implemented method according to any one of claims 1 to 14, wherein: the computer-implemented method further comprises generating a 3D tool cleaning path based on the one or more identified continuous smooth surface segments.
16. A method of identifying one or more continuous smooth surface segments of an object to be cleaned by a 3D cleaning tool, the method comprising: generating an electronic representation of an outer surface S of the object to be cleaned, the representation comprising a plurality of points Pi, each point representative of a point on the outer surface of the object to be cleaned; transmitting the generated electronic representation to a computer or other data processing unit; and performing the computer-implemented method of any one of claims 1 to 15.
17. A method according to claim 16, wherein: generating the electronic representation comprises one or more of the following: obtaining a CAD representation of the object to be cleaned; Or obtaining a 3D representation from a 3D camera.
18. An input generation system for a control system for a 3D cleaning tool, the system comprising: an input generation module configured to generate a representation of an outer surface S of an object to be cleaned, the representation comprising a plurality of points each point representative of a point on the outer surface of the object to be cleaned; a transmitter; and a computer or other data processing unit, wherein: the transmitter is configured to transmit the generated electronic representation to a computer or other data processing unit; and the computer or other data processing unit is configured to perform the computer-implemented method of any one of claims 1 to 15.