GB2621906A

GB2621906A - A system and method for processing image data

Info

Publication number: GB2621906A
Application number: GB2304764.0A
Authority: GB
Inventors: Edmunds Matt; Morgan Huw
Original assignee: Gpc Systems Ltd
Current assignee: Gpc Systems Ltd
Priority date: 2023-03-30
Filing date: 2023-03-30
Publication date: 2024-02-28
Anticipated expiration: 2043-03-30
Also published as: GB202304764D0; GB2621906B

Abstract

3D scene image comprising data points having 2D coordinates plus depth is captured and displayed, and a user defines a start region of a plane of interest (166). Start region data points are assigned to processing data points, and the plane is approximated by determining a processing data points mean position and fitting a proposed plane through the processing data points using a least-squares fit. The approximation is improved by a) altering the processing points to include 3D image data points which are positioned within a radius to the mean position, b) determining a revised mean position, c) measuring a distance from each of the data points to the proposed plane, d) calculating a current value of standard deviation of the measured distances, e) removing processing data points outside of a multiple of the standard deviation (172), f) refitting the proposed plane (174) through the processing data points using the least-squares fit, and repeating a) to f) until an end condition. The end condition comprises either meeting a maximum number of iterations or the value of the standard deviation and a difference between the current and a preceding value of the standard deviation is lower than a minimum deviation parameter.

Description

A System and Method for Processing Image Data

TECHNICAL FIELD

The present description generally relates to the technical field of image data processing, and it relates in particular, though not specifically, to a system and method of image data processing for determining planes using point cloud data.

BACKGROUND

In transport and logistic industries, freight is constantly moved from warehouses in shipping containers or trucks via land, air, or water. Therefore, it is important that the dimensions and the weight of the freight are accurately determined as this can help maximise the amount of freight that can be loaded into the container or truck by arranging it in the most efficient way for handling and distribution.

With the development of three-dimensional (3D) cameras, depth measurement has played a significant part in accurately measuring the dimensions of the freight, specifically in fitting a plane to a point cloud. However, in order to operate optimally, current systems have a limitation in that they rely on a set of data points representing a scene that has a low level of noise. This often means that the set of data points captured by the 3D camera needs to discretely represent only the collection of points to be fit. However, in reality, most scene data include large quantifies of points which are both noisy and are not part of the surface of interest. This can cause large inaccuracies when using known methods to fit the point cloud data captured by the 3D camera to the plane. Furthermore, solutions that look at entire scenes or encompass areas of a scene that are not the plane of interest, are not only computationally expensive, but given a limited amount of processing power, they inaccurately calculate the location of the plane.

Figure la shows an example of an image captured by a 3D camera showing raw point cloud data. The image is taken of a surface from above and at an angle. There are two planes shown in Figure la, an upper plane 100 and a lower plane 102. The upper plane 100 is plane of interest and a small portion of that plane is indicated by the dark square 103 shown in that plane 100. As can be seen from Figure la, the plane of interest is slightly raised from the lower plane 102, and it is evident that the figure shows some of the noise and inaccuracies obtained by the camera. The plane of interest 100 where the dark square 103 is marked, is in fact flat Figure lb shows an image captured by a 3D camera showing raw point cloud data of the same surface as Figure la but taken from the side of the surface. The upper and lower planes 100 and 102 shown in Figure la, can also be seen as can the dark square 103 in the plane of interest. As can be seen from Figure lb to the left of the dark square 103 on the plane of interest 100, bumps 104 can be seen which appear to suggest the upper surface is undulating and raised. The raised area is, in fact, a flat plane and does not comprise any bumps to the left contrary to the representation in the image of Figure lb. This clearly illustrates noisy and inaccurate data which is obtained from 3D cameras when capturing a scene.

Accordingly, data obtained from existing 3D cameras is often of low quality and i.e., noisy, and current plane fitting methods for identifying the position and orientation of a wall or floor are not accurate or are computationally expensive.

In addition, current solutions are limited to measuring freight in structured environments and require costly hardware for their operation. For instance, systems can only determine the dimensions of regular shaped objects in known positions and thus, cannot cater for irregular shaped objects in environments where geometrical planes are unknown, lighting is limited, or objects are reflective. Moreover, these solutions are typically awkward to use as they must be positioned at the level of the object or overhead.

The present invention has been devised to address, mitigate, or overcome at least some the above-mentioned limitations.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided a computer-implemented method for processing noisy image data to determine a plane of interest within the image data, the method comprising: capturing a three-dimensional (3D) image of a scene, the 3D image comprising a plurality of data points each having two-dimensional coordinates and depth information; displaying the 3D image of the scene on a screen of a portable electronic device; receiving a user input defining a start region, the user input indicating a region of the plane of interest in the scene; assigning data points of the start region to a set of processing data points; determining a first approximation of the plane of interest by: determining a mean position of the set of processing data points; fitting a proposed plane through the set of processing data points using a least-squares fit through the data points; and improving the first approximation of the plane of interest by: a) altering the set of processing points to include data points from the 3D image which are positioned within a radius of the mean position; b) determining a revised mean position of the set of processing data points; c) measuring a distance from each of the data points within the set of processing data points to the proposed plane; d) calculating a current value of standard deviation of the measured distances; e) removing processing data points outside of a multiple of the standard deviation from the set of processing data points; f) refitting the proposed plane through the set of processing data points using the least-squares fit; and repeating steps a) to f) until an end condition is reached, wherein the end condition comprises either the number of iterations equalling a number of maximum iterations parameter or the value of the standard deviation and a difference between the current value and a preceding value of the standard deviation is lower than a minimum deviation parameter.

The method of the present invention differentiates itself from and improves on existing plane determination methods primarily because the method is able to deal with low-quality or noisy data which is generated from 3D cameras or sensors. Such noise may be caused by characteristics of the sensor or camera or in some cases poor image capture for example poor environmental conditions such a poor lighting or image capture which includes many other features which may make the plane of interest more difficult to determine.

One important aspect is the use of a statistical distribution measure of image points to refine each iteration of the method of plane fitting. This enables outliers to be readily removed and prevent them from distorting the results.

Another key advantage of the present invention is that the algorithm does not consume a great amount of processing power. This enables the method to be implemented on low processing power mobile devices, such as smart phones or tablet computers which do not have the same computational processing power as a Personal Computer (PC). Furthermore, advantageously there is no requirement for any data to be uploaded to the cloud for cloud data processing.

As an example, the present invention provides results in less than 1 second even if data is low quality and the device has limited processing capability. An example of the device that has been used to generate this performance is a Samsung Tab® A10.1 with Exynos 7904 chipset and 2GB RAM.

As has been mentioned previously, current algorithms to determine planes are either too computationally expensive (which either requires processing in the cloud or on a PC for example) or such algorithms are inaccurate when presented with low quality point cloud, Figure 9a (described later) provides a good illustrative example of this.

The method described herein, when compared to existing plane fit methods is more accurate. Given the same processing power, conventional algorithms which simply fit a plane to a set of captured points, are on average 25 degrees more inaccurate when compared to a plane calculated from the iterative method of the present invention.

Preferably the end condition comprises either: the number of iterations equalling the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is higher than a minimum deviation parameter; or the number of iterations is less than the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is lower than a minimum deviation parameter.

The method may further comprise setting the maximum number of iterations parameter and a difference parameter specifying the difference between the between the current value and a preceding value of the standard deviation.

The method may further comprise setting a radius parameter specifying the radius to be used in the altering step.

The method may comprise setting a Sigma parameter specifying the multiple of standard deviations to be used in the removing step.

Providing any of these selected parameters advantageously enables the method be adjusted to different types of applications, whilst at the same time placing control of the amount of processing required to generate a result. For example the number of iterations places a limit on the amount of processing that will be applied to get a result in the worst case.

The step of determining a first approximation of the plane of interest preferably comprises: calculating a covariance matrix of the set of processing data points within the start region; applying linear regression on an eigensystem of the covariance matrix; and outputting an eigenvector and the mean of the set of processing data points.

In order to correct for bias or noise generation within sensors or cameras the method preferably further comprises: examining raw output data from a camera and determining noise deviations from a known position of a reference plane; determining an offset value to compensate for the deviations from the known position of the reference plane; and applying the offset, to the proposed plane once the end condition has been reached to bias the proposed plane relative to a distribution of noise.

The method may further comprise outputting the proposed plane as a determined plane once the end condition has been reached.

The present invention may extend to a method of calculating the size of an object within a scene, the method may comprise carrying out the above described method of processing noisy image data to determine a plane of interest within the image data; where the receiving step comprises receiving user input defines a plane of interest which is adjacent the object to be measured within the scene; and using the plane of interest as a reference plane within the scene from which to take measurements of the object.

The ability to measure the size of an object is particularly useful for logistical applications involving freight. Here dimensions of the freight can be accurately determined as this can help maximise the amount of freight that can be loaded into the container or truck by arranging it in the most efficient way for handling and distribution.

According to another aspect of the present invention there is provided a portable electronic device for processing noisy image data to determine a plane of interest within the image data, the device comprising: a sensor or camera for capturing a three-dimensional (3D) image of a scene, the 3D image comprising a plurality of data points each having two-dimensional coordinates and depth information; a display for displaying the 3D image of the scene; input means for receiving a user input defining a start region, the user input indicating a region of the plane of interest in the scene; and processing means configured to: assign data points of the start region to a set of processing data points; determine a first approximation of the plane of interest by: determining a mean position of the set of processing data points; fitting a proposed plane through the set of processing data points using a least-squares fit through the data points; and improve the first approximation of the plane of interest by: a) altering the set of processing points to include data points from the 3D image which are positioned within a radius of the mean position; b) determining a revised mean position of the set of processing data points; c) measuring a distance from each of the data points within the set of processing data points to the proposed plane; d) calculating a current value of standard deviation of the measured distances; e) removing processing data points outside of a multiple of the standard deviation from the set of processing data points; f) refitting the proposed plane through the set of processing data points using the least-squares fit; and repeating steps a) to f) until an end condition is reached, wherein the end condition comprises either the number of iterations equalling a number of maximum iterations parameter or the value of the standard deviation and a difference between the current value and a preceding value of the standard deviation is lower than a minimum deviation parameter.

The portable electronic device may further comprise means for setting the maximum number of iterations parameter and a difference parameter specifying the difference between the between the current value and a preceding value of the standard deviation.

The portable electronic device may further comprise means for setting a radius parameter specifying the radius to be used in the altering step and/or means for setting a Sigma parameter specifying the multiple of standard deviations to be used in the removing step.

In an embodiment, the processing means is configured to determine a first approximation of the plane of interest by: calculating a covariance matrix of the set of processing data points within the start region; applying linear regression on an eigensystem of the covariance matrix; and outputting an eigenvector and the mean of the set of processing data points.

The processing means may be configured to: examine raw output data from the sensor or camera and determining noise deviations from a known position of a reference plane; and determine an offset value to compensate for the deviations from the known position of the reference plane; and apply the offset, to the proposed plane once the end condition has been reached to bias the proposed plane relative to a distribution of noise.

The processor is preferably configured to output the proposed plane as a determined plane once the end condition has been reached.

The portable electronic device may be further configured to calculate the size of an object within a scene, wherein the input means is configured to receive user input defining a plane of interest which is adjacent the object to be measured within the scene; and the processing means may be configured to use the plane of interest as a reference plane within the scene from which to take measurements of the object.

BRIEF DESCRIPTIONS OF DRAWINGS

Specific embodiments of the disclosure will now be described, by way of example, with reference to the accompanying drawings of which: Figure la is an image comprised of a raw point cloud data taken from a 3D camera indicating a surface having a plane of interest taken from above and at an angle to the surface; Figure lb is an image of the raw point cloud data of the surface of Figure 1a taken from the side, to illustrate noise and inaccurate data; Figure 2a is a schematic block diagram showing a portable computing device for determining a plane from captured image data, and Figure 2b shows a schematic representation of an environment in which the portable computing device is used to help measure the size of an object within that environment; Figure 3 is a flowchart showing a method for determining a plane from low quality point cloud data captured by the camera of the portable computing device of Figure 2a; Figure 4 is a flowchart showing the step of determining a first 3D plane of Figure 3 in greater detail; Figure 5 is a flowchart showing the step of improving plane determination of Figure 3 in greater detail; Figure 6a is a graphical plot of positions of a calculated floor plane affected by outlying points of data from a prior art analysis of a whole scene;

B

Figure 6b is a graphical plot of positions of a calculated floor plane with outlying points of data corresponding to Figure 6a removed from an analysis of a known area within the scene according to an embodiment of the present invention; Figure 7a is a graphical plot of positions of a calculated floor plane with noisy floor points; Figure 7b is a graphical plot of positions of a calculated floor plane corresponding to Figure 7a with additional data points obtained by capturing all points up to a radius according to an embodiment of the present invention; Figure 7c is a graphical plot of positions of a calculated floor plane corresponding to Figure 7b with noisy floor points removed as being outside a sigma value of overall distribution in accordance with an embodiment of the present invention; Figure 8a is a graphical plot of positions of a calculated floor plane with data points above a floor plane as determined by conventional camera technology; Figure 8b is a graphical plot of positions of a calculated floor plane corresponding to Figure 8a with a statistical height bias offset to compensate for the problem shown in Figure 8a; Figure Sc is a graphical plot of positions of a corrected calculated floor plane corresponding to Figure 8a adjusted by the statistical height bias offset of Figure 8b; Figure 9a is an image of the raw point cloud data of Figure la showing the result of a plane

fitting algorithm according to the prior art; and

Figure 9b is an image of the raw point cloud data of Figure la showing the result of a plane fitting algorithm according to an embodiment the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

In Figure 2a, a system for determining a plane from low quality image data (point cloud data) according to an embodiment of the present invention, is illustrated. The system in this embodiment comprises a mobile (portable) electronic device 106 (for example a smart phone or a tablet computer) with a depth camera 114 or depth sensor that is configured to capture an image of an observed scene and generate a set of 3D point data (image data) describing the observed scene. The camera/sensor 114 is operatively coupled to a processor 116 which processes the raw camera data according to an algorithm 111 stored in a memory 110 of the device 106 and a database 112 configured to store sensor data gathered from the camera or sensor 114 is coupled to the processor 116. The device 106 also has a screen or display 108 coupled to the processor 116 which displays an image of the captured data, and which allows user selection of a portion of the image as a reference start region (or start point). As will be described further below, the processor 116 running the algorithm 111, determines and improves plane calculations from the captured raw camera data.

As shown in Figure 2b, in an exemplary embodiment, a scene 117 has multiple planes (floor 119 and walls 121) with a freight in the form of a box 120 located on the floor 119. In order to accurately determine the size of the box 120, it is necessary to first determine the surface on which the box 120 is located. To determine the plane against which the box 120 is located (the plane of interest), a user defines the reference start region 118 within that plane of interest, illustrated by a screen box (B1), by indicating on the screen 108 of the electronic device 106, a region of the plane in the image of the observed scene 117. In this example, the plane of interest is the floor 119, but any of the planes in the image displayed on the mobile device 106 can be determined to be the plane of interest. For example, if the box 120 is against a wall 121, the wall 121 may be selected as the plane of interest to enable measurement to be carried out.

Figure 3 shows a flowchart summarising a method M for determining a plane from low-quality point cloud data. The method M commences, at Step 122, with the camera sensor (such as the depth camera 114 shown in Figure 2a) being used to capture an image of the scene 117 which includes an object of interest (for example, the box 120 shown in Figure 2b). The captured data provides a set of three-dimensional point data (or data points) describing the observed scene 117. Each data point comprises x and y coordinates and a depth coordinate.

The image is displayed on the screen 108 of the device 106 and then the user defines the reference start region 118 by identifying a small region B1 of the observed floor 119 (the plane of interest) in the captured image of the scene by using the screen 108 of the electronic device 106. The screen 108 can be touch sensitive and user identification of the region 118 can be simply achieved by the user touching the screen at a point in the displayed image which is considered to be in the plane of interest, namely the floor 119 in this instance. Alternatively, in other embodiments, a cursor can be used to select the small region B1 within the displayed image of the scene 117.

The reference start region 118 is illustrated with the screen box Bl. At Step 124, 3D points each providing x, y, and z coordinates representing the respective location of the screen box B1, captured using the camera or sensor 114 of the electronic device 106, are determined.

Using the captured points, a first 3D plane (P1) is determined, at Step 126, using the technique of least squares fit of a plane to the data points representing the screen box B1 and this discussed in greater detail later with reference to Figure 4. Once a first 3D plane (P1) has been determined, its determination is improved, at Step 128, using an iterative process (discussed later with reference to Figure 5). This process advantageously maintains a relatively low computational cost compared to other processes known in the art.

Improvement of the initial 3D plane calculation is carried out because the first 3D plane (P1) may not be an accurate representation of the actual floor 119 due to noisy and additional unrequired data from the camera or sensor 114 (see earlier description of Figures la and lb). Noisy data can be seen as a set of virtual points which represent a geometrical feature but deviates from the real geometry by varying orthogonal distances from a nominal. The amount of noise is reference to the quantity and deviation of points from the nominal geometry. Noise can also refer to missing data points and quantity of missing data points.

Once the plane calculation has been improved to a sufficient degree (described later), a mean bias (0) is applied, at Step 130, to the calculated plane to raise or lower the plane in response to any noisy data. This offsets the mean of a distribution for the plane to better coincide with a ground truth. This is described in greater detail later with reference to Figures 8a, 8b and 8c. Finally, once the correct plane has been determined, the dimensions of an object of interest within the scene can be measured, at Step 132, correctly with reference to the correct plane of interest. To measure the dimensions of an object within the captured scene, the resulting calculated plane allows points that are not of interest (for example below the floor, or behind a wall) to be removed from the scene, leaving only the object to be measured. In another embodiment, this technique can also be applied to measuring defects in highways; as roads are predominantly flat, they can be used to better describe disruptions in the surface of the road, such as potholes, even though the captured images include noisy data. There are, in fact, many different uses of the above described plane determination technique which can be carried out in different embodiments and as such measuring the dimensions of the object at Step 132, is optional part of this embodiment as it is only one of several possible applications of the plane determination technique.

Referring to Figure 4, the step of determining a first (initial) 3D plane (P1) (shown as Step 126 in Figure 3) is described in greater detail. The plane can be represented by a point coincident with the plane, and a vector of unit length normal the plane. Generally the steps involved to determine the plane are as follows. The centre of the point cloud (i.e., the mean) is calculated, followed by a covariance matrix of the point cloud. Next, eigen vectors and values of the covariance matrix are calculated using linear regression with the least squares method. The mean or centre point is coincident with the plane, and the minimum eigen is normal to the plain.

Looking at these steps in more detail, an input into the process, controlled by the algorithm 111, is a point set as an (x, y, z) three tuple of positive real numbers representing coordinates in three dimensional space of the screen box (B1). At Step 134, the mean of the points is calculated, that is, i.e., the centre point of the point set, also a three tuple of real numbers (x, y, z). A covariance matrix for the dataset is then calculated, at Step 136. The covariance matrix determines the variance exhibited by the tuples in a dataset The covariance matrix comprises the variances of the variables along the main diagonal and the covariances between each pair of variables in the other matrix positions.

Next, linear regression is applied, at Step 138, to an eigensystem, specifically to a list of eigenvalues and eigenvectors of the matrix calculated in the previous step (Step 136). The resultant eigensystem is a representation of the distribution of points. For a symmetric covariance matrix, as defined above, the eigen vectors will be orthogonal to each other in the directions most and least variations of the dataset. The eigen values scale the eigen vectors with respect to the magnitude of variation. The eigenvector in the direction of the smallest variance (eigen value) will be normal to the plane, while the mean or centre point of the dataset will be coincident with the plane, The eigenvector and the mean of all the points is output, at Step 140.The result of this process is an approximation to the actual plane of the floor plane 119. It is typically within a 25 degrees angle of the actual floor plane.

Referring to Figure 5, the step of improving plane determination of Figure 3 (Step 128) is described in greater detail below. As the first plane determination, or fit, may not be as accurate as desired, due to noisy and additional unrequired data from the camera or sensor 114, the first plane (P1) determined may not be consistent with the plane of interest (floor 119 in this instance). To improve the plane fitting, additional points are added, at Step 142, to the dataset by expanding the data points of interest (which will contribute to plane fitting) outside of the screen box B1 up to radius (r) value from the calculated centre point of the point set (point cloud) (i.e., the calculated mean of the point set). This may be achieved by adding, at Step 142, all data points in the captured image and then filtering out, at Step 143, data points which lie beyond the radius r, namely are located at a Euclidian distance greater than r from the centre point of the point set. Then, the distance from each of the new points to the first plane (P1) is measured, at Step 144. It is important to note that the plane extends beyond the boundary of the screen box B1 to a larger set of points B2. Once the distance has been measured, a distribution curve of the standard deviation of distances from P1 to B2 are plotted, at Step 146. The results are distributed as a bell curve. The standard deviation is used as it can handle variations in noise between different datasets or regions of the data better than other methods. Next, points that are not within a defined sigma value (S) of the distribution are removed at Step 148, and a new centre point of the point cloud is calculated, also at Step 148.The sigma value (S) is an input parameter which can be selected to suit the noise distribution associated with the given camera or sensor. At Step 150, a second plane (a new plane) is fitted to the data based on the remaining points and this results in a closer (more accurate) fit to the plane of interest (floor 119 in this instance).

Subsequently, a check is carried out, at Step 152, to determine if the planes have converged. Here, convergence is defined as when the mean distance to the plane of all 3D points being averaged is less than the previously set tolerance (T) for any further iterations. The tolerance value (T) is set to determine the level of accuracy required by the method. This has to be balanced by the time it will take to get a result as the higher the accuracy required the more time the process will take (as more iterations would typically be required). Regardless, an acceptable level of accuracy can be obtained without having to set the tolerance value to be very small. If the planes have not converged, then a check is carried out at Step 153 to determine if the number of iterations of the process (Steps 143 to 152) is equal to the input parameter N (see below). If the number of iterations is less than N, then the process is reiterated, starting again by removing any points which are located at a distance greater than r from the previously calculated centre of the point cloud. Note the centre point of each distribution typically varies on each iteration. -If the planes are considered to have converged (at Step 152), or if the number of iterations equals N (as determined at Step 153) the plane determination is output, at Step 154, and it can then be used, at Step 132, in this embodiment, to measure the dimensions of the object 120 within the scene 117.

The number of iterations (N), tolerance (T), sigma value (S) and radius (r) parameters can be changed to suit the type of sensor or camera or as the quality of the data decreases over distance. Example non-limiting values for a typical camera used include N = 20, T = 0.0005, S = 3.5, 1=1.4, where S is the number of standard deviations of the bell curve and the tolerance and radius (r) are in metres.

The processes described above are now further described with reference to graphical representations.

In Figure 6a, a graphical plot of distribution of positions of a calculated floor plane 158 affected by outlying points of data 156 from a prior art analysis of a whole scene is shown. Here, it can be seen that a calculated floor plane 158 is affected by outlying data points 156.

This is an example of a plane 158 that has been calculated inaccurately due to encompassing areas of a scene that are not in the plane of interest 155. In this case the calculated floor plane 158 is affected by the points of data 156 related to the wall plane 160 and so the position if the calculated plane 158 is inaccurate.

Figure 6b shows is a graphical plot of positions of a calculated floor plane 162 where the effect of outlying points of data 156 on the calculation of the surface position can be mitigated. The calculated floor plane 162 can be determined with outlying points of data 156 corresponding to Figure 6a removed. The removal is achieved by using a small known area of the plane of interest (the floor in this example) as a starting point 164. By using a small area 164 on the floor, outlying points, in this case points 156 on the wall plane 160, are iteratively removed if they lie outside of the defined sigma value of the current distribution of the points set The plane is then fitted to the new subset of points and the process is reiterated. In this way, the Euclidean distance of the plane reduces every iteration using the fixed sigma distance as the size of the standard deviation of each recalculated distribution reduces and the noisy points are discarded. Although the sigma value remains the same, this equates to a reduced Euclidean distance from the plane as outlying points are removed and the distribution is recalculated. Using the new distribution, the points outside same sigma value are discarded. This results in a more accurate plane determination.

In Figure 7a, a graphical plot of positions of a calculated floor plane 166 using only the start region B1 is shown. Here the data includes noisy floor points 168 and the present embodiment also deals with these types of noisy data points even within the start region B1 as is described with reference to Figures 7b and 7c. In this example, the first calculated floor plane 166 is not accurate after the first fit due to noisy and additional unrequired data from the camera or sensor 114.

Figure 7b is a graphical plot of positions of a calculated floor plane 166 corresponding to Figure 7a, with additional data points 170 of the captured image being added up to the radius r value. To improve the accuracy of the calculate planed 166, additional floor points B2 are added to the dataset by effectively expanding (Step 142) the points of interest outside of the screen box 81 up to the radius (r) from the centre of the point cloud within the screen box Bl.

However, the first plane remains inaccurate and so, the shortest distance from each of the new points to the first plane (perpendicular to the plane) is measured.

Figure 7c shows a graphical plot of positions of a newly calculated floor plane 174 corresponding to Figure 7a with noisy floor points 172 removed as being a distance from the newly calculated floor plane which is outside a sigma value of overall point distribution. The noisy floor points 172 that are not within the defined sigma value of the distribution are removed. The second plane (newly calculated) 174 on the remaining data points in the image, is then fitted to the remaining points and this results in a closer fit to the floor 176 as can be clearly seen by comparison of Figures 7b and 7c.

Figure 8a is a graphical plot of positions of a calculated floor plane 178 with data points 180 above a floor plane 182 as determined by conventional camera technology through examining raw data from the camera and determining the offset. When using particular camera technology, points may not be distributed evenly around the floor plane height, and some may, for example, have more noisy data above the floor position than below the floor position. The present embodiments mitigate this problem by calculating and applying an offset. More specifically, referring to Figure 8b a graphical plot of positions of a calculated floor plane 178 corresponding to Figure 8a with a statistical height bias offset is shown. To account for noisy points 180, the statistically determined bias (0) is applied to the mean of the data points to compensate for the data points 180 with a relative mean higher than the ground truth. The ground truth is determined by measuring a known plane and box and examining point cloud data generated from an image of the plane and box. Different cameras operating in different environments (including different viewing distances) typically have different offsets. The offset can raise or lower the plane 178 determined by the method described above in response to any a specific camera response which generates noisy data. The bias (offset which is applied) is the sigma (different to the value Sigma (S) described above) which offsets the mean of the distribution for the plane 178 to better coincide with the ground truth. It is to be appreciated that the bias is not a geometrical offset as this does not reliably transfer to different scenes using the same camera or parameters due to data noise variations between scenes. The bias is a sigma valued offset which biases the plane 178 relative to the distribution of noise. This has been found to be a more reliable method to consistently apply an offset for a particular camera and scene type.

Figure 8c is a graphical plot of positions of a corrected calculated floor plane 184 corresponding to Figure 8a adjusted by the statistical height bias offset of Figure 8b. The calculated floor plane 184 here has been adjusted by the statistically determined bias (0), which is in this case, is a negative value indicating a downwards adjustment such that the calculated floor plane 184 is at the same height as the actual floor plane 186.

To illustrate the effect that the present embodiments have on captured image data Figures 9a and 9b are now referred to. Figure 9a shows an image of the raw point cloud data of Figure la showing the result of a plane fitting algorithm according to the prior art methods. As can be seen here the prior art methods result in a poor plane fitting as the calculated plane 188 is clearly not aligned with the point data (including the dark square 103) which represents the first plane 100. A Figure 9b is an image of the raw point cloud data of Figure la showing the result of a plane fitting algorithm according to the above described embodiment of the present invention. Here, the calculated plane 190 is shown to clearly have a greater correspondence (alignment) to the plane of interest, the first plane 100. Also, the dark square 103 of the point data is also shown to be in alignment with the calculated plane 190.

Having described several exemplary embodiments of the present invention and the implementation of different functions of the device in detail, it is to be appreciated that the skilled addressee will readily be able to adapt the basic configuration of the system to carry out described functionality without requiring detailed explanation of how this would be achieved. Therefore, in the present specification several functions of the system have been described in different places without an explanation of the required detailed implementation as this not necessary given the abilities of the skilled addressee to implement functionality into the system.

For example, the above-described method could be applied to improve any plane determination, not necessarily floors or walls. It is best suited to any application where the data is noisy.

One application is to determine floors in logistics as part of an algorithm to measure an item on a floor. In this case, floor determination is used because it enables the floor to be removed leaving only the item to be measured. Also, the method is used to remove walls in an image On a logistics application) as part of an algorithm to measure items in the back of a truck. In this case, the planes are then used to exclude the truck walls and floor leaving only the item of freight to be measured.

In another application, as was mentioned previously, the present invention can be used in a highways application for measuring the size of potholes or other defects in the road surface. As roads are predominantly flat, the present invention can be used to better describe the condition of the road even though the camera data will contain noisy data.

In this regard, it is to be appreciated that any flat surface where a plane is to be determined from 3D camera data and the data is noisy, the present invention can be used to offset the plane if the noisy data is biased, i.e., more points are above rather than below the surface of interest. We can raise or lower the plane by Y, but never outside the noise boundary of the data.

Furthermore, it will be understood that features, advantages, and functionality of the different embodiments described herein may be combined where context allows.

Claims

CLAIMS: 1. A computer-implemented method for processing noisy image data to determine a plane of interest within the image data, the method comprising: capturing a three-dimensional (3D) image of a scene, the 3D image comprising a plurality of data points each having two-dimensional coordinates and depth information; displaying the 3D image of the scene on a screen of a portable electronic device; receiving a user input defining a start region, the user input indicating a region of the plane of interest in the scene; assigning data points of the start region to a set of processing data points; determining a first approximation of the plane of interest by: determining a mean position of the set of processing data points; fitting a proposed plane through the set of processing data points using a least-squares fit through the data points; and improving the first approximation of the plane of interest by: a) altering the set of processing points to include data points from the 3D image which are positioned within a radius of the mean position; b) determining a revised mean position of the set of processing data points; c) measuring a distance from each of the data points within the set of processing data points to the proposed plane; d) calculating a current value of standard deviation of the measured distances; e) removing processing data points outside of a multiple of the standard deviation from the set of processing data points; refitting the proposed plane through the set of processing data points using the least-squares fit; and repeating steps a) to f) until an end condition is reached, wherein the end condition comprises either the number of iterations equalling a number of maximum iterations parameter or the value of the standard deviation and a difference between the current value and a preceding value of the standard deviation is lower than a minimum deviation parameter.
2. The method as claimed in Claim 1, wherein the end condition comprises either: the number of iterations equalling the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is higher than a minimum deviation parameter; or the number of iterations is less than the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is lower than a minimum deviation parameter.
3. The method of Claim 1 or 2, further comprising setting the maximum number of iterations parameter and a difference parameter specifying the difference between the between the current value and a preceding value of the standard deviation.
4. The method of any preceding claim, further comprising setting a radius parameter specifying the radius to be used in the altering step.
5. The method of any preceding claim, further comprising setting a Sigma parameter specifying the multiple of standard deviations to be used in the removing step.
6. The method of any preceding claim, wherein the step of determining a first approximation of the plane of interest comprises: calculating a covariance matrix of the set of processing data points within the start region; applying linear regression on an eigensystem of the covariance matrix; and outputting an eigenvector and the mean of the set of processing data points.
7. The method as claimed in any preceding claim, further comprising: examining raw output data from a camera and determining noise deviations from a known position of a reference plane; determining an offset value to compensate for the deviations from the known position of the reference plane; and applying the offset, to the proposed plane once the end condition has been reached to bias the proposed plane relative to a distribution of noise.
8. The method of any preceding claim, further comprising outputting the proposed plane as a determined plane once the end condition has been reached.
9. A method of calculating the size of an object within a scene, the method comprising carrying out the method processing noisy image data to determine a plane of interest within the image data according to any preceding claim; wherein the receiving step comprises receiving user input defines a plane of interest which is adjacent the object to be measured within the scene; and using the plane of interest as a reference plane within the scene from which to take measurements of the object.
10. A portable electronic device for processing noisy image data to determine a plane of interest within the image data, the device comprising: a sensor or camera for capturing a three-dimensional (3D) image of a scene, the 3D image comprising a plurality of data points each having two-dimensional coordinates and depth information; a display for displaying the 3D image of the scene; input means for receiving a user input defining a start region, the user input indicating a region of the plane of interest in the scene; and processing means configured to: assign data points of the start region to a set of processing data points; determine a first approximation of the plane of interest by: determining a mean position of the set of processing data points; fitting a proposed plane through the set of processing data points using a least-squares fit through the data points; and improve the first approximation of the plane of interest by: a) altering the set of processing points to include data points from the 3D image which are positioned within a radius of the mean position; b) determining a revised mean position of the set of processing data points; c) measuring a distance from each of the data points within the set of processing data points to the proposed plane; d) calculating a current value of standard deviation of the measured distances; e) removing processing data points outside of a multiple of the standard deviation from the set of processing data points; refitting the proposed plane through the set of processing data points using the least-squares fit; and repeating steps a) tot) until an end condition is reached, wherein the end condition comprises either the number of iterations equalling a number of maximum iterations parameter or the value of the standard deviation and a difference between the current value and a preceding value of the standard deviation is lower than a minimum deviation parameter.
11. The portable electronic device as claimed in Claim 10, wherein the end condition comprises either: the number of iterations equalling the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is higher than a minimum deviation parameter; or the number of iterations is less than the number of maximum iterations parameter and the difference between the current value and the preceding value of the standard deviation is lower than a minimum deviation parameter.
12. The portable electronic device as claimed in Claim 10 or 11, further comprising means for setting the maximum number of iterations parameter and a difference parameter specifying the difference between the between the current value and a preceding value of the standard deviation.
13. The portable electronic device as claimed in any of Claims 10 to 12, further comprising means for setting a radius parameter specifying the radius to be used in the altering step.
14. The portable electronic device as claimed in any of Claims 10 to 13, further comprising means for setting a Sigma parameter specifying the multiple of standard deviations to be used in the removing step.
15. The portable electronic device as claimed in any of Claims 10 to 14, wherein the processing means is configured to determine a first approximation of the plane of interest by: calculating a covariance matrix of the set of processing data points within the start region; applying linear regression on an eigensystem of the covariance matrix; and outputting an eigenvector and the mean of the set of processing data points.
16. The portable electronic device as claimed in any of Claims 10 to 15, wherein the processing means is configured to: examine raw output data from the sensor or camera and determining noise deviations from a known position of a reference plane; and determine an offset value to compensate for the deviations from the known position of the reference plane; and apply the offset, to the proposed plane once the end condition has been reached to bias the proposed plane relative to a distribution of noise.
17. The portable electronic device as claimed in any of Claims 10 to 16, wherein the processor is configured to output the proposed plane as a determined plane once the end condition has been reached.
18. The portable electronic device as claimed in any of Claims 10 to 17, further configured to calculate the size of an object within a scene, wherein the input means is configured to receive user input defining a plane of interest which is adjacent the object to be measured within the scene; and the processing means is configured to use the plane of interest as a reference plane within the scene from which to take measurements of the object.