WO2010142929A1 - 3d image generation - Google Patents
3d image generation Download PDFInfo
- Publication number
- WO2010142929A1 WO2010142929A1 PCT/GB2009/001473 GB2009001473W WO2010142929A1 WO 2010142929 A1 WO2010142929 A1 WO 2010142929A1 GB 2009001473 W GB2009001473 W GB 2009001473W WO 2010142929 A1 WO2010142929 A1 WO 2010142929A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- depth
- frame
- distribution
- image
- camera
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
- H04N13/221—Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Definitions
- the present invention relates the fields of 3D image generation and 3D depth profiling. More specifically, the present invention is concerned with a system which can generate 3D images and depth profiles in real time.
- the ability to produce 3D images is useful in a number of technical fields.
- the new generation of parking sensors use images of the surroundings to guide the driver while parking.
- In the field of animation and special effects for films and games there is a need to be able to quickly produce an accurate depth map for a 3D object which is to be placed into the film or game.
- the provision of a 3D model of the article which is to be sold allows a potential buyer to study the article.
- the ability to generate a 3D model can also be used for surveying, in architecture, the automotive industry or even in fields such as archaeology where it is desirable to be able to store and send 3D mages of artefacts.
- 3D imaging technology exists most of the current systems allow the user to take the data to produce the image and then process the data at a later time. This has the drawback that it does not give a 3D image in real time, which makes it largely useless for parking sensors. Further, since the 3D image is produced after data acquisition, if more data is required to produce a good image, it is either too late or the process has to be started again. Finally, most current 3D systems require very large memory capacity and processing power.
- SLAM simultaneous localisation and mapping
- the present invention addresses the above problems and in a first aspect provides:
- a system for generating a 3D depth profile of an object comprising: a moveable video camera configured to continually capture 2D image data of said object as said camera is moved, said image data being captured as a plurality of
- each frame comprises a plurality of pixels
- a processor configured: to receive said 2D image data; to determine the position of the camera for each frame captured; to calculate the depth of the part of an object shown in a pixel in a first frame, said depth being determined with respect to a reference, said depth being determined by identifying said part of the object in at least one further frame and calculating the depth using the camera positions of said first image and said at least one further image; and to determine a depth profile of said object, the depth profile being determined for a plurality of parts of the object shown in said pixels, the system further comprises a display configured to display said depth profile and an indication in real time as data is still being gathered, the parts of the object for which the depth profile has been calculated.
- a depth profile which can be used to either generate a 3D image of the object or can be used as data for animation etc.
- no regularisation or other type of smoothing needs to be performed on the depth map.
- the depth profile will be dense, in a preferred embodiment the processor attempts to determine the depth for adjacent pixels in said first frame.
- said depth is calculated by determining the similarity of a pixel in a first frame with a plurality of pixels in at least one further image of said plurality of frames, said plurality of pixels in a further image being located along a epipolar line in said further image, said epipolar line representing the possible depths of said part of the object.
- the depth is preferably measured from a reference located along said epipolar line. More preferably, the reference point is located at a general mid-point along said line. For example, if the object is located on a mat, then the reference point may be take to be lying on the epipolar line and a plane through the middle of said mat.
- the distance through which the camera has moved between said first frame and said further frame is used to determine the number of pixels along said epipolar line which are processed to determine their similarity in said further frame. This may be achieved by either specifically calculating the position of the camera or assuming that for adjacent frames the pixel position of the part of the object will have only moved by n number of pixels.
- the value of n can be chosen dependent on the processing constraints of the computer used. If processing power is limited then n can be chosen to be a smaller value.
- the camera is preferably hand held.
- the system may be configured to indicated the parts of the object for which the depth profile has been calculated, by showing the object being constructed on a screen.
- the display may give more sophisticated information, for example a confidence level on the points calculated so far.
- a matching score indicating the similarity of a pixel in the first image with a pixel in said second image is determined for the pixels along said epipolar line, and potential matches for said pixel in the first image are selected as local maxima for said matching score when dependent on depth.
- the matching scores as a function of depth for a plurality of images can be combined to determine a single value for the depth of said part of object.
- the matching scores from further images for a given part of the object are continually collected until a depth is established.
- a plurality of calculations one for each different part of the object will be performed at the same time.
- a fixed number of calculations will be performed at any one time, when one of said calculations converges to a depth measurement, a calculation for a further part of said object is initiated.
- a criteria is also provided to determine if said distribution will not evolve to a substantially unimodal distribution such that said calculation may be stopped and a new calculation started.
- the depth of a part of said object is calculated when the distribution of matching scores against depth for a plurality of images becomes substantially unimodal in character.
- the present invention provides a depth sensor for determining the depth of a part of an object from a reference, said system comprising: a moveable video camera configured to continually capture 2D image data of said object as said camera is moved, said image data being captured as a plurality of 2D image frames where each frame comprises a plurality of pixels; a processor configured: to receive said 2D image data; to determine the position of the camera for each frame captured; to identify the part of the object whose depth is to be measured in a first frame; to determine matching scores indicating the similarity of a pixel showing said part of the object in a first frame with each of a plurality of pixels in a further frame as a function of the depth of said part of the object, said depth being determined using the camera positions of said first frame and said further frame; obtaining the matching scores as a function of depth for a plurality of further frames and combining said matching scores as a function of depth with previously obtained matching scores; continually monitoring the distribution of the matching scores as a function of depth as data
- the distribution of the matching score against depth is modelled as a mixture of a uniform distribution and a unimodal distribution.
- the unimodal distribution is a Gaussian distribution
- a weighting may be given to said unimodal distribution, said weighting indicating the probability that the depth indicated by said unimodal distribution is a good measurement.
- the weighting may be used as an indication of confidence in the calculated depth.
- the weighting may be shown on the display as a colour, the colour indicating the extent to which the distribution has converged to a unimodal distribution.
- the weighting may be compared with a threshold to determine if the distribution has converged.
- the weighting may also be compared with a second threshold to see if it is unlikely to converge and hence the calculation of the depth may be halted.
- the parameters of said model are estimated using Bayesian Posterior Estimation.
- the model may be approximated by a distribution of the type:
- X t is the t th point
- Z is the depth
- ⁇ is the weighting
- a and b are parameters of a beta distribution
- ⁇ and ⁇ 2 are the mean and standard deviation of said unimodal distribution.
- the standard deviation may also be compared with a reference to determine if the depth value is a good measurement.
- the matching score may be determined by a number of techniques such as Normalized Cross Correlation or Sum of Absolute differences.
- the position of the camera may be determined by a number of techniques, for example techniques which derive the camera position from the object itself such as those used in SLAM systems.
- the object is located, said mat having a known pattern which is used to calculate the position of the camera for each frame.
- said mat has a pattern comprising a plurality of circles.
- the present invention provides a method for generating a 3D depth profile of an object, the method comprising: continually capturing 2D image data of said object using a moveable video camera said image data being captured as a plurality of 2D image frames where each frame comprises a plurality of pixels; determining the position of the camera when each frame is captured; calculating the depth of the part of an object shown in a pixel in a first frame, said depth being determined with respect to a reference, said depth being determined by identifying said part of the object in at least one further frame and calculating the depth using the camera positions of said first image and said at least one further image; determining a depth profile of said object, the depth profile being determined for a plurality of parts of the object shown in said pixels, and displaying said depth profile and an indication in real time as data is still being gathered, the parts of the object for which the depth profile has been calculated.
- the present invention provides a method of determining the depth of a part of an object from a reference, said method comprising: continually capturing 2D image data of said object using a moveable video, said image data being captured as a plurality of 2D image frames where each frame comprises a plurality of pixels; determining the position of the camera when each frame is captured; identifying the part of the object whose depth is to be measured in a first frame; determining matching scores indicating the similarity of a pixel showing said part of the object in a first frame with each of a plurality of pixels in a further frame as a function of the depth of said part of the object, said depth being determined using the camera positions of said first frame and said further frame; obtaining the matching scores as a function of depth for a plurality of further frames and combining said matching scores as a function of depth with previously obtained matching scores; continually monitoring the distribution of the matching scores as a function of depth as data from further frames is combined; and determining the depth of said part of said object when said distribution of
- the present invention can be implemented either in hardware or on software in a general purpose computer. Further the present invention can be implemented in a combination of hardware and software. The present invention can also be implemented by a single processing apparatus or a distributed network of processing apparatuses.
- the present invention can be implemented by software, the present invention encompasses computer code provided to a general purpose computer on any suitable carrier medium.
- the carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signal e.g. an electrical, optical or microwave signal.
- the present invention can be used in a number of different fields.
- the above system may form part of an e-commerce system.
- the present invention provides an e-commerce system comprising a system according to the first or second aspects of the present invention configured to produce a depth profile or image of an article for sale on said commerce system.
- the present invention may also form part of a parking system, where a depth profile of the surroundings of a vehicle may be constructed.
- a parking system comprising a system in accordance with the first aspect of the present invention configured to produce a depth profile of the surroundings of the vehicle.
- the present invention may also be used for surveying, architecture, archaeology where a 3D image or depth profile of surroundings, building or article are required.
- the present invention may also be used in the field of animation and special effects where it is required to animate or otherwise place a 3D object in a film or game, a depth profile of said object being generated using any of the above systems or methods.
- Figure 1 shows an imaging system in accordance with an embodiment of the present invention
- Figure 2 is a schematic demonstrating operation of a system in accordance with an embodiment of the present invention where an image of point x is taken at two different positions by the same camera;
- Figures 3a to 3d show plots of the matching score of a plurality of pixels against depth from a reference point for a pixel in a reference image I and further images, figure 3a shows the results for 1 further frame, figure 3b for 6 further frames, figure 3c for 15 further frames and figure 3d for 60 further frames;
- Figure 4 is a flow diagram showing steps in accordance with an embodiment of the present invention
- Figures 5a (i) to 5h(i) are histograms showing matching score against depth for a pixel in the reference image
- figure 5a (iii) to 5h (iii) shows the results of a probabilistic depth model of the data of row (i)
- figures 5a (ii) to figure 5h (ii) show a parametric model approximation to the model results of row (iii);
- Figure 6a shows a plot of how the estimate of the depth plotted along the y axis converges over data from further images plotted along the x axis
- figure 6b shows a plot of the probability that the depth being derived in figure 6a is good against the number of images along the x axis
- figure 6c shows the measurement histogram with matching probability for a plurality of images against possible depth
- Figure 7 is a schematic used to explain the calculations performed by a system in accordance with an embodiment of the present invention.
- Figure 8a is a picture of a model which is to be imaged
- figure 8b is benchmark 3D image generated from the model of figure 8a
- figure 8c shows the results of a prior art method where 30 2D frames are combined to form the 3D image
- figure 8d shows the results of the continuous method of the present invention produced after 30 frames
- figure 8e shows the results of the present invention after 600 frames;
- Figures 9a-f show a display generated in accordance with an embodiment of the present invention in figure 9a, a small number of 2D images have been captured and the number of images captured increases to produce the figures shown in figure 9f;
- Figures 10a-d show images produced from a method in accordance with an embodiment of the present invention.
- Figure 1 shows a system in accordance with an embodiment of the present invention.
- the system basically comprises a camera 1, a computer 3 and a display 5.
- the camera 1 is a standard video camera and can be moved by a user. In operation, the camera 1 will be freely moved around object 7 which is to be imaged. In a preferred embodiment, the camera will not be mounted on a tripod or other mechanical support device, it is simply handheld.
- the computer 3 comprises a section for receiving the camera data from camera 1.
- the camera data receiving section 9 is in communication with processor 11.
- Processor 11 is configured to process the data in accordance with an embodiment of the present invention. Once the data has been processed, it is then converted into the form of a depth map and output via data output section 13 to display 5.
- the display shows the 3D image as it is slowly being built up. Although the figure is in black and white, in a preferred embodiment, when displayed on a computer screen the possible data points change colour as a confidence level in the point increases. Once the point reaches a pre-determined colour, the user knows that enough data concerning that part of the object have been gathered and that the camera can be moved on to another part of the object. The system will determine the depth of many points at once.
- FIG. 2 is a schematic which is used to explain a method in accordance with an embodiment of the present invention.
- Camera 1 is moved between first position 21 which will be referred to as the first image position and second position 23 which will be referred to as the further image position.
- first image I a pixel p is shown.
- point x (Z) is shown on the object.
- the point x (Z) lies at a distance (Z) from a reference point.
- the reference point is camera 1.
- the reference point could be any point.
- the point x which is shown at pixel p lies along epipolar line 25. From the data in 2D image I, it is impossible to judge the depth Z. However, the position of the line along which Z lies can be determined.
- the position of projected line 27 can easily be determined once the position of the camera at the first position 21 and the second position 23 are known. Further, as the images are captured by a continually moving video camera, the distance between position 21 and position 23 is very small. In order to provide a clear diagram, in figure 2, the difference between these two positions has been exaggerated. In reality, this difference will be very small therefore the pixel p at which point x is shown in the reference image will only move within a small area w from the image taken in the first position I to the image in the second position I'.
- This area w when projected onto the second image I' as w' then means that it is only pixels which fall along the projection of epi-polar line 27 within the projection of area w' that need to be processed to look for similarity with the pixel p.
- Correspondence scores can be evaluated using systems such as normalized cross correlation (NCC), sum of absolute differences (SAD) or another metric on w and w'.
- FIG. 3a A plot of the matching score or similarity score is shown in figure 3a for distances between band Z min and Z max. It has been found that in well textured scenes, the correct depth is typically very close to a local maxima of a matching score. Therefore, just the local maxima which will be denoted as Xi , XN are considered from hereon.
- the distance Z can be projected onto the second image I'.
- the first approximation of the distance Z will be based on some information concerning the general size of the object.
- the camera will then move to a third position (not shown in figure 2), in the third position, the same analysis can be performed and a similarity score can be performed in the same manner as described with reference to figure 2 for position 23.
- the two similarity scores can be then added together.
- the scores for both further images are represented in terms of Z along the epi-polar line 25.
- the similarity scores from six frames are added together.
- the local maxima is heavily multi- modal. This is due to various pathologies in the problem domain such as occlusion, time warping, repetitive texture etc.
- figure 3c further similarity scores the similarity scores from 15 frames are added and in figure 3d the similarity scores from 60 frames are added.
- FIG. 4 is a flow chart showing steps of the method in accordance with an embodiment of the present invention.
- step S101 two 2 dimensional image data is collected by a video camera as explained with reference to figures 1 to 3.
- One of the frames is denoted as a first frame in step S103.
- the first frame will be used to construct the epipolar line 25 shown in figure 2 for a pixel p. However, as image capture continues, further frames may be denoted as the first frame for other pixels.
- step S105 the camera position for each frame is determined. This may be done by a number of methods. For example, it is possible to determine the position of the cameras purely from multiple measurements of the object itself and matching points on the objects across multiple frames. It is also possible to determine the position of the cameras if the system is provided with information concerning parts of the object. In a preferred embodiment, the object is provided on a marker mat, the size of features on the marker mat and their position being known to the system to allow calibration.
- step S107 matching scores are then obtained which indicate the similarity between a pixel which shows a part of said object in the reference frame and a plurality of pixels in a further frame. This is described with reference to figure 2.
- the matching scores are obtained for a plurality of frames.
- the depth of said feature is determined from a plurality of further images.
- the histogram evolves from being multimodal to substantially unimodal in character. Also, when many images are available, in addition to the unimodal distribution around the depth, the histogram has an additional DC component. This comes from the aforementioned failure cases of the matching score function that typically do not cluster in the same depth locations.
- a probabilistic depth sensor is constructed.
- the role of the probabilistic depth sensor is to provide a parameter based approach to determine this transition.
- the local maxima which are observed in the matching score for example see figure 3a are defined as x-i, ...XN as a set of noisy measurements coming from a depth sensor.
- the depth sensor can produce two types of measurements with a probability ⁇ and 1- ⁇ respectively:
- the limit Z min and Z max can be determined by some prior knowledge of the geometry of the object, for example, the longest dimensions. For example, if the object is located on a mat, then the dimensions of the mat can be used. In this example the distance 2 is measured from a midpoint in the object along the epipolar line.
- the object of interest is guaranteed to be entirely contained between Z m i n and Z max .
- the following Gaussian plus uniform mixture model describes the probability of obtaining the t h measurement given the correct depth location Z and the inlier probability ⁇
- the variance of a good measurement ⁇ ] can be reliably calculated from the relative position of the cameras at frame I and I' that produce the measurement. This is because it is assumed that the measurement x t has a fixed variance of 1 pixel when projected in I'. This variance is then lifted in 3D space to produce the variance of the measurement in distance units. This will be described with reference to figure 7.
- the estimation process is formulated as Bayesian posterior estimation for the following reasons:
- the posterior probability can be obtained from:
- the first row (i) is a histogram showing the row data gathered as described with reference to figure 3a-d.
- Figures 5a-d show results for a pixel p where the results converge.
- Figures 5e-h show results which do not converge.
- the results are shown after five updates, i.e.one first image and five further images.
- Figures 5b and f show the results after 10 updates
- figures 5c and g show the results up to 20 updates
- figures 5d and h show the results after 100 updates.
- Row (iii) shows snap shots of the evolution of P[Z, ⁇ x ⁇ xj over time, where the distribution is modelled using a dense 2D histogram as described above. The distribution converges to the correct values for Z and ⁇ . However, modelling posterior with a full 2D histogram for each pixel is also impractical due to memory limitations.
- Figures 6a, b and c demonstrate further evolution of how the depth estimate is achieved.
- Figure 6a the depth estimate is shown as a mean (thick line) and standard deviation (2 outer lines).
- Figure 6c shows the histogram data with a parametric model data (solid line) superimposed on top thus showing that the parametric model has a converged to the correct depth.
- the profile of the object can be determined as described with reference to step S109 of figure 4.
- step S111 the 3D image can is constructed in step S111.
- a planar pattern is used where the features are detected and tracked a camera frame-rate of 60 hertz with a mean projection error of about 0.1 pixels.
- the object is located on a mat having a pattern of a plurality of circles. A pattern with a plurality of circles allows easy recognition of the correct camera position (see figure 8a).
- the problem of estimating the depth Z alone an optic ray passing through pixel p away from the image plane I is an inference problem.
- an inference problem When constructing a 3 dimensional depth profile of a sample, it is necessary to solve many of these inference problems. As each inference problem will require computing power and memory to perform all inference problems for all points on the object as once.
- Each inference problem is termed a "seed".
- Each seed is associated with a set of parameter values (a t , b t , ⁇ t , ⁇ 2 t , W).
- Parameters a t , b t , ⁇ t, ⁇ 2 t describe the depth Z and the inlier ratio ⁇ for that pixel.
- the first two a t and b t are two parameters of the Beta distribution of equation 3. These probabilistic counters of how many inlier and outlier measurements have occurred during the lifetime of the seed.
- the other two parameters ⁇ t and ⁇ 2 t represent the mean and variance of the Gaussian depth estimate.
- a reference image patch w at the location of the corresponding pixel at the reference image of the seed is also stored. This patch is used to compared against target matches to obtain depth measurements.
- the depth parameter is ⁇ t and ⁇ t are set such that 99% of the prior probability mass is between preset Z m ⁇ n and Z max . These limits define a bounding volume inside which the object of interest must lie.
- w is set at 3 pixels for a 2 million pixel camera.
- the local maxima of the matching score collected in Xi to x N .
- the new posterior distribution parameters a t+ i, b b+ i, ⁇ t +i and ⁇ 2 t+ i are calculated by matching the first and second order moments at Z and ⁇ between
- each seed is faced with three possibilities:
- the above framework provides such a decision criteria in the form of a variance of the depth posterior ⁇ 2 t and the estimated inlier probability ⁇ .
- Figure 9a-f show the convergence process.
- the points rendered represent the mean of the current depth distributions for each seed. As this is a black and white figure, the colours cannot be seen. However, in the preferred embodiment, each point will be associated with a colour which represents the mean inlier ratio. In the beginning this is close to 0.5 (brown) for most points. Eventually, it becomes much higher (red) for seeds that get many inliers and much lower (green) for seeds that do not get inliers. These typically correspond to pixels that are locations in regions of low focus or low texture variation. These seeds would eventually be discarded.
- the generated 3D points are collected into a octree structure that is graphically rendered with a Z-buffer shading in real time.
- Store in W a square patch at the seed's pixel location in the current image I
- Step 2 For each seed (a t , b t , ⁇ t , ⁇ t 2 , W)
- Step 3 Remove all seeds (a t , b t , ⁇ t, ⁇ t 2 , W) such that ⁇ ou tifer with 99% probability.
- Step 4 Convert into 3d points (and remove from seed list) all seeds (a t , b t , ⁇ t , ⁇ t 2 , W) such that (1) ⁇ > ⁇ j n iier with 99% probability (2) and ⁇ t ⁇ .
- sequence of regularised 2-view stereo depth maps performs worse than the un- regularised continuous depth estimation in accordance with an embodiment of the present invention which operates in the same set of images.
- Figure 8a shows one of the frames of the video sequence.
- Figure 8b shows the benchmark reconstruction using a high definition camera.
- Figure 8c shows the results of fusing 30 two-view depth maps.
- Figure 8d shows the results of a method in accordance with the present invention using a continuous depth estimate result operating on the same images as one and figure 8e shows the results of the algorithm on all 600 images.
- the sequence of regularised two-view stereo depth maps performs worse than the un- regularised continuous depth estimation operating on the same set of images. This is due to the fact that the present invention fully exploits the temporal continuity of the depth maps. The performance in accuracy improves even more when all 600 frames are added to the depth estimation.
- Figures 10(a) to (d) show images which were collected within about 40 seconds.
- the models have not been regularised or post-processed in any way.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
Description
Claims
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/377,712 US9189859B2 (en) | 2009-06-11 | 2009-06-11 | 3D image generation |
GB1200452.9A GB2483213B8 (en) | 2009-06-11 | 2009-06-11 | 3D image generation |
PCT/GB2009/001473 WO2010142929A1 (en) | 2009-06-11 | 2009-06-11 | 3d image generation |
JP2012514524A JP5487298B2 (en) | 2009-06-11 | 2009-06-11 | 3D image generation |
US14/931,505 US20160196658A1 (en) | 2009-06-11 | 2015-11-03 | 3d image generation |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/GB2009/001473 WO2010142929A1 (en) | 2009-06-11 | 2009-06-11 | 3d image generation |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/377,712 A-371-Of-International US9189859B2 (en) | 2009-06-11 | 2009-06-11 | 3D image generation |
US14/931,505 Continuation US20160196658A1 (en) | 2009-06-11 | 2015-11-03 | 3d image generation |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010142929A1 true WO2010142929A1 (en) | 2010-12-16 |
Family
ID=41510610
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/GB2009/001473 WO2010142929A1 (en) | 2009-06-11 | 2009-06-11 | 3d image generation |
Country Status (4)
Country | Link |
---|---|
US (2) | US9189859B2 (en) |
JP (1) | JP5487298B2 (en) |
GB (1) | GB2483213B8 (en) |
WO (1) | WO2010142929A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
NL2008490C2 (en) * | 2012-03-15 | 2013-09-18 | Ooms Otto Bv | METHOD, DEVICE AND COMPUTER PROGRAM FOR EXTRACTING INFORMATION ON ONE OR MULTIPLE SPATIAL OBJECTS. |
CN110249215A (en) * | 2017-02-08 | 2019-09-17 | 联达科技设备私人有限公司 | For generating the 3D composograph of object and determining the method and systems of object properties based on the 3D composograph |
Families Citing this family (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8503720B2 (en) | 2009-05-01 | 2013-08-06 | Microsoft Corporation | Human body pose estimation |
JP2011104906A (en) * | 2009-11-18 | 2011-06-02 | Mitsubishi Heavy Ind Ltd | Inspection method, method for producing composite material parts, inspection apparatus and apparatus for producing composite material parts |
US20120044320A1 (en) * | 2010-03-11 | 2012-02-23 | Trex Enterprises Corp. | High resolution 3-D holographic camera |
US8942917B2 (en) | 2011-02-14 | 2015-01-27 | Microsoft Corporation | Change invariant scene recognition by an agent |
JP5170276B2 (en) * | 2011-03-31 | 2013-03-27 | カシオ計算機株式会社 | Image reconstruction device, image reconstruction method, and image reconstruction program |
KR101275823B1 (en) * | 2011-04-28 | 2013-06-18 | (주) 에투시스템 | Device for detecting 3d object using plural camera and method therefor |
WO2013025151A1 (en) * | 2011-08-12 | 2013-02-21 | Telefonaktiebolaget L M Ericsson (Publ) | Signaling of camera and/or depth parameters |
CN104904200B (en) | 2012-09-10 | 2018-05-15 | 广稹阿马斯公司 | Catch the unit and system of moving scene |
US9332243B2 (en) * | 2012-10-17 | 2016-05-03 | DotProduct LLC | Handheld portable optical scanner and method of using |
US10674135B2 (en) * | 2012-10-17 | 2020-06-02 | DotProduct LLC | Handheld portable optical scanner and method of using |
US9576183B2 (en) * | 2012-11-02 | 2017-02-21 | Qualcomm Incorporated | Fast initialization for monocular visual SLAM |
US9417910B2 (en) * | 2012-12-20 | 2016-08-16 | Oracle International Corporation | System and method for implementing shared probabilistic counters storing update probability values |
US9857470B2 (en) | 2012-12-28 | 2018-01-02 | Microsoft Technology Licensing, Llc | Using photometric stereo for 3D environment modeling |
US9940553B2 (en) | 2013-02-22 | 2018-04-10 | Microsoft Technology Licensing, Llc | Camera/object pose from predicted coordinates |
KR20190021496A (en) * | 2013-10-07 | 2019-03-05 | 애플 인크. | Method and system for providing position or movement information for controlling at least one function of a vehicle |
US10937187B2 (en) | 2013-10-07 | 2021-03-02 | Apple Inc. | Method and system for providing position or movement information for controlling at least one function of an environment |
CN107636728B (en) * | 2015-05-21 | 2022-03-01 | 皇家飞利浦有限公司 | Method and apparatus for determining a depth map for an image |
EP3272110B1 (en) * | 2015-05-22 | 2019-07-31 | Huawei Technologies Co., Ltd. | Apparatus and method for color calibration |
US10242455B2 (en) * | 2015-12-18 | 2019-03-26 | Iris Automation, Inc. | Systems and methods for generating a 3D world model using velocity data of a vehicle |
US10417781B1 (en) | 2016-12-30 | 2019-09-17 | X Development Llc | Automated data capture |
EP3379814A1 (en) * | 2017-03-23 | 2018-09-26 | Christian Rymarenko | Converting media using mobile devices |
NL2019516B1 (en) * | 2017-09-08 | 2019-03-19 | Fugro N V | System and method for determination of a spatial property of a submerged object in a 3D-space |
US10460512B2 (en) * | 2017-11-07 | 2019-10-29 | Microsoft Technology Licensing, Llc | 3D skeletonization using truncated epipolar lines |
US10534837B2 (en) | 2017-11-13 | 2020-01-14 | Samsung Electronics Co., Ltd | Apparatus and method of low complexity optimization solver for path smoothing with constraint variation |
JP2022143412A (en) | 2021-03-17 | 2022-10-03 | 株式会社東芝 | Image processing device and image processing method |
US20230077427A1 (en) * | 2021-09-12 | 2023-03-16 | NexTech AR Solutions Corp. | Material estimation for three-dimensional (3d) modeling |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1193646A2 (en) * | 2000-09-27 | 2002-04-03 | Canon Kabushiki Kaisha | Image processing apparatus |
WO2007084589A2 (en) * | 2006-01-20 | 2007-07-26 | 3M Innovative Properties Company | Three-dimensional scan recovery |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5179441A (en) * | 1991-12-18 | 1993-01-12 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Near real-time stereo vision system |
JPH09187038A (en) | 1995-12-27 | 1997-07-15 | Canon Inc | Three-dimensional shape extract device |
US6163337A (en) * | 1996-04-05 | 2000-12-19 | Matsushita Electric Industrial Co., Ltd. | Multi-view point image transmission method and multi-view point image display method |
US6094215A (en) * | 1998-01-06 | 2000-07-25 | Intel Corporation | Method of determining relative camera orientation position to create 3-D visual images |
US6146830A (en) * | 1998-09-23 | 2000-11-14 | Rosetta Inpharmatics, Inc. | Method for determining the presence of a number of primary targets of a drug |
JP4221808B2 (en) * | 1999-03-30 | 2009-02-12 | コニカミノルタセンシング株式会社 | Three-dimensional data input method and apparatus |
JP4306006B2 (en) * | 1999-03-30 | 2009-07-29 | コニカミノルタセンシング株式会社 | Three-dimensional data input method and apparatus |
US7085409B2 (en) * | 2000-10-18 | 2006-08-01 | Sarnoff Corporation | Method and apparatus for synthesizing new video and/or still imagery from a collection of real video and/or still imagery |
JP4075418B2 (en) * | 2002-03-15 | 2008-04-16 | ソニー株式会社 | Image processing apparatus, image processing method, printed material manufacturing apparatus, printed material manufacturing method, and printed material manufacturing system |
US7724379B2 (en) | 2005-05-12 | 2010-05-25 | Technodream21, Inc. | 3-Dimensional shape measuring method and device thereof |
JP2008275366A (en) * | 2007-04-26 | 2008-11-13 | Tokyo Institute Of Technology | Stereoscopic 3-d measurement system |
JP5120926B2 (en) * | 2007-07-27 | 2013-01-16 | 有限会社テクノドリーム二十一 | Image processing apparatus, image processing method, and program |
-
2009
- 2009-06-11 US US13/377,712 patent/US9189859B2/en active Active
- 2009-06-11 GB GB1200452.9A patent/GB2483213B8/en active Active
- 2009-06-11 JP JP2012514524A patent/JP5487298B2/en active Active
- 2009-06-11 WO PCT/GB2009/001473 patent/WO2010142929A1/en active Application Filing
-
2015
- 2015-11-03 US US14/931,505 patent/US20160196658A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1193646A2 (en) * | 2000-09-27 | 2002-04-03 | Canon Kabushiki Kaisha | Image processing apparatus |
WO2007084589A2 (en) * | 2006-01-20 | 2007-07-26 | 3M Innovative Properties Company | Three-dimensional scan recovery |
Non-Patent Citations (5)
Title |
---|
H.P. MORAVEC: "Rover visual obstacle avoidance", INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 1981, pages 785 - 790, XP002569539 * |
KU J S ET AL: "Multi-image matching for a general motion stereo camera model", PATTERN RECOGNITION, ELSEVIER, GB, vol. 34, no. 9, 1 September 2001 (2001-09-01), pages 1701 - 1712, XP004362596, ISSN: 0031-3203 * |
LANE R A ET AL: "Tutorial: Overview of Stereo Matching Research", TINA MEMO. INTERNAL, R, 1 December 1998 (1998-12-01), pages 1 - 10, XP002355354 * |
POPESCU ET AL: "The ModelCamera", GRAPHICAL MODELS, ELSEVIER, SAN DIEGO, CA, US, vol. 68, no. 5-6, 1 November 2006 (2006-11-01), pages 385 - 401, XP005731783, ISSN: 1524-0703 * |
S. RUSINKIEWICZ ET AL: "Real-time 3D model acquisition", ACM TRANSACTIONS ON GRAPHICS, PROCEEDINGS OF ACM SIGGRAPH 2002, vol. 21, no. 3, July 2002 (2002-07-01), pages 438 - 446, XP002569538 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
NL2008490C2 (en) * | 2012-03-15 | 2013-09-18 | Ooms Otto Bv | METHOD, DEVICE AND COMPUTER PROGRAM FOR EXTRACTING INFORMATION ON ONE OR MULTIPLE SPATIAL OBJECTS. |
WO2013137733A1 (en) * | 2012-03-15 | 2013-09-19 | Otto Ooms B.V. | Method, device and computer programme for extracting information about one or more spatial objects |
EP2825841B1 (en) | 2012-03-15 | 2016-11-23 | Otto Ooms B.V. | Method, device and computer programme for extracting information about a staircase |
EP3153816A1 (en) * | 2012-03-15 | 2017-04-12 | Otto Ooms B.V. | Method, device and computer programme for extracting information about one or more spatial objects |
US9885573B2 (en) | 2012-03-15 | 2018-02-06 | Otto Ooms B.V. | Method, device and computer programme for extracting information about one or more spatial objects |
CN110249215A (en) * | 2017-02-08 | 2019-09-17 | 联达科技设备私人有限公司 | For generating the 3D composograph of object and determining the method and systems of object properties based on the 3D composograph |
Also Published As
Publication number | Publication date |
---|---|
GB201200452D0 (en) | 2012-02-22 |
GB2483213B8 (en) | 2016-09-14 |
US20160196658A1 (en) | 2016-07-07 |
GB2483213A8 (en) | 2016-09-14 |
JP5487298B2 (en) | 2014-05-07 |
US9189859B2 (en) | 2015-11-17 |
US20120147152A1 (en) | 2012-06-14 |
GB2483213B (en) | 2014-04-30 |
GB2483213A (en) | 2012-02-29 |
JP2012529691A (en) | 2012-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9189859B2 (en) | 3D image generation | |
Schops et al. | Bad slam: Bundle adjusted direct rgb-d slam | |
CN109190508B (en) | Multi-camera data fusion method based on space coordinate system | |
Whelan et al. | Deformation-based loop closure for large scale dense RGB-D SLAM | |
KR101532864B1 (en) | Planar mapping and tracking for mobile devices | |
JP6095018B2 (en) | Detection and tracking of moving objects | |
US10163256B2 (en) | Method and system for generating a three-dimensional model | |
CN110807809B (en) | Light-weight monocular vision positioning method based on point-line characteristics and depth filter | |
US20190141247A1 (en) | Threshold determination in a ransac algorithm | |
Padua et al. | Linear sequence-to-sequence alignment | |
CN112367514A (en) | Three-dimensional scene construction method, device and system and storage medium | |
CN103218799B (en) | The method and apparatus tracked for camera | |
KR20120031012A (en) | Piecewise planar reconstruction of three-dimensional scenes | |
CN110827321B (en) | Multi-camera collaborative active target tracking method based on three-dimensional information | |
CN112630469B (en) | Three-dimensional detection method based on structured light and multiple light field cameras | |
Yuan et al. | 3D reconstruction of background and objects moving on ground plane viewed from a moving camera | |
WO2022127181A1 (en) | Passenger flow monitoring method and apparatus, and electronic device and storage medium | |
Stentoumis et al. | A local adaptive approach for dense stereo matching in architectural scene reconstruction | |
CN117456114B (en) | Multi-view-based three-dimensional image reconstruction method and system | |
JP2007025863A (en) | Photographing system, photographing method, and image processing program | |
Argiles et al. | Dense multi-planar scene estimation from a sparse set of images | |
Mahmoud et al. | Fast 3d structure from motion with missing points from registration of partial reconstructions | |
WO2016170332A1 (en) | A method of generating a 3d representation of an environment and related apparatus | |
Vandewiele et al. | Occlusion management strategies for pedestrians tracking across fisheye camera networks | |
Alahi et al. | A sparsity constrained inverse problem to locate people in a network of cameras |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09784602 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2012514524 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 1200452 Country of ref document: GB Kind code of ref document: A Free format text: PCT FILING DATE = 20090611 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1200452.9 Country of ref document: GB |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13377712 Country of ref document: US |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 09784602 Country of ref document: EP Kind code of ref document: A1 |