US20190294893A9 - Monocular cued detection of three-dimensional structures from depth images - Google Patents
Monocular cued detection of three-dimensional structures from depth images Download PDFInfo
- Publication number
- US20190294893A9 US20190294893A9 US14/180,548 US201414180548A US2019294893A9 US 20190294893 A9 US20190294893 A9 US 20190294893A9 US 201414180548 A US201414180548 A US 201414180548A US 2019294893 A9 US2019294893 A9 US 2019294893A9
- Authority
- US
- United States
- Prior art keywords
- image
- model
- depth map
- feature
- image frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 20
- 238000000034 method Methods 0.000 claims description 28
- 239000003550 marker Substances 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 2
- 230000004888 barrier function Effects 0.000 description 58
- 230000006870 function Effects 0.000 description 7
- 238000013459 approach Methods 0.000 description 3
- 239000002184 metal Substances 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000010426 asphalt Substances 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G06K9/00805—
-
- H04N13/0271—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
Definitions
- the present disclosure relates to detection of three-dimensional structures from two-dimensional images and particularly related to application in driver assistance systems
- Structure-from-Motion refers to methods for recovering three-dimensional information of a scene that has been projected onto the back focal plane of a camera.
- the structural information derived from a SfM algorithm may take the form of a set of projection matrices, one projection matrix per image frame, representing the relationship between a specific two-dimensional point in the image plane and its corresponding three-dimensional point.
- SfM algorithms rely on tracking specific image features from image frame to image frame to determine structural information concerning the scene.
- stereo or multi-ocular disparity analysis may be used to determine three-dimensional points from two-dimensional images. Finding structure from motion presents a similar problem as finding structure from stereo vision. In both instances, the correspondence between images and the reconstruction of three-dimensional object is found.
- a depth map is an image that contains information relating to the distance Z of surfaces of objects from a viewpoint; the viewpoint generally being the position of a camera.
- Depth maps may also lack the full spatial resolution of the original gray scale images.
- the epipolar geometry of the cameras is known and dense correspondences are computed by performing a search along the epipolar lines.
- Structure from Motion (SfM) techniques typically precede the dense epipolar search by computing the camera motion and epipolar geometry.
- a smoothness function is explicitly or implicitly assumed so as to regularize the search and give robustness to noise and brightness changes.
- the search is performed by matching images patches rather than individual pixels.
- the depth error E z in multi-camera stereo is a function of forward distance Z to the cameras, the correspondence error (E d ), image resolution (or focal length f in pixels) and the baseline b:
- driver assistance system and corresponding method adapted to detect three dimensional structures such as guardrails, curbs or other three dimensional objects in the road environment from depth map images while avoiding intensive computational overhead normally associated with computing depth maps to allow for real time processing in the driver assistance system.
- the systems are mountable in a host vehicle and a camera connectible to a processor.
- Multiple image frames are captured in the field of view of the camera.
- an imaged feature is detected of an object in the environment of the vehicle.
- the image frames are portioned locally around the imaged feature to produce imaged portions of the image frames including the imaged feature.
- the image frames are processed to compute a depth map locally around the detected imaged feature in the image portions.
- the depth map includes an image of the feature with a color or grayscale coordinate related to a function of distance from the camera to the object. Responsive to the depth map, it is determined if the object is an obstacle to the motion of the vehicle.
- the object may be represented with multiple models.
- the depth map of the detected feature may be compared with the model depth maps and based on the comparison, it may be determined that the object is an obstacle or not an obstacle to the motion of the vehicle.
- the models may be a horizontal planar model, a vertical planar model, a mixed model including horizontal and vertical portions, a spherical model, a circular model, a model of a guard rail, a model of lane marker, a model of a road curb and a model of an upright pedestrian.
- the computation of the depth map may be performed only locally around the detected feature in the image portions. The resolution of the computation of the depth map may be adjusted only to achieve an accuracy required based on the imaged feature.
- FIGS. 1 and 2 illustrate a system including a camera or image sensor mountable in a vehicle, according to an aspect of the present invention.
- FIG. 3 a shows a typical partial road scene, according to a feature of the present invention.
- FIG. 3 b shows further details of the same road scene as shown in FIG. 3 a , according to a feature of the present invention.
- FIG. 4 shows a method, according to a feature of the present invention.
- FIG. 5 a shows an associated (inverse) depth map, according to a feature of the present invention.
- FIG. 5 b shows the depth map patch around one candidate line, according to a feature of the present invention.
- FIGS. 5 c , 5 d , and 5 e show three models of vertical surface, L shape model and horizontal surface, according to a feature of the present invention.
- FIGS. 5 f , 5 g and 5 h show difference images between the depth map patch and the three models of FIGS. 5 c , 5 d , and 5 e respectively, according to a feature of the present invention.
- FIGS. 1 and 2 illustrate a system 16 including a camera or image sensor 12 mountable in a vehicle 18 , according to an aspect of the present invention.
- Image sensor 12 imaging a field of view in the forward direction typically delivers images in real time and the images may be captured in a time series of image frames 15 .
- An image processor 14 may be used to process image frames 15 simultaneously and/or in parallel to serve a number of driver assistance systems.
- Image sensor 12 is typically monochrome or black-white, i.e. without color separation.
- FIG. 1 and 2 illustrate a system 16 including a camera or image sensor 12 mountable in a vehicle 18 , according to an aspect of the present invention.
- Image sensor 12 imaging a field of view in the forward direction typically delivers images in real time and the images may be captured in a time series of image frames 15 .
- An image processor 14 may be used to process image frames 15 simultaneously and/or in parallel to serve a number of driver assistance systems.
- Image sensor 12 is typically monochrome or black-white, i.e
- image frames 15 may be used to serve a warning system 23 which may include collision warning 17 , lane keeping 19 , traffic sign recognition (TSR) 21 and three-dimensional structure, e.g. barrier and guard rai, 1 detection 401 .
- Image processor 14 is used to process image frames 15 to detect three-dimensional structure in the forward field of view of camera 12 .
- the terms “camera” and “image sensor” are used herein interchangeably.
- the detection of curbs, guard rails, structural barriers, e.g. concrete lane dividers is important for many driver assistance functions. Aspects of the present invention may include exchange of information between three-dimensional structure detection 401 and other driver assistance functions and/or systems including but not limited by FCW 17 and lane keeping 19 .
- a lane keeping 19 as part of system memory 23 may respond more strongly to a lane departure towards a guard rail or a barrier rather than a lane marker or even a white line.
- a Forward Collision Warning (FCW) system 19 may trigger sooner if the path to either side of in-path vehicle is blocked by a guard rail or another vehicle.
- FCW Forward Collision Warning
- object refers to an object in real space being viewed by a camera.
- a curb along the edge of a road and a lane marker in the road are examples of objects.
- image refers to the image of one or more objects in image space at the focal plane of camera 12 .
- Image coordinates (x,y) in small letters refer to image space and may be in arbitrary units or numbers of picture elements in the horizontal and vertical directions with the pixel dimensions assumed.
- image motion refers to motion of an image of an object in image space. From image frame 15 to a subsequent image frame 15 the points of the image of the object may map from one set of coordinates (x1,y1) to a different set of coordinates (x2,y2).
- image motion refers to the mapping of coordinates of an image from image frame to image frame or a function of the mapping.
- projection or “projecting” as used herein refers to camera or perspective projection from three dimensional space to a two dimensional image unless otherwise indicated by the context.
- embodiments of the present invention may be directed to an efficient method for detection of three-dimensional structures from depth map images using monocular cues.
- Monocular cues are used to detect candidate regions where a particular depth structure is expected and then the depth map is used verify the existence of the structure.
- the depth maps may be derived from a monocular sequence of images using ‘structure from motion’ (SfM) techniques or from multi-camera stereo.
- Embodiments of the present invention may include methods which search for particular three-dimensional structures or classes of three-dimensional structure.
- Three-dimensional objects have both distinctive three-dimensional structure and two-dimensional image characteristics.
- a sphere for example has three-dimensional shape and also appears in the image as a circle.
- the first step is to detect circles in the gray scale or color image which is a monocular cue for sphere detection. Once candidate circles are detected, the diameter of the circle in the image and the distance Z from the camera to the real-space sphere may define the three-dimensional diameter of the respective three-dimensional sphere.
- a depth map of the imaged circle may be computed using either SfM or multi-ocular (stereo vision) techniques.
- curb and barrier detection 401 has many uses in Driver Assistance Systems (DAS) such as lane keeping 19 , escape route determination, forward collision warning 17 , and automatic headlight control (AHC).
- DAS Driver Assistance Systems
- AHC automatic headlight control
- FIG. 3 a shows a typical partial road scene 10 , according to a feature of the present invention.
- Road scene 10 shows a partial image frame 15 (the sky portion of the image frame 15 is only partially shown) as captured by camera 12 mounted in host vehicle 18 .
- barriers 32 a and 32 b In road scene 10 are barriers 32 a and 32 b .
- Barrier 32 a has concrete base and vertical metal upright posts protruding out form the concrete base onto which is mounted a horizontal metal rail.
- Barrier 32 b appears to be just a barrier made from concrete.
- Road scene 10 also includes examples road markings 30 as well as lead vehicles 34 and oncoming vehicles 36 .
- FIG. 3 b shows further details of the same road scene as 10 shown in FIG. 3 a , according to a feature of the present invention.
- a gray scale value map 39 with gray scale values.
- Distinctive lines associated with lane markings, curbs, barriers, hill contours and the roof of an oncoming vehicle 36 in the road environment appear as almost straight in the image frame 15 as indicated by dotted lines 38 .
- the approximate position of the horizon 37 in image frame 15 is also shown by a solid line.
- Curbs and barriers 32 along the road may be associated with three-dimensional structures with an L shaped cross-section and with lines in the gray scale image. A line appears where the three-dimensional structure changes from horizontal road to vertical curb edge or barrier 32 .
- Image lines associated with lane markings 30 , curbs and barriers 32 in the road environment appear most often almost straight in the image. Even on curved roads, due to perspective, lane markings 30 and curbs appear straight closer to the vehicle, between twenty and thirty meters, and the curve appears only at longer ranges.
- the example as follows therefore uses features in the road environment which are represented by straight lines in image space as models.
- FIG. 4 shows a method 401 , according to a feature of the present invention.
- Method 401 may be used for detection of three dimensional obstacles such as barriers 32 a or 32 b for example.
- Method 401 is for straight lines but may be easily be extended to curved lines ‘parallel’ to the road skeleton or elongated lines in general as per techniques described for detecting the road skeleton in U.S. Pat. No. 7,151,996.
- image frames 15 are captured in the field of view of camera 12 .
- image features such as lines indicated by dotted lines 38 ( FIG. 3 b ) of a minimum length may be detected in image frame 15 .
- a strip in image frame 15 may be portioned around a line to give an imaged portion 409 .
- imaged portion 409 may be processed.
- a depth map 413 for each candidate image portions 409 may include a color or gray scale coordinate related to a function of distance (Z) from the camera 12 to the lines.
- three models (a), (b) and (c) may be computed by way of example:
- a horizontal surface model ( FIG. 5 e ) if both sides of the imaged line are road images (horizontal surface). For example, all points on the image row have the same depth value associated with the line pixel on that row.
- a vertical surface model ( FIG. 5 c ) if both sides of the imaged line are imaged barrier (vertical surface). For example, all points on the same image column have the same depth value associated with the line pixel on that column.
- An L-shaped model FIG. 5 d
- a barrier or curb model if the part of the patch inner from the imaged line is imaged road and patch outer of the imaged line is imaged vertical barrier 32 or curb. For example all points along a row inner from the line have a depth value associated with the depth value of the line pixel on that row AND all points along a column outer from the line have a depth value associated with the depth value of the line pixel on that column.
- depth map 413 of imaged portions 409 around an image feature, e.g. line 38 is compared with each of the three models (a), (b) and (c).
- the comparison may be performed by correlation or by computing the sum of squared differences (SSD) or sum of absolute differences (SAD) between actual depth map 413 of imaged portion 409 and each of the three models.
- the best score gives the correct real-space structure whether the imaged portions 409 include a three-dimensional structure: barrier or curb, a vertical surface or a road surface. If imaged portions 409 include an object which is an obstacle to the motion of vehicle 18 a warning may be issued to the driver of vehicle 18 in step 417 , or collision avoidance may be initiated by providing a control feature of host vehicle 18 such as automatic breaking or steering of host vehicle 18 to avoid the obstacle found in decision block 415 . Otherwise capturing of image frames 15 continues in step 403 .
- the depth value for the pixels on a line can be taken directly from a low resolution depth image.
- the gray scale line might not exactly correspond to the bend in the L shape (e.g. This happens when the curbstone is a molded L shape in concrete and the lane is asphalt), the same process can be repeated for line shifted a few pixels laterally in and out to get better coverage.
- the depth values of the vertical and horizontal models can also be derived by fitting the depth map 413 to planar models:
- x and y are the image column and row respectively and A, B, C, D are the derived model parameters.
- Robust fitting using random sample consensus (RANSAC) can be performed.
- FIG. 5 a shows an associated (inverse) depth map 50 , according to a feature of the present invention.
- Depth map 50 is computed for the full image frame 15 of FIGS. 3 a , 3 b and a few preceding image frames 15 . Light gray is close and dark gray is far away.
- FIG. 5 b shows the depth map patch around one candidate line, according to a feature of the present invention.
- the three models of vertical surface, L shape model and horizontal surface are show in FIGS. 5 c , 5 d , 5 e and the difference images between the depth map patch and the three models are shown respectively in FIGS. 5 f , 5 g and 5 h .
- Dark gray indicates small difference and lighter gray a large difference.
- the L shaped model shown is FIG. 5 d is clearly a better fit as shown by higher uniformity in FIG. 5 g.
- Robust estimate of the dominant ground plane can be performed using both lower spatial resolution and lower accuracy.
- higher accuracy depth maps can be computed along the predicted host vehicle 18 path. This can be used for obstacle detection. Higher accuracy still, can be used along the exact wheel tracks to be used for road profile estimation for adjusting active and semi-active suspensions.
- Sphere detection could use a rough distance and the diameter of the circle in the image to estimate the expected three-dimensional sphere diameter.
- the rough distance and the diameter of the circle in the image to estimate the expected three-dimensional sphere diameter values can be used to adjust the resolution used for the dense map to be sufficient to differentiate from a flat surface but not more than required.
- a similar approach is useful for pedestrian detection. If a suspect pedestrian target is detected with a suspect vehicle detected behind, the accuracy can be adjusted so as to be able to confirm the presence of both object and differentiate between them but not more than required.
- the depth map would be limited in information at spatial frequencies.
- the most practical solution is to use gray scale image information at the highest available resolution (eg. 640 ⁇ 480) to compute a depth map image of lower resolution (e.g. 320 ⁇ 240).
- depth map is an image that contains information relating to the distance Z of object surfaces from a viewpoint; the viewpoint generally being the position of the camera.
- structural barrier as used herein is a road barrier installed and extending along a road at the side of a road, at the median of a divided highway or as a lane divider.
- the structural barrier may be a concrete barrier, Jersey barrier, a metal guard rail or a cable barrier.
- structural barrier and vertical structural barrier are used herein interchangeably.
- the term “locally” as used herein refers to portioning an image to include only a detected feature and no other image features.
- high resolution refers to single pixel or sub-pixel resolution.
- acceleration refers to the determined position of an obstacle relative to a camera as compared with the true or actual position. of the obstacle.
- posts refers to an imageable vertical texture in a structural barrier and may include any vertical structure or surface texture, e.g painted stripes, or the anti-glare slats. Hence, the terms “vertical texture” and “post” are used herein interchangeably.
- capture refers to the real time recording and/or storage of image frames in a memory, for example volatile and/or non-volatile memory accessible by a processor.
- warping refers to a transform from image space to image space.
- homography refers to an invertible transformation from a projective space to itself that maps straight lines to straight lines.
- two images of the same planar surface in space are related by a homography assuming a pinhole camera model.
- relative scale change refers to the overall scale change in image coordinates dependent upon distance Z to a camera.
- the driver assistance system may include a camera operatively connected to a processor. Multiple image frames may be captured in the forward field of view of the camera. In the image frames, motion of images of the barrier are processed to detect the barrier.
- the camera may be a single camera. The motion of the images may be responsive to forward motion of the host vehicle and/or the motion of the images may be responsive to lateral motion of the host vehicle.
- the structural barrier may include multiple posts. Multiple linear image structures are hypothesized in an image frame as projections of the barrier onto the road surface and multiple vertical image coordinates are obtained respectively from the linear image structures.
- the linear image structures may be image lines which run parallel to the image of the road and intersect the vanishing point of the image of the lane markers.
- Multiple forward distances and corresponding lateral distances to the posts are computed based on the vertical image coordinates. Based on the known forward motion of the host vehicle and horizontal image coordinates of the linear image structures new horizontal image coordinates of the linear image structures are computed. The horizontal image coordinate in a second image frame of one of the images of the linear image structures is selected to align an image of one of the posts.
- forward distances from the host vehicle to the posts may be determined based on the motion of the images and the forward motion of the host vehicle. Lateral distances to the posts from the host vehicle may be determined from the forward distances and the horizontal image coordinates of the posts.
- Road plane lines at the lateral distances may be hypothesized to form multiple hypothesized road plane lines as projections of the vertical structural barrier onto the road surface. The hypothesized road plane lines at the lateral distances may be projected onto an image of the vertical structural barrier in an image frame.
- the correct road plane line is selected from the hypothesized road plane lines by aligning the correct road plane line with the image of the vertical structural barrier in the image frame.
- an image patch may be located in one of the image frames on an image line intersecting the vanishing point in the image frame.
- the image patch may be warped based on a vertical surface model.
- the vertical structural barrier may be detected by ascertaining that the patch is an image of the vertical structural barrier when points in columns of the patch scale vertically with host vehicle motion.
- the image patch may be warped based on a road surface model, and the patch may be an image of the road surface when points in rows of the patch scale horizontally with host vehicle motion.
- Various driver assistance systems may be provided for detecting a structural barrier extending along a road, The driver assistance system may be mountable in a host vehicle.
- the camera may capture multiple image frames in the forward field of view of the camera.
- a processor may process motion of images of the barrier in the image frames.
- the camera may be a single camera.
- the camera may be configured to view in the direction of forward motion of the host vehicle.
- the motion of the images may be responsive to forward motion of the host vehicle and/or the motion of the images may be responsive to lateral motion of the host vehicle.
- the motion of the images of the structural barrier may correlate with an image line in the direction of the vanishing point of the road, wherein the image line corresponds to a vertical projection of the structural barrier onto the road plane.
- the processor may be operable to hypothesize linear image structures as projections of the structural barrier onto the road plane to produce multiple hypotheses. Each of the hypotheses gives a lateral position of the barrier relative to the host vehicle. For each hypothesis, the lateral positions and host vehicle motion are used to predict image motion. The predicted image motion is compared to the actual image motion to verify the hypothesis and to derive the actual lateral position of the structural barrier relative to the host vehicle.
- Motion of the host vehicle may have a lateral component relative to the road direction and the image motion is of an image line in the image that is above the linear image structure.
- the image line may be that of the top of the barrier. Vertical motion or looming of the image line may be used to determine lateral distance between the host vehicle and the structural barrier to determine whether the image line is of the same lateral distance as the linear image structure (the barrier) or on the road surface farther away.
- the processor may be operable to hypothesize multiple linear image structures in an image frame as projections of the barrier onto the road surface and obtain thereby multiple vertical image coordinates respectively from the linear image structures.
- the processor may be operable to compute multiple forward distances and corresponding lateral distances to the posts based on the vertical image coordinates.
- the processor may be operable to compute new horizontal image coordinates of the linear image structures to select the horizontal image coordinate in a second image frame of one of the images of the linear image structures and to align an image of one of the posts.
- the processor may be operable to determine forward distances from the host vehicle to the posts based on the motion of the images and the forward motion of the host vehicle.
- the processor may be operable to compute lateral distances to the posts from the host vehicle from the forward distance and horizontal image coordinates x of the posts.
- the processor may be operable to hypothesize road plane lines at the lateral distances, to form multiple hypothesized road plane lines as projections of the structural barrier onto the road surface; to project the hypothesized road plane lines at the lateral distances onto an image of the structural barrier in an image frame.
- the processor may be operable to select the correct road plane line from the hypothesized road plane lines by aligning the correct road plane line with the image of the structural barrier in the image frame Alternatively for a barrier without substantial vertical image texture, the processor may be operable to locate in one of the image frames an image patch on an image line intersecting the vanishing point in an image frame, to warp said image patch based on a vertical surface model and to detect the structural barrier by ascertaining that the patch may be an image of the structural barrier when points in columns of the patch scale vertically with host vehicle motion. Or, the processor may be operable to ascertain that the patch may be an image of the road surface if or when points in rows of the patch scale horizontally with host vehicle motion.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
Abstract
Description
- The present application is a continuation in part application of U.S. patent application Ser. No. 13/237,163 filed on 20 Sep. 2011. The present application claims priority from U.S. provisional patent application 61/765,748 filed on 17 Feb. 2013 by the same inventors, the disclosures of which are incorporated by reference as if entirely set fort herein.
- 1. Technical Field
- The present disclosure relates to detection of three-dimensional structures from two-dimensional images and particularly related to application in driver assistance systems
- 2. Description of Related Art
- Structure-from-Motion (SfM) refers to methods for recovering three-dimensional information of a scene that has been projected onto the back focal plane of a camera. The structural information derived from a SfM algorithm may take the form of a set of projection matrices, one projection matrix per image frame, representing the relationship between a specific two-dimensional point in the image plane and its corresponding three-dimensional point. SfM algorithms rely on tracking specific image features from image frame to image frame to determine structural information concerning the scene.
- Similarly, stereo or multi-ocular disparity analysis may be used to determine three-dimensional points from two-dimensional images. Finding structure from motion presents a similar problem as finding structure from stereo vision. In both instances, the correspondence between images and the reconstruction of three-dimensional object is found.
- In three-dimensional computer graphics, a depth map is an image that contains information relating to the distance Z of surfaces of objects from a viewpoint; the viewpoint generally being the position of a camera.
- The computation of depth maps from multiple images, either from a motion sequence and/or from multiple cameras is the subject of extensive research and numerous systems have been demonstrated. These systems are capable of producing dense depth map information but at considerable computational expense. Depth maps may also lack the full spatial resolution of the original gray scale images.
- In multi-camera stereo, the epipolar geometry of the cameras is known and dense correspondences are computed by performing a search along the epipolar lines. Structure from Motion (SfM) techniques typically precede the dense epipolar search by computing the camera motion and epipolar geometry. In both cases, a smoothness function is explicitly or implicitly assumed so as to regularize the search and give robustness to noise and brightness changes. In the most straightforward approach, the search is performed by matching images patches rather than individual pixels.
- The depth error Ez in multi-camera stereo is a function of forward distance Z to the cameras, the correspondence error (Ed), image resolution (or focal length f in pixels) and the baseline b:
-
- In SfM depth error is similar, however, the baseline b is replaced by the motion of the camera. Since the correspondence error (Ed) is finite, at best on the order of 0.25 pixels in optimal conditions, it may be considered to compute the depth map using data from the highest available image resolution. However, due to the computational cost, it is often inhibitive to compute a depth map at such a large resolution.
- Thus there is a need for and it would be advantageous to have a driver assistance system and corresponding method adapted to detect three dimensional structures such as guardrails, curbs or other three dimensional objects in the road environment from depth map images while avoiding intensive computational overhead normally associated with computing depth maps to allow for real time processing in the driver assistance system.
- Various methods and systems are disclosed herein for detection of three dimensional obstacles. The systems are mountable in a host vehicle and a camera connectible to a processor. Multiple image frames are captured in the field of view of the camera. In the image frames, an imaged feature is detected of an object in the environment of the vehicle. The image frames are portioned locally around the imaged feature to produce imaged portions of the image frames including the imaged feature. The image frames are processed to compute a depth map locally around the detected imaged feature in the image portions. The depth map includes an image of the feature with a color or grayscale coordinate related to a function of distance from the camera to the object. Responsive to the depth map, it is determined if the object is an obstacle to the motion of the vehicle. The object may be represented with multiple models. Multiple model depth maps may be computed respectively of the multiple models. The depth map of the detected feature may be compared with the model depth maps and based on the comparison, it may be determined that the object is an obstacle or not an obstacle to the motion of the vehicle. The models may be a horizontal planar model, a vertical planar model, a mixed model including horizontal and vertical portions, a spherical model, a circular model, a model of a guard rail, a model of lane marker, a model of a road curb and a model of an upright pedestrian. The computation of the depth map may be performed only locally around the detected feature in the image portions. The resolution of the computation of the depth map may be adjusted only to achieve an accuracy required based on the imaged feature.
- The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:
-
FIGS. 1 and 2 illustrate a system including a camera or image sensor mountable in a vehicle, according to an aspect of the present invention. -
FIG. 3a shows a typical partial road scene, according to a feature of the present invention. -
FIG. 3b shows further details of the same road scene as shown inFIG. 3a , according to a feature of the present invention. -
FIG. 4 shows a method, according to a feature of the present invention. -
FIG. 5a shows an associated (inverse) depth map, according to a feature of the present invention. -
FIG. 5b shows the depth map patch around one candidate line, according to a feature of the present invention. -
FIGS. 5c, 5d, and 5e show three models of vertical surface, L shape model and horizontal surface, according to a feature of the present invention. -
FIGS. 5f, 5g and 5h show difference images between the depth map patch and the three models ofFIGS. 5c, 5d, and 5e respectively, according to a feature of the present invention. - Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures.
- Referring now to the drawings, reference is now made to
FIGS. 1 and 2 which illustrate asystem 16 including a camera orimage sensor 12 mountable in avehicle 18, according to an aspect of the present invention.Image sensor 12, imaging a field of view in the forward direction typically delivers images in real time and the images may be captured in a time series of image frames 15. Animage processor 14 may be used to process image frames 15 simultaneously and/or in parallel to serve a number of driver assistance systems.Image sensor 12 is typically monochrome or black-white, i.e. without color separation. By way of example inFIG. 2 , image frames 15 may be used to serve awarning system 23 which may include collision warning 17, lane keeping 19, traffic sign recognition (TSR) 21 and three-dimensional structure, e.g. barrier and guard rai, 1detection 401.Image processor 14 is used to process image frames 15 to detect three-dimensional structure in the forward field of view ofcamera 12. The terms “camera” and “image sensor” are used herein interchangeably. The detection of curbs, guard rails, structural barriers, e.g. concrete lane dividers is important for many driver assistance functions. Aspects of the present invention may include exchange of information between three-dimensional structure detection 401 and other driver assistance functions and/or systems including but not limited byFCW 17 and lane keeping 19. For example, a lane keeping 19 as part ofsystem memory 23, may respond more strongly to a lane departure towards a guard rail or a barrier rather than a lane marker or even a white line. A Forward Collision Warning (FCW)system 19 may trigger sooner if the path to either side of in-path vehicle is blocked by a guard rail or another vehicle. - The term “object” as used herein refers to an object in real space being viewed by a camera. A curb along the edge of a road and a lane marker in the road are examples of objects. The term “image” refers to the image of one or more objects in image space at the focal plane of
camera 12. Image coordinates (x,y) in small letters refer to image space and may be in arbitrary units or numbers of picture elements in the horizontal and vertical directions with the pixel dimensions assumed. The term “image motion” refers to motion of an image of an object in image space. Fromimage frame 15 to asubsequent image frame 15 the points of the image of the object may map from one set of coordinates (x1,y1) to a different set of coordinates (x2,y2). The term “image motion” refers to the mapping of coordinates of an image from image frame to image frame or a function of the mapping. The term “projection” or “projecting” as used herein refers to camera or perspective projection from three dimensional space to a two dimensional image unless otherwise indicated by the context. - Before explaining embodiments of the invention in detail, it is to be understood that the invention is not limited in its application to the details of design and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
- By way of introduction, embodiments of the present invention may be directed to an efficient method for detection of three-dimensional structures from depth map images using monocular cues. Monocular cues are used to detect candidate regions where a particular depth structure is expected and then the depth map is used verify the existence of the structure. The depth maps may be derived from a monocular sequence of images using ‘structure from motion’ (SfM) techniques or from multi-camera stereo.
- Embodiments of the present invention may include methods which search for particular three-dimensional structures or classes of three-dimensional structure. Three-dimensional objects have both distinctive three-dimensional structure and two-dimensional image characteristics. A sphere for example has three-dimensional shape and also appears in the image as a circle. In the ‘sphere’ example, the first step is to detect circles in the gray scale or color image which is a monocular cue for sphere detection. Once candidate circles are detected, the diameter of the circle in the image and the distance Z from the camera to the real-space sphere may define the three-dimensional diameter of the respective three-dimensional sphere. A depth map of the imaged circle may be computed using either SfM or multi-ocular (stereo vision) techniques. One may compute a first model associated with a three-dimensional depth model of a convex sphere, a second model associated with a flat circle, and possibly a third model three-dimensional depth model associated with a concave sphere. By comparing the computed depth map to the three models one can simply and inexpensively determine whether or not the imaged circle is a sphere in three-dimensional space.
- The next section shows in detail how the same technique can be applied to curb and barrier detection by a
monocular camera 12 mounted onvehicle 18. As discussed above, curb andbarrier detection 401 has many uses in Driver Assistance Systems (DAS) such as lane keeping 19, escape route determination, forward collision warning 17, and automatic headlight control (AHC). - Reference is now made to
FIG. 3a which shows a typicalpartial road scene 10, according to a feature of the present invention.Road scene 10 shows a partial image frame 15 (the sky portion of theimage frame 15 is only partially shown) as captured bycamera 12 mounted inhost vehicle 18. Inroad scene 10 arebarriers Barrier 32 a has concrete base and vertical metal upright posts protruding out form the concrete base onto which is mounted a horizontal metal rail.Barrier 32 b appears to be just a barrier made from concrete.Road scene 10 also includesexamples road markings 30 as well aslead vehicles 34 and oncomingvehicles 36. - Reference is now made to
FIG. 3b which shows further details of the same road scene as 10 shown inFIG. 3a , according to a feature of the present invention. To the right ofroad scene 10 is shown a grayscale value map 39 with gray scale values. Distinctive lines associated with lane markings, curbs, barriers, hill contours and the roof of an oncomingvehicle 36 in the road environment appear as almost straight in theimage frame 15 as indicated bydotted lines 38. The approximate position of thehorizon 37 inimage frame 15 is also shown by a solid line. - Detecting Curbs and Barriers Structures from Monocular Cues
- Curbs and barriers 32 along the road may be associated with three-dimensional structures with an L shaped cross-section and with lines in the gray scale image. A line appears where the three-dimensional structure changes from horizontal road to vertical curb edge or barrier 32.
- Image lines associated with
lane markings 30, curbs and barriers 32 in the road environment appear most often almost straight in the image. Even on curved roads, due to perspective,lane markings 30 and curbs appear straight closer to the vehicle, between twenty and thirty meters, and the curve appears only at longer ranges. The example as follows therefore uses features in the road environment which are represented by straight lines in image space as models. - Reference is now made to
FIG. 4 which shows amethod 401, according to a feature of the present invention.Method 401 may be used for detection of three dimensional obstacles such asbarriers Method 401 is for straight lines but may be easily be extended to curved lines ‘parallel’ to the road skeleton or elongated lines in general as per techniques described for detecting the road skeleton in U.S. Pat. No. 7,151,996. Instep 403, image frames 15 are captured in the field of view ofcamera 12. Instep 405, image features such as lines indicated by dotted lines 38 (FIG. 3b ) of a minimum length may be detected inimage frame 15. Lines which are below thehorizon 37 are candidates for being on the road, at the bottom of barriers 32 or curb. Instep 407, a strip inimage frame 15 may be portioned around a line to give an imagedportion 409. Instep 411, imagedportion 409 may be processed. Adepth map 413 for eachcandidate image portions 409 may include a color or gray scale coordinate related to a function of distance (Z) from thecamera 12 to the lines. For eachcandidate image portion 409, three models (a), (b) and (c) may be computed by way of example: - (a) A horizontal surface model (
FIG. 5e ) if both sides of the imaged line are road images (horizontal surface). For example, all points on the image row have the same depth value associated with the line pixel on that row.
(b) A vertical surface model (FIG. 5c ) if both sides of the imaged line are imaged barrier (vertical surface). For example, all points on the same image column have the same depth value associated with the line pixel on that column.
(c) An L-shaped model (FIG. 5d ) A barrier or curb model if the part of the patch inner from the imaged line is imaged road and patch outer of the imaged line is imaged vertical barrier 32 or curb. For example all points along a row inner from the line have a depth value associated with the depth value of the line pixel on that row AND all points along a column outer from the line have a depth value associated with the depth value of the line pixel on that column. - Further in
decision block 415,depth map 413 of imagedportions 409 around an image feature,e.g. line 38 is compared with each of the three models (a), (b) and (c). The comparison may be performed by correlation or by computing the sum of squared differences (SSD) or sum of absolute differences (SAD) betweenactual depth map 413 of imagedportion 409 and each of the three models. - In
decision 415, the best score gives the correct real-space structure whether the imagedportions 409 include a three-dimensional structure: barrier or curb, a vertical surface or a road surface. If imagedportions 409 include an object which is an obstacle to the motion of vehicle 18 a warning may be issued to the driver ofvehicle 18 instep 417, or collision avoidance may be initiated by providing a control feature ofhost vehicle 18 such as automatic breaking or steering ofhost vehicle 18 to avoid the obstacle found indecision block 415. Otherwise capturing of image frames 15 continues instep 403. - The depth value for the pixels on a line can be taken directly from a low resolution depth image. Alternatively one can perform a robust fit of
depth map 413 values to a line in three-dimensional space and then the depth value of the three-dimensional line may be used. - Because the gray scale line might not exactly correspond to the bend in the L shape (e.g. This happens when the curbstone is a molded L shape in concrete and the lane is asphalt), the same process can be repeated for line shifted a few pixels laterally in and out to get better coverage.
- The depth values of the vertical and horizontal models can also be derived by fitting the
depth map 413 to planar models: -
- for vertical surfaces or:
-
- for horizontal surfaces, where x and y are the image column and row respectively and A, B, C, D are the derived model parameters. Robust fitting using random sample consensus (RANSAC) can be performed.
- Reference is now made to
FIG. 5a which shows an associated (inverse)depth map 50, according to a feature of the present invention.Depth map 50 is computed for thefull image frame 15 ofFIGS. 3a, 3b and a few preceding image frames 15. Light gray is close and dark gray is far away. -
FIG. 5b shows the depth map patch around one candidate line, according to a feature of the present invention. The three models of vertical surface, L shape model and horizontal surface are show inFIGS. 5c, 5d, 5e and the difference images between the depth map patch and the three models are shown respectively inFIGS. 5f, 5g and 5h . Dark gray indicates small difference and lighter gray a large difference. The L shaped model shown isFIG. 5d is clearly a better fit as shown by higher uniformity inFIG. 5 g. - As discussed earlier in the background, computing a dense depth map using full resolution gray scale data can be very computationally expensive, using the full resolution gives better depth accuracy however, such accuracy is not required uniformly over the whole image and for all applications.
- There are two more efficient approaches:
- 1. Compute the dense depth map only where necessary
If the application is curb detection then one need only compute the dense depth map around the line candidates (steps 407 and 411). Starting from the innermost line candidates, compute a dense depth map using the highest resolution available only for the patches around those candidates. If they are determined to be ‘road’ then move to the next outer candidate. If they are determined to be L or ‘vertical’ there is no need to compute the accurate dense depth map for candidates further out.
2. Adjust the accuracy of the depth map to the task required
Curb detection requires very accurate depth map computation. However, such accuracy is often unnecessary for other applications. Detecting barriers 32 of 0.5 m height or above can be performed using lower accuracy depth maps 413. Higher accuracy would be applied to line candidates for curb detection. - There are other examples, Robust estimate of the dominant ground plane can be performed using both lower spatial resolution and lower accuracy. After the dominant ground plane is computed, higher accuracy depth maps can be computed along the predicted
host vehicle 18 path. This can be used for obstacle detection. Higher accuracy still, can be used along the exact wheel tracks to be used for road profile estimation for adjusting active and semi-active suspensions. - Sphere detection could use a rough distance and the diameter of the circle in the image to estimate the expected three-dimensional sphere diameter. The rough distance and the diameter of the circle in the image to estimate the expected three-dimensional sphere diameter values can be used to adjust the resolution used for the dense map to be sufficient to differentiate from a flat surface but not more than required.
- A similar approach is useful for pedestrian detection. If a suspect pedestrian target is detected with a suspect vehicle detected behind, the accuracy can be adjusted so as to be able to confirm the presence of both object and differentiate between them but not more than required.
- Furthermore, due to the smoothness constraints used in the dense correspondence computation the depth map would be limited in information at spatial frequencies. Thus, the most practical solution is to use gray scale image information at the highest available resolution (eg. 640×480) to compute a depth map image of lower resolution (e.g. 320×240).
- The term “depth map” as used herein is an image that contains information relating to the distance Z of object surfaces from a viewpoint; the viewpoint generally being the position of the camera.
- The term “structural barrier” as used herein is a road barrier installed and extending along a road at the side of a road, at the median of a divided highway or as a lane divider. The structural barrier may be a concrete barrier, Jersey barrier, a metal guard rail or a cable barrier. The terms “structural barrier” and “vertical structural barrier” are used herein interchangeably.
- The term “locally” as used herein refers to portioning an image to include only a detected feature and no other image features.
- The term “high resolution” as used herein refers to single pixel or sub-pixel resolution.
- The term “accurate” as used herein refers to the determined position of an obstacle relative to a camera as compared with the true or actual position. of the obstacle.
- The term “posts” as used herein refers to an imageable vertical texture in a structural barrier and may include any vertical structure or surface texture, e.g painted stripes, or the anti-glare slats. Hence, the terms “vertical texture” and “post” are used herein interchangeably.
- The term “capture” as used herein refers to the real time recording and/or storage of image frames in a memory, for example volatile and/or non-volatile memory accessible by a processor.
- The term “warping” as used herein refers to a transform from image space to image space.
- The term “homography” as used herein refers to an invertible transformation from a projective space to itself that maps straight lines to straight lines. In the field of computer vision, two images of the same planar surface in space are related by a homography assuming a pinhole camera model.
- The term “relative scale change” as used herein refers to the overall scale change in image coordinates dependent upon distance Z to a camera.
- The indefinite articles “a” and “an” is used herein, such as “a depth map”, “a line” have the meaning of “one or more” that is “one or more depth maps” or “one or more lines”.
- Summary of US Patent Publication 2012/0069185, Application Ser. No. 13/237,163
- Various methods are disclosed herein for detecting a structural barrier extending along a road. The methods are performable by a driver assistance system mountable in a host vehicle. The driver assistance system may include a camera operatively connected to a processor. Multiple image frames may be captured in the forward field of view of the camera. In the image frames, motion of images of the barrier are processed to detect the barrier. The camera may be a single camera. The motion of the images may be responsive to forward motion of the host vehicle and/or the motion of the images may be responsive to lateral motion of the host vehicle.
- The structural barrier may include multiple posts. Multiple linear image structures are hypothesized in an image frame as projections of the barrier onto the road surface and multiple vertical image coordinates are obtained respectively from the linear image structures. The linear image structures may be image lines which run parallel to the image of the road and intersect the vanishing point of the image of the lane markers.
- Multiple forward distances and corresponding lateral distances to the posts are computed based on the vertical image coordinates. Based on the known forward motion of the host vehicle and horizontal image coordinates of the linear image structures new horizontal image coordinates of the linear image structures are computed. The horizontal image coordinate in a second image frame of one of the images of the linear image structures is selected to align an image of one of the posts.
- Alternatively, for each of the posts, forward distances from the host vehicle to the posts may be determined based on the motion of the images and the forward motion of the host vehicle. Lateral distances to the posts from the host vehicle may be determined from the forward distances and the horizontal image coordinates of the posts. Road plane lines at the lateral distances may be hypothesized to form multiple hypothesized road plane lines as projections of the vertical structural barrier onto the road surface. The hypothesized road plane lines at the lateral distances may be projected onto an image of the vertical structural barrier in an image frame. The correct road plane line is selected from the hypothesized road plane lines by aligning the correct road plane line with the image of the vertical structural barrier in the image frame.
- Alternatively for a barrier without substantial vertical image texture, an image patch may be located in one of the image frames on an image line intersecting the vanishing point in the image frame. The image patch may be warped based on a vertical surface model. The vertical structural barrier may be detected by ascertaining that the patch is an image of the vertical structural barrier when points in columns of the patch scale vertically with host vehicle motion. Alternatively, the image patch may be warped based on a road surface model, and the patch may be an image of the road surface when points in rows of the patch scale horizontally with host vehicle motion. Various driver assistance systems may be provided for detecting a structural barrier extending along a road, The driver assistance system may be mountable in a host vehicle.
- The camera may capture multiple image frames in the forward field of view of the camera. A processor may process motion of images of the barrier in the image frames. The camera may be a single camera. The camera may be configured to view in the direction of forward motion of the host vehicle. The motion of the images may be responsive to forward motion of the host vehicle and/or the motion of the images may be responsive to lateral motion of the host vehicle.
- The motion of the images of the structural barrier may correlate with an image line in the direction of the vanishing point of the road, wherein the image line corresponds to a vertical projection of the structural barrier onto the road plane. The processor may be operable to hypothesize linear image structures as projections of the structural barrier onto the road plane to produce multiple hypotheses. Each of the hypotheses gives a lateral position of the barrier relative to the host vehicle. For each hypothesis, the lateral positions and host vehicle motion are used to predict image motion. The predicted image motion is compared to the actual image motion to verify the hypothesis and to derive the actual lateral position of the structural barrier relative to the host vehicle. Motion of the host vehicle may have a lateral component relative to the road direction and the image motion is of an image line in the image that is above the linear image structure.
- The image line may be that of the top of the barrier. Vertical motion or looming of the image line may be used to determine lateral distance between the host vehicle and the structural barrier to determine whether the image line is of the same lateral distance as the linear image structure (the barrier) or on the road surface farther away. The processor may be operable to hypothesize multiple linear image structures in an image frame as projections of the barrier onto the road surface and obtain thereby multiple vertical image coordinates respectively from the linear image structures. The processor may be operable to compute multiple forward distances and corresponding lateral distances to the posts based on the vertical image coordinates. Based on the known forward motion of the host vehicle and horizontal image coordinates of the linear image structures, the processor may be operable to compute new horizontal image coordinates of the linear image structures to select the horizontal image coordinate in a second image frame of one of the images of the linear image structures and to align an image of one of the posts.
- Alternatively, for each of the posts, the processor may be operable to determine forward distances from the host vehicle to the posts based on the motion of the images and the forward motion of the host vehicle. The processor may be operable to compute lateral distances to the posts from the host vehicle from the forward distance and horizontal image coordinates x of the posts. The processor may be operable to hypothesize road plane lines at the lateral distances, to form multiple hypothesized road plane lines as projections of the structural barrier onto the road surface; to project the hypothesized road plane lines at the lateral distances onto an image of the structural barrier in an image frame. The processor may be operable to select the correct road plane line from the hypothesized road plane lines by aligning the correct road plane line with the image of the structural barrier in the image frame Alternatively for a barrier without substantial vertical image texture, the processor may be operable to locate in one of the image frames an image patch on an image line intersecting the vanishing point in an image frame, to warp said image patch based on a vertical surface model and to detect the structural barrier by ascertaining that the patch may be an image of the structural barrier when points in columns of the patch scale vertically with host vehicle motion. Or, the processor may be operable to ascertain that the patch may be an image of the road surface if or when points in rows of the patch scale horizontally with host vehicle motion.
- Although selected embodiments of the present invention have been shown and described, it is to be understood the present invention is not limited to the described embodiments. Instead, it is to be appreciated that changes may be made to these embodiments, the scope of which is defined by the claims and the equivalents thereof
Claims (10)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/180,548 US10776635B2 (en) | 2010-09-21 | 2014-02-14 | Monocular cued detection of three-dimensional structures from depth images |
US17/011,798 US11062155B2 (en) | 2010-09-21 | 2020-09-03 | Monocular cued detection of three-dimensional structures from depth images |
US17/366,377 US11763571B2 (en) | 2010-09-21 | 2021-07-02 | Monocular cued detection of three-dimensional structures from depth images |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US38512210P | 2010-09-21 | 2010-09-21 | |
US13/237,163 US9280711B2 (en) | 2010-09-21 | 2011-09-20 | Barrier and guardrail detection using a single camera |
US201361765748P | 2013-02-17 | 2013-02-17 | |
US14/180,548 US10776635B2 (en) | 2010-09-21 | 2014-02-14 | Monocular cued detection of three-dimensional structures from depth images |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/237,163 Continuation-In-Part US9280711B2 (en) | 2010-09-21 | 2011-09-20 | Barrier and guardrail detection using a single camera |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/011,798 Continuation US11062155B2 (en) | 2010-09-21 | 2020-09-03 | Monocular cued detection of three-dimensional structures from depth images |
Publications (3)
Publication Number | Publication Date |
---|---|
US20140160244A1 US20140160244A1 (en) | 2014-06-12 |
US20190294893A9 true US20190294893A9 (en) | 2019-09-26 |
US10776635B2 US10776635B2 (en) | 2020-09-15 |
Family
ID=50880528
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/180,548 Active 2034-03-21 US10776635B2 (en) | 2010-09-21 | 2014-02-14 | Monocular cued detection of three-dimensional structures from depth images |
US17/011,798 Active US11062155B2 (en) | 2010-09-21 | 2020-09-03 | Monocular cued detection of three-dimensional structures from depth images |
US17/366,377 Active US11763571B2 (en) | 2010-09-21 | 2021-07-02 | Monocular cued detection of three-dimensional structures from depth images |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/011,798 Active US11062155B2 (en) | 2010-09-21 | 2020-09-03 | Monocular cued detection of three-dimensional structures from depth images |
US17/366,377 Active US11763571B2 (en) | 2010-09-21 | 2021-07-02 | Monocular cued detection of three-dimensional structures from depth images |
Country Status (1)
Country | Link |
---|---|
US (3) | US10776635B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180131924A1 (en) * | 2016-11-07 | 2018-05-10 | Samsung Electronics Co., Ltd. | Method and apparatus for generating three-dimensional (3d) road model |
US20180339670A1 (en) * | 2017-05-24 | 2018-11-29 | Toyota Jidosha Kabushiki Kaisha | Collision preventing control device |
US10777081B2 (en) * | 2017-03-13 | 2020-09-15 | Toyota Jidosha Kabushiki Kaisha | Collision preventing control device |
CN112258509A (en) * | 2020-12-18 | 2021-01-22 | 之江实验室 | Method for extracting white line in green field under complex light condition |
WO2021101045A1 (en) * | 2019-11-20 | 2021-05-27 | Samsung Electronics Co., Ltd. | Electronic apparatus and method for controlling thereof |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5877897A (en) | 1993-02-26 | 1999-03-02 | Donnelly Corporation | Automatic rearview mirror, vehicle lighting control and vehicle interior monitoring system using a photosensor array |
US6822563B2 (en) | 1997-09-22 | 2004-11-23 | Donnelly Corporation | Vehicle imaging system with accessory control |
US7655894B2 (en) | 1996-03-25 | 2010-02-02 | Donnelly Corporation | Vehicular image sensing system |
WO2003093857A2 (en) | 2002-05-03 | 2003-11-13 | Donnelly Corporation | Object detection system for vehicle |
US7526103B2 (en) | 2004-04-15 | 2009-04-28 | Donnelly Corporation | Imaging system for vehicle |
WO2008024639A2 (en) | 2006-08-11 | 2008-02-28 | Donnelly Corporation | Automatic headlamp control system |
US10776635B2 (en) | 2010-09-21 | 2020-09-15 | Mobileye Vision Technologies Ltd. | Monocular cued detection of three-dimensional structures from depth images |
US20140363073A1 (en) * | 2013-06-11 | 2014-12-11 | Microsoft Corporation | High-performance plane detection with depth camera data |
US9285230B1 (en) * | 2013-12-20 | 2016-03-15 | Google Inc. | Methods and systems for detecting road curbs |
JP5989701B2 (en) * | 2014-03-24 | 2016-09-07 | トヨタ自動車株式会社 | Boundary detection device and boundary detection method |
DE102014114221A1 (en) * | 2014-09-30 | 2016-03-31 | Valeo Schalter Und Sensoren Gmbh | Method for detecting an object in a surrounding area of a motor vehicle, driver assistance system and motor vehicle |
US9904855B2 (en) * | 2014-11-13 | 2018-02-27 | Nec Corporation | Atomic scenes for scalable traffic scene recognition in monocular videos |
US9870437B2 (en) * | 2014-11-24 | 2018-01-16 | Google Llc | Systems and methods for detecting and modeling curb curves in complex urban scenes |
EP4030378A1 (en) | 2015-05-10 | 2022-07-20 | Mobileye Vision Technologies Ltd. | Road profile along a predicted path |
US9898671B2 (en) | 2015-05-11 | 2018-02-20 | Magna Electronics Inc. | Vehicle vision system with structure and motion estimation |
US20170032676A1 (en) * | 2015-07-30 | 2017-02-02 | Illinois Institute Of Technology | System for detecting pedestrians by fusing color and depth information |
US9869863B2 (en) * | 2015-10-05 | 2018-01-16 | Unity IPR ApS | Systems and methods for processing a 2D video |
EP3217374A1 (en) * | 2016-03-10 | 2017-09-13 | Volvo Car Corporation | Method and system for estimating a boundary of a road technical field |
EP3229173B1 (en) * | 2016-04-05 | 2018-10-03 | Conti Temic microelectronic GmbH | Method and apparatus for determining a traversable path |
CN107610152B (en) * | 2017-08-31 | 2020-02-28 | 杭州视氪科技有限公司 | Passage detection method for avoiding water surface and obstacles |
EP3495219B1 (en) * | 2017-12-11 | 2023-07-05 | Volvo Car Corporation | Path prediction for a vehicle |
CN110706271B (en) * | 2019-09-30 | 2022-02-15 | 清华大学 | Vehicle-mounted vision real-time multi-vehicle-mounted target transverse and longitudinal distance estimation method |
US11483547B2 (en) * | 2019-12-04 | 2022-10-25 | Nxp Usa, Inc. | System and method for adaptive correction factor subsampling for geometric correction in an image processing system |
Family Cites Families (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6215898B1 (en) * | 1997-04-15 | 2001-04-10 | Interval Research Corporation | Data processing system and method |
AU2001253619A1 (en) * | 2000-04-14 | 2001-10-30 | Mobileye, Inc. | Generating a model of the path of a roadway from an image recorded by a camera |
ATE321689T1 (en) * | 2002-04-19 | 2006-04-15 | Iee Sarl | SAFETY DEVICE FOR A VEHICLE |
US7489812B2 (en) * | 2002-06-07 | 2009-02-10 | Dynamic Digital Depth Research Pty Ltd. | Conversion and encoding techniques |
US7263209B2 (en) * | 2003-06-13 | 2007-08-28 | Sarnoff Corporation | Vehicular vision system |
US7660436B2 (en) * | 2003-06-13 | 2010-02-09 | Sarnoff Corporation | Stereo-vision based imminent collision detection |
US7117090B2 (en) * | 2003-08-28 | 2006-10-03 | Siemens Aktiengesellschaft | Moving vehicle in cuboidal panorama |
US8134637B2 (en) * | 2004-01-28 | 2012-03-13 | Microsoft Corporation | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing |
WO2007043036A1 (en) * | 2005-10-11 | 2007-04-19 | Prime Sense Ltd. | Method and system for object reconstruction |
US20100259539A1 (en) * | 2005-07-21 | 2010-10-14 | Nikolaos Papanikolopoulos | Camera placement and virtual-scene construction for observability and activity recognition |
WO2007124502A2 (en) * | 2006-04-21 | 2007-11-01 | Sarnoff Corporation | Apparatus and method for object detection and tracking and roadway awareness using stereo cameras |
US7786898B2 (en) * | 2006-05-31 | 2010-08-31 | Mobileye Technologies Ltd. | Fusion of far infrared and visible images in enhanced obstacle detection in automotive applications |
DE602008005318D1 (en) | 2007-08-22 | 2011-04-14 | Honda Res Inst Europe Gmbh | ESTIMATE THE PROPER MOVEMENT OF OBJECTS USING OPTICAL RIVER, KINEMATICS AND DEEP INFORMATION |
DE102008004632A1 (en) * | 2008-01-16 | 2009-07-23 | Robert Bosch Gmbh | Device and method for measuring a parking space |
KR101188588B1 (en) * | 2008-03-27 | 2012-10-08 | 주식회사 만도 | Monocular Motion Stereo-Based Free Parking Space Detection Apparatus and Method |
US8442355B2 (en) * | 2008-05-23 | 2013-05-14 | Samsung Electronics Co., Ltd. | System and method for generating a multi-dimensional image |
DE102009009047A1 (en) * | 2009-02-16 | 2010-08-19 | Daimler Ag | Method for object detection |
BRPI1013339B1 (en) * | 2009-02-19 | 2021-09-21 | Interdigital Madison Patent Holdings | METHOD AND APPARATUS FOR ENCODING VIDEO, METHOD AND APPARATUS FOR DECODING VIDEO, VIDEO SIGNAL FORMATTED TO INCLUDE INFORMATION AND MEDIA READABLE BY PROCESSOR |
EP2256690B1 (en) | 2009-05-29 | 2016-08-17 | Honda Research Institute Europe GmbH | Object motion detection system based on combining 3D warping techniques and a proper object motion detection |
JP4587237B1 (en) * | 2009-06-17 | 2010-11-24 | Necカシオモバイルコミュニケーションズ株式会社 | Terminal device and program |
US8744661B2 (en) * | 2009-10-21 | 2014-06-03 | Berthold K. P. Horn | Method and apparatus for reducing motor vehicle traffic flow instabilities and increasing vehicle throughput |
US8861842B2 (en) * | 2010-02-05 | 2014-10-14 | Sri International | Method and apparatus for real-time pedestrian detection for urban driving |
US20120056982A1 (en) * | 2010-09-08 | 2012-03-08 | Microsoft Corporation | Depth camera based on structured light and stereo vision |
US10776635B2 (en) * | 2010-09-21 | 2020-09-15 | Mobileye Vision Technologies Ltd. | Monocular cued detection of three-dimensional structures from depth images |
US8432434B2 (en) | 2011-07-08 | 2013-04-30 | Mitsubishi Electric Research Laboratories, Inc. | Camera and method for focus based depth reconstruction of dynamic scenes |
-
2014
- 2014-02-14 US US14/180,548 patent/US10776635B2/en active Active
-
2020
- 2020-09-03 US US17/011,798 patent/US11062155B2/en active Active
-
2021
- 2021-07-02 US US17/366,377 patent/US11763571B2/en active Active
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180131924A1 (en) * | 2016-11-07 | 2018-05-10 | Samsung Electronics Co., Ltd. | Method and apparatus for generating three-dimensional (3d) road model |
US10863166B2 (en) * | 2016-11-07 | 2020-12-08 | Samsung Electronics Co., Ltd. | Method and apparatus for generating three-dimensional (3D) road model |
US20210058608A1 (en) * | 2016-11-07 | 2021-02-25 | Samsung Electronics Co., Ltd. | Method and apparatus for generating three-dimensional (3d) road model |
US11632536B2 (en) * | 2016-11-07 | 2023-04-18 | Samsung Electronics Co., Ltd. | Method and apparatus for generating three-dimensional (3D) road model |
US10777081B2 (en) * | 2017-03-13 | 2020-09-15 | Toyota Jidosha Kabushiki Kaisha | Collision preventing control device |
US20180339670A1 (en) * | 2017-05-24 | 2018-11-29 | Toyota Jidosha Kabushiki Kaisha | Collision preventing control device |
WO2021101045A1 (en) * | 2019-11-20 | 2021-05-27 | Samsung Electronics Co., Ltd. | Electronic apparatus and method for controlling thereof |
US11417007B2 (en) | 2019-11-20 | 2022-08-16 | Samsung Electronics Co., Ltd. | Electronic apparatus and method for controlling thereof |
CN112258509A (en) * | 2020-12-18 | 2021-01-22 | 之江实验室 | Method for extracting white line in green field under complex light condition |
Also Published As
Publication number | Publication date |
---|---|
US20140160244A1 (en) | 2014-06-12 |
US20210334557A1 (en) | 2021-10-28 |
US20200401822A1 (en) | 2020-12-24 |
US11062155B2 (en) | 2021-07-13 |
US11763571B2 (en) | 2023-09-19 |
US10776635B2 (en) | 2020-09-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11062155B2 (en) | Monocular cued detection of three-dimensional structures from depth images | |
US11170466B2 (en) | Dense structure from motion | |
US11854272B2 (en) | Hazard detection from a camera in a scene with moving shadows | |
US11087148B2 (en) | Barrier and guardrail detection using a single camera | |
US10834324B2 (en) | Image distortion correction of a camera with a rolling shutter | |
US9846812B2 (en) | Image recognition system for a vehicle and corresponding method | |
EP2993654B1 (en) | Method and system for forward collision warning | |
US20090041337A1 (en) | Image processing apparatus and method | |
CN107423675B (en) | Advanced warning system for forward collision warning of traps and pedestrians | |
CN107491065B (en) | Method and apparatus for detecting side surface of object using ground boundary information of obstacle | |
EP3803790B1 (en) | Motion segmentation in video from non-stationary cameras | |
JP6552448B2 (en) | Vehicle position detection device, vehicle position detection method, and computer program for vehicle position detection | |
Ponsa et al. | On-board image-based vehicle detection and tracking | |
JP4144300B2 (en) | Plane estimation method and object detection apparatus using stereo image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MOBILEYE TECHNOLOGIES LIMITED, CYPRUS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BERBERIAN, ODED;STEIN, GIDEON;REEL/FRAME:032218/0623 Effective date: 20140213 |
|
AS | Assignment |
Owner name: MOBILEYE VISION TECHNOLOGIES LTD., ISRAEL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MOBILEYE TECHNOLOGIES LIMITED;REEL/FRAME:034305/0993 Effective date: 20140723 |
|
STCV | Information on status: appeal procedure |
Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS |
|
FEPP | Fee payment procedure |
Free format text: PETITION RELATED TO MAINTENANCE FEES GRANTED (ORIGINAL EVENT CODE: PTGR); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCV | Information on status: appeal procedure |
Free format text: BOARD OF APPEALS DECISION RENDERED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |