CN111898396A - Obstacle detection method and device - Google Patents
Obstacle detection method and device Download PDFInfo
- Publication number
- CN111898396A CN111898396A CN201910370949.3A CN201910370949A CN111898396A CN 111898396 A CN111898396 A CN 111898396A CN 201910370949 A CN201910370949 A CN 201910370949A CN 111898396 A CN111898396 A CN 111898396A
- Authority
- CN
- China
- Prior art keywords
- view
- image
- matching
- cloud data
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 33
- 230000008030 elimination Effects 0.000 claims abstract description 11
- 238000003379 elimination reaction Methods 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 3
- 238000003062 neural network model Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention provides a method and a device for detecting an obstacle. The obstacle detection method of the present invention includes: the method comprises the steps of carrying out distortion elimination and line alignment on an acquired left view and an acquired right view to obtain a first view and a second view, determining an N-layer pyramid of the first view according to the first view, carrying out traversal search on the N-layer pyramid of the first view, determining the pixel position of a matching point of the first view, carrying out normalized product correlation NCC matching on the first view and the second view according to the pixel position of the matching point to obtain a disparity map, wherein the disparity map comprises the disparity of each matching point, and detecting obstacles according to the disparity map. The embodiment of the invention can solve the problem of high characteristic point selection condition.
Description
Technical Field
The embodiment of the invention relates to an image processing technology, in particular to a method and a device for detecting an obstacle.
Background
In the fields of unmanned driving, driving auxiliary early warning and the like, various technical means such as ultrasonic waves, laser radars, machine vision, infrared rays and the like are generally used for acquiring driving environment information, and a control unit is used for monitoring and intelligently predicting the road running condition so as to avoid vehicle collision accidents or reduce the collision damage degree of accidents.
The laser radar has the advantages of high precision and high resolution, and can be widely applied to Advanced Driving Assistance Systems (ADAS), such as Adaptive Cruise Control (ACC), Forward Collision Warning (FCW), and Automatic Emergency Braking (AEB). The laser radar is composed of a transmitting system, a receiving system, information processing and the like, and the realization cost is high. In order to reduce the cost, the left image and the right image can be shot by two cameras from different angles in a binocular stereo vision mode, the disparity map is obtained through a matching algorithm, three-dimensional point cloud information is reconstructed according to the disparity map, and then the obstacle is judged according to the three-dimensional point cloud information.
However, in the process of reconstructing the three-dimensional point cloud information by the binocular stereo vision and determining the obstacle, feature points of the left image or the right image need to be detected, and the disparity map is obtained by matching descriptors of the feature points. The characteristic point selection condition is high, the characteristic points are very sparse under the condition of poor image quality, and a clear space structure cannot be constructed after reconstruction.
Disclosure of Invention
The embodiment of the invention provides an obstacle detection method and device, and aims to solve the problem of high feature point selection conditions.
In a first aspect, an embodiment of the present invention provides an obstacle detection method, including:
carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first view and a second view;
determining an N-layer pyramid of the first view according to the first view;
traversing and searching the N layers of pyramids of the first view, and determining the pixel position of the matching point of the first view;
according to the pixel positions of the matching points, carrying out normalized product correlation NCC matching on the first view and the second view to obtain a disparity map, wherein the disparity map comprises the disparity of each matching point;
and detecting the obstacle according to the disparity map.
In a second aspect, an embodiment of the present invention provides an obstacle detection apparatus, including:
the first processing module is used for carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first view and a second view;
the second processing module is used for determining an N-layer pyramid of the first view according to the first view;
the third processing module is used for performing traversal search on the N layers of pyramids of the first view and determining the pixel position of the matching point of the first view;
the fourth processing module is used for carrying out normalized product correlation NCC matching on the first view and the second view according to the pixel positions of the matching points to obtain a disparity map, and the disparity map comprises the disparity of each matching point;
and the fifth processing module is used for detecting the obstacles according to the disparity map.
In a third aspect, an embodiment of the present invention provides an obstacle detection apparatus, including:
a memory and a processor;
the memory is configured to store instructions to cause the processor to execute the instructions to implement the obstacle detection method according to the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer storage medium, including: the computer storage medium is for storing a computer program which, when executed, is for implementing the method as described in the first aspect.
According to the obstacle detection method and device provided by the embodiment of the invention, the distortion elimination and line alignment are carried out on the acquired left view and right view to obtain a first view and a second view, the N-layer pyramid of the first view is determined according to the first view, the N-layer pyramid of the first view is subjected to traversal search, the pixel position of the matching point of the first view is determined, the first view and the second view are subjected to normalized product correlation NCC matching according to the pixel position of the matching point to obtain a disparity map, the disparity map comprises the disparity of each matching point, and the obstacle is detected according to the disparity map. The matching points of the first view are determined by adopting a mode of searching N layers of pyramids of the first view in a layered mode, so that the selected matching points are relatively robust, relatively rich matching points can be selected on the first view with different image qualities, and the problem of relatively high feature point selection conditions is solved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic diagram of an application scenario according to an embodiment of the present invention;
fig. 2 is a flowchart of an obstacle detection method according to an embodiment of the present invention;
FIG. 3 is a flow chart of another obstacle detection method according to an embodiment of the present invention;
fig. 4 is a flowchart of a matching point obtaining method according to an embodiment of the present invention;
FIG. 5 is a flow chart of a method for obstacle detection according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a first embodiment of a terminal device according to the present invention;
fig. 8 is a schematic structural diagram of a chip according to a first embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," and the like (if any) used herein are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Fig. 1 is a schematic diagram of an application scenario according to an embodiment of the present invention, as shown in fig. 1, the application scenario may include an obstacle detection apparatus 1 and a binocular camera 2, the binocular camera may take two images from different angles, a left view and a right view, and the obstacle detection apparatus 1 may perform the obstacle detection method of the present invention to process the left view and the right view to determine whether an obstacle exists and determine a distance of the obstacle, and a specific implementation manner of the method may be referred to the following explanation of the embodiment.
The obstacle detection device 1 and the binocular camera 2 may be provided in combination or separately.
This obstacle detection device 1 and binocular camera 2 can be applied to in equipment such as car, unmanned aerial vehicle, and it can carry out nimble setting according to the demand.
Fig. 2 is a flowchart of an obstacle detection method according to an embodiment of the present invention, where an execution main body of the method may be an obstacle detection device, and as shown in fig. 2, the method according to the embodiment may include:
The left view and the right view can be acquired through the binocular camera, the binocular camera is not limited to the above, the binocular camera can comprise a left camera and a right camera, and the left camera and the right camera are used for shooting the same target to acquire the left view and the right view. Distortion elimination and line alignment can be carried out to left view and right view according to the internal reference data of left camera, the internal reference data and the binocular relative position relation of right camera for the formation of image origin coordinates of left view and right view are unanimous, and left camera and right camera optical axis are parallel, and left view and right view imaging plane coplane aligns to the polar line.
The internal reference data may include a focal length, an imaging origin, a distortion coefficient, etc., and the binocular relative positional relationship may be represented using binocular external references, which may include a rotation matrix and a translation vector. The internal reference data and the binocular external reference can be acquired after binocular calibration.
The first view may be a left view after distortion removal and line alignment, or a right view after distortion removal and line alignment, and when the first view is a left view after distortion removal and line alignment, the second view is a right view after distortion removal and line alignment. The embodiment of the present invention is exemplified by taking the first view as the left view after distortion removal and line alignment.
And 102, determining an N-layer pyramid of the first view according to the first view.
In a possible implementation, the N-level pyramid of the first view is obtained according to the first view, the first-level image of the N-level pyramid may be the first view, that is, the original image, and the second-level image of the N-level pyramid is the first view
That is, the size of the second layer image is half of the size of the first view, each pixel value in the second layer image is the average value of the pixels in the 2 x 2 region of the first view, and so on, the nth layer image is the first view
For example, when N is 4, the first layer image of the 4-layer pyramid is the first view, and the second layer image of the 4-layer pyramid is the first view
The third layer image of the 4-layer pyramid is the first view
The fourth layer image of the 4-layer pyramid is of the first view
N is an integer greater than 1.
And traversing and searching the N layers of pyramids of the first view, and searching the matching points of the first view. The N layers of pyramids of the first view are searched in a layered mode, the visual characteristics of the human eyes for watching objects from coarse to fine can be simulated, and the first view is searched in a layered mode from coarse to fine until an accurate matching point of the first view is found.
And 104, performing Normalized Cross Correlation (NCC) matching on the first view and the second view according to the pixel positions of the matching points to obtain a disparity map, wherein the disparity map comprises the disparity of each matching point.
And searching each matching point in the first view in the direction of the second view by taking the coordinate of the first view as a starting point on the horizontal epipolar line to perform NCC matching of the L-L neighborhood, and acquiring a disparity map after matching is successful.
And 105, detecting the obstacle according to the parallax map.
The embodiment of the present application does not limit how the obstacle is detected according to the disparity map. Alternatively, the obstacle may be detected according to the point cloud data after the depth map is converted into the point cloud data by acquiring the depth map according to the disparity map and then converting the depth map into the point cloud data. And determining whether an obstacle exists according to the point cloud data, and determining the distance of the obstacle.
In this embodiment, distortion removal and line alignment are performed on an acquired left view and an acquired right view to obtain a first view and a second view, an N-layer pyramid of the first view is determined according to the first view, traversal search is performed on the N-layer pyramid of the first view to determine a pixel position of a matching point of the first view, normalization product correlation NCC matching is performed on the first view and the second view according to the pixel position of the matching point to obtain a disparity map, the disparity map includes disparity of each matching point, and an obstacle is detected according to the disparity map. The matching points are determined by adopting a mode of searching N layers of pyramids of the first view in a layered mode, so that the selected matching points are relatively robust, relatively rich matching points can be selected on the first view with different image qualities, and the problem of relatively high feature point selection conditions is solved.
Optionally, in an implementation manner of step 101, distortion removal and line alignment are performed on the acquired left view and right view to obtain a first calibration view and a second calibration view, the first calibration view is input into the neural network model, the road surface area map is output, and the road surface area map is used as the first view. That is, the obstacle can be detected in the target area (for example, the road surface area) by the processing from step 102 to step 105, so that the amount of calculation can be reduced and the obstacle detection speed can be increased.
Fig. 3 is a flowchart of another obstacle detection method according to an embodiment of the present invention, where an execution main body of the method may be an obstacle detection device, and as shown in fig. 3, the method according to the embodiment may include:
For specific explanation of steps 201 to 202, reference may be made to steps 101 to 102 in the embodiment shown in fig. 2, which is not described herein again.
Where M is a positive integer greater than 1, e.g., M ═ 24. I.e., the first layer image is gridded and the threshold for each block is determined, in one implementation, the median of the gradient values of the pixels within each block is used as the threshold for that block.
And 204, selecting matching points from the N-layer pyramid of the first view according to the threshold value of each block, and acquiring the pixel positions of the matching points.
Specifically, matching points are selected in the N-level pyramid of the first view according to the threshold of each block. As illustrated by M-24 and N-4, a 24 × 24 block of the first layer image corresponds to a 12 × 12 block of the second layer image, a 6 × 6 block of the third layer image, and a 3 × 3 block of the fourth layer image. The embodiment of the invention can search from the fourth layer image by the threshold value to find a proper matching point. For example, the matching point is a pixel whose gradient value is larger than the threshold value.
And step 206, detecting the obstacle according to the disparity map.
For the detailed explanation of step 205 to step 206, reference may be made to step 104 to step 105 in the embodiment shown in fig. 2, which is not described herein again.
In this embodiment, distortion removal and line alignment are performed on an acquired left view and an acquired right view to obtain a first view and a second view, an N-layer pyramid of the first view is determined according to the first view, traversal search is performed on the N-layer pyramid of the first view to determine a pixel position of a matching point of the first view, normalization product correlation NCC matching is performed on the first view and the second view according to the pixel position of the matching point to obtain a disparity map, the disparity map includes disparity of each matching point, and an obstacle is detected according to the disparity map. The matching points are determined by adopting a mode of searching N layers of pyramids of the first view in a layered mode, so that the selected matching points are relatively robust, relatively rich matching points can be selected on the first view with different image qualities, and the problem of relatively high feature point selection conditions is solved.
Fig. 4 is a flowchart of a method for acquiring a matching point according to an embodiment of the present invention, which is used in this embodiment to explain a specific implementation manner of the step 204 on the basis of the foregoing embodiment, and as shown in fig. 4, the method according to this embodiment may include:
step A: searching the position with the maximum pixel gradient in each target area in the Nth layer of image, comparing the maximum pixel gradient in each target area with a preset threshold value of a corresponding block of the Nth layer of image, taking the Nth layer of image as an image to be processed, wherein the target area is a preset area of the image to be processed, and the preset threshold value of the corresponding block of the Nth layer of image is the threshold value of the corresponding block of the first layer of image.
For example, by further exemplifying that M is 24 and N is 4, a 24 × 24 block of the first layer image, a 12 × 12 block corresponding to the second layer image, a 6 × 6 block corresponding to the third layer image, and a 3 × 3 block corresponding to the fourth layer image, the preset region of the nth layer image may be a 3 × 3 block, that is, a 3 × 3 block of the target region, and the calculation is performed in the 3 × 3 region of the 4 th layer image to find the position where the gradient is maximum, and the maximum value of the gradient in the target region is compared with the threshold value of the 24 × 24 block of the first layer image.
And B: and if the pixel gradient is larger than the preset threshold value of the corresponding block, taking the position with the maximum pixel gradient as the pixel position of a matching point.
And C: and if the pixel gradient is not greater than the preset threshold of the corresponding block, taking the image to be processed as a processed image, taking the next layer image of the processed image as the image to be processed, searching the position with the maximum pixel gradient in each target area in the image to be processed, and comparing the maximum pixel gradient in each target area with the preset threshold of the corresponding block of the image to be processed.
Step D: and C-B is repeatedly executed until the first layer image is completely processed.
By way of further example, if the value is not greater than the preset threshold of the corresponding block, the nth layer image is taken as the processed image, the N-1 th layer image, which is the next layer image of the nth layer image, is taken as the image to be processed, and then the position where the pixel gradient is maximum in each target region is searched in the N-1 th layer image, the preset region of the N-1 th layer image may be different from the preset region of the nth layer image, for example, a block of 6 × 6 of the third layer image is taken as the target region, and in addition, the present application does not limit the setting manner of the preset threshold of the corresponding block of the N-1 th layer image, for example, the preset threshold of the corresponding block of the N-1 th layer image is s times the preset threshold of the corresponding block of the nth layer image, and s takes a positive number smaller than 1, for example, s is 0.75. Searching the position with the maximum pixel gradient in each target area in the image of the layer N-1, and comparing the maximum pixel gradient in each target area with a preset threshold value of a corresponding block of the image of the layer N-1. If the pixel gradient is larger than the preset threshold value of the corresponding block, the position with the maximum pixel gradient is taken as the pixel position of a matching point, if the pixel gradient is not larger than the preset threshold value of the corresponding block, the next layer image of the N-1 layer image, namely the N-2 layer image is taken as an image to be processed, the position with the maximum pixel gradient in each target area is searched in the N-2 layer image, the maximum pixel gradient in each target area is compared with the preset threshold value of the corresponding block of the N-2 layer image, and the like. Taking the above example as an example, the N-2 th layer image is the second layer image, and the maximum value of the gradient within the 12 × 12 block of the second layer image is compared with s × s times of the preset threshold of the corresponding block of the N-th layer image. If the maximum gradient value of the pixel is not greater than the s-s times of the preset threshold value of the corresponding block of the Nth-layer image, the maximum gradient value in the block of the 24-s-24 of the first-layer image is compared with the s-s times of the preset threshold value of the corresponding block of the Nth-layer image, if the maximum gradient value of the pixel is greater than the s-s times of the preset threshold value of the corresponding block of the Nth-layer image, the maximum gradient value of the pixel is taken as the pixel position of one matching point, and if the maximum gradient value of the pixel is not greater than the s-s times of the preset threshold value of the corresponding block of the Nth-layer image, the next 3-s block of the fourth-layer image is returned for searching to obtain all matching points.
The first layer image is the first view, namely the image of the bottommost layer, the Nth layer image is the image of the topmost layer, and the next layer image of the processed images is selected from top to bottom.
Optionally, before matching, an appropriate number of matching points may be selected from all matching points for matching, so as to improve the matching speed. For example, 0.005 matching points of the total number of pixels of the first view can be selected for subsequent matching to determine the disparity map. The selection can be random or according to a preset rule.
In this embodiment, by searching a position where a pixel gradient in each target area is maximum in an nth layer image, comparing the maximum pixel gradient value in each target area with a preset threshold value of a corresponding block of the nth layer image, taking the nth layer image as an image to be processed, taking the target area as a preset area of the image to be processed, taking a preset threshold value of a corresponding block of the nth layer image as a threshold value of a corresponding block of a first layer image, if the maximum pixel gradient value is greater than the preset threshold value of the corresponding block, taking the position where the pixel gradient is maximum as a pixel position of a matching point, if the maximum pixel gradient value is not greater than the preset threshold value of the corresponding block, taking the image to be processed as a processed image, taking a next layer image of the processed image as the image to be processed, searching the position where the pixel gradient is maximum in each target area in the image to be processed, comparing the maximum pixel gradient value in each target area with the preset threshold value of the corresponding block of the image to be processed, and repeating the steps until the first layer image is processed, so that the matching points can be quickly and accurately selected.
Fig. 5 is a flowchart of an obstacle detection method according to an embodiment of the present invention, which explains a specific implementation of the step 105 on the basis of the above embodiment, and as shown in fig. 5, the method of the embodiment may include:
step S401, point cloud data are obtained according to the disparity map.
In one possible embodiment, the depth map may be converted into point cloud data by obtaining the depth map according to the disparity map, for example, the depth d of each matching point may be determined by using the formula d ═ f × b/(xl-xr), where f is the focal length, b is the baseline of the binocular camera, and xl-xr is the disparity of the matching points, so as to obtain the depth map. And converting the depth map into point cloud data according to the formula of (u-cx) × d/fx, (v-cy) × d/fy, and (Z) ═ d, wherein fx, fy, cx and cy are internal parameters of the binocular camera. The embodiment of the present application does not limit this.
The point cloud data can be clustered by using any clustering algorithm, the categories of the point cloud data are output, and the normal vector of the point cloud data of each category is calculated.
And 403, determining whether each type of point cloud data is an obstacle according to the included angle between the normal vector of each type of point cloud data and the preset vector and the distance between the maximum value and the minimum value of the y axis of each type of point cloud data.
In one implementation, the predetermined vector is (0,0, 1).
And when the included angle between the normal vector and the preset vector of the point cloud data is smaller than a first preset value and the distance between the maximum value and the minimum value of the y axis of the point cloud data is larger than a second preset value, determining that the point cloud data is an obstacle.
In this embodiment, the normal vector of each type of point cloud data is calculated by clustering the point cloud data, and whether each type of point cloud data is an obstacle or not is determined according to an included angle between the normal vector of each type of point cloud data and a preset vector and a distance between the maximum value and the minimum value of the y axis of each type of point cloud data, so that the obstacle and the distance between the obstacles can be accurately determined.
Fig. 6 is a schematic structural diagram of an obstacle detection device according to an embodiment of the present invention, and as shown in fig. 6, the device according to the embodiment may include:
and the first processing module 61 is configured to perform distortion removal and line alignment on the acquired left view and right view to obtain a first view and a second view.
A second processing module 62, configured to determine, according to the first view, an N-level pyramid of the first view.
And a third processing module 63, configured to perform traversal search on the N-level pyramid of the first view, and determine a pixel position of a matching point of the first view.
A fourth processing module 64, configured to perform normalized product correlation NCC matching on the first view and the second view according to the pixel position of the matching point, and obtain a disparity map, where the disparity map includes a disparity of each matching point.
And a fifth processing module 65, configured to detect an obstacle according to the disparity map.
Optionally, the third processing module 63 is specifically configured to.
The first layer image is divided into M-size blocks, and the threshold value of each block is determined according to the gradient value of pixels in each block. And selecting a matching point in the pyramid of the N layers of the first view according to the threshold value of each block, and acquiring the pixel position of the matching point.
Optionally, the third processing module 63 is further specifically configured to:
step A: searching the position with the maximum pixel gradient in each target area in the Nth layer image, comparing the maximum pixel gradient in each target area with a preset threshold value of a corresponding block of the Nth layer image, taking the Nth layer image as an image to be processed, wherein the target area is a preset area of the image to be processed, and the preset threshold value of the corresponding block of the Nth layer image is the threshold value of the corresponding block of the first layer image.
And B: and if the pixel gradient is larger than the preset threshold value of the corresponding block, taking the position with the maximum pixel gradient as the pixel position of a matching point.
And C: and if the pixel gradient is not greater than the preset threshold of the corresponding block, taking the image to be processed as a processed image, taking the next layer image of the processed image as the image to be processed, searching the position with the maximum pixel gradient in each target area in the image to be processed, and comparing the maximum pixel gradient in each target area with the preset threshold of the corresponding block of the image to be processed.
Step D: and C-B is repeatedly executed until the first layer image is completely processed.
Optionally, the fifth processing module 65 is specifically configured to:
acquiring point cloud data according to the disparity map; clustering the point cloud data, and calculating a normal vector of each type of point cloud data; and determining whether each type of point cloud data is an obstacle or not according to the included angle between the normal vector of each type of point cloud data and the preset vector and the distance between the maximum value and the minimum value of the y axis of each type of point cloud data.
Optionally, the fifth processing module 65 is further specifically configured to:
and when the included angle between the normal vector and the preset vector of the point cloud data of the same type is smaller than a first preset value and the distance between the maximum value and the minimum value of the y axis of the point cloud data of the same type is larger than a second preset value, determining the point cloud data of the same type as the obstacle.
Optionally, the first processing module 61 is specifically configured to:
carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first calibrated view and a second calibrated view; and inputting the calibration first view into the neural network model, outputting a road surface area map, and taking the road surface area map as a first view.
The apparatus of this embodiment may be configured to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Embodiments of the present invention also provide a computer storage medium, on which a computer program or instructions are stored, which, when executed by a processor or a computer, implement the method according to the embodiment shown in fig. 4.
It should be noted that the obstacle detection apparatus according to the embodiment of the present invention may be a terminal device, or may be a component in the terminal device, such as a chip.
Fig. 7 is a schematic structural diagram of a first terminal device according to the present invention, and as shown in fig. 7, the terminal device according to the present embodiment includes: a processor 711, a memory 712, a transceiver 713, and a bus 714. Wherein the processor 711, the memory 712, and the transceiver 713 are connected to each other through a bus 714. The bus 714 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 714 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 7, but this is not intended to represent only one bus or type of bus.
In terms of hardware implementation, the functional modules shown in fig. 6 above may be embedded in the processor 211 of the terminal device or may be independent of the processor.
The transceiver 713 may include necessary radio frequency communication devices such as a mixer. The processor 711 may include at least one of a Central Processing Unit (CPU), a Digital Signal Processor (DSP), a Microcontroller (MCU), an Application Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA).
The memory 712 is used for storing program instructions and the processor 711 is used for calling the program instructions in the memory 712 to execute the above-mentioned scheme.
The program instructions may be implemented in the form of software functional units and may be sold or used as a stand-alone product, and the memory 712 may be any form of computer-readable storage medium. Based on such understanding, all or part of the technical solutions of the present application may be embodied in the form of a software product, which includes several instructions to enable a computer device, specifically, the processor 711, to execute all or part of the steps of the first terminal in the embodiments of the present application. And the aforementioned computer-readable storage media comprise: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The terminal device described above in this embodiment may be configured to execute the technical solution in the foregoing method embodiment, and the implementation principle and the technical effect are similar, where the function of each device may refer to the corresponding description in the method embodiment, and is not described here again.
Fig. 8 is a schematic structural diagram of a first chip embodiment of the invention, and as shown in fig. 8, the chip of this embodiment includes: a memory 81 and a processor 82; the memory 81 is used for storing instructions to make the processor 82 execute the instructions to implement the technical solution in the above method embodiments, which is similar in implementation principle and technical effect.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. An obstacle detection method, comprising:
carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first view and a second view;
determining an N-layer pyramid of the first view according to the first view;
traversing and searching the N layers of pyramids of the first view, and determining the pixel position of the matching point of the first view;
according to the pixel position of the matching point, carrying out normalized product correlation NCC matching on the first view and the second view to obtain a disparity map, wherein the disparity map comprises the disparity of each matching point;
and detecting the obstacle according to the disparity map.
2. The method of claim 1, wherein performing a traversal search of the N-level pyramid of the first view to determine pixel locations of matching points comprises:
dividing the first layer image into M-by-M blocks, and determining a threshold value of each block according to the gradient value of pixels in each block;
and selecting a matching point in the pyramid of the N layers of the first view according to the threshold value of each block, and acquiring the pixel position of the matching point.
3. The method of claim 2, wherein selecting a matching point in the N-level pyramid of the first view according to the threshold value of each block, and obtaining a pixel position of the matching point comprises:
step A: searching the position with the maximum pixel gradient in each target area in the Nth layer of image, comparing the maximum pixel gradient in each target area with a preset threshold value of a corresponding block of the Nth layer of image, taking the Nth layer of image as an image to be processed, wherein the target area is a preset area of the image to be processed, and the preset threshold value of the corresponding block of the Nth layer of image is the threshold value of the corresponding block of the first layer of image;
and B: if the pixel gradient is larger than the preset threshold value of the corresponding block, taking the position with the maximum pixel gradient as the pixel position of a matching point;
and C: if the pixel gradient is not greater than the preset threshold of the corresponding block, taking the image to be processed as a processed image, taking the next layer image of the processed image as the image to be processed, searching the position with the maximum pixel gradient in each target area in the image to be processed, and comparing the maximum pixel gradient in each target area with the preset threshold of the corresponding block of the image to be processed;
step D: and C-B is repeatedly executed until the first layer image is completely processed.
4. The method according to any one of claims 1 to 3, wherein the detecting an obstacle according to the disparity map comprises:
acquiring point cloud data according to the disparity map;
clustering the point cloud data, and calculating a normal vector of each type of point cloud data;
and determining whether each type of point cloud data is an obstacle or not according to the included angle between the normal vector of each type of point cloud data and the preset vector and the distance between the maximum value and the minimum value of the y axis of each type of point cloud data.
5. The method of claim 4, wherein the determining whether each type of point cloud data is an obstacle according to an included angle between a normal vector of each type of point cloud data and a preset vector and a distance between a maximum value and a minimum value of a y-axis of each type of point cloud data comprises:
and when the included angle between the normal vector and the preset vector of the point cloud data of the same type is smaller than a first preset value and the distance between the maximum value and the minimum value of the y axis of the point cloud data of the same type is larger than a second preset value, determining that the point cloud data of the same type is an obstacle.
6. The method of claim 1, wherein the de-distorting and line aligning the acquired left and right views to obtain a first view and a second view comprises:
carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first calibrated view and a second calibrated view;
inputting the calibration first view into a neural network model, outputting a road surface area map, and taking the road surface area map as the first view.
7. An obstacle detection device, comprising:
the first processing module is used for carrying out distortion elimination and line alignment on the acquired left view and right view to obtain a first view and a second view;
the second processing module is used for determining an N-layer pyramid of the first view according to the first view;
the third processing module is used for performing traversal search on the N-layer pyramid of the first view and determining the pixel position of the matching point of the first view;
the fourth processing module is configured to perform normalized product correlation NCC matching on the first view and the second view according to the pixel position of the matching point, and obtain a disparity map, where the disparity map includes a disparity of each matching point;
and the fifth processing module is used for detecting the obstacles according to the disparity map.
8. The apparatus of claim 7, wherein the third processing module is configured to:
dividing the first layer image into M-by-M blocks, and determining a threshold value of each block according to the gradient value of pixels in each block;
and selecting a matching point in the pyramid of the N layers of the first view according to the threshold value of each block, and acquiring the pixel position of the matching point.
9. A chip, comprising:
a memory and a processor;
the memory is for storing instructions to cause the processor to execute the instructions to implement the obstacle detection method of any of claims 1 to 6.
10. A computer storage medium, comprising: the computer storage medium is for storing a computer program which, when executed, is for implementing the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910370949.3A CN111898396A (en) | 2019-05-06 | 2019-05-06 | Obstacle detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910370949.3A CN111898396A (en) | 2019-05-06 | 2019-05-06 | Obstacle detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111898396A true CN111898396A (en) | 2020-11-06 |
Family
ID=73169386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910370949.3A Pending CN111898396A (en) | 2019-05-06 | 2019-05-06 | Obstacle detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111898396A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115049980A (en) * | 2022-06-16 | 2022-09-13 | 威海经济技术开发区天智创新技术研究院 | Target object determination method and device based on image and electronic equipment |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102881014A (en) * | 2012-09-07 | 2013-01-16 | 航天恒星科技有限公司 | Quick stereo matching method based on graph cut |
| CN104364796A (en) * | 2012-06-01 | 2015-02-18 | 罗伯特·博世有限公司 | Method and device for processing stereoscopic data |
| CN105160702A (en) * | 2015-08-20 | 2015-12-16 | 武汉大学 | Stereoscopic image dense matching method and system based on LiDAR point cloud assistance |
| CN106504237A (en) * | 2016-09-30 | 2017-03-15 | 上海联影医疗科技有限公司 | Determine method and the image acquiring method of matching double points |
| CN106529495A (en) * | 2016-11-24 | 2017-03-22 | 腾讯科技（深圳）有限公司 | Obstacle detection method of aircraft and device |
| CN107179768A (en) * | 2017-05-15 | 2017-09-19 | 上海木爷机器人技术有限公司 | A kind of obstacle recognition method and device |
| CN107341454A (en) * | 2017-06-21 | 2017-11-10 | 海信集团有限公司 | The detection method and device of barrier, electronic equipment in a kind of scene |
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
| CN107636679A (en) * | 2016-12-30 | 2018-01-26 | 深圳前海达闼云端智能科技有限公司 | A kind of obstacle detection method and device |
| CN108230392A (en) * | 2018-01-23 | 2018-06-29 | 北京易智能科技有限公司 | A kind of dysopia analyte detection false-alarm elimination method based on IMU |
| WO2018120027A1 (en) * | 2016-12-30 | 2018-07-05 | 深圳前海达闼云端智能科技有限公司 | Method and apparatus for detecting obstacles |
| CN108876861A (en) * | 2018-05-25 | 2018-11-23 | 中国空间技术研究院 | A kind of solid matching method of objects outside Earth rover |
| CN109242769A (en) * | 2018-12-13 | 2019-01-18 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
-
2019
- 2019-05-06 CN CN201910370949.3A patent/CN111898396A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104364796A (en) * | 2012-06-01 | 2015-02-18 | 罗伯特·博世有限公司 | Method and device for processing stereoscopic data |
| CN102881014A (en) * | 2012-09-07 | 2013-01-16 | 航天恒星科技有限公司 | Quick stereo matching method based on graph cut |
| CN105160702A (en) * | 2015-08-20 | 2015-12-16 | 武汉大学 | Stereoscopic image dense matching method and system based on LiDAR point cloud assistance |
| CN106504237A (en) * | 2016-09-30 | 2017-03-15 | 上海联影医疗科技有限公司 | Determine method and the image acquiring method of matching double points |
| CN106529495A (en) * | 2016-11-24 | 2017-03-22 | 腾讯科技（深圳）有限公司 | Obstacle detection method of aircraft and device |
| CN107636679A (en) * | 2016-12-30 | 2018-01-26 | 深圳前海达闼云端智能科技有限公司 | A kind of obstacle detection method and device |
| WO2018120027A1 (en) * | 2016-12-30 | 2018-07-05 | 深圳前海达闼云端智能科技有限公司 | Method and apparatus for detecting obstacles |
| CN107179768A (en) * | 2017-05-15 | 2017-09-19 | 上海木爷机器人技术有限公司 | A kind of obstacle recognition method and device |
| CN107341454A (en) * | 2017-06-21 | 2017-11-10 | 海信集团有限公司 | The detection method and device of barrier, electronic equipment in a kind of scene |
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
| CN108230392A (en) * | 2018-01-23 | 2018-06-29 | 北京易智能科技有限公司 | A kind of dysopia analyte detection false-alarm elimination method based on IMU |
| CN108876861A (en) * | 2018-05-25 | 2018-11-23 | 中国空间技术研究院 | A kind of solid matching method of objects outside Earth rover |
| CN109242769A (en) * | 2018-12-13 | 2019-01-18 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 胡颖;马国军;何康;王亚军;: "卷积神经网络双目视觉路面障碍物检测", 计算机工程与设计, no. 10, 16 October 2018 (2018-10-16), pages 286 - 291 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115049980A (en) * | 2022-06-16 | 2022-09-13 | 威海经济技术开发区天智创新技术研究院 | Target object determination method and device based on image and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111222395B (en) | Target detection method and device and electronic equipment | |
| US9338437B2 (en) | Apparatus and method for reconstructing high density three-dimensional image | |
| US8406510B2 (en) | Methods for evaluating distances in a scene and apparatus and machine readable medium using the same | |
| CN112889071B (en) | System and method for determining depth information in a two-dimensional image | |
| CN105335955A (en) | Object detection method and object detection apparatus | |
| CN109520480B (en) | Distance measurement method and distance measurement system based on binocular stereo vision | |
| US20220180476A1 (en) | Systems and methods for image feature extraction | |
| CN111738071B (en) | Inverse perspective transformation method based on motion change of monocular camera | |
| US20210125361A1 (en) | Systems and methods for stereoscopic imaging | |
| CN114495064A (en) | Monocular depth estimation-based vehicle surrounding obstacle early warning method | |
| CN116310673A (en) | Three-dimensional target detection method based on fusion of point cloud and image features | |
| CN114089329A (en) | Target detection method based on fusion of long and short focus cameras and millimeter wave radar | |
| CN115909268A (en) | Dynamic obstacle detection method and device | |
| CN112184700B (en) | Monocular camera-based agricultural unmanned vehicle obstacle sensing method and device | |
| CN111898396A (en) | Obstacle detection method and device | |
| JP2013069045A (en) | Image recognition device, image recognition method, and image recognition program | |
| JP2018124963A (en) | Image processing device, image recognition device, image processing program, and image recognition program | |
| CN113591890A (en) | Clustering method and device | |
| CN114359891B (en) | Three-dimensional vehicle detection method, system, device and medium | |
| CN115761425A (en) | Target detection method, device, terminal equipment and computer readable storage medium | |
| CN112686155A (en) | Image recognition method, image recognition device, computer-readable storage medium and processor | |
| CN113840130A (en) | Depth map generation method, device and storage medium | |
| CN113591891A (en) | Clustering method and device | |
| CN115049822B (en) | Three-dimensional imaging method and device | |
| CN116778262B (en) | Three-dimensional target detection method and system based on virtual point cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |