CN107506711B - Convolutional neural network-based binocular vision barrier detection system and method - Google Patents

Convolutional neural network-based binocular vision barrier detection system and method Download PDF

Info

Publication number
CN107506711B
CN107506711B CN201710697239.2A CN201710697239A CN107506711B CN 107506711 B CN107506711 B CN 107506711B CN 201710697239 A CN201710697239 A CN 201710697239A CN 107506711 B CN107506711 B CN 107506711B
Authority
CN
China
Prior art keywords
image
parallax
disparity map
neural network
convolutional neural
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710697239.2A
Other languages
Chinese (zh)
Other versions
CN107506711A (en
Inventor
马国军
胡颖
夏健
卫春军
郑威
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu University of Science and Technology
Original Assignee
Jiangsu University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu University of Science and Technology filed Critical Jiangsu University of Science and Technology
Priority to CN201710697239.2A priority Critical patent/CN107506711B/en
Publication of CN107506711A publication Critical patent/CN107506711A/en
Application granted granted Critical
Publication of CN107506711B publication Critical patent/CN107506711B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a binocular vision barrier detection system and method based on a convolutional neural network. The system consists of an image acquisition module and an obstacle detection module, wherein the image acquisition module acquires binocular images and transmits the binocular images to the obstacle detection module to perform corresponding data processing on the acquired image data so as to obtain an accurate obstacle region. The detection method comprises the following steps: firstly, performing median filtering processing on an acquired original image; correcting the binocular image according to the camera parameters; designing a new convolution kernel to be applied to a convolution neural network structure for generating an accurate disparity map; finally, the precise obstacle area in the image is detected by using an improved V parallax method. The invention has good obstacle detection precision under the conditions of complex light, small obstacles and the like, and has good robustness.

Description

Convolutional neural network-based binocular vision barrier detection system and method
Technical Field
The invention relates to the technical field of binocular vision image processing, in particular to a binocular vision barrier detection system based on a convolutional neural network and a detection method thereof.
Background
With the progress of computer technology, intelligent vehicles are rapidly developed and widely applied to the fields of national defense, scientific research, daily life and the like. Among them, obstacle detection is a core problem in intelligent vehicle navigation.
The convolutional neural network is a feed-forward network, and the artificial neuron can respond to units in a coverage range, including convolutional layers and pooling layers, and has excellent performance on image processing. In recent years, people have attracted more and more attention, and the application field is more and more extensive.
At present, a binocular vision-based obstacle detection method gradually becomes a hot problem of research, the same scene in binocular images is searched through binocular matching, a disparity relation graph of a changed space scene is generated, and a specific detection method is applied to process the disparity graph to obtain a corresponding obstacle detection area.
However, when complex light and small obstacles appear in the scene, it is difficult for the existing method to achieve accurate obstacle region detection and maintain the robustness of the system. Therefore, for the complex environment of the smart car, how to accurately detect the obstacle area and maintain the robustness of the system becomes an urgent problem to be solved.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide a binocular vision barrier detection system based on a convolutional neural network and a detection method thereof, so as to solve the defects of the conventional binocular vision barrier detection method in the aspects of barrier detection accuracy, system robustness, etc.
In order to achieve the purpose, the invention adopts the following technical scheme:
a binocular vision obstacle detection system based on a convolutional neural network is composed of an image acquisition module and an obstacle detection module which are connected. The image acquisition module is used for acquiring a left image and a right image in a scene, wherein the left image and the right image are both optical images. The obstacle detection module is used for carrying out corresponding data processing on the acquired image data to acquire a final accurate obstacle area.
Furthermore, the image acquisition modules are two industrial cameras which are horizontally arranged in parallel and have the model of Pike F-100, and image data are transmitted to a computer for subsequent processing through an IEEE-1394b interface by an acquisition card.
Further, the obstacle detection module is a computer equipped with a GPU of NVIDIAGTX 1070.
In order to achieve the above object, the present invention is achieved by the following another technical means.
A detection method of a binocular vision barrier detection system based on a convolutional neural network comprises the following steps:
(1) acquiring a binocular image from an image acquisition module, and preprocessing the binocular image to eliminate noise in the image;
(2) the calibration image acquisition module acquires parameters of internal parameters and distortion of the image acquisition module and corrects the binocular image;
(3) designing a twin convolution neural network to generate a disparity map of an accurate binocular image;
(4) the disparity map is processed using an improved V-disparity method to detect an obstacle region in the image.
The specific method for generating the accurate binocular image disparity map by designing the twin convolutional neural network in the step (3) comprises the following steps of:
a) and designing a twin convolutional neural network structure, wherein the left and right branch parameters are shared. The twin convolutional neural network is composed of a feature extraction sub-network and a feature classification sub-network. The left branch and the right branch of the feature extraction sub-network can respectively extract corresponding feature descriptions from the input image square and the input image long strip; and the feature classification sub-network performs dot product operation on the extracted left and right branch feature descriptions to obtain the similarity score of the pixel points to be matched in the parallax search range, and then the similarity score is used as the input of the softmax layer to obtain the parallax probability distribution.
The convolution kernel is designed to have the same receptive field as the traditional convolution kernel (2n-1) × (2n-1) parameters are reduced to (2n-3) × (2n-3) parameters, the calculated amount is reduced, and the overfitting phenomenon can be overcome by less parameter amount.
b) And (6) cutting a sample. Selecting a pixel point p (x) with real parallax in the left image according to real parallax data in the KITTI data seti,yi) Extracting an image block with the pixel point as a center; selecting a pixel point q in the right image, and enabling the coordinate to be (x)i,yi) And selecting image blocks of the same size with q as the center. According to the parallax search range, selecting the left boundary of the right image block with q as the center in the right imageSide image stripes. Sample cropping is accomplished in this way.
c) And standardizing sample data and constructing a training set. And (b) transforming the gray value of the image block clipped in the step b) into a range of [ -1,1] by using a formula (1) as an input of the network.
Figure GDA0001453637400000021
Wherein the content of the first and second substances,
Figure GDA0001453637400000031
Figure GDA0001453637400000032
in the formula, U is an image block after standardization; x is a cropped original image block;
Figure GDA0001453637400000033
is the average of the image blocks X; x is the number ofiIs the pixel value in image block X; s is the standard deviation of the image block X; n is the number of pixels in the image block.
d) And training the twin convolutional neural network. Training the twin convolutional neural network designed by step a) with an adaptive moment-estimated stochastic gradient descent algorithm (Adam). According to the application scenario of the invention, the mutual entropy loss function is modified as follows:
Figure GDA0001453637400000034
wherein the content of the first and second substances,
Figure GDA0001453637400000035
where j (ω) represents the mutual entropy loss for a set of samples; p is a radical ofi(diω) represents the probability distribution of the ith sample; diIs a predicted disparity value;
Figure GDA0001453637400000036
is the true disparity value; lambda [ alpha ]1,λ2,λ3Is a preset value. In the present invention, λ is set individually1＝0.5,λ2＝0.2,λ3＝0.05。
e) And (5) calculating a disparity map. Taking binocular image pairs and standardizing the binocular image pairs by using the step c). Extracting 64-dimensional features of pixel points in the left image and the right image by using the network model trained in the step d), and respectively recording the features as SL(p) and SR(q); will SL(p) and SR(q) performing dot product operation to obtain an image pair similarity score, and taking an inverse number as a matching cost:
CCNN(p,d)＝-s(<SL(p),SR(q)>) (6)
wherein s: (<SL(p),SR(q)>) Representing the image pair similarity score. And finally, in the parallax searching range, selecting the point with the minimum matching cost as the matching point for parallax selection, and further generating a parallax map.
The specific method for detecting an obstacle region in an image by processing the disparity map obtained in step (3) with the improved V-disparity method in step (4) above includes:
f) and (3) calculating the gradient of the parallax image array direction generated in the step (4) by adopting a Prewitt operator, reserving the parallax at the position where the gradient is a negative value, setting the parallax at other positions to be 0, filtering out the pixel points of the obstacle, and generating the filtered parallax image.
g) Counting the same gray value number of each line of pixels of the filtered disparity map generated in the step f) to generate a V disparity map I1。
h) Search for V disparity map I1The maximum value of each row in the image is reserved, the gray value of the rest pixel points is set to be 0, and a maximum value V parallax image I is generated2。
i) Setting a threshold value T, and2setting the gray value of the pixel larger than T to be 1, and setting the rest to be 0, and generating a V parallax binary image I only containing road information3The calculation formula of the threshold value T is as follows:
Figure GDA0001453637400000041
in the formula, xiIs I2The value of each pixel; n is I2The total number of middle pixels; 1{ xiNot equal to 0} denotes xiAnd if not, taking 1, otherwise, taking 0.
j) Extraction of I by hough line detection method3And (5) making the middle road straight.
k) And calculating the slope k and the intercept b of the image coordinate system of the road straight line extracted in the step j) in the V disparity map by using a straight line two-point equation.
l) the disparity map D (x, y, D) is scanned point by point from bottom to top from left to right in accordance with a raster scan method, and f ═ kd + b is calculated for each pixel point.
m) if (f-y)>T1And projecting the pixel point in the disparity map above a straight line of the road surface in the V disparity map, namely, being higher than the road surface, and taking the pixel point as an obstacle point. The embodiment of the invention takes the threshold value T1＝5。
And n) repeating the steps l) and m) until the parallax image is completely scanned, and obtaining the obstacle area in the image.
The invention has the following advantages and beneficial effects:
(1) the method uses the convolutional neural network to calculate the disparity map, has good robustness under the conditions of complex light, small obstacles and the like, and can obtain the accurate disparity map; therefore, accurate barrier regions can be obtained and good robustness can be achieved through the improved V parallax method.
(2) Compared with the traditional convolution kernel, the convolution kernel has the same receptive field as the traditional convolution kernel, and simultaneously greatly reduces the parameter quantity, thereby reducing the calculated quantity and being beneficial to overcoming the overfitting phenomenon by less parameter quantity.
(3) The invention improves the V parallax method for detecting the obstacle area, can effectively and stably extract the road straight line in the V parallax image, further realizes the obstacle area detection in the image, and has higher detection precision and detection robustness.
Drawings
Figure 1 is a schematic view of the obstacle detection system disclosed in the present invention,
figure 2 is a general flow chart of the obstacle detection method disclosed in the present invention,
figure 3 is a flow chart of a portion of a design twin convolutional neural network to generate a disparity map,
figure 4 is a flow chart of a portion of an improved V-parallax method for detecting obstacles,
figure 5 is a schematic diagram of the structure of a convolutional neural network designed by the present invention,
fig. 6 is a schematic diagram of a new convolution kernel structure designed by the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
Referring to fig. 1, a schematic structural diagram of a binocular vision barrier detection system based on a convolutional neural network according to the present invention is shown. This binocular vision obstacle detection system based on convolutional neural network includes: the device comprises an image acquisition module and an obstacle detection module. In the embodiment, two industrial cameras which are horizontally arranged in parallel and have the models of PikeF-100 are adopted in the image acquisition module for acquiring left and right optical images in an actual scene, and an IEEE-1394b interface is used for transmitting image data to a computer through an acquisition card for subsequent processing. In the embodiment of the invention, the obstacle detection module adopts a computer provided with a GPU of NVIDIAGTX1070 to process the acquired binocular image data, so as to fulfill the aim of obstacle area detection.
The invention also discloses a binocular vision obstacle detection method based on the convolutional neural network, and FIG. 2 is a general flow chart of the obstacle detection method disclosed by the invention; FIG. 3 is a partial flow diagram of a design of a twin convolutional neural network to generate a disparity map; fig. 4 is a partial flow chart of the obstacle detection by the improved V-parallax method of the present invention, which specifically includes the following steps:
(1) acquiring an image pair of a scene to be detected by using a binocular camera, and then transmitting the acquired image to a computer for processing;
(2) the method adopts a window with the size of 3 × 3 to carry out median filtering on the image pair, effectively removes salt-pepper noise and speckle noise in the image, and retains the contour and the details of the image;
(3) the embodiment of the invention calibrates the binocular camera by using a Zhang Zhengyou calibration method, obtains the internal parameters and the distortion parameters, and corrects the binocular image pair to obtain the coplanar and line-aligned image pair;
(4) designing a twin convolutional neural network to generate a disparity map of an accurate binocular image, which comprises the following specific steps:
a) a twin convolutional neural network structure as shown in FIG. 5 is designed, and parameters of left and right branches are shared. The twin convolutional neural network is composed of a feature extraction sub-network (L1-L9) and a feature classification sub-network (L10-L11). The left branch and the right branch of the feature extraction sub-network can respectively extract corresponding feature descriptions from the input image square and the input image long strip; and the feature classification sub-network performs dot product operation on the extracted left and right branch feature descriptions to obtain the similarity score of the pixel points to be matched in the parallax search range, and then the similarity score is used as the input of the softmax layer to obtain the parallax probability distribution.
The convolutional kernel is designed to have the same receptive field as the traditional convolutional kernel, 25 parameters of the traditional convolutional kernel are reduced to 9 parameters, the calculated amount is reduced, and less parameter amount can also reduce overfitting.
b) And (6) cutting a sample. Selecting a pixel point p (x) with real parallax in the left image according to real parallax data in the KITTI data seti,yi) Extracting a 37 × 37 image block with the pixel point as the center;selecting a pixel point q in the right image, and enabling the coordinate to be (x)i,yi) And selecting 37 × 37 image block with q as center, selecting 37 × 237 image strip with the left side of the right boundary of the image block with q as center in the right image according to the parallax search range, and completing sample clipping by the method.
c) And standardizing sample data and constructing a training set. And (b) transforming the gray value of the image block clipped in the step b) into a range of [ -1,1] by using a formula (1) as an input of the network.
Figure GDA0001453637400000061
Wherein the content of the first and second substances,
Figure GDA0001453637400000067
Figure GDA0001453637400000062
in the formula, U is an image block after standardization; x is a cropped original image block;
Figure GDA0001453637400000063
is the average of the image blocks X; x is the number ofiIs the pixel value in image block X; s is the standard deviation of the image block X; n is the number of pixels in the image block.
d) And training the twin convolutional neural network. Training the twin convolutional neural network designed by step a) with an adaptive moment-estimated stochastic gradient descent algorithm (Adam). According to the application scenario of the invention, the mutual entropy loss function is modified as follows:
Figure GDA0001453637400000064
wherein the content of the first and second substances,
Figure GDA0001453637400000065
where j (ω) represents the mutual entropy loss for a set of samples; p is a radical ofi(diω) represents the probability distribution of the ith sample; diIs a predicted disparity value;
Figure GDA0001453637400000066
is the true disparity value; lambda [ alpha ]1,λ2,λ3Is a preset value. In the present invention, λ is set individually1＝0.5,λ2＝0.2,λ3＝0.05。
e) And (5) calculating a disparity map. Taking binocular image pairs and standardizing the binocular image pairs by using the step c). Extracting 64-dimensional features of pixel points in the left image and the right image by using the network model trained in the step d), and respectively recording the features as SL(p) and SR(q); will SL(p) and SR(q) performing dot product operation to obtain an image pair similarity score, and taking an inverse number as a matching cost:
CCNN(p,d)＝-s(<SL(p),SR(q)>) (6)
wherein s: (<SL(p),SR(q)>) Representing the image pair similarity score. And finally, selecting a point with the minimum matching cost as a matching point for parallax selection in a parallax search range by a Winner-Take-All strategy (Winner-Take-All, WTA), and further generating a parallax map.
(5) The method comprises the following steps of processing a parallax image by an improved V parallax method so as to detect an obstacle area in the image:
f) and (3) calculating the gradient of the parallax image array direction generated in the step (4) by adopting a Prewitt operator, reserving the parallax at the position where the gradient is a negative value, setting the parallax at other positions to be 0, filtering out the pixel points of the obstacle, and generating the filtered parallax image.
g) Counting the same gray value number of each line of pixels of the filtered disparity map generated in the step f) to generate a V disparity map I1。
h) Search for V disparity map I1The maximum value of each row in the image is reserved, the gray value of the rest pixel points is set to be 0, and a maximum value V parallax image I is generated2。
i) Setting a threshold value T, and2the gray value of the pixel greater than T is set to 1,setting the rest to 0 to generate a V parallax binary image I only containing road information3The calculation formula of the threshold value T is as follows:
Figure GDA0001453637400000071
in the formula, xiIs I2The value of each pixel; n is I2The total number of middle pixels; 1{ xiNot equal to 0} denotes xiAnd if not, taking 1, otherwise, taking 0.
j) Extraction of I by hough line detection method3And (5) making the middle road straight.
k) And calculating the slope k and the intercept b of the image coordinate system of the road straight line extracted in the step j) in the V disparity map by using a straight line two-point equation.
l) scanning the disparity map D (x, y, D) point by point from bottom to top from left to right according to a raster scanning method, and calculating f ═ kd + b for each pixel point, wherein D is a disparity value, and f is a vertical coordinate of the pixel point of the disparity map projected in the V disparity map.
m) if (f-y)>T1And projecting the pixel point in the disparity map above a straight line of the road surface in the V disparity map, namely, being higher than the road surface, and taking the pixel point as an obstacle point. The embodiment of the invention takes the threshold value T1＝5。
And n) repeating the steps l) and m) until the parallax image is completely scanned, and obtaining the obstacle area in the image.
The foregoing is only a preferred embodiment of the present invention. The present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof, and it is therefore intended that all such equivalent changes and modifications as would be obvious to one skilled in the art be included herein are deemed to be within the scope and spirit of the present invention as defined by the appended claims.

Claims (1)

1. A detection method of a binocular vision barrier detection system based on a convolutional neural network is characterized in that the detection system based on the convolutional neural network consists of an image acquisition module and a barrier detection module which are connected; the image acquisition module is used for acquiring a left image and a right image in a scene; the obstacle detection module is used for carrying out corresponding data processing on the acquired image data to acquire a final accurate obstacle area; the image acquisition modules are two industrial cameras which are horizontally arranged in parallel and have the models of Pike F-100, and adopt IEEE-1394b interfaces to transmit image data to a computer through an acquisition card for subsequent processing; the obstacle detection module is a computer provided with a GPU of NVIDIAGTX 1070; the detection method is characterized by comprising the following steps:
(1) acquiring a binocular image from an image acquisition module, and preprocessing the binocular image to eliminate noise in the image;
(2) calibrating the image acquisition module, acquiring parameters of internal parameters and distortion of the image acquisition module, and correcting the binocular image acquired in the step (1);
(3) designing a twin convolution neural network to generate a disparity map of an accurate binocular image;
(4) processing the disparity map acquired in the step (3) by using an improved V disparity method so as to detect an obstacle region in the image;
the specific method for designing the twin convolutional neural network to generate the disparity map of the accurate binocular image in the step (3) comprises the following steps of:
a) designing a twin convolutional neural network structure, wherein the left branch and the right branch of the twin convolutional neural network structure are shared by parameters, and the twin convolutional neural network is composed of a feature extraction sub-network and a feature classification sub-network; the left branch and the right branch of the feature extraction sub-network can respectively extract corresponding feature descriptions from an input image square and an input image long strip; the feature classification sub-network performs dot product operation on the extracted left and right branch feature descriptions to obtain similarity scores of pixel points to be matched in a parallax search range, and then the similarity scores are used as input of a softmax layer to obtain parallax probability distribution;
the feature extraction sub-networks are all composed of convolution layers, convolution kernels in the convolution layers adopt a mode of transmitting information at intervals, meanwhile, each layer uses a BatchNormalization technology and a PRELU activation function, and the last layer of convolution layer does not use the activation function;
b) cutting a sample; selecting real parallax from the left image according to the real parallax data in the KITTI data setPixel point p (x)i,yi) Extracting an image block with the pixel point as a center; selecting a pixel point q in the right image, and enabling the coordinate to be (x)i,yi) Selecting an image block by taking q as a center; according to the parallax search range, selecting a left image strip of the right boundary of an image square with q as the center in the right image, and finishing sample cutting by the method;
c) standardizing sample data and constructing a training set; transforming the grey value of the image block cut in the step b) into a range of [ -1,1], and taking the grey value as the input of the network;
d) training a twin convolutional neural network; training the twin convolutional neural network designed by the step a) by adopting a random gradient descent algorithm (Adam) of adaptive moment estimation;
e) calculating a disparity map; taking a binocular image pair, standardizing the binocular image pair by adopting the step c), extracting pixel point characteristic descriptions in the left image and the right image by using the network model trained in the step d), and respectively recording the pixel point characteristic descriptions as SL(p) and SR(q); will SL(p) and SR(q) performing dot product operation to obtain an image pair similarity score, and taking an inverse number as a matching cost:
CCNN(p,d)＝-s(＜SL(p),SR(q)＞)
wherein S (< S)L(p),SR(q) >) represents an image pair similarity score; finally, in the parallax searching range, selecting a point with the minimum matching cost as a matching point for parallax selection, and further generating a parallax map;
the specific method for detecting an obstacle region in an image by processing the disparity map acquired in step (3) by using the improved V-disparity method in step (4) includes:
f) calculating the gradient of the parallax image in the column direction by adopting a Prewitt operator, reserving the parallax at the position where the gradient is a negative value, filtering out the pixel points of the obstacle, and generating a filtered parallax image;
g) counting the number of the same gray values of pixels in each line of the filtered disparity map generated in the step f) to generate a V disparity map I1；
h) Search for V disparity map I1The maximum value of each row in the image is reserved, and the gray scales of the rest pixel points are keptSetting the value to 0 to generate a maximum value V disparity map I2；
i) Setting a threshold value T, and2setting the gray value of the pixel larger than T to be 1, and setting the rest to be 0, and generating a V parallax binary image I only containing road information3The calculation formula of the threshold value T is as follows:
Figure FDA0002457476100000021
in the formula, xiIs I2The value of each pixel; n is I2The total number of middle pixels; 1{ xiNot equal to 0} denotes xiTaking 1 when not equal to 0, otherwise, taking 0;
j) extraction of I by hough line detection method3A middle road straight line;
k) calculating the slope k and the intercept b of the image coordinate system of the road straight line extracted in the step j) in the V parallax map by using a straight line two-point equation;
l) scanning a disparity map D (x, y, D) point by point from bottom to top from left to right according to a raster scanning method, and calculating f ═ kd + b for each pixel point, wherein D is a disparity value, and f is a longitudinal axis coordinate of the disparity map pixel point projected on a V disparity map;
m) when (f-y) > T1If the pixel point in the disparity map is projected above the straight line of the road surface in the V disparity map, namely higher than the road surface, the pixel point is judged to be an obstacle point;
n) repeating the steps l) and m) until the parallax map is completely scanned, and obtaining the obstacle area in the image.
CN201710697239.2A 2017-08-15 2017-08-15 Convolutional neural network-based binocular vision barrier detection system and method Active CN107506711B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710697239.2A CN107506711B (en) 2017-08-15 2017-08-15 Convolutional neural network-based binocular vision barrier detection system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710697239.2A CN107506711B (en) 2017-08-15 2017-08-15 Convolutional neural network-based binocular vision barrier detection system and method

Publications (2)

Publication Number Publication Date
CN107506711A CN107506711A (en) 2017-12-22
CN107506711B true CN107506711B (en) 2020-06-30

Family

ID=60690941

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710697239.2A Active CN107506711B (en) 2017-08-15 2017-08-15 Convolutional neural network-based binocular vision barrier detection system and method

Country Status (1)

Country Link
CN (1) CN107506711B (en)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108596009A (en) * 2017-12-29 2018-09-28 西安智加科技有限公司 A kind of obstacle detection method and system for agricultural machinery automatic Pilot
CN108734693B (en) * 2018-03-30 2019-10-25 百度在线网络技术（北京）有限公司 Method and apparatus for generating information
CN110443841B (en) * 2018-05-02 2022-08-05 杭州海康威视数字技术股份有限公司 Method, device and system for measuring ground depth
CN108648161B (en) * 2018-05-16 2020-09-01 江苏科技大学 Binocular vision obstacle detection system and method of asymmetric kernel convolution neural network
CN109241855B (en) * 2018-08-10 2022-02-11 西安交通大学 Intelligent vehicle travelable area detection method based on stereoscopic vision
JP7166108B2 (en) 2018-08-31 2022-11-07 株式会社小松製作所 Image processing system, display device, image processing method, trained model generation method, and training data set
CN109460709B (en) * 2018-10-12 2020-08-04 南京大学 RTG visual barrier detection method based on RGB and D information fusion
CN111353331B (en) * 2018-12-20 2023-09-05 浙江欣奕华智能科技有限公司 Target object detection method, detection device and robot
CN109631850B (en) * 2019-01-03 2021-01-01 甘肃大禹九洲空间信息科技有限公司 Inclined camera shooting relative positioning method based on deep learning
CN109887019B (en) * 2019-02-19 2022-05-24 北京市商汤科技开发有限公司 Binocular matching method and device, equipment and storage medium
CN111723926B (en) * 2019-03-22 2023-09-12 北京地平线机器人技术研发有限公司 Training method and training device for neural network model for determining image parallax
CN111898396A (en) * 2019-05-06 2020-11-06 北京四维图新科技股份有限公司 Obstacle detection method and device
CN112926368B (en) * 2019-12-06 2024-01-16 北京京东乾石科技有限公司 Method and device for identifying obstacle
CN110989636B (en) * 2020-02-26 2020-08-07 北京三快在线科技有限公司 Method and device for predicting track of obstacle
CN111399505B (en) * 2020-03-13 2023-06-30 浙江工业大学 Mobile robot obstacle avoidance method based on neural network
WO2021226776A1 (en) * 2020-05-11 2021-11-18 华为技术有限公司 Vehicle drivable area detection method, system, and automatic driving vehicle using system
CN112233136B (en) * 2020-11-03 2021-10-22 上海西井信息科技有限公司 Method, system, equipment and storage medium for alignment of container trucks based on binocular recognition
CN112348293A (en) * 2021-01-07 2021-02-09 北京三快在线科技有限公司 Method and device for predicting track of obstacle
CN112861976B (en) * 2021-02-11 2024-01-12 温州大学 Sensitive image identification method based on twin graph convolution hash network
CN114638898A (en) * 2022-05-23 2022-06-17 中国人民解放军国防科技大学 Small-sized flight target detection method and device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102164296A (en) * 2011-06-16 2011-08-24 上海大学 System and method for full-angular parallax stereoscopic imaging based on single DLP (digital light processing) projection

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103679127B (en) * 2012-09-24 2017-08-04 株式会社理光 The method and apparatus for detecting the wheeled region of pavement of road
KR20150120805A (en) * 2014-04-18 2015-10-28 한양대학교 산학협력단 Method and system for detecting human in range image
CN105335955B (en) * 2014-07-17 2018-04-10 株式会社理光 Method for checking object and object test equipment
CN105550665B (en) * 2016-01-15 2019-01-25 北京理工大学 A kind of pilotless automobile based on binocular vision can lead to method for detecting area
CN105956597A (en) * 2016-05-04 2016-09-21 浙江大学 Binocular stereo matching method based on convolution neural network
CN106952274B (en) * 2017-03-14 2019-06-21 西安电子科技大学 Pedestrian detection and distance measuring method based on stereoscopic vision

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102164296A (en) * 2011-06-16 2011-08-24 上海大学 System and method for full-angular parallax stereoscopic imaging based on single DLP (digital light processing) projection

Also Published As

Publication number Publication date
CN107506711A (en) 2017-12-22

Similar Documents

Publication Publication Date Title
CN107506711B (en) Convolutional neural network-based binocular vision barrier detection system and method
CN110569704B (en) Multi-strategy self-adaptive lane line detection method based on stereoscopic vision
CN110487562B (en) Driveway keeping capacity detection system and method for unmanned driving
CN108648161B (en) Binocular vision obstacle detection system and method of asymmetric kernel convolution neural network
CN107274445B (en) Image depth estimation method and system
CN111723721A (en) Three-dimensional target detection method, system and device based on RGB-D
CN109034184B (en) Grading ring detection and identification method based on deep learning
CN116258817B (en) Automatic driving digital twin scene construction method and system based on multi-view three-dimensional reconstruction
CN111209840B (en) 3D target detection method based on multi-sensor data fusion
CN111832461A (en) Non-motor vehicle riding personnel helmet wearing detection method based on video stream
CN111738071B (en) Inverse perspective transformation method based on motion change of monocular camera
CN114972968A (en) Tray identification and pose estimation method based on multiple neural networks
CN107944350B (en) Monocular vision road identification method based on appearance and geometric information fusion
CN114120283A (en) Method for distinguishing unknown obstacles in road scene three-dimensional semantic segmentation
CN115308732A (en) Multi-target detection and tracking method integrating millimeter wave radar and depth vision
CN112836573A (en) Lane line image enhancement and completion method based on confrontation generation network
CN116279592A (en) Method for dividing travelable area of unmanned logistics vehicle
CN116978009A (en) Dynamic object filtering method based on 4D millimeter wave radar
CN117284320A (en) Vehicle feature recognition method and system for point cloud data
CN115100618B (en) Multi-source heterogeneous perception information multi-level fusion characterization and target identification method
CN113723432B (en) Intelligent identification and positioning tracking method and system based on deep learning
CN105740805B (en) One kind combining method for detecting lane lines based on multizone
CN113569803A (en) Multi-mode data fusion lane target detection method and system based on multi-scale convolution
EP4070278A1 (en) Method to determine the depth from images by self-adaptive learning of a neural network and system thereof
CN113345035A (en) Binocular camera-based gradient real-time prediction method and system and computer-readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant