CN108596009A - A kind of obstacle detection method and system for agricultural machinery automatic Pilot - Google Patents
A kind of obstacle detection method and system for agricultural machinery automatic Pilot Download PDFInfo
- Publication number
- CN108596009A CN108596009A CN201711483388.5A CN201711483388A CN108596009A CN 108596009 A CN108596009 A CN 108596009A CN 201711483388 A CN201711483388 A CN 201711483388A CN 108596009 A CN108596009 A CN 108596009A
- Authority
- CN
- China
- Prior art keywords
- obstacle detection
- information
- image data
- detection method
- range information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 42
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 25
- 238000012886 linear function Methods 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 4
- 230000004888 barrier function Effects 0.000 abstract description 40
- 238000010586 diagram Methods 0.000 description 14
- 238000000034 method Methods 0.000 description 12
- 238000012549 training Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 5
- 239000013256 coordination polymer Substances 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000011900 installation process Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 241000196324 Embryophyta Species 0.000 description 1
- 241000607479 Yersinia pestis Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 244000144972 livestock Species 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012372 quality testing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000611 regression analysis Methods 0.000 description 1
- 230000001932 seasonal effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Electromagnetism (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of obstacle detection methods for agricultural machinery automatic Pilot, including：The image data with range information is obtained using binocular vision collector；Image data is input in depth convolutional neural networks, image data is divided into multiple subgraphs by depth convolutional neural networks, each subgraph is handled, it predicts the classification information of object block message and object block that may include in each subgraph, the confidence level score value for being predicted object is calculated according to object block message and classification information；It determines the believable position coordinates for identifying object and range information and exports.The invention also discloses a kind of obstacle detection systems for agricultural machinery automatic Pilot, including：Image data acquisition module, identification module and output module.The present invention exports the location information for recognizing object and range information according to the image information got, is effectively detected to farmland barrier, reduces background and misidentifies and send out alarm according to the danger zone of setting.
Description
Technical field
The present invention relates to technical field of information processing more particularly to a kind of detection of obstacles sides for agricultural machinery automatic Pilot
Method and system.
Background technology
During farmland operation, inevitably there are various barriers in the fields, if cannot be by obstacle quality testing
It measures and, it may occur that major accident causes casualties or economic loss, it is therefore desirable to have to the barrier inside farmland
The detection of effect.But there are its specific difficult point and challenge for the identification of the barrier inside farmland.On the one hand, crops and barrier
Hindering object, often in color and texture etc., there are similitudes, to bring very high choose in the accuracy in detection of barrier
War；On the other hand, the various objects in field, including the color and texture in crops and field can also become with seasonal variations
Change；The difficulty to detection of obstacles is more increased in this way.
A kind of existing farmland obstacle detection method is：Farmland figure is acquired by field navigation picture collecting device first
Then farmland image is carried out the operations such as denoising gray processing, wavelet transformation, maximum between-cluster variance segmentation, finally obtains image by picture
Drop shadow curve judges farmland barrier according to the trip point of curve.This method is led to based on traditional image-recognizing method
It crosses wavelet transformation and corresponding partitioning algorithm (such as OTSU) detaches object, this method needs to rely on crops and barrier
Hinder object in difference of both color and frequency to be detected, once the two similarity in terms of color or frequency is excessively high,
It can then fail to the detection method of barrier.
Also a kind of detection method to farmland barrier, this method only rely on the detection that frequency information completes barrier,
When the frequency that barrier and crop generate is close, it is not easy to detect that barrier, detection result are bad.
Another farmland obstacle detection method, this method are：First farmland image is carried out based on frequency difference
Segmentation, then detached based on colouring information progress barrier, field-crop, this method can promote the recall rate of barrier,
But to segmentation stage of the barrier based on frequency information, changeable background frequency can lead to a part at maximum interference
Barrier was because interfering the missing inspection that mostly occurs.
Therefore, these obstacle detection methods have the following problems in the prior art：
1, background misidentifies：Above-mentioned detection method needs to rely on crops and barrier in color and/or frequency
Difference is detected, and when the two is when similarity is excessively high in terms of color or frequency, is just not easy to detect barrier, therefore
Background becomes greatly interference, and background is caused to misidentify.
2, the object space and range information of missing detection：Above-mentioned detection method only simply knows object
Not, specific object space and range information are not provided, can not effectively find barrier, make accurate judgement.
Invention content
The purpose of the present invention is exactly to propose that a kind of reduction background is accidentally known to solve above-mentioned problems of the prior art
The position of object and the obstacle detection method of range information and system can be obtained not and accurately.
For overcome the deficiencies in the prior art, it includes field-crop and obstacle that the present invention is obtained by binocular vision collector
The picture point cloud with depth (distance) information of object, then use depth convolutional neural networks to the image collected carry out from
Line training and On-line accoun, according to the image information got by recognize object location information and object relative to vehicle
Range information exports, and to be effectively detected to farmland barrier, and then can also send out report according to the danger zone of setting
It is alert.
In order to achieve the above object, the technical solution adopted by the present invention is：
A kind of obstacle detection method for agricultural machinery automatic Pilot includes the following steps：
Step 1, the image data with range information is obtained using binocular vision collector；
Step 2, described image data are input in depth convolutional neural networks, the depth convolutional neural networks are figure
As data are divided into multiple subgraphs, each subgraph is handled, predicts the object block that may include in each subgraph
The classification information of information and object block, and the confidence level point for being predicted object is calculated according to the object block message and classification information
The confidence level score value and predetermined threshold are relatively judged whether the prediction object is believable identification object by value；
Step 3, the believable position coordinates for identifying object and range information are determined and is exported.
Further, further comprising the steps of：Step 4, according to the position coordinates and distance of the object of the step 3 output
Information, judges whether the object is in danger zone and sends out alarm signal.
Further, the object block message includes being predicted relative coordinate in the subgraph of the center of object, quilt
Predict the confidence level of the width and height and the object block of object.
Further, the classification information is the conditional probability that the object block belongs to particular category.
Further, the depth convolutional neural networks include multiple convolutional layers and multiple maximum pond layers, wherein convolution
The activation primitive of layer includes that linear function is corrected in leakage.
Further, the position coordinates of the object include the coordinate of the coordinate and rightmost point of the Far Left point of object.
Further, the right boundary and forward length of the danger zone are determined according to vehicle dimension.
A kind of obstacle detection system for agricultural machinery automatic Pilot, including：
Image data acquisition module, for obtaining the image data with range information using binocular vision collector；
Identification module, for described image data to be input in depth convolutional neural networks, the depth convolutional Neural
Image data is divided into multiple subgraphs by network, is handled each subgraph, predict may include in each subgraph
Object block message and object block classification information, and according to the object block message and classification information calculating be predicted object
The confidence level score value and predetermined threshold are relatively judged whether the prediction object is believable identification object by confidence level score value
Body；
Output module, for determining the believable position coordinates for identifying object and range information and exporting.
Further, further include alarm module, for according to the position coordinates and range information of the object, described in judgement
Whether object is in danger zone and sends out alarm signal.
Further, the position coordinates of the object include the coordinate of the coordinate and rightmost point of the Far Left point of object.
The beneficial effects of the invention are as follows：
1, it has been obviously improved the recall rate of barrier：The present invention is got by binocular vision collector comprising crops
With the picture point cloud with depth information of barrier, and using depth convolutional neural networks the image collected is carried out offline
Training and On-line accoun, to be obviously improved the recall rate of barrier.
2, the position of object and the missing of range information are compensated for：In the prior art, not for the barrier that detects
Position and range information can be provided, cause subsequent anti-collision warning module that analytical judgment can not be effectively performed, made rationally
Early warning.The present invention utilizes binocular vision collector, obtains position and the depth information of each pixel, is then associated with each
On identified object, position and the range information of object are exported, and then effective barrier anti-collision warning can be carried out.
3, to the detection of unknown object：The present invention uses the depth convolutional neural networks algorithm based on global image information,
It can be good at identifying the object not in training data, even if can be accurate if specific classification information can not be provided
It really identifies object, and provides position and the range information of object, thus allow for effective barrier anti-collision warning.
4, background misrecognition is reduced：The depth convolutional neural networks algorithm that the present invention uses is carried out based on global image information
Reasoning, can effectively the contextual information of coded object classification and other be based on sliding window (DPM) or region (R-
CNN deep neural network algorithm) is compared, and the generation of background misrecognition can be greatly reduced.
5, the speed of service is fast：It is typically that first object is identified in the prior art, then again to the object classification of identification,
Realized by former and later two steps, and the present invention is to be carried out at the same time the identification and classification of object, so its speed of service is very
Soon, reach 25-50 milliseconds of single frames, that is, the image per second that can at most predict 40 frames, much disclosure satisfy that this in farmland
Demand in the case of low speed working scene.
6, sensor, easy for installation, information collection precision height are reduced：The present invention believes object using binocular vision collector
Breath is acquired, and avoids the sensor that buying repeats, to reduce cost, while its installation process is also more convenient；And
And the object information precision of acquisition is high, disclosure satisfy that the needs of agricultural machinery working scene.
Description of the drawings
The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a kind of flow chart of one embodiment of obstacle detection method for agricultural machinery automatic Pilot of the invention；
Fig. 2 is a kind of flow of another embodiment of obstacle detection method for agricultural machinery automatic Pilot of the invention
Figure；
Fig. 3 is a kind of structural representation of one embodiment of obstacle detection system for agricultural machinery automatic Pilot of the invention
Figure；
Fig. 4 is that a kind of structure of another embodiment of obstacle detection system for agricultural machinery automatic Pilot of the invention is shown
It is intended to；
Fig. 5 is the configuration diagram of the depth convolutional neural networks in one embodiment of the present of invention；
Fig. 6 is that the vehicle of one embodiment of the present of invention acquires image schematic diagram；
Fig. 7 is the coordinate schematic diagram of the Image Acquisition of one embodiment of the present of invention；
Fig. 8 is the image segmentation schematic diagram of one embodiment of the present of invention；
Fig. 9 a are schematic diagram of a scenario of one embodiment of the present of invention barrier inside danger zone；
Fig. 9 b are schematic diagram of a scenario of an alternative embodiment of the invention barrier not in danger zone.
Specific implementation mode
It is described below for the open present invention so that those skilled in the art can realize the present invention.It is excellent in being described below
Embodiment is selected to be only used as illustrating, it may occur to persons skilled in the art that other obvious deformations.It defines in the following description
The present invention basic principle can be applied to other embodiments, deformation scheme, improvement project, equivalent program and do not carry on the back
Other technologies scheme from the spirit and scope of the present invention.
It is understood that term " one " is interpreted as " at least one " or " one or more ", i.e., in one embodiment,
The quantity of one element can be one, and in a further embodiment, the quantity of the element can be multiple, and term " one " is no
It can be interpreted as the limitation to quantity.；Term used herein is only used for describing the purpose of various embodiments and is not intended to limit.
As used herein, singulative is intended to also include plural form, makes an exception unless the context clearly dictates.It will further be understood that art
Language " comprising " and/or " having " specify the feature, number, step, operation, component, member when being used in this specification
The presence of part or combinations thereof, and it is not excluded for one or more of the other feature, number, step, operation, component, element or its group
In the presence of or it is additional.
Fig. 1 is a kind of one embodiment flow chart of obstacle detection method for agricultural machinery automatic Pilot of the present invention,
Referring to Fig.1, in the present embodiment, a kind of obstacle detection method for agricultural machinery automatic Pilot is provided, is included the following steps：
Step 1, the image data with range information is obtained using binocular vision collector；In the present embodiment, it uses
Camera is as binocular vision collector, but the present invention is not limited thereto, and other image collecting devices can also be used to realize
The binocular vision collector；
Step 2, feature extraction, mapping and sampling are carried out to described image data, and pre- using depth convolutional neural networks
The classification information of object block message and object block therein is surveyed, and then obtains the confidence level score value for being predicted object, and can by this
Reliability score value and a predetermined threshold relatively judge whether the prediction object is believable identification object；
Step 3, the believable position coordinates for identifying object and range information are determined and is exported.
Further, the image data obtained in step 1 is the picture point cloud with depth information, each image pixel table
It is shown as (x, y, d), wherein (x, y) is the coordinate in image coordinate system, d is actual object corresponding to the pixel (x, y) to taking the photograph
As the distance of head (the present embodiment includes left camera and right camera), that is, object is to the distance of vehicle.
It is that the vehicle of one embodiment of the present of invention acquires the schematic diagram of image, wherein binocular camera referring to Fig. 6, Fig. 6
Mounted on vehicle roof position, face vehicle front, A, B are two objects of vehicle front.
Fig. 7 is the coordinate schematic diagram of the Image Acquisition of one embodiment of the present of invention, it will be seen in fig. 7 that objects in front A
The coordinate of mapping point of the LB points in the lower left corner in the image coordinate system that binocular camera acquires be (x1,y1), d1It is LB points to double
Mesh camera central point (corresponds to the point (x in image coordinate system0,y0)) actual range (not shown).
Including following sub-step further, in step 2,：
Step 2.1：The picture segmentation of input at multiple subgraphs, in the present embodiment by taking the grid for being divided into N × N as an example
(but it is not limited to this, can also be divided into other shapes or quantity).
Step 2.2：Each grid is predicted, if obtaining the information of dried object block.Here, object block refers to
According to prediction obtain, there may be the regions of object, shown in FIG as a region contour.If in an object block
The heart is fallen within some grid (subgraph), then this grid is just responsible for predicting this object；It is limited in the present embodiment every
A grid is only responsible for the most B objects of prediction, and wherein B is an empirical value, and the value of B is, for example, 2 in the present embodiment.
It is described to predict that the information of obtained object block may include five output valves：(x′,y′,W,H,Sconfidence)。
Wherein, (x ', y ') indicates to be predicted the relative coordinate of the center of object in the grid, and W and H are objects in grid
Width in image and height value, SconfidenceBy prediction object block confidence level, the confidence level is for reflecting the object
Block includes the possibility and accuracy of object, is defined as follows：
Wherein, P (object) is that object is present in object probability in the block,For the object block of prediction
(predict) and the friendship of actual object block (ground truth) and than (intersection over union).
In the present embodiment, trained depth convolutional neural networks model can determine for each object block above-mentioned automatically
P (object) andValue, to obtain the confidence level S of each object blockconfidence。
Step 2.3：While predicting object block, each grid also predicts that the object that it is included belongs to some
The conditional probability of classification, and the highest classification of probability and its probability value that predict are exported as a result.Here in order to
Optimize computational efficiency, no matter may include how many a objects or object block in a grid, only predicts the property in the grid
Body belongs to the probability of some classification.
In this way, on the basis of step 2.2 and step 2.3, so that it may to continue step 2.4：
Step 2.4：The class probability that the confidence level and step 2.3 obtained according to step 2.2 obtains, calculates in each grid
There are the confidence level score value CP that object and the object belong to classification C,
CP=P (ClassC|Object)×Sconfidence
Wherein, P (ClassC| Object) it is the conditional probability that each object belongs to classification C.
The possibility classification of the object for example including：People, livestock, electric pole, the barriers such as vehicle, crops.
Fig. 8 is the image segmentation schematic diagram of one embodiment of the present of invention, and wherein Fig. 8 a are side of the picture according to 4X4
The figure of lattice segmentation shows that the two possible object blocks for predicting and, Fig. 8 c shows to the prediction of each grid in Fig. 8 b
Classification obtaining, belonging to object, identical color indicate identical classification；Fig. 8 d obtain for the result of complex chart 8b, 8c
Final output, that is, carry classification information object block schematic diagram, wherein the classification of the object block and Fig. 8 c Fig. 8 b
Information is merged.
Regression analysis is used to the entire identification process of object in the present embodiment, certainly, the present invention can also be used poly-
Whole object is identified in the known methods such as alanysis.
Fig. 5 is the schematic diagram of the depth convolutional neural networks framework of the present embodiment, and specifically design is as shown in figure 5, in figure
24 layers of convolutional layer (Conv.Layer) and 5 maximum pond layers (Maxpool Layer) are shared, wherein preceding 23 layers of convolutional layer swashs
Function living is that linear function (Leaky Rectified Linear Function) is corrected in leakage：
The activation primitive of last layer of convolutional layer is linear function (Linear Function)：
φ (x)=λ x, the value of λ can be selected as 1 in the present embodiment, and but it is not limited to this, can also be set as other constants.
Meanwhile in order to effectively identify object of different sizes, cumulative width and height inside loss function
Square root uses in the present embodiment corrected and variance function as loss function (Loss Function/Error
Function), for example,：
Wherein, N is the number of subgraph, and B is the number of block, and obj is the number of the block comprising object, and na is not comprising object
The number of the block of body, Pi(C) it is to predict that object belongs to the probability of some classification C, (xi,yi) it is to predict the center of object at i-th
Coordinate in grid, W and H are the width and height value of object.To be identified in i-th of grid in training pattern
Object actual coordinate value,WithDeveloped width and height for the object identified in i-th of grid in training pattern
Degree, CPiThe probability occurred for object block in i-th of grid；Indicate the practical probability occurred of object block, example in i-th of grid
If value is 1 or 0, if object block occurs, otherwise value 1 is 0；Indicate that object belongs to some in i-th of grid
The actual probability of classification, such as value are 1 or 0, if object belongs to the category, otherwise value 1 is 0.
Value is 0 or 1, if j-th of object block inside i-th of grid is responsible for predicting object,Value
It is 1, is otherwise 0.
Value is 1 or 0, if j-th of object block inside i-th of grid is responsible for predicting object,Value be
0, it is otherwise 1.
Likewise,Value is also 0 or 1, if object appears in i-th of grid,Value be 1, otherwise
It is 0.
In view of to optimize the identification of object block and the classification of object simultaneously, brought so to reduce the block not comprising object
Influence of noise.Include the block of object and not comprising the block of object to distinguish, introduces two coefficients in above-mentioned formula, respectively
For λobjAnd λna, wherein λobjFor the coefficient of the block comprising object, λnaFor the coefficient of the block not comprising object, include to increase
The weight of the block of object reduces the weight of the block not comprising object.
λobjAnd λnaValue rule of thumb set, in the present embodiment, λ can be setobjIt is much larger than λ for onena's
Value, such as λobjCan value be 4, λnaCan value be 0.4.
In addition, in the reasonable scope, it will be understood by those skilled in the art that the number of plies of depth convolutional neural networks and every layer
The parameter of convolutional layer can all change；Parameter in the parameter and loss function of activation primitive is also that can change, this
A little change can achieve the purpose that the present invention will realize, all within protection scope of the present invention.
Step 2.5：The confidence level score value CP for each object that prediction obtains is compared with the empirical value of setting, is judged
Whether the object identified is credible, and all believable identification objects are added in a recognized list.
Step 3：It determines geographical location and the range information of each believable identification object, and exports each object of identification
The geographical coordinate and range information of Far Left point (leftmost) the rightmost point (rightmost) of body, are embodied as
(xleftmost,yleftmost,xrightmost,yrightmost, d), wherein d is distance of the object to binocular camera.
With reference to Fig. 2, another embodiment of the present invention also provides a kind of obstacle detection method for agricultural machinery automatic Pilot,
This method and method shown in FIG. 1 are essentially identical, and difference lies in further include step 4 further, according in above-mentioned steps 3
The geographical location of identified object and range information judge the object whether on the travel path in current vehicle, if
It then alarms on travel path.
In one embodiment, if meeting the following conditions, judge that the object is on the travel path of current vehicle：
(xdanger-zone-left≤xrightmostOr xdanger-zone-right≥
xleftmost) and (d≤Ddanger-zone-distance),
Wherein xdanger-zone-leftAnd xdanger-zone-rightIt is the left and right side of the danger zone determined according to the width of vehicle body
Boundary, such as：
xdanger-zone-left=-λv×Wv,
xdanger-zone-right=λv×Wv,
Wherein WvFor the width of vehicle body, Ddanger-zone-distanceIt is according to danger area defined in vehicle real work situation
The forward length in domain；λvFor constant, value range is, for example, 1.5~2.5.
It is schematic diagram of a scenario of the barrier A in danger zone in one embodiment of the present of invention referring to Fig. 9 a, Fig. 9 a, this
When can send out alarm signal, λ in the present embodimentvPreferably value is 2.
Fig. 9 b are schematic diagram of a scenario of the barrier B not in danger zone in an alternative embodiment of the invention, at this moment not
Alarm signal can be sent out, λ in the present embodimentvPreferably value is 2.
Referring to Fig. 3, Fig. 3 is a kind of one embodiment of obstacle detection system for agricultural machinery automatic Pilot of the invention
Structural schematic diagram, including：
Image data acquisition module, for obtaining the image data with range information using binocular vision camera；
Identification module for carrying out feature extraction, mapping and sampling to described image data, and utilizes depth convolutional Neural
The classification information of neural network forecast object block message and object block therein, and then the confidence level score value for being predicted object is obtained, and
The confidence level score value and a predetermined threshold are relatively judged whether the prediction object is believable identification object；
Output module, for determining the believable position coordinates for identifying object and range information and exporting.
Referring to Fig. 4, another embodiment of the present invention also provides a kind of obstacle detection system for agricultural machinery automatic Pilot,
Its as shown in figure 3 essentially identical, difference lies in, further include alarm module, for according to the position coordinates of the object and away from
From information, judge whether the object is in danger zone and sends out alarm signal.
Further, the position coordinates of the object include the coordinate of the coordinate and rightmost point of object Far Left point.
The method using binocular vision collector and depth convolutional neural networks of the present invention is equally applicable to pest and disease damage knowledge
Not, other scenes such as weed identification, maturity.
In conclusion the present invention is obtained by binocular vision collector carries depth comprising field-crop and barrier
Then the picture point cloud of (distance) information uses depth convolutional neural networks to carry out off-line training and online to the image collected
Reasoning, it is according to the image information got that the location information and object that recognize object is defeated relative to the range information of vehicle
Go out, to be effectively detected to farmland barrier, and then can also send out alarm according to the danger zone of setting.
Present invention obtains following advantageous effects as a result,：
1, it has been obviously improved the recall rate of barrier：The present invention is got by binocular vision collector comprising crops
With the picture point cloud with depth information of barrier, and using depth convolutional neural networks the image collected is carried out offline
Training and On-line accoun, to be obviously improved the recall rate of barrier.
2, the position of object and the missing of range information are compensated for：In the prior art, not for the barrier that detects
Position and range information can be provided, cause subsequent anti-collision warning module that can not effectively be analyzed and determined, made rationally
Early warning.The present invention utilizes binocular vision collector, obtains position and the depth information of each pixel, is then associated with each
On identified object, position and the range information of object are exported, and then effective barrier anti-collision warning can be carried out.
3, to the detection of unknown object：The present invention uses the deep neural network algorithm based on global image information, can
The object not in training data is identified well, even if can be accurately if can not providing specific classification information
It identifies object, and provides position and the range information of object, thus allow for effective barrier anti-collision warning.
4, background misrecognition is reduced：The deep neural network algorithm that the present invention uses is pushed away based on global image information
Reason, can effectively the contextual information of coded object classification and other be based on sliding window (DPM) or region (R-CNN)
Deep neural network algorithm compare, can greatly reduce background misrecognition generation.
5, the speed of service is fast：It is typically that first object is identified in the prior art, then again to the object classification of identification,
Realized by former and later two steps, and the present invention is to be carried out at the same time the identification and classification of object, so its speed of service is very
Soon, reach 25-50 milliseconds of single frames, that is, the image per second that can at most predict 40 frames, much disclosure satisfy that this in farmland
Demand in the case of low speed working scene.
6, sensor, easy for installation, information collection precision height are reduced：The present invention believes object using binocular vision collector
Breath is acquired, and avoids the sensor that buying repeats, to reduce cost, while its installation process is also more convenient；And
And the object information precision of acquisition is high, disclosure satisfy that the needs of agricultural machinery working scene.
Certainly, the invention may also have other embodiments, without deviating from the spirit and substance of the present invention, ripe
It knows those skilled in the art and makes various corresponding change and deformations, but these corresponding changes and change in accordance with the present invention
Shape should all belong to the protection domain of appended claims of the invention.
Claims (10)
1. a kind of obstacle detection method for agricultural machinery automatic Pilot, which is characterized in that include the following steps：
Step 1, the image data with range information is obtained using binocular vision collector；
Step 2, described image data are input in depth convolutional neural networks, the depth convolutional neural networks are picture number
According to multiple subgraphs are divided into, each subgraph is handled, predicts the object block message that may include in each subgraph
With the classification information of object block, and the confidence level score value for being predicted object is calculated according to the object block message and classification information,
The confidence level score value and predetermined threshold are relatively judged whether the prediction object is believable identification object；
Step 3, the believable position coordinates for identifying object and range information are determined and is exported.
2. obstacle detection method according to claim 1, which is characterized in that further comprising the steps of：
Step 4, according to the position coordinates and range information of the object of the step 3 output, judge the object whether in danger
In the domain of danger zone and send out alarm signal.
3. obstacle detection method according to claim 1, which is characterized in that the object block message includes being predicted object
Relative coordinate, the confidence level of the width and height and the object block that are predicted object of the center of body in the subgraph.
4. obstacle detection method according to claim 1, which is characterized in that the classification information is the object block category
In the conditional probability of particular category.
5. obstacle detection method according to claim 1, it is characterised in that：The depth convolutional neural networks include more
A convolutional layer and multiple maximum pond layers, the wherein activation primitive of convolutional layer include that linear function is corrected in leakage.
6. obstacle detection method according to claim 1, which is characterized in that the position coordinates of the object include object
Far Left point coordinate and rightmost point coordinate.
7. obstacle detection method according to claim 2, which is characterized in that determine the danger area according to agricultural machinery size
The right boundary and forward length in domain.
8. a kind of obstacle detection system for agricultural machinery automatic Pilot, which is characterized in that including：
Image data acquisition module, for obtaining the image data with range information using binocular vision collector；
Identification module, for described image data to be input in depth convolutional neural networks, the depth convolutional neural networks
Image data is divided into multiple subgraphs, each subgraph is handled, predicts the object that may include in each subgraph
The classification information of body block message and object block, and the credible of object is predicted according to the object block message and classification information calculating
It spends score value, the confidence level score value and predetermined threshold is relatively judged whether the prediction object is believable identification object；
Output module, for determining the believable position coordinates for identifying object and range information and exporting.
9. obstacle detection system according to claim 8, which is characterized in that further include：Alarm module, for according to institute
The position coordinates and range information for stating object, judge whether the object is in danger zone and sends out alarm signal.
10. obstacle detection system according to claim 8, which is characterized in that the position coordinates of the object include object
The coordinate of the coordinate and rightmost point of the Far Left point of body.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711483388.5A CN108596009A (en) | 2017-12-29 | 2017-12-29 | A kind of obstacle detection method and system for agricultural machinery automatic Pilot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711483388.5A CN108596009A (en) | 2017-12-29 | 2017-12-29 | A kind of obstacle detection method and system for agricultural machinery automatic Pilot |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108596009A true CN108596009A (en) | 2018-09-28 |
Family
ID=63633144
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711483388.5A Pending CN108596009A (en) | 2017-12-29 | 2017-12-29 | A kind of obstacle detection method and system for agricultural machinery automatic Pilot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108596009A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109409244A (en) * | 2018-09-29 | 2019-03-01 | 维沃移动通信有限公司 | A kind of object puts the output method and mobile terminal of scheme |
| CN109540138A (en) * | 2018-11-12 | 2019-03-29 | 中南大学 | Indoor navigation method, system and the readable memory of view-based access control model neural network |
| CN109711279A (en) * | 2018-12-08 | 2019-05-03 | 南京赫曼机器人自动化有限公司 | A kind of obstacle detection method for agricultural environment |
| CN109740443A (en) * | 2018-12-12 | 2019-05-10 | 歌尔股份有限公司 | Detect the method, apparatus and sports equipment of barrier |
| CN110084133A (en) * | 2019-04-03 | 2019-08-02 | 百度在线网络技术（北京）有限公司 | Obstacle detection method, device, vehicle, computer equipment and storage medium |
| CN110263628A (en) * | 2019-05-09 | 2019-09-20 | 杭州飞步科技有限公司 | Obstacle detection method, device, electronic equipment and storage medium |
| CN110646574A (en) * | 2019-10-08 | 2020-01-03 | 张家港江苏科技大学产业技术研究院 | Unmanned ship-based water quality conductivity autonomous detection system and method |
| CN110764512A (en) * | 2019-11-18 | 2020-02-07 | 江苏农林职业技术学院 | Agricultural machinery driving control system |
| CN110900611A (en) * | 2019-12-13 | 2020-03-24 | 合肥工业大学 | Novel mechanical arm target positioning and path planning method |
| CN112418003A (en) * | 2020-11-05 | 2021-02-26 | 湖南中联重科智能高空作业机械有限公司 | Working platform obstacle identification method and system and anti-collision method and system |
| WO2022078463A1 (en) * | 2020-10-16 | 2022-04-21 | 爱驰汽车（上海）有限公司 | Vehicle-based obstacle detection method and device |
| CN116823812A (en) * | 2023-08-25 | 2023-09-29 | 中国农业大学 | Silage corn field life detection method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102538766A (en) * | 2011-12-21 | 2012-07-04 | 武汉科技大学 | Obstacle test method for active intelligent vehicle |
| CN106004884A (en) * | 2016-07-11 | 2016-10-12 | 南昌工学院 | Method and system for realizing real-time identification and danger judgment of road conditions based on complex sensing |
| CN107392965A (en) * | 2017-07-24 | 2017-11-24 | 王华锋 | A kind of distance-finding method being combined based on deep learning and binocular stereo vision |
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
-
2017
- 2017-12-29 CN CN201711483388.5A patent/CN108596009A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102538766A (en) * | 2011-12-21 | 2012-07-04 | 武汉科技大学 | Obstacle test method for active intelligent vehicle |
| CN106004884A (en) * | 2016-07-11 | 2016-10-12 | 南昌工学院 | Method and system for realizing real-time identification and danger judgment of road conditions based on complex sensing |
| CN107392965A (en) * | 2017-07-24 | 2017-11-24 | 王华锋 | A kind of distance-finding method being combined based on deep learning and binocular stereo vision |
| CN107506711A (en) * | 2017-08-15 | 2017-12-22 | 江苏科技大学 | Binocular vision obstacle detection system and method based on convolutional neural networks |
Non-Patent Citations (1)
| Title |
|---|
| JOSEPH REDMON ET AL.: "《You Only Look Once:Unified, Real-Time Object Detection》", 《2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109409244B (en) * | 2018-09-29 | 2021-03-09 | 维沃移动通信有限公司 | Output method of object placement scheme and mobile terminal |
| CN109409244A (en) * | 2018-09-29 | 2019-03-01 | 维沃移动通信有限公司 | A kind of object puts the output method and mobile terminal of scheme |
| CN109540138A (en) * | 2018-11-12 | 2019-03-29 | 中南大学 | Indoor navigation method, system and the readable memory of view-based access control model neural network |
| CN109711279A (en) * | 2018-12-08 | 2019-05-03 | 南京赫曼机器人自动化有限公司 | A kind of obstacle detection method for agricultural environment |
| CN109711279B (en) * | 2018-12-08 | 2023-06-20 | 南京赫曼机器人自动化有限公司 | Obstacle detection method for agricultural environment |
| CN109740443A (en) * | 2018-12-12 | 2019-05-10 | 歌尔股份有限公司 | Detect the method, apparatus and sports equipment of barrier |
| CN110084133B (en) * | 2019-04-03 | 2022-02-01 | 百度在线网络技术（北京）有限公司 | Obstacle detection method, obstacle detection apparatus, vehicle, computer device, and storage medium |
| CN110084133A (en) * | 2019-04-03 | 2019-08-02 | 百度在线网络技术（北京）有限公司 | Obstacle detection method, device, vehicle, computer equipment and storage medium |
| CN110263628B (en) * | 2019-05-09 | 2021-11-23 | 杭州飞步科技有限公司 | Obstacle detection method, obstacle detection device, electronic apparatus, and storage medium |
| CN110263628A (en) * | 2019-05-09 | 2019-09-20 | 杭州飞步科技有限公司 | Obstacle detection method, device, electronic equipment and storage medium |
| CN110646574A (en) * | 2019-10-08 | 2020-01-03 | 张家港江苏科技大学产业技术研究院 | Unmanned ship-based water quality conductivity autonomous detection system and method |
| CN110646574B (en) * | 2019-10-08 | 2022-02-08 | 张家港江苏科技大学产业技术研究院 | Unmanned ship-based water quality conductivity autonomous detection system and method |
| CN110764512A (en) * | 2019-11-18 | 2020-02-07 | 江苏农林职业技术学院 | Agricultural machinery driving control system |
| CN110900611A (en) * | 2019-12-13 | 2020-03-24 | 合肥工业大学 | Novel mechanical arm target positioning and path planning method |
| WO2022078463A1 (en) * | 2020-10-16 | 2022-04-21 | 爱驰汽车（上海）有限公司 | Vehicle-based obstacle detection method and device |
| CN112418003A (en) * | 2020-11-05 | 2021-02-26 | 湖南中联重科智能高空作业机械有限公司 | Working platform obstacle identification method and system and anti-collision method and system |
| CN112418003B (en) * | 2020-11-05 | 2023-09-29 | 湖南中联重科智能高空作业机械有限公司 | Work platform obstacle recognition method and system and anti-collision method and system |
| CN116823812A (en) * | 2023-08-25 | 2023-09-29 | 中国农业大学 | Silage corn field life detection method |
| CN116823812B (en) * | 2023-08-25 | 2023-10-31 | 中国农业大学 | Silage corn field life detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108596009A (en) | A kind of obstacle detection method and system for agricultural machinery automatic Pilot | |
| JP6759474B2 (en) | Vessel automatic tracking methods and systems based on deep learning networks and average shifts | |
| Wu et al. | Detection and counting of banana bunches by integrating deep learning and classic image-processing algorithms | |
| CN104821056B (en) | Intelligent warning method based on radar and video fusion | |
| CN110068818A (en) | The working method of traffic intersection vehicle and pedestrian detection is carried out by radar and image capture device | |
| CN109145836B (en) | Ship target video detection method based on deep learning network and Kalman filtering | |
| CN112395987B (en) | SAR image target detection method based on unsupervised domain adaptive CNN | |
| CA3094424A1 (en) | Safety monitoring and early-warning method for man-machine interaction behavior of underground conveyor belt operator | |
| CN111079518B (en) | Ground-falling abnormal behavior identification method based on law enforcement and case handling area scene | |
| CN115345908B (en) | Human body posture recognition method based on millimeter wave radar | |
| CN109949229A (en) | A kind of target cooperative detection method under multi-platform multi-angle of view | |
| CN109506628A (en) | Object distance measuring method under a kind of truck environment based on deep learning | |
| CN113092807B (en) | Urban overhead road vehicle speed measuring method based on multi-target tracking algorithm | |
| CN106780551A (en) | A kind of Three-Dimensional Moving Targets detection method and system | |
| CN111091101A (en) | High-precision pedestrian detection method, system and device based on one-step method | |
| CN111048208A (en) | Indoor solitary old man walking health detection method based on laser radar | |
| CN107316024A (en) | perimeter alarm algorithm based on deep learning | |
| CN115761534A (en) | Method for detecting and tracking small target of infrared unmanned aerial vehicle under air background | |
| CN112116635A (en) | Visual tracking method and device based on rapid human body movement | |
| CN116486287A (en) | Target detection method and system based on environment self-adaptive robot vision system | |
| CN111724356A (en) | Image processing method and system for CT image pneumonia identification | |
| CN113723833B (en) | Method, system, terminal equipment and storage medium for evaluating quality of forestation actual results | |
| CN116052099A (en) | Small target detection method for unstructured road | |
| Fukui et al. | Development of a tomato volume estimating robot that autonomously searches an appropriate measurement position–basic feasibility study using a tomato bed mock-up– | |
| CN116109047A (en) | Intelligent scheduling method based on three-dimensional intelligent detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20181116 Address after: Room 601-A06, Tiancheng Information Building, 88 Nantiancheng Road, Xiangcheng High-speed Railway New Town, Suzhou City, Jiangsu Province Applicant after: Suzhou Zhijia Technology Co., Ltd. Address before: 710000 Room 2301, 23rd Floor, Block B, Green Territory Building, No. 1 Jinye Road, Xi'an High-tech Zone, Shaanxi Province Applicant before: Xi'an Zhi Jia Technology Co., Ltd. |