CN115424240A - Ground obstacle detection method, system, medium, equipment and terminal - Google Patents
Ground obstacle detection method, system, medium, equipment and terminal Download PDFInfo
- Publication number
- CN115424240A CN115424240A CN202211001143.5A CN202211001143A CN115424240A CN 115424240 A CN115424240 A CN 115424240A CN 202211001143 A CN202211001143 A CN 202211001143A CN 115424240 A CN115424240 A CN 115424240A
- Authority
- CN
- China
- Prior art keywords
- image
- rgb
- contour
- depth
- map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 70
- 238000001914 filtration Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 37
- 238000007781 pre-processing Methods 0.000 claims abstract description 22
- 238000005516 engineering process Methods 0.000 claims abstract description 21
- 238000004364 calculation method Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 15
- 230000000694 effects Effects 0.000 claims description 14
- 235000002566 Capsicum Nutrition 0.000 claims description 4
- 239000006002 Pepper Substances 0.000 claims description 4
- 235000016761 Piper aduncum Nutrition 0.000 claims description 4
- 235000017804 Piper guineense Nutrition 0.000 claims description 4
- 235000008184 Piper nigrum Nutrition 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 230000000877 morphologic effect Effects 0.000 claims description 4
- 150000003839 salts Chemical class 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 230000007797 corrosion Effects 0.000 claims description 3
- 238000005260 corrosion Methods 0.000 claims description 3
- 230000014759 maintenance of location Effects 0.000 claims description 3
- 230000006740 morphological transformation Effects 0.000 claims description 3
- 244000203593 Piper nigrum Species 0.000 claims 1
- 238000003909 pattern recognition Methods 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 35
- 238000004519 manufacturing process Methods 0.000 description 8
- 230000004888 barrier function Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 238000013507 mapping Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 238000012549 training Methods 0.000 description 4
- 241000722363 Piper Species 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008092 positive effect Effects 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of computer vision and pattern recognition, and discloses a ground obstacle detection method, a system, a medium, equipment and a terminal, wherein a RGB-D depth camera is used for respectively acquiring a 2D RGB image and a 3D depth image; preprocessing the image by adopting a median filtering method, and then preprocessing the image subjected to median filtering by adopting a Gaussian filtering method to filter noise; and respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology, and realizing obstacle avoidance based on the depth map and the RGB map. The method utilizes the 3D information image for real-time processing, immediately judges whether the obstacle exists once the current image is updated, can send the instruction for controlling the AGV to move without updating the map, and is efficient and safe. The invention not only utilizes the 3D information to calculate, but also assists in adding the 2D RGB image information, and the two are combined, so that the final detection data is more stable, and the result is more reliable.
Description
Technical Field
The invention belongs to the technical field of computer vision and pattern recognition, and particularly relates to a ground obstacle detection method, a system, a medium, equipment and a terminal.
Background
At present, along with the improvement of the automation degree, the production mode of the traditional manufacturing industry is deeply changed, the automation of warehouse logistics is trended, and the production is transited from a single fixed mode to a flexible production line with stronger adaptability. In order to save cost and improve economic benefit, advanced production modes such as a flexible manufacturing system and an automatic warehousing system are widely applied, and the flexible manufacturing system and the automatic warehousing system take a high-intelligence system as a decision center and take a robot, an automatic stereoscopic warehouse, an unmanned carrying vehicle and the like as main equipment. The unmanned carrying equipment takes an Automatic Guided Vehicle (AGV) as a mark, integrates advanced technologies such as electronic computer science, image signal processing, automatic control and the like, and is key equipment of a modern logistics system and a flexible production organization system.
The AGV has the main guiding and navigation modes as follows: coordinate guidance AGV, electromagnetic guidance AGV, optical guidance AGV, laser infrared navigation AGV, vision navigation AGV. The visual obstacle avoidance is an important module in the visual navigation AGV, and the current common mode for visual obstacle avoidance is based on various filtering algorithms of depth images, removes image noise, detects obstacles, positions the obstacles, and then carries out obstacle avoidance operation. Although depth image filtering methods have advanced greatly in denoising in recent years, there are still some problems in this field: noise is generated for a plurality of reasons, and it is obviously unrealistic to completely remove the noise through an image technology, and the noise point can be misjudged as an obstacle; in order to find the nearest barrier, all area pixel points of the barrier need to be traversed, so that computing resources are consumed very much, and the computing efficiency is low.
Aiming at the problem that the conventional image filtering method cannot completely remove the depth image noise, the main solutions at present are as follows:
a. hardware scheme is promoted, and 3D camera with better imaging effect and low noise is adopted
The 3D depth camera is replaced to enable the depth picture to be clearer in imaging, the noise is less, and the adverse effect of image noise on subsequent judgment can be overcome to a certain extent.
b. Adding more filter layers to reject more noise
And filtering the obtained depth picture by more layers, and removing more noises by layer-by-layer superposition.
Aiming at the problem that calculation resources are consumed by traversing pixel points in all areas of an obstacle at present, the following main solutions are provided:
(1) Replacement of higher-level, high-performance CPU
By using the CPU with better performance, the calculation efficiency can be improved to a certain extent under the condition of equivalent calculation amount.
(2) Adding GPU, DSP and other devices in processing terminal
By using the GPU, the DSP and other devices, multi-core parallel computation such as the GPU can be used for greatly improving the computation efficiency.
(3) Obstacle detection using deep learning algorithm
The method can better identify the obstacle in the fixed scene, but depends on the training model excessively, and if the object not contained in the training picture appears, the obstacle cannot be identified. And the cost of the existing edge deep learning equipment is higher, and the execution efficiency is lower.
The prior patent CN201710151701.9 discloses a method for avoiding an obstacle by using a depth camera, which obtains coordinates of an obstacle on a multi-angle virtual plane when the multi-angle virtual plane contacts the obstacle; mapping the obstacle coordinates to a horizontal plane where the height of the camera is located to generate mapping coordinates; and replacing the original coordinate data on the coordinate on the horizontal plane where the height of the camera is positioned with the data of the mapping coordinate to generate a new obstacle avoidance map. When the multi-angle virtual plane contacts with the obstacle, acquiring obstacle coordinates on the multi-angle virtual plane, mapping the obstacle coordinates to a horizontal plane with the height of the camera to generate mapping coordinates, and finally generating an updated obstacle avoidance map. The method has large calculation amount, needs strong calculation engine support, cannot update the obstacle avoidance map in real time, and can cause the situation of obstacle avoidance failure in the non-updating stage.
The prior patent 201710233985.6 discloses a robot active obstacle avoidance method based on machine vision, which comprises the following steps: acquiring visual information, resolving the visual information, detecting obstacles and identifying the types of the obstacles. The prior patent also relates to a robot active obstacle avoidance device based on machine vision, which combines the detection, identification and obstacle avoidance of obstacles and adopts different obstacle avoidance strategies according to the difference of the obstacles. Particularly, the obstacle is divided into two types of pedestrians and non-pedestrians based on the obstacle recognition of the depth camera, the obstacle is actively avoided, the obstacle avoidance safety is guaranteed, and the intelligence and the interactivity of the robot are improved. The method is huge in calculation amount, excessively depends on the 3D point cloud information, the 3D point cloud information is not preprocessed, and the obstacle avoidance algorithm is invalid under the condition that the 3D point cloud information is inaccurate.
The prior patent CN110826512A discloses a method, device and computer readable storage medium for detecting obstacles on the ground, which uses image information obtained by a camera to process and divide the obstacles in the image into a first contour and a second contour, and processes and analyzes the height and distance of the obstacles in the image by grading the contour information. According to the method, depth information is combined with information such as a horizontal view field angle, an installation height, an installation angle, a vertical view field angle, effective pixel column numbers and effective pixel row numbers to calculate the outline information of each level to obtain polar coordinates of each level, and then various image processing algorithms are carried out. This method is obviously computationally intensive and has poor real-time performance.
The prior patent CN110526058A discloses a system for controlling the opening and closing of an elevator door by using a sensor to acquire RGB image and depth image information in an elevator hall and judging whether an obstacle exists in an elevator hall area. The author uses background structure modeling, foreground detection, background subtraction, and shape filtering for both RGB images and depth images. The trial scene is a relatively fixed elevator car scene. The invention takes RGB image information processing as a first processing module and depth image information processing as a second processing module, and only when the first and second processing modules judge yes at the same time, the obstacle is considered to exist, thereby controlling the opening and closing of the elevator door. However, the processing method of background structure modeling, foreground detection, background subtraction and shape filtering processing of the invention may obtain better results in a fixed scene (such as an elevator room), is not suitable for a general scene (such as an obstacle avoidance scene of an AGV trolley warehouse), and a large amount of calculation amount brought by operation affects the real-time performance of the system, and two images need to be modeled; only morphological filtering is carried out, image filtering is not carried out on original images (RGB images and depth images), and common noise is removed; the RGB image information is processed as a first processing module, the depth image information is processed as a second processing module, only when the first processing module and the second processing module judge to be yes at the same time, the obstacle is considered to exist, and the weighted value in the selectable items is used for obtaining the final judgment result.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) The noise cannot be completely removed through the existing image technology, and the noise point can be misjudged as an obstacle; in order to find the nearest barrier, all area pixel points of the barrier need to be traversed, so that computing resources are consumed, and computing efficiency is low.
(2) The existing obstacle detection method excessively depends on a training model, and if an object which is not contained in a training picture appears, an obstacle cannot be identified; and the cost of the edge deep learning equipment is high, and the execution efficiency is low.
(3) The existing obstacle detection method has large calculation amount, needs strong calculation engine support, cannot update the obstacle avoidance map in real time, and can cause the obstacle avoidance failure in the non-updating stage.
(4) The existing obstacle detection method is poor in real-time performance, excessively depends on 3D point cloud information, the 3D point cloud information is not preprocessed, and the algorithm fails when the 3D point cloud information is inaccurate.
Disclosure of Invention
The invention provides a ground obstacle detection method, a system, a medium, equipment and a terminal, and particularly relates to a ground obstacle detection method, a system, a medium, equipment and a terminal based on a depth camera.
The present invention is achieved in this way, a ground obstacle detection method including:
filtering the original image by adopting a mode of combining median filtering and Gaussian filtering to filter out image noise, and eliminating a minimum noise point by judging the area of an image connected domain; and scaling the 3D depth information image, and processing the 3D depth image and the 2D RGB image by utilizing various morphological transformation combinations in the image processing technology to realize the ground obstacle detection based on the depth image and the RGB image.
Further, the ground obstacle detection method includes the steps of:
step one, image acquisition: acquiring image information by using an RGB-D depth camera;
step two, image preprocessing: preprocessing the RGB and 3D images by adopting a median filtering method, and then preprocessing the median filtered images by adopting a Gaussian filtering method to filter noise;
step three, obstacle avoidance of the 3D depth map: and performing efficient image algorithm operation on the depth map foreground contour and the RGB map foreground contour based on the image processing technology to realize obstacle avoidance based on the depth map and the RGB map.
Further, the image information in the first step comprises a 2D RGB image and a 3D depth image; each pixel point in the 3D depth image is the distance from the camera to an object in the image, and the distance is depth information. The obstacle detection is to detect an object distance closest to the camera.
And the median filtering in the second step is to set the gray value of each pixel point as the median of the gray values of all the pixel points in a certain neighborhood window of the point.
Further, the constructing of the depth map foreground contour model in step three includes:
(1) Median filtering: carrying out median filtering on the depth map, and filtering noise points of images similar to salt and pepper;
(2) Gaussian filtering: performing Gaussian filtering on the median filtering result to filter Gaussian noise of the image;
(3) Binarization: carrying out binarization by adopting Otsu method, and setting the gray value of pixel points on the image to be 0 or 255, thereby realizing the process of presenting the whole image with obvious black and white effect;
(4) Opening operation: performing morphological operation opening operation on the obtained binary image, wherein the opening operation is corrosion operation and then expansion operation, isolated small points and burrs are removed, and the total position and shape are unchanged;
(5) Outputting foreground contour information: and outputting the contour information after the opening operation is finished.
Further, the construction of the RGB map foreground contour model in the third step includes:
(1) Graying: graying the RGB image to obtain a grayscale image;
(2) Binarization: carrying out binarization by adopting Otsu method, and setting the gray value of pixel points on the image to be 0 or 255, thereby realizing the process of presenting the whole image with obvious black and white effect;
(3) Outputting contour information: and after the binarization is finished, outputting the contour information.
Further, the obstacle avoidance based on the depth map and the RGB map in the third step includes:
(1) After 2 contour information images A and B are obtained by respectively utilizing a depth image foreground contour model and an RGB image foreground contour model, the image A and the image B are subjected to AND operation to obtain a common foreground contour region of the two images, and a contour m is obtained 1 ,m 2 …m n N contour regions;
(2) Setting a threshold value, traversing the areas of n contour regions, reserving the contour regions larger than the threshold value, deleting the contour regions smaller than the threshold value, and deleting the reserved contours n 1 ,n 2 …n n And sequentially calculating the nearest distance points in the contour.
Wherein the pair of retention profiles n 1 ,n 2 …n n Sequentially calculating the nearest distance points in the contour comprises the following steps:
1) Ordering all points on each profile;
2) Obtaining the minimum value and the maximum value of (x, y) coordinates on the contour as (xmin, ymin), (xmax, ymax);
3) For (xmin, ymin), (xmax, ymax) two point coordinates are equally divided into ten parts, and 11 point coordinates are obtained.
The method for calculating the tenths is as follows:
if (x 0, y 0) = (xmin, ymin), (x 10, y 10) = (xmax, ymax), then (x 1, y 1) = (xmin + (xmax-xmin)/10, xmin + (xmax-xmin)/10), 11 points calculation method and so on;
and acquiring 11 corresponding pixel values of the 11 points in the depth map according to the coordinates of the 11 points, and taking the minimum value as the actual distance from the camera to the nearest obstacle.
Another object of the present invention is to provide a ground obstacle detecting system to which the ground obstacle detecting method is applied, the ground obstacle detecting system including:
the image acquisition module is used for acquiring image information by utilizing the RGB-D depth camera;
the image preprocessing module is used for preprocessing the image by adopting a median filtering method, then preprocessing the image subjected to median filtering by adopting a Gaussian filtering method, and filtering noise;
and the 3D depth map obstacle avoidance module is used for respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology and realizing obstacle avoidance based on the depth map and the RGB map.
It is a further object of the invention to provide a computer arrangement comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of the ground obstacle detection method.
It is a further object of the present invention to provide a computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the ground obstacle detection method.
Another object of the present invention is to provide an information data processing terminal for implementing the ground obstacle detection system.
In combination with the technical solutions and the technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected in the present invention from the following aspects:
first, aiming at the technical problems and difficulties in solving the problems in the prior art, the technical problems to be solved by the technical scheme of the present invention are closely combined with results, data and the like in the research and development process, and some creative technical effects are brought after the problems are solved. The specific description is as follows:
the invention provides a method for realizing real-time obstacle avoidance of a 3D depth camera, which adopts an image processing technology to efficiently complete a calculation process. The invention provides a method for realizing accurate positioning of an obstacle, which realizes accurate positioning of the obstacle by adopting a method of combining 3D information and 2D information. The invention also provides a method for reducing the false detection rate of the obstacle, which adopts 2 filtering modes to filter the original image, can well filter two kinds of image noise, and can remove partial very small noise points by judging the area of the image connected domain.
The invention provides a new image processing algorithm, which avoids performing 3D modeling processing on two images. The invention properly zooms the 3D depth information image, efficiently processes the 3D depth image and the 2D RGB image by utilizing various ingenious and novel morphological transformation combinations in the image processing technology, basically does not process each pixel point independently, ensures that the utilization rate of the CPU of the whole algorithm is low and fast, and the algorithm can run and detect each frame of the image collected by the camera in real time.
According to the method, 2 filtering modes are adopted for filtering the 3D depth picture, firstly, median filtering is adopted, and through experiments, a proper filtering template is selected, so that punctiform noise similar to salt and pepper noise can be well removed; and Gaussian filtering is adopted, so that the most common Gaussian noise in the nature can be filtered. The invention further deeply combines the texture contour information of the two images, provides a new method for estimating the nearest distance according to the depth map by ten equal parts, and has better robustness and usability.
Secondly, considering the technical scheme as a whole or from the perspective of products, the technical effect and advantages of the technical scheme to be protected by the invention are specifically described as follows:
the method utilizes the 3D information image and the 2D RGB image for real-time processing, immediately judges whether the obstacle exists once the current image is updated, can send the instruction for controlling the AGV to move without updating the map, and is efficient and safe. The invention not only utilizes 3D information to calculate, but also adds 2D RGB image information in an auxiliary way, and the two are combined, so that the final detection data is more stable, and the result is more reliable. Meanwhile, the invention also carries out image preprocessing operation on the 3D information. The algorithm of the invention is simple and efficient, the size of the calculation graph can be adjusted according to different scene requirements, and the real-time performance in engineering practice can be well ensured. The invention can be operated on the edge machine only provided with the CPU in real time, and does not need to rely on expensive edge equipment such as DSP, NPU, GPU and the like.
Third, as inventive supplementary proof of the claims of the present invention, there are several important aspects as follows:
the expected income and commercial value after the technical scheme of the invention is converted are as follows: the method can conveniently and non-limitedly perfectly transplant and operate the side end platforms (linux and win), greatly reduces the cost problem of hardware in the field of AGV, and can perfectly operate in an actual industrial scene by using a low-power and high-efficiency real-time algorithm under the same detection effect. At present, most of AVG navigation trolleys can support huge expenses on software algorithms only by high-cost hardware (GPU, DSP, NPU and the like), the algorithm is simple and easy to calculate, the calculation can be efficiently completed on a CPU, and the cost of the whole vehicle is greatly reduced.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for detecting a ground obstacle according to an embodiment of the present invention;
FIG. 2 is a flowchart of a depth map foreground contour algorithm based on an image processing technique according to an embodiment of the present invention;
fig. 3 is a flowchart of an RGB image obstacle avoidance algorithm based on an image processing technology according to an embodiment of the present invention;
fig. 4 is a flowchart of an obstacle avoidance algorithm based on a depth map and an RGB map according to an embodiment of the present invention;
fig. 5 is a diagram of the final effect of the detection provided by the embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a method, a system, a medium, a device and a terminal for detecting a ground obstacle, which are described in detail below with reference to the accompanying drawings.
1. The embodiments are explained. This section is an explanatory embodiment expanding on the claims so as to fully understand how the present invention is embodied by those skilled in the art.
As shown in fig. 1, the ground obstacle detection method provided by the embodiment of the present invention includes the following steps:
s101, image acquisition: acquiring RGB images and 3D image information by using an RGB-D depth camera;
s102, image preprocessing: preprocessing the image by adopting a median filtering method, and then preprocessing the image subjected to median filtering by adopting a Gaussian filtering method to filter noise;
s103, obstacle avoidance of the 3D depth map: and respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology, and realizing obstacle avoidance based on the depth map and the RGB map.
As a preferred embodiment, the ground obstacle detection method provided in the embodiment of the present invention specifically includes:
1. image acquisition
A 2D RGB image and a 3D depth image are acquired separately with an RGB-D depth camera.
Each pixel point in the 3D depth image acquired by the RGB-D depth camera is the distance from the camera to an object in the image, namely depth information. The obstacle detection is to detect an object distance closest to the camera.
2. Median filtering
In order to effectively filter noise signals in the image, the embodiment of the invention adopts a median filtering method to carry out preprocessing operation on the image, so that the noise is filtered, and meanwhile, the edges of the signals are protected from being blurred. The median filtering method is a non-linear smoothing technique, and sets the gray value of each pixel point as the median of the gray values of all pixel points in a certain neighborhood window of the point.
3. Gaussian filtering
In order to effectively filter the most common gaussian noise signals in the image, the embodiment of the invention adopts a gaussian filtering method to perform preprocessing operation on the image after median filtering, so as to further filter the noise.
4. 3D depth map obstacle avoidance algorithm
4.1 introduction to depth map obstacle avoidance Algorithm
4.1.1 depth map basic concept
A Depth map is a two-dimensional picture in which each pixel records the distance from a viewpoint (viewpoint) to the surface of an obstruction (an obstruction is a shadow-generating object), and the vertices of these pixels are "visible" to the viewer.
4.1.2 basic idea of obstacle avoidance algorithm
The obstacle avoidance algorithm, as the name implies, is to avoid an obstacle, and what needs to be done first is to detect the position of the obstacle closest to the camera. For example, a cup held in the hand, i.e., the object closest to the camera (foreground object). And through image transformation of the depth map, the position of the cup in the map is positioned, and the pixel value of the cup is taken out, so that the position of the barrier away from the camera can be found, and the detection process is completed.
4.2 depth map foreground contour algorithm based on image processing technology
Fig. 2 shows a flow of a depth map foreground contour algorithm based on an image processing technology. The depth map foreground contour algorithm based on the image processing technology provided by the embodiment of the invention is described in detail as follows:
(1) Median filtering
And performing median filtering on the depth map to filter noise points of the image similar to salt and pepper.
(2) Gauss filtering
And performing Gaussian filtering on the median filtering result to filter Gaussian noise of the image.
(3) Binarization method
The binarization method is that the Otsu method is used for binarization, namely, the gray value of a pixel point on an image is set to be 0 or 255, namely, the whole image is subjected to a process of obvious black and white effect.
(4) Open operation
And performing morphological operation opening operation on the obtained binary image, wherein the opening operation is firstly corrosion operation and then expansion operation, so that isolated dots and burrs can be removed without changing the total position and shape.
(5) Outputting foreground contour information
And after the opening operation is finished, outputting the contour information.
4.3 RGB map foreground contour algorithm based on image processing technology
Fig. 3 is a flow of an RGB image obstacle avoidance algorithm based on an image processing technology according to an embodiment of the present invention.
The detailed steps are described as follows:
(1) Graying
And graying the RGB image to obtain a grayscale image.
(2) Binarization method
The binarization method is that the Otsu method is used for binarization, namely, the gray value of a pixel point on an image is set to be 0 or 255, namely, the whole image is subjected to a process of obvious black and white effect.
(3) Outputting profile information
And after the binarization is finished, outputting the contour information.
4.4 obstacle avoidance algorithm based on depth map and RGB map
Fig. 4 is a flow chart of an obstacle avoidance algorithm based on a depth map and an RGB map, and after 2 contour information maps a and B are obtained by using the 3 and 4 algorithms, detailed operation steps are as follows:
(1) And (5) carrying out AND operation on the graph A and the graph B to obtain a common foreground contour region of the two graphs and obtain a contour m 1 ,m 2 …m n N contour regions.
(2) Setting a threshold, traversing the areas of the n contour regions, reserving the contour regions larger than the threshold, and deleting the contour regions smaller than the threshold.
(3) For the retention profile n 1 ,n 2 …n n Sequentially calculating the nearest distance points in the contour in the following specific calculation mode:
1) Ordering all points on each contour;
2) Obtaining the minimum value and the maximum value of (x, y) coordinates on the contour as (xmin, ymin), (xmax, ymax);
3) For (xmin, ymin), (xmax, ymax) two points are equally divided into ten parts, and 11 point coordinates are obtained.
The ten-point calculation method is as follows:
let (x 0, y 0) = (xmin, ymin), (x 10, y 10) = (xmax, ymax),
then (x 1, y 1) = (xmin + (xmax-xmin)/10, xmin + (xmax-xmin)/10), 11 point calculation methods, and so on.
And acquiring 11 pixel values corresponding to the 11 points in the depth map according to the coordinates of the 11 points, wherein the minimum value is the actual distance of the closest obstacle to the camera.
Fig. 5 is a 3D depth map of the final effect of detecting an obstacle, the three contours that are finally retained are framed, and finally, the distance measurement of the nearest obstacle is completed by only calculating the nearest distance in the frame of the 3 contours, namely the depth information of the cup.
The ground obstacle detection system provided by the embodiment of the invention comprises:
the image acquisition module is used for acquiring image information by using the RGB-D depth camera;
the image preprocessing module is used for preprocessing the image by adopting a median filtering method, and then preprocessing the image subjected to median filtering by adopting a Gaussian filtering method to filter noise;
and the 3D depth map obstacle avoidance module is used for respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology and realizing obstacle avoidance based on the depth map and the RGB map.
2. Application examples. In order to prove the creativity and the technical value of the technical scheme of the invention, the part is an application example of the technical scheme of the claims to a specific product or related technology.
The obstacle avoidance algorithm is successfully applied to a factory scene, and is in charge of transporting the AGV of the goods, and the AGV runs for multiple days without faults. The AGV trolley emergency braking pause function is provided, due to the fact that the calculation force requirement is low, detection can be conducted on each frame in real time, and when an emergency situation occurs, such as an obstacle suddenly falling before the AGV, the obstacle can be detected in time.
3. Evidence of the relevant effects of the examples. The embodiment of the invention achieves some positive effects in the process of research and development or use, and has great advantages compared with the prior art, and the following contents are described by combining data, diagrams and the like in the test process.
According to the existing obstacle avoidance algorithm, the obstacle avoidance algorithm based on the deep learning target detection is to detect the obstacle through the 2D RGB image, and find the position of the obstacle in the depth map after detecting the obstacle. Before the actual algorithm is operated, a large number of pictures of different obstacle types are needed, and the pictures need to be labeled, according to the prior empirical data, each type needs 500 pictures, so that N types need 500 pictures, manual labeling is time-consuming, and the labeling of 500 pictures needs about 2-3 hours. And the obstacle avoidance algorithm can only identify the N obstacles through deep learning, and the limitation is very obvious. The present algorithm does not limit the kind of obstacles.
In terms of algorithm execution efficiency, the time complexity of the algorithm is not an order of magnitude with the time complexity of the 4 patents mentioned in the background, and the speed is obviously improved.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided on a carrier medium such as a disk, CD-or DVD-ROM, programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the embodiments of the present invention, and the scope of the present invention should not be limited thereto, and any modifications, equivalents and improvements made by those skilled in the art within the technical scope of the present invention as disclosed in the present invention should be covered by the scope of the present invention.
Claims (10)
1. A ground obstacle detection method, characterized by comprising:
filtering the original image by adopting a mode of combining median filtering and Gaussian filtering to filter image noise, and eliminating a very small noise point by judging the area of an image connected domain; and scaling the 3D depth information image, and processing the 3D depth image and the 2D RGB image by utilizing various morphological transformation combinations in the image processing technology to realize the ground obstacle detection based on the depth image and the RGB image.
2. The ground obstacle detection method according to claim 1, characterized by comprising the steps of:
step one, image acquisition: acquiring image information by using an RGB-D depth camera;
step two, image preprocessing: after the image is preprocessed by adopting a median filtering method, preprocessing the image subjected to median filtering by adopting a Gaussian filtering method, and filtering noise;
step three, obstacle avoidance is performed on the 3D depth map: and respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology, and realizing obstacle avoidance based on the depth map and the RGB map.
3. The ground obstacle detection method according to claim 2, wherein the image information in the first step includes a 2D RGB image and a 3D depth image; each pixel point in the 3D depth image is the distance from the camera to an object in the image, and the distance is depth information; detecting an obstacle as detecting the object distance closest to the camera;
and the median filtering in the second step is to set the gray value of each pixel point as the median of the gray values of all the pixel points in a certain neighborhood window of the point.
4. The ground obstacle detection method of claim 2, wherein the construction of the depth map foreground profile model in step three comprises:
(1) Median filtering: carrying out median filtering on the depth map, and filtering noise points of images similar to salt and pepper;
(2) Gaussian filtering: performing Gaussian filtering on the median filtering result to filter Gaussian noise of the image;
(3) Binarization: carrying out binarization by adopting an Otsu method, and setting the gray value of a pixel point on the image to be 0 or 255 so as to realize the process of showing an obvious black-white effect on the whole image;
(4) Opening operation: performing morphological operation opening operation on the obtained binary image, wherein the opening operation is corrosion operation and then expansion operation, and removing isolated small points and burrs without changing the total position and shape;
(5) Outputting foreground contour information: and after the opening operation is finished, outputting the contour information.
5. The ground obstacle detection method according to claim 2, wherein the construction of the RGB map foreground contour model in the third step includes:
(1) Graying: graying the RGB image to obtain a grayscale image;
(2) Binarization: carrying out binarization by adopting an Otsu method, and setting the gray value of a pixel point on the image to be 0 or 255 so as to realize the process of showing an obvious black-white effect on the whole image;
(3) Outputting contour information: and after the binarization is finished, outputting the contour information.
6. The method for detecting ground obstacle according to claim 2, wherein the step three of avoiding obstacles based on the depth map and the RGB map comprises:
(1) After 2 contour information graphs A and B are obtained by respectively utilizing a depth graph foreground contour model and an RGB graph foreground contour model, and operation is carried out on the graph A and the graph B to obtain a common foreground contour area of the two graphs and a contour m 1 ,m 2 …m n N contour regions;
(2) Setting a threshold value, traversing the areas of n contour regions, reserving the contour regions larger than the threshold value, deleting the contour regions smaller than the threshold value, and reserving the contour n 1 ,n 2 …n n Sequentially calculating the nearest distance points in the contour;
wherein the pair of retention profiles n 1 ,n 2 …n n Sequentially calculating the nearest distance points in the contour comprises the following steps:
1) Ordering all points on each contour;
2) Obtaining the minimum value and the maximum value of (x, y) coordinates on the contour as (xmin, ymin), (xmax, ymax);
3) Dividing two points (xmin, ymin), (xmax, ymax) into ten equal parts to obtain 11 point coordinates;
the method for calculating the scores comprises the following steps:
if (x 0, y 0) = (xmin, ymin), (x 10, y 10) = (xmax, ymax), then (x 1, y 1) = (xmin + (xmax-xmin)/10, xmin + (xmax-xmin)/10), 11 points calculation method and so on;
and acquiring 11 corresponding pixel values of the 11 points in the depth map according to the coordinates of the 11 points, and taking the minimum value as the actual distance from the camera to the nearest obstacle.
7. A ground obstacle detection system to which the ground obstacle detection method according to any one of claims 1 to 6 is applied, characterized by comprising:
the image acquisition module is used for acquiring image information by using the RGB-D depth camera;
the image preprocessing module is used for preprocessing the image by adopting a median filtering method, and then preprocessing the image subjected to median filtering by adopting a Gaussian filtering method to filter noise;
and the 3D depth map obstacle avoidance module is used for respectively constructing a depth map foreground contour model and an RGB map foreground contour model based on an image processing technology and realizing obstacle avoidance based on the depth map and the RGB map.
8. A computer arrangement, characterized in that the computer arrangement comprises a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of the ground obstacle detection method according to any one of claims 1-6.
9. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the ground obstacle detection method according to any one of claims 1 to 6.
10. An information data processing terminal characterized by being used to implement the ground obstacle detection system according to claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211001143.5A CN115424240A (en) | 2022-08-19 | 2022-08-19 | Ground obstacle detection method, system, medium, equipment and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211001143.5A CN115424240A (en) | 2022-08-19 | 2022-08-19 | Ground obstacle detection method, system, medium, equipment and terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115424240A true CN115424240A (en) | 2022-12-02 |
Family
ID=84198482
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211001143.5A Pending CN115424240A (en) | 2022-08-19 | 2022-08-19 | Ground obstacle detection method, system, medium, equipment and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115424240A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274265A (en) * | 2023-11-22 | 2023-12-22 | 山东高烽畜牧科技有限公司 | Method and system for pathological unhairing tracking of beef cattle in livestock breeding based on images |
| CN117496189A (en) * | 2024-01-02 | 2024-02-02 | 中国石油大学(华东) | Rectangular tray hole identification method and system based on depth camera |
-
2022
- 2022-08-19 CN CN202211001143.5A patent/CN115424240A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274265A (en) * | 2023-11-22 | 2023-12-22 | 山东高烽畜牧科技有限公司 | Method and system for pathological unhairing tracking of beef cattle in livestock breeding based on images |
| CN117274265B (en) * | 2023-11-22 | 2024-02-02 | 山东高烽畜牧科技有限公司 | Method and system for pathological unhairing tracking of beef cattle in livestock breeding based on images |
| CN117496189A (en) * | 2024-01-02 | 2024-02-02 | 中国石油大学(华东) | Rectangular tray hole identification method and system based on depth camera |
| CN117496189B (en) * | 2024-01-02 | 2024-03-22 | 中国石油大学(华东) | Rectangular tray hole identification method and system based on depth camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107507167B (en) | Cargo tray detection method and system based on point cloud plane contour matching | |
| Eveland et al. | Background modeling for segmentation of video-rate stereo sequences | |
| CN115424240A (en) | Ground obstacle detection method, system, medium, equipment and terminal | |
| Zhou et al. | Efficient road detection and tracking for unmanned aerial vehicle | |
| CN113370977B (en) | Intelligent vehicle forward collision early warning method and system based on vision | |
| EP2386998B1 (en) | A Two-Stage Correlation Method for Correspondence Search | |
| CN106780560B (en) | Bionic robot fish visual tracking method based on feature fusion particle filtering | |
| CN116503803A (en) | Obstacle detection method, obstacle detection device, electronic device and storage medium | |
| CN113282088A (en) | Unmanned driving method, device and equipment of engineering vehicle, storage medium and engineering vehicle | |
| CN113269838B (en) | Obstacle visual detection method based on FIRA platform | |
| Foresti | Object detection and tracking in time-varying and badly illuminated outdoor environments | |
| Głowacz et al. | Video detection algorithm using an optical flow calculation method | |
| Kanhere et al. | Real-time detection and tracking of vehicle base fronts for measuring traffic counts and speeds on highways | |
| CN111591284A (en) | Visual field blind area obstacle avoidance method and device based on monocular recognition | |
| Chun-Zhao et al. | Drivable road boundary detection for intelligent vehicles based on stereovision with plane-induced homography | |
| CN115035492B (en) | Vehicle identification method, device, equipment and storage medium | |
| Zhang et al. | Insulator profile detection of transmission line based on traditional edge detection algorithm | |
| CN115546764A (en) | Obstacle detection method, device, equipment and storage medium | |
| CN111915632B (en) | Machine learning-based method for constructing truth database of lean texture target object | |
| Guo et al. | Drivable road region detection based on homography estimation with road appearance and driving state models | |
| Li et al. | Depth image enhancement algorithm based on RGB image fusion | |
| Zhang et al. | Dynamic Semantics SLAM Based on Improved Mask R-CNN | |
| Thu et al. | An effective vision-based self-navigation system for autonomous indoor vehicle | |
| He et al. | Steering Angle Measurement of UAV Navigation Based on Improved Image Processing. | |
| CN114783172B (en) | Parking lot empty space recognition method and system and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |