CN115187946B - Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data - Google Patents
Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data Download PDFInfo
- Publication number
- CN115187946B CN115187946B CN202210796916.7A CN202210796916A CN115187946B CN 115187946 B CN115187946 B CN 115187946B CN 202210796916 A CN202210796916 A CN 202210796916A CN 115187946 B CN115187946 B CN 115187946B
- Authority
- CN
- China
- Prior art keywords
- obstacle
- detection
- monorail crane
- image
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000004927 fusion Effects 0.000 title claims description 37
- 238000001514 detection method Methods 0.000 claims abstract description 101
- 230000000007 visual effect Effects 0.000 claims abstract description 19
- 230000007246 mechanism Effects 0.000 claims abstract description 18
- 238000012549 training Methods 0.000 claims description 48
- 238000011897 real-time detection Methods 0.000 claims description 20
- 230000004888 barrier function Effects 0.000 claims description 9
- 238000011156 evaluation Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 230000001629 suppression Effects 0.000 claims description 3
- 238000005286 illumination Methods 0.000 abstract description 6
- 230000001965 increasing effect Effects 0.000 abstract description 6
- 239000003245 coal Substances 0.000 abstract description 4
- 239000011435 rock Substances 0.000 abstract description 4
- 238000000605 extraction Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003014 reinforcing effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Traffic Control Systems (AREA)
- Geophysics And Detection Of Objects (AREA)
Abstract
The invention relates to a multi-scale intelligent sensing method for fusing underground obstacle point cloud and image data, which is characterized in that aiming at an underground complex environment, a single sensor obtains limited information, and laser point cloud depth and height information and visual image RGB three-channel information are fused to form five-channel data characteristics; the underground roadway has a plurality of influencing factors such as roadway deformation, coal gangue drop, turnout faults, personnel flow, uneven illumination and the like, the saliency of the obstacle is influenced, so that the problem of difficult detection is caused, and the saliency in the image is enhanced by introducing a convolution attention mechanism on the basis of the original network YOLOv 5; the small targets such as falling coal gangue and falling rocks are difficult to detect due to the fact that the obstacles in the monorail crane track are small targets, and the detection precision of the model on the small targets can be improved by increasing the output scale of the model; through the scheme, the obstacle in the advancing track of the monorail crane can be rapidly and accurately detected, and unmanned autonomous obstacle avoidance of the monorail crane is realized.
Description
Technical Field
The invention belongs to the technical field of autonomous environment sensing of underground auxiliary transportation equipment of a coal mine, and particularly relates to a multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data.
Background
The auxiliary transportation of the coal mine is an important link of a mine production system, the monorail crane and the like are important equipment forms of underground auxiliary transportation, and unmanned development of the monorail crane is an important technical guarantee for realizing efficient continuous auxiliary transportation. Because the underground space is limited and the environment is complex, the autonomous obstacle avoidance of the monorail crane is an important unmanned premise. The underground unmanned monorail crane has the following problems in the aspect of intelligent perception of environment: 1) Multiple factors influencing the normal running of the monorail crane, such as roadway deformation, scattered ore sundries, turnout faults, personnel flow and the like, exist in the underground limited space, and the obstacle needs to be perceived in real time and whether the normal running is influenced or not is judged; 2) The illumination in the pit is uneven, light is insufficient, a small target obstacle exists, information acquired by a single sensor is limited, intelligent obstacle sensing of a monorail crane in the running process is difficult, and safety accidents are easy to cause.
At present, research on intelligent perception of underground obstacles is mainly focused on two directions of visual images and three-dimensional laser radars. Planar information, namely two-dimensional pixel points, in front of the track running of the monorail crane can be captured through vehicle-mounted visual camera shooting, and point cloud images formed by the coordinate points, namely height information and depth information of a model in front of the track running of the monorail crane, can be captured through vehicle-mounted laser radar equipment. In the face of a complex underground environment, a single laser radar device is greatly interfered by environmental factors when acquiring environmental information, so that a modeling effect is poor, and intelligent perception of obstacles is affected; the underground illumination is uneven, the light is insufficient, and the problems that the characteristics of a visual image are incomplete, not obvious and the like exist when the characteristics in front of a track are acquired; the traditional image processing technology has complex flow, can not classify the obstacle, has a plurality of defects, and a two-stage target detection network represented by Fast R-CNN can not meet the requirement of real-time detection of the obstacle in the running process of the monorail crane. Therefore, in view of the above problems, there is a need for an intelligent obstacle sensing method that can adapt to a complex transportation environment downhole and detect quickly.
Disclosure of Invention
The invention provides a multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data, which aims to solve the problem of intelligent sensing of an unmanned monorail crane in environment. Aiming at a complex underground environment, the information acquired by a single sensor is limited, and the forward obstacle characteristic extraction of the monorail crane is carried out by fusing laser point cloud and visual image data characteristics; the underground roadway has a plurality of influencing factors such as roadway deformation, turnout faults, personnel flowing, uneven illumination and the like, the saliency of the obstacle is influenced, so that the problem of difficult detection is solved, and the saliency in the image is enhanced by introducing a convolution attention mechanism on the basis of the original network YOLOv 5; the small targets such as falling rocks and the like are difficult to detect due to the fact that the obstacles in the monorail crane track are difficult to detect, and the detection precision of the model on the small targets can be improved by increasing the output scale of the model; through the scheme, the obstacle detection device can be used for rapidly detecting the obstacle in the forward track of the monorail crane, and has a good detection effect on the small target obstacle in the track.
In order to achieve the above purpose, the specific technical scheme adopted by the invention is as follows:
the multi-scale intelligent sensing method for fusing underground obstacle point cloud and image data is used for detecting obstacles in a forward track of a monorail crane in real time, and the sensing method comprises the following steps:
step S1: constructing a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S1-1: introducing a convolution attention mechanism;
step S1-2: adding a small obstacle target detection layer;
step S1-3: adding a priori frame detection anchor points;
step S2: establishing an obstacle detection data set;
step S2-1: acquiring road scene information of the track advancing direction by utilizing a laser radar and a vision module carried by the monorail crane;
step S2-2: fusing laser radar point cloud information and visual image information;
step S2-3: dividing the fusion data set;
step S3: training a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S3-1: setting training parameters;
step S3-2: setting a network training self-adaptive scaling image, namely a five-channel obstacle fusion image;
step S3-3: training an obstacle detection model;
step S4: evaluating a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S4-1: performing performance evaluation and evaluation on the model trained in the step S3-3, inputting the test set divided in the step S2-3 into the detection model trained in the step S3-3, and evaluating the model from single target detection precision, small target detection precision, average detection precision and detection speed evaluation indexes;
step S4-2: judging whether the detection precision and the detection speed of the detection model trained in the step S3-3 meet the real-time detection requirement of the obstacle of the monorail crane, namely the actual working condition requirement, if the detection precision and the detection speed of the detection model trained in the step S3-3 meet the actual working condition requirement of the obstacle detection of the monorail crane, executing the step 6, and if the detection precision and the detection speed of the detection model trained in the step S3-3 can not meet the actual working condition requirement of the obstacle detection of the monorail crane, executing the step 5;
step S5: correcting a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm, adjusting the width and depth parameters of a network model, and turning to the step 3 to retrain the monorail crane obstacle real-time detection model based on the improved YOLOv5 algorithm;
step S6: the training model is used for unmanned autonomous obstacle avoidance of the monorail crane, the forward direction track scene information is obtained through vehicle-mounted vision and laser radar in the running process of the monorail crane, the five-channel fusion image is obtained by fusing the visual RGB three-channel image with the laser radar depth image and the altitude image obtained through preprocessing, and the five-channel fusion image is input into the training detection model, so that the type, the position and the distance information of the obstacle in the current track scene information can be obtained in real time, and unmanned autonomous obstacle avoidance of the monorail crane is realized.
Preferably, a convolution attention mechanism is introduced in step S1-1, a channel attention mechanism and a space attention mechanism which are connected in parallel are added after the original YOLOv5 backbone network, the output of the backbone network respectively enters a channel attention module and a space attention module, the feature images respectively output by the two modules are added, fusion operation is carried out with the output feature images of the backbone network, the feature images after the fusion operation are input to a neg layer of the YOLOv5 network, and the convolution attention mechanism is used for enhancing the extraction of barrier features of the backbone network and further improving the significance of the extraction of the barrier features.
Preferably, in the step S1-2, a small target detection layer is added, and a small target detection layer is added on the basis of the three-scale layer output by the original network, so that four-scale prediction of obstacle detection is realized.
Preferably, a priori frame detection anchor point is added in the step S1-3, and an anchor point frame which accords with the target scale characteristics of the obstacle is obtained by adopting a K-means++ self-adaptive algorithm; according to the small target detection layer added in step S1-2, the corresponding small scale anchor is required to be added to the divided small scale grid.
Preferably, in the step S2-1, the visual image and the laser radar sensor data acquisition module of the existing calibrated underground monorail crane are utilized to acquire the road scene information in the forward track of the monorail crane; and processing the three-dimensional point cloud information acquired by the vehicle-mounted laser radar by using the designed algorithm to acquire a depth map capable of reflecting the depth information of the road scene in the forward track of the monorail crane and a height map of the height information.
Preferably, in step S2-2, image information is fused with the depth map and the height map of the processed laser radar through visual image RGB three-channel image information acquired at the same time to form a five-channel fusion training data set, a mosaic algorithm is adopted to realize data enhancement of an obstacle fusion image, and four images are randomly scaled, randomly cut and randomly arranged to be spliced, so that the background of obstacle target detection is enriched.
Preferably, step S3 sets training parameters, performs optimization training by using a random optimization algorithm Adam, sets batch size batch for training to 32, momentum parameter momentum to 0.9, learning rate to 0.001, training iteration number epoch to 1000, loss function adopts GIoU, prediction frame screening method, performs optimization by using a non-maximum suppression NMS method, and selects an optimal prediction frame.
Preferably, in step S3, the size of the network training adaptive scaling image is set to 608x608x5, adaptive scaling of the images of the adaptive obstacle fusion image detection training set and the verification set is realized according to the size of the input size set by the network, the model learning rate and the iteration times are adjusted according to the obstacle fusion data set divided in step S2-3 and the training parameters and the training model set in step S3-1, and the model learning rate and the iteration times are adjusted according to the change trend of the loss function until the change of the loss function tends to be stable, thereby determining the final training model.
The invention has the beneficial effects that:
1. according to the invention, by fusing the laser point cloud and the visual image data characteristics, the barrier characteristics in the forward track of the monorail crane are extracted, the five-channel data type characteristics are formed, the problem that the information acquired by a single sensor is limited due to a complex underground environment is solved, the accurate barrier limit and characteristics can be acquired, and the advantages of the laser point cloud and the visual are fully fused.
2. The invention can effectively overcome the problems that the significance of the barrier is influenced and difficult to detect due to the fact that a plurality of influencing factors such as roadway deformation, turnout faults, personnel flow, uneven illumination and the like exist in underground roadways by introducing a convolution attention mechanism to enhance the significance of the image on the basis of the original network YOLOv 5.
3. According to the invention, the detection precision of the model to the small targets can be improved by increasing the output scale of the model, the problem that the detection difficulty is increased by the small targets such as falling rocks and the like existing in the obstacles in the track of the monorail crane can be effectively solved, and the multi-target and multi-size real-time detection of the obstacles in the forward track of the monorail crane can be realized.
Drawings
For a clearer description of an embodiment of the invention or of the solutions of the prior art, reference will be made to the accompanying drawings, which are used in the embodiments and which are intended to illustrate, but not to limit the invention in any way, the features and advantages of which can be obtained according to these drawings without inventive labour for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of monorail crane obstacle target detection;
FIG. 2 is a block diagram of a modified YOLOv5 algorithm.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings of the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Unless defined or otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
In one embodiment, the invention provides a multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data, which aims at underground complex environments, has limited information acquired by a single sensor, and performs single-track crane forward-direction in-track obstacle feature extraction by fusing laser point clouds and visual image data features; the underground roadway has a plurality of influencing factors such as roadway deformation, turnout faults, personnel flowing, uneven illumination and the like, the saliency of the obstacle is influenced, so that the problem of difficult detection is solved, and the saliency in the image is enhanced by introducing a convolution attention mechanism on the basis of the original network YOLOv 5; the small targets such as falling rocks and the like are difficult to detect due to the fact that the obstacles in the monorail crane track are difficult to detect, and the detection precision of the model on the small targets can be improved by increasing the output scale of the model; through the scheme, the obstacle detection device can be used for rapidly detecting the obstacle in the forward track of the monorail crane, and has a good detection effect on the small target obstacle in the track.
As shown in fig. 1, the multi-scale intelligent sensing method for fusing underground obstacle point cloud and image data in this embodiment specifically includes:
s1, constructing a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm
In general, the object detection network structure mainly comprises four parts, namely Input, a backbone network Backbone, neck, a prediction layer, namely Head, as shown in fig. 2, an improved YOLOv5 algorithm structure diagram, wherein an Input end Input obstacle fusion image is 608x608x5, and the Input obstacle fusion image is Input into the backbone network through data enhancement, self-adaptive anchor frame calculation and self-adaptive image scaling; the backbone network mainly comprises a Focus structure and a CSP structure, the Focus structure is used for slicing an image and then connecting the image as shown in the figure, the CSP module divides the feature map of the base layer into two parts, and then the feature map of the base layer is combined through a cross-stage hierarchical structure, so that the calculation amount is reduced, and meanwhile, the accuracy can be ensured; the Neck is mainly composed of FPN and PAN structures; and adding a small target detection layer on the basis of the original three-scale prediction by the Head to form four-scale prediction.
Step S1-1: introducing a convolution attention mechanism, adding a channel attention mechanism and a space attention mechanism which are connected in parallel after an original YOLOv5 backbone network, enabling the output of the backbone network to enter a channel attention module and a space attention module respectively, adding the characteristic diagrams respectively output by the two modules, carrying out fusion operation with the output characteristic diagrams of the backbone network, inputting the characteristic diagrams after the fusion operation into a neg layer of the YOLOv5 network, wherein the convolution attention mechanism is used for reinforcing the extraction of barrier characteristics of the backbone network and further improving the significance of the extraction of the barrier characteristics;
step S1-2: adding a small obstacle target detection layer, adding a small obstacle target detection layer on the basis of the original network output three-scale layer, realizing four-scale prediction of obstacle detection, taking 608x608x5 as an input obstacle fusion image, respectively passing through 4 times, 8 times, 16 times and 32 times of up-sampling characteristic layers, and adding four characteristic scales obtained after adding one detection layer as follows: the 152x152 scale feature map, the 76x76 scale feature map, the 38x38 scale feature map and the 19x19 scale feature map are used as an increased obstacle small target detection layer, and the 152x152 scale feature map is used for realizing multi-scale detection of the falling stone image, so that the detection precision of the improved network is further improved.
Step S1-3: adding a priori frame detection anchor point, and acquiring an anchor point frame conforming to the target scale characteristics of the obstacle by adopting a K-means++ self-adaptive algorithm; according to the small target detection layer, i.e. the 152x152 scale feature map, in step S1-2, the small scale grids are divided to be added with corresponding small scale anchors, so that the prior frames anchors of the improved network model are added into 12 groups of frames with corresponding 4 detection scales.
Step S2, establishing an obstacle detection data set
Step S2-1: the method comprises the steps of obtaining road scene information in a forward track of a monorail crane by using a visual image and a laser radar sensor data acquisition module of the existing calibrated underground monorail crane; processing three-dimensional point cloud information acquired by a vehicle-mounted laser radar by using a designed algorithm to acquire a depth map capable of reflecting road scene depth information in a forward track of a monorail crane and a height map of the height information;
step S2-2: image information is fused with the depth map and the height map of the laser radar after processing through visual image RGB three-channel image information acquired at the same time to form a five-channel fusion training data set, a mosaic algorithm is adopted to realize data enhancement of obstacle fusion images, and four images are spliced in a random zooming, random clipping and random arrangement mode, so that the background of obstacle target detection is enriched;
step S2-3: dividing the fusion data set, and establishing a monorail crane obstacle detection training set, a verification set and a test set, wherein the monorail crane obstacle detection training set, the verification set and the test set are as follows: 2:2, dividing an obstacle data set in proportion, namely randomly selecting 60% of obstacle fusion images as a training set, randomly selecting 20% of obstacle fusion images as a verification set, randomly selecting 20% of obstacle fusion images as a test set, and setting the category as the name of the obstacle according to the requirement of a detection target;
step S3: monorail crane obstacle real-time detection model based on improved YOLOv5 algorithm
Step S3-1: setting training parameters, performing optimization training by adopting a random optimization algorithm Adam, setting the batch size batch for training to be 32, setting the momentum parameter momentum to be 0.9, setting the learning rate to be 0.001, setting the training iteration times epoch to be 1000, optimizing a loss function by adopting a GIoU, adopting a prediction frame screening method, and adopting a non-maximum suppression NMS method to select an optimal prediction frame;
step S3-2: setting the size of a network training self-adaptive scaling image to 608x608, and realizing self-adaptive scaling of images of the self-adaptive obstacle fusion image detection training set and the verification set according to the size of the input size set by the network;
training an obstacle detection model, training the model according to the obstacle fusion data set divided in the step S2-3 and the training parameters set in the step S3-1, and adjusting the model learning rate and the iteration times according to the change trend of the loss function until the change of the loss function tends to be stable, thereby determining a final training model;
step S4: monorail crane obstacle real-time detection model for evaluating based on improved YOLOv5 algorithm
Step S4-1: performing performance evaluation and evaluation on the model trained in the step S3-3, inputting the test set divided in the step S2-3 into the model trained in the step S3-3, and evaluating the model from evaluation indexes such as single target detection precision, small target detection precision, average detection precision, detection speed and the like;
step S4-2: judging whether the detection precision and the detection speed of the model trained in the step S3-3 meet the real-time detection requirement, namely the actual working condition requirement, of the monorail crane obstacle, if the detection precision and the detection speed of the model trained in the step S3-3 meet the actual working condition requirement of the monorail crane obstacle detection, executing the step 6, and if the detection precision and the detection speed of the model trained in the step S3-3 can not meet the actual working condition requirement of the monorail crane obstacle detection, executing the step 5;
step S5: correcting a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm, adjusting parameters such as width, depth and the like of a network model, and turning to the step 3 to retrain the monorail crane obstacle real-time detection model based on the improved YOLOv5 algorithm;
step S6: the training model is used for unmanned autonomous obstacle avoidance of the monorail crane, the forward direction track scene information obtained through vehicle-mounted vision and laser radar in the running process of the monorail crane is used for fusing the visual RGB three-channel image with the laser radar depth image and the altitude image obtained through preprocessing to obtain a five-channel fusion image, and the five-channel fusion image is input into the training model, so that the information of the type, the position, the distance and the like of the obstacle in the current track scene information can be obtained in real time, and unmanned autonomous obstacle avoidance of the monorail crane is realized.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 1 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur sequentially, but may be performed alternately or alternately with at least a portion of other steps or other steps.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present disclosure describes embodiments, not every embodiment is provided with a separate embodiment, and that this description is provided for clarity only, and that the disclosure is not limited to the embodiments described in detail below, and that the embodiments described in the examples may be combined as appropriate to form other embodiments that will be apparent to those skilled in the art.
Claims (8)
1. The multi-scale intelligent sensing method for fusing the underground obstacle point cloud and the image data is characterized by comprising the following steps of:
step S1: constructing a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S1-1: introducing a convolution attention mechanism;
step S1-2: adding a small obstacle target detection layer;
step S1-3: adding a priori frame detection anchor points;
step S2: establishing an obstacle detection data set;
step S2-1: acquiring road scene information of the track advancing direction by utilizing a laser radar and a vision module carried by the monorail crane;
step S2-2: fusing laser radar point cloud information and visual image information;
step S2-3: dividing the fusion data set;
step S3: training a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S3-1: setting training parameters;
step S3-2: setting a network training self-adaptive scaling image, namely a five-channel obstacle fusion image;
step S3-3: training an obstacle detection model;
step S4: evaluating a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm;
step S4-1: performing performance evaluation and evaluation on the model trained in the step S3-3, inputting the test set divided in the step S2-3 into the detection model trained in the step S3-3, and evaluating the model from single target detection precision, small target detection precision, average detection precision and detection speed evaluation indexes;
step S4-2: judging whether the detection precision and the detection speed of the detection model trained in the step S3-3 meet the real-time detection requirement of the obstacle of the monorail crane, namely the actual working condition requirement, if the detection precision and the detection speed of the detection model trained in the step S3-3 meet the actual working condition requirement of the obstacle detection of the monorail crane, executing the step 6, and if the detection precision and the detection speed of the detection model trained in the step S3-3 can not meet the actual working condition requirement of the obstacle detection of the monorail crane, executing the step 5;
step S5: correcting a monorail crane obstacle real-time detection model based on an improved YOLOv5 algorithm, adjusting the width and depth parameters of a network model, and turning to the step 3 to retrain the monorail crane obstacle real-time detection model based on the improved YOLOv5 algorithm;
step S6: the training model is used for unmanned autonomous obstacle avoidance of the monorail crane, the forward direction track scene information is obtained through vehicle-mounted vision and laser radar in the running process of the monorail crane, the five-channel fusion image is obtained by fusing the visual RGB three-channel image with the laser radar depth image and the altitude image obtained through preprocessing, and the five-channel fusion image is input into the training detection model, so that the type, the position and the distance information of the obstacle in the current track scene information can be obtained in real time, and unmanned autonomous obstacle avoidance of the monorail crane is realized.
2. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: and introducing a convolution attention mechanism in the step S1-1, adding a channel attention mechanism and a space attention mechanism which are connected in parallel after the original YOLOv5 backbone network, enabling the output of the backbone network to enter a channel attention module and a space attention module respectively, adding the characteristic graphs output by the two modules respectively, carrying out fusion operation with the output characteristic graphs of the backbone network, inputting the characteristic graphs after the fusion operation into a neg layer of the YOLOv5 network, and using the convolution attention mechanism to strengthen the backbone network to extract barrier characteristics and further improve the significance of extracting the barrier characteristics.
3. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: and step S1-2, adding a small target detection layer of the obstacle, and adding a small target detection layer on the basis of outputting a three-scale layer by the original network to realize four-scale prediction of obstacle detection.
4. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: s1-3, adding a priori frame detection anchor point, and acquiring an anchor point frame conforming to the target scale characteristics of the obstacle by adopting a K-means++ self-adaptive algorithm; according to the small target detection layer added in step S1-2, the corresponding small scale anchor is required to be added to the divided small scale grid.
5. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: s2-1, acquiring road scene information in a forward track of a monorail crane by using a visual image and a laser radar sensor data acquisition module of the existing calibrated underground monorail crane; and processing the three-dimensional point cloud information acquired by the vehicle-mounted laser radar by using the designed algorithm to acquire a depth map capable of reflecting the depth information of the road scene in the forward track of the monorail crane and a height map of the height information.
6. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: in the step S2-2, image information is fused with the depth map and the height map of the processed laser radar through visual image RGB three-channel image information acquired at the same time to form a five-channel fusion training data set, a mosaic algorithm is adopted to realize data enhancement of obstacle fusion images, and four images are randomly scaled, randomly cut and randomly arranged to be spliced, so that the background of obstacle target detection is enriched.
7. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: step S3, setting training parameters, performing optimization training by adopting a random optimization algorithm Adam, setting the batch size of training as 32, setting the momentum parameter momentum as 0.9, setting the learning rate as 0.001, setting the training iteration number epoch as 1000, optimizing a loss function by adopting a GIoU, adopting a prediction frame screening method, adopting a non-maximum suppression NMS method, and selecting an optimal prediction frame.
8. The multi-scale intelligent sensing method for fusing underground obstacle point clouds and image data according to claim 1, wherein the method comprises the following steps: in the step S3, the size of a network training self-adaptive scaling image is set to 608x608x5, the self-adaptive scaling of the images of the self-adaptive obstacle fusion image detection training set and the verification set is realized according to the size of the input size set by the network, the model learning rate and the iteration times are adjusted according to the obstacle fusion data set divided in the step S2-3 and the training parameters and the training model set in the step S3-1 and the change trend of the loss function until the change of the loss function tends to a stable state, and the final training model is determined.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210796916.7A CN115187946B (en) | 2022-07-06 | 2022-07-06 | Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210796916.7A CN115187946B (en) | 2022-07-06 | 2022-07-06 | Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115187946A CN115187946A (en) | 2022-10-14 |
| CN115187946B true CN115187946B (en) | 2023-06-27 |
Family
ID=83517217
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210796916.7A Active CN115187946B (en) | 2022-07-06 | 2022-07-06 | Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115187946B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115620263B (en) * | 2022-10-25 | 2023-08-15 | 四川吉利学院 | Intelligent vehicle obstacle detection method based on image fusion of camera and laser radar |
| CN115620153A (en) * | 2022-12-16 | 2023-01-17 | 成都理工大学 | Method and device for grading surface acoustic wave mill of track |
| CN116796210B (en) * | 2023-08-25 | 2023-11-28 | 山东莱恩光电科技股份有限公司 | Barrier detection method based on laser radar |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| KR20210026412A (en) * | 2019-08-30 | 2021-03-10 | 순천향대학교 산학협력단 | Method for Real-time Object Detection Based on Lidar Sensor and Camera Using CNN |
| US10955855B1 (en) * | 2019-11-23 | 2021-03-23 | Ha Q Tran | Smart vehicle |
| CN113486726A (en) * | 2021-06-10 | 2021-10-08 | 广西大学 | Rail transit obstacle detection method based on improved convolutional neural network |
| WO2022108745A1 (en) * | 2020-11-23 | 2022-05-27 | Argo AI, LLC | Systems and methods for object detection with lidar decorrelation |
| CN114693909A (en) * | 2022-03-31 | 2022-07-01 | 苏州蓝图智慧城市科技有限公司 | Microcosmic vehicle track sensing equipment based on multi-sensor machine vision fusion |
-
2022
- 2022-07-06 CN CN202210796916.7A patent/CN115187946B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20210026412A (en) * | 2019-08-30 | 2021-03-10 | 순천향대학교 산학협력단 | Method for Real-time Object Detection Based on Lidar Sensor and Camera Using CNN |
| US10955855B1 (en) * | 2019-11-23 | 2021-03-23 | Ha Q Tran | Smart vehicle |
| CN111928862A (en) * | 2020-08-10 | 2020-11-13 | 廊坊和易生活网络科技股份有限公司 | Method for constructing semantic map on line by fusing laser radar and visual sensor |
| WO2022108745A1 (en) * | 2020-11-23 | 2022-05-27 | Argo AI, LLC | Systems and methods for object detection with lidar decorrelation |
| CN113486726A (en) * | 2021-06-10 | 2021-10-08 | 广西大学 | Rail transit obstacle detection method based on improved convolutional neural network |
| CN114693909A (en) * | 2022-03-31 | 2022-07-01 | 苏州蓝图智慧城市科技有限公司 | Microcosmic vehicle track sensing equipment based on multi-sensor machine vision fusion |
Non-Patent Citations (5)
| Title |
|---|
| Detection of Farmland Obstacles Based on an IMproved YOLOv5s Algorithm by Using CIou and Anchor Box Scale Clustering;Jinlin Xue等;Sensor;全文 * |
| ObjectDetect：A Real-Time Object Detection Framework for Advanced Driver Assistant Systems Using YOLOv5;Jamuna S.Murthy等;Machine Learning for Energy Efficient Wireless Communications and Mobile Computing;全文 * |
| 基于图像与点云的三维障碍物检测;张耀威等;计算机工程与设计(04);全文 * |
| 基于毫米波雷达与视觉融合的无人驾驶目标检测;谢正宇;中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑(第05期);全文 * |
| 煤矿井下视频运动目标的检测与识别研究;徐凤志;中国优秀硕士学位论文全文数据库 工程科技Ⅰ辑(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115187946A (en) | 2022-10-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115187946B (en) | Multi-scale intelligent sensing method for fusion of underground obstacle point cloud and image data | |
| CN102270301B (en) | Method for detecting unstructured road boundary by combining support vector machine (SVM) and laser radar | |
| EP3171292B1 (en) | Driving lane data processing method, device, storage medium and apparatus | |
| CN110738121A (en) | front vehicle detection method and detection system | |
| CN114782626B (en) | Transformer substation scene map building and positioning optimization method based on laser and vision fusion | |
| CN117274749B (en) | Fused 3D target detection method based on 4D millimeter wave radar and image | |
| CN115187964A (en) | Automatic driving decision-making method based on multi-sensor data fusion and SoC chip | |
| CN112818935B (en) | Multi-lane congestion detection and duration prediction method and system based on deep learning | |
| CN103206957A (en) | Detecting and tracking method for lane lines of autonomous vehicle navigation | |
| Filatov et al. | Any motion detector: Learning class-agnostic scene dynamics from a sequence of lidar point clouds | |
| CN106529404A (en) | Imaging principle-based recognition method for pilotless automobile to recognize road marker line | |
| CN103700106A (en) | Distributed-camera-based multi-view moving object counting and positioning method | |
| CN105405297A (en) | Traffic accident automatic detection method based on monitoring video | |
| CN103927523B (en) | Fog level detection method based on longitudinal gray features | |
| CN112149471A (en) | Loopback detection method and device based on semantic point cloud | |
| CN118053322A (en) | Engineering vehicle collision detection early warning system and method based on computer vision | |
| CN116343513B (en) | Rural highway beyond-sight-distance risk point safety monitoring and early warning method and system thereof | |
| CN116935304A (en) | Self-adaptive detection and tracking method based on crowd concentration | |
| Yang et al. | A novel vision-based framework for real-time lane detection and tracking | |
| CN104268889A (en) | Vehicle tracking method based on automatic characteristic weight correction | |
| Xie et al. | Circular Accessible Depth: A Robust Traversability Representation for UGV Navigation | |
| Zhang et al. | End-to-end BEV perception via Homography matrix | |
| CN116580369B (en) | Lane end-to-end real-time detection method for automatic driving | |
| Zhu et al. | Real-time vehicle queue estimation of large-scale traffic scene | |
| Wang et al. | Multi-Scale Feature Extraction and Monocular Depth Estimation Algorithm for Traffic Images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |