CN114445572A - Deeplab V3+ based method for instantly positioning obstacles and constructing map in unfamiliar sea area - Google Patents
Deeplab V3+ based method for instantly positioning obstacles and constructing map in unfamiliar sea area Download PDFInfo
- Publication number
- CN114445572A CN114445572A CN202111642668.2A CN202111642668A CN114445572A CN 114445572 A CN114445572 A CN 114445572A CN 202111642668 A CN202111642668 A CN 202111642668A CN 114445572 A CN114445572 A CN 114445572A
- Authority
- CN
- China
- Prior art keywords
- coordinate system
- visible light
- obstacle
- laser radar
- laser
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000006243 chemical reaction Methods 0.000 claims abstract description 37
- 230000011218 segmentation Effects 0.000 claims abstract description 30
- 238000000605 extraction Methods 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 57
- 230000004888 barrier function Effects 0.000 claims description 26
- 238000013507 mapping Methods 0.000 claims description 13
- 239000011159 matrix material Substances 0.000 claims description 10
- 238000011176 pooling Methods 0.000 claims description 10
- 238000013519 translation Methods 0.000 claims description 9
- 101100295091 Arabidopsis thaliana NUDT14 gene Proteins 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 8
- 230000000877 morphologic effect Effects 0.000 claims description 8
- 230000001133 acceleration Effects 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 5
- 239000011800 void material Substances 0.000 claims description 5
- 239000003086 colorant Substances 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000013508 migration Methods 0.000 claims description 4
- 230000005012 migration Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 abstract description 4
- 238000004519 manufacturing process Methods 0.000 abstract description 2
- 230000007547 defect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Remote Sensing (AREA)
- Computer Graphics (AREA)
- Image Processing (AREA)
- Optical Radar Systems And Details Thereof (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses a method for instantly positioning an obstacle and constructing a map in an unfamiliar sea area based on DeeplabV3+, which comprises the steps of firstly projecting laser point clouds onto an image based on a conversion relation between a laser radar coordinate system and a visible light camera coordinate system to obtain laser point cloud projection coordinates in the image coordinate system, then carrying out segmentation prediction on the image based on a Deeplabv3+ semantic segmentation network model to obtain segmented class images, manufacturing an obstacle mask by using obstacle classes to obtain laser point clouds corresponding to the obstacle, and finally constructing a surrounding obstacle map with an unmanned ship as a map center according to the conversion relation between the laser radar coordinate system and an IMU coordinate system, coordinates of the laser point clouds corresponding to the obstacle in the laser radar coordinate system and unmanned ship posture information obtained by the IMU. According to the invention, the accurate extraction of the outline information of the predicted and unpredictable obstacles is realized by utilizing the Deeplabv3+ semantic segmentation network model, and the accurate detection and positioning of the marine predicted and unpredictable obstacles are realized by combining the visible light camera and the laser radar.
Description
Technical Field
The invention belongs to the technical field of unmanned boats, and particularly relates to a method for instantly positioning obstacles and constructing a map in a strange sea area based on Deeplab V3 +.
Background
Unmanned ship obtains more and more extensive application in recent years, for example in the military field, unmanned ship can be widely used in fields such as naval vessel protection, anti-submarine operation, logistics supply, information investigation, and in civilian field, unmanned ship can be widely used in fields such as marine environment monitoring, seabed survey and drawing, emergent accident processing and sea area search and rescue work. Because the unmanned ship and the equipment carried by the unmanned ship are often expensive in cost, if the unmanned ship collides with objects in the sea during navigation, the unmanned ship and the equipment can be seriously damaged, and if the unmanned ship collides with the ship, the human consequences can be further beyond the assumption. At present, technical schemes for safe obstacle avoidance of unmanned boats are developed successively, however, a large lifting space is still provided for active positioning and obstacle avoidance of unpredictable obstacles on the sea.
Research on autonomous obstacle avoidance of unmanned boats is increasing. At present, a YOLO series network model is generally adopted for detecting the marine obstacles, and the model has a better detection result aiming at the predicted obstacles but cannot identify unpredictable obstacles; the method for positioning the marine barrier mostly adopts a laser radar or binocular camera positioning method. The laser clustering method is easily affected by noise, the false alarm rate is high, and the clustering method is time-consuming for large-size obstacles. The binocular positioning method has the defects of long base line, poor stability, poor matching precision for a reflective object and a weak texture obstacle and low positioning precision. Therefore, how to improve the accuracy and real-time performance of the detection and positioning of unpredictable marine obstacles becomes a hot problem for the next research.
Disclosure of Invention
The invention aims to overcome the defects and provides a method for instantly positioning obstacles and constructing a map in a strange sea area based on DeeplabV3+, firstly, based on the conversion relation between a laser radar coordinate system and a visible light camera coordinate system, projecting laser point cloud of a laser radar onto an image obtained by the visible light camera to obtain laser point cloud projection coordinates in an image coordinate system, then, based on a trained Deeplabv3+ semantic segmentation network model, carrying out segmentation prediction on the image to obtain a segmented category image, and finally, according to the conversion relation between a laser radar coordinate system and an IMU coordinate system, the coordinates of the laser point clouds corresponding to the obstacles in the laser radar coordinate system and unmanned ship posture information acquired by the IMU, constructing a surrounding obstacle map with the unmanned ship as a map center. According to the method, the contour information of the predicted and unpredictable obstacles is accurately extracted by using the Deeplabv3+ semantic segmentation network model, and the accurate detection and positioning of the predicted and unpredictable obstacles on the sea are realized by combining the visible light camera and the laser radar.
In order to achieve the above purpose, the invention provides the following technical scheme:
a method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+, which comprises the following steps:
s1, obtaining a conversion relation between a laser radar coordinate system and a visible light camera coordinate system through combined calibration of the laser radar and the visible light camera;
s2, converting the coordinates of the laser point cloud in the laser radar coordinate system into the coordinates in the visible light camera coordinate system based on the conversion relation between the laser radar coordinate system and the visible light camera coordinate system, and projecting the laser point cloud of the laser radar onto the image obtained by the visible light camera according to the coordinates of the laser point cloud in the visible light camera coordinate system and the projection model of the visible light camera to obtain the laser point cloud projection coordinates in the visible light camera image coordinate system;
s3, constructing a Deeplabv3+ semantic segmentation network model, and training the Deeplabv3+ semantic segmentation network model;
s4, carrying out segmentation prediction on the image based on the trained Deeplabv3+ semantic segmentation network model to obtain a segmented class image; the category images include 4 types of sky, ocean, bird, and obstacle;
s5, making an obstacle mask by using the obstacles in the category image, processing the projection coordinates of the laser point clouds in the image coordinate system through the obstacle mask to obtain the laser point clouds corresponding to the obstacles, and filtering the laser point clouds corresponding to the non-obstacles;
s6, obtaining the conversion relation between the laser radar coordinate system and the IMU coordinate system through the combined calibration of the laser radar and the IMU;
s7, based on the conversion relation between the laser radar coordinate system and the IMU coordinate system, the conversion relation between the IMU coordinate system and the northeast coordinate system, the coordinates of the laser point cloud corresponding to the obstacle in the laser radar coordinate system and the unmanned ship attitude information acquired by the IMU, the position coordinates of the obstacle in the northeast coordinate system are calculated, and a surrounding obstacle map with the unmanned ship as the map center is constructed.
Further, in the step S1, the conversion relationship between the laser radar coordinate system and the visible light camera coordinate system includes a rotation matrix and a translation vector from the laser radar coordinate system to the visible light camera coordinate system;
in the step S6, the conversion relationship between the lidar coordinate system and the IMU coordinate system includes a rotation matrix and a translation vector from the lidar coordinate system to the IMU coordinate system;
in step S7, the unmanned surface vehicle attitude information acquired by the IMU includes a yaw angle, a pitch angle, and a roll angle.
Further, the specific method of step S1 is as follows:
s1.1, calibrating a visible light camera to obtain internal parameters of the visible light camera;
s1.2, synchronously acquiring checkerboard data of the checkerboard calibration plate by a visible light camera and a laser radar at the same acquisition frequency to respectively obtain an image and laser point cloud data of the checkerboard calibration plate;
s1.3, extracting coordinates of the checkerboard angular points of the image by using an angular point extraction algorithm to obtain coordinates of the checkerboard angular points in a visible light coordinate system;
s1.4, fitting by using laser point cloud data to obtain four corner point coordinates of the checkerboard calibration plate, and calculating the checkerboard corner point coordinates according to the four corner point coordinates to obtain coordinates of the checkerboard corner points in a laser radar coordinate system;
s1.5, solving to obtain the conversion relation between the laser radar coordinate system and the visible light camera coordinate system according to the internal parameters of the visible light camera, the coordinates of the checkerboard angular points in the visible light coordinate system and the coordinates of the checkerboard angular points in the laser radar coordinate system.
Further, in the step S1.2, the acquisition frequencies of the visible light camera and the laser radar are both 10 HZ; in step S1.2, no white edge is left at the edge of the checkerboard calibration board.
Further, in the step S2, laser points of the laser point cloud of the laser radar at an angle of 0-180 ° are projected onto an image obtained by the visible light camera.
Further, the specific method of step S3 is:
s3.1, collecting training samples by using a visible light camera to obtain a training sample set, and marking each training sample as 4 types of sky, ocean, bird and obstacle;
s3.2, selecting n images in different time periods and different weather as target images, converting each training sample into n training samples with colors similar to the colors of the n target images by using a color migration algorithm, wherein the n training samples and the original training samples form a new training sample set together; n is more than or equal to 6;
s3.3, performing data enhancement on the new training sample set obtained in the step S3.2 by using an image enhancement technology to obtain an enhanced training sample set;
s3.4, constructing a Deeplabv3+ semantic segmentation network model comprising an encoding module and a decoding module;
s3.5, loading weights of a pre-training model, and training a Deeplab V3+ semantic segmentation network model by using the training sample set obtained in the step S3.3; in the training process, the front 358 layers of the network model are frozen to perform iterative training, unfreezing is performed after 50 iterations, and the iterative training is continued, wherein the maximum iteration number is not less than 100.
Further, in the step S3.4, the coding module is composed of a backbone network Xception and a cavity space pyramid pooling module ASPP cascaded thereto; the cavity space pyramid pooling module ASPP comprises 1 multiplied by 1 convolution layers, cascaded cavity convolution modules and average pooling layers; the cascaded cavity convolution module comprises 3 cavity convolution layers with cavity convolution rates of 6, 12 and 18 respectively; the decoding module includes a 1 × 1 convolutional layer, a 3 × 3 convolutional layer, and two 4-fold bilinear interpolation upsampling modules.
Further, the specific method of step S5 is as follows:
s5.1, assigning the pixel corresponding to the obstacle in the class image to be 1, and assigning the pixel corresponding to other types in the class image to be 0 to obtain an obstacle mask;
s5.2, performing morphological opening operation and closing operation on the barrier mask; the size of a convolution kernel used in the morphological opening operation and the morphological closing operation is 3 multiplied by 3;
s5.3, judging whether the pixel point closest to the projection coordinate of the laser point cloud is located in the barrier mask; if the pixel point is not 1, judging that the laser point cloud is outside the obstacle, and filtering the laser point cloud, otherwise, judging that the laser point cloud corresponds to the obstacle, and keeping the laser point cloud projection.
Further, in the step S6, the method for jointly calibrating the laser radar and the IMU includes driving the unmanned ship at a constant speed, and synchronously acquiring the laser point cloud of the laser radar and the speed, acceleration and position data of the IMU, wherein the time for synchronous acquisition is 10-20 seconds each time; and obtaining the conversion relation between the laser radar coordinate system and the IMU coordinate system according to the laser point cloud and the velocity, acceleration and position data of the IMU.
Further, in step S7, the size of the obstacle map is m × m, where m is greater than or equal to 400; in the barrier map, the risk coefficient of the pixel closest to the position coordinate of the barrier in the northeast coordinate system is greater than 0, and the risk coefficients of the other pixels are 0; the resolution of the obstacle map is 1 m;
the risk factor of the pixel closest to the position coordinate of the obstacle in the northeast coordinate system is 100.
Compared with the prior art, the invention has at least one of the following beneficial effects:
(1) according to the invention, the image pixel level is divided into sky, ocean, bird and obstacle four categories based on the DeeplabV3+ semantic division network, so that the accurate extraction of the outline information of the predicted and unpredictable obstacles is realized; the situation that the unmanned boat starts to stop suddenly or avoid obstacles when meeting the birds is avoided by distinguishing the obstacles from the birds;
(2) in the training process of the Deeplab V3+ semantic segmentation network, the influence of time intervals and weather is fully considered, the diversity of training data in time and weather is enriched, and the training result of the Deeplab V3+ semantic segmentation network is optimized;
(3) according to the method, the conversion relation between the visible light camera coordinate system and the radar laser coordinate system is calculated, the laser point cloud is accurately projected to the visible light image, noise points of the laser point cloud are filtered by the barrier mask, the point cloud corresponding to the barrier is accurately reserved, and a foundation is provided for the accurate construction of the barrier map;
(4) the method is based on combined calibration of the laser radar and the IMU, and the northeast coordinate of the obstacle point cloud is accurately calculated; the cloud coordinates of the obstacle points are projected onto a map with an unmanned boat as the center, so that the rapid separation and positioning of the obstacles are realized, and the obstacle map generated by the method can provide real-time reliable reference for path planning and autonomous obstacle avoidance.
Drawings
FIG. 1 is a flow chart of an instant positioning and mapping method for obstacles in an unfamiliar sea area based on Deeplab V3 +;
FIG. 2 is a flowchart of image segmentation based on Deeplabv3+ semantic segmentation network model according to the present invention;
FIG. 3 is a flow chart of generating an obstacle map according to the present invention;
fig. 4 is a schematic diagram of an obstacle map according to the present invention.
Detailed Description
The features and advantages of the present invention will become more apparent and appreciated from the following detailed description of the invention.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
A method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ is realized by the following steps:
s1, obtaining a conversion relation between a laser radar coordinate system and a visible light camera coordinate system through combined calibration of the laser radar and the visible light camera;
s2, projecting the laser point cloud onto an image based on the conversion relation of the coordinate systems of the laser radar and the visible light camera and the projection model of the camera obtained by calibration of S1 to obtain the point cloud projection coordinates in the image coordinate system;
s3, constructing a Deeplabv3+ semantic segmentation network model, and training the Deeplabv3+ semantic segmentation network model;
s4, segmenting the visible light image based on the trained Deeplabv3+ network model to obtain a segmented category image;
s5, taking the barrier obtained by semantic segmentation as a mask, filtering laser point cloud projections on the image except the barrier, and keeping the laser point cloud projections corresponding to the barrier so as to obtain the coordinates of the laser point cloud corresponding to the barrier in a laser radar coordinate system;
s6, obtaining a conversion relation between a laser radar coordinate system and an IMU (inertial measurement unit) coordinate system through the joint calibration of the laser radar and the IMU;
s7, calculating the position coordinates of the obstacle in the northeast sky coordinate system based on the conversion relation between the laser radar and the IMU coordinate system, the yaw (yaw) angle, the pitch angle and the roll angle information of the unmanned ship obtained by the IMU, the conversion relation between the IMU coordinate system and the northeast sky coordinate system, the coordinates of the laser point cloud corresponding to the obstacle in the laser radar coordinate system, and constructing a surrounding obstacle map with the unmanned ship as the map center.
Further, in step S1, the conversion relationship between the laser radar coordinate system and the visible light camera coordinate system includes a rotation matrix and a translation vector from the laser radar coordinate system to the visible light camera coordinate system;
in step S6, the transformation relationship between the lidar coordinate system and the IMU coordinate system includes a rotation matrix and a translation vector from the lidar coordinate system to the IMU coordinate system.
Further, S1 specifically includes the following steps:
s1.1, calibrating a camera to obtain internal parameters, namely internal orientation elements, of the camera;
s1.2, preparing a checkerboard calibration board without leaving white edges on the edge, setting the acquisition frequency of the laser radar and the acquisition frequency of the visible light camera to be 10HZ, and synchronously acquiring checkerboard data of the visible light camera and the laser radar by utilizing a Rosbag command in ros (a robot operating system);
s1.3, extracting coordinates of the checkerboard angular points of the image by using an angular point extraction algorithm to obtain coordinates of the checkerboard angular points in a visible light coordinate system;
s1.4, extracting laser point cloud data corresponding to the checkerboard calibration plate from the laser point cloud, fitting a rectangle corresponding to the boundary of the checkerboard calibration plate, and obtaining three-dimensional point cloud coordinates of each checkerboard angular point of the checkerboard calibration plate, namely three-dimensional coordinates of the checkerboard angular point in a laser radar coordinate system, based on four angular point coordinates of the rectangle;
s1.5, based on the camera internal parameters, the coordinates of the checkerboard angular points on the image and the three-dimensional point cloud coordinates of the checkerboard angular points, the conversion relation between a camera coordinate system and a radar coordinate system is solved.
Further, in S2, selecting a laser point with a positive angle of 0-180 degrees from the laser point cloud for projection.
Further, S3 specifically includes the following steps:
s3.1, collecting training samples by using a visible light camera, and marking each image as 4 types of sky, ocean, bird and barrier;
s3.2, 6 images in different time periods and different weathers are selected as target images, each original training image is converted into 6 images with the color similar to that of the target image by using a color migration algorithm, and the diversity of training samples in time and weather is enriched;
s3.3, data enhancement is carried out on the training samples by using an image enhancement technology, so that the number of the samples is increased, and the diversity of the samples is enriched;
s3.4, constructing a Deeplabv3+ semantic segmentation network model comprising an encoding module and a decoding module; the coding module consists of a backbone network Xception and a cavity space pyramid pooling module ASPP cascaded with the backbone network Xception; in a preferred embodiment, the void space pyramid pooling module ASPP comprises a 1 × 1 convolution layer, a cascaded void convolution module and an average pooling layer; the cavity convolution module comprises 3 cavity convolution layers with cavity convolution rates of 6, 12 and 18 respectively; the decoding module comprises a 1 × 1 convolutional layer, a 3 × 3 convolutional layer and two 4-time bilinear interpolation up-sampling modules;
s3.5, loading weights of the pre-training model, and training the Deeplab V3+ network model by using training samples; in the training process, the front 358 layers of the network are firstly frozen to carry out iterative training, unfreezing is carried out after 50 times of iteration, and the iterative training is continued, wherein the maximum iteration time is not less than 100 times.
Further, S4 carries out segmentation prediction on the image by using the trained Deeplab V3+ network model to obtain the classification results of sky, ocean, bird and obstacle.
Further, S5 specifically includes the following steps:
s5.1, assigning the pixel corresponding to the obstacle in the segmented image to be 1, assigning the other classes to be 0, and manufacturing an obstacle mask;
s5.2, performing morphological opening operation and closing operation on the barrier mask, wherein the size of a convolution kernel is 3 multiplied by 3, so that a fine area is eliminated, and a cavity area is filled;
s5.3, selecting a pixel point nearest to the projection coordinate of the laser point cloud, and judging whether the pixel point is located in the barrier mask, namely whether the pixel point is 1; if not, filtering the laser point; otherwise, the laser point is reserved; obtaining a point cloud corresponding to the obstacle;
further, in S6, the unmanned ship runs at a constant speed on a calm sea surface, the ship speed is less than or equal to 6 knots, the laser point cloud and the IMU speed, acceleration and position data are synchronously acquired by using the rossbag, and the acquisition time is 10-20 seconds;
further, in S7, an image with a size of > 400 × 400 is created, pixels closest to the northeast coordinate point of the obstacle in the image are assigned as 100, and the remaining pixels are assigned as 0, so as to obtain an obstacle map of the boat. The obstacle map size is preferably 500 × 500. The pixel assignments represent the risk factor, 100 represents the highest risk, and 0 represents the lowest risk. The resolution of the obstacle map is 1 m.
Example 1:
as shown in fig. 1, an implementation step of an immediate obstacle positioning and mapping method in an unfamiliar sea area based on deepabv 3+ is as follows:
A. obtaining a conversion relation between a laser radar coordinate system and a visible light camera coordinate system, namely a rotation matrix and a translation vector from the laser radar coordinate system to the visible light camera coordinate system, through combined calibration of the laser radar and the visible light camera;
calibrating the camera by using a checkerboard calibration board with a white edge to obtain internal parameters of the camera; preparing a checkerboard calibration board without leaving white edges on the edge, setting the acquisition frequency of the laser radar and the acquisition frequency of the visible light camera to be 10HZ, and synchronously acquiring checkerboard data of the visible light camera and the laser radar by utilizing a Rosbag command in ros; extracting coordinates of checkerboard corner points in the visible light image by using a corner point extraction algorithm; extracting point cloud data corresponding to a calibration plate in laser point cloud by using laser point cloud processing software, fitting a rectangle corresponding to the boundary of the checkerboard calibration plate, and obtaining three-dimensional point cloud coordinates of each checkerboard angular point of the calibration plate based on four angular point coordinates of the rectangle; solving a conversion relation between a camera coordinate system and a radar coordinate system based on camera internal parameters, checkerboard corner point coordinates on the image and three-dimensional point cloud coordinates of the checkerboard corner points;
B. b, calibrating to obtain a coordinate system conversion relation between the laser radar and the visible light camera and a projection model of the camera based on the step A, and projecting the laser point cloud onto the image to obtain a point cloud projection coordinate in an image coordinate system;
in the step, laser points which are at right angles of 0-180 degrees in the laser point cloud are selected for projection, and the calculation method of the projection points comprises the following steps:
wherein, p is the projection coordinate of the laser point on the image, namely the projection coordinate in the image coordinate system;
k is an internal parameter matrix of the visible light camera;
p is a three-dimensional coordinate of the point cloud under a laser radar coordinate system;
Tclthe translation vector from the laser radar coordinate system to the visible light camera coordinate system;
C. constructing a Deeplabv3+ semantic segmentation network model, and training the Deeplabv3+ semantic segmentation network model;
as shown in fig. 2, training samples were collected with a camera, each image labeled 4 types of sky, ocean, bird, and obstacle; selecting 6 images in different time periods and different weather as target images, converting each original training image into 6 images with the color similar to that of the target images by using a color migration algorithm, adding the original images, and expanding the sample data amount to 7 times of the original sample data amount; performing data enhancement on training data by using an image enhancement technology, and increasing the number of samples; constructing a Deeplabv3+ semantic segmentation network model comprising an encoding module and a decoding module; the coding module consists of a backbone network Xception and a cavity space pyramid pooling module ASPP cascaded with the backbone network Xception; the ASPP module comprises a 1 multiplied by 1 convolution layer, a cascaded cavity convolution module and an average pooling layer; the cavity convolution module comprises 3 cavity convolution layers with cavity convolution rates of 6, 12 and 18 respectively; the decoding module comprises a 1 × 1 convolutional layer, a 3 × 3 convolutional layer and two 4-time bilinear interpolation up-sampling modules; loading the weight of a pre-training model, and training a Deeplab V3+ network model; in the training process, the front 358 layers of the network are firstly frozen to carry out iterative training, unfreezing is carried out after 50 times of iteration, and the iterative training is continued, wherein the maximum iteration time is not less than 100 times.
D. Segmenting the visible light image based on a trained Deeplabv3+ network model to obtain a segmented class image; in the step, the trained model (h5 format) is converted into pb format, and inference prediction is carried out on the image based on the dnn module of opencv to obtain the classification results of sky, ocean, bird and obstacle.
E. And taking the barrier obtained by semantic segmentation as a mask, filtering laser point cloud projection points outside the barrier on the image, and keeping the laser point cloud corresponding to the barrier.
In the step, the pixel corresponding to the barrier in the segmented image is assigned to be 1, and the other classes are assigned to be 0, so that a barrier mask is manufactured; performing morphological opening operation and closing operation on the barrier mask, wherein the size of a convolution kernel is 3 multiplied by 3, so that a fine area is eliminated, and a cavity area is filled;
selecting a pixel point nearest to the projection coordinate of the laser point cloud, and judging whether the pixel point is positioned in the barrier mask, namely whether the pixel point is 1; if not, filtering the laser point; otherwise, the laser point is reserved; obtaining laser point clouds corresponding to the obstacles;
F. obtaining a rotation matrix and a translation vector from a laser radar coordinate system to an IMU coordinate system through combined calibration of a laser radar and the IMU;
requiring the boat to run at a constant speed and a low speed on a calm sea surface, and synchronously acquiring laser point cloud and IMU speed, acceleration and position data by utilizing the rossbag, wherein the acquisition time is 10 seconds; separating message data of laser radar point cloud and velocity, acceleration and position message data of the IMU from bag files recorded by the Rosbag; and inputting data, establishing a calibration model according to the conversion relation between the laser radar and the IMU, and obtaining the conversion relation between the laser radar and the IMU through iterative optimization of a loss function.
G. Based on the conversion relation between the laser radar and the IMU coordinate system, the yaw, pitch and roll angle information acquired by the IMU, the coordinates of the obstacle corresponding to the laser point cloud in the laser radar coordinate system and the conversion relation between the MU coordinate system and the northeast coordinate system, the position coordinates of the obstacle in the northeast coordinate system are calculated, and a surrounding obstacle map with the ship as the map center is constructed.
As shown in fig. 3, calculating coordinates of the obstacle point cloud in the IMU coordinate system according to the conversion relationship between the lidar and the IMU; calculating the northeast coordinates of the obstacle point cloud according to the unmanned ship attitude information obtained by the IMU; a map two-dimensional image with the size of 500 x 500 is established, the nearest pixel of the northeast coordinate point of the obstacle in the image is assigned as 100, and the rest pixels are assigned as 0, so that the obstacle map of the boat is obtained as shown in fig. 4.
The invention has been described in detail with reference to specific embodiments and illustrative examples, but the description is not intended to be construed in a limiting sense. Those skilled in the art will appreciate that various equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its embodiments without departing from the spirit and scope of the present invention, which fall within the scope of the present invention. The scope of the invention is defined by the appended claims.
Those skilled in the art will appreciate that those matters not described in detail in the present specification are well known in the art.
Claims (10)
1. A method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ is characterized by comprising the following steps:
s1, obtaining a conversion relation between a laser radar coordinate system and a visible light camera coordinate system through combined calibration of the laser radar and the visible light camera;
s2, converting the coordinates of the laser point cloud in the laser radar coordinate system into the coordinates in the visible light camera coordinate system based on the conversion relation between the laser radar coordinate system and the visible light camera coordinate system, and projecting the laser point cloud of the laser radar onto the image obtained by the visible light camera according to the coordinates of the laser point cloud in the visible light camera coordinate system and the projection model of the visible light camera to obtain the laser point cloud projection coordinates in the visible light camera image coordinate system;
s3, constructing a Deeplabv3+ semantic segmentation network model, and training the Deeplabv3+ semantic segmentation network model;
s4, carrying out segmentation prediction on the image based on the trained Deeplabv3+ semantic segmentation network model to obtain a segmented class image; the category images include 4 types of sky, ocean, bird, and obstacle;
s5, making an obstacle mask by using the obstacles in the category image, processing the projection coordinates of the laser point clouds in the image coordinate system through the obstacle mask to obtain the laser point clouds corresponding to the obstacles, and filtering the laser point clouds corresponding to the non-obstacles;
s6, obtaining a conversion relation between a laser radar coordinate system and an IMU coordinate system through the combined calibration of the laser radar and the IMU;
s7, based on the conversion relation between the laser radar coordinate system and the IMU coordinate system, the conversion relation between the IMU coordinate system and the northeast coordinate system, the coordinates of the laser point cloud corresponding to the obstacle in the laser radar coordinate system and the unmanned ship attitude information acquired by the IMU, the position coordinates of the obstacle in the northeast coordinate system are calculated, and a surrounding obstacle map with the unmanned ship as the map center is constructed.
2. The method for instantly positioning and mapping obstacles in the unfamiliar sea area based on Deeplab V3+ as claimed in claim 1, wherein in step S1, the transformation relationship between the lidar coordinate system and the visible light camera coordinate system comprises a rotation matrix and a translation vector from the lidar coordinate system to the visible light camera coordinate system;
in the step S6, the conversion relationship between the lidar coordinate system and the IMU coordinate system includes a rotation matrix and a translation vector from the lidar coordinate system to the IMU coordinate system;
in step S7, the unmanned surface vehicle attitude information acquired by the IMU includes a yaw angle, a pitch angle, and a roll angle.
3. The method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ as claimed in claim 1, wherein the specific method in step S1 is:
s1.1, calibrating a visible light camera to obtain internal parameters of the visible light camera;
s1.2, synchronously acquiring checkerboard data of the checkerboard calibration plate by a visible light camera and a laser radar at the same acquisition frequency to respectively obtain an image and laser point cloud data of the checkerboard calibration plate;
s1.3, extracting coordinates of the checkerboard angular points of the image by using an angular point extraction algorithm to obtain coordinates of the checkerboard angular points in a visible light coordinate system;
s1.4, fitting by using laser point cloud data to obtain four corner point coordinates of the checkerboard calibration plate, and calculating the checkerboard corner point coordinates according to the four corner point coordinates to obtain coordinates of the checkerboard corner points in a laser radar coordinate system;
s1.5, solving to obtain a conversion relation between a laser radar coordinate system and a visible light camera coordinate system according to the internal parameters of the visible light camera, the coordinates of the checkerboard angular points in the visible light coordinate system and the coordinates of the checkerboard angular points in the laser radar coordinate system.
4. The method for instantly positioning and mapping the obstacles in the unfamiliar sea area based on Deeplab V3+ as claimed in claim 3, wherein in step S1.2, the acquisition frequency of the visible light camera and the acquisition frequency of the lidar are both 10 HZ; in step S1.2, no white edge is left at the edge of the checkerboard calibration board.
5. The method for instantly positioning and mapping the obstacles in the unfamiliar sea area based on Deeplab V3+ as claimed in claim 1, wherein in step S2, laser points of the laser point cloud of the laser radar at an angle of 0-180 ° are projected onto an image obtained by the visible light camera.
6. The method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ as claimed in claim 1, wherein the specific method in step S3 is:
s3.1, collecting training samples by using a visible light camera to obtain a training sample set, and marking each training sample as 4 types of sky, ocean, bird and obstacle;
s3.2, selecting n images in different time periods and different weather as target images, converting each training sample into n training samples with colors similar to the colors of the n target images by using a color migration algorithm, wherein the n training samples and the original training samples form a new training sample set together; n is more than or equal to 6;
s3.3, performing data enhancement on the new training sample set obtained in the step S3.2 by using an image enhancement technology to obtain an enhanced training sample set;
s3.4, constructing a Deeplabv3+ semantic segmentation network model comprising an encoding module and a decoding module;
s3.5, loading weights of a pre-training model, and training a Deeplab V3+ semantic segmentation network model by using the training sample set obtained in the step S3.3; in the training process, the front 358 layers of the network model are firstly frozen to carry out iterative training, unfreezing is carried out after 50 times of iteration, and the iterative training is continued, wherein the maximum iteration time is not less than 100 times.
7. The method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ as claimed in claim 6, wherein in step S3.4, the coding module comprises a backbone network Xception and a cavity space pyramid pooling module ASPP cascaded thereto; the ASPP comprises a 1 multiplied by 1 convolution layer, a cascaded cavity convolution module and an average pooling layer; the cascaded void convolution module comprises 3 void convolution layers with void convolution rates of 6, 12 and 18 respectively; the decoding module includes a 1 × 1 convolutional layer, a 3 × 3 convolutional layer, and two 4-fold bilinear interpolation upsampling modules.
8. The method for instantly positioning and mapping obstacles in an unfamiliar sea area based on Deeplab V3+ as claimed in claim 1, wherein the specific method in step S5 is:
s5.1, assigning the pixel corresponding to the obstacle in the class image to be 1, and assigning the pixel corresponding to other types in the class image to be 0 to obtain an obstacle mask;
s5.2, performing morphological opening operation and closing operation on the barrier mask; the size of a convolution kernel used in the morphological opening operation and the morphological closing operation is 3 multiplied by 3;
s5.3, judging whether the pixel point closest to the projection coordinate of the laser point cloud is located in the barrier mask; if the pixel point is not 1, judging that the laser point cloud is outside the obstacle, and filtering the laser point cloud, otherwise, judging that the laser point cloud corresponds to the obstacle, and keeping the laser point cloud projection.
9. The method for instantly positioning the obstacles and constructing the map in the unfamiliar sea area based on the Deeplab V3+ as claimed in claim 1, wherein in the step S6, the method for jointly calibrating the laser radar and the IMU comprises the steps of enabling the unmanned ship to run at a constant speed, synchronously acquiring the laser point cloud of the laser radar and the speed, acceleration and position data of the IMU, and the time for synchronously acquiring each time is 10-20 seconds; and obtaining the conversion relation between the laser radar coordinate system and the IMU coordinate system according to the laser point cloud and the velocity, acceleration and position data of the IMU.
10. The method for instantly positioning and mapping obstacles in unfamiliar sea areas based on Deeplab V3+ as claimed in claim 1, wherein in step S7, the size of the obstacle map is m x m, wherein m is greater than or equal to 400; in the barrier map, the risk coefficient of the pixel closest to the position coordinate of the barrier in the northeast coordinate system is greater than 0, and the risk coefficients of the other pixels are 0; the resolution of the obstacle map is 1 m;
the risk factor of the pixel closest to the position coordinate of the obstacle in the northeast coordinate system is 100.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111642668.2A CN114445572B (en) | 2021-12-29 | 2021-12-29 | DeeplabV3+ based method for immediately positioning and constructing map of obstacle in strange sea area |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111642668.2A CN114445572B (en) | 2021-12-29 | 2021-12-29 | DeeplabV3+ based method for immediately positioning and constructing map of obstacle in strange sea area |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114445572A true CN114445572A (en) | 2022-05-06 |
| CN114445572B CN114445572B (en) | 2024-05-14 |
Family
ID=81365095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111642668.2A Active CN114445572B (en) | 2021-12-29 | 2021-12-29 | DeeplabV3+ based method for immediately positioning and constructing map of obstacle in strange sea area |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114445572B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116416319A (en) * | 2022-11-17 | 2023-07-11 | 南京理工大学 | Intelligent driving multi-type sensor calibration-oriented one-time combined calibration method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200301013A1 (en) * | 2018-02-09 | 2020-09-24 | Bayerische Motoren Werke Aktiengesellschaft | Methods and Apparatuses for Object Detection in a Scene Based on Lidar Data and Radar Data of the Scene |
| CN111709343A (en) * | 2020-06-09 | 2020-09-25 | 广州文远知行科技有限公司 | Point cloud detection method and device, computer equipment and storage medium |
| WO2021104497A1 (en) * | 2019-11-29 | 2021-06-03 | 广州视源电子科技股份有限公司 | Positioning method and system based on laser radar, and storage medium and processor |
| CN113705375A (en) * | 2021-08-10 | 2021-11-26 | 武汉理工大学 | Visual perception device and method for ship navigation environment |
| CN113743391A (en) * | 2021-11-08 | 2021-12-03 | 江苏天策机器人科技有限公司 | Three-dimensional obstacle detection system and method applied to low-speed autonomous driving robot |
-
2021
- 2021-12-29 CN CN202111642668.2A patent/CN114445572B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200301013A1 (en) * | 2018-02-09 | 2020-09-24 | Bayerische Motoren Werke Aktiengesellschaft | Methods and Apparatuses for Object Detection in a Scene Based on Lidar Data and Radar Data of the Scene |
| WO2021104497A1 (en) * | 2019-11-29 | 2021-06-03 | 广州视源电子科技股份有限公司 | Positioning method and system based on laser radar, and storage medium and processor |
| CN111709343A (en) * | 2020-06-09 | 2020-09-25 | 广州文远知行科技有限公司 | Point cloud detection method and device, computer equipment and storage medium |
| CN113705375A (en) * | 2021-08-10 | 2021-11-26 | 武汉理工大学 | Visual perception device and method for ship navigation environment |
| CN113743391A (en) * | 2021-11-08 | 2021-12-03 | 江苏天策机器人科技有限公司 | Three-dimensional obstacle detection system and method applied to low-speed autonomous driving robot |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116416319A (en) * | 2022-11-17 | 2023-07-11 | 南京理工大学 | Intelligent driving multi-type sensor calibration-oriented one-time combined calibration method |
| CN116416319B (en) * | 2022-11-17 | 2023-11-24 | 南京理工大学 | Intelligent driving multi-type sensor calibration-oriented one-time combined calibration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114445572B (en) | 2024-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108152831B (en) | Laser radar obstacle identification method and system | |
| CN113359810B (en) | Unmanned aerial vehicle landing area identification method based on multiple sensors | |
| KR102530691B1 (en) | Device and method for monitoring a berthing | |
| CN110222612B (en) | Dynamic target identification and tracking method for autonomous landing of unmanned aerial vehicle | |
| US20220024549A1 (en) | System and method for measuring the distance to an object in water | |
| CN112001226B (en) | Unmanned 3D target detection method, device and storage medium | |
| CN111753677B (en) | Multi-angle remote sensing ship image target detection method based on characteristic pyramid structure | |
| CN113177593B (en) | Fusion method of radar point cloud and image data in water traffic environment | |
| CN109857144A (en) | Unmanned plane, unmanned aerial vehicle control system and control method | |
| CN114926739B (en) | Unmanned collaborative acquisition processing method for geographical space information on water and under water of inland waterway | |
| CN114612769B (en) | Integrated sensing infrared imaging ship detection method integrated with local structure information | |
| KR102466804B1 (en) | Autonomous navigation method using image segmentation | |
| CN113743385A (en) | Unmanned ship water surface target detection method and device and unmanned ship | |
| CN114004977A (en) | Aerial photography data target positioning method and system based on deep learning | |
| Zhang et al. | A object detection and tracking method for security in intelligence of unmanned surface vehicles | |
| CN110427030B (en) | Unmanned ship autonomous docking recovery method based on Tiny-YolOship target detection algorithm | |
| CN116087982A (en) | Marine water falling person identification and positioning method integrating vision and radar system | |
| le Fevre Sejersen et al. | Safe vessel navigation visually aided by autonomous unmanned aerial vehicles in congested harbors and waterways | |
| Thompson | Maritime object detection, tracking, and classification using lidar and vision-based sensor fusion | |
| CN114445572B (en) | DeeplabV3+ based method for immediately positioning and constructing map of obstacle in strange sea area | |
| CN114049362A (en) | Transform-based point cloud instance segmentation method | |
| CN113933828A (en) | Unmanned ship environment self-adaptive multi-scale target detection method and system | |
| CN115187959B (en) | Method and system for landing flying vehicle in mountainous region based on binocular vision | |
| CN116486252A (en) | Intelligent unmanned search and rescue system and search and rescue method based on improved PV-RCNN target detection algorithm | |
| CN115792912A (en) | Method and system for sensing environment of unmanned surface vehicle based on fusion of vision and millimeter wave radar under weak observation condition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |