CN117115170A - Self-adaptive SAR ship detection method and system in unsupervised domain - Google Patents
Self-adaptive SAR ship detection method and system in unsupervised domain Download PDFInfo
- Publication number
- CN117115170A CN117115170A CN202311385351.4A CN202311385351A CN117115170A CN 117115170 A CN117115170 A CN 117115170A CN 202311385351 A CN202311385351 A CN 202311385351A CN 117115170 A CN117115170 A CN 117115170A
- Authority
- CN
- China
- Prior art keywords
- ship
- sar
- image
- target
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 114
- 230000003287 optical effect Effects 0.000 claims abstract description 44
- 238000012360 testing method Methods 0.000 claims abstract description 30
- 238000012549 training Methods 0.000 claims abstract description 21
- 238000000034 method Methods 0.000 claims description 33
- 238000000605 extraction Methods 0.000 claims description 24
- 230000007246 mechanism Effects 0.000 claims description 15
- 230000003044 adaptive effect Effects 0.000 claims description 13
- 230000004927 fusion Effects 0.000 claims description 11
- 230000002776 aggregation Effects 0.000 claims description 7
- 238000004220 aggregation Methods 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 5
- 230000002708 enhancing effect Effects 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 11
- 230000006978 adaptation Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000012512 characterization method Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 5
- 230000006872 improvement Effects 0.000 description 5
- 230000004913 activation Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000011056 performance test Methods 0.000 description 4
- 238000005215 recombination Methods 0.000 description 3
- 230000006798 recombination Effects 0.000 description 3
- 238000002679 ablation Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000013140 knowledge distillation Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an unsupervised domain self-adaptive SAR ship detection method and system, comprising the following steps: acquiring an optical ship image and an SAR ship image; constructing a UDA-SARDet model; training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model; and detecting the SAR image ship by using the UDA-SARDet model passing the test. According to the SAR image detection method, the ship detection task of the SAR image is efficiently completed by using the unlabeled SAR ship image and the existing labeled optical data set. Meanwhile, a brand new network model structure and IoU loss function are designed, so that the problems that SAR images are multi-scale, few in characteristics, easy to lose information and the like are solved, and the detection performance of the model is improved.
Description
Technical Field
The invention belongs to the technical field of SAR image target detection, and particularly relates to an unsupervised domain self-adaptive SAR ship detection method and system.
Background
Synthetic Aperture Radar (SAR) is a high-resolution imaging radar, and has the characteristics of realizing all-day and all-weather earth observation without being limited by illumination, climate conditions and the like. SAR image ship detection has important research value in the military field, but the SAR image target detection task is very challenging due to the complex imaging mechanism of the SAR image. The traditional detection method based on the structure, the gray features and the texture features has great influence on the detection performance when the problems of complex scenes, multiple scales of targets and the like are faced. The SAR image ship detection method based on the convolutional neural network is influenced by factors such as seriously deficient multi-angle and multi-parameter SAR ship data volume in a real application scene, high real-time requirements of the SAR image ship detection, and the like, so that the SAR image ship detection still faces significant challenges.
Domain adaptation is a method for implementing migration of a model of a source domain to a target domain by learning the difference between the source domain and the target domain. The method solves the problems of incomplete data set, unbalance, inaccurate marking and the like in practical application due to higher data acquisition and marking cost. The field self-adaptive proposal provides ideas for SAR image ship detection. In practical application, the field self-adaptive method achieves good effect in the optical image target detection task, but is still blank in the SAR image ship detection task and faces a great challenge. First, the domain difference between the optical data set and the SAR data set is much larger than the domain difference between different optical data sets, and the huge domain difference may cause the performance of the domain adaptive method in SAR image target detection to be reduced. Secondly, due to a special imaging mechanism of the SAR image, the common target detection algorithm cannot effectively extract the characteristics of the SAR target, and the requirements on detection performance in practical application are difficult to meet. Finally, most field adaptive methods are still based on a two-stage detector, namely a fast R-CNN, and in SAR image ship detection tasks with limited resources and urgent time, a single-stage detector with high real-time performance and high precision is more hoped to be used.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides an unsupervised domain self-adaptive SAR ship detection method and system, which efficiently complete the ship detection task of SAR images by using unlabeled SAR ship images and the existing labeled optical data sets. Meanwhile, a brand new network model structure and IoU loss function are designed, so that the problems that SAR images are multi-scale, few in characteristics, easy to lose information and the like are solved, and the detection performance of the model is improved.
In order to achieve the above object, the present invention provides the following solutions:
an unsupervised domain self-adaptive SAR ship detection method comprises the following steps:
acquiring an optical ship image and an SAR ship image;
constructing a UDA-SARDet model;
training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and detecting the SAR image ship by using the UDA-SARDet model passing the test.
Preferably, the UDA-sarset model includes: self-attention feature extraction backbone network SAFEBckebone, lightweight feature fusion neck LFNeck, image generation network CycleGAN-SAR and Loss。
preferably, the image generation network CycleGAN-SAR is: an image generation network based on CycleGAN adapted for use between an optical image and a SAR image; the method comprises the following steps: the SCA attention mechanism is added to the CycleGAN arbiter.
Preferably, the self-attention feature extraction backbone network SAFEBckebone is: based on CSPDarknet53, a Swim-Transformer Block module and a SCA attention mechanism module in a Swim-Transformer are introduced to generate the self-attention feature extraction backbone network SAFEBckebone.
Preferably, the lightweight feature fusion neck LFFNeck is: based on PANet, adding a feature map up-sampling module CARAFE and a context aggregation module SCABlock.
Preferably, the saidThe expression for Loss is:
。
wherein the method comprises the steps ofRepresenting the loss function when the intersection ratio of the predicted frame and the actual frame is calculated using EIOU.
Preferably, the method for completing SAR image ship detection by using the UDA-SARDet model passing the test comprises the following steps:
taking the optical ship image as source domain data and taking the SAR ship image as target domain data;
inputting the source domain data and the target domain data into the image generation network CycleGAN-SAR to obtain source-like domain data and target-like domain data;
cutting the source domain data, the target domain data, the class source domain data and the class target domain data, inputting the cut source domain data, the target domain data and the class target domain data into the self-attention feature extraction backbone network SAFEBcKebone, and extracting ship target features;
based on the lightweight characteristic fusion neck LFFNeck, enhancing the ship target characteristic;
and inputting the enhanced ship target characteristics into a detection layer to finish ship target detection of the target domain image.
The invention also provides an unsupervised domain self-adaptive SAR ship detection system, which comprises: the device comprises an acquisition module, a construction module, a test module and a detection module;
the acquisition module is used for acquiring an optical ship image and an SAR ship image;
the building module is used for building a UDA-SARDet model;
the test module is used for training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and the detection module is used for completing detection of SAR image ships by using the UDA-SARDet model passing the test.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides an unsupervised domain adaptive SAR ship detection method UDA-SARDet with high precision and high real-time performance from the optical domain to the SAR domain, which is a single-stage detection method, wherein an optical ship image containing rich information is taken as a source domain, the SAR ship image is taken as a target domain, and source domain data are completely marked while target domain data are not marked. To reduce the gap between the source domain and the target domain, a new image generation network CycleGAN-SAR based on CycleGAN design is introduced to realize pseudo-image generation of the optical image and the SAR image. And a knowledge distillation framework is adopted, and a Mean Teacher is used for guiding Teacher network detection to detect unlabeled target domain images and generate pseudo labels, so that the student network is updated relatively unbiased. The method for detecting the self-adaptive SAR ship from the optical domain to the SAR is experimentally compared with other latest self-adaptive methods in the non-supervision domain on the SAR data set SSDD and the optical data set DIOR, and experimental results show that the method is greatly improved. The innovation of the invention mainly comprises the following aspects:
1. the invention provides an unsupervised domain self-adaptive SAR ship detection method from optics to SAR for the first time, and reduces the domain difference between a source domain and a target domain through a CycleGAN-SAR. The advantages of large data volume of the optical image, rich target information and the like are fully exerted, the problem of low model performance caused by lack of labeling data volume in SAR image ship detection is solved, and the performance of SAR image ship detection is improved under the condition that expensive labeling cost is not required to be generated.
2. The invention designs a feature extraction backbone network SAFEBckebone which can utilize a self-attention mechanism to mine feature characterization potential. The method can capture global information and rich context information, reduce the influence of complex background of SAR images on ship detection, and enable the model to still play good performance under the condition of high-density targets.
3. The invention designs a brand new lightweight feature fusion neck LFFNock, which aims at the problems of low resolution, few features, easy information loss and the like of a small-size ship, introduces a more efficient up-sampling module and a context aggregation module, learns global space content and improves the detection effect on the small-size ship target.
4. The invention designs a simple and efficient deviceLoss。/>The Loss has the characteristic of balancing the contribution of the high-quality sample and the low-quality sample to the Loss, improves the contribution of the high-quality sample, suppresses the contribution of the low-quality sample, and is more suitable for SAR image ship target detection tasks based on unsupervised domain adaptation.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are needed in the embodiments are briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of an unsupervised domain adaptive SAR ship detection method in an embodiment of the present invention;
FIG. 2 is a schematic diagram of a network structure of a UDA-SARDet according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a network structure of SAFEBckebone according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an original CA module and a modified SCA module structure according to an embodiment of the present invention, where (a) is a schematic diagram of an original CA module structure, and (b) is a schematic diagram of a modified SCA module structure;
fig. 5 is a schematic diagram of a network structure of LFFNeck in an embodiment of the present invention;
FIG. 6 is a schematic view of CABACK and SCABlock structures according to an embodiment of the present invention, wherein (a) is a schematic view of CABACK structure, and (b) is a schematic view of SCABlock structure;
FIG. 7 shows EIoU Loss, focal-EIoU Loss and Focal-EIoU Loss in an embodiment of the inventionThe function diagram of Loss is compared with a schematic diagram.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1
As shown in fig. 1, the method for detecting the self-adaptive SAR ship in the unsupervised domain comprises the following steps:
acquiring an optical ship image and an SAR ship image;
constructing a UDA-SARDet model;
training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and (3) detecting the SAR image ship by using the UDA-SARDet model passing the test.
In this embodiment, the self-attention feature extraction backbone network SAFEBcKebone, lightweight feature fusion neck LFNeck, image generation network CycleGAN-SAR, and simple and efficientLoss is the main module and strategy of the UDA-sarset model, and the overall network structure is shown in fig. 2.
In this embodiment, the self-attention feature extraction backbone network SAFEBckebone is: based on CSPDarknet53, a Swim-Transformer Block module in a Swim-Transformer and a newly designed SCA attention mechanism module are introduced to generate the self-attention feature extraction backbone network SAFEBckebone. As shown in fig. 3.
SAFEBckebone combines the advantages of CNN network local feature extraction and transform global relation modeling, and deep image feature characterization potential is mined through global information and rich context information, so that ship target features are extracted more effectively.
The invention replaces the last two C3 modules in CSPDarknet53 with Swim-Transformer Block modules and renames them as C3STR modules. After the C3STR module, an improved version SCA of Coordinate Attention attention mechanism (CA) is added, replacing the sigmoid activation function in CA with a smoother SiLU activation function, as shown in (a), (b) of fig. 4.
In the embodiment, in the SAR image ship detection task, a brand new lightweight characteristic recombination and enhanced neck LFFNECK is designed aiming at the problems of multi-scale ship targets, few small-size target characteristics, easy information loss and the like. LFNeck is based on PANet, and adds a feature map up-sampling module CARAFE and a context aggregation module SCABlock. The network structure is shown in fig. 5.
The invention replaces the original upsampling module in PANet with the feature map upsampling module CARAFE, and has the advantages of less CARAFE calculation cost and high calculation speed, thus being a lightweight and efficient upsampling module. CARAFE has a large receptive field, can better utilize surrounding information, supports instance-specific content-aware processing, dynamically generates an adaptive kernel, and upsamples based on semantic information of input content. The problem of information loss of small and medium-sized targets in SAR image ship detection is solved, and characteristic information of the ship targets is utilized more efficiently. Meanwhile, before target detection is carried out on three feature graphs with different scales, a context aggregation module SCABlock is added, and features are further enhanced by learning global space contexts of each scale. The SCABlock is a modified version of CABlock, as shown in (a), (b) of fig. 6. In SAR ship detection, the SCABlock can effectively fuse local and global features, reduce information confusion, enhance feature extraction of feature graphs with different scales, alleviate the problems of multi-scale and few features of ship targets, and improve the detection precision of ships in SAR images.
In this embodiment, the EIoU Loss has a faster convergence speed and a better positioning result in the calculation of various IoU Loss. Meanwhile, focal-EIoU Loss is provided in combination with Focal Loss, so that the problem of sample unbalance in a bounding box regression task is optimized, and the regression process is more focused on a high-quality prediction box, and the formula is as follows:
wherein the method comprises the steps ofIs a super-parameter for controlling the radian of a curve.
In the SAR image ship detection task, the target characteristic information is difficult to extract, so that the number of high-quality samples is far less than that of low-quality samples, and the training samples are extremely unbalanced. Therefore, the invention designs a simpler and more efficient regression frame Loss based on the idea of EIoU LossLoss, the form of which is as follows:
because the functional expression of Focal-EIoU Loss is complex, the functional expression of the Focal-EIoU Loss and the functional expression of Focal-EIoU Loss are more intuitively compared,Differences in Loss, approximate IoU in the Focal-EIoU Loss function expression was replaced with EIoU, and the changes in three IoU Loss with EIoU are shown in fig. 7. As can be seen from fig. 7, in the Focal-EIoU Loss, the contribution of the low-quality samples to the Loss is strongly suppressed, compared to the improvement of the contribution of the high-quality samples to the Loss. Although this design may focus the network model more on the detection of high quality samples, in small target-based SAR image ship detection tasks, low quality prediction frames and low confidence predictions may be ignored or assigned to small targets, affecting the performance of the model. But->The Loss is improved on the basis, the contribution of the high-quality sample and the low-quality sample to the Loss is balanced, the attention to the low-quality sample is enhanced, the detection performance of the model to the small target ship is improved, and the SAR image ship detection task is more suitable. At the same time (I)>The computational complexity of Loss is lower, and compared with Focal-EIoU Loss, the computational parameters are fewer, so that the convergence rate of the model is increased.
In this embodiment, the image generation network CycleGAN-SAR is: an image generation network based on CycleGAN adapted for use between an optical image and a SAR image; the method comprises the following steps: the SCA attention mechanism is added to the CycleGAN arbiter.
In SAR ship detection based on unsupervised domain adaptation, the optical image of the source domain and the SAR image of the target domain are different in imaging principle, so that the source domain and the target domain have great differences in texture, color, appearance and the like, namely the domain difference between the source domain and the target domain is large. In the field-adaptive task, domain differences are important factors that lead to reduced model performance. In order to alleviate the influence of domain differences on model performance, the invention designs an image generation network CycleGAN-SAR applicable between an optical image and an SAR image based on CycleGAN. The invention adds an SCA attention mechanism in a CycleGAN discriminator and designs a new image generation network named CycleGAN-SAR. The SCA can accurately capture the position information of the region of interest and effectively capture the relation among channels. The distinguishing capability of the model to different areas of the image is enhanced, and the quality of the generated pseudo image is improved. Meanwhile, the high-quality target domain pseudo image and the source domain sample share a common label, so that the number of trainable samples is enriched, and the generalization capability of the model is enhanced.
In this embodiment, the method for completing the detection of the SAR image ship by using the UDA-SARDet model passing the test comprises the following steps:
taking the optical ship image as source domain data and taking the SAR ship image as target domain data;
inputting the source domain data and the target domain data into the image generation network CycleGAN-SAR to obtain source-like domain data and target-like domain data;
cutting the source domain data, the target domain data, the class source domain data and the class target domain data, inputting the cut source domain data, the target domain data and the class target domain data into the self-attention feature extraction backbone network SAFEBcKebone, and extracting ship target features;
based on the lightweight characteristic fusion neck LFFNeck, enhancing the ship target characteristic;
and inputting the enhanced ship target characteristics into a detection layer to finish ship target detection of the target domain image.
Specifically, experimental verification is performed on a commonly used optical image dataset DIOR and a SAR image ship target detection dataset SSDD to evaluate the performance of the UDA-SARDet model. The DIOR dataset is a large-scale, publicly available optical remote sensing image target detection reference dataset proposed by the northwest university of industry, covering 20 types of remote sensing ground object targets, including 23463 images and 190288 instances. The SSDD data set is the first data set disclosed at home and abroad and specially used for SAR image ship target detection, the resolution is 1-15m, and ship targets are arranged in large sea areas and near-shore areas.
In the experiment, only the images containing ship targets in the DIOR dataset are used, and the 1169 Zhang Jianchuan images are selected as source fields. 928 images in the SSDD are selected as a target domain training set, and the remaining 232 images are selected as a target domain testing set.
Firstly, source domain data and target domain training data are generated into high-quality class source domain data and class target domain data through a CycleGAN-SAR module, wherein the source domain data and the class source domain data share labels, the target domain training data and the class target domain data do not have labels, then the four types of data are uniformly cut into sizes of 512 x 512 and input into a UDA-SARDet model, and finally performance test of the model is carried out on target domain test data.
In the whole experimental process, firstly, source domain data and target domain training data pass through a CycleGAN-SAR module. The CycleGAN-SAR is an image generation module more suitable for use between an optical image and a SAR image, and the generated source-like domain data and target-like domain data are more similar to the real image data in style. Of the four types of image data, the source domain data and the class source domain data share labels, while the target domain training data and the class target domain data do not have labels. And then, uniformly cutting the four types of data into 512 x 512 sizes, inputting the 512 x 512 data into a self-attention feature extraction backbone network SAFEBckebone, and combining the SAFEBckebone with the deep mining image feature characterization potential of global information and contextual information to more effectively extract ship target features so that the subsequent network can better utilize the features. The lightweight feature rebinning and enhancement neck LFFNeck then further enhances the features by learning the global spatial context for each scale. The influence of the problems of multi-scale ship targets, small-size targets with few characteristics, easy information loss and the like on the model detection performance is relieved. And finally, outputting a ship target detection result of the target domain image through the detection layer, and performing performance test of the model on the target domain test data. In the process of calculating the loss function of ship target detection, the invention provides a new simple and efficient regression frame lossLoss replaces the original regression box Loss. />The Loss balances the contribution of the high-quality sample and the low-quality sample to the Loss, strengthens the attention of the low-quality sample, improves the detection performance of the model on the small target ship, and is more suitable for SAR image ship detection tasks.
Example two
The invention also provides an unsupervised domain self-adaptive SAR ship detection system, which comprises: the device comprises an acquisition module, a construction module, a test module and a detection module;
the acquisition module is used for acquiring an optical ship image and an SAR ship image;
the building module is used for building a UDA-SARDet model;
the test module is used for training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and the detection module is used for completing detection of SAR image ships by using the UDA-SARDet model passing the test.
In this embodiment, the self-attention feature extraction backbone network SAFEBcKebone, lightweight feature fusion neck LFNeck, image generation network CycleGAN-SAR, and simple and efficientLoss is the main module and strategy of the UDA-sarset model, and the overall network structure is shown in fig. 2.
In this embodiment, the self-attention feature extraction backbone network SAFEBckebone is: based on CSPDarknet53, a Swim-Transformer Block module in a Swim-Transformer and a newly designed SCA attention mechanism module are introduced to generate the self-attention feature extraction backbone network SAFEBckebone. As shown in fig. 3.
SAFEBckebone combines the advantages of CNN network local feature extraction and transform global relation modeling, and deep image feature characterization potential is mined through global information and rich context information, so that ship target features are extracted more effectively.
The invention replaces the last two C3 modules in CSPDarknet53 with Swim-Transformer Block modules and renames them as C3STR modules. After the C3STR module, an improved version SCA of Coordinate Attention attention mechanism (CA) is added, replacing the sigmoid activation function in CA with a smoother SiLU activation function, as shown in (a), (b) of fig. 4.
In the embodiment, in the SAR image ship detection task, a brand new lightweight characteristic recombination and enhanced neck LFFNECK is designed aiming at the problems of multi-scale ship targets, few small-size target characteristics, easy information loss and the like. LFNeck is based on PANet, and adds a feature map up-sampling module CARAFE and a context aggregation module SCABlock. The network structure is shown in fig. 5.
The invention replaces the original upsampling module in PANet with the feature map upsampling module CARAFE, and has the advantages of less CARAFE calculation cost and high calculation speed, thus being a lightweight and efficient upsampling module. CARAFE has a large receptive field, can better utilize surrounding information, supports instance-specific content-aware processing, dynamically generates an adaptive kernel, and upsamples based on semantic information of input content. The problem of information loss of small and medium-sized targets in SAR image ship detection is solved, and characteristic information of the ship targets is utilized more efficiently. Meanwhile, before target detection is carried out on three feature graphs with different scales, a context aggregation module SCABlock is added, and features are further enhanced by learning global space contexts of each scale. SCABlock is a modified version of CABLock, as shown in FIG. 6. In SAR ship detection, the SCABlock can effectively fuse local and global features, reduce information confusion, enhance feature extraction of feature graphs with different scales, alleviate the problems of multi-scale and few features of ship targets, and improve the detection precision of ships in SAR images.
In this embodiment, the EIoU Loss has a faster convergence speed and a better positioning result in the calculation of various IoU Loss. Meanwhile, focal-EIoU Loss is provided in combination with Focal Loss, so that the problem of sample unbalance in a bounding box regression task is optimized, and the regression process is more focused on a high-quality prediction box, and the formula is as follows:
wherein the method comprises the steps ofIs a super-parameter for controlling the radian of a curve.
In the SAR image ship detection task, the target characteristic information is difficult to extract, so that the number of high-quality samples is far less than that of low-quality samples, and the training samples are extremely unbalanced. Therefore, the invention designs a simpler and more efficient regression frame Loss based on the idea of EIoU LossLoss, the form of which is as follows:
because the functional expression of Focal-EIoU Loss is complex, the functional expression of the Focal-EIoU Loss and the functional expression of Focal-EIoU Loss are more intuitively compared,Differences in Loss, approximate IoU in the Focal-EIoU Loss function expression was replaced with EIoU, and the changes in three IoU Loss with EIoU are shown in fig. 7. As can be seen from fig. 7, in the Focal-EIoU Loss, the contribution of the low-quality samples to the Loss is strongly suppressed, compared to the improvement of the contribution of the high-quality samples to the Loss. Although this design may focus the network model more on the detection of high quality samples, in small target-based SAR image ship detection tasks, low quality prediction frames and low confidence predictions may be ignored or assigned to small targets, affecting the performance of the model. But->Loss is improved on the basis of the improvement, and the high-quality sample and low-quality sample pair are balancedThe contribution of loss strengthens the attention to low-quality samples, improves the detection performance of a model to a small target ship, and is more suitable for SAR image ship detection tasks. At the same time (I)>The computational complexity of Loss is lower, and compared with Focal-EIoU Loss, the computational parameters are fewer, so that the convergence rate of the model is increased.
In this embodiment, the image generation network CycleGAN-SAR is: an image generation network based on CycleGAN adapted for use between an optical image and a SAR image; the method comprises the following steps: the SCA attention mechanism is added to the CycleGAN arbiter.
In SAR ship detection based on unsupervised domain adaptation, the optical image of the source domain and the SAR image of the target domain are different in imaging principle, so that the source domain and the target domain have great differences in texture, color, appearance and the like, namely the domain difference between the source domain and the target domain is large. In the field-adaptive task, domain differences are important factors that lead to reduced model performance. In order to alleviate the influence of domain differences on model performance, the invention designs an image generation network CycleGAN-SAR applicable between an optical image and an SAR image based on CycleGAN. The invention adds an SCA attention mechanism in a CycleGAN discriminator and designs a new image generation network named CycleGAN-SAR. The SCA can accurately capture the position information of the region of interest and effectively capture the relation among channels. The distinguishing capability of the model to different areas of the image is enhanced, and the quality of the generated pseudo image is improved. Meanwhile, the high-quality target domain pseudo image and the source domain sample share a common label, so that the number of trainable samples is enriched, and the generalization capability of the model is enhanced.
In this embodiment, the method for completing the detection of the SAR image ship by using the UDA-SARDet model passing the test comprises the following steps:
taking the optical ship image as source domain data and taking the SAR ship image as target domain data;
inputting the source domain data and the target domain data into the image generation network CycleGAN-SAR to obtain source-like domain data and target-like domain data;
cutting the source domain data, the target domain data, the class source domain data and the class target domain data, inputting the cut source domain data, the target domain data and the class target domain data into the self-attention feature extraction backbone network SAFEBcKebone, and extracting ship target features;
based on the lightweight characteristic fusion neck LFFNeck, enhancing the ship target characteristic;
and inputting the enhanced ship target characteristics into a detection layer to finish ship target detection of the target domain image.
Specifically, experimental verification is performed on a commonly used optical image dataset DIOR and a SAR image ship target detection dataset SSDD to evaluate the performance of the UDA-SARDet model. The DIOR dataset is a large-scale, publicly available optical remote sensing image target detection reference dataset proposed by the northwest university of industry, covering 20 types of remote sensing ground object targets, including 23463 images and 190288 instances. The SSDD data set is the first data set disclosed at home and abroad and specially used for SAR image ship target detection, the resolution is 1-15m, and ship targets are arranged in large sea areas and near-shore areas.
In the experiment, only the images containing ship targets in the DIOR dataset are used, and the 1169 Zhang Jianchuan images are selected as source fields. 928 images in the SSDD are selected as a target domain training set, and the remaining 232 images are selected as a target domain testing set.
Firstly, source domain data and target domain training data are generated into high-quality class source domain data and class target domain data through a CycleGAN-SAR module, wherein the source domain data and the class source domain data share labels, the target domain training data and the class target domain data do not have labels, then the four types of data are uniformly cut into sizes of 512 x 512 and input into a UDA-SARDet model, and finally performance test of the model is carried out on target domain test data.
In the whole experimental process, firstly, source domain data and target domain training data pass through a CycleGAN-SAR module. The CycleGAN-SAR is an image generation module which is more suitable for the optical image and SAR image, and generates source-like domain data, target-like domain data and trueThe image data style is more similar. Of the four types of image data, the source domain data and the class source domain data share labels, while the target domain training data and the class target domain data do not have labels. And then, uniformly cutting the four types of data into 512 x 512 sizes, inputting the 512 x 512 data into a self-attention feature extraction backbone network SAFEBckebone, and combining the SAFEBckebone with the deep mining image feature characterization potential of global information and contextual information to more effectively extract ship target features so that the subsequent network can better utilize the features. The lightweight feature rebinning and enhancement neck LFFNeck then further enhances the features by learning the global spatial context for each scale. The influence of the problems of multi-scale ship targets, small-size targets with few characteristics, easy information loss and the like on the model detection performance is relieved. And finally, outputting a ship target detection result of the target domain image through the detection layer, and performing performance test of the model on the target domain test data. In the process of calculating the loss function of ship target detection, the invention provides a new simple and efficient regression frame lossLoss replaces the original regression box Loss. />The Loss balances the contribution of the high-quality sample and the low-quality sample to the Loss, strengthens the attention of the low-quality sample, improves the detection performance of the model on the small target ship, and is more suitable for SAR image ship detection tasks.
Example III
In order to verify the high performance of the proposed UDA-sarset, the present invention compares UDA-sarset with the latest unsupervised domain adaptation method on DIOR and SSDD datasets. In the comparison experiment, the experimental parameters are set to be the same as much as possible, so that the fairness of the experiment is ensured, and the experimental result is shown in the comparison of the table 1 and the latest unsupervised domain adaptation method.
TABLE 1
Where source only represents the YOLOv5s detection network using only source image training without domain adaptation, baseline is the network before UDA-sarset design, DA and DTCN are the latest unsupervised domain adaptation methods, upper bound is the fully supervised YOLOv5s detection method. Analysis of the data in the tables shows that UDA-SARDet is significantly better than all comparison methods. Compared with DA and DTCN, UDA-SARDet is more accurate in detecting small targets and has lower false detection rate and omission rate. The application of the UDA-SARDet in SAR image ship detection tasks is effective, and compared with other unsupervised domain adaptation methods, the UDA-SARDet is designed aiming at the characteristics of SAR ship targets, so that the performance of the model is improved to a great extent.
To demonstrate that we designed CycleGAN-SAR, SAFEBckebone, LFFNeck andeffectiveness of Loss we performed ablation experiments on the DIOR and SSDD datasets. We combine these designs differently to more fully understand the impact of different designs on model detection performance. The ablation experimental results are shown in table 2, and from the table, it can be seen that the image generation network CycleGAN-SAR, the self-attention feature extraction backbone network SAFEBckebone, the lightweight feature recombination and enhancement neck LFFNeck, and the simplicity and high efficiency are shown>The detection performance of the Loss on the model is improved.
TABLE 2
The model provided by the invention can be combined with the labeled optical ship data set and the unlabeled SAR ship image data for training, and the trained weight is saved. And then inputting the unknown SAR ship image to be processed into a model, applying a trained weight file, performing target detection on the unknown SAR ship image, and marking the ship target.
The above embodiments are merely illustrative of the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, but various modifications and improvements made by those skilled in the art to which the present invention pertains are made without departing from the spirit of the present invention, and all modifications and improvements fall within the scope of the present invention as defined in the appended claims.
Claims (8)
1. An unsupervised domain self-adaptive SAR ship detection method is characterized by comprising the following steps:
acquiring an optical ship image and an SAR ship image;
constructing a UDA-SARDet model;
training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and detecting the SAR image ship by using the UDA-SARDet model passing the test.
2. The unsupervised domain adaptive SAR ship detection method according to claim 1, wherein the UDA-sarset model comprises: self-attention feature extraction backbone network SAFEBckebone, lightweight feature fusion neck LFNeck, image generation network CycleGAN-SAR and Loss。
3. the unsupervised domain adaptive SAR ship detection method according to claim 2, wherein the image generation network CycleGAN-SAR is: an image generation network based on CycleGAN adapted for use between an optical image and a SAR image; the method comprises the following steps: the SCA attention mechanism is added to the CycleGAN arbiter.
4. The unsupervised domain adaptive SAR ship detection method according to claim 2, wherein the self-attention feature extraction backbone network safebckene is: based on CSPDarknet53, a Swim-Transformer Block module and a SCA attention mechanism module in a Swim-Transformer are introduced to generate the self-attention feature extraction backbone network SAFEBckebone.
5. The unsupervised domain adaptive SAR ship detection method according to claim 2, wherein the lightweight feature fusion neck LFFNeck is: based on PANet, adding a feature map up-sampling module CARAFE and a context aggregation module SCABlock.
6. The unsupervised domain adaptive SAR ship detection method of claim 2, wherein said method comprisesThe expression for Loss is:
wherein->Representing the loss function when the intersection ratio of the predicted frame and the actual frame is calculated using EIOU.
7. The unsupervised domain adaptive SAR ship detection method according to claim 2, wherein the method for performing SAR image ship detection using the UDA-sarset model passing the test comprises:
taking the optical ship image as source domain data and taking the SAR ship image as target domain data;
inputting the source domain data and the target domain data into the image generation network CycleGAN-SAR to obtain source-like domain data and target-like domain data;
cutting the source domain data, the target domain data, the class source domain data and the class target domain data, inputting the cut source domain data, the target domain data and the class target domain data into the self-attention feature extraction backbone network SAFEBcKebone, and extracting ship target features;
based on the lightweight characteristic fusion neck LFFNeck, enhancing the ship target characteristic;
and inputting the enhanced ship target characteristics into a detection layer to finish ship target detection of the target domain image.
8. An unsupervised domain adaptive SAR ship detection system, comprising: the device comprises an acquisition module, a construction module, a test module and a detection module;
the acquisition module is used for acquiring an optical ship image and an SAR ship image;
the building module is used for building a UDA-SARDet model;
the test module is used for training and testing the optical ship image and the SAR ship image as input data of the UDA-SARDet model;
and the detection module is used for completing detection of SAR image ships by using the UDA-SARDet model passing the test.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311385351.4A CN117115170B (en) | 2023-10-25 | 2023-10-25 | Self-adaptive SAR ship detection method and system in unsupervised domain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311385351.4A CN117115170B (en) | 2023-10-25 | 2023-10-25 | Self-adaptive SAR ship detection method and system in unsupervised domain |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117115170A true CN117115170A (en) | 2023-11-24 |
| CN117115170B CN117115170B (en) | 2024-01-12 |
Family
ID=88813220
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311385351.4A Active CN117115170B (en) | 2023-10-25 | 2023-10-25 | Self-adaptive SAR ship detection method and system in unsupervised domain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117115170B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190356684A1 (en) * | 2018-05-17 | 2019-11-21 | Adobe Inc. | Detecting robotic internet activity across domains utilizing one-class and domain adaptation machine-learning models |
| CN113869361A (en) * | 2021-08-20 | 2021-12-31 | 深延科技(北京)有限公司 | Model training method, target detection method and related device |
| EP4170605A1 (en) * | 2021-10-22 | 2023-04-26 | University Industry Foundation, Yonsei University (UIF) | Multi-level transferable region-based domain adaptive object detection apparatus and method |
| CN116343045A (en) * | 2023-03-30 | 2023-06-27 | 南京理工大学 | Lightweight SAR image ship target detection method based on YOLO v5 |
| CN116503732A (en) * | 2023-03-30 | 2023-07-28 | 北京理工大学 | Domain-adaptation-based unsupervised SAR image ship target detection method |
| CN116912680A (en) * | 2023-06-25 | 2023-10-20 | 西南交通大学 | SAR ship identification cross-modal domain migration learning and identification method and system |
-
2023
- 2023-10-25 CN CN202311385351.4A patent/CN117115170B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190356684A1 (en) * | 2018-05-17 | 2019-11-21 | Adobe Inc. | Detecting robotic internet activity across domains utilizing one-class and domain adaptation machine-learning models |
| CN113869361A (en) * | 2021-08-20 | 2021-12-31 | 深延科技(北京)有限公司 | Model training method, target detection method and related device |
| EP4170605A1 (en) * | 2021-10-22 | 2023-04-26 | University Industry Foundation, Yonsei University (UIF) | Multi-level transferable region-based domain adaptive object detection apparatus and method |
| CN116343045A (en) * | 2023-03-30 | 2023-06-27 | 南京理工大学 | Lightweight SAR image ship target detection method based on YOLO v5 |
| CN116503732A (en) * | 2023-03-30 | 2023-07-28 | 北京理工大学 | Domain-adaptation-based unsupervised SAR image ship target detection method |
| CN116912680A (en) * | 2023-06-25 | 2023-10-20 | 西南交通大学 | SAR ship identification cross-modal domain migration learning and identification method and system |
Non-Patent Citations (2)
| Title |
|---|
| KAI LEI ETC: ""Detecting Malicious Domains with Behavioral Modeling and Graph Embedding"", 《2019 IEEE 39TH INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS (ICDCS)》 * |
| 周立君;刘宇;白璐;茹志兵;于帅;: "一种基于GAN和自适应迁移学习的样本生成方法", 应用光学, no. 01 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117115170B (en) | 2024-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Scale adaptive proposal network for object detection in remote sensing images | |
| CN112966684A (en) | Cooperative learning character recognition method under attention mechanism | |
| Wan et al. | AFSar: An anchor-free SAR target detection algorithm based on multiscale enhancement representation learning | |
| CN108846446A (en) | The object detection method of full convolutional network is merged based on multipath dense feature | |
| CN106408030A (en) | SAR image classification method based on middle lamella semantic attribute and convolution neural network | |
| CN109886147A (en) | A kind of more attribute detection methods of vehicle based on the study of single network multiple-task | |
| Sun et al. | Image recognition technology in texture identification of marine sediment sonar image | |
| Sun et al. | IRDCLNet: Instance segmentation of ship images based on interference reduction and dynamic contour learning in foggy scenes | |
| Wang et al. | Accurate real-time ship target detection using Yolov4 | |
| Lou et al. | Generative knowledge transfer for ship detection in SAR images | |
| Hao et al. | Research on underwater object detection based on improved YOLOv4 | |
| Zhao et al. | A target detection algorithm for remote sensing images based on a combination of feature fusion and improved anchor | |
| Yan et al. | 3MNet: Multi-task, multi-level and multi-channel feature aggregation network for salient object detection | |
| CN117115170B (en) | Self-adaptive SAR ship detection method and system in unsupervised domain | |
| Zhu et al. | Application of pornographic images recognition based on depth learning | |
| CN114898290A (en) | Real-time detection method and system for marine ship | |
| Tian et al. | Weighted Pseudo-labels and Bounding Boxes for Semi-supervised SAR Target Detection | |
| Wang et al. | ALF-YOLO: Enhanced YOLOv8 based on multiscale attention feature fusion for ship detection | |
| Luo et al. | Infrared Road Object Detection Based on Improved YOLOv8. | |
| Zhang et al. | Parallel Connecting Deep and Shallow CNNs for Simultaneous Detection of Big and Small Objects | |
| Liang et al. | SCDFMixer: Spatial-Channel Dual-Frequency Mixer Based on Satellite Optical Sensors for Remote Sensing Multi-Object Detection | |
| Sun et al. | Real-time Underwater Target Tracking Using PP-YOLO and Cloud Computing | |
| Guo et al. | An attention model based on spatial transformers for scene recognition | |
| Qu et al. | Research on UAV Image Detection Method in Urban Low-altitude Complex Background | |
| Zhu et al. | Discriminative feature pyramid network for object detection in remote sensing images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |