CN113240017B - Multispectral and panchromatic image classification method based on attention mechanism - Google Patents
Multispectral and panchromatic image classification method based on attention mechanism Download PDFInfo
- Publication number
- CN113240017B CN113240017B CN202110542410.9A CN202110542410A CN113240017B CN 113240017 B CN113240017 B CN 113240017B CN 202110542410 A CN202110542410 A CN 202110542410A CN 113240017 B CN113240017 B CN 113240017B
- Authority
- CN
- China
- Prior art keywords
- sample
- multispectral
- panchromatic
- network
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, and belongs to the technical field of image processing. Initial feature extraction is carried out on the multispectral sample and the panchromatic sample in the training sample set by constructing the training sample set and the testing sample set, so that initial features of the multispectral sample and initial features of the panchromatic sample with the same feature size are obtained; continuously and respectively inputting the deep features into a lightweight sharing-separating network to extract deep features of the multispectral sample and deep features of the panchromatic sample; continuously and respectively carrying out self-adaptive feature fusion and outputting fusion features; classifying the fusion features, and training the network according to the classification result; and based on the trained network, obtaining classification results of the multispectral sample and the panchromatic sample in the training sample set. The problems of poor classification time cost and multi-sensor fusion effect in the prior art are solved.
Description
Technical Field
The invention belongs to the technical field of image processing, and relates to a multispectral and full-color image classification method based on an attention mechanism.
Background
The rapid development of aerospace technology generates a large number of remote sensing images of different sensors, so that the combination of a plurality of sensor images to effectively classify ground objects becomes a research hot spot, and the classification research of multispectral and full-color images is particularly carried out. A typical remote sensing satellite has a full color sensor that can spectrally respond to a wide range of spectra to form a full color image. The full-color image is a gray-scale image and has high spatial resolution, but because of only one spectral band, the spectral resolution is low, the type of the ground object cannot be determined, and the identification of the ground object type is extremely unfavorable. To make up for the shortage of full-color images, a multispectral sensor (red, green, blue, near infrared, etc. are commonly carried on the satellite) at the same time. Multispectral sensors have high spectral resolution, but lower spatial resolution due to physical device limitations. Therefore, how to fully utilize complementary features of multispectral and panchromatic images to improve classification accuracy is a challenge.
Methods for classifying in conjunction with multispectral and panchromatic images generally fall into two categories: the first is to use full-color images to improve the spatial resolution of multispectral images, obtain fused multispectral images, and classify the fused multispectral images; the second category is to extract the features of multispectral and panchromatic images, respectively, and then fusion classify the extracted features. The first class of methods focuses more on how to acquire valid fused images, while the second class of methods focuses more on how to extract features efficiently. In recent years, in order to improve the classification accuracy, deep neural networks have achieved desirable effects in remote sensing image classification. Several typical deep learning networks, such as stacked auto encoders, convolutional neural networks, generating countermeasure networks, and recurrent neural networks, have been widely used in remote sensing image classification. The current multispectral and panchromatic image classification method based on the deep neural network mostly belongs to a second class of methods, namely: deep neural networks are respectively adopted to extract deep features of multispectral images and full-color images, and stacked connection is adopted to fuse and classify the deep features of the multispectral images and the full-color images. This type of method is simple and effective, but also has certain limitations: (1) Extracting the features of multispectral and panchromatic images requires two separate networks to complete, but training two separate networks can create a significant time penalty. (2) The simple stacked feature fusion approach considers that the multispectral image and the panchromatic image contribute to the classification to the same extent, and ignores the problem that the contribution of different images to the classification is unequal.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a multispectral and panchromatic image classification method based on an attention mechanism, which solves the problems of poor classification time cost and multi-sensor fusion effect in the prior art so as to balance the classification precision and time cost of multispectral and panchromatic images.
In order to achieve the above purpose, the invention is realized by adopting the following technical scheme:
the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which comprises the following steps:
firstly, respectively constructing a training sample set and a testing sample set which are composed of a multispectral sample and a full-color sample, and extracting initial characteristics of the multispectral sample and the full-color sample in the training sample set to respectively obtain initial characteristics of the multispectral sample and initial characteristics of the full-color sample with the same characteristic size;
constructing a lightweight sharing-separating network; respectively inputting the initial characteristics of the multispectral sample and the initial characteristics of the panchromatic sample in the training sample set obtained in the step one into a lightweight shared-separation network, and extracting deep characteristics to obtain the deep characteristics of the multispectral sample and the deep characteristics of the panchromatic sample respectively;
step three, respectively carrying out self-adaptive feature fusion on the deep features of the multispectral sample obtained in the step two and the deep features of the full-color sample, and outputting fusion features;
classifying the fusion characteristics obtained by the output of the step three, and training the network according to classification results; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in the training sample set; the multi-spectral image and full-color image classification method based on the attention mechanism is completed.
Preferably, in the first step, a training sample set and a test sample set composed of a multispectral sample and a panchromatic sample are constructed, and initial feature extraction is performed on the multispectral sample and the panchromatic sample in the training sample set, including the following steps:
according to each labeled pixel (some pixels are not labeled) on the multispectral image, respectively extracting multispectral samples and full-color samples in pairs on the multispectral image and the full-color image, and forming sample pairs of the labeled pixels; forming a sample set according to the obtained sample pairs; selecting part of sample pairs in the sample set to form a training sample set, and selecting the rest sample pairs in the sample set to form a test sample set;
constructing a multispectral image initial feature extraction network consisting of 2 convolution layers and 2 downsampling layers, and initializing parameters of each layer; taking a multispectral sample in the training sample set as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral sample through forward propagation;
constructing a full-color image initial feature extraction network consisting of 3 convolution layers and 3 downsampling layers, and initializing parameters of each layer; and taking the full-color samples in the training sample set as the input of the full-color image initial feature extraction network, and obtaining initial features of the full-color samples through forward propagation.
Further preferably, the specific network design of the multispectral sample initial feature extraction network is: the convolution kernel size of the first convolution layer is 3 multiplied by 3, the number of output features is 32, the convolution kernel size of the second convolution layer is 3 multiplied by 3, the number of output features is 64, the downsampling layers all adopt the maximum downsampling method, and the sampling kernel is 2 multiplied by 2.
Further preferably, the specific network of the full-color sample initial feature extraction network is designed as follows: the convolution kernel size of the first convolution layer is 3×3, the number of output features is 16, the convolution kernel size of the second convolution layer is 3×3, the number of output features is 32, the convolution kernel size of the second convolution layer is 3×3, the number of output features is 64, the downsampling layers all adopt the maximum downsampling method, and the sampling kernel is 2×2.
Preferably, in the second step, a lightweight shared-separate network is constructed, including the steps of:
constructing a shared sub-network consisting of 1 convolution layer and 1 downsampling layer, and initializing parameters of each layer;
respectively constructing a spectrum separation sub-network and a full-color separation sub-network; the spectrum separation sub-network and the full-color separation sub-network have the same network structure and both comprise a compression part and an excitation part;
the compression is to perform global average operation on the input features to obtain global compression feature vectors of the input features, the excitation is to input the global compression feature vectors into 2 fully connected layers, the weights of all channels in the initial features are obtained through output, finally, the obtained weights are used for carrying out linear weighted summation on all channels of the input features, and the features of the linear weighted summation are used as the input of a next-layer network.
Further preferably, the compression is a global compression feature vector z of the input features obtained by performing a global averaging operation on the input features c Wherein the global averaging operation is according to the following formula:
wherein f c (i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively;
excitation is to compress the feature vector z globally c The method comprises the steps of inputting the multi-spectral sample into 2 full-connection layers, wherein the number of the features of a first full-connection layer is 4, the number of the features of a second full-connection layer is 64, outputting a 64-dimensional weight vector, namely the weight of each channel in the initial features of the multi-spectral sample and the initial features of the full-color sample, and finally carrying out linear weighted summation on each channel of the input features by the obtained weight, and taking the linear weighted summation features as the input of a next-layer network.
Further preferably, in the second step, the specific step of deep feature extraction is as follows:
firstly, respectively inputting initial characteristics of a multispectral sample and initial characteristics of a panchromatic sample into an established shared sub-network, and respectively outputting the shared characteristics of the multispectral sample and the shared characteristics of the panchromatic sample through forward propagation;
then, the sharing characteristics of the obtained multispectral sample are input into a built spectrum separation sub-network, and the separation characteristics of the multispectral sample are output through forward propagation; inputting the shared characteristics of the obtained full-color sample into a built full-color separation sub-network, and outputting the separation characteristics of the full-color sample through forward propagation;
and finally, taking the separation characteristics of the output multispectral sample and the separation characteristics of the panchromatic sample as the input of the established shared sub-network, and circulating the steps for three times to respectively obtain the deep characteristics of the multispectral sample and the deep characteristics of the panchromatic sample.
Preferably, in the third step, the specific steps for performing adaptive feature fusion are as follows:
adding the deep features of the obtained multispectral sample and the deep features of the panchromatic sample to obtain initial fusion features;
performing global compression and excitation on the obtained initial fusion characteristics to respectively obtain fusion weights of the multispectral samples and fusion weights of the panchromatic samples; carrying out linear weighted summation on the fusion weight of the obtained multispectral sample and the deep features of the obtained multispectral sample according to the channel to obtain the linear weighted summation features of the multispectral sample; carrying out linear weighted summation on the fusion weight of the obtained panchromatic sample and the deep layer characteristic of the panchromatic sample according to the channel to obtain the linear weighted summation characteristic of the panchromatic sample; the linear weighted sum feature of the multispectral sample and the linear weighted sum feature of the panchromatic sample are added to output an adaptive fusion feature.
Preferably, in the fourth step, training the network according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in the training sample set, comprising the following steps:
inputting the obtained fusion characteristics into a softmax classifier for classification, and outputting the probability of each sample pair belonging to each class in a training sample set;
constructing a cross entropy loss function according to the output probability and the class mark of the sample pair; adopting a gradient descent method to counter-propagate the cross entropy loss function obtained by construction, and training all networks involved in the steps 1 to 4; repeating the network training in the steps 1 to 4 until reaching the preset iteration step, and stopping iteration;
taking the multispectral sample and the panchromatic sample in the test sample set as the input of the step 1, executing the steps 1 to 4, outputting the probability of each sample pair in the test sample set belonging to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the sample pair in the test sample set to finish classification.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which can effectively extract the characteristic differences of different sensors through capturing the correlation of characteristic channels; the problem of high calculation cost in the traditional multispectral and panchromatic image classification method is solved by the design of a lightweight sharing-separating network, and the training efficiency of the network is improved; by considering the difference of contributions of different sensors to classification results, a attention mechanism is introduced, so that a self-adaptive feature fusion network is realized, the problem that input images are unbalanced in contribution to classification in the traditional feature fusion method is solved, the fusion effect is improved, and the classification accuracy is further improved. Therefore, the multispectral and full-color image classification method based on the attention mechanism improves the effect of feature fusion by designing a light-weight network structure so as to better balance time cost and classification effect.
Drawings
FIG. 1 is a flow diagram of a method for classifying multispectral and panchromatic images based on an attention mechanism of the present invention;
FIG. 2 is a data set used in the experiments of the present invention, wherein (a) is a Sichuan suburban multispectral and panchromatic image data set and (b) is a Sichuan suburban multispectral and panchromatic image data set;
FIG. 3 is a classification diagram of different methods on a suburban Western Annulus dataset; wherein, (a) is an EMAP method, (b) is a CAE method, (c) is an RNN method, (d) is an SCPF-ResNet method, (e) is a CNN-MS method, (f) is a CNN-PAN method, (g) is an SFNet method, and (h) is the method of the invention;
FIG. 4 is a classification diagram of different methods on a western An urban dataset; wherein, (a) is an EMAP method, (b) is a CAE method, (c) is an RNN method, (d) is an SCPF-ResNet method, (e) is a CNN-MS method, (f) is a CNN-PAN method, (g) is an SFNet method, and (h) is the method of the invention;
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme and effect of the present invention are described in further detail below with reference to the accompanying drawings.
Referring to fig. 1, the implementation steps of the present invention are as follows:
step 1, multispectral and panchromatic images are input.
A multispectral image and a full-color image are input separately, as shown in fig. 2.
And 2, acquiring a training sample set and a test sample set.
In step 2, respectively extracting multispectral samples and full-color samples in pairs on multispectral images and full-color images to form sample pairs, and performing the following steps:
for each labeled pixel on the multispectral image:
2a) Defining a space window with the size of 32 multiplied by 32 on the multispectral image by taking the pixel as the center to obtain a multispectral sample;
2b) Multiplying the position of the pixel on the multispectral image by 4, mapping the pixel to a corresponding position on the panchromatic image, and defining a space window with the size of 128 multiplied by 128 on the panchromatic image by taking the pixel on the mapped position as the center to obtain a panchromatic sample corresponding to the multispectral sample, wherein the panchromatic sample has the same class label as the multispectral sample;
2c) Combining the multispectral sample and the panchromatic sample into a sample pair;
all sample pairs are formed into a sample set, and 100 sample pairs are randomly selected for each class in the sample set to form a training sample set; the rest samples form a test sample set;
and 3, extracting initial characteristics.
3a) Initial characteristic extraction of a multispectral sample:
3a1) Constructing a multispectral image initial feature extraction network consisting of 2 convolution layers and 2 downsampling layers, wherein the convolution kernel of the first convolution layer is 3×3, the number of output features is 32, the convolution kernel of the second convolution layer is 3×3, the number of output features is 64, the downsampling layers adopt the maximum downsampling method, and the sampling kernel is 2×2;
3a2) Taking a multispectral sample in a training sample pair as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral sample through forward propagation;
3b) Full color sample initial feature extraction:
3b1) Constructing a full-color image initial feature extraction network consisting of 3 convolution layers and 3 downsampling layers, wherein the convolution kernel of the first convolution layer is 3×3, the number of output features is 16, the convolution kernel of the second convolution layer is 3×3, the number of output features is 32, the convolution kernel of the second convolution layer is 3×3, the number of output features is 64, the downsampling layers adopt the maximum downsampling method, and the sampling kernel is 2×2;
3b2) Taking a corresponding panchromatic sample in the training sample pair as input of a panchromatic image initial feature extraction network, and performing forward propagation to obtain initial features of the panchromatic sample;
and 4, building a sharing-separating network.
4a) Building a shared sub-network: the shared sub-network consists of 1 convolution layer and one downsampling layer, wherein the convolution kernel of the convolution layer is 3 multiplied by 3, the number of output features is 64, and the maximum downsampling method is adopted, and the sampling kernel is 2 multiplied by 2;
4b) Respectively constructing a spectrum separation sub-network and a full-color separation sub-network: each separation sub-network has the same network structure, and the separation sub-network is divided into a compression part and an excitation part, and the network structures are respectively as follows:
4b1) Compression is the feature f to the input c Global average operation is carried out to obtain global compressed feature vector z of input features c Wherein the global averaging operation is according to the following formula:
wherein f c (i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively.
4b2) The excitation is to input global compressed feature vectors into 2 fully connected layers, wherein the number of features of a first fully connected layer is 4, the number of features of a second fully connected layer is 64, a 64-dimensional weight vector is output, the vector dimension is the weight of each channel in the initial features obtained in the step 3, and finally, each channel of the input features is subjected to linear weighted summation by the obtained weight, and the linear weighted summation features are used as the input of a next-layer network;
step 5, extracting deep features f of the multispectral sample and the panchromatic sample respectively according to the established sharing-separating network MS And f PAN 。
5a) Inputting the initial characteristics of the multispectral sample obtained in the step 3 into the shared sub-network constructed in the step 4 a), and outputting the shared characteristics of the multispectral sample through forward propagation;
5b) Inputting the initial characteristics of the full-color sample obtained in the step 3 into the shared sub-network built in the step 4 a), and outputting the shared characteristics of the full-color sample through forward propagation;
note that: the shared sub-networks in step 5 a) and step 5 b) are the same network, sharing parameters;
5c) Inputting the shared characteristics of the multispectral sample as input characteristics into the spectrum separation network constructed in the step 4 b), and outputting the separation characteristics of the multispectral sample through forward propagation;
5d) Inputting the shared characteristic of the full-color sample as an input characteristic into the full-color separation ion network constructed in the step 4 b), and outputting the separation characteristic of the full-color sample through forward propagation;
note that: the spectral separation sub-network and the full color separation sub-network in step 5 c) and step 5 d) have the same network structure, but are not the same network, and the parameters are not shared;
5e) Taking the separation characteristic of the multispectral sample output in 5 c) and the separation characteristic of the full-color sample output in 5 d) as the output characteristics of 5 a) and 5 b), respectively, and circulating the steps 5 a) -5 d) three times to obtain deep layer characteristics f of the multispectral sample MS And deep features f of panchromatic samples PAN ；
And 6, self-adaptive feature fusion.
6a) Deep features f for multispectral samples MS And deep features f of panchromatic samples PAN Performing addition operation to obtain initial fusion feature f add Wherein the adding operation is:
f add ＝f MS +f PAN ； (B)
6b) For the initial fusion feature f add Performing the whole processLocal compression and excitation to obtain fusion weight w of multispectral and panchromatic samples respectively (1) And w (2) The method is characterized by comprising the following steps:
6b1) For the initial fusion feature f add Performing global average operation to obtain a global compressed feature vector of the initial fusion feature, wherein the global average operation is shown in a formula (A);
6b2) Inputting the global compressed feature vector of the initial fusion feature into 1 full-connection layer with 6 features, and outputting a fusion compressed feature through forward propagation;
6b3) Building a multispectral full-connecting layer with 64 characteristics, taking the fusion compression characteristics obtained in the step 6 c) as the input of the multispectral full-connecting layer, performing forward calculation, and outputting multispectral fusion excitation characteristics;
6b4) Building a full-color full-connecting layer with 64 characteristics, taking the fusion compression characteristics obtained in the step 6 c) as the input of the full-color full-connecting layer, calculating forward, and outputting full-color fusion excitation characteristics;
note that: the multispectral full-connecting layer and the full-color full-connecting layer are two networks;
6b5) The multispectral fusion excitation feature and the panchromatic fusion excitation feature are connected in a stacked mode, the connected feature is used as the input of a softmax classifier, the feature output by the classifier is equally divided into two parts, and multispectral sample fusion weight w is generated respectively (1) Fusion weight w with panchromatic sample (2) ；
6c) Weighting w of multispectral samples (1) Deep features f with multispectral samples MS Linear weighted summation is carried out according to the channels to obtain linear weighted summation characteristic F of the multispectral sample (1) And the weight of the full-color sample is related to the deep layer characteristic f of the full-color sample PAN Linear weighted summation is carried out according to the channels to obtain linear weighted summation characteristic F of the panchromatic sample (2) ；
6d) Will F (1) And F (2) Adding and outputting an adaptive fusion characteristic F;
step 7, taking the self-adaptive fusion characteristic F as the input of a softmax classifier, and outputting the probability of each multispectral and panchromatic image sample pair belonging to each class;
and 8, training the network.
Constructing a cross entropy loss function according to the probability output in the step 7) and class labels of multispectral and panchromatic sample pairs in a training sample set, reversely propagating the cross entropy loss function by adopting a gradient descent method, training all networks involved in the steps 3) to 7), repeating the steps 3 to 8 until reaching a preset iteration step, and stopping iteration;
and 9, taking the multispectral and panchromatic sample pairs in the test sample set as the input of the step 3, executing the steps 3 to 7, outputting the probability that each sample pair in the test sample set belongs to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the test sample pair to finish classification.
Examples
The effect of the invention can be further illustrated by the following simulation experiments:
(1) Simulation conditions
The simulated hardware conditions of the invention are as follows: windows XP, SPI, CPU Pentium (R) 4, the fundamental frequency is 2.4GHZ; the software platform is as follows: matlabR2016a, pytorch;
the simulated image sources are multispectral and panchromatic image data sets of the western security suburban area and the western security urban area, wherein the multispectral and panchromatic image data set of the western security suburban area comprises 8 types of features as shown in fig. 2 (a), and the multispectral and panchromatic image data set of the western security urban area comprises 7 types of features as shown in fig. 2 (b); in the invention, 100 pixel points are randomly selected for each class as initial training pixels.
Simulation content and results
Simulation 1, the two data sets shown in fig. 2 are respectively classified and simulated by using the present invention and the seven existing technologies, and the result is shown in the figure, wherein:
FIGS. 3 (a) through (h) are graphs of classification effects of the present technique on a western-style suburban dataset for EMAP, CAE, RNN, SCPF-ResNet, CNN-MS, CNN-PAN, SFNet, respectively;
FIGS. 4 (a) through (h) are graphs of classification effects of the present technique on the Western An urban dataset for EMAP, CAE, RNN, SCPF-ResNet, CNN-MS, CNN-PAN, SFNet, respectively;
as can be seen from the classification result diagrams of fig. 3-4, the classification method of the present invention has better accuracy and classification effect. Tables 1 and 2 show the index values in terms of values of the classification method of the present invention and other seven classification methods, and also show that the classification accuracy obtained by the present invention is better. And the magnitude of the training parameters of various methods in the training process is also shown in the table 3, so that the training parameters of the method are obviously reduced, and the time cost and the classification effect are better balanced.
TABLE 1 index values of the multi-spectral and panchromatic image classification method (the method of the present invention) and the other seven classification methods based on the attention mechanism of the present invention
TABLE 2 index values of the multi-spectral and panchromatic image classification method (the method of the present invention) and the other seven classification methods based on the attention mechanism of the present invention
TABLE 3 training parameters for the attention-based multispectral and panchromatic image classification methods of the present invention
The experimental results show that: compared with the prior art, the method has the advantages of improving the classification precision, reducing the network training parameters, balancing the time cost and the classification effect.
In summary, the invention discloses a multispectral and panchromatic image classification method based on an attention mechanism, which mainly solves the problems of high classification time cost and poor multi-sensor fusion effect in the prior art. The implementation steps are as follows: 1) Selecting a training sample set and a testing sample set on the multispectral and panchromatic images, respectively; 2) Respectively extracting initial features; 3) Building a sharing-separating network, and respectively extracting deep features of multispectral and full-color images; 4) Carrying out self-adaptive fusion on the extracted deep features; 5) Inputting the fusion characteristics into a classifier, and constructing a loss function to train a network; 6) And inputting the test sample into a trained network, outputting class labels and finishing classification. The invention can utilize the lightweight network to extract and multispectral and panchromatic image characteristics, and effectively fuse, can better balance time cost and classification effect, and can be used for meteorological monitoring, environmental monitoring, urban planning, disaster prevention and reduction.
The above is only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited by this, and any modification made on the basis of the technical scheme according to the technical idea of the present invention falls within the protection scope of the claims of the present invention.
Claims (5)
1. A method for classifying multispectral and panchromatic images based on an attention mechanism, comprising the steps of:
firstly, respectively constructing a training sample set and a testing sample set which are composed of a multispectral sample and a full-color sample, and extracting initial characteristics of the multispectral sample and the full-color sample in the training sample set to respectively obtain initial characteristics of the multispectral sample and initial characteristics of the full-color sample with the same characteristic size;
constructing a lightweight sharing-separating network; respectively inputting the initial characteristics of the multispectral sample and the initial characteristics of the panchromatic sample in the training sample set obtained in the step one into a lightweight shared-separation network, and extracting deep characteristics to obtain the deep characteristics of the multispectral sample and the deep characteristics of the panchromatic sample respectively;
step three, respectively carrying out self-adaptive feature fusion on the deep features of the multispectral sample obtained in the step two and the deep features of the full-color sample, and outputting fusion features;
classifying the fusion characteristics obtained by the output of the step three, and training the network according to classification results; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in the training sample set; completing a multispectral image and panchromatic image classification method based on an attention mechanism;
in the second step, a lightweight shared-separate network is constructed, comprising the steps of:
constructing a shared sub-network consisting of 1 convolution layer and 1 downsampling layer, and initializing parameters of each layer;
respectively constructing a spectrum separation sub-network and a full-color separation sub-network; the spectrum separation sub-network and the full-color separation sub-network have the same network structure and both comprise a compression part and an excitation part;
the compression is to perform global average operation on input features to obtain global compression feature vectors of the input features, the excitation is to input the global compression feature vectors into 2 full-connection layers, the weights of all channels in the initial features are obtained through output, finally, the obtained weights are used for carrying out linear weighted summation on all channels of the input features, and the features of the linear weighted summation are used as the input of a next-layer network;
the compression is to perform global average operation on the input features to obtain global compression feature vectors z of the input features c Wherein the global averaging operation is according to the following formula:
wherein f c (i, j) represents the feature value of the input feature in the ith row and jth column, and H and W represent the height and width of the input feature, respectively;
excitation is to compress the feature vector z globally c Is input into 2 full connection layers, wherein the number of features of the first full connection layer is 4, and the second full connection layerThe number of the characteristics of the joint layer is 64, a 64-dimensional weight vector is output, the vector dimension is the weight of each channel in the initial characteristics of the multispectral sample and the initial characteristics of the panchromatic sample, finally, the obtained weight is used for carrying out linear weighted summation on each channel of the input characteristics, and the characteristics of the linear weighted summation are used as the input of a next layer network;
in the second step, the specific steps for deep feature extraction are as follows:
firstly, respectively inputting initial characteristics of a multispectral sample and initial characteristics of a panchromatic sample into an established shared sub-network, and respectively outputting the shared characteristics of the multispectral sample and the shared characteristics of the panchromatic sample through forward propagation;
then, the sharing characteristics of the obtained multispectral sample are input into a built spectrum separation sub-network, and the separation characteristics of the multispectral sample are output through forward propagation; inputting the shared characteristics of the obtained full-color sample into a built full-color separation sub-network, and outputting the separation characteristics of the full-color sample through forward propagation;
finally, taking the separation characteristics of the output multispectral sample and the separation characteristics of the panchromatic sample as the input of the established shared sub-network, and circulating the steps for three times to respectively obtain the deep characteristics of the multispectral sample and the deep characteristics of the panchromatic sample;
in the third step, the specific steps for performing adaptive feature fusion are as follows:
adding the deep features of the obtained multispectral sample and the deep features of the panchromatic sample to obtain initial fusion features;
performing global compression and excitation on the obtained initial fusion characteristics to respectively obtain fusion weights of the multispectral samples and fusion weights of the panchromatic samples; carrying out linear weighted summation on the fusion weight of the obtained multispectral sample and the deep features of the obtained multispectral sample according to the channel to obtain the linear weighted summation features of the multispectral sample; carrying out linear weighted summation on the fusion weight of the obtained panchromatic sample and the deep layer characteristic of the panchromatic sample according to the channel to obtain the linear weighted summation characteristic of the panchromatic sample; the linear weighted sum feature of the multispectral sample and the linear weighted sum feature of the panchromatic sample are added to output an adaptive fusion feature.
2. The method of classifying multispectral and panchromatic images based on an attention mechanism of claim 1, wherein in step one, a training sample set and a test sample set are constructed from multispectral samples and panchromatic samples, and initial feature extraction is performed on multispectral samples and panchromatic samples in the training sample set, comprising the steps of:
according to each labeled pixel (some pixels are not labeled) on the multispectral image, respectively extracting multispectral samples and full-color samples in pairs on the multispectral image and the full-color image, and forming sample pairs of the labeled pixels; forming a sample set according to the obtained sample pairs; selecting part of sample pairs in the sample set to form a training sample set, and selecting the rest sample pairs in the sample set to form a test sample set;
constructing a multispectral image initial feature extraction network consisting of 2 convolution layers and 2 downsampling layers, and initializing parameters of each layer; taking a multispectral sample in the training sample set as input of a multispectral image initial feature extraction network, and obtaining initial features of the multispectral sample through forward propagation;
constructing a full-color image initial feature extraction network consisting of 3 convolution layers and 3 downsampling layers, and initializing parameters of each layer; and taking the full-color samples in the training sample set as the input of the full-color image initial feature extraction network, and obtaining initial features of the full-color samples through forward propagation.
3. The attention mechanism based multispectral and panchromatic image classification method of claim 2 wherein the specific network design of the multispectral sample initial feature extraction network is: the convolution kernel size of the first convolution layer is 3 multiplied by 3, the number of output features is 32, the convolution kernel size of the second convolution layer is 3 multiplied by 3, the number of output features is 64, the downsampling layers all adopt the maximum downsampling method, and the sampling kernel is 2 multiplied by 2.
4. The attention mechanism based multispectral and panchromatic image classification method of claim 2 wherein the specific network design of the panchromatic sample initial feature extraction network is: the convolution kernel size of the first convolution layer is 3×3, the number of output features is 16, the convolution kernel size of the second convolution layer is 3×3, the number of output features is 32, the convolution kernel size of the second convolution layer is 3×3, the number of output features is 64, the downsampling layers all adopt the maximum downsampling method, and the sampling kernel is 2×2.
5. The method of classifying multispectral and panchromatic images based on an attention mechanism of claim 1, wherein in step four, the network is trained according to the classification result; based on the trained network, obtaining classification results of multispectral samples and panchromatic samples in the training sample set, comprising the following steps:
inputting the obtained fusion characteristics into a softmax classifier for classification, and outputting the probability of each sample pair belonging to each class in a training sample set;
constructing a cross entropy loss function according to the output probability and the class mark of the sample pair; adopting a gradient descent method to counter-propagate the cross entropy loss function obtained by construction, and training all networks involved in the first to fourth steps; repeating the network training in the first step to the fourth step until reaching the preset iteration step, and stopping iteration;
taking the multispectral sample and the panchromatic sample in the test sample set as the input of the step one, executing the step one to the step four, outputting the probability of each sample pair in the test sample set belonging to each class, and setting the position corresponding to the maximum value of the probability value as the class mark of the sample pair in the test sample set to finish classification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110542410.9A CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110542410.9A CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113240017A CN113240017A (en) | 2021-08-10 |
| CN113240017B true CN113240017B (en) | 2023-09-12 |
Family
ID=77135131
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110542410.9A Active CN113240017B (en) | 2021-05-18 | 2021-05-18 | Multispectral and panchromatic image classification method based on attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113240017B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758361B (en) * | 2023-08-22 | 2023-10-27 | 中国铁路设计集团有限公司 | Engineering rock group remote sensing classification method and system based on spatial and spectral joint characteristics |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830330A (en) * | 2018-06-22 | 2018-11-16 | 西安电子科技大学 | Classification of Multispectral Images method based on self-adaptive features fusion residual error net |
| CN111199214A (en) * | 2020-01-04 | 2020-05-26 | 西安电子科技大学 | Residual error network multispectral image ground feature classification method |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
-
2021
- 2021-05-18 CN CN202110542410.9A patent/CN113240017B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830330A (en) * | 2018-06-22 | 2018-11-16 | 西安电子科技大学 | Classification of Multispectral Images method based on self-adaptive features fusion residual error net |
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN111199214A (en) * | 2020-01-04 | 2020-05-26 | 西安电子科技大学 | Residual error network multispectral image ground feature classification method |
Non-Patent Citations (1)
| Title |
|---|
| 刘启超 ; 肖亮 ; 刘芳 ; 徐金环 ; .SSCDenseNet:一种空-谱卷积稠密网络的高光谱图像分类算法.电子学报.2020,(第04期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113240017A (en) | 2021-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109993220B (en) | Multi-source remote sensing image classification method based on double-path attention fusion neural network | |
| CN108510532B (en) | Optical and SAR image registration method based on deep convolution GAN | |
| CN110084159B (en) | Hyperspectral image classification method based on combined multistage spatial spectrum information CNN | |
| CN111123257B (en) | Radar moving target multi-frame joint detection method based on graph space-time network | |
| CN107330437B (en) | Feature extraction method based on convolutional neural network target real-time detection model | |
| CN108564606B (en) | Heterogeneous image block matching method based on image conversion | |
| CN109598306B (en) | Hyperspectral image classification method based on SRCM and convolutional neural network | |
| CN111539316A (en) | High-resolution remote sensing image change detection method based on double attention twin network | |
| CN112434745B (en) | Occlusion target detection and identification method based on multi-source cognitive fusion | |
| CN108960404B (en) | Image-based crowd counting method and device | |
| CN111353531B (en) | Hyperspectral image classification method based on singular value decomposition and spatial spectral domain attention mechanism | |
| CN110082821B (en) | Label-frame-free microseism signal detection method and device | |
| CN107944483B (en) | Multispectral image classification method based on dual-channel DCGAN and feature fusion | |
| CN105825511A (en) | Image background definition detection method based on deep learning | |
| CN109446894B (en) | Multispectral image change detection method based on probability segmentation and Gaussian mixture clustering | |
| CN107992891A (en) | Based on spectrum vector analysis multi-spectral remote sensing image change detecting method | |
| CN111738268A (en) | Semantic segmentation method and system for high-resolution remote sensing image based on random block | |
| CN107967474A (en) | A kind of sea-surface target conspicuousness detection method based on convolutional neural networks | |
| CN113569788B (en) | Building semantic segmentation network model training method, system and application method | |
| CN115240072B (en) | Hyperspectral multi-class change detection method based on multidirectional multi-scale spectrum-space residual convolution neural network | |
| CN111783555A (en) | Deep learning satellite data cloud detection algorithm using unified samples | |
| CN113240017B (en) | Multispectral and panchromatic image classification method based on attention mechanism | |
| CN113902646A (en) | Remote sensing image pan-sharpening method based on depth layer feature weighted fusion network | |
| CN107437091A (en) | Multilayer limits the positive and negative class change detecting method of SAR image of Boltzmann machine | |
| CN113362322A (en) | Distinguishing auxiliary and multi-mode weighted fusion salient object detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |