CN111814875B - Ship sample expansion method in infrared image based on pattern generation countermeasure network - Google Patents
Ship sample expansion method in infrared image based on pattern generation countermeasure network Download PDFInfo
- Publication number
- CN111814875B CN111814875B CN202010650897.8A CN202010650897A CN111814875B CN 111814875 B CN111814875 B CN 111814875B CN 202010650897 A CN202010650897 A CN 202010650897A CN 111814875 B CN111814875 B CN 111814875B
- Authority
- CN
- China
- Prior art keywords
- layer
- convolution
- network
- setting
- pattern
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a ship sample expansion method in an infrared image based on a pattern generation countermeasure network. The method mainly solves the problems of poor reality caused by complex simulation modeling of an infrared image generated in the prior art, high acquisition difficulty of a visible light-infrared image photoelectric conversion training sample and lack of diversity of the expanded infrared image caused by small number of training sets, and comprises the following steps: (1) selecting a real shot infrared image to form a training set; (2) building a generator network; (3) constructing a discriminator network; (4) build style generation antagonism network; (5) training a arbiter network; (6) a training generator network; (7) training patterns to generate an countermeasure network; (8) And outputting an infrared image sample by using the trained generator network, and completing expansion of the infrared image sample. The invention can generate a large amount of infrared ship samples, and effectively improves the sense of reality and diversity of the expanded samples.
Description
Technical Field
The invention belongs to the technical field of image processing, and further relates to a ship sample expansion method in an infrared image based on style generation countermeasure network style GAN (style-based Generative Adversarial Network) in the field of deep learning. The invention can expand the ship sample in the infrared image so as to provide a rich data set for the training of the infrared target detection and identification algorithm.
Background
The infrared imaging technology is commonly used for detecting, identifying and tracking targets due to the characteristics of strong target detection capability, strong anti-interference capability and the like. Because the infrared characteristics of the targets are complex, the change along with the temperature condition is obvious, the detection and the identification of the infrared targets are difficult to realize, and a large number of infrared images are generally required to train and learn the detection and identification algorithm in order to improve the capability of detecting and identifying the infrared targets. However, the infrared image is usually obtained by photographing the target scene through a thermal imager, so that the means for obtaining infrared images of some specific targets is limited, which results in serious shortages of the number of infrared samples. At present, researchers have proposed a sample expansion method based on infrared simulation, which generates an infrared simulation image by modeling a target area scene and performing radiation calculation, however, the modeling needs target information to be limited, so that the generated infrared image has low sense of reality and limited quantity.
An infrared simulation-based target sample generation method is proposed in patent literature (2018105060882, NC 110162812A) applied by Beijing institute of mechanical and electrical engineering. The method comprises the steps of modeling a target area scene, completing area division and material assignment of the target area scene, generating a material image of the target area scene, constructing a three-dimensional scene of the target area scene according to the material image, a model of the target area scene and data of the target area scene, setting external environment conditions of the target area scene, calculating a temperature field, infrared radiation and atmospheric transmittance of the target area scene to generate an infrared simulation scene under zero viewing distance of the target area scene, and generating an infrared simulation target sample according to viewing distance parameters and imaging system parameters on the basis of the infrared simulation scene of the target area. The method has the following defects: the radiation calculation needs to be performed for the external environment of the target area scene. Although the radiation calculation can solve the problem of infrared simulation image generation, in a complex external environment, the factors to be considered are numerous, the modeling process is complex, the simulation result is poor in sense of reality, and the infrared sample of the required target scene cannot be accurately obtained.
Chen Foji et al in its published paper "based on generating infrared image data enhancement against a network" (computer application, 2020:1-7) propose a sample augmentation method for generating infrared images based on visible light images of a photoelectric image conversion model. The method firstly constructs paired data sets through the existing visible light image and infrared image data, and then constructs a generator and a discriminator for generating an countermeasure network based on a convolutional neural network. The paired data sets are then used to train the generation of the countermeasure network until an equilibrium state is reached between the generator and the arbiter. Finally, the method uses the trained generator that generates the countermeasure network to transform the visible light image from the visible light domain to the infrared domain, thereby completing sample expansion of the infrared image. The method has the following defects: the paired visible light images and the corresponding infrared images are required to be collected as data sets to train the generation countermeasure network, the paired visible light-infrared image data acquisition difficulty is high, the number of training sets is small, and the generated infrared image samples lack diversity; when the infrared image sample is expanded, a large amount of extra visible light images still need to be collected as input for photoelectric conversion, the expansion mode is complex, and the number of the expanded samples is limited.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a ship sample expansion method in an infrared image based on a pattern generation countermeasure network, and aims to solve the problems that the simulation process is complex when an infrared image sample is expanded, the reality of the expanded infrared sample is poor, the acquisition difficulty of a training set of a visible light-to-infrared photoelectric conversion method is high, the expanded infrared sample lacks diversity and the expansion quantity is limited.
The idea of the invention for achieving the above purpose is: constructing a pattern with random feature vectors as input to generate an countermeasure network, using real-time infrared image ship samples as training sets, and updating parameters of the network in an alternating training mode of a discriminator and a generator. And under the condition that the loss value updated to the generator network is smaller than 20 and the average value of the loss value of the discriminator network is smaller than 10, saving the weight parameters of each layer of the trained generator network. And finally, inputting the randomly generated feature vectors into a trained generator network for calculation to obtain a generated infrared image, and adding the generated infrared image into a training set to complete the expansion of the ship sample of the infrared image.
The specific steps of the invention include the following steps:
(1) Acquiring a training set:
selecting at least 2000 infrared images to be photographed, wherein each image comprises a ship target; scaling and cutting the size of each image into 256 multiplied by 256 to form a training set;
(2) Constructing a generator network:
(2a) A generator network is built, and the structure of the generator network is as follows: constant matrix layer- & gt 1 st noise modulation layer- & gt 1 st self-adaptive pattern modulation layer- & gt 1 st deconvolution layer- & gt 2 nd noise modulation layer- & gt activation function layer- & gt 2 nd self-adaptive pattern modulation layer- & gt pattern convolution block combination- & gt 2 nd deconvolution layer- & gt output layer;
the adaptive pattern modulation layer structure is as follows: feature vector input layer- & gt normalization layer- & gt 1 st full connection layer- & gt 2 nd full connection layer- & gt 3 rd full connection layer- & gt 4 th full connection layer- & gt 5 th full connection layer- & gt 6 th full connection layer- & gt 7 th full connection layer- & gt 8 th full connection layer- & gt scaling translation conversion layer- & gt output layer;
the pattern convolution block combination is formed by cascading 6 pattern convolution blocks with the same structure, and the structure of each pattern convolution block is as follows: deconvolution layer 1 → noise modulation layer 1 → active function layer 1 → adaptive pattern modulation layer 1 → deconvolution layer 2 → noise modulation layer 2 → active function layer 2 → adaptive pattern modulation layer 2;
the normalization layer is realized by adopting an example normalization function; the activation function layers are all realized by adopting a LeakyReLU function;
(2b) Setting each layer of parameters of the generator network:
setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer to be 3 multiplied by 3, setting the number of the convolution kernels to be 512 and 3 respectively, setting the convolution step length to be 1, and initializing the weight to be a random value meeting normal distribution with the standard deviation of 0.02;
setting the slopes of the leak ReLU functions of the activation function layer to 0.2;
setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer of the 1 st pattern convolution block to the 3 rd pattern convolution block in the pattern convolution block combination to be 3 multiplied by 3, setting the number of the convolution kernels to be 512, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 4 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 256, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 5 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 128, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 6 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 64, and setting the convolution step length to be 1; initializing the weight of each convolution kernel of each deconvolution layer in the 1 st to 6 th pattern convolution blocks to a random value meeting normal distribution with a standard deviation of 0.02; setting the slope of each leak ReLU function of the activation function layer in the 1 st to 6 th pattern convolution blocks to 0.2;
setting the number of neurons of the 1 st to 8 th full-connection layers in the adaptive pattern modulation layer to be 512, and initializing weights to be random values meeting normal distribution with standard deviation of 0.02;
(3) Constructing a discriminator network:
(3a) Constructing a discriminator network, and sequentially constructing the following structures: input layer, convolution layer, activation function layer, convolution block combination, full connection layer and output layer;
the convolution block combination is formed by cascading 6 convolution blocks with the same structure, and the structure of each convolution block is as follows: convolution layer 1- & gt activation function layer 1- & gt convolution layer 2- & gt pooling layer 1- & gt activation function layer 2;
the activation function layers are all realized by adopting a Leaky ReLU function, and the pooling layer is realized by adopting global average pooling;
(3b) Setting each layer of parameters of a discriminator network:
the size of convolution kernels of the convolution layers is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 512 respectively, the convolution step length is set to be 1, and the weight is initialized to be a random value meeting normal distribution with standard deviation of 0.02;
setting the slope of each leak ReLU function of the activation function layer to 0.2;
setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 1 st convolution block and the 2 nd convolution block in the convolution block combination to be 3 multiplied by 3, setting the number of the convolution kernels to be 512, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 3 rd convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 256, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 4 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 128, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 5 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 64, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 6 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 32, and setting the convolution step length to be 1; initializing each convolution kernel weight of each convolution layer in the 1 st to 6 th convolution blocks to a random value satisfying a normal distribution with a standard deviation of 0.02; setting the slope of the leak ReLU function of each activation function layer in the 1 st to 6 th convolution blocks to 0.2;
setting the number of neurons of the full-connection layer as 1, and initializing the weight to a random value meeting normal distribution with a standard deviation of 0.02;
(4) Build style generation antagonism network:
cascading the generator network and the discriminator network to form a pattern to generate an countermeasure network;
(5) Training a arbiter network:
fixing the current weight parameters of a generator network, inputting the randomly generated feature vectors into the generator network, outputting random infrared images, respectively inputting the generated random infrared images and infrared images in a training set into a discriminator network, respectively outputting corresponding evaluation scores after evaluating the sequentially input infrared images by the discriminator network, and calculating the loss value of the discriminator network by utilizing the evaluation scores of the discriminator network and a loss function of the discriminator network;
calculating the gradient of each convolution kernel of each convolution layer of the discriminator network and the gradient of the full connection layer by using a loss value and gradient descent method of the discriminator network;
updating the weight of each convolution kernel of each convolution layer and each convolution kernel of the deconvolution layer of the identifier network and the weight of the full connection layer by using an Adam optimizer with a learning rate of 0.005 by using the gradient of each convolution kernel of each convolution layer of the identifier network and the gradient of the full connection layer;
(6) Training generator network:
fixing the current weight parameters of the discriminator network, inputting the randomly generated feature vectors into the generator network, outputting infrared images, inputting the generated random infrared images into the discriminator network, evaluating the input generated images by the discriminator network, outputting evaluation scores, and calculating a generator network loss value by using the evaluation scores of the discriminator and a generator network loss function;
calculating the gradient of each convolution kernel of each deconvolution layer of the generator network, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the self-adaptive pattern modulation layer by using a loss value and gradient descent method of the generator network;
iteratively updating the weight of each convolution kernel, the weight of the full connection layer, the weight of the modulation noise layer and the weight of the adaptive pattern modulation layer of each deconvolution layer of the generator network by using an Adam optimizer with the learning rate of 0.005 by using the gradient of each convolution kernel, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the adaptive pattern modulation layer of each deconvolution layer of the generator network;
(7) Training patterns are generated against the network:
repeating the steps (5) and (6), alternately training the discriminator network and the generator network until the loss value of the generator network obtained by the current iteration is smaller than 20 and the average value of the loss values of the discriminator network is smaller than 10, obtaining the trained generator network weight, and storing the trained pattern to generate the weight of each convolution kernel of each deconvolution layer, the weight of the full-connection layer, the weight of the modulation noise layer and the weight of the self-adaptive pattern modulation layer of each deconvolution layer of the counter-generation network;
(8) Expanding an infrared image sample:
and inputting the randomly generated feature vectors into a trained generator network for calculation, outputting an infrared image containing ship samples, adding the infrared image into a training set, and completing expansion of the infrared image ship samples.
Compared with the prior art, the invention has the following advantages:
first, the invention adopts the pattern convolution block, uses the random characteristic vector as the input of the pattern convolution block to modulate the convolution layer result, generates the infrared image ship samples with various ship target postures and sea surface relief, overcomes the problem of poor infrared image diversity generated by the prior art, and improves the diversity of the infrared image ship samples when expanding the infrared image ship samples.
Secondly, because the invention adopts the pattern generation countermeasure network, takes the real shot infrared image ship sample as the training set, adopts the mode of alternately training the discriminator and the generator to train the network, the infrared characteristic of the output infrared image ship sample is more similar to the real shot image, the problem of poor sense of reality when the infrared image is generated by simulation in the prior art is overcome, and the sense of reality of the infrared image sample is improved when the infrared image sample is expanded.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a schematic diagram of the architecture of a generator network of the present invention;
FIG. 3 is a schematic diagram of a discriminator network according to the invention;
FIG. 4 is a schematic diagram of a pattern generation countermeasure network according to the present invention;
fig. 5 is a simulation diagram of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The implementation steps of the present invention will be described in further detail with reference to fig. 1.
Step 1, obtaining a training set.
Selecting at least 2000 infrared images to be photographed, wherein each image comprises a ship target; the size of each image is scaled to 256×256, constituting a training set.
And 2, constructing a generator network.
A generator network is constructed and each layer parameter of the network is set, and the generator network of the countermeasure network is generated as a pattern.
The structure of the pattern generation countermeasure network constructed by the present invention will be described in further detail with reference to fig. 2.
The structure of the generator network is as follows: constant matrix layer- & gt 1 st noise modulation layer- & gt 1 st self-adaptive pattern modulation layer- & gt 1 st deconvolution layer- & gt 2 nd noise modulation layer- & gt activation function layer- & gt 2 nd self-adaptive pattern modulation layer- & gt pattern convolution block combination- & gt 2 nd deconvolution layer- & gt output layer.
The adaptive pattern modulation layer structure is as follows: feature vector input layer, normalization layer, full connection layer, scaling translation conversion layer and output layer.
The pattern convolution block combination is formed by cascading 6 pattern convolution blocks with the same structure, and the structure of each pattern convolution block is as follows: deconvolution layer 1 → noise modulation layer 1 → active function layer 1 → adaptive pattern modulation layer 1 → deconvolution layer 2 → noise modulation layer 2 → active function layer 2 → adaptive pattern modulation layer 2.
The normalization layer is realized by adopting an example normalization function. The activation function layers are all realized by adopting a LeakyReLU function.
The ellipses in fig. 2 represent pattern 2-6 convolutions, ten deconvolution layers, ten noise modulation layers, ten activation function layers, and ten scaling translation transform layers. The random noise in fig. 2 represents the input of the noise modulation layer. The unidirectional arrows in fig. 2 represent the characteristic connection relationships.
Each layer of parameters of the generator network is set as follows:
the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 512 and 3 respectively, the convolution step length is set to be 1, and the weight is initialized to be a random value meeting normal distribution with the standard deviation of 0.02.
The slopes of the leak ReLU functions of the activation function layer are all set to 0.2.
The size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer of the 1 st pattern convolution block to the 3 rd pattern convolution block in the pattern convolution block combination is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 512, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 4 th pattern convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 256, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 5 th pattern convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 128, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 6 th pattern convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 64, and the convolution step length is set to be 1. The weight of each convolution kernel of each deconvolution layer in the 1 st to 6 th pattern convolution blocks is initialized to a random value satisfying a normal distribution with a standard deviation of 0.02. The slope of each leak ReLU function of the activation function layer in the 1 st to 6 th pattern convolution blocks is set to 0.2.
And initializing the weights of the full-connection layers in the adaptive pattern modulation layer to random values meeting normal distribution with standard deviation of 0.02.
And 3, constructing a discriminator network.
Constructing a discriminator network, setting each layer parameter of the network, and generating the discriminator network of the countermeasure network by using the parameters as patterns.
The construction of the pattern generation countermeasure network according to the present invention will be described in further detail with reference to fig. 3.
The network structure of the discriminator is as follows: input layer, convolution layer, activation function layer, convolution block combination, full connection layer and output layer.
The convolution block combination is formed by cascading 6 convolution blocks with the same structure, and the structure of each convolution block is as follows: convolution layer 1- & gt activation function layer 1- & gt convolution layer 2- & gt pooling layer 1- & gt activation function layer 2.
The activation function layers are all realized by adopting a Leaky ReLU function, and the pooling layer is realized by adopting global average pooling.
The ellipses in fig. 3 represent the 2 nd to 6 th convolution blocks, ten convolution layers, ten activation function layers, and five pooling layers, and the unidirectional arrows in fig. 3 represent the characteristic connection relationships.
Each layer of parameters of the arbiter network is set as follows:
the convolution kernel sizes of the convolution layers are all set to be 3 multiplied by 3, the number of the convolution kernels is respectively set to be 512, the convolution step sizes are all set to be 1, and the weight is initialized to be a random value meeting normal distribution with the standard deviation of 0.02.
The slope of each leak ReLU function of the activation function layer is set to 0.2.
The size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 1 st convolution block and the 2 nd convolution block in the convolution block combination is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 512, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 3 rd convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 256, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 4 th convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 128, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 5 th convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 64, and the convolution step length is set to be 1. The size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 6 th convolution block is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 32, and the convolution step length is set to be 1. Each convolution kernel weight of each convolution layer in the 1 st to 6 th convolution blocks is initialized to a random value satisfying a normal distribution with a standard deviation of 0.02. The slope of the leak ReLU function of each activation function layer in the 1 st to 6 th convolution blocks is set to 0.2.
The weights of the fully connected layers are initialized to random values satisfying a normal distribution with a standard deviation of 0.02.
And 4, building a style generation countermeasure network.
The structure of the pattern generation countermeasure network constructed by the present invention will be described in further detail with reference to fig. 4.
The pattern generation countermeasure network is composed of a cascade of a generator network and a discriminator network.
The generator in fig. 4 represents the generator network described in step 2, the arbiter in fig. 4 represents the arbiter network described in step 3, and the unidirectional arrow in fig. 4 represents the characteristic connection relationship.
And 5, training a discriminator network.
Fixing the current weight parameters of a generator network, inputting a randomly generated feature vector into the generator network, outputting a random infrared image, respectively inputting the generated random infrared image and an infrared image in a training set into a discriminator network, respectively outputting corresponding evaluation scores after evaluating the sequentially input infrared images by the discriminator network, and calculating a loss value of the discriminator network by utilizing the evaluation scores of the discriminator network and a loss function of the discriminator network, wherein the loss function calculation formula of the discriminator network is as follows:
wherein L is D Representing a loss function of the arbiter network, E [. Cndot.]Representing a desired operation, D (·) representing the output of a graph generating a network of discriminators in the countermeasure network, G (·) representing the output of a network of generators in the graph generating the countermeasure network, z representing randomly generated feature vectors, x representing 16 Zhang Shi beats of infrared images taken from the training set, γ representing square term coefficients, λ representing constraint term coefficients,the constraint term infrared image obtained by corresponding fusion of 16 infrared images output by the generator and 16 Zhang Shi infrared images in the training set according to random proportion is represented 2 Representing a 2-norm operation, and v represents a derivative operation.
And calculating the gradient of each convolution kernel of each convolution layer of the discriminator network and the gradient of the full connection layer by using a loss value and gradient descent method of the discriminator network.
And updating the weight of each convolution kernel of each convolution layer and each convolution kernel of the deconvolution layer of the discriminant network and the weight of the full connection layer by using an Adam optimizer with a learning rate of 0.005 by using the gradient of each convolution kernel of each convolution layer of the discriminant network and the gradient of the full connection layer.
And 6, training a generator network.
Fixing the current weight parameters of a discriminator network, inputting a randomly generated feature vector into a generator network, outputting an infrared image, inputting the generated random infrared image into the discriminator network, evaluating the input generated image by the discriminator network, outputting an evaluation score, and calculating a generator network loss value by using the evaluation score of the discriminator and a loss function of the generator network, wherein the calculation formula of the generator network is as follows:
L G ＝-E[D(G(z))]
wherein L is G Representing a loss function of the generator network, E [. Cndot.]Representing a desired operation, D (-) represents the output of a discriminator network in the pattern generation countermeasure network, G (-) represents the output of a generator network in the pattern generation countermeasure network, and z represents a randomly generated feature vector.
And calculating the gradient of each convolution kernel of each deconvolution layer of the generator network, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the self-adaptive pattern modulation layer by using the loss value and the gradient descent method of the generator network.
And iteratively updating the weight of each convolution kernel of each deconvolution layer of the generator network, the weight of the full connection layer, the weight of the modulation noise layer and the weight of the adaptive pattern modulation layer by using an Adam optimizer with the learning rate of 0.005 by using the gradient of each convolution kernel of each deconvolution layer of the generator network, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the adaptive pattern modulation layer.
And 7, generating a training pattern to combat the network.
And (5) repeating the step (5) and the step (6) to train the discriminator network and the generator network alternately until the loss value of the generator network obtained by the current iteration is smaller than 20 and the average value of the loss value of the discriminator network is smaller than 10, obtaining the trained generator network weight, and storing the trained pattern to generate the weight of each convolution kernel of each deconvolution layer of the counter-generator network in the network, the weight of the full connection layer, the weight of the modulation noise layer and the weight of the self-adaptive pattern modulation layer.
And 8, expanding the infrared image sample.
And inputting the randomly generated feature vectors into a trained generator network for calculation, outputting an infrared image containing ship samples, adding the infrared image into a training set, and completing expansion of the infrared image ship samples.
The effects of the present invention will be further described with reference to simulation experiments.
1. Simulation conditions:
the simulation experiment of the invention is carried out under a single GPU model NVIDIA RTX 2080, a hardware environment running 128GB of memory and a software environment of PyTorrch1.1.0.
2. Simulation content and result analysis:
the simulation experiment of the invention uses the method of the invention and a prior art to expand 2000 infrared image ship samples.
The prior art is as follows: an infrared simulation-based target sample generation method is proposed in patent literature (2018105060882, NC 110162812A) applied by Beijing mechanical and electrical engineering research institute.
Fig. 5 is a simulation diagram of the present invention, wherein fig. 5 (a) is a diagram of 1 real shot infrared image ship sample randomly selected from the training set used in the simulation experiment of the present invention. 5 (b) is a ship expansion sample diagram of 1 infrared image generated by adopting the prior art, and fig. 5 (c) is a ship expansion sample diagram of 9 infrared images generated by adopting the method of the invention.
Comparing fig. 5 (a), fig. 5 (b) and fig. 5 (c) of fig. 5, it can be seen that the infrared image ship sample generated by the prior art in fig. 5 (b) is significantly different from the real-time infrared image ship sample in fig. 5 (a). Compared with the real-time infrared image ship sample in FIG. 5 (a), the infrared image ship sample generated by the method in FIG. 5 (c) has high consistency in the aspects of infrared texture detail characteristics, radiation characteristics of targets and the like.
Comparing the 9 infrared image ship expansion samples generated by the method of the invention with the fig. 5 (c), the infrared image ship expansion samples generated by the method of the invention can be seen to have ship targets with various sizes and angles and sea surface bright bands with various forms, and the infrared image ship samples generated by the method of the invention can be seen to have strong diversity.
Therefore, the method of the invention overcomes the problems in the prior art, improves the sense of realism of the infrared image ship expansion sample, and increases the diversity of the infrared image ship sample.
Claims (3)
1. A ship sample expansion method in infrared image based on pattern generation countermeasure network is characterized in that a pattern of alternately training a discriminator and a generator is adopted to generate the countermeasure network, each layer of the generator is modulated by random feature vectors, the random feature vectors output the infrared image ship sample through the generator for expansion, and the method comprises the following specific steps:
(1) Acquiring a training set:
selecting at least 2000 infrared images to be photographed, wherein each image comprises a ship target; scaling and cutting the size of each image into 256 multiplied by 256 to form a training set;
(2) Constructing a generator network:
(2a) A generator network is built, and the structure of the generator network is as follows: constant matrix layer- & gt 1 st noise modulation layer- & gt 1 st self-adaptive pattern modulation layer- & gt 1 st deconvolution layer- & gt 2 nd noise modulation layer- & gt activation function layer- & gt 2 nd self-adaptive pattern modulation layer- & gt pattern convolution block combination- & gt 2 nd deconvolution layer- & gt output layer;
the adaptive pattern modulation layer structure is as follows: feature vector input layer- & gt normalization layer- & gt 1 st full connection layer- & gt 2 nd full connection layer- & gt 3 rd full connection layer- & gt 4 th full connection layer- & gt 5 th full connection layer- & gt 6 th full connection layer- & gt 7 th full connection layer- & gt 8 th full connection layer- & gt scaling translation conversion layer- & gt output layer;
the pattern convolution block combination is formed by cascading 6 pattern convolution blocks with the same structure, and the structure of each pattern convolution block is as follows: deconvolution layer 1 → noise modulation layer 1 → active function layer 1 → adaptive pattern modulation layer 1 → deconvolution layer 2 → noise modulation layer 2 → active function layer 2 → adaptive pattern modulation layer 2;
the normalization layer is realized by adopting an example normalization function; the method comprises the steps of carrying out a first treatment on the surface of the The activation function layers are all realized by adopting a Leaky ReLU function;
(2b) Setting each layer of parameters of the generator network:
setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer to be 3 multiplied by 3, setting the number of the convolution kernels to be 512 and 3 respectively, setting the convolution step length to be 1, and initializing the weight to be a random value meeting normal distribution with the standard deviation of 0.02;
setting the slopes of the leak ReLU functions of the activation function layer to 0.2;
setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer of the 1 st pattern convolution block to the 3 rd pattern convolution block in the pattern convolution block combination to be 3 multiplied by 3, setting the number of the convolution kernels to be 512, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 4 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 256, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 5 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 128, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st deconvolution layer and the 2 nd deconvolution layer in the 6 th pattern convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 64, and setting the convolution step length to be 1; initializing the weight of each convolution kernel of each deconvolution layer in the 1 st to 6 th pattern convolution blocks to a random value meeting normal distribution with a standard deviation of 0.02; setting the slope of each leak ReLU function of the activation function layer in the 1 st to 6 th pattern convolution blocks to 0.2;
setting the number of neurons of the 1 st to 8 th full-connection layers in the adaptive pattern modulation layer to be 512, and initializing weights to be random values meeting normal distribution with standard deviation of 0.02;
(3) Constructing a discriminator network:
(3a) Constructing a discriminator network, and sequentially constructing the following structures: input layer, convolution layer, activation function layer, convolution block combination, full connection layer and output layer;
the convolution block combination is formed by cascading 6 convolution blocks with the same structure, and the structure of each convolution block is as follows: convolution layer 1- & gt activation function layer 1- & gt convolution layer 2- & gt pooling layer 1- & gt activation function layer 2;
the activation function layers are all realized by adopting a Leaky ReLU function, and the pooling layer is realized by adopting global average pooling;
(3b) Setting each layer of parameters of a discriminator network:
the size of convolution kernels of the convolution layers is set to be 3 multiplied by 3, the number of the convolution kernels is set to be 512 respectively, the convolution step length is set to be 1, and the weight is initialized to be a random value meeting normal distribution with standard deviation of 0.02;
setting the slope of each leak ReLU function of the activation function layer to 0.2;
setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 1 st convolution block and the 2 nd convolution block in the convolution block combination to be 3 multiplied by 3, setting the number of the convolution kernels to be 512, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 3 rd convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 256, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 4 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 128, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 5 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 64, and setting the convolution step length to be 1; setting the size of each convolution kernel of the 1 st convolution layer and the 2 nd convolution layer in the 6 th convolution block to be 3 multiplied by 3, setting the number of the convolution kernels to be 32, and setting the convolution step length to be 1; initializing each convolution kernel weight of each convolution layer in the 1 st to 6 th convolution blocks to a random value satisfying a normal distribution with a standard deviation of 0.02; setting the slope of the leak ReLU function of each activation function layer in the 1 st to 6 th convolution blocks to 0.2;
setting the number of neurons of the full-connection layer as 1, and initializing the weight to a random value meeting normal distribution with a standard deviation of 0.02;
(4) Build style generation antagonism network:
cascading the generator network and the discriminator network to form a pattern to generate an countermeasure network;
(5) Training generator network:
fixing the current weight parameters of a generator network, inputting the randomly generated feature vectors into the generator network, outputting random infrared images, respectively inputting the generated random infrared images and infrared images in a training set into a discriminator network, respectively outputting corresponding evaluation scores after evaluating the sequentially input infrared images by the discriminator network, and calculating the loss value of the discriminator network by utilizing the evaluation scores of the discriminator network and a loss function of the discriminator network;
calculating the gradient of each convolution kernel of each convolution layer of the discriminator network and the gradient of the full connection layer by using a loss value and gradient descent method of the discriminator network;
updating the weight of each convolution kernel of each convolution layer and each convolution kernel of the deconvolution layer of the identifier network and the weight of the full connection layer by using an Adam optimizer with a learning rate of 0.005 by using the gradient of each convolution kernel of each convolution layer of the identifier network and the gradient of the full connection layer;
(6) Training a arbiter network:
fixing the current weight parameters of the discriminator network, inputting the randomly generated feature vectors into the generator network, outputting infrared images, inputting the generated random infrared images into the discriminator network, evaluating the input generated images by the discriminator network, outputting evaluation scores, and calculating a generator network loss value by using the evaluation scores of the discriminator and a generator network loss function;
calculating the gradient of each convolution kernel of each deconvolution layer of the generator network, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the self-adaptive pattern modulation layer by using a loss value and gradient descent method of the generator network;
iteratively updating the weight of each convolution kernel, the weight of the full connection layer, the weight of the modulation noise layer and the weight of the adaptive pattern modulation layer of each deconvolution layer of the generator network by using an Adam optimizer with the learning rate of 0.005 by using the gradient of each convolution kernel, the gradient of the full connection layer, the gradient of the modulation noise layer and the gradient of the adaptive pattern modulation layer of each deconvolution layer of the generator network;
(7) Training patterns are generated against the network:
repeating the steps (5) and (6), alternately training the discriminator network and the generator network until the loss value of the generator network obtained by the current iteration is smaller than 20 and the average value of the loss values of the discriminator network is smaller than 10, obtaining the trained generator network weight, and storing the trained pattern to generate the weight of each convolution kernel of each deconvolution layer, the weight of the full-connection layer, the weight of the modulation noise layer and the weight of the self-adaptive pattern modulation layer of each deconvolution layer of the counter-generation network;
(8) Expanding an infrared image sample:
and inputting the randomly generated feature vectors into a trained generator network for calculation, outputting an infrared image containing ship samples, adding the infrared image into a training set, and completing expansion of the infrared image ship samples.
2. The method for expanding ship samples in infrared images based on pattern generation countermeasure network according to claim 1, wherein the loss function of the discriminator network in step (5) is as follows:
wherein L is D Representing a loss function of the arbiter network, E [. Cndot.]Representing a desired operation, D (-) representing the output of a arbiter network in a pattern generation countermeasure network, G (-) representing the output of a arbiter network in a pattern generation countermeasure networkZ represents a randomly generated feature vector, x represents a 16 Zhang Shi beat infrared image in the training set, γ represents a square term coefficient, λ represents a constraint term coefficient,the constraint term infrared image obtained by corresponding fusion of 16 infrared images output by the generator and 16 Zhang Shi infrared images in the training set according to random proportion is represented 2 Representing a 2-norm operation, and v represents a derivative operation.
3. The method of augmenting ship samples in infrared images based on pattern generation of countermeasure network of claim 2, wherein the generator network in step (6) has a loss function as follows:
L G ＝-E[D(G(z))]
wherein L is G Representing the loss function of the generator network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010650897.8A CN111814875B (en) | 2020-07-08 | 2020-07-08 | Ship sample expansion method in infrared image based on pattern generation countermeasure network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010650897.8A CN111814875B (en) | 2020-07-08 | 2020-07-08 | Ship sample expansion method in infrared image based on pattern generation countermeasure network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111814875A CN111814875A (en) | 2020-10-23 |
| CN111814875B true CN111814875B (en) | 2023-08-01 |
Family
ID=72842044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010650897.8A Active CN111814875B (en) | 2020-07-08 | 2020-07-08 | Ship sample expansion method in infrared image based on pattern generation countermeasure network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111814875B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465736B (en) * | 2020-11-18 | 2023-03-24 | 武汉理工大学 | Infrared video image enhancement method for port ship monitoring |
| CN112835709B (en) * | 2020-12-17 | 2023-09-22 | 华南理工大学 | Cloud load time sequence data generation method, system and medium based on generation countermeasure network |
| CN112700408B (en) * | 2020-12-28 | 2023-09-08 | ***股份有限公司 | Model training method, image quality evaluation method and device |
| CN112884003A (en) * | 2021-01-18 | 2021-06-01 | 中国船舶重工集团公司第七二四研究所 | Radar target sample expansion generation method based on sample expander |
| CN112819914B (en) * | 2021-02-05 | 2023-07-18 | 北京航空航天大学 | PET image processing method |
| CN112950505B (en) * | 2021-03-03 | 2024-01-23 | 西安工业大学 | Image processing method, system and medium based on generation countermeasure network |
| CN112784930B (en) * | 2021-03-17 | 2022-03-04 | 西安电子科技大学 | CACGAN-based HRRP identification database sample expansion method |
| CN113792764A (en) * | 2021-08-24 | 2021-12-14 | 北京遥感设备研究所 | Sample expansion method, system, storage medium and electronic equipment |
| CN113516656B (en) * | 2021-09-14 | 2021-12-14 | 浙江双元科技股份有限公司 | Defect image data processing simulation method based on ACGAN and Cameralink cameras |
| CN114022724A (en) * | 2021-10-08 | 2022-02-08 | 郑州大学 | Pipeline disease image data enhancement method for generating countermeasure network based on style migration |
| CN114881884B (en) * | 2022-05-24 | 2024-03-29 | 河南科技大学 | Infrared target sample enhancement method based on generation countermeasure network |
| CN115086674B (en) * | 2022-06-16 | 2024-04-02 | 西安电子科技大学 | Image steganography method based on generation of countermeasure network |
| CN115393242A (en) * | 2022-09-30 | 2022-11-25 | 国网电力空间技术有限公司 | Method and device for enhancing foreign matter image data of power grid based on GAN |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018028255A1 (en) * | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| CN111027454A (en) * | 2019-12-06 | 2020-04-17 | 西安电子科技大学 | SAR (synthetic Aperture Radar) ship target classification method based on deep dense connection and metric learning |
| CN111260594A (en) * | 2019-12-22 | 2020-06-09 | 天津大学 | Unsupervised multi-modal image fusion method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11170256B2 (en) * | 2018-09-26 | 2021-11-09 | Nec Corporation | Multi-scale text filter conditioned generative adversarial networks |
-
2020
- 2020-07-08 CN CN202010650897.8A patent/CN111814875B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018028255A1 (en) * | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| CN111027454A (en) * | 2019-12-06 | 2020-04-17 | 西安电子科技大学 | SAR (synthetic Aperture Radar) ship target classification method based on deep dense connection and metric learning |
| CN111260594A (en) * | 2019-12-22 | 2020-06-09 | 天津大学 | Unsupervised multi-modal image fusion method |
Non-Patent Citations (2)
| Title |
|---|
| 基于双重对抗自编码网络的红外目标建模方法;苗壮;张湧;李伟华;;光学学报(第11期);全文 * |
| 基于稀疏编码的红外显著区域超分重建算法;黄硕;胡勇;巩彩兰;郑付强;;红外与毫米波学报(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111814875A (en) | 2020-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111814875B (en) | Ship sample expansion method in infrared image based on pattern generation countermeasure network | |
| CN108564119B (en) | Pedestrian image generation method in any posture | |
| CN106981080A (en) | Night unmanned vehicle scene depth method of estimation based on infrared image and radar data | |
| CN108509910A (en) | Deep learning gesture identification method based on fmcw radar signal | |
| CN110348059B (en) | Channel internal flow field reconstruction method based on structured grid | |
| CN112257741B (en) | Method for detecting generative anti-false picture based on complex neural network | |
| CN113724149B (en) | Weak-supervision visible light remote sensing image thin cloud removing method | |
| CN116343330A (en) | Abnormal behavior identification method for infrared-visible light image fusion | |
| CN114049434A (en) | 3D modeling method and system based on full convolution neural network | |
| Chen et al. | A wavelet based deep learning method for underwater image super resolution reconstruction | |
| CN113420639A (en) | Method and device for establishing near-ground infrared target data set based on generation countermeasure network | |
| CN113989100A (en) | Infrared texture sample expansion method based on pattern generation countermeasure network | |
| CN116258816A (en) | Remote sensing image simulation method based on nerve radiation field | |
| Reed et al. | Coupling rendering and generative adversarial networks for artificial SAS image generation | |
| CN115761117A (en) | Three-dimensional human body reconstruction method and system based on STAR model | |
| CN110148083B (en) | Image fusion method based on rapid BEMD and deep learning | |
| Zhang et al. | Ship HRRP target recognition based on CNN and ELM | |
| CN109658508A (en) | A kind of landform synthetic method of multiple dimensioned details fusion | |
| CN117036875B (en) | Infrared weak and small moving target generation algorithm based on fusion attention GAN | |
| CN117036901A (en) | Small sample fine adjustment method based on visual self-attention model | |
| CN116721201A (en) | Infrared sea surface background simulation method and system for intelligent texture generation | |
| CN115830462A (en) | SAR image reconstruction method and device based on cycle consistency countermeasure network | |
| CN116486244A (en) | Underwater target detection method based on detail enhancement | |
| CN116188847A (en) | Intelligent processing method and system for military toy model | |
| CN115471516A (en) | Optical-radar ISAR image conversion method and device based on generation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |