CN113569960B - Small sample image classification method and system based on domain adaptation - Google Patents
Small sample image classification method and system based on domain adaptation Download PDFInfo
- Publication number
- CN113569960B CN113569960B CN202110866395.3A CN202110866395A CN113569960B CN 113569960 B CN113569960 B CN 113569960B CN 202110866395 A CN202110866395 A CN 202110866395A CN 113569960 B CN113569960 B CN 113569960B
- Authority
- CN
- China
- Prior art keywords
- domain
- feature
- image
- representing
- small sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000006978 adaptation Effects 0.000 title claims abstract description 28
- 238000012549 training Methods 0.000 claims abstract description 69
- 238000000605 extraction Methods 0.000 claims abstract description 59
- 238000012360 testing method Methods 0.000 claims abstract description 39
- 238000010801 machine learning Methods 0.000 claims abstract description 35
- 230000006870 function Effects 0.000 claims description 27
- 239000011159 matrix material Substances 0.000 claims description 20
- 230000004913 activation Effects 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 4
- 238000011478 gradient descent method Methods 0.000 claims description 3
- 238000011160 research Methods 0.000 abstract description 6
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 238000011426 transformation method Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the field of image classification and discloses a small sample image classification method and a system based on domain adaptation, wherein the method comprises the steps of constructing a feature extractor and loading a pre-training parameter initialization model; obtaining a plurality of feature extractors by using a domain feature extraction module; obtaining a final feature extractor suitable for the target data domain by using a weight training module; the final performance of the method is obtained by using the test module, so that the problem that the current small sample machine learning method cannot really solve the small sample in the field of computer vision is solved. The invention ensures that the small sample machine learning method does not need a strict meta training set, and the source domain and the target domain data can be dissimilar, thereby the small sample machine learning method can be better applied to actual scenes and has the advantage of expanding the application range of the current research.
Description
Technical Field
The invention relates to the field of image classification, in particular to a small sample image classification method and system based on domain adaptation.
Background
In the field of computer vision, training a currently mainstream neural network model often requires an image dataset with a very large sample number, because machine learning requires a large amount of training data to fit the data distribution of a target task, and insufficient sample data volume can significantly affect the performance of the machine learning model. An image dataset typically includes tens of thousands of image samples, each image category including hundreds of samples. In practical application scenarios, a sufficient amount of sample data cannot be obtained, so that the problem of classifying small sample images gradually becomes a research hot spot, and in order to solve the problem, a small sample machine learning method is proposed.
The small sample machine learning method mainly aims at solving the situation that the quantity of sample data available for model training is extremely insufficient (each class only comprises 1-5 samples), under the scene, the performance of a general neural network model and the machine learning method is extremely poor, while the small sample machine learning method can achieve better performance under the extreme scene by using a special neural network model and a training thought, so the field gradually becomes the main direction of current research. The small sample machine learning approach typically artificially divides the available data set into a meta-training data set and a meta-testing data set and separates the training process into meta-training and meta-testing phases. During the meta-training phase, using a sufficiently large annotated meta-training dataset for training the method model; in the meta-test phase, a meta-test dataset containing different classes than the meta-training dataset is used to evaluate the ability of the method model to learn and classify these new classes. In order to simulate the actual application scene, the small sample machine learning method takes out 1-5 images from each category in the meta-test set to form a support set for model learning by a method, and the rest images form a query set for testing the classification performance of the model of the method. The current research on the small sample machine learning method is mainly carried out from two directions of model optimization and element learning, wherein the model optimization direction is optimized for a neural network model used by the small sample machine learning method, so that the model can adapt to the scene of the small sample, for example, a graph convolution nerve is introduced, and an countermeasure network is generated as a backbone network so as to achieve better classification effect; the meta-learning direction is to introduce the idea of meta-learning into a small sample machine learning method, train a movable meta-parameter (such as gradient, initial parameters of model) in the meta-training stage, and then use these parameters to make the method model obtain better classification performance in the meta-testing stage.
When the small sample machine learning method is used, the small sample machine learning method basically comprises a meta training process, a method model needs to be trained on a meta training set which is sufficient in the number of image sample data and the number of categories and is highly similar to that of a meta test data set, so that the current small sample machine learning method usually only performs experiments on a plurality of image data sets which are specially used for classifying small sample images, the data which can be obtained in an actual scene usually only has a plurality of categories and only has a plurality of sample images in each category, the meta training set and the meta test set cannot be divided, and a public data set which is similar to the current task data cannot be easily obtained, so that the research on the small sample machine learning method is basically in a theoretical research stage, and practical application cannot be performed, namely the current small sample machine learning method cannot really solve the problem of the small sample in the computer vision field, and needs to be improved.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention aims to provide a small sample image classification method and a small sample image classification system based on domain adaptation, wherein the domain adaptation method is creatively used in the small sample machine learning method, so that the small sample machine learning method does not need to require a strict meta training set any more, a plurality of feature extractors are trained on a source domain, then the small sample machine learning method is used on a target domain support set to train to obtain the optimal weight for combining the feature extractors, so that a final feature extractor which is most suitable for a target domain is obtained, and the target domain query set is tested to finally obtain the optimal performance.
In order to achieve the above purpose, the technical scheme of the invention is as follows:
a small sample image classification method based on domain adaptation, comprising the steps of:
constructing a feature extractor, and loading a pre-training parameter initialization model;
obtaining a plurality of feature extractors by using a domain feature extraction module;
obtaining a final feature extractor suitable for the target data domain by using a weight training module;
and obtaining the final performance of the method by using a test module.
By adopting the technical scheme, the small sample image classification method based on domain adaptation creatively uses the domain adaptation method in the small sample machine learning method by means of the domain feature extraction module, the weight training module and the prediction module, specifically, after a plurality of feature extractors which are fully trained on a source domain and inserted with the domain feature extraction module are obtained by using the domain feature extraction module, the weight training module trains and combines the optimal weights of the feature extractors on a support set of a target domain, so that a final feature extractor which is most suitable for the target domain is obtained, and a final small sample image classification result is obtained by the prediction module so as to measure the performance of the small sample image classification method based on domain adaptation.
The invention is further provided with: the domain feature extraction module transforms the feature map X output by the feature extractor to extract domain-specific global features of the input image using the following formula:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; as indicated by the letter "".
By adopting the technical scheme, the domain feature extraction module can transform the feature map X output by the feature extractor to better extract domain-specific global features of the input image.
The invention is further provided with: the domain feature extraction module also performs a local attention operation on the feature map X output by the feature extractor to extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, and when the input value m is less than or equal to 0, reLU (m) =0, and when m > 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution layer operation of the input feature map X by the point convolution.
By adopting the technical scheme, the domain feature extraction module also performs local attention operation on the feature map X output by the feature extractor by using the following formula so as to better extract domain-specific local features of the input image.
The invention is further provided with: the final output of the domain feature extraction module is expressed as follows:
wherein,is the final output of the domain feature extraction module.
By adopting the technical scheme, the X is obtained through local attention operation treatment 2 The method has the same dimension as the input feature map X, can reserve and highlight fine details in the bottom layer features, and can better extract, save and migrate domain-specific local features. Because the dimension of the input feature map X is not changed in the transformation operation and the local attention operation, the two parts of output can be directly added, namely, the domain-specific local feature and the domain-specific global feature of the input image are simultaneously extracted, so that the capability of the feature extractor for extracting the domain-specific feature is greatly improved, more feature information is migrated from a source domain, assistance is provided for a small sample machine learning method in a target domain, and the final small sample image classification performance is improved.
The invention is further provided with: the weight training module obtains the weight lambda using the following formula:
where λ is the weight of combining the feature extractors, λ is a vector of dimension 1×n; n is the number of feature extractors that contain domain-specific features (including domain-specific global features and domain-specific local features); s is a support set in the target domain, and the image tag pairs in the support set s are (x i ，y i ) Wherein x is i Representing the ith image sample, y i Representing its corresponding category label; j represents the image category in the support set s, N in total s Class, n s Image samples s j An image sample subscript set representing an image class label equal to j, p being a prototype, f (·) representing a feature extractor, d (·, ·) representing a distance function for estimating the similarity between different parameters; l (λ) represents the loss function on the support set with respect to λ.
By adopting the technical scheme, the weight training module uses the target domain support set data to train the weights of the plurality of feature extractors obtained by the combined domain feature extraction module, and combines the best weight obtained by training to obtain the final feature extractor which is most suitable for the target domain.
The invention is further provided with: the test module obtains a predicted result of the test sample using the following formula:
wherein x is k Representing the kth image sample, yk represents its corresponding class label,representing the predicted outcome for the test sample, j represents the image class in the support set s, N in total s Class, n s Image samples s j The set of image sample subscripts representing an image class label equal to j, p being the prototype, f (·) representing the feature extractor, d (·, ·) representing the distance function for estimating the similarity between different parameters.
By adopting the technical scheme, the testing module can predict the category label of the target domain query set sample data so as to test the final performance of the method.
The invention is further provided with: after the feature extractor is constructed and the pre-training parameter initialization model is loaded, the method further comprises the following steps:
the domain feature extraction module trains a plurality of feature extractors on the source domain;
the weight training module trains all the feature extractors on the target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for the target domain;
the test module tests final image classification performance on the target domain query set using the final feature extractor.
By adopting the technical scheme, the plurality of feature extractors are trained on the source domain, the model parameters of the feature extractors are stored and carry domain specific features of the domain, the optimal weights of the plurality of feature extractors are obtained by combining support set data of the target domain with training of a small sample machine learning method, all the feature extractors are combined according to the optimal weights to obtain a final feature extractor capable of obtaining optimal performance on the target domain, and finally, the final feature extractor can be utilized to test the final image classification performance on a target domain query set.
The invention is further provided with: each data field in the source field trains a feature extractor.
By adopting the technical scheme, the model parameters of one feature extractor trained on each data domain are stored and carry the domain specific features of the domain, so that all the feature extractors can extract the domain specific features of all the data domains in the source domain, the utilization rate of the source domain is improved, and the final image classification performance is also improved.
The invention is further provided with: wherein the model structure of the plurality of feature extractors trained on the source domain is identical.
By adopting the technical scheme, the model structures of the feature extractors are identical, so that subsequent combination is facilitated, and the operation difficulty of the combined feature extractors is reduced.
The invention also provides a small sample image classification system based on domain adaptation, which comprises a domain feature extraction module, a weight training module and a testing module, wherein the domain feature extraction module transforms a feature map X of an input image by using the following formula to extract domain specific global features of the input image:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; the element multiplication;
the domain feature extraction module also performs a local attention operation on a feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, and when the input value m is less than or equal to 0, reLU (m) =0, and when m > 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution operation of the input feature map X by the point convolution;
the final output of the domain feature extraction module is expressed as follows:
wherein,is the final output of the domain feature extraction module.
By adopting the technical scheme, the domain feature extraction module can be conveniently embedded into an original neural network, and the domain specific local feature and the domain specific global feature of the input image can be simultaneously extracted by combining the local attention and the feature map transformation method, so that the capability of the feature extractor for extracting the domain specific feature is greatly improved, more feature information is migrated from a source domain, and finally the classification performance of the final small sample image is improved.
In summary, the beneficial effects achieved by the invention are as follows:
(1) The small sample image classification method based on domain adaptation uses a domain adaptation method in a small sample machine learning method creatively by means of a domain feature extraction module, a weight training module and a prediction module, after a plurality of feature extractors are obtained by the domain feature extraction module, the weight training module obtains the optimal weight for combining the feature extractors and obtains a final feature extractor which is most suitable for a target domain, and the prediction module obtains a final small sample image classification result to measure the performance of the small sample image classification method based on domain adaptation;
(2) Training a plurality of feature extractors on a source domain, storing model parameters of the feature extractors and carrying domain specific features of the domain, training support set data of a target domain by combining a small sample machine learning method to obtain optimal weights for combining the plurality of feature extractors, combining all the feature extractors according to the optimal weights to obtain a final feature extractor capable of obtaining optimal performance on the target domain, and finally testing final image classification performance on a target domain query set by utilizing the final feature extractor;
(3) The domain feature extraction module can be conveniently embedded into an original neural network, combines a local attention and a feature map transformation method, can simultaneously extract domain-specific local features and domain-specific global features of an input image, greatly improves the capability of a feature extractor for extracting the domain-specific features, further migrates more feature information from a source domain, and finally improves the classification performance of a final small sample image.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
FIG. 1 is a flow chart of a small sample image classification method based on domain adaptation in the first embodiment;
FIG. 2 is a diagram of a key model structure of the present invention;
FIG. 3 is a block diagram of a neural network basic block with a domain feature extraction module inserted;
fig. 4 is a flow chart of a small sample image classification method based on domain adaptation in the second embodiment.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
As shown in fig. 1, a small sample image classification method based on domain adaptation includes:
s101: constructing a feature extractor, and loading a pre-training parameter initialization model;
as shown in FIG. 2, which is a key model structure of the present invention, a Resnet-18 convolutional neural network (i.e., resnet model of 18 layers of neural network layers) is used as a backbone network in this embodiment, the neural network being divided by 4 basic blocks and partitionsClass layer composition. The basic block structure is shown in fig. 3, where a represents two domain feature extraction modules inserted on each basic block according to the present invention, 3×3 represents a convolution layer with a convolution kernel of 3×3, BN represents a batch normalization layer, reLU represents an activation function,representing element multiplication->Representing the element addition.
Let the source domain be r= { R 1 ，r 2 ，…，r N }，r i Representing an ith image data field, N total;
let the target domain be t=s ≡q, where s represents the support set for training the small sample machine learning method, q represents the query set for performance testing, andin this example, training was performed using the CUB dataset (i.e., image dataset containing 200 different birds), the air dataset (i.e., image dataset containing 100 classes of Aircraft) and the VGG-Flower dataset (i.e., image dataset containing 102 different flowers) as source fields, the three datasets being set to r, respectively 1 ，r 2 ，r 3 And randomly taking out 5 images from each class of CIFAR-10 data set (namely, image data set containing 10 classes of common articles) to form a support set s, and taking a query set q formed by the residual image samples as a target domain to train and test a small sample machine learning method. The classification network consists of a pooling layer, a linear classification layer and a softmax (normalized exponential function) layer for outputting the final prediction result.
According to the key model structure in fig. 2, a neural network serving as a feature extractor is constructed, a pre-training parameter initialization model is loaded, gamma parameters are initialized to a full 1 matrix, beta parameters are initialized to a full 0 matrix, and a random parameter initialization classification network is used.
S102: obtaining a plurality of feature extractors by using a domain feature extraction module;
the domain feature extraction module transforms the feature map X output by the feature extractor to better extract domain-specific global features of the input image using the following formula:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; as indicated by the letter "".
Meanwhile, in order to better extract the domain-specific local features, the invention also introduces local attention operation, which consists of two point-wise convolution (Point-wise convolution) layers and a ReLU activation function. The point convolution is a special convolution operation, which can realize the fusion of cross-channel information to increase the nonlinearity of the network and realize the dimension increase and dimension decrease of the channel number, and is realized as a point convolution layer in the embodiment. ReLU is an activation function, and when the input value m is less than or equal to 0, reLU (m) =0, and when m > 0, reLU (m) =m. Thus, the domain feature extraction module performs a local attention operation on the feature map X output by the feature extractor to better extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, and when the input value m is less than or equal to 0, reLU (m) =0, and when m > 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution layer operation of the input feature map X by the point convolution.
Finally, the output of the domain feature extraction module is expressed as follows:
wherein,is the final output of the domain feature extraction module. X obtained by local attention operation processing 2 The method has the same dimension as the input feature map X, can reserve and highlight fine details in the bottom layer features, and can better extract, save and migrate domain-specific local features. Because the dimension of the input feature map X is not changed in the transformation operation and the local attention operation, the two parts of output can be directly added, namely, the domain-specific local feature and the domain-specific global feature of the input image are simultaneously extracted, so that the capability of the feature extractor for extracting the domain-specific feature is greatly improved, more feature information is migrated from a source domain, assistance is provided for a small sample machine learning method in a target domain, and the final small sample image classification performance is improved.
Using the above feature extractor to respectively locate in the source domain data domain r 1 ，r 2 ，r 3 Performing sufficient training, respectively storing model parameters obtained by training, and setting the feature extractor obtained by training as f 1 (·)，f 2 (·)，f 3 (. Cndot.) is used as an output to sufficiently extract domain-specific feature information of each domain, and these feature information are stored in the model parameters.
S103: obtaining a final feature extractor suitable for the target data domain by using a weight training module;
the weight training module obtains the weight λ using the following formula:
where λ is the weight of combining the feature extractors, λ is a vector of dimension 1×n; n is the number of feature extractors that contain domain-specific features (including domain-specific global features and domain-specific local features)An amount of; s is a support set in the target domain, and the image tag pairs in the support set s are (x i ，y i ) Wherein x is i Representing the ith image sample, y i Representing its corresponding category label; j represents the image category in the support set s, N in total s Class, n s Image samples s j The image sample subscript set for representing the image category label equal to j, p is a prototype, f (·) represents a feature extractor, d (·, ·) represents a distance function for estimating similarity between different parameters, and various choices such as Euclidean distance, manhattan distance and the like can be adopted in practical application; l (λ) represents the loss function on the support set with respect to λ.
The loss function shown by the formula can be used for training on a support set by using methods such as gradient descent and the like to obtain the optimal combination weight lambda, so as to obtain a final feature extractor f which is most suitable for a target domain λ (·)。
Specifically, in the step of obtaining a final feature extractor suitable for the target data domain using the weight training module, the method further comprises the following sub-steps:
s1031: output feature extractor f accepting a multi-domain feature extraction module 1 (·)，f 2 (·)，f 3 (. Cndot.) as input, the combining weight parameters λ of these feature extractors are initialized to full 1 vector, the initial f is obtained from the following equation λ (·)：
Wherein x represents a target domain image, f λ (x) The other settings in the formula representing the features extracted by the feature extractor for x are the same as those in step S103.
S1032: using f λ (. Cndot.) processing of image samples on support set s, initial support set per-class prototypes were obtained from the following equation
The setting in the formula is the same as that in step S103.
S1033: traversing all image samples on the support set s using a loss function shown in the following formula to obtain a training loss of the current lambda, and training lambda using a random gradient descent method:
where λ is the weight of combining the feature extractors, λ is a vector of dimension 1×n; n is the number of feature extractors that contain domain-specific features (including domain-specific global features and domain-specific local features); s is a support set in the target domain, and the image tag pairs in the support set s are (x i ，y i ) Wherein x is i Representing the ith image sample, y i Representing its corresponding category label; j represents the image category in the support set s, N in total s Class, n s Image samples s j The image sample subscript set for representing the image category label equal to j, p is a prototype, f (·) represents a feature extractor, d (·, ·) represents a distance function for estimating similarity between different parameters, and various choices such as Euclidean distance, manhattan distance and the like can be adopted in practical application; l (λ) represents the loss function on the support set with respect to λ.
S1034: repeating step S1033 until training loss of lambda is not reduced, and finally obtaining optimal weight parameter lambda of combining multiple feature extractors, namely obtaining final feature extractor f which is most suitable for target data field λ (·)。
S104: and obtaining the final performance of the method by using a test module.
The test module obtains a predicted result of the test sample using the following formula:
wherein x is k Representing the kth image sample, yk represents its corresponding class label,representing the predicted outcome for the test sample, j represents the image class in the support set s, N in total s Class, n s Image samples s j The image sample subscript set representing the image category label equal to j, p is a prototype, f (·) represents a feature extractor, d (·, ·) represents a distance function, and is used for estimating similarity between different parameters, and various choices such as euclidean distance, manhattan distance and the like can be available in practical application.
Obtaining a final feature extractor f which is most suitable for the target data field by a weight training module λ After (-), traversing the query set q by using a test module to obtain the final performance of the classification method.
The small sample image classification method based on domain adaptation in the embodiment creatively uses a domain adaptation method in a small sample machine learning method by means of a domain feature extraction module, a weight training module and a prediction module, specifically, after a plurality of feature extractors which are fully trained on a source domain and inserted with the domain feature extraction module are obtained by using the domain feature extraction module, the weight training module trains and combines the optimal weights of the feature extractors on a support set of a target domain, so that a final feature extractor which is most suitable for the target domain is obtained, and a final small sample image classification result is obtained by the prediction module so as to measure the performance of the small sample image classification method based on domain adaptation.
The embodiment also provides a small sample image classification system based on domain adaptation, which comprises a domain feature extraction module, a weight training module and a testing module, wherein the domain feature extraction module transforms a feature map X of an input image to extract domain specific global features of the input image by using the following formula:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; the element multiplication;
the domain feature extraction module also performs a local attention operation on a feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, and when the input value m is less than or equal to 0, reLU (m) =0, and when m > 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution operation of the input feature map X by the point convolution;
the final output of the domain feature extraction module is expressed as follows:
wherein,is the final output of the domain feature extraction module.
The domain feature extraction module can be conveniently embedded into an original neural network, combines a local attention and a feature map transformation method, can simultaneously extract domain-specific local features and domain-specific global features of an input image, greatly improves the capability of a feature extractor for extracting the domain-specific features, further migrates more feature information from a source domain, and finally improves the classification performance of a final small sample image.
Example two
As shown in fig. 4, in order to disclose a small sample image classification method based on domain adaptation, unlike the first embodiment, after constructing the feature extractor and loading the pre-training parameter initialization model, the method further includes the following steps:
s201: the domain feature extraction module trains a plurality of feature extractors on the source domain;
s202: the weight training module trains all the feature extractors on the target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for the target domain;
s203: the test module tests final image classification performance on the target domain query set using the final feature extractor.
In step S201, one feature extractor is trained for each data field in the source field.
The model parameters of one feature extractor trained on each data domain are stored and carry the domain specific features of the domain, so that all the feature extractors can extract the domain specific features of all the data domains in the source domain, the utilization rate of the source domain is improved, and the final image classification performance is also improved.
Further, the model structures of the plurality of feature extractors trained on the source domain are the same, so that the subsequent combination of the plurality of feature extractors is facilitated, and the operation difficulty of the combined feature extractors is reduced.
In the embodiment, training is performed on a source domain through a plurality of feature extractors, model parameters of the feature extractors are stored and carry domain specific features of the domain, the optimal weights of the feature extractors are obtained by combining support set data of a target domain with training of a small sample machine learning method, all the feature extractors are combined according to the optimal weights to obtain a final feature extractor capable of obtaining optimal performance on the target domain, and finally, the final feature extractor can be utilized to test the final image classification performance on a target domain query set.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (9)
1. A small sample image classification method based on domain adaptation, comprising the steps of:
constructing a feature extractor, and loading a pre-training parameter initialization model;
obtaining a plurality of feature extractors by using a domain feature extraction module;
obtaining a final feature extractor suitable for the target data domain by using a weight training module;
obtaining the final performance of the method by using a test module;
in the step of obtaining a final feature extractor suitable for the target data domain using the weight training module, the method further comprises the following sub-steps:
s1: output feature extractor f accepting a multi-domain feature extraction module 1 (·),f 2 (·),f 3 (. Cndot.) as input, the combining weight parameters λ of these feature extractors are initialized to full 1 vector, the initial f is obtained from the following equation λ (·)：
Wherein x represents a target domain image, f λ (x) Representing the features extracted by the feature extractor for x;
s2: using f λ (. Cndot.) processing of image samples on support set s, initial support set per-class prototypes were obtained from the following equation
S3: traversing all image samples on the support set s using a loss function shown in the following formula to obtain a training loss of the current lambda, and training lambda using a random gradient descent method:
where λ is the weight of combining the feature extractors, λ is a vector of dimension 1×n; n is the number of feature extractors that contain domain-specific features; s is a support set in the target domain, and the image tag pairs in the support set s are (x i ,y i ) Wherein x is i Representing the ith image sample, y i Representing its corresponding category label; j represents the image category in the support set s, N in total s Class, n s Image samples s j An image sample subscript set representing an image class label equal to j, p being a prototype, f (·) representing a feature extractor, d (·, ·) representing a distance function for estimating the similarity between different parameters; l (λ) represents a loss function on the support set with respect to λ;
s4: repeating the step S3 until the training loss of lambda is not reduced, and finally obtaining the optimal weight parameter lambda combining a plurality of feature extractors, namely obtaining the final feature extractor f which is most suitable for the target data field λ (·)。
2. The small sample image classification method based on domain adaptation according to claim 1, wherein the domain feature extraction module transforms the feature map X output by the feature extractor to extract domain-specific global features of the input image using the following formula:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; as indicated by the letter "".
3. The small sample image classification method based on domain adaptation according to claim 2, wherein the domain feature extraction module further performs a local attention operation on the feature map X output by the feature extractor to extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, when the input value m is less than or equal to 0, reLU (m) =0, when m>At 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution layer operation of the input feature map X by the point convolution.
4. A small sample image classification method based on domain adaptation according to claim 3, characterized in that the final output of said domain feature extraction module is formulated as follows:
wherein,is the final output of the domain feature extraction module.
5. The domain adaptation based small sample image classification method of claim 1, wherein the test module obtains the prediction result of the test sample using the following formula:
wherein x is k Representing the kth image sample, y k Indicating the category label to which it corresponds,representing the predicted outcome for the test sample, j represents the image class in the support set s, N in total s Class, n s Image samples s j The set of image sample indices representing the image class labels equal to j, p being the prototype, f (·) representing the feature extractor, d (·.) representing the distance function used to estimate the similarity between the different parameters.
6. The domain-adaptive small sample image classification method according to claim 1, further comprising the steps of, after constructing the feature extractor and loading the pre-training parameter initialization model:
the domain feature extraction module trains a plurality of feature extractors on the source domain;
the weight training module trains all the feature extractors on the target domain support set by using a small sample machine learning method to obtain weights for combining all the feature extractors and combines all the feature extractors according to the weights to obtain a final feature extractor suitable for the target domain;
the test module tests final image classification performance on the target domain query set using the final feature extractor.
7. A domain-adaptive small sample image classification method according to claim 5, wherein one feature extractor is trained per data domain in the source domain.
8. The domain-adaptive small sample image classification method as claimed in claim 5, wherein the model structure of a plurality of feature extractors trained on the source domain is identical.
9. The small sample image classification system based on domain adaptation is characterized by comprising a domain feature extraction module, a weight training module and a testing module, wherein the domain feature extraction module transforms a feature map X of an input image to extract domain-specific global features of the input image by using the following formula:
X 1 ＝γ⊙X+β
wherein X is a matrix with a dimension HW×C, wherein H, W and C respectively represent the height, width and channel number, and the domain feature extraction module takes X as input and processes the X; x is X 1 For the transformed matrix; gamma and beta are trainable parameter matrixes, and the dimensions are the same as X; the element multiplication;
the domain feature extraction module also performs a local attention operation on a feature map X of the input image to extract domain-specific local features of the input image using the following formula:
X 2 ＝PWConv 2 (ReLU(PWConv 1 (X)))
wherein X is 2 A matrix after performing a local attention operation; pwconv 1 Representing the first point convolution layer, PWConv 2 Representing a second point convolution layer; reLU is an activation function, when the input value m is less than or equal to 0, reLU (m) =0, when m>At 0, reLU (m) =m; pwconv 1 (X) represents the result of the point convolution operation of the input feature map X by the point convolution;
the final output of the domain feature extraction module is expressed as follows:
wherein,the final output of the domain feature extraction module;
in the step of obtaining a final feature extractor suitable for the target data domain using the weight training module, the method further comprises the following sub-steps:
s1: output feature extractor f accepting a multi-domain feature extraction module 1 (·),f 2 (·),f 3 (. Cndot.) as input, the combining weight parameters λ of these feature extractors are initialized to full 1 vector, the initial f is obtained from the following equation λ (·)：
Wherein x represents a target domain image, f λ (x) Representing the features extracted by the feature extractor for x;
s2: using f λ (. Cndot.) processing of image samples on support set s, initial support set per-class prototypes were obtained from the following equation
S3: traversing all image samples on the support set s using a loss function shown in the following formula to obtain a training loss of the current lambda, and training lambda using a random gradient descent method:
where λ is the weight of combining the feature extractors, λ is a vector of dimension 1×n; n is the number of feature extractors that contain domain-specific features; s is a support set in the target domain, and the image tag pairs in the support set s are (x i ,y i ) Wherein x is i Representing the ith image sample, y i Representing its corresponding category label; j represents the image category in the support set s, N in total s Class, n s Image samples s j An image sample subscript set representing an image class label equal to j, p being a prototype, f (·) representing a feature extractor, d (·, ·) representing a distance function for estimating the similarity between different parameters; l (λ) represents a loss function on the support set with respect to λ;
s4: repeating the step S3 until the training loss of lambda is not reduced, and finally obtaining the optimal weight parameter lambda combining a plurality of feature extractors, namely obtaining the final feature extractor f which is most suitable for the target data field λ (·)。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110866395.3A CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110866395.3A CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113569960A CN113569960A (en) | 2021-10-29 |
| CN113569960B true CN113569960B (en) | 2023-12-26 |
Family
ID=78169151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110866395.3A Active CN113569960B (en) | 2021-07-29 | 2021-07-29 | Small sample image classification method and system based on domain adaptation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113569960B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117150282B (en) * | 2023-09-16 | 2024-01-30 | 石家庄正和网络有限公司 | Secondhand equipment recycling evaluation method and system based on prediction model |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961089A (en) * | 2019-02-26 | 2019-07-02 | 中山大学 | Small sample and zero sample image classification method based on metric learning and meta learning |
| CN111738301A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image data identification method based on two-channel learning |
| CN112419321A (en) * | 2021-01-25 | 2021-02-26 | 长沙理工大学 | X-ray image identification method and device, computer equipment and storage medium |
| CN112668494A (en) * | 2020-12-31 | 2021-04-16 | 西安电子科技大学 | Small sample change detection method based on multi-scale feature extraction |
| CN112784879A (en) * | 2020-12-31 | 2021-05-11 | 前线智能科技(南京)有限公司 | Medical image segmentation or classification method based on small sample domain self-adaption |
| CN112784764A (en) * | 2021-01-27 | 2021-05-11 | 南京邮电大学 | Expression recognition method and system based on local and global attention mechanism |
| CN112801146A (en) * | 2021-01-13 | 2021-05-14 | 华中科技大学 | Target detection method and system |
| CN112990097A (en) * | 2021-04-13 | 2021-06-18 | 电子科技大学 | Face expression recognition method based on countermeasure elimination |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10546232B2 (en) * | 2017-07-04 | 2020-01-28 | Microsoft Technology Licensing, Llc | Image recognition with promotion of underrepresented classes |
| CN110472483B (en) * | 2019-07-02 | 2022-11-15 | 五邑大学 | SAR image-oriented small sample semantic feature enhancement method and device |
-
2021
- 2021-07-29 CN CN202110866395.3A patent/CN113569960B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961089A (en) * | 2019-02-26 | 2019-07-02 | 中山大学 | Small sample and zero sample image classification method based on metric learning and meta learning |
| CN111738301A (en) * | 2020-05-28 | 2020-10-02 | 华南理工大学 | Long-tail distribution image data identification method based on two-channel learning |
| CN112668494A (en) * | 2020-12-31 | 2021-04-16 | 西安电子科技大学 | Small sample change detection method based on multi-scale feature extraction |
| CN112784879A (en) * | 2020-12-31 | 2021-05-11 | 前线智能科技(南京)有限公司 | Medical image segmentation or classification method based on small sample domain self-adaption |
| CN112801146A (en) * | 2021-01-13 | 2021-05-14 | 华中科技大学 | Target detection method and system |
| CN112419321A (en) * | 2021-01-25 | 2021-02-26 | 长沙理工大学 | X-ray image identification method and device, computer equipment and storage medium |
| CN112784764A (en) * | 2021-01-27 | 2021-05-11 | 南京邮电大学 | Expression recognition method and system based on local and global attention mechanism |
| CN112990097A (en) * | 2021-04-13 | 2021-06-18 | 电子科技大学 | Face expression recognition method based on countermeasure elimination |
Non-Patent Citations (2)
| Title |
|---|
| 基于卷积神经网络的小样本图像识别方法;段萌;王功鹏;牛常勇;;计算机工程与设计(第01期);全文 * |
| 基于深度神经网络的少样本学习综述;李新叶;龙慎鹏;朱婧;;计算机应用研究(第08期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113569960A (en) | 2021-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110163234B (en) | Model training method and device and storage medium | |
| CN108805185B (en) | Face recognition method and device, storage medium and computer equipment | |
| CN110969250B (en) | Neural network training method and device | |
| Su et al. | Vitas: Vision transformer architecture search | |
| CN109558942B (en) | Neural network migration method based on shallow learning | |
| US20190361972A1 (en) | Method, apparatus, device for table extraction based on a richly formatted document and medium | |
| CN116134454A (en) | Method and system for training neural network models using knowledge distillation | |
| CN110362723A (en) | A kind of topic character representation method, apparatus and storage medium | |
| CN110245683B (en) | Residual error relation network construction method for less-sample target identification and application | |
| CN114359563B (en) | Model training method, device, computer equipment and storage medium | |
| US20180349766A1 (en) | Prediction guided sequential data learning method | |
| CN112906865B (en) | Neural network architecture searching method and device, electronic equipment and storage medium | |
| CN111767962A (en) | One-stage target detection method, system and device based on generation countermeasure network | |
| CN108830236A (en) | A kind of recognition methods again of the pedestrian based on depth characteristic | |
| CN107590505A (en) | The learning method of joint low-rank representation and sparse regression | |
| CN112231516B (en) | Training method of video abstract generation model, video abstract generation method and device | |
| CN116310425B (en) | Fine-grained image retrieval method, system, equipment and storage medium | |
| CN113128478A (en) | Model training method, pedestrian analysis method, device, equipment and storage medium | |
| CN114565807A (en) | Method and device for training target image retrieval model | |
| CN113609337A (en) | Pre-training method, device, equipment and medium of graph neural network | |
| CN113569960B (en) | Small sample image classification method and system based on domain adaptation | |
| CN111178196B (en) | Cell classification method, device and equipment | |
| CN116089883A (en) | Training method for improving classification degree of new and old categories in existing category increment learning | |
| CN114556364A (en) | Neural architecture search based on similarity operator ordering | |
| CN111079011A (en) | Deep learning-based information recommendation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |