CN110378366B - Cross-domain image classification method based on coupling knowledge migration - Google Patents

Cross-domain image classification method based on coupling knowledge migration Download PDF

Info

Publication number
CN110378366B
CN110378366B CN201910482559.5A CN201910482559A CN110378366B CN 110378366 B CN110378366 B CN 110378366B CN 201910482559 A CN201910482559 A CN 201910482559A CN 110378366 B CN110378366 B CN 110378366B
Authority
CN
China
Prior art keywords
data
target domain
domain data
domain
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910482559.5A
Other languages
Chinese (zh)
Other versions
CN110378366A (en
Inventor
孟敏
兰孟城
武继刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN201910482559.5A priority Critical patent/CN110378366B/en
Publication of CN110378366A publication Critical patent/CN110378366A/en
Application granted granted Critical
Publication of CN110378366B publication Critical patent/CN110378366B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24147—Distances to closest patterns, e.g. nearest neighbour classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a cross-domain image classification method based on coupling knowledge migration, which is characterized in that a common low-dimensional subspace of a source domain and a target domain is searched based on a maximum mean difference criterion, and the difference between the edge distribution and the class condition distribution of data of the source domain and the target domain is eliminated; constructing respective adjacent maps according to label information of source domain data and pseudo label information of target domain data, keeping structural consistency of the data from an original space to a low-dimensional subspace, and simultaneously dynamically adjusting the structure of the adjacent maps to promote the forward migration of domain knowledge; training a nearest neighbor classifier by using source domain data with label information in a low-dimensional subspace, and continuously iterating and optimizing pseudo label information of target domain data to obtain final label information of the target domain data, namely finishing cross-domain image classification; in addition, the method of the invention endows different confidence degrees of the pseudo label of the target domain image by designing a sample reweighting strategy, effectively reduces the negative transfer of the knowledge in the domain, and improves the precision of cross-domain image classification.

Description

Cross-domain image classification method based on coupling knowledge migration
Technical Field
The invention relates to the technical field of computer vision image classification, in particular to a cross-domain image classification method based on coupling knowledge migration.
Background
Conventional machine learning algorithms typically require a large number of labeled data samples and require that the training samples and the test samples follow separate and identical distributions. A straightforward approach is to migrate an existing tagged data set onto a new unknown data set, i.e. from the source domain to the target domain. However, the distribution must be different between different data sets, and the assumption that "independent equal distribution is obeyed" is often not true. Therefore, the conventional image classification method has difficulty in obtaining good classification performance.
In order to overcome the difference of sample distribution between a source domain and a target domain and improve the generalization capability of a classification algorithm, currently, domain adaptation is mainly used as a means for processing the problem of knowledge migration in cross-domain image classification, and the method can be generally divided into a method based on feature transformation, a method based on an example and a method based on neural network migration. However, there are many problems to be solved and improved in the current cross-domain learning, for example, in the case of a large difference between the source domain and the target domain, the classification performance of the domain adaptation method based on feature transformation is far from reaching the original expectation of people; the method based on the example needs to calculate additional sample weight, has high calculation complexity, and cannot carry out rapid cross-field learning on a large data set; the neural network-based approach requires a large number of labeled source domain samples for long training and has a large limitation in practical applications.
Disclosure of Invention
The invention provides a cross-domain image classification method based on coupling knowledge migration, which aims to solve the problems of long calculation time, insufficient classification accuracy and the like of the existing cross-domain image classification method.
In order to realize the purpose of the invention, the technical means adopted is as follows:
a cross-domain image classification method based on coupling knowledge migration comprises the following steps:
s1, obtaining a source domain image and a target domain image, extracting characteristics, respectively obtaining a source domain characteristic matrix as source domain data, and obtaining a target domain characteristic matrix as target domain data;
s2, respectively constructing an adjacency graph of source domain data and an adjacency graph of target domain data according to label information of the source domain data, and respectively obtaining a weight matrix of the source domain and a weight matrix of the target domain by calculating edge weights of the adjacency graphs;
s3, performing inter-domain data distribution alignment in a common low-dimensional subspace of the source domain and the target domain, and performing knowledge migration of intra-domain data by using a weight matrix of the source domain and a weight matrix of the target domain;
s4, in the low-dimensional subspace, utilizing source domain data with label information to train a nearest neighbor classifier and predicting the category information of target domain data;
s5, reweighing the label information of the predicted target domain data;
and S6, judging whether the maximum iteration number is reached, if not, returning to the step S2, and if so, outputting the label information of the predicted target domain image.
In the scheme, the common low-dimensional subspace of the source domain and the target source is searched, the difference of sample distribution is reduced, meanwhile, the adjacent map is used for carrying out the in-domain knowledge positive transfer, the structural consistency from the original space to the common low-dimensional subspace is kept, the data distribution is aligned by eliminating the difference of the edge distribution and class condition distribution of the data of the source domain and the target domain, therefore, the source domain data with label information is used for training a nearest neighbor classifier in the low-dimensional subspace to predict the label information of the data of the target domain, and finally, the predicted label information of the data of the target domain is subjected to weighting to inhibit the knowledge negative transfer, so that the accurate classification of the cross-domain image is realized.
Preferably, the specific steps of step S1 are:
s1.1. Input n s A source domain image with label and n t A label-free target domain image; wherein n is s 、n t Is a positive integer;
s1.2, respectively extracting the features of the source domain image and the target domain image, and arranging the feature vectors obtained by extracting the source domain image in columns to obtain a source domain data feature matrix
Figure BDA0002084313960000021
Where m is the dimension of the feature vector, n s The number of the feature vectors;
arranging the characteristic vectors extracted from the target domain image according to columns to obtain a target domain data characteristic matrix
Figure BDA0002084313960000022
Where m is the dimension of the feature vector, n t The number of the feature vectors;
the feature vector of the source domain image represents source domain data of the source domain image, and the feature vector of the target domain image represents target domain data of the target domain image.
Preferably, the feature extraction in step S1 is to extract feature vectors of the source domain image and the target domain image respectively through a pre-trained neural network. In the preferred embodiment, the neural network may use VGG16, *** lenet, resNet50, or the like.
Preferably, the feature extraction in step S1 is specifically to arrange the gray values of the source domain image into feature vectors and the gray values of the target domain image into feature vectors after graying the source domain image and the target domain image respectively.
Preferably, the specific step S2 is:
s2.1, judging whether the iteration is the first iteration, if not, directly carrying out the step S2.2, if so, constructing a nearest neighbor classifier and training the nearest neighbor classifier by utilizing the source domain data for predicting pseudo label information of the target domain data
Figure BDA0002084313960000031
S2.2, constructing an adjacency graph of the source domain data according to the label information of the source domain data, namely establishing adjacent edges among the same type data in the source domain data, and calculating the weights of the adjacent edges to obtain a weight matrix
Figure BDA0002084313960000032
Wherein W s A weight matrix for the source domain, calculated by:
Figure BDA0002084313960000033
wherein
Figure BDA0002084313960000034
Is the feature vector of the ith source domain image,
Figure BDA0002084313960000035
the label information corresponding to the ith source domain image is represented by the bandwidth of a Gaussian kernel, and | · | | | represents a vector two-norm;
constructing an adjacency graph of the target domain data according to the pseudo label information of the target domain data, establishing an adjacency edge between the same type data in the target domain data, and calculating the weight of the adjacency edge to obtain a weight matrix
Figure BDA0002084313960000036
Wherein W t Is the weight matrix of the target domain and is calculated by:
Figure BDA0002084313960000037
wherein
Figure BDA0002084313960000038
Is the feature vector of the ith target domain image,
Figure BDA0002084313960000039
for the pseudo label corresponding to the ith target field image, beta i Is composed of
Figure BDA00020843139600000310
The initial value of (1) is set to be 1.
In the preferred scheme, by means of the method for constructing the adjacent maps in the source domain and the target domain respectively, the structural consistency of the data from an original space to an invariant subspace can be maintained, the semantic information of the source domain data label information is utilized to construct the adjacent maps, the structure of the adjacent maps can be dynamically adjusted, so that the more essential local structure of the data is mined, and the forward migration of the knowledge in the domain is promoted.
Preferably, the specific step S3 is:
s3.1, in a common low-dimensional subspace of the source domain and the target domain, performing inter-domain data distribution alignment including edge distribution and conditional probability distribution by using a maximum mean difference criterion, specifically, minimizing the following first objective function:
Figure BDA00020843139600000311
wherein C represents the number of categories of label information, x s,c And x t,c Respectively indicating that the category of the source domain data label information belongs to the set of categories c and the category of the target domain data label information belongs to the set of categories c,
Figure BDA00020843139600000312
and
Figure BDA00020843139600000313
respectively indicating the number of data of which the source domain data belongs to the category c of the tag information and the number of data of which the destination domain data belongs to the category c of the tag information,
Figure BDA0002084313960000041
is a projection matrix, where d < m, for projecting data from m dimensions to d-dimensional subspace;
s3.2. Using weight matrix W s And W t Performing knowledge migration of data within the domain, specifically by minimizing the following second objective function:
Figure BDA0002084313960000042
s3.3, combining the first objective function and the second objective function and adding a regularization
Figure BDA0002084313960000043
The following third objective function is obtained:
Figure BDA0002084313960000044
s.t.P T XHX T P＝I
wherein λ and α are balance factors, λ > 0 and α > 0, X = [ X ] = s ,X t ]H is the centering matrix:
Figure BDA0002084313960000045
i is a unit matrix, and the unit matrix is,
Figure BDA0002084313960000046
is a full 1 column vector of value 1;
solving a projection matrix P through generalized eigen decomposition, and obtaining the expressions of the source domain data and the target domain data in the low-dimensional subspace as follows:
Figure BDA0002084313960000047
and
Figure BDA0002084313960000048
in the preferred scheme, the difference between the edge distribution and the class condition distribution of the data of the source domain and the target domain is eliminated by searching the common low-dimensional subspace of the source domain and the target domain based on the maximum mean difference criterion.
Preferably, the specific steps of step S4 are:
obtaining the source domain data with label information in the low-dimensional subspace according to the representation of the source domain data in the low-dimensional subspace
Figure BDA0002084313960000049
For training the nearest neighbor classifier and predicting target domain data in a low-dimensional space
Figure BDA00020843139600000410
Class information of
Figure BDA00020843139600000411
In the preferred scheme, a nearest neighbor classifier is trained by using source domain data with label information in a low-dimensional subspace, and the pseudo label information of the target domain data is continuously optimized in an iterative manner, so that the final label information of the target domain data is obtained.
Preferably, the specific step of step S5 includes:
s5.1. for target domain data in a low-dimensional space
Figure BDA00020843139600000412
Constructing a new adjacency graph, i.e. each target domain data point in the low-dimensional space
Figure BDA00020843139600000413
Respectively establishing adjacent edges with k target domain data points with the shortest Euclidean distances;
s5.2, based on the adjacency graph in the step S5.1, for the ith target domain data point on the granted adjacency graph
Figure BDA0002084313960000051
If and with
Figure BDA0002084313960000052
The connected k target domain data points are all connected with
Figure BDA0002084313960000053
If they belong to one category, the target domain data point
Figure BDA0002084313960000054
Is marked with a label
Figure BDA0002084313960000055
Confidence of (beta) i =1, otherwise β i =0, wherein k is a positive integer.
In the preferred scheme, different confidences are given to the target domain samples based on a sample re-weighting strategy, so that the negative migration of the knowledge in the domain is effectively inhibited.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
the method of the invention carries out inter-domain data distribution alignment comprising edge distribution and conditional probability distribution by utilizing the maximum mean difference criterion, searches a common low-dimensional subspace of a source domain and a target domain, eliminates the difference between the edge distribution and the class conditional distribution of the data of the source domain and the target domain, and leads the data distribution to be aligned;
constructing respective adjacency graphs according to the label information of the source domain data and the pseudo label information of the target domain data, keeping the structural consistency of the data from an original space to a low-dimensional subspace, and simultaneously dynamically adjusting the structure of the adjacency graphs to promote the forward migration of the knowledge in the domain; training a nearest neighbor classifier by using source domain data with label information in a low-dimensional subspace, and continuously iterating and optimizing pseudo label information of target domain data to obtain final label information of the target domain data, namely finishing cross-domain image classification;
in addition, the method of the invention endows different confidence coefficients of the pseudo labels of the target domain images by designing a sample re-weighting strategy, effectively reduces the negative transfer of the knowledge in the domain, and improves the accuracy of cross-domain image classification.
The method is based on an iterative method, and can effectively combine data distribution alignment to promote the intra-domain knowledge positive migration and inhibit the intra-domain knowledge negative migration, thereby achieving the purpose of coupling knowledge migration and greatly improving the performance of cross-domain image classification. The method solves the problems of long calculation time, insufficient classification accuracy and the like of the conventional cross-domain image classification method.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Fig. 2 is an effect diagram after performing inter-domain data distribution alignment in embodiment 2.
Fig. 3 is a diagram illustrating the distribution effect of the label information types in the low-dimensional space for the target domain in this embodiment 2.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent;
for the purpose of better illustrating the present embodiments, certain elements of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The technical solution of the present invention is further described with reference to the drawings and the embodiments.
Example 1
A cross-domain image classification method based on coupled knowledge migration, as shown in FIG. 1, includes the following steps:
s1, obtaining a source domain image and a target domain image, extracting features, respectively obtaining a source domain feature matrix as source domain data, and obtaining a target domain feature matrix as target domain data:
s1.1. Input n s A tagged source domain image and n t Opening an image of the target domain without a label; wherein n is s 、n t Is a positive integer;
s1.2, respectively extracting the features of the source domain image and the target domain image, respectively extracting the feature vectors of the source domain image and the target domain image through any one of pre-trained neural networks of VGG16, ***Ne and ResNet50, and arranging the feature vectors extracted from the source domain image in columns to obtain a source domain data feature matrix
Figure BDA0002084313960000061
Where m is the dimension of the feature vector, n s The number of the feature vectors;
arranging the feature vectors extracted from the target domain image according to columns to obtain a target domain data feature matrix
Figure BDA0002084313960000062
Where m is the dimension of the feature vector, n t The number of the feature vectors;
the feature vector of the source domain image represents source domain data of the source domain image, and the feature vector of the target domain image represents target domain data of the target domain image.
S2, respectively constructing an adjacency graph of source domain data and an adjacency graph of target domain data according to label information of the source domain data, and respectively obtaining a weight matrix of the source domain and a weight matrix of the target domain by calculating edge weights of the adjacency graphs:
s2.1, judging whether the iteration is the first iteration or not, if not, directly carrying out the step S2.2, if so, constructing a nearest neighbor classifier and training the nearest neighbor classifier by utilizing the source domain data for predicting pseudo label information of target domain data
Figure BDA0002084313960000063
S2.2, constructing an adjacency graph of the source domain data according to the label information of the source domain data, namely establishing adjacent edges among the same-class data in the source domain data, and calculating the weights of the adjacent edges to obtain a weight matrix
Figure BDA0002084313960000064
Wherein W s A weight matrix for the source domain, calculated by:
Figure BDA0002084313960000065
wherein
Figure BDA0002084313960000066
Is the feature vector of the ith source domain image,
Figure BDA0002084313960000067
the label information corresponding to the ith source domain image is represented by the bandwidth of a Gaussian kernel, and | · | | | represents a vector two-norm;
constructing an adjacency graph of the target domain data according to the pseudo label information of the target domain data, establishing an adjacency edge between the same type data in the target domain data, and calculating the weight of the adjacency edge to obtain a weight matrix
Figure BDA0002084313960000071
Wherein W t Is the weight matrix of the target domain and is calculated by:
Figure BDA0002084313960000072
wherein
Figure BDA0002084313960000073
Is the feature vector of the ith target domain image,
Figure BDA0002084313960000074
for the pseudo label corresponding to the ith target field image, beta i Is composed of
Figure BDA0002084313960000075
The initial value of (1) is set.
S3, performing inter-domain data distribution alignment in a common low-dimensional subspace of the source domain and the target domain, and simultaneously performing knowledge migration of intra-domain data by using a weight matrix of the source domain and a weight matrix of the target domain:
s3.1, in a common low-dimensional subspace of the source domain and the target domain, performing inter-domain data distribution alignment including edge distribution and conditional probability distribution by using a maximum mean difference criterion, specifically, minimizing the following first objective function:
Figure BDA0002084313960000076
wherein C represents the number of categories of label information, x s,c And x t,c Respectively indicating that the category of the source domain data label information belongs to the set of categories c and the category of the target domain data label information belongs to the set of categories c,
Figure BDA0002084313960000077
and
Figure BDA0002084313960000078
respectively indicating the number of data of which the source domain data belongs to the category c of the tag information and the number of data of which the destination domain data belongs to the category c of the tag information,
Figure BDA0002084313960000079
is a projection matrix, where d < m, for projecting data from m dimensions to d-dimensional subspace;
s3.2. Using weight matrix W s And W t Performing knowledge migration of data in the domain, specifically by minimizing the following second objective function:
Figure BDA00020843139600000710
s3.3, combining the first objective function and the second objective function and adding a regularization
Figure BDA00020843139600000711
The following third objective function is obtained:
Figure BDA00020843139600000712
s.t.P T XHX T P＝I
wherein λ and α are balance factors, λ > 0 and α > 0, X = [ X ] = s ,X t ]H is the centering matrix:
Figure BDA0002084313960000081
i is a unit matrix, and the unit matrix is,
Figure BDA0002084313960000082
is a full 1 column vector of value 1;
solving a projection matrix P through generalized eigen decomposition to obtain the expressions of the source domain data and the target domain data in the low-dimensional subspace, wherein the expressions are respectively as follows:
Figure BDA0002084313960000083
and
Figure BDA0002084313960000084
s4, in the low-dimensional subspace, utilizing the source domain data with the label information to train a nearest neighbor classifier, and predicting the category information of the target domain data: obtaining the source domain data with label information in the low-dimensional subspace according to the representation of the source domain data in the low-dimensional subspace
Figure BDA0002084313960000085
For training the nearest neighbor classifier and predicting target domain data in a low-dimensional space
Figure BDA0002084313960000086
Class information of
Figure BDA0002084313960000087
S5, reweighing the label information of the predicted target domain data:
s5.1. for target domain data in a low-dimensional space
Figure BDA0002084313960000088
Constructing a new adjacency graph, i.e. each target domain data point in the low-dimensional space
Figure BDA0002084313960000089
Respectively establishing adjacent edges with k target domain data points with the shortest Euclidean distance;
s5.2, based on the adjacency graph in the step S5.1, for the ith target domain data point on the granted adjacency graph
Figure BDA00020843139600000810
If and with
Figure BDA00020843139600000811
The connected k target domain data points are all connected with
Figure BDA00020843139600000812
If they belong to one category, the target domain data point
Figure BDA00020843139600000813
Of (2)
Figure BDA00020843139600000814
Confidence of (beta) i =1, otherwise β i =0, wherein k is a positive integer.
S6, judging whether the maximum iteration times is reached or not, and if not, returning to the step S2, if yes, outputting the label information of the predicted target domain image
Figure BDA00020843139600000815
Example 2
In this embodiment 2, 980 face images each of which is composed of face images of 10 persons randomly selected from the CMU PIE data set were selected, and each image was cut to 32 × 32 resolution. Further, 490 images in the C05 subset are used as a source domain data set, and 490 images in the C27 subset are used as a target domain data set to be classified.
S1, obtaining a source domain image in a source domain data set and a target domain image in a target domain data set, after graying the source domain image and the target domain image respectively, arranging gray values of the source domain image into a feature vector and arranging gray values of the target domain image into a feature vector, in this embodiment 2, gray values of each image are arranged into a feature vector, that is, each image is represented as a 1024-dimensional column vector, and correspondingly, source domain data feature matrix matrixes can be obtained respectively by arranging the images in columns
Figure BDA00020843139600000816
And a target domain feature matrix
Figure BDA00020843139600000817
S2, respectively constructing an adjacency graph of source domain data and an adjacency graph of target domain data according to label information of the source domain data, and respectively obtaining a weight matrix of the source domain and a weight matrix of the target domain by calculating edge weights of the adjacency graphs:
s2.1, judging whether the iteration is the first iteration or not, if not, directly carrying out the step S2.2, if so, constructing a nearest neighbor classifier and utilizing the source domain data with the labeled information
Figure BDA0002084313960000091
Training nearest neighbor classifiers for predicting target domain data
Figure BDA0002084313960000092
Pseudo category information of
Figure BDA0002084313960000093
S2.2. Tag information y according to source domain data s Constructing an adjacency graph of the source domain data, namely establishing an adjacency edge between homogeneous data in the source domain data, and calculating the weight of the adjacency edge to obtain a weight matrix
Figure BDA0002084313960000094
Wherein W s A weight matrix for the source domain, calculated by:
Figure BDA0002084313960000095
wherein
Figure BDA0002084313960000096
Is the feature vector of the ith source domain image,
Figure BDA0002084313960000097
for the label information corresponding to the ith source domain image, σ is the bandwidth of the gaussian kernel, which is set to 1 in this embodiment 1, and | · | | | represents a vector two-norm;
pseudo label information according to the target domain data
Figure BDA0002084313960000098
Constructing an adjacent graph of target domain data, establishing adjacent edges among the same-class data in the target domain data, and calculating the weights of the adjacent edges to obtain a weight matrix
Figure BDA0002084313960000099
Wherein W t A weight matrix for the target domain and calculated by:
Figure BDA00020843139600000910
wherein
Figure BDA00020843139600000911
Is the feature vector of the ith target domain image,
Figure BDA00020843139600000912
for the pseudo label corresponding to the ith target field image, beta i Is composed of
Figure BDA00020843139600000913
The initial value of (1) is set to be 1.
S3, performing inter-domain data distribution alignment in a common low-dimensional subspace of the source domain and the target domain, and simultaneously performing knowledge migration of intra-domain data by using a weight matrix of the source domain and a weight matrix of the target domain;
by minimizing the following objective function:
Figure BDA00020843139600000914
s.t.P T XHX T P＝I
where λ =0.1 and α =0.01 are balance factors. X = [ X = s ,X t ]H is the centering matrix:
Figure BDA00020843139600000915
I 980 is a unit matrix, 1 980 Is a full 1 column vector. Solving the above formula through generalized eigen decomposition to obtain a projection matrix
Figure BDA00020843139600000916
The dimension of the low-dimensional subspace is set to d =100 in this embodiment 2. And finally, the representation of the source domain data and the target domain data in the low-dimensional subspace is obtained:
Figure BDA0002084313960000101
and
Figure BDA0002084313960000102
s4, according to the representation of the source domain data in the low-dimensional subspace, obtaining the source domain data with label information in the low-dimensional subspace
Figure BDA0002084313960000103
For training the nearest neighbor classifier and predicting target domain data in a low-dimensional space
Figure BDA0002084313960000104
Class information of
Figure BDA0002084313960000105
S5, reweighing the label information of the predicted target domain data;
s5.1. For target domain data in a low-dimensional space
Figure BDA0002084313960000106
Constructing a new adjacency graph, i.e. each target domain data point in the low-dimensional space
Figure BDA0002084313960000107
Respectively establishing adjacent edges with k =5 target domain data points with the shortest Euclidean distance;
s5.2, based on the adjacency graph in the step S5.1, for the ith target domain data point on the granted adjacency graph
Figure BDA0002084313960000108
If and with
Figure BDA0002084313960000109
The 5 connected target domain data points are all connected with
Figure BDA00020843139600001010
If they belong to one category, the target domain data point
Figure BDA00020843139600001011
Is markedLabel (Bao)
Figure BDA00020843139600001012
Confidence of (beta) i =1, otherwise beta i ＝0。
S6, judging whether the maximum iteration number is reached to 10, if not, returning to the step S2, and if so, outputting the label information of the predicted target domain image
Figure BDA00020843139600001013
Thereby obtaining a classification result.
In this embodiment 2, the experimental platform is MATLAB R2017a software on WIN10 system, and the CPU model is Intel i7-6700k @4.00ghz. The experimental result diagram of this embodiment 2 is shown in fig. 2 and 3, where in fig. 2, a circle represents source domain data, a pentagon represents target domain data, and a left diagram of fig. 2 is an original distribution of the source domain data and the target domain data, and it can be obviously seen that the distribution of the source domain data and the target domain data has a great difference; the right diagram of fig. 2 is data distribution after inter-domain data distribution alignment is performed on source domain data and data target domain data in a low-dimensional subspace in the method of embodiment 2, and it can be obviously seen that the difference of the source domain data and the target domain data distribution is greatly reduced. In fig. 3, labels of different shapes represent different label information categories, for a total of ten label information categories. The left sub-graph in fig. 3 is the distribution of each label information of the original target domain data, and the category distribution is relatively disordered; fig. 3 is a right sub-graph of the distribution of the target domain data in the low-dimensional subspace obtained by the method of this embodiment 2, which shows that the category distribution of the target domain data is clear, and each data with the same tag information forms a cluster. This demonstrates the high efficiency of the process of the invention.
The terms describing positional relationships in the drawings are for illustrative purposes only and are not to be construed as limiting the patent;
it should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.

Claims (7)

1. A cross-domain image classification method based on coupling knowledge migration is characterized by comprising the following steps:
s1, obtaining a source domain image and a target domain image, extracting characteristics, respectively obtaining a source domain characteristic matrix as source domain data, and obtaining a target domain characteristic matrix as target domain data;
s2, respectively constructing an adjacency graph of source domain data and an adjacency graph of target domain data according to label information of the source domain data, and respectively obtaining a weight matrix of the source domain and a weight matrix of the target domain by calculating edge weights of the adjacency graphs;
s3, performing inter-domain data distribution alignment in a common low-dimensional subspace of the source domain and the target domain, and performing knowledge migration of intra-domain data by using a weight matrix of the source domain and a weight matrix of the target domain;
s4, in the low-dimensional subspace, training a nearest neighbor classifier by using source domain data with label information, and predicting the label information of target domain data;
s5, reweighing the label information of the predicted target domain data;
s6, judging whether the maximum iteration number is reached, if not, returning to the step S2, and if so, outputting label information of the predicted target domain image so as to obtain a classification result;
the S3 comprises the following specific steps:
s3.1, in a common low-dimensional subspace of the source domain and the target domain, performing inter-domain data distribution alignment including edge distribution and conditional probability distribution by using a maximum mean difference criterion, specifically, minimizing the following first objective function:
Figure FDA0003962172090000011
wherein C represents the number of categories of label information, x s,c And x t,c Respectively indicating that the category of the source domain data label information belongs to the set of categories c and the category of the target domain data label information belongs to the set of categories c,
Figure FDA0003962172090000012
and
Figure FDA0003962172090000013
respectively indicating the number of data of which the source domain data belongs to the category c of the tag information and the number of data of which the destination domain data belongs to the category c of the tag information,
Figure FDA0003962172090000014
is a projection matrix, where d < m, for projecting data from m dimensions into a d-dimensional subspace, m representing the dimensions of the eigenvectors;
Figure FDA0003962172090000015
the feature vector of the ith source domain image is obtained; n is s Number of source domain images for which a label is applied; n is t Representing the number of unlabeled target domain images;
Figure FDA0003962172090000016
the feature vector of the ith target domain image is obtained;
s3.2. Using weight matrix W s And W t Performing knowledge migration of data within the domain, specifically by minimizing the following second objective function:
Figure FDA0003962172090000021
s3.3, combining the first objective function and the second objective function and adding a regularization
Figure FDA0003962172090000022
The following third objective function is obtained:
Figure FDA0003962172090000023
s.t.P T XHX T P＝I
wherein λ and α are balance factors, λ > 0 and α > 0, X = [ X ] = s ,X t ]H is the centering matrix:
Figure FDA0003962172090000024
i is an identity matrix and is a matrix of the identity,
Figure FDA0003962172090000025
is a full 1-column vector of value 1; x s Representing a source domain data feature matrix, X t Representing a target domain data feature matrix;
solving a projection matrix P through generalized eigen decomposition, and obtaining the expressions of the source domain data and the target domain data in the low-dimensional subspace as follows:
Figure FDA0003962172090000026
and
Figure FDA0003962172090000027
2. the cross-domain image classification method based on coupling knowledge migration according to claim 1, wherein the specific steps of the step S1 are:
s1.1. Input n s A source domain image with label and n t Opening an image of the target domain without a label; wherein n is s 、n t Is a positive integer;
s1.2, respectively extracting the features of the source domain image and the target domain image, and arranging the feature vectors obtained by extracting the source domain image in columns to obtain a source domain data feature matrix
Figure FDA0003962172090000028
Wherein m is the dimension of the feature vector;
arranging the characteristic vectors extracted from the target domain image according to columns to obtain a target domain data characteristic matrix
Figure FDA0003962172090000029
Wherein m is the dimension of the feature vector;
the feature vector of the source domain image represents source domain data of the source domain image, and the feature vector of the target domain image represents target domain data of the target domain image.
3. The method for cross-domain image classification based on coupling knowledge migration according to claim 2, wherein the feature extraction in step S1 is to extract feature vectors of the source domain image and the target domain image respectively through a pre-trained neural network.
4. The cross-domain image classification method based on coupling knowledge migration according to claim 2, wherein the feature extraction in step S1 specifically is to arrange gray values of the source domain image into feature vectors and gray values of the target domain image into feature vectors after graying the source domain image and the target domain image respectively.
5. The cross-domain image classification method based on coupling knowledge migration according to claim 3 or 4, wherein the specific step S2 is:
s2.1, judging whether the iteration is the first iteration or not, if not, directly carrying out the step S2.2, if so, constructing a nearest neighbor classifier and training the nearest neighbor classifier by utilizing the source domain data for predicting pseudo label information of target domain data
Figure FDA0003962172090000031
S2.2. According to the source domain data markLabeling information, constructing an adjacency graph of source domain data, establishing an adjacency edge between the same type data in the source domain data, and calculating the weight of the adjacency edge to obtain a weight matrix
Figure FDA0003962172090000032
Wherein W s A weight matrix for the source domain, calculated by:
Figure FDA0003962172090000033
wherein
Figure FDA0003962172090000034
Is the feature vector of the ith source domain image,
Figure FDA0003962172090000035
the label information corresponding to the ith source domain image is represented by the bandwidth of a Gaussian kernel, and | · | | | represents a vector two-norm;
constructing an adjacency graph of the target domain data according to the pseudo label information of the target domain data, establishing an adjacency edge between the same type data in the target domain data, and calculating the weight of the adjacency edge to obtain a weight matrix
Figure FDA0003962172090000036
Wherein W t Is the weight matrix of the target domain and is calculated by:
Figure FDA0003962172090000037
wherein
Figure FDA0003962172090000038
Is the feature vector of the ith target domain image,
Figure FDA0003962172090000039
for the pseudo label information, beta, corresponding to the ith target field image i Is composed of
Figure FDA00039621720900000310
The initial value of (1) is set.
6. The cross-domain image classification method based on coupling knowledge migration according to claim 5, wherein the specific steps of the step S4 are as follows:
obtaining the source domain data with label information in the low-dimensional subspace according to the representation of the source domain data in the low-dimensional subspace
Figure FDA00039621720900000311
For training the nearest neighbor classifier and predicting target domain data in a low-dimensional space
Figure FDA00039621720900000312
Pseudo tag information of
Figure FDA00039621720900000313
7. The cross-domain image classification method based on coupling knowledge migration according to claim 6, wherein the specific steps of the step S5 comprise:
s5.1. For target domain data in a low-dimensional space
Figure FDA0003962172090000041
Constructing a new adjacency graph, and setting each target domain data point in a low-dimensional space
Figure FDA0003962172090000042
Respectively establishing adjacent edges with k target domain data points with the shortest Euclidean distance; i =1,2, \8230;, n t ；
S5.2. Based onFor the adjacency graph stated in step S5.1, for the ith target domain data point on the granted adjacency graph
Figure FDA0003962172090000043
If and with
Figure FDA0003962172090000044
The k target domain data points connected with
Figure FDA0003962172090000045
If the same belongs to a category, the target domain data point
Figure FDA0003962172090000046
Is marked with a label
Figure FDA0003962172090000047
Weight of beta i =1, otherwise β i =0, wherein k is a positive integer.
CN201910482559.5A 2019-06-04 2019-06-04 Cross-domain image classification method based on coupling knowledge migration Active CN110378366B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910482559.5A CN110378366B (en) 2019-06-04 2019-06-04 Cross-domain image classification method based on coupling knowledge migration

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910482559.5A CN110378366B (en) 2019-06-04 2019-06-04 Cross-domain image classification method based on coupling knowledge migration

Publications (2)

Publication Number Publication Date
CN110378366A CN110378366A (en) 2019-10-25
CN110378366B true CN110378366B (en) 2023-01-17

Family

ID=68249801

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910482559.5A Active CN110378366B (en) 2019-06-04 2019-06-04 Cross-domain image classification method based on coupling knowledge migration

Country Status (1)

Country Link
CN (1) CN110378366B (en)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110781970B (en) * 2019-10-30 2024-04-26 腾讯科技（深圳）有限公司 Classifier generation method, device, equipment and storage medium
CN111312403A (en) * 2020-01-21 2020-06-19 山东师范大学 Disease prediction system, device and medium based on instance and feature sharing cascade
CN111814871B (en) * 2020-06-13 2024-02-09 浙江大学 Image classification method based on reliable weight optimal transmission
CN111929053B (en) * 2020-07-07 2022-02-01 中国矿业大学 DA-RVFLN-based pneumatic control valve field adaptive fault diagnosis method
CN112085085B (en) * 2020-09-03 2024-02-23 哈尔滨工业大学 Multi-source migration learning method based on graph structure
CN112733970B (en) * 2021-03-31 2021-06-18 腾讯科技（深圳）有限公司 Image classification model processing method, image classification method and device
CN113420775B (en) * 2021-03-31 2024-03-29 中国矿业大学 Picture classification method based on nonlinearity self-adaptive subdomain field adaptation under extremely small quantity of training samples
CN113469405B (en) * 2021-05-18 2024-04-02 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) Subimage segmentation-based cross-city tail gas migration prediction method, system and medium
CN113657651B (en) * 2021-07-27 2024-07-09 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) Deep migration learning-based diesel vehicle emission prediction method, medium and equipment
CN113836996B (en) * 2021-08-10 2024-02-02 中国地质大学（武汉） Hyperspectral remote sensing image part migration method and hyperspectral remote sensing image part migration system
CN114743043B (en) * 2022-03-15 2024-04-26 北京迈格威科技有限公司 Image classification method, electronic device, storage medium and program product
CN115393362B (en) * 2022-10-28 2023-02-03 中南大学 Method, equipment and medium for selecting automatic glaucoma recognition model
CN117216507A (en) * 2023-10-08 2023-12-12 深圳大学 Deep neural network model mobility measurement method based on geographic partition

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103971129A (en) * 2014-05-27 2014-08-06 浙江大学 Classification method and device based on learning image content recognition in cross-data field subspace
CN107895177A (en) * 2017-11-17 2018-04-10 南京邮电大学 A kind of migration classification learning method for keeping image classification sparsity structure
CN109389174A (en) * 2018-10-23 2019-02-26 四川大学 A kind of crowd massing Sensitive Image Detection Method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103971129A (en) * 2014-05-27 2014-08-06 浙江大学 Classification method and device based on learning image content recognition in cross-data field subspace
CN107895177A (en) * 2017-11-17 2018-04-10 南京邮电大学 A kind of migration classification learning method for keeping image classification sparsity structure
CN109389174A (en) * 2018-10-23 2019-02-26 四川大学 A kind of crowd massing Sensitive Image Detection Method

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
迁移学习方法研究及其在跨领域数据分类中的应用;覃姜维;《中国博士学位论文全文数据库 信息科技辑》;20111215(第12期);I140-39 *

Also Published As

Publication number Publication date
CN110378366A (en) 2019-10-25

Similar Documents

Publication Publication Date Title
CN110378366B (en) Cross-domain image classification method based on coupling knowledge migration
Liu et al. Connectionist temporal classification with maximum entropy regularization
Wang et al. Exploring domain-invariant parameters for source free domain adaptation
CN110532542B (en) Invoice false invoice identification method and system based on positive case and unmarked learning
US20220382553A1 (en) Fine-grained image recognition method and apparatus using graph structure represented high-order relation discovery
CN110224987B (en) Method for constructing network intrusion detection model based on transfer learning and detection system
Xiao et al. Mcapsnet: Capsule network for text with multi-task learning
US20220076074A1 (en) Multi-source domain adaptation with mutual learning
CN113326731A (en) Cross-domain pedestrian re-identification algorithm based on momentum network guidance
CN112906770A (en) Cross-modal fusion-based deep clustering method and system
CN110751027B (en) Pedestrian re-identification method based on deep multi-instance learning
Shah et al. Max-margin contrastive learning
CN113554100B (en) Web service classification method for enhancing attention network of special composition picture
CN112861929B (en) Image classification method based on semi-supervised weighted migration discriminant analysis
Liu et al. Semi-supervised change detection based on graphs with generative adversarial networks
CN114863091A (en) Target detection training method based on pseudo label
CN114048295A (en) Cross-modal retrieval method and system for data processing
CN115114409A (en) Civil aviation unsafe event combined extraction method based on soft parameter sharing
Zhang et al. Projected-prototype based classifier for text categorization
CN114792114B (en) Unsupervised domain adaptation method based on black box multi-source domain general scene
Sener et al. Unsupervised transductive domain adaptation
CN113392876B (en) Small sample image classification method based on graph neural network
CN114676755A (en) Non-supervision domain self-adaptive classification method based on graph convolution network
Yu et al. Graph-structure based multi-label prediction and classification for unsupervised person re-identification
CN113344189A (en) Neural network training method and device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant