CN112651492A - Self-connection width graph convolution neural network model and training method thereof - Google Patents
Self-connection width graph convolution neural network model and training method thereof Download PDFInfo
- Publication number
- CN112651492A CN112651492A CN202011610969.2A CN202011610969A CN112651492A CN 112651492 A CN112651492 A CN 112651492A CN 202011610969 A CN202011610969 A CN 202011610969A CN 112651492 A CN112651492 A CN 112651492A
- Authority
- CN
- China
- Prior art keywords
- self
- neural network
- graph
- order
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 102
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000003062 neural network model Methods 0.000 title claims abstract description 44
- 230000004927 fusion Effects 0.000 claims abstract description 39
- 238000007781 pre-processing Methods 0.000 claims abstract description 35
- 238000011176 pooling Methods 0.000 claims abstract description 21
- 239000011159 matrix material Substances 0.000 claims description 89
- 238000013527 convolutional neural network Methods 0.000 claims description 32
- 230000004913 activation Effects 0.000 claims description 17
- 238000011478 gradient descent method Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 abstract description 30
- 238000012360 testing method Methods 0.000 abstract description 8
- 238000004364 calculation method Methods 0.000 description 11
- 230000002829 reductive effect Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 241000689227 Cora <basidiomycete fungus> Species 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 101001013832 Homo sapiens Mitochondrial peptide methionine sulfoxide reductase Proteins 0.000 description 1
- 102100031767 Mitochondrial peptide methionine sulfoxide reductase Human genes 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012821 model calculation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- QVRVXSZKCXFBTE-UHFFFAOYSA-N n-[4-(6,7-dimethoxy-3,4-dihydro-1h-isoquinolin-2-yl)butyl]-2-(2-fluoroethoxy)-5-methylbenzamide Chemical compound C1C=2C=C(OC)C(OC)=CC=2CCN1CCCCNC(=O)C1=CC(C)=CC=C1OCCF QVRVXSZKCXFBTE-UHFFFAOYSA-N 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a self-connection width chart convolution neural network model and a training method thereof, wherein the self-connection width chart convolution neural network model sequentially comprises an input layer, a self-connection high-order chart convolution layer for simultaneously capturing multi-order neighborhood information, an SP information fusion pooling layer for fusing the multi-order neighborhood information and a softmax function multi-classification output layer, and the training method corresponding to the self-connection width chart convolution neural network model is an efficient model training method for firstly carrying out preprocessing to obtain preprocessing characteristics and then inputting the preprocessing characteristics into training. When the model and the training method thereof are applied to actual classification tests, the self-connection high-order graph convolution layer is adopted to increase the width of the model, reduce the depth of the model, reduce the number of parameters and complexity, meanwhile, the multi-order neighborhood node information is gathered, and the self node is given higher weight, so that the model receptive field is widened, the over-fitting risk is avoided, and meanwhile, the learning capability, the stability and the classification precision of the model are further improved.
Description
Technical Field
The invention relates to the technical field of image processing and deep learning, in particular to a self-connection width map convolution neural network model and a training method thereof.
Background
With the continuous deepening of the graph convolution neural network research, various high-order graph convolution network models capable of gathering more and richer node information are paid attention to due to the improvement of the classification performance of the high-order graph convolution network models in order to improve the receptive field and the learning capacity of the models. The existing high-order graph convolution network model adopts a convolution mode from a first-order graph to a P-order graph, different-order convolutions use different weight parameters, the relation between high-order nodes is learned by utilizing two high-order graph convolution layers, the high-order graph convolutions gather neighborhood node information of different distances, after the neighborhood information of different distances is gathered by each high-order graph convolution, the neighborhood information is spliced by utilizing column connection, and finally the neighborhood information is fused through a full connection layer. Although the high-order graph convolution network learns the relationship between high-order nodes by using a plurality of high-order graph convolution layers with different weights, the receptive field of the model is widened to a certain extent, and the classification performance of the model is also improved, the high-order graph convolution network increases the calculation complexity and the parameter number along with the increase of the layer number due to the different convolution weights of different orders, the model becomes more complex and harder to train, and the risk of overfitting is increased along with the increase of the parameter number of the model.
Therefore, it is very meaningful how to improve the existing high-order graph convolution network, so that the classification performance is improved, the computation complexity and the parameter quantity can be reduced, and the risk of overfitting is avoided.
Disclosure of Invention
The invention aims to reduce the calculation complexity and parameter quantity of the conventional high-order graph convolution network, avoid the risk of overfitting and further improve the classification performance of the high-order graph convolution network.
In order to achieve the above objective, it is necessary to provide a self-connected width map convolution neural network model and a training method thereof.
In a first aspect, an embodiment of the present invention provides a self-connected width map convolutional neural network model, where the self-connected width map convolutional neural network model sequentially includes an input layer, a self-connected high-order map convolutional layer, an information fusion pooling layer, and an output layer;
the input layer is used for receiving graph characteristics of a training data set;
the self-connection high-order graph convolution layer is used for performing zero-order to k-order self-connection graph convolution operation according to the graph characteristics to obtain graph convolution data;
the information fusion pooling layer is used for carrying out zero-order to k-order feature fusion according to the graph convolution data to obtain fusion data;
and the output layer is used for outputting a model result according to the fusion data.
Further, the self-connected high-order graph convolution layer is obtained by introducing new self-connection in any order graph convolution layer.
Further, the self-join high-order graph convolution layer includes zero-order graph convolution to k-order graph convolution based on weight sharing, which is expressed as:
where X is the input matrix of the graph, w is the parameter matrix,is the regularized adjacency matrix of the graph, k is the highest order of graph convolution,
go toStep one, the output layer HGCN of the self-connection width map convolution neural network modelSLCan be expressed as:
wherein, σ (-) is an activation function, SP (-) is an information fusion function, and softmax (-) is a multi-classification output function.
Further, the sigma activation function is a ReLU nonlinear activation function.
Further, the information fusion pooling layer adopts SP summation information fusion pooling, and the calculation formula is as follows:
in a second aspect, an embodiment of the present invention provides a training method for a self-connected width map convolutional neural network model, where the training method includes:
obtaining an input matrix and a regularization adjacency matrix of the graph of the self-connection width graph convolution neural network model according to a training data set;
performing characteristic preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain preprocessing characteristics;
inputting the preprocessing features into the self-connection width chart convolution neural network model, and performing feature training to obtain a training result.
Further, the step of obtaining the input matrix and the regularized adjacency matrix of the graph of the self-connected width graph convolution neural network model according to the training data set includes:
acquiring the training data set and determining the type of the training data set;
and selecting a specific method to obtain an input matrix and a regularized adjacency matrix of the graph of the self-connection width graph convolution neural network model according to the type of the training data set.
Further, the step of performing feature preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain a preprocessed feature includes:
summing the regularized adjacency matrixes of the graphs of all different orders to obtain a preprocessed adjacency matrix;
and performing product on the preprocessing adjacent matrix and the input matrix of the graph to obtain preprocessing characteristics.
Further, the step of inputting the preprocessed features into the self-connected width map convolution neural network model to perform feature training to obtain a training result includes:
randomly initializing a parameter matrix of the self-connection width map convolution neural network model;
inputting the preprocessing characteristics into the self-connection width chart convolution neural network model, and training by adopting a loss function and a gradient descent method to obtain a convergent parameter matrix.
The application provides a self-connection width map convolution neural network model and a training method thereof, and the self-connection width map convolution neural network model and the training method thereof realize that the self-connection width map convolution neural network model only with an input layer, a self-connection high-order map convolution layer, an SP information fusion pooling layer and a softmax function output layer is adopted, and the self-connection width map convolution neural network model is combined with a feature preprocessing method before the model training and accordingly obtains an accurate classification effect. Compared with the prior art, the model and the training method thereof not only gather more abundant node information among more levels of adjacent domains by adopting the high-level graph convolution which introduces self-connection and gives higher weight to self nodes in actual classification application, and well improve the learning capability and classification precision of the model, but also effectively reduce the parameter quantity, reduce the complexity and training difficulty of the model and avoid the risk of over-fitting by designing a layer of self-connection high-level graph convolution and adopting a weight sharing mechanism among different levels of graph convolution.
Drawings
FIG. 1 is a schematic diagram of an application scenario of a self-connected width map convolutional neural network model and a training method thereof according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a self-connected width map convolutional neural network model;
FIG. 3 is a schematic diagram of a self-connected width map convolutional neural network model based on SP information fusion pooling layers;
FIG. 4 is a schematic flow diagram of a training method of the self-connected width map convolutional neural network model of FIG. 3;
FIG. 5 is a schematic flow chart of the training set data obtained in step S11 in FIG. 4 being preprocessed to obtain the input matrix and the regularized adjacency matrix of the graph;
FIG. 6 is a schematic flow chart of the step S12 in FIG. 4 for obtaining pre-processing features according to the input matrix and the regularized adjacency matrix of the graph;
FIG. 7 is a schematic flowchart of the step S13 of FIG. 4 of inputting the preprocessed features into the connected width map convolutional neural network model for feature training;
fig. 8 is an internal structural diagram of a computer device in the embodiment of the present invention.
Detailed Description
In order to make the purpose, technical solution and advantages of the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments, and it is obvious that the embodiments described below are part of the embodiments of the present invention, and are used for illustrating the present invention only, but not for limiting the scope of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The self-connection width graph convolution neural network provided by the invention is an improvement of the existing high-order graph convolution neural network, and the model and the training method thereof can be applied to a terminal or a server as shown in figure 1. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server can be implemented by an independent server or a server cluster formed by a plurality of servers. The server may employ a self-connected breadth map convolutional neural network model (HGCN)SL) And the corresponding training method completes the prior artAnd the high-order graph convolution neural network model can complete a classification task, and a classification prediction result of the model is sent to the terminal for a user of the terminal to check and analyze.
In one embodiment, as shown in fig. 2, a self-connected width map convolutional neural network model is provided that includes an input layer 1, a self-connected high-order map convolutional layer 2, an information fusion pooling layer 3, and an output layer 4; the input layer 1 is used for receiving graph characteristics of a training data set; the self-connection high-order graph convolution layer 2 is used for performing zero-order to k-order self-connection graph convolution operation according to the graph characteristics to obtain graph convolution data; the information fusion pooling layer 3 is used for carrying out zero-order to k-order feature fusion according to the graph convolution data to obtain fusion data; and the output layer 4 is used for outputting a model result according to the fusion data.
The number of the self-connection high-order graph convolution layers and the number of the information fusion pooling layers are only 1, namely the structure of the self-connection width graph convolution neural network model is as follows: the input layer 1 is connected with the self-connection high-order graph convolution layer 2, the self-connection high-order graph convolution layer 2 is connected with the information fusion pooling layer 3, and the information fusion pooling layer 3 is connected with the output layer 4 which adopts the softmax function to carry out multi-classification output. The self-connection high-order graph convolutional layer is a new self-connection high-order graph convolutional layer introduced into any order graph convolutional layer, namely, a regularization adjacent matrix of any order graph is added with a unit matrix of the same dimension to serve as a new graph regularization adjacent matrix input model. It should be noted that, considering that the element value of the diagonal position of the adjacency matrix where the self node has no self-connection (that is, the self node and the self node have one edge) is 0, when the adjacency matrix is input into the model for training, the classification effect of the model may be affected by omitting the self node information, the regularized adjacency matrix of the original graph in this embodiment contains self-connection, and the self-connection emphasized in this application means that the self-connection is introduced again on the basis of the regularized adjacency matrix of the original self-connection, so as to further increase the weight of the self node, that is:
wherein ,for the ith order regularized adjacency matrix containing self-joins,for zero order regularized adjacency matrices containing self-joins, I is ANDA co-dimensional identity matrix. The effect as when second order graph convolution introduces new self-joins isOf course, it is also possible to choose to introduce self-concatenation in other orders of the graph convolution, and so on, and not described in detail here. In addition, the method of introducing self-join in the present embodiment is also applicable to other convolution models of higher-order graphs in principle.
In the embodiment, by constructing the network architecture with only one layer of self-connection high-order graph convolution layer, the parameter quantity and the calculation quantity of the model are reduced, the model training efficiency is effectively improved, in addition, the influence of the self node characteristics on the classification prediction is considered to be larger, and the classification effect of the model is further improved by introducing the self-connection mode to increase the weight of the self node information.
The output of the output layer of the self-connection width map convolution neural network model is assumed to be HGCNSLThen, it can be expressed as:
where X is the input matrix of the graph, w is the parameter matrix,is the regularized adjacency matrix of the graph, k is the highest order of graph convolution,σ (-) is an activation function, SP (-) is an information fusion function, and softmax (-) is a multi-class output function.
The self-connected high-order graph convolution layer in the embodiment comprises zero-order graph convolution to k-order graph convolution based on weight sharing, so that the parameter quantity of the self-connected high-order graph convolution is consistent with the parameter quantity of the first-order graph convolution, namelyThe self-join high-order graph convolution is not only realized by introducing new self-join in any order graph convolutionGiving higher weight to self node and further through first-order to k-order graph convolutionThe high-order neighborhood information of the nodes is captured, and meanwhile, the neighborhood information of different orders is gathered. In order to compensate the learning ability of the one-layer framework, in practical application, a larger k value, that is, a larger width can be selected according to requirements, that is, the receptive field of the model is increased by increasing the width (increasing the order) instead of increasing the depth (increasing the number of layers), so that the learning ability of the model is enhanced. The order k of the self-connected high-order graph convolutional layer can be one of the orders of the second order and above, or a combination of any plurality of orders. When k is 2, the adopted self-connection width graph convolution neural network model is HGCN with mixed 1-order and 2-order neighborhoodsSL-2, the formula is as follows:
when k is 3, the adopted self-connection width graph convolution neural network model is the HGCN with mixed 1, 2 and 3-order neighborhoodsSL-3, the formula is as follows:
when k is n, the adopted self-connection width graph convolution neural network model is HGCN mixed with 1 st-n th order neighborhoodSL-n, the formula:
in the model, the same weight parameter is adopted in each order neighborhood of the convolution layer of the same graph to realize weight sharing and parameter quantity reduction, and the selection of the parameter W in the formulas (1) to (4) is embodied.
When the method is actually applied to large-scale classification training, the calculation is needed firstlyDue to the fact thatIs generally a sparse matrix with m nonzero elements, and adopts a weight sharing mechanism based on self-connection high-order graph convolution and adopts multiplication from right to left to calculateE.g. when k is 2, useBy multiplication to obtainIn the same way, the method for preparing the composite material,and so on throughLeft-multiplying k-1 order graph convolutionCalculating convolution of the k-th order graph, i.e.The calculation method effectively reduces the calculation complexity. In addition, because different order graph convolutions adopt a weight sharing mechanism, the parameter quantity of the self-connection high order graph convolution and the parameter quantity of the first order graph convolution are the same, and the high efficiency of self-connection high order graph convolution calculation is guaranteed to a certain extent.
Preferably, the sigma activation function may be a ReLU (-) nonlinear activation function.
The ReLU activation function is used for non-linear processing. Because the linear model has insufficient expressive force and some data features are not necessarily linearly separable, in order to solve the problem, an activation function is used for performing nonlinear processing after the information fusion pooling layer, commonly used nonlinear activation functions include sigmoid, tanh, ReLU, ElU, prilu and the like, the activation functions can be used, but have advantages and disadvantages, and can be selected according to the characteristics of data to be processed in practical application, while the most ReLU functions in the neural network are used in the embodiment, and the formula is defined as follows:
f(x)=max(0,x),
that is, values equal to or greater than 0 are retained, and all the remaining values smaller than 0 are directly rewritten to 0. By mapping the values in the feature map generated after convolution through the method, the irrelevant data can be directly abandoned during feature extraction, and the operation is more convenient.
The nonlinear activation function can improve the expression capability of the model, but is not very useful for the graph classification task, and particularly for the self-connected width graph convolutional neural network model with only one layer of self-connected high-order graph convolutional layer in the embodiment, the nonlinear activation function can be omitted, the complexity of model calculation is further reduced, but a little precision can be lost, and the influence on the classification precision of the model is not large overall.
Preferably, the information fusion pooling layer in the formula (1) of the present invention adopts SP summation information fusion pooling to fuse node information of different order neighborhoods from zero order to k order, and the specific calculation formula is as follows:
the corresponding self-connected high-order graph volume adopting SP information fusion can gather more and richer neighborhood information to obtain global graph structure information, and simultaneously considers more important factors of self nodes in classification prediction, as shown in FIG. 3, the expression of the model is as follows:
wherein, H is the output value of the self-connection high-order graph convolution layer, namely the input value of the softmax function output layer of the model.
The information fusion mode in the above embodiment is described by a specific third-order embodiment, and the higher-order case is similar. Suppose the order k of the neighborhood is 3 and the zeroth order neighborhood is H0First order neighborhood is H1The second order neighborhood is H2Third order neighborhood is H3Then, the SP summation information fusion process is:
The self-connected high-order graph convolution algorithm adopting SP information fusion is implemented as follows:
information fusion: h0k=SP(H0,H1,H2,…,Hk)
Nonlinear activation: h ═ σ (H)0k)
In the embodiment, the graph network is firstly input into the self-connection high-order graph convolution for the algorithm processing, then the SP summation information fusion is used for mixing the characteristics from zero order to high order of different neighborhoods, and the characteristic is input into the softmax function output layer after nonlinear activation to obtain the classification probability result.
In one embodiment, as shown in fig. 4, the steps of the training method of any of the above self-connected width map convolutional neural network models include:
s11, obtaining an input matrix and a regularized adjacency matrix of the self-connection width graph convolution neural network model according to a training data set;
as shown in fig. 5, the step S11 of obtaining the input matrix and the regularized adjacency matrix of the graph of the self-connected width graph convolutional neural network model according to the training data set includes:
s111, acquiring the training data set and determining the type of the training data set;
the training data set is selected according to actual classification requirements, for example, the data set for text classification may be selected from R52 and R8 of Reuters21578, 20-Newsgroups (20NG), ohsumed (oh), and moviereview (mr), the semi-supervised classification may be selected from Cora, cieseer, Pubmed, and the multi-view classification may be selected from Modelnet10 and Modelnet40, and the data sets for each classification task are different in content and type.
And S112, selecting a specific method to obtain an input matrix and a regularized adjacency matrix of the graph of the self-connection width graph convolution neural network model according to the type of the training data set.
The types of the training data sets include the text data set, the semi-supervised classification data set, the multi-view classification data set and the like, and for each type of training set data, when the self-connection width graph convolution neural network model is used, corresponding preprocessing is carried out to obtain an input matrix of a graph of the model and a regularized adjacency matrix of the graph. If text classification is needed, the corpus data including documents and titles needs to be processed to construct a corresponding corpus text map network, and an input matrix of a graph and a regularized adjacency matrix of the graph used for model training are obtained according to the corpus text map network. For data sets in other situations, such as semi-supervised data sets or multi-view classification data sets, corresponding preprocessing methods are provided, and when the model in the example is used for classification, the data set corresponding to the task is converted into the input matrix of the graph and the regularized adjacency matrix of the graph according to the conventional method corresponding to the classification task type. In the following embodiments of the present application, the semi-supervised data set shown in table 1 is taken as an example for relevant description.
TABLE 1 semi-supervised classification classical dataset information Table
Data set | Cora | Citeseer | Pubmed |
Number of nodes | 2708 | 3327 | 19717 |
Number of edges | 5429 | 4732 | 44338 |
Characteristic word | 1433 | 3703 | 500 |
Number of classification | 7 | 6 | 3 |
Training node | 140 | 120 | 60 |
Verifying node | 500 | 500 | 500 |
Test node | 1000 | 1000 | 1000 |
Rate of labels | 0.052 | 0.036 | 0.003 |
S12, performing feature preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain preprocessing features;
as shown in fig. 6, the step S12 of performing feature preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain a preprocessed feature includes:
s121, summing the regularized adjacency matrixes of the graphs with different orders to obtain a preprocessed adjacency matrix;
the preprocessing adjacency matrix is a great advantage of the self-connection width graph convolution neural network model in calculation, and because only one layer of self-connection high-order graph convolution layer is constructed in the method, the SP summation information fusion calculation formula (5) can be optimized as follows:
and due to regularization of adjacency matricesAnd(i.e., I) are known, then are readily obtained by matrix multiplicationIs easily obtained by matrix additionIt can be seen thatIs obtained by element-by-element operation, the spatial position sum of each elementIs the same, is an operator that preserves the graph topology, i.e., canThe adjacency matrix which is used as a preprocessed graph is used for subsequent model training.
And S122, performing product on the preprocessing adjacent matrix and the input matrix of the graph to obtain preprocessing characteristics.
Obtaining a preprocessed adjacent matrix through the preprocessingThen, since the input matrix X is known, it is possible to directly connectThe pre-processed feature matrix is sent to the model for training, so that the complexity and difficulty of machine training are reduced to a certain extent, and the high efficiency of model training is ensured.
And S13, inputting the preprocessing features into the self-connection width chart convolution neural network model, and performing feature training to obtain a training result.
As shown in fig. 7, the step S13 of inputting the preprocessed features into the self-connected width map convolutional neural network model to perform feature training to obtain a training result includes:
s131, randomly initializing a parameter matrix of the self-connection width map convolution neural network model;
the method for randomly initializing the model parameter matrix comprises the following steps: the weight is initialized by Gaussian with Gaussian distribution, by Xavier with uniform distribution, and by MSRA with mean 0 and variance 2/n. In the embodiment, when the parameter matrix of the self-connection width map convolution neural network model is randomly initialized, the parameter matrix can be selected according to actual classification requirements by combining the three initialization characteristics, and the application effect of the model cannot be influenced.
S132, inputting the preprocessing characteristics into the self-connection width chart convolution neural network model, and training by adopting a loss function and a gradient descent method to obtain a converged parameter matrix;
the training process of the self-connection width chart convolution neural network model is as follows: training the preprocessing characteristics obtained by preprocessing the effective characteristic data in the training data set, obtaining a classification result through forward propagation, calculating the cross entropy through a loss function, and training until convergence by using a gradient descent algorithm of updating a parameter matrix through backward propagation to obtain the parameter matrix during convergence.
In this embodiment, the training data set used for model training is a training data set of semi-supervised classification nodes, and according to the characteristics of the training data set, the selected loss function is as follows:
xlfor a set of labeled vertices (nodes), M is the number of classes, YlmReal labels, Z, representing label nodeslmAnd represents the probability value between 0 and 1 predicted by softmax (input tag node). After the parameter matrix is initialized, obtaining an initial loss function value based on all training data sets, if the loss function is larger, indicating that the performance of the neural network is not good, a gradient descent method is needed, namely calculating the partial derivative of the loss function about the parameters of the neural network model, continuously adjusting and updating weight parameters, and then training until the loss function is reduced to an acceptable range, ending the whole training process of the model to obtain a converged parameter matrix, namely obtaining a more stable neural network model, and storing the currently obtained converged parameter matrix to determine the high-efficiency width graph convolutional neural network training model which can be used for subsequent classification tests. Practice of the present applicationIn the example, firstly, based on the generalization ability consideration of the model, the public reference data set is adopted for parameter training during model training, and the data set does not have repeated data, so that the workload of model training can be reduced to a certain extent, and the efficiency of model training is improved; secondly, a self-connection width graph convolution neural network model with only one layer of self-connection high-order graph convolution and a model training method for characteristic preprocessing are designed, the self-connection width graph convolution is adopted to replace the depth graph convolution, under the condition that multilayer graph convolution layers do not exist, the complexity of the model, the parameter quantity of the model and the training difficulty are reduced, the interactive relation among multiple neighborhood nodes and the importance of neighborhood information of the nodes are highlighted, and the characteristic preprocessing method during model training is combined, so that the receptive field of the model is improved, the training efficiency and the classification precision of the model are improved, the over-smooth phenomenon of the trained model is avoided, and the stability of the model is improved.
In the embodiment of the present application, classification training is performed based on a semi-supervised classification dataset and is compared with the classification precision of the existing graph convolution neural model, and the result is shown in table 2 below:
TABLE 2 HGCNSLTest precision comparison table based on same semi-supervised data set as conventional graph convolution model
Table 2 illustrates: the accuracy in the table is expressed as a percentage and the number is the average of 10 runs.
Based on the above experimental results, this embodiment proposes a self-connected width graph convolution network model HGCN with only one layer of SP information fusion pooling layer that can both simultaneously gather neighborhood node information of different orders and consider self-node convolution of self-node more important for classification prediction, and mix different neighborhood characteristicsSLThe method simplifies the conventional high-order graph convolution model, reduces the complexity of the model, reduces the parameter quantity and the training difficulty of the model, further improves the model training efficiency, and avoids the model passingFitting risk, reserving more and richer neighborhood characteristic information in classification learning, learning global graph topology, widening the receptive field, introducing new self-connection to increase the weight of self nodes, further improving model expression capability and learning capability, and based on experimental results on three groups of reference semi-supervised classification data sets, the self-connection width graph convolution network model has greater advantages in the aspects of classification precision, parameter quantity, complexity, stability and the like compared with the classification training by adopting the existing high-order graph convolution network model.
In order to study the maximum order of the self-connection width map convolution model corresponding to the three groups of data in the embodiment and determine the importance of the self-node in model classification prediction, a contrast experiment is also performed based on the classification accuracy of the self-connection width map convolution and the self-connection width map convolution of different orders. As shown in tables 3-5, it is determined through research that the maximum orders of the non-self-connected width graph convolutional neural network model based on the Pubmed, Cora and cieser data sets are respectively 21, 5 and 4, the maximum orders of the self-connected width graph convolutional neural network model based on the Pubmed, Cora and cieser data sets are respectively 21, 8 and 4, and the classification accuracy is reduced after the corresponding maximum orders are exceeded; in addition, research also shows that the classification precision and stability of the width graph convolutional neural network model with self-connection introduced are improved, and further it is determined that the fact that the nodes are endowed with higher weights by introducing new self-connection is more practical and meaningful.
TABLE 3 No self-attach HGCNSLWith HGCNSLTest precision comparison table based on Pubmed data set
Order (k) | HGCNSL(No self-ligation)/Epochs | HGCNSL(ours)/Epochs |
1 | 71.6±0.6/50 | 74.0±0.1/300 |
2 | 75.0±0.1/150 | 76.4±0.1/250 |
3 | 76.5±0.1/200 | 78.3±0.1/300 |
4 | 77.5±0.1/150 | 79.1±0.1/100 |
5 | 77.7±0.1/200 | 79.6±0.0/300 |
10 | 79.4±0.1/120 | 79.9±0.0/150 |
15 | 80.0±0.1/150 | 80.4±0.0/220 |
20 | 80.3±0.1/200 | 80.78±0.0/250 |
21 | 80.4±0.1/170 | 80.80±0.0/260 |
22 | 80.4±0.1/120 | 80.7±0.0/270 |
25 | 80.4±0.1/220 | 80.8±0.0/280 |
Table 3 illustrates: the accuracy in the table is expressed as a percentage and the number is the average of 10 runs.
TABLE 4 non-self-attaching HGCNSLWith HGCNSLTest precision comparison table based on Cora data set
Order (k) | HGCNSL (self-Linked)/Epochs | HGCNSL(ours)/Epochs |
1 | 75.2±0.1/30,000 | 74.6±0.2/25,000 |
2 | 80.0±0.2/50,000 | 78.9±0.1/50,000 |
3 | 81.2±0.1/25,000 | 81.1±0.1/45,000 |
4 | 82.0±0.1/35,000 | 81.7±0.1/45,000 |
5 | 82.6±0.1/25,000 | 82.4±0.1/25,000 |
6 | 82.2±0.1/45,000 | 82.5±0.2/20,000 |
7 | 81.9±0.1/45,000 | 82.9±0.1/50,000 |
8 | 81.7±0.1/25,000 | 83.1±0.1/30,000 |
9 | 81.7±0.0/30,000 | 83.0±0.1/35,000 |
Table 4 illustrates: the accuracy in the table is expressed as a percentage and the number is the average of 10 runs.
TABLE 5 non-self-attaching HGCNSLWith HGCNSLTest precision comparison table based on Cora data set
Order (k) | HGCNSL (self-Linked)/Epochs | HGCNSL(ours)/Epochs |
1 | 69.2±0.2/500 | 69.6±0.1/1,500 |
2 | 70.6±0.0/3,000 | 71.7±0.1/1,500 |
3 | 71.5±0.0/2,500 | 72.7±0.0/1,500 |
4 | 72.1±0.0/700 | 73.2±0.0/2,000 |
5 | 71.9±0.0/1,500 | 73.0±0.0/2,500 |
Table 5 illustrates: the accuracy in the table is expressed as a percentage and the number is the average of 10 runs.
It should be noted that, although the steps in the above-described flowcharts are shown in sequence as indicated by arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in the above-described flowcharts may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least a portion of the sub-steps or stages of other steps.
Fig. 8 shows an internal structure diagram of a computer device in one embodiment, and the computer device may be specifically a terminal or a server. As shown in fig. 8, the computer apparatus includes a processor, a memory, a network interface, a display, and an input device, which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of training a self-connected width map convolutional neural network model. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those of ordinary skill in the art that the architecture shown in FIG. 8 is a block diagram of only a portion of the architecture associated with the subject application, and is not intended to limit the computing devices to which the subject application may be applied, as a particular computing device may include more or less components than those shown, or may combine certain components, or have a similar arrangement of components.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the steps of the training method of the self-connected width map convolutional neural network model when executing the computer program.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the above-mentioned training method of the self-connected width map convolutional neural network model.
In summary, the embodiment of the present invention provides a self-connected width map convolutional neural network model and a training method thereof, which provide a self-connected width map convolutional neural network model including a self-connected high-order map convolutional layer capturing multi-order neighborhood information of nodes, an SP information fusion pooling layer mixing different-order neighborhood features, and a softmax classification output layer, and an efficient model training method corresponding to the model, in which feature preprocessing is performed before training, based on fully considering that the existing high-order map convolutional neural network model adopts different-order convolutions, adopts different weights, stacks a plurality of high-order map convolutional layers, and introduces over-fitting risks and other problems of low training efficiency caused by excessive parameters and complexity of the model due to a method mechanism of fusing different distance neighborhood information by full connection. When the model and the training method thereof are applied to actual classification tests, the self-connection high-order graph convolution layer is adopted to increase the width of the model, reduce the depth of the model and reduce the number of parameters, meanwhile, multi-order neighborhood information can be gathered, higher weight is given to nodes of the model, the receptive field of the model is widened, the overfitting risk of the model is avoided, and the learning capacity, stability and classification accuracy of the model are further improved.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above.
The embodiments in this specification are described in a progressive manner, and all the same or similar parts of the embodiments are directly referred to each other, and each embodiment is described with emphasis on differences from other embodiments. It should be noted that, the technical features of the embodiments may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express some preferred embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for those skilled in the art, various modifications and substitutions can be made without departing from the technical principle of the present invention, and these should be construed as the protection scope of the present application. Therefore, the protection scope of the present patent shall be subject to the protection scope of the claims.
Claims (10)
1. A self-connection width map convolution neural network model is characterized by comprising an input layer, a self-connection high-order map convolution layer, an information fusion pooling layer and an output layer in sequence;
the input layer is used for receiving graph characteristics of a training data set;
the self-connection high-order graph convolution layer is used for performing zero-order to k-order self-connection graph convolution operation according to the graph characteristics to obtain graph convolution data;
the information fusion pooling layer is used for carrying out zero-order to k-order feature fusion according to the graph convolution data to obtain fusion data;
and the output layer is used for outputting a model result according to the fusion data.
2. The self-connected width map convolutional neural network model of claim 1, wherein the self-connected high order map convolutional layer is obtained by introducing new self-connections at any order of map convolutional products.
3. The self-connected width map convolutional neural network model of claim 2, wherein the self-connected high order map convolutional layer comprises a zero order map convolution to a k order map convolution based on weight sharing, represented as:
4. the self-connected width map convolutional neural network model of claim 3, wherein an output layer HGCN of the self-connected width map convolutional neural network modelSLExpressed as:
wherein, σ (-) is an activation function, SP (-) is an information fusion function, and softmax (-) is a multi-classification output function.
5. The self-connected width map convolutional neural network model of claim 3, wherein the σ activation function is a ReLU nonlinear activation function.
7. the method for training a self-connected width map convolutional neural network model as claimed in any one of claims 1-6, wherein the step of training the method comprises:
obtaining an input matrix and a regularization adjacency matrix of the graph of the self-connection width graph convolution neural network model according to a training data set;
performing characteristic preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain preprocessing characteristics;
inputting the preprocessing features into the self-connection width chart convolution neural network model, and performing feature training to obtain a training result.
8. The method of claim 7, wherein the deriving the input matrix and the regularized adjacency matrix for the graph of the self-connected width map convolutional neural network model based on a training dataset comprises:
acquiring the training data set and determining the type of the training data set;
and selecting a specific method to obtain an input matrix and a regularized adjacency matrix of the graph of the self-connection width graph convolution neural network model according to the type of the training data set.
9. The method of claim 7, wherein the step of performing feature preprocessing according to the input matrix and the regularized adjacency matrix of the graph to obtain preprocessed features comprises:
summing the regularized adjacency matrixes of the graphs of all different orders to obtain a preprocessed adjacency matrix;
and performing product on the preprocessing adjacent matrix and the input matrix of the graph to obtain preprocessing characteristics.
10. The method according to claim 7, wherein the step of inputting the preprocessed features into the self-connected width convolution neural network model for feature training to obtain the training result comprises:
randomly initializing a parameter matrix of the self-connection width map convolution neural network model;
inputting the preprocessing characteristics into the self-connection width chart convolution neural network model, and training by adopting a loss function and a gradient descent method to obtain a convergent parameter matrix.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011610969.2A CN112651492B (en) | 2020-12-30 | 2020-12-30 | Self-connection width graph convolution neural network model system and training method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011610969.2A CN112651492B (en) | 2020-12-30 | 2020-12-30 | Self-connection width graph convolution neural network model system and training method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112651492A true CN112651492A (en) | 2021-04-13 |
CN112651492B CN112651492B (en) | 2023-10-13 |
Family
ID=75364267
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011610969.2A Active CN112651492B (en) | 2020-12-30 | 2020-12-30 | Self-connection width graph convolution neural network model system and training method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112651492B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114050975A (en) * | 2022-01-10 | 2022-02-15 | 苏州浪潮智能科技有限公司 | Heterogeneous multi-node interconnection topology generation method and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107622276A (en) * | 2017-08-21 | 2018-01-23 | 北京精密机电控制设备研究所 | A kind of deep learning training method combined based on robot simulation and physics sampling |
CN108764292A (en) * | 2018-04-27 | 2018-11-06 | 北京大学 | Deep learning image object mapping based on Weakly supervised information and localization method |
US20200285944A1 (en) * | 2019-03-08 | 2020-09-10 | Adobe Inc. | Graph convolutional networks with motif-based attention |
CN112132818A (en) * | 2020-09-29 | 2020-12-25 | 北京航空航天大学 | Image processing method for constructing three stages based on graph convolution neural network |
-
2020
- 2020-12-30 CN CN202011610969.2A patent/CN112651492B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107622276A (en) * | 2017-08-21 | 2018-01-23 | 北京精密机电控制设备研究所 | A kind of deep learning training method combined based on robot simulation and physics sampling |
CN108764292A (en) * | 2018-04-27 | 2018-11-06 | 北京大学 | Deep learning image object mapping based on Weakly supervised information and localization method |
US20200285944A1 (en) * | 2019-03-08 | 2020-09-10 | Adobe Inc. | Graph convolutional networks with motif-based attention |
CN112132818A (en) * | 2020-09-29 | 2020-12-25 | 北京航空航天大学 | Image processing method for constructing three stages based on graph convolution neural network |
Non-Patent Citations (2)
Title |
---|
ANH VIET PHAN等: "DGCNN: A convolutional neural network over large-scale labeled graphs", 《ELSEVIER SCIENCE》, pages 1 - 11 * |
张桐: "基于时空神经网络的动态情感识别研究", 《万方数据知识服务平台学位论文库》, pages 1 - 118 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114050975A (en) * | 2022-01-10 | 2022-02-15 | 苏州浪潮智能科技有限公司 | Heterogeneous multi-node interconnection topology generation method and storage medium |
CN114050975B (en) * | 2022-01-10 | 2022-04-19 | 苏州浪潮智能科技有限公司 | Heterogeneous multi-node interconnection topology generation method and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN112651492B (en) | 2023-10-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112598080A (en) | Attention-based width map convolutional neural network model and training method thereof | |
Sun et al. | What and how: generalized lifelong spectral clustering via dual memory | |
CN112529168B (en) | GCN-based attribute multilayer network representation learning method | |
Zheng et al. | Learning in energy-efficient neuromorphic computing: algorithm and architecture co-design | |
CN112633482B (en) | Efficient width graph convolution neural network model system and training method | |
WO2023065859A1 (en) | Item recommendation method and apparatus, and storage medium | |
Penkovsky et al. | Efficient design of hardware-enabled reservoir computing in FPGAs | |
Wan et al. | Deep matrix factorization for trust-aware recommendation in social networks | |
CN112633481B (en) | Multi-jump graph convolution neural network model system and training method | |
CN112529069B (en) | Semi-supervised node classification method, system, computer equipment and storage medium | |
CN114637923B (en) | Data information recommendation method and device based on hierarchical attention-graph neural network | |
CN113987200A (en) | Recommendation method, system, terminal and medium combining neural network with knowledge graph | |
Venturelli et al. | A Kriging-assisted multiobjective evolutionary algorithm | |
CN112529071B (en) | Text classification method, system, computer equipment and storage medium | |
CN115221413B (en) | Sequence recommendation method and system based on interactive graph attention network | |
Concolato et al. | Data science: A new paradigm in the age of big-data science and analytics | |
Jiang et al. | Convolutional neural network pruning based on multi-objective feature map selection for image classification | |
Gallicchio et al. | Deep reservoir neural networks for trees | |
Reimann et al. | Neural logic rule layers | |
CN116992151A (en) | Online course recommendation method based on double-tower graph convolution neural network | |
CN112668700A (en) | Width map convolutional network model based on grouping attention and training method thereof | |
CN111178986A (en) | User-commodity preference prediction method and system | |
CN114417172A (en) | Deep interest evolution recommendation method, device, equipment and storage medium | |
CN114511387A (en) | Product recommendation method and device, electronic equipment and storage medium | |
Shen et al. | StructBoost: Boosting methods for predicting structured output variables |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |