CN116883746A - Graph node classification method based on partition pooling hypergraph neural network - Google Patents
Graph node classification method based on partition pooling hypergraph neural network Download PDFInfo
- Publication number
- CN116883746A CN116883746A CN202310861005.2A CN202310861005A CN116883746A CN 116883746 A CN116883746 A CN 116883746A CN 202310861005 A CN202310861005 A CN 202310861005A CN 116883746 A CN116883746 A CN 116883746A
- Authority
- CN
- China
- Prior art keywords
- node
- matrix
- hypergraph
- pooling
- nodes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011176 pooling Methods 0.000 title claims abstract description 81
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000005192 partition Methods 0.000 title claims abstract description 23
- 239000013598 vector Substances 0.000 claims abstract description 60
- 239000011159 matrix material Substances 0.000 claims description 200
- 230000008569 process Effects 0.000 claims description 16
- 239000002131 composite material Substances 0.000 claims description 8
- 101100481876 Danio rerio pbk gene Proteins 0.000 claims description 6
- 101100481878 Mus musculus Pbk gene Proteins 0.000 claims description 6
- 230000020411 cell activation Effects 0.000 claims description 2
- 239000012633 leachable Substances 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000012545 processing Methods 0.000 abstract description 7
- 238000010276 construction Methods 0.000 description 10
- 238000013527 convolutional neural network Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 102000004169 proteins and genes Human genes 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013316 zoning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a Graph node classification method based on a partition pooling hypergraph neural network, which relates to the technical field of image processing, wherein entity targets in judicial images are regarded as nodes in a Graph, the relationship between the entity targets is modeled as a Graph (Graph), a target classification task is converted into a node classification task of the Graph, so that the association relationship between the entity targets is fully utilized, deep feature vectors extracted by FastRCNN are regarded as feature vectors of the nodes, then the adjacent relationship between the nodes is constructed according to the similarity of the node feature vectors, and finally the target recognition task of the judicial images is completed by using the Graph node classification technology. The invention improves the image classification recognition speed and accuracy through the pooling technology.
Description
Technical Field
The invention relates to the technical field of image classification, in particular to a graph node classification method based on a partition pool hypergraph neural network.
Background
When analyzing judicial images, it is necessary to identify physical objects in the images, such as persons, cars, daggers, and the like. Conventional target recognition methods use convolutional neural networks (Convolutional Neural Networks, CNNs) to accomplish target recognition tasks, such as regional convolutional neural networks (Region-CNN, RCNN), fast regional convolutional neural networks (Fast Region-CNN, fast RCNN), and other target recognition techniques, but these techniques do not utilize the association between physical targets in an image, but rather, predict independently. The graph is a non-euclidean data composed of nodes and their connection relations, which can be represented by an adjacency matrix. A large amount of data in life can be modeled as a graph, such as a quotation network, social network, protein structure, etc. The graph node classification refers to predicting the types of all nodes in the graph by utilizing the structural information of the graph and the feature matrix of the nodes, and is an important research field in the image recognition classification.
The research field of researching graph node classification tasks by using a deep learning method has emerged with a great deal of work, and the methods can be classified into graph neural networks and hypergraph neural networks according to the types of the neural networks used. The graph neural network mainly comprises a graph convolutional neural network (Graph Convolutional Network, GCN), a graph injection force neural network (Graph Attention Network, GAT), a graph isomorphic neural network (Graph Isomorphism Network, GIN) and the like, wherein the graph neural network takes a characteristic matrix of nodes and an adjacent matrix of a graph as inputs of the neural network, adopts a thought of 'neighborhood information aggregation', utilizes characteristics of neighborhood nodes to update own characteristics, and finally obtains classification prediction results of all the nodes through a simple linear classifier; the hypergraph neural network is a neural network structure modeled according to the hypergraph, the hypergraph is a generalized graph structure and consists of nodes and hyperedges, one edge of the simple graph can only be connected with two nodes, and one hyperedge can be connected with a plurality of nodes, so that the defect that the simple graph can only represent binary relations between the nodes is overcome, and the connection relation between the nodes and the hyperedges in the hypergraph can be represented by an association matrix.
In recent years, hypergraph neural networks (Hypergraph Neural Networks, HGNN) based on hypergraph modeling show good effects in research fields such as citation networks, text classification, attitude estimation and the like. The HGNN takes the characteristic matrix and the association matrix of the node as input, and updates the characteristic representation of the node by using the super-graph Laplace transformation. Because the hypergraph neural network cannot be directly used for processing graph data, in order to process the graph data, the HGNN translates the graph adjacent matrix of the node self-connection into a hypergraph correlation matrix, converts the common graph into the hypergraph, and then utilizes the HGNN to process the converted hypergraph. By converting graph modeling data into hypergraph modeling, hypergraph neural networks can be used for node classification tasks of the graph.
To enhance the generalization ability of neural networks, reduce the computational effort and obtain information at different levels, it is often necessary to introduce pooling techniques. The pooling technology is a technical method for obtaining coarsened data, reducing the calculated amount and enhancing the generalization capability. In the graph neural network, there are mainly clustering-based and node selection-based pooling methods, typical representatives of which are DIFFPOOL and SAGPOOL, which reduce the scale of the graph by means of node clustering and node selection, respectively. In hypergraph neural networks, there are hyperedge pooling-based methods, such as HM-GNN, that reduce hypergraph size by pooling hyperedges. The pooling technology is often used as an important structural component of the neural network, plays an important role in the content understanding task of the large-scale image, and can effectively improve the efficiency and effect of judicial image analysis.
However, existing pooling methods cannot be directly applied to graph node classification tasks. Because the graph node classification task requires the neural network to retain all node characteristics, the existing graph pooling technology can reduce the number of nodes, and the pooling technology of the hypergraph neural network can also lead to the reduction of the number of nodes, so that the graph node classification task cannot be applied to node classification and is not suitable for judicial image analysis tasks of graph modeling. Moreover, the hypergraph neural network based on hyperedge pooling takes hyperedges as pooling objects, and a plurality of hyperedges are pooled each time, so that the hypergraph information is possibly seriously lost. When applied to judicial image entity target recognition, the entity target recognition effect may be poor.
Therefore, how to realize accurate identification and classification of images based on pooled hypergraph neural networks is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of this, the present invention provides a Graph node classification method based on a partition pool hypergraph neural network, which regards entity targets in judicial images as nodes in the Graph, models the relationship between the entity targets as graphs (Graph), and converts the target classification task into a node classification task of the Graph, so as to make full use of the association relationship between the entity targets, regards deep feature vectors extracted by Fast RCNN as feature vectors of the nodes, constructs an adjacency relationship between the nodes according to the similarity of the node feature vectors, and finally completes the target recognition task of the judicial images by using the Graph node classification technology.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a graph node classification method based on a partition pooling hypergraph neural network comprises the following steps:
step 1: obtaining a judicial image, and extracting high-order feature vectors of all entity targets in the judicial image by using Fast RCNN;
step 2: taking an entity target in the judicial image as a node, taking a high-order feature vector as a node feature vector, and forming a node feature matrix by the node feature vectors of all the nodes; calculating cosine similarity of any two node feature vectors, taking two nodes corresponding to the node feature vectors with the cosine similarity exceeding a set similarity threshold as adjacent nodes, obtaining an initial adjacency matrix, and constructing an original graph;
step 3: converting the original graph into a hypergraph, constructing the hypergraph according to the adjacent matrix, obtaining a hypergraph association matrix, and initializing the iteration times; initializing the iteration number to be 0;
step 4: the hypergraph convolution layer of the partition pooling hypergraph neural network calculates a hypergraph Laplace matrix according to the hypergraph incidence matrix, and updates the node characteristic matrix by using the hypergraph Laplace matrix;
step 5: the regional pooling layer of the regional pooling hypergraph neural network calculates node comprehensive scores according to the hypergraph incidence matrix and the node characteristic matrix, selects nodes to be reserved in the hyperedge of the hypergraph according to the node comprehensive scores, deletes other nodes, and updates the hypergraph incidence matrix and the node characteristic matrix;
step 6: the method comprises the steps that a reading layer of a partition pooling hypergraph neural network obtains a hyperedge feature matrix according to a hypergraph incidence matrix and a node feature matrix, and the iteration number is increased by 1;
step 7: if the iteration number is equal to the maximum iteration number, the step 8 is entered, if not, the step 4 is replaced; the maximum number of iterations is set to 3;
step 8: and (3) classifying the nodes in the step (2) according to the superside feature matrix to realize judicial image classification.
Preferably, the original graph constructed in the step 2 is represented by a node feature matrix and an adjacency matrix, which are composed of node feature vectors of all nodes.
Preferably, in step 3, the original graph is converted into the hypergraph, and each node in the original graph is connected with its adjacent nodes by a hyperedge; the hypergraph correlation matrix is calculated from the adjacency matrix, h=a+i,representing a hypergraph incidence matrix, wherein E is the number of hyperedges, N is the number of entity targets, and E=N in the initial stage; a represents an adjacency matrix; i is the identity matrix.
Preferably, the specific implementation process of the step 4 is as follows:
step 41: calculating node degree and superside degree of the hypergraph according to the hypergraph incidence matrix, wherein the node degree and the superside degree are expressed as follows:
wherein d (v) represents the node degree of the v-th node; d (e) represents the superlimit degree of the e-th superlimit; the node degrees of all the nodes form a node degree matrix, and the superedge degrees of all the superedges form a superedge degree matrix; the method comprises the steps of carrying out a first treatment on the surface of the W (e) represents the hyperedge weight of the e-th hyperedge (default 1); h (v, e) represents a value in the hypergraph correlation matrix corresponding to the v-th node and the e-th hyperedge, H (v, e) =1 represents that the v-th node is connected with the e-th hyperedge, and H (v, e) =0 represents that the v-th node is disconnected with the e-th hyperedge;
step 42: and calculating a super-graph Laplace matrix delta according to the node degree matrix and the super-edge degree matrix, wherein the super-graph Laplace matrix delta is expressed as:
wherein Representing a node degree matrix; />Representing a superlimit matrix; />A weight matrix representing the superside; w default to a diagonal matrix of all 1;
step 43: and inputting the hypergraph Laplace matrix into a hypergraph convolution layer to perform hypergraph convolution, and obtaining an updated node characteristic matrix, wherein the updated node characteristic matrix is expressed as:
X (k) ＝σ(ΔX (k-1) Θ)
wherein ,X(k-1) Representing a node characteristic matrix before hypergraph convolution;representing a learnable parameter, F representing a dimension of a node feature vector of the entity target; u represents the hidden layer dimension of the partition-pooling hypergraph neural network; sigma is a correct linear cell activation function, +.>And (5) representing the node characteristic matrix after the hypergraph convolution.
Preferably, the specific implementation process of the step 5 is as follows:
step 51: calculating a node type score S 1 The expression is:
S 1 ＝XW s1 +b s1
wherein ,learnable parameters scoring node types, +.>Is a leachable offset for node type scoring; x represents the node characteristic matrix after the hypergraph convolution update; u represents the hidden layer dimension of the partition-pooling hypergraph neural network;
step 52: calculating importance scores of the nodes to the supersides, wherein the expression is as follows:
S 2 (v，e)＝((X(v)||X E (e))W s2 +b s2 )*H(v，e)
wherein ,S2 (v, e) represents the importance of the v node to the e-th superedge;a learnable parameter representing the importance score of a node for a superside, < ->Is a leavable offset of the node's importance score for the superside; x (v) represents a node feature vector of a v-th node; x is X E (e) The superside feature vector representing the e-th superside is the average value of all node feature vectors in the superside e; spliced superside feature vector X E (e) And node feature vector X (v), and obtaining importance score S of node v to superside e through a full connection layer 2 (v，e)；
Step 53: calculating a node comprehensive score S according to the node type score and the importance score of the node pair supersides, wherein the expression is as follows:
S(v，e)＝sigmoid(S 1 (v)+S 2 (v，e))*H(v，e)
wherein S2 (v, e) represents the node composite score of the v-th node in the e-th superside; s is S 1 (v) Node type score representing the v-th node;
step 54: updating the node characteristic matrix X by using the node comprehensive score, wherein the expression is as follows:
wherein X (v) represents a node feature vector of the v-th node; the ";
step 55: performing in-edge pooling operation on all supersides in the supergraph, and determining to reserve or delete nodes in the supersides according to the node comprehensive score S, wherein the reserved nodes are as follows:
idx(e)＝topk(S[：，e]，keep(e))，
wherein idx (e) represents the node reserved in the e-th superside; pr represents the pooling rate; topk is a function of the node number corresponding to the highest keep (e) scores; s [: e represents the node composite score for all nodes connected to the e-th superside, S [: e represents { S1, e, S2, e, …, S n, e };
step 56: in the hypergraph incidence matrix, if the node v epsilon idx (e) is set to H (v, e) =1, otherwise, H (v, e) =0, deleting all zero rows in the hypergraph incidence matrix H, deleting corresponding node feature vectors in the node feature matrix X, and finishing the regional pooling operation of the hypergraph.
Preferably, the specific implementation process of the step 6 is as follows:
step 61: and calculating the superside characteristic vector recorded by each superside in each iteration, wherein the expression is as follows:
wherein ,XR (e) (k) Representing the superside characteristic vector read by the reading layer in the kth iteration; n represents the number of entity targets; x (v) represents a node feature vector of a v-th node; all the superside feature vectors form a superside feature matrix.
Preferably, in step 8, a plurality of sets of superside feature matrices are obtained through the iteration, a final superside feature matrix is obtained by summing the plurality of sets of superside feature matrices, and the final superside feature matrix outputs node classification results through the full-connection layer of the partition-pool supergraph neural network, corresponds to entity target classification results, and completes judicial image classification. And (5) performing iteration for several times to obtain several groups of superside feature matrixes.
Compared with the prior art, the method for classifying the graph nodes based on the partition pooling hypergraph neural network provided by the invention has the advantages that the task of classifying the graph nodes is completed by adopting the partition pooling hypergraph neural network model, and the defect that the task of classifying the graph nodes cannot apply pooling technology is overcome. By applying the pooling technology, the running speed of the model can be increased, the accuracy of the model is improved, and the memory consumption of a computer is reduced, so that the classification and recognition speed and accuracy of judicial images are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of the overall architecture of a graph node classification system based on a partition pooling hypergraph neural network provided by the invention;
FIG. 2 is a schematic diagram of hypergraph construction provided by the present invention;
FIG. 3 is a schematic diagram of a hypergraph convolution operation provided by the present invention;
FIG. 4 is a schematic diagram of a node comprehensive score calculation process provided by the invention;
FIG. 5 is a schematic diagram of a node process according to the node comprehensive score pooling provided by the invention;
FIG. 6 is a schematic diagram of updating an incidence matrix and reading out an over-edge feature matrix provided by the invention;
fig. 7 is a schematic diagram of the final classification process provided by the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses a graph node classification method based on a partition pooling hypergraph neural network, which comprises the following steps of:
s1: obtaining a judicial image, and extracting high-order feature vectors of all entity targets in the judicial image by using Fast RCNN;
s2: taking an entity target in the judicial image as a node, taking a high-order feature vector as a node feature vector, calculating cosine similarity of any two node feature vectors, taking two nodes corresponding to the node feature vectors with the cosine similarity exceeding a set similarity threshold as adjacent nodes, and constructing an initial adjacency matrix;
s3: constructing a hypergraph association matrix according to the adjacency matrix;
s4: calculating a hypergraph Laplace matrix according to the hypergraph incidence matrix;
s5: updating the node characteristic matrix according to the supergraph Laplace matrix;
s6: calculating node comprehensive scores according to the hypergraph incidence matrix and the node characteristic matrix;
s7: selecting nodes to be reserved in the superside according to the node comprehensive scores, deleting other nodes, and updating the supergraph incidence matrix and the node characteristic matrix;
s8: obtaining a superside feature matrix according to the supergraph incidence matrix and the node feature matrix;
s9: iterating S4 to S83 times;
s10: and classifying the nodes according to the superside feature matrix to realize judicial image classification.
The invention converts the entity target identification task in the judicial image into the graph node classification task.
Examples
Fig. 1 shows the overall architecture of a graph node classification method based on a partition-pooling hypergraph neural network. The construction of the model comprises 4 processing stages, namely an image feature extraction stage, a hypergraph construction stage, a convolution pooling stage and a node classification stage.
Image feature extraction: extracting a physical target feature vector in the image by using Fast RCNN;
hypergraph construction stage: firstly, judging the adjacent relation between nodes according to the entity target feature vector so as to construct a common graph, and then converting the common graph into a hypergraph;
convolution pooling stage: performing hypergraph convolution and regional pooling;
node classification stage: and calculating a final node classification result.
Wherein the convolution pooling stage is formed by stacking three identical modules, each of which can be divided into hypergraph convolution layers (S4 to S5), zoned pooling layers (S6 to S7) and readout layers (S8).
1) Image feature extraction stage
Given a judicial image needing to classify an entity target, a Fast-RCNN is used for extracting a high-order feature vector of the entity target in the image. And regarding the entity target as a node in the graph, regarding a higher-order feature vector of the entity target as an initial node feature vector of the node, calculating cosine similarity of any two node feature vectors, and if the cosine similarity is higher than a threshold value, regarding the nodes as adjacent, and using an adjacent matrix to represent an adjacent relation of the node to complete construction of the initial graph.
Giving a judicial image needing entity target classification, assuming N entities in the image, and extracting a high-order feature matrix corresponding to the entity targets in the image by using Fast RCNNWhere F is the dimension of the node feature vector of the entity target extracted by Fast RCNN. Each entity in the image is regarded as a node in the image, a high-order feature matrix formed by node feature vectors of all entity targets is regarded as an initial node feature matrix of the node, and an adjacent matrix A epsilon {0,1} of the image is constructed N×N . And calculating cosine similarity of any two node feature vectors in the node feature matrix, wherein sim (i, j) =cos_similarity (X (i), X (j)), wherein X (i) represents an initial node feature vector of an ith node, and sim (i, j) represents cosine similarity of the ith node and the jth node. If sim (i, j) > γ, A (i, j) =1 is set, whereas A (i, j) =0, where γ ε [0,1]Is a threshold set by man. And the construction of the original graph is completed through the processing.
2) Hypergraph construction stage
Given the adjacency matrix and node feature matrix of the original graph, this stage converts the original graph into a hypergraph. Firstly, converting an adjacent matrix of an original graph into an adjacent matrix with nodes connected by self, regarding nodes in the original graph and neighbor nodes thereof as nodes connected by a superside, converting the original graph into the supergraph according to the adjacent matrix, representing the supergraph by using an association matrix, and regarding a node characteristic matrix of the original graph as a node characteristic matrix of the supergraph, wherein the process is shown in fig. 2. And after the processing is finished, the associated matrix and the node characteristic matrix are used as input of the next stage.
After the image feature extraction is completed, the original image is extracted by an initial node feature matrixAnd adjacency matrix A epsilon {0,1} N×N And (3) representing. To convert a graph into a hypergraph, each node in the original graph and its neighbor nodes are considered to be connected by a hyperedge, thereby constructing the hypergraph. According to the hypergraphThe construction method is H=A+I, wherein +.>Representing the hypergraph correlation matrix, E is the number of hyperedges, and initially e=n, I is the identity matrix.
If there are 5 nodes in the original graph, the connection relationship between the nodes is as shown in the original graph in fig. 2. Firstly, representing the structure of an original graph by using an adjacent matrix A, and then calculating H=A+I to obtain a graph correlation matrix H, thereby completing the hypergraph construction process.
3) Convolution pooling stage
And according to the input hypergraph incidence matrix and the node characteristic matrix, performing hypergraph convolution and regional pooling operation at the stage, updating the node characteristic matrix, reducing the hypergraph scale, and reading out the hyperedge characteristic matrix for final classification.
Firstly, the hypergraph convolution layer calculates a hypergraph Laplace matrix according to the incidence matrix, and updates the node characteristic matrix by using the hypergraph Laplace matrix. And inputting the association matrix and the node characteristic matrix into a regional pooling layer to pool the nodes in the superside. And the regional pooling layer takes the superside as a unit, and selects reserved or deleted nodes according to the comprehensive scores of the nodes in the superside. The composite score of a node is made up of two parts: node type scoring and node importance to supersides scoring. The node type scoring scores the importance of the nodes according to the node type; the importance score of a node pair superside measures the importance of the node pair on the superside. After the node comprehensive scores are calculated, k nodes with the highest comprehensive scores in the supersides are reserved by taking the supersides as units, the rest nodes are deleted, each superside is ensured to be reserved at least one node, and the association matrix and the node characteristic matrix are updated according to the selection result of the nodes, so that the regional pooling operation is completed. And inputting the incidence matrix and the node characteristic matrix into a reading layer, and calculating and recording the superside characteristic matrix by the reading layer according to the incidence matrix and the node characteristic matrix. The association matrix and node feature matrix are then input to the next module, continuing 3) the operation in the convolution pooling stage until there is no next module.
(1) Hypergraph convolutional layer
Calculating a hypergraph Laplace matrix according to the hypergraph incidence matrix, and updating a node characteristic matrix according to the hypergraph Laplace matrix; after the hypergraph convolution layer processing is completed, the node characteristic matrix X is obtained (k) And the incidence matrix H as the input of the regional pooling layer, X for the convenience of representation (k) Still noted as X.
And taking the obtained hypergraph incidence matrix H and the initial node characteristic matrix X as inputs of a hypergraph convolution layer. The hypergraph convolution layer updates the feature representation of X with a hypergraph convolution operation in the hypergraph neural network. Hypergraph convolution can be understood as the aggregation of information from node to hyperedge and the transfer of information from hyperedge to node, thereby updating the node feature matrix in the hypergraph. The formula of the hypergraph convolution is:
X (k) ＝σ(ΔX (k-1) Θ) (1)
wherein delta represents a super-graph Laplace matrix, X (k-1) Representing the node feature matrix before the hypergraph convolution,is a learnable parameter, U represents the hidden layer dimension of the neural network, σ is the correction linear unit (Rectified Linear Units, reLU) activation function, +.>And (5) representing the node characteristic matrix after the hypergraph convolution. The calculation formula of the hypergraph Laplace matrix is as follows:
wherein Representing a node degree matrix, ">Representing a superside matrix,/->Weight matrix representing superedges, W defaults to pairs of all 1' sAn angular matrix. The node degree of the v-th node is as follows:
the e-th superside degree is:
for example, as shown in fig. 3, the input hypergraph includes 5 hyperedges and 5 nodes, the hypergraph incidence matrix is H, and the input node characteristic matrix is X (k-1) . Firstly, according to the formula (3), calculating a node degree matrix D v The method comprises the steps of carrying out a first treatment on the surface of the Calculating an over-edge matrix D according to the formula (4) e The method comprises the steps of carrying out a first treatment on the surface of the Then according to the formula (2), calculating a super-graph Laplace matrix delta; finally, according to the formula (1), calculating an updated node characteristic matrix X (k) 。
(2) Zoned pooling layer
The regional pooling layer takes the superside as a unit and takes the comprehensive grading of nodes in the superside as a unitNodes within the superedge are deleted or retained. The composite score S of a node is scored by node type +.>And node importance scoring for supersidesTogether, the comprehensive score S (v, e) of the v-th node in the e-th superside is:
S(v，e)＝sigmoid(S 1 (vD+S 2 (v，e))*H(v，e) (5)
the composite score of the node also takes into account the score for the node type and the importance score of the node for the superside. The calculation of the node type score and the node importance score for the superside will be described below.
Node type scores are first calculated. The node type score reflects an assessment of the importance of that type of node. For example, taking natural language as an example, a sentence semantic key often resides in a few words. The node type scoring calculation formula is:
S 1 ＝XW n +b s1 (6)
wherein For learning parameters->Is a learnable offset.
The importance score of the node to the superside is calculated. Taking natural language as an example, the same word has different importance in different sentences, so that the importance of the node to different supersides can be calculated respectively to reflect the importance of the node. Importance S of the v node to the e-th superedge 2 (v, e) is calculated as follows:
S 2 (v，e)＝((XCv)||X E (e))W s2 +b s2 )*H(v，e) (7)
wherein Representing a parameter that can be learned, < >>Is a learnable offset. Equation (8) shows that the average value of the eigenvectors of all the nodes in the hyperedge e is taken as the eigenvector X of the hyperedge E (e) Then splicing the superside feature vector X E (e) And node feature vector X (v), and obtaining importance score S of node v to superside e through a full connection layer 2 (v，e)。
The node feature matrix X and the association matrix H as in fig. 3 serve as inputs to the regional pooling layer. Firstly, calculating according to a formula (6) to obtain a node type score S 1 . Then calculating according to formula (8) to obtain the superb feature matrix X E . Then calculating according to a formula (7) to obtain importance scores S of the nodes on the supersides 2 . And finally, calculating according to a formula (5) to obtain the comprehensive score S of the node. Fig. 4 demonstrates the process of calculating a node composite score S from a node feature matrix X and an association matrix H.
After the comprehensive score S of the node is obtained, the node characteristic matrix X is updated by utilizing the comprehensive score S of the node. The updating process of the node characteristic matrix comprises the following steps:
wherein ". Sup.indicates Hadamard multiplication.
Finally, determining to reserve or delete the nodes in the superside according to the comprehensive score S of the nodes, wherein the reserved nodes are as follows:
idx(e)＝topk(S[：，e]，keep(e))，
where idx (e) represents the node reserved in the e-th superside, pr represents the pooling rate, and topk is a function of the node number corresponding to the highest keep (e) score. And (3) carrying out intra-edge pooling operation shown in the formula (10) on all the superedges, setting H (v, e) =1 if the node v epsilon idx (e), and otherwise setting H (v, e) =0. And finally deleting all zero rows in the hypergraph incidence matrix H, and deleting corresponding feature vectors in the node feature matrix X to complete the regional pooling operation of the hypergraph.
Updating the feature matrix of the node according to formula (9). And (3) calculating the number of the nodes to be reserved for each superside according to a formula (10), and selecting the nodes idx to be reserved for each superside. Fig. 5 demonstrates the process of updating the node feature matrix X and selecting keep (e) retention nodes based on the node synthesis score.
The partition pooling hypergraph neural network comprises three convolution pooling modules, wherein each convolution pooling module comprises a hypergraph convolution layer and a partition pooling layer and performs one hypergraph convolution-partition pooling operation
The convolution pooling module can perform three hypergraph convolution-regional pooling operations
(3) Readout layer
The readout layer records hierarchical information of the hypergraph and is finally used for node classification. Superside e superside feature vector recorded by kth convolution pooling module is X R (e) (k) The calculation formula is as follows:
thus, the hypergraph convolution-zoning pooling operation of one convolution pooling module in the convolution pooling stage is completed, and the same operation is performed on all the convolution pooling modules in the convolution pooling stage until all the modules are traversed, as shown in fig. 6.
4) Node classification stage
The module processing stage records 3 groups of characteristic matrixes of the superedges, and because 1) the supergraph construction method adopted in the image characteristic extraction stage can treat one superedge as one node in the original graph, the classification of the superedges can be regarded as the classification of the nodes in the original graph. Firstly, summing 3 groups of superside feature matrixes to obtain a final superside feature matrix, and then outputting a final node classification prediction result through a full-connection layer. In this application, the classification result of the node is the classification result of the entity target in the judicial image, as shown in fig. 7.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. The graph node classification method based on the partition pooling hypergraph neural network is characterized by comprising the following steps of:
step 1: obtaining a judicial image, and extracting high-order feature vectors of all entity targets in the judicial image by using Fast RCNN;
step 2: taking an entity target in the judicial image as a node, taking a high-order feature vector as a node feature vector, and forming a node feature matrix by the node feature vectors of all the nodes; calculating cosine similarity of any two node feature vectors, taking two nodes corresponding to the node feature vectors with the cosine similarity exceeding a set similarity threshold as adjacent nodes, obtaining an initial adjacency matrix, and constructing an original graph;
step 3: converting the original graph into a hypergraph, constructing the hypergraph according to the adjacent matrix, obtaining a hypergraph association matrix, and initializing the iteration times;
step 4: the hypergraph convolution layer of the partition pooling hypergraph neural network calculates a hypergraph Laplace matrix according to the hypergraph incidence matrix, and updates the node characteristic matrix by using the hypergraph Laplace matrix;
step 5: the regional pooling layer of the regional pooling hypergraph neural network calculates node comprehensive scores according to the hypergraph incidence matrix and the node characteristic matrix, selects nodes to be reserved in the hyperedge of the hypergraph according to the node comprehensive scores, deletes other nodes, and updates the hypergraph incidence matrix and the node characteristic matrix;
step 6: the method comprises the steps that a reading layer of a partition pooling hypergraph neural network obtains a hyperedge feature matrix according to a hypergraph incidence matrix and a node feature matrix, and the iteration number is increased by 1;
step 7: if the iteration number is equal to the maximum iteration number, the step 8 is entered, if not, the step 4 is replaced;
step 8: and (3) classifying the nodes in the step (2) according to the superside feature matrix to realize judicial image classification.
2. The method for classifying graph nodes based on the partition pooling hypergraph neural network according to claim 1, wherein the original graph constructed in the step 2 is represented by a node feature matrix and an adjacency matrix, which are composed of node feature vectors of all nodes.
3. The method for classifying graph nodes based on the partition pooling hypergraph neural network according to claim 1, wherein in the step 3, the original graph is converted into the hypergraph by adopting a hyperedge to connect each node in the original graph and the adjacent nodes thereof; the hypergraph correlation matrix is calculated from the adjacency matrix, h=a+i,representing a hypergraph incidence matrix, wherein E is the number of hyperedges, N is the number of entity targets, and E=N in the initial stage; a represents an adjacency matrix; i is the identity matrix.
4. The method for classifying graph nodes based on the partition pooling hypergraph neural network according to claim 1, wherein the specific implementation process of the step 4 is as follows:
step 41: calculating node degree and superside degree of the hypergraph according to the hypergraph incidence matrix, wherein the node degree and the superside degree are expressed as follows:
wherein d (v) represents the node degree of the v-th node; d (e) represents the superlimit degree of the e-th superlimit; the node degrees of all the nodes form a node degree matrix, and the superedge degrees of all the superedges form a superedge degree matrix; w (e) represents the hyperedge weight of the e-th hyperedge; h (v, e) represents a value in the hypergraph correlation matrix corresponding to the v-th node and the e-th hyperedge; e represents the number of superedges; n is the number of entity targets;
step 42: and calculating a super-graph Laplace matrix alpha according to the node degree matrix and the super-edge degree matrix, wherein the super-graph Laplace matrix alpha is expressed as:
wherein ,representing a node degree matrix; />Representing a superlimit matrix; />A weight matrix representing the superside;
step 43: and inputting the hypergraph Laplace matrix into a hypergraph convolution layer to perform hypergraph convolution, and obtaining an updated node characteristic matrix, wherein the updated node characteristic matrix is expressed as:
X (k) ＝σ(ΔX (k-1) Θ)
wherein ,X(k-1) Representing a node characteristic matrix before hypergraph convolution;representing a learnable parameter, F representing a dimension of a node feature vector of the entity target; u represents the hidden layer dimension of the partition-pooling hypergraph neural network; σ is the correct linear cell activation function; />And (5) representing the node characteristic matrix after the hypergraph convolution.
5. The method for classifying graph nodes based on the partition pooling hypergraph neural network according to claim 1, wherein the specific implementation process of the step 5 is as follows:
step 51: calculating a node type score S 1 The expression is:
S 1 ＝XW s1 +b s1
wherein ,learnable parameters scoring node types, +.>Is a leachable offset for node type scoring; x represents the node characteristic matrix after the hypergraph convolution update; u represents the hidden layer dimension of the partition-pooling hypergraph neural network;
step 52: calculating importance scores of the nodes to the supersides, wherein the expression is as follows:
S 2 (v，e)＝((X(v)||X E (e))W s2 +b s2 )*H(v，e)
wherein ,S2 (v, e) represents the importance of the v node to the e-th superedge;a learnable parameter representing the importance score of a node for a superside,/>Is a leavable offset of the node's importance score for the superside; x (v) represents a node feature vector of a v-th node; x is X E (e) A superside feature vector representing an e-th superside; d (e) represents the superlimit degree of the e-th superlimit; h (v, e) represents a value in the hypergraph correlation matrix corresponding to the v-th node and the e-th hyperedge;
step 53: calculating a node comprehensive score S according to the node type score and the importance score of the node pair supersides, wherein the expression is as follows:
S(v，e)＝sigmoid(S 1 (v)+S 2 (v，e))*H(v，e)
wherein S (v, e) represents the node composite score of the v-th node in the e-th superside; s is S 1 (v) Node type score representing the v-th node;
step 54: updating the node characteristic matrix by using the node comprehensive score, wherein the expression is as follows:
wherein X (v) represents a node feature vector of the v-th node; the "; e represents the number of superedges; d (v) represents the node degree of the v-th node;
step 55: performing in-edge pooling operation on all supersides in the supergraph, and determining to reserve or delete nodes in the supersides according to the node comprehensive score S, wherein the reserved nodes are as follows:
idx(e)＝topk(S[：，e]，keep(e))，
wherein idx (e) represents the node reserved in the e-th superside; pr represents the pooling rate; topk is a function of the node number corresponding to the highest key (e); s [: e represents the node composite score of all nodes connected to the e-th superside;
step 56: in the hypergraph incidence matrix, if the node v epsilon idx (e) is set to H (v, e) =1, otherwise, H (v, e) =0, deleting all zero rows in the hypergraph incidence matrix, deleting the corresponding node feature vectors in the node feature matrix, and finishing the regional pooling operation of the hypergraph.
6. The method for classifying graph nodes based on the partition pooling hypergraph neural network according to claim 1, wherein the specific implementation process of the step 6 is as follows:
and calculating the superside characteristic vector recorded by each superside in each iteration, wherein the expression is as follows:
wherein ,XR (e) (k) Representing the superside characteristic vector read by the reading layer in the kth iteration; n represents the number of entity targets; x (v) represents a node feature vector of a v-th node; all the superside feature vectors form a superside feature matrix; d (e) represents the superlimit degree of the e-th superlimit; h (v, e) represents the value in the hypergraph correlation matrix corresponding to the v-th node and the e-th hyperedge.
7. The method for classifying graph nodes based on the partition-pooling hypergraph neural network according to claim 1, wherein in the step 8, a plurality of sets of hyperedge feature matrices are obtained through iteration, a final hyperedge feature matrix is obtained by summing the plurality of sets of hyperedge feature matrices, and a node classification result is output by the final hyperedge feature matrix through a full-connection layer of the partition-pooling hypergraph neural network, and the classification result of an entity target is corresponding to the final hyperedge feature matrix, so that the classification of judicial images is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310861005.2A CN116883746A (en) | 2023-07-13 | 2023-07-13 | Graph node classification method based on partition pooling hypergraph neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310861005.2A CN116883746A (en) | 2023-07-13 | 2023-07-13 | Graph node classification method based on partition pooling hypergraph neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116883746A true CN116883746A (en) | 2023-10-13 |
Family
ID=88269347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310861005.2A Pending CN116883746A (en) | 2023-07-13 | 2023-07-13 | Graph node classification method based on partition pooling hypergraph neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116883746A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117131438A (en) * | 2023-10-27 | 2023-11-28 | 深圳市迪博企业风险管理技术有限公司 | Litigation document analysis method, model training method, device, equipment and medium |
-
2023
- 2023-07-13 CN CN202310861005.2A patent/CN116883746A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117131438A (en) * | 2023-10-27 | 2023-11-28 | 深圳市迪博企业风险管理技术有限公司 | Litigation document analysis method, model training method, device, equipment and medium |
| CN117131438B (en) * | 2023-10-27 | 2024-02-13 | 深圳市迪博企业风险管理技术有限公司 | Litigation document analysis method, model training method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112163426B (en) | Relationship extraction method based on combination of attention mechanism and graph long-time memory neural network | |
| CN111695415B (en) | Image recognition method and related equipment | |
| CN109284406B (en) | Intention identification method based on difference cyclic neural network | |
| CN112733866B (en) | Network construction method for improving text description correctness of controllable image | |
| CN111291556B (en) | Chinese entity relation extraction method based on character and word feature fusion of entity meaning item | |
| CN110969020A (en) | CNN and attention mechanism-based Chinese named entity identification method, system and medium | |
| CN111414461A (en) | Intelligent question-answering method and system fusing knowledge base and user modeling | |
| CN112949740B (en) | Small sample image classification method based on multilevel measurement | |
| CN112308115B (en) | Multi-label image deep learning classification method and equipment | |
| CN108595558B (en) | Image annotation method based on data equalization strategy and multi-feature fusion | |
| CN114444600A (en) | Small sample image classification method based on memory enhanced prototype network | |
| CN113255366B (en) | Aspect-level text emotion analysis method based on heterogeneous graph neural network | |
| CN115035418A (en) | Remote sensing image semantic segmentation method and system based on improved deep LabV3+ network | |
| CN113378938B (en) | Edge transform graph neural network-based small sample image classification method and system | |
| CN115966010A (en) | Expression recognition method based on attention and multi-scale feature fusion | |
| CN116610778A (en) | Bidirectional image-text matching method based on cross-modal global and local attention mechanism | |
| CN111898665A (en) | Cross-domain pedestrian re-identification method based on neighbor sample information guidance | |
| CN111026887A (en) | Cross-media retrieval method and system | |
| CN116883746A (en) | Graph node classification method based on partition pooling hypergraph neural network | |
| CN112784921A (en) | Task attention guided small sample image complementary learning classification algorithm | |
| CN108805280B (en) | Image retrieval method and device | |
| CN116206327A (en) | Image classification method based on online knowledge distillation | |
| CN113792594B (en) | Method and device for locating language fragments in video based on contrast learning | |
| CN113535928A (en) | Service discovery method and system of long-term and short-term memory network based on attention mechanism | |
| CN117349494A (en) | Graph classification method, system, medium and equipment for space graph convolution neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |