CN113781385A - Joint attention-seeking convolution method for brain medical image automatic classification - Google Patents

Joint attention-seeking convolution method for brain medical image automatic classification Download PDF

Info

Publication number
CN113781385A
CN113781385A CN202110299393.0A CN202110299393A CN113781385A CN 113781385 A CN113781385 A CN 113781385A CN 202110299393 A CN202110299393 A CN 202110299393A CN 113781385 A CN113781385 A CN 113781385A
Authority
CN
China
Prior art keywords
brain
network
attention
nodes
fmri
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110299393.0A
Other languages
Chinese (zh)
Other versions
CN113781385B (en
Inventor
朱旗
张耕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Aeronautics and Astronautics
Original Assignee
Nanjing University of Aeronautics and Astronautics
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Aeronautics and Astronautics filed Critical Nanjing University of Aeronautics and Astronautics
Priority to CN202110299393.0A priority Critical patent/CN113781385B/en
Publication of CN113781385A publication Critical patent/CN113781385A/en
Application granted granted Critical
Publication of CN113781385B publication Critical patent/CN113781385B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0012Biomedical image inspection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10072Tomographic images
    • G06T2207/10088Magnetic resonance imaging [MRI]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10072Tomographic images
    • G06T2207/10088Magnetic resonance imaging [MRI]
    • G06T2207/10092Diffusion tensor magnetic resonance imaging [DTI]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30004Biomedical image processing
    • G06T2207/30016Brain

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a medical image automatic identification technology based on a graph convolution neural network. Based on brain diffusion tensor imaging and functional magnetic resonance imaging, a machine learning automatic diagnosis method which can fuse two modal data and reasonably reserve and utilize brain graph structure information is designed. The current brain image classification lacks an effective method for fusing multi-modal images, and has poor learning effect on the graph structure information in the data. The invention discloses a medical image automatic identification technology fusing diffusion tensor imaging and functional magnetic resonance imaging based on a graph convolution neural network. Specifically, neighbor information among nodes of the brain areas is aggregated on brain diffusion and relaxation imaging, and then brain area joint attention is extracted. And then performing graph convolution operation on the brain function magnetic resonance image by taking the attention of the brain region as a reference. And finally, inputting the characteristics into a multilayer perceptron to perform automatic identification and classification.

Description

Joint attention-seeking convolution method for brain medical image automatic classification
The technical field is as follows: machine learning field, in particular to automatic identification technology of brain medical images
Background art: the brain network is constructed through the correlation of brain intervals so as to observe the functional characteristics of the brain, and the technology is applied to the field of automatic identification of medical images through machine learning. Scientists in the field of machine learning have proposed many novel technical models for implementing automatic recognition analysis of medical images of the brain. The graph convolution neural network technology is a deep neural network technology suitable for graph structure data analysis processing. Researchers in the field of deep learning solve the problem of processing graph structure data by designing a graph convolution neural network framework.
Disclosure of Invention
Object of the Invention
The automatic identification technology of the medical image has an important auxiliary effect on the work of doctors, and the invention and the application of a good technology in the related field can greatly improve the diagnosis level of hospitals. The existing automatic identification method of the brain image has the defects of poor robustness and insufficient interpretability. The existing machine learning technology model cannot adapt to the image structure characteristics of the brain function network well, and a multi-mode framework organically integrating Functional Magnetic Resonance Imaging (FMRI) and Diffusion Tensor Imaging (DTI) is lacked.
In order to solve the problems, a technical scheme is explored, the FMRI and the DTI of the brain can be organically fused, the structural characteristics of a brain network diagram can be perfectly adapted, and the method needs to have robustness and good interpretability.
Technical scheme
In order to achieve the purpose, the technical scheme adopted by the invention comprises the following four steps:
the method comprises the steps that (I) a high-order functional brain network is built based on FMRI, the high-order functional brain network is built by brain region FMRI signal vectors, and the fully-connected brain network built among all brain regions based on sparse representation can better reflect high-order relations and potential implicit information on brain region functions.
And (II) constructing a structural network for extracting brain region attention information based on DTI, and constructing a brain network graph by taking DTI structural data as edges between nodes and taking FMRI as node feature vectors in order to extract brain network attention weight.
And (III) calculating first-order, second-order and third-order attention scores of the brain regions through node aggregation, extracting joint attention on a DTI mode and an FMRI mode, and using the joint attention on the pooling of the GCN. The method for calculating the attention does not need to set additional parameters, so that the calculation is very convenient. The attention score extracted by the above equation is then applied to the pooling layer of the GCN. We measure the strength of the functional activity of each edge on the structural network represented by the DTI, and then evaluate the attention score for each brain region node. If the attention scores of the nodes on the structural network are higher, we can understand that the brain area carries out more functional activities through the edges and other brain areas in the DTI structural network.
And (IV) executing graph volume and pooling operation and classifying the output characteristics by using a multilayer perceptron to obtain a final identification result. And (4) sorting the nodes by combining the joint attention scores learned in the DTI and FMRI modes, screening out unimportant nodes and reserving the nodes with high attention. In the lower sampling middle pooling layer of the GCN, the fusion efficiency of the high-order neighbors of the nodes is improved by adopting a mode of discarding the nodes layer by layer, and the method breaks through the structural integrity of a brain network graph, so that the lower network lacks the capability of fusing all the nodes. Therefore, a reading layer is added behind each pooling layer of the GCN, the reading layer completes one-time global information aggregation on the graph node of the current layer, and the result is fed back to the multi-layer perceptron for final classification.
Drawings
FIG. 1 construction of a high-order functional brain network diagram
FIG. 2 DTI structural brain network map construction
FIG. 3 aggregation of node attention
Figure 4 GCN network structure diagram
Detailed Description
The following describes an implementation of the technical scheme specifically:
the work firstly constructs a high-order brain network by using a sparse representation method, and the high-order brain network is used as side information in a brain network graph. And the original signal vector of the FMRI time sequence of the brain area is used as node information, so that a complete graph structure is defined. Different from the traditional brain network classification diagnosis method and framework, the work constructs a high-order graph functional brain network by using brain FMRI data, simultaneously constructs a structural brain network graph by using DTI structural data as edges between nodes and FMRI as a node feature vector, and is used for extracting attention weight of a brain region.
Construction of high-order functional brain network based on FMRI
By X ═ X1,x2,...,x90]∈R240×90To represent the feature vector of the node in the brain network graph, i.e. the FMRI data of a single sample in the sample set, where 90 is the number of brain regions corresponding to each sample, xiThe FMRI signature direction of the ith brain region of the sample. 240 is the length of the brain region feature vector.
A brain region FMRI signal vector is used for constructing a high-order functional brain network, and the high-order relation and potential implicit information on brain region functions can be better embodied by the fully-connected brain network constructed based on sparse representation among all brain regions.
Figure BSA0000236586790000021
In the expression AiRepresenting a dictionary corresponding to the ith brain region, consisting of the features of the other N-1 brain regions, AiThe value of the element in the midbrain region corresponding to the ith column is zero. EiIs a 90 × 1 vector, representing the indicated vector of the dictionary representation corresponding to the ith brain region, here representing the weight of the edge between the brain region node i and other nodes. By calculating the indication vectors of the sparse representation corresponding to all brain regions, an indication vector matrix can be obtained:
E={E1,E2,......,E90}
e is the adjacency matrix of the graph, representing the edges of the higher-order functional brain network graph. Since E is constructed using a sparse representation, the values of many elements in the matrix are zero, which means that there is no edge connection between corresponding nodes. The process of constructing a high-order functional brain network diagram is shown in FIG. 1. We name this network map as a higher order functional brain network map.
(II) constructing a structural network for extracting brain region attention information based on DTI
Meanwhile, in order to extract attention weight of the brain network, DTI structural data is used as edges between nodes, and FMRI is used as a node feature vector to construct a brain network graph. As shown in figure 2, the edge matrix of the brain network graph constructed by DTI structural information
Figure BSA0000236586790000039
It is shown that the characteristics of the nodes are X ═ X as in the functional brain network diagram1,x2,...,x90]. We name this network map as a structural brain network map. The purpose of constructing the structural brain network graph is to extract attention weights of the brain region nodes.
(III) calculating the first-order, second-order and third-order attention scores of the brain areas through node aggregation
This section describes how we extract joint attention on the DTI and FMRI modes and use it on pooling of GCNs. Firstly, in the first two steps, we define a DTI structure brain network graph with FMRI signal vectors as node characteristics, wherein the node characteristics of the graph are represented by X, and the edges of the graph are represented by X
Figure BSA00002365867900000310
And (4) showing. We adopt the mode of aggregating nodes to obtain the node joint attention score Z.
Figure BSA0000236586790000031
Wherein<xi,xj>The inner product of the node i and the node j represents the correlation calculation of the characteristics of the two nodes and reflects the functional connection between brain regions on the edge of the brain network diagram. This means that we measure the strength of the functional activity of each edge on the structural network represented by the DTI, and thus evaluate the attention score for each brain region node. If the attention scores of the nodes on the structural network are higher, we can understand that the brain area carries out more functional activities through the edges and other brain areas in the DTI structural network. Attention score for layer 1 polymerization we used ZlTo express, the calculation method of the l +1 th layer is:
Figure BSA0000236586790000032
the method for calculating the attention does not need to set additional parameters, so that the calculation is very convenient. The attention score extracted by the above equation is then applied to the pooling layer of the GCN. Node screening We adopt a topK mechanism
(IV) executing graph volume and pooling operation and classifying output characteristics by a multilayer perceptron to obtain a final identification result
The pooling mechanism of TopK is to screen out the most valuable information by following the idea of maximal pooling operation in CNN. In the work, the nodes are sorted by combining the joint attention scores learned in the DTI mode and the FMRI mode, unimportant nodes are screened out, and the nodes with high attention are reserved.
After the subgraph division and the corresponding adjacency matrix are determined, the information without the grid subgraph needs to be integrated and extracted, and in the last step, an aggregation method of node attention is defined. We have first introduced the node aggregation approach of convolutional layers.
h=GNN(X,E)
Specific can be written as follows:
Figure BSA0000236586790000033
where W is the parameter to be trained,
Figure BSA0000236586790000034
Figure BSA0000236586790000035
is that
Figure BSA0000236586790000036
The degree matrix of (c). Layer 1+1 is defined as
Figure BSA0000236586790000037
Because the pooling layer adopts a topK method in the downsampling of the GCN, the fusion efficiency of the high-order neighbors of the nodes is improved in a mode of discarding the nodes layer by layer, the method breaks through the structural integrity of a brain network graph, and the lower-layer network lacks the capability of fusing all the nodes. Therefore, a reading layer is added behind each pooling layer of the GCN, the reading layer completes one-time global information aggregation on the graph node of the current layer, and the result is fed back to the multi-layer perceptron for final classification. The readout layer calculation is as follows:
Figure BSA0000236586790000038
the above equation represents the result of stitching together the global average pooling and the global maximum pooling to get the readout layer, | | | represents the stitching operation. Max pooling is a method commonly used in neural networks to extract features and reduce the effects of unwanted information, and mean pooling is a supplement to max pooling to preserve background information and overfitting due to uneven distribution of feature data. The readout layer structure of each layer is added to give a representation of the overall graph:
Figure BSA0000236586790000041
the role of the readout layer is similar to the global pooling of convolutional layers commonly used in CNN network models, and it is their common feature to obtain global expression by a one-time aggregation of global inputs. Also, similar to global pooling in CNN, the readout layer may also employ common operations of summing, averaging, maximizing, etc. The resulting GCN network comprising the readout layer is shown in FIG. 4
Aggregating neighbor nodes on a DTI structural brain network diagram to obtain combined attention in DTI and FMRI modes, then pooling unimportant brain area nodes by using topK according to the attention value, setting a readout layer behind each pooling layer to aggregate information of global nodes in order to learn global information of each layer, and finally adding the readout layer results of each layer and transmitting the readout layer results to a multilayer perceptron for classification.

Claims (5)

1. A joint attention-seeking convolution method for brain medical image automatic classification is characterized in that: the method comprises the following steps:
the method comprises the steps that (I) a high-order functional brain network is built based on FMRI, the high-order functional brain network is built by using a sparse representation method through correlation among FMRI signal vectors in a brain area, and an original vector is used as a node feature of a network node.
And (II) constructing a structural network for extracting brain region attention information based on DTI, and constructing a brain network graph by taking DTI structural data as edges between nodes and taking FMRI as node feature vectors in order to extract brain network attention weight.
And (III) calculating first-order, second-order and third-order attention scores of the brain regions through node aggregation, extracting joint attention on a DTI mode and an FMRI mode, and using the joint attention on the pooling of the GCN.
And (IV) executing graph volume and pooling operation and classifying the output characteristics by using a multilayer perceptron to obtain a final identification result. And (4) sequencing the nodes by combining joint attention scores learned in two modes of DTI and FMRI, screening out unimportant nodes by using a TopK strategy, and reserving the nodes with high attention.
2. The method for convolution of joint attention maps for automatic classification of brain medical images according to 1 is characterized in that a high-order functional brain network map described in the first step is constructed in a manner that the original vectors are used as node features of network nodes after a brain network is constructed by using a sparse representation method.
3. The method for convolving joint attention maps of brain medical image automatic classification according to 1 is characterized in that the brain structure network map described in the second step is constructed in a manner that DTI is used as the edge of the brain network map and FMRI vector is used as the node feature of the network node.
4. According to the method for the convolution of the joint attention map for the automatic classification of the medical brain image in the step 1, which is characterized in that the method for extracting the attention of the brain region nodes described in the step three is adopted, the invention innovatively calculates the importance degree of the brain region in a mode of using the node aggregation of graph convolution.
5. The method for the convolution of the joint attention map of the brain medical image automatic classification according to 1 is characterized in that the process of the brain structure and function information learning and the feature extraction described in the step four is characterized in that a functional network and a structural network are fused in a graph convolution neural network by simultaneously utilizing two modal data of DTI and FMRI, so that the performance of automatic diagnosis is improved.
CN202110299393.0A 2021-03-19 2021-03-19 Combined attention-seeking convolution method for automatic classification of brain medical images Active CN113781385B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110299393.0A CN113781385B (en) 2021-03-19 2021-03-19 Combined attention-seeking convolution method for automatic classification of brain medical images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110299393.0A CN113781385B (en) 2021-03-19 2021-03-19 Combined attention-seeking convolution method for automatic classification of brain medical images

Publications (2)

Publication Number Publication Date
CN113781385A true CN113781385A (en) 2021-12-10
CN113781385B CN113781385B (en) 2024-03-08

Family

ID=78835561

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110299393.0A Active CN113781385B (en) 2021-03-19 2021-03-19 Combined attention-seeking convolution method for automatic classification of brain medical images

Country Status (1)

Country Link
CN (1) CN113781385B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114743053A (en) * 2022-04-14 2022-07-12 电子科技大学 Magnetic resonance image auxiliary processing system based on graph neural network and self attention
CN115359297A (en) * 2022-08-24 2022-11-18 南京航空航天大学 High-order brain network-based classification method, system, electronic device and medium
CN116206752A (en) * 2023-02-22 2023-06-02 南通大学 Mental disease identification method based on structure-function brain network

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110522448A (en) * 2019-07-12 2019-12-03 东南大学 A kind of brain network class method based on figure convolutional neural networks
CN111127441A (en) * 2019-12-25 2020-05-08 兰州大学 Multi-modal brain image depression recognition method and system based on graph node embedding
CN112329801A (en) * 2020-12-03 2021-02-05 中国石油大学(华东) Convolutional neural network non-local information construction method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110522448A (en) * 2019-07-12 2019-12-03 东南大学 A kind of brain network class method based on figure convolutional neural networks
CN111127441A (en) * 2019-12-25 2020-05-08 兰州大学 Multi-modal brain image depression recognition method and system based on graph node embedding
CN112329801A (en) * 2020-12-03 2021-02-05 中国石油大学(华东) Convolutional neural network non-local information construction method

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114743053A (en) * 2022-04-14 2022-07-12 电子科技大学 Magnetic resonance image auxiliary processing system based on graph neural network and self attention
CN114743053B (en) * 2022-04-14 2023-04-25 电子科技大学 Magnetic resonance image auxiliary processing system based on graph neural network and self-attention
CN115359297A (en) * 2022-08-24 2022-11-18 南京航空航天大学 High-order brain network-based classification method, system, electronic device and medium
CN115359297B (en) * 2022-08-24 2024-01-26 南京航空航天大学 Classification method, system, electronic equipment and medium based on higher-order brain network
CN116206752A (en) * 2023-02-22 2023-06-02 南通大学 Mental disease identification method based on structure-function brain network

Also Published As

Publication number Publication date
CN113781385B (en) 2024-03-08

Similar Documents

Publication Publication Date Title
CN110428428B (en) Image semantic segmentation method, electronic equipment and readable storage medium
CN108446730B (en) CT pulmonary nodule detection device based on deep learning
CN113616184B (en) Brain network modeling and individual prediction method based on multi-mode magnetic resonance image
Louati et al. Deep convolutional neural network architecture design as a bi-level optimization problem
CN107506797A (en) One kind is based on deep neural network and multi-modal image alzheimer disease sorting technique
CN113724880A (en) Abnormal brain connection prediction system, method and device and readable storage medium
CN107133651A (en) The functional magnetic resonance imaging data classification method of subgraph is differentiated based on super-network
CN114242236A (en) Structure-function brain network bidirectional mapping model construction method and brain network bidirectional mapping model
CN110852393A (en) Remote sensing image segmentation method and system
CN113781385A (en) Joint attention-seeking convolution method for brain medical image automatic classification
CN113505239B (en) Entity alignment method combining graph attention and attribute clustering
WO2022205502A1 (en) Image classification model construction method, image classification method, and storage medium
CN113693563A (en) Brain function network classification method based on hypergraph attention network
CN112308115A (en) Multi-label image deep learning classification method and equipment
CN115049629A (en) Multi-mode brain hypergraph attention network classification method based on line graph expansion
CN112508181A (en) Graph pooling method based on multi-channel mechanism
CN114743037A (en) Deep medical image clustering method based on multi-scale structure learning
CN114495210A (en) Posture change face recognition method based on attention mechanism
Dhawan et al. Deep Learning Based Sugarcane Downy Mildew Disease Detection Using CNN-LSTM Ensemble Model for Severity Level Classification
CN114743053A (en) Magnetic resonance image auxiliary processing system based on graph neural network and self attention
CN112990270B (en) Automatic fusion method of traditional feature and depth feature
CN116633639B (en) Network intrusion detection method based on unsupervised and supervised fusion reinforcement learning
Le et al. DR-Unet: Rethinking the ResUnet++ Architecture with Dual ResPath skip connection for Nuclei segmentation
CN116630777A (en) Medical index missing data complement method for information bottleneck graph compression
CN117173595A (en) Unmanned aerial vehicle aerial image target detection method based on improved YOLOv7

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant