CN114862881A - Cross-modal attention tumor segmentation method and system based on PET-CT - Google Patents

Cross-modal attention tumor segmentation method and system based on PET-CT Download PDF

Info

Publication number
CN114862881A
CN114862881A CN202210807701.0A CN202210807701A CN114862881A CN 114862881 A CN114862881 A CN 114862881A CN 202210807701 A CN202210807701 A CN 202210807701A CN 114862881 A CN114862881 A CN 114862881A
Authority
CN
China
Prior art keywords
image
pet
modal
cross
attention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210807701.0A
Other languages
Chinese (zh)
Inventor
章毅
李林
胡俊杰
蔡华伟
皮勇
赵祯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan University
Original Assignee
Sichuan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan University filed Critical Sichuan University
Priority to CN202210807701.0A priority Critical patent/CN114862881A/en
Publication of CN114862881A publication Critical patent/CN114862881A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10104—Positron emission tomography [PET]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a PET-CT (positron emission tomography-computed tomography) -based cross-modal attention tumor segmentation method, a system and equipment, which relate to the PET-CT-based tumor segmentation in the technical field of image processing and aim to solve the problems that in the prior art, the fusion efficiency of various modal image features is low and the accurate segmentation of a tumor region is difficult to realize when a PET-CT-based multi-modal image is segmented, and mainly comprises the steps of firstly respectively extracting the features in a PET image and a CT image by using a self-attention mechanism, and then fusing the single-modal features in the PET image and the CT image by using the self-attention mechanism in a cross-modal manner to obtain the cross-modal fusion image features; and finally, segmenting the tumor region based on the cross-modal fusion image characteristics. The self-attention mechanism realizes the expression of the single-mode characteristics through the interaction between the characteristics of different areas
Figure DEST_PATH_IMAGE002
To fusion of image features
Figure DEST_PATH_IMAGE004
By fusing image features
Figure 600949DEST_PATH_IMAGE004
The method has different dimension information, realizes cross-mode efficient fusion of the PET image and the CT image, and realizes accurate segmentation of the tumor region.

Description

Cross-modal attention tumor segmentation method and system based on PET-CT
Technical Field
The invention relates to the technical field of image processing, in particular to a PET-CT-based tumor segmentation method, a PET-CT-based cross-modal attention tumor segmentation system and a PET-CT-based cross-modal attention tumor segmentation method.
Background
Pet (positron Emission tomography), the chinese name of which is positron Emission computed tomography, is a molecular level imaging technique. In the PET imaging process, firstly, a radioactive isotope tracer is injected into a human body, the radioactive isotope generates positrons in the decay process of the human body, and a probe of a PET scanner reflects the metabolism condition of each tissue and organ of the human body by reconstructing the concentration distribution of the radioactive isotope in the human body. The PET imaging technology has the characteristics of high sensitivity and strong specificity in detection of pathological tissues, and can detect metabolic changes of the pathological tissues before morphological changes of the pathological tissues, for example, 18F-labeled Fluorodeoxyglucose (18F-FDG) is a PET tracer commonly used in the field of oncology at present, and the absorption amount of the 18F-FDG by malignant tissues is far greater than that of normal tissues and organs, so that the concentration of radionuclide in the malignant regions is higher, and the image intensity of the malignant regions is higher than that of the normal tissues and organs when the radioactive nuclide is reflected in a PET image. Therefore, compared with the traditional imaging technology, PET is more sensitive to the tumor region, PET imaging can detect the diseased tissue of the human body earlier, and the method has obvious advantages in early diagnosis and treatment of cancer. However, the PET image has low spatial resolution, and has the characteristics of image blurring and high noise.
Ct (computed tomography), which is called electronic computed tomography (ct) in chinese name, scans a human body with X-rays, receives the transmitted X-rays by a detector, and finally processes the X-rays into an image by a computer. Compared with the PET image, the CT image has higher image resolution, but the CT image has a complex structure, the image intensity of a tumor region and that of a normal soft tissue region in the CT image are similar, and the tumor region is difficult to distinguish through the CT image.
Based on the characteristics of sensitivity of PET to tumor regions and high spatial resolution of CT, more and more tumor segmentation models based on PET-CT multi-modal images appear for segmenting the tumor regions, so that quantitative reference basis is provided for patient condition evaluation and treatment scheme formulation, and finally the effect of treatment schemes of patients such as surgery, radiotherapy, chemotherapy and the like is improved. However, these models generally use a simple image fusion strategy to fuse information in images of different modalities, and use the same weight for all voxels of the same fault, which cannot fully utilize the advantages and features of images of various modalities; the research on how to efficiently fuse the complementary information of PET and CT images and the reasonable and effective PET-CT multi-modal tumor segmentation method have important significance on the evaluation and treatment of tumors.
Disclosure of Invention
The invention aims to: the invention provides a cross-modal attention tumor segmentation method, a cross-modal attention tumor segmentation system and a cross-modal attention tumor segmentation device based on PET-CT, and aims to solve the problems that in the prior art, fusion efficiency of image features of each modality is low and accurate segmentation of a tumor region is difficult to achieve when multi-modal image segmentation is based on PET-CT.
The invention specifically adopts the following technical scheme for realizing the purpose:
a cross-modal attention tumor segmentation method based on PET-CT comprises the following steps:
step S1, acquiring a PET image and a CT image;
step S2, respectively extracting the characteristics in the PET image and the CT image by using a self-attention mechanism;
step S3, using a self-attention mechanism to perform cross-modal fusion on the single-modal characteristics in the PET image and the CT image to obtain cross-modal fusion image characteristics;
step S4, segmenting the tumor region based on the cross-modality fusion image features.
In step S1, a PET/CT scanner is used to acquire a registered PET image and a CT image.
In step S2, the PET image and the CT image acquired in step S1 are respectively segmented, and the segmented images are converted from a matrix to a vector form to obtain vectors
Figure 335613DEST_PATH_IMAGE001
(ii) a Amount of coincidence
Figure 331382DEST_PATH_IMAGE001
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image.
In step S3, the single-mode image features of the PET image obtained in step S2 are linearly transformed
Figure 376699DEST_PATH_IMAGE002
、
Figure 99804DEST_PATH_IMAGE003
And
Figure 930488DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 80847DEST_PATH_IMAGE005
、
Figure 613459DEST_PATH_IMAGE006
And
Figure 890988DEST_PATH_IMAGE007
express the vector
Figure 825446DEST_PATH_IMAGE005
、
Figure 894508DEST_PATH_IMAGE008
And
Figure 914417DEST_PATH_IMAGE007
rewriting to matrix form Q 1 、K 1 And V 1 Then, the matrix form is obtained by self-attention calculation
Figure 979325DEST_PATH_IMAGE009
：
Figure 784601DEST_PATH_IMAGE010
Performing linear transformation on the single-mode image characteristics of the CT image obtained in step S2
Figure 480025DEST_PATH_IMAGE002
、
Figure 49546DEST_PATH_IMAGE003
And
Figure 606561DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 515611DEST_PATH_IMAGE005
、
Figure 178673DEST_PATH_IMAGE011
And
Figure 923907DEST_PATH_IMAGE012
express the vector
Figure 533880DEST_PATH_IMAGE005
、
Figure 359753DEST_PATH_IMAGE013
And
Figure 144782DEST_PATH_IMAGE012
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self-attention calculation
Figure 423317DEST_PATH_IMAGE014
：
Figure 571401DEST_PATH_IMAGE015
Form of matrix
Figure 2514DEST_PATH_IMAGE009
In matrix form
Figure 210641DEST_PATH_IMAGE014
Are superposed and fused to obtain a matrix form C, i.e.
Figure 727204DEST_PATH_IMAGE016
Then, linear transformation is performed on the matrix form C
Figure 413400DEST_PATH_IMAGE002
、
Figure 417129DEST_PATH_IMAGE003
And
Figure 609207DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 800016DEST_PATH_IMAGE005
、
Figure 352221DEST_PATH_IMAGE017
And
Figure 223837DEST_PATH_IMAGE018
express the vector
Figure 39346DEST_PATH_IMAGE005
、
Figure 717452DEST_PATH_IMAGE011
And
Figure 824080DEST_PATH_IMAGE018
rewriting to matrix form Q 3 、K 3 And V 3 Then proceed with self-attentionCalculating to obtain matrix form
Figure 536821DEST_PATH_IMAGE019
，
Figure 992073DEST_PATH_IMAGE019
Namely, the image characteristics are fused across the modes:
Figure 970524DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 67793DEST_PATH_IMAGE021
is to fuse image features
Figure 635041DEST_PATH_IMAGE022
T is the transpose of the matrix,
Figure 74244DEST_PATH_IMAGE002
、
Figure 992521DEST_PATH_IMAGE003
and
Figure 627902DEST_PATH_IMAGE004
is a learnable parameter matrix.
A PET-CT based cross-modal attention tumor segmentation system comprising:
the image acquisition module is used for acquiring a PET image and a CT image;
the characteristic extraction module is used for respectively extracting the characteristics in the PET image and the CT image acquired by the image acquisition module by using a self-attention mechanism;
the characteristic fusion module is used for extracting and obtaining single-mode characteristics in the PET image and the CT image by using the self-attention mechanism cross-mode fusion characteristic extraction module to obtain cross-mode fusion image characteristics;
and the tumor segmentation module is used for segmenting the tumor region based on the cross-modal fusion image features obtained by the feature fusion module.
The image acquisition module acquires a registered PET image and a registered CT image by using a PET/CT scanner.
The characteristic extraction module is used for respectively blocking the PET image and the CT image acquired by the image acquisition module, and converting the blocked images into a vector form from a matrix to obtain a vector
Figure 594196DEST_PATH_IMAGE023
(ii) a Amount of coincidence
Figure 656830DEST_PATH_IMAGE023
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image.
The characteristic fusion module carries out linear transformation on the single-mode image characteristics of the PET image obtained by the characteristic extraction module
Figure 344295DEST_PATH_IMAGE024
、
Figure 783366DEST_PATH_IMAGE003
And
Figure 121944DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 371791DEST_PATH_IMAGE005
、
Figure 733502DEST_PATH_IMAGE013
And
Figure 773002DEST_PATH_IMAGE018
express the vector
Figure 654501DEST_PATH_IMAGE005
、
Figure 324517DEST_PATH_IMAGE017
And
Figure 235841DEST_PATH_IMAGE018
rewriting to matrix form Q 1 、K 1 And V 1 Then, the matrix form is obtained by self-attention calculation
Figure 498939DEST_PATH_IMAGE009
：
Figure 15371DEST_PATH_IMAGE010
The characteristic fusion module carries out linear transformation on the single-mode image characteristics of the CT image obtained by the characteristic extraction module
Figure 387447DEST_PATH_IMAGE002
、
Figure 740062DEST_PATH_IMAGE003
And
Figure 121364DEST_PATH_IMAGE025
to obtain different vector expressions
Figure 226724DEST_PATH_IMAGE005
、
Figure 786012DEST_PATH_IMAGE006
And
Figure 609612DEST_PATH_IMAGE026
express the vector
Figure 279758DEST_PATH_IMAGE005
、
Figure 505203DEST_PATH_IMAGE008
And
Figure 219082DEST_PATH_IMAGE026
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self-attention calculation
Figure 543359DEST_PATH_IMAGE014
：
Figure 204148DEST_PATH_IMAGE015
Feature fusion module realigning matrix form
Figure 284099DEST_PATH_IMAGE009
In matrix form
Figure 450769DEST_PATH_IMAGE014
Are superposed and fused to obtain a matrix form C, namely
Figure 983382DEST_PATH_IMAGE016
Then, linear transformation is performed on the matrix form C
Figure 510178DEST_PATH_IMAGE002
、
Figure 195368DEST_PATH_IMAGE003
And
Figure 719891DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 802116DEST_PATH_IMAGE005
、
Figure 821019DEST_PATH_IMAGE027
And
Figure 609983DEST_PATH_IMAGE012
express the vector
Figure 367724DEST_PATH_IMAGE005
、
Figure 177327DEST_PATH_IMAGE013
And
Figure 983609DEST_PATH_IMAGE012
rewriting to matrix form Q 3 、K 3 And V 3 Then, the matrix form is obtained by self-attention calculation
Figure 158239DEST_PATH_IMAGE019
，
Figure 572034DEST_PATH_IMAGE019
Namely, the cross-modal fusion image characteristics:
Figure 566534DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 723977DEST_PATH_IMAGE021
is to fuse image features
Figure 487534DEST_PATH_IMAGE022
T is the transpose of the matrix,
Figure 321498DEST_PATH_IMAGE002
、
Figure 554027DEST_PATH_IMAGE003
and
Figure 764429DEST_PATH_IMAGE004
is a learnable parameter matrix.
A computer device comprising a memory storing a computer program and a processor implementing the steps of a PET-CT based cross-modality attention tumor segmentation method as described above when the computer program is executed.
A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of a PET-CT based cross-modal attention tumor segmentation method as described above.
The invention has the following beneficial effects:
1、compared with convolution operation, the self-attention mechanism can better model the spatial relationship among the features, avoids the loss of characteristic information caused by pooling, and has greater application potential in a multi-modal segmentation task; the self-attention mechanism in the application realizes the expression of the single-mode characteristics through the interaction between the characteristics of different areas
Figure 648071DEST_PATH_IMAGE023
To fusion of image features
Figure 604001DEST_PATH_IMAGE022
By fusing image features
Figure 369832DEST_PATH_IMAGE022
The method has different dimension information, realizes cross-mode efficient fusion of the PET image and the CT image, and realizes accurate segmentation of the tumor region.
2. In the invention, the PET/CT scanner is an integrated imaging device which is manufactured by integrating two imaging devices of PET and CT, and the obtained PET and CT images are well registered, thereby being more beneficial to the subsequent cross-modal characteristic fusion, improving the efficiency and effect of the cross-modal fusion and being more beneficial to the accurate segmentation of a tumor region.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a self-attention mechanism calculation schematic of the present invention.
Detailed Description
Example 1
The present embodiment provides a PET-CT based cross-modal attention tumor segmentation method, as shown in fig. 1, which includes 4 steps, respectively: step S1, acquiring a PET image and a CT image of a patient; step S2 of extracting features (single-mode image features) in the PET image and the CT image using the self-attention mechanism on the basis of the image acquired in step S1; step S3, on the basis of the single-mode image features extracted in the step S2, cross-mode fusion PET images and single-mode features in CT images are fused by using a self-attention mechanism to obtain cross-mode fusion image features; and step S4, accurately segmenting the tumor region based on the cross-modal fusion image characteristics after fusion. The steps will be explained in detail below:
step S1, acquiring a PET image and a CT image of a patient;
different from the traditional manual feature-based segmentation method, the deep neural network automatically learns how to extract task-related abstract features from data, and the extracted features have stronger expression capability and higher translation invariance. Therefore, when the PET/CT scanner is used for acquiring images, the PET/CT scanner is an integrated imaging device formed by integrating two imaging devices of PET and CT, and the obtained PET images and the obtained CT images are well registered.
Step S2, on the basis of the images, respectively extracting the characteristics in the PET images and the CT images by using a self-attention mechanism;
respectively blocking the PET image and the CT image obtained in step S1 (for example, 3 × 3), and converting the blocked images from a matrix to a vector form to obtain vectors
Figure 321607DEST_PATH_IMAGE023
(ii) a And aligning the vectors by applying a self-attention method layer by layer
Figure 607226DEST_PATH_IMAGE023
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image. The self-attention method is a method of performing nonlinear transformation on a vector.
Step S3, on the basis of the single-mode image features extracted in the step S2, cross-mode fusion PET images and single-mode features in CT images are fused by using a self-attention mechanism to obtain cross-mode fusion image features;
performing linear transformation on the single-mode image features of the PET image obtained in step S2
Figure 517413DEST_PATH_IMAGE002
、
Figure 724535DEST_PATH_IMAGE003
And
Figure 214422DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 869394DEST_PATH_IMAGE005
、
Figure 435636DEST_PATH_IMAGE011
And
Figure 113742DEST_PATH_IMAGE018
express the vector
Figure 938479DEST_PATH_IMAGE005
、
Figure 930181DEST_PATH_IMAGE013
And
Figure 713329DEST_PATH_IMAGE018
rewriting into matrix form Q1, K1 and V1, and performing self-attention calculation to obtain matrix form
Figure 629464DEST_PATH_IMAGE009
：
Figure 523470DEST_PATH_IMAGE010
Performing linear transformation on the single-mode image characteristics of the CT image obtained in step S2
Figure 825139DEST_PATH_IMAGE002
、
Figure 529921DEST_PATH_IMAGE003
And
Figure 448198DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 818000DEST_PATH_IMAGE005
、
Figure 52803DEST_PATH_IMAGE011
And
Figure 381016DEST_PATH_IMAGE018
express the vector
Figure 255431DEST_PATH_IMAGE005
、
Figure 504622DEST_PATH_IMAGE013
And
Figure 780883DEST_PATH_IMAGE018
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self-attention calculation
Figure 748839DEST_PATH_IMAGE014
：
Figure 658020DEST_PATH_IMAGE015
Form of matrix
Figure 104045DEST_PATH_IMAGE009
In matrix form
Figure 297129DEST_PATH_IMAGE014
Are superposed and fused to obtain a matrix form C, namely
Figure 249036DEST_PATH_IMAGE016
Then, linear transformation is performed on the matrix form C
Figure 832464DEST_PATH_IMAGE002
、
Figure 144496DEST_PATH_IMAGE003
And
Figure 129770DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 252578DEST_PATH_IMAGE005
、
Figure 385619DEST_PATH_IMAGE017
And
Figure 439026DEST_PATH_IMAGE018
express the vector
Figure 88925DEST_PATH_IMAGE005
、
Figure 631902DEST_PATH_IMAGE011
And
Figure 924343DEST_PATH_IMAGE018
rewriting to matrix form Q 3 、K 3 And V 3 Then, the matrix form is obtained by self-attention calculation
Figure 594490DEST_PATH_IMAGE019
，
Figure 288777DEST_PATH_IMAGE019
Namely, the cross-modal fusion image characteristics:
Figure 2655DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 861020DEST_PATH_IMAGE021
is to fuse image features
Figure 256230DEST_PATH_IMAGE022
T is the transpose of the matrix,
Figure 867340DEST_PATH_IMAGE002
、
Figure 955381DEST_PATH_IMAGE003
and
Figure 35464DEST_PATH_IMAGE004
is a learnable parameter matrix.
As shown in figure 2 of the drawings, in which,
Figure 31102DEST_PATH_IMAGE023
is a single-mode feature of different regions of a PET image and a CT image, which comprises
Figure 699980DEST_PATH_IMAGE028
、
Figure 769043DEST_PATH_IMAGE029
And
Figure 257793DEST_PATH_IMAGE030
3 parts, by applying linear transformations
Figure 322701DEST_PATH_IMAGE002
、
Figure 659136DEST_PATH_IMAGE003
And
Figure 823401DEST_PATH_IMAGE004
obtaining different vector representations of the data
Figure 127343DEST_PATH_IMAGE005
、
Figure 481095DEST_PATH_IMAGE017
And
Figure 858987DEST_PATH_IMAGE018
then the vector is expressed
Figure 522050DEST_PATH_IMAGE005
、
Figure 798441DEST_PATH_IMAGE011
And
Figure 142835DEST_PATH_IMAGE018
the fusion image characteristics are obtained by rewriting into matrix forms Q, K and V and finally calculating based on the self-attention mechanism
Figure 703129DEST_PATH_IMAGE022
In the form of a matrix
Figure 209197DEST_PATH_IMAGE021
。
And step S4, accurately segmenting the tumor region based on the cross-modal fusion image characteristics after fusion.
In the embodiment, the self-attention mechanism realizes expression through interaction between different regional characteristics
Figure 235534DEST_PATH_IMAGE031
To
Figure 180357DEST_PATH_IMAGE022
In which the expression is
Figure 532840DEST_PATH_IMAGE031
And
Figure 22859DEST_PATH_IMAGE032
with different dimensions. Compared with convolution operation, the spatial relationship among the features can be better modeled by the self-attention mechanism, loss of feature information caused by pooling is avoided, and the self-attention mechanism has a great application potential in a multi-modal segmentation task. In addition, the self-attention mechanism used in this patent is to fuse PET and CT image features in a learnable way, where
Figure 460793DEST_PATH_IMAGE002
、
Figure 209306DEST_PATH_IMAGE003
And
Figure 229346DEST_PATH_IMAGE004
both represent learnable parameter matrixes, and the maximum value pooling and the mean value pooling are both fixed operator calculations, and have weak feature fusion capability.
Example 2
A PET-CT based cross-modal attention tumor segmentation system comprising:
the image acquisition module is used for acquiring a PET image and a CT image;
the characteristic extraction module is used for respectively extracting the characteristics in the PET image and the CT image acquired by the image acquisition module by using a self-attention mechanism;
the characteristic fusion module is used for extracting and obtaining single-mode characteristics in the PET image and the CT image by using the self-attention mechanism cross-mode fusion characteristic extraction module to obtain cross-mode fusion image characteristics;
and the tumor segmentation module is used for segmenting the tumor region based on the cross-modal fusion image features obtained by the feature fusion module.
The image acquisition module acquires a registered PET image and a registered CT image by using a PET/CT scanner.
The characteristic extraction module is used for respectively blocking the PET image and the CT image acquired by the image acquisition module, and converting the blocked images into a vector form from a matrix to obtain a vector
Figure 342796DEST_PATH_IMAGE023
(ii) a And applying the self-attention method layer by layer to vector
Figure 330343DEST_PATH_IMAGE023
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image. The self-attention method is a method of performing nonlinear transformation on a vector.
The characteristic fusion module carries out linear transformation on the single-mode image characteristics of the PET image obtained by the characteristic extraction module
Figure 367701DEST_PATH_IMAGE024
、
Figure 694777DEST_PATH_IMAGE003
And
Figure 838182DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 264091DEST_PATH_IMAGE005
、
Figure 292090DEST_PATH_IMAGE013
And
Figure 535989DEST_PATH_IMAGE018
express the vector
Figure 991241DEST_PATH_IMAGE005
、
Figure 704114DEST_PATH_IMAGE017
And
Figure 332541DEST_PATH_IMAGE018
rewriting to matrix form Q 1 、K 1 And V 1 Then, the matrix form is obtained by self-attention calculation
Figure 368630DEST_PATH_IMAGE009
：
Figure 807833DEST_PATH_IMAGE010
The characteristic fusion module carries out linear transformation on the single-mode image characteristics of the CT image obtained by the characteristic extraction module
Figure 991690DEST_PATH_IMAGE002
、
Figure 361491DEST_PATH_IMAGE003
And
Figure 330715DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 127770DEST_PATH_IMAGE005
、
Figure 798923DEST_PATH_IMAGE013
And
Figure 540393DEST_PATH_IMAGE018
express the vector
Figure 551074DEST_PATH_IMAGE005
、
Figure 581347DEST_PATH_IMAGE017
And
Figure 677479DEST_PATH_IMAGE018
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self-attention calculation
Figure 202132DEST_PATH_IMAGE014
：
Figure 395216DEST_PATH_IMAGE015
Feature fusion module realigning matrix form
Figure 534074DEST_PATH_IMAGE009
In matrix form
Figure 664972DEST_PATH_IMAGE014
Are superposed and fused to obtain a matrix form C, namely
Figure 242584DEST_PATH_IMAGE016
Then, linear transformation is performed on the matrix form C
Figure 40907DEST_PATH_IMAGE002
、
Figure 412982DEST_PATH_IMAGE003
And
Figure 218127DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 81653DEST_PATH_IMAGE005
、
Figure 983750DEST_PATH_IMAGE027
And
Figure 464410DEST_PATH_IMAGE018
express the vector
Figure 835480DEST_PATH_IMAGE005
、
Figure 426998DEST_PATH_IMAGE013
And
Figure 449181DEST_PATH_IMAGE018
rewriting to matrix form Q 3 、K 3 And V 3 Then, the matrix form is obtained by self-attention calculation
Figure 913791DEST_PATH_IMAGE019
，
Figure 693528DEST_PATH_IMAGE019
Namely, the cross-modal fusion image characteristics:
Figure 151055DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 965427DEST_PATH_IMAGE021
is to fuse image features
Figure 132097DEST_PATH_IMAGE022
T is the transpose of the matrix,
Figure 664710DEST_PATH_IMAGE002
、
Figure 939309DEST_PATH_IMAGE003
and
Figure 873767DEST_PATH_IMAGE004
is a learnable parameter matrix.
Example 3
The present embodiment provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the PET-CT based cross-modal attention tumor segmentation method according to embodiment 1 when executing the computer program.
Example 4
The present embodiment provides a computer readable storage medium having stored thereon a computer program which, when being executed by a processor, implements the steps of the PET-CT based cross-modal attention tumor segmentation method of embodiment 1.

Claims (8)

1. A PET-CT-based cross-modal attention tumor segmentation method is characterized by comprising the following steps:
step S1, acquiring a PET image and a CT image;
step S2, respectively extracting the characteristics in the PET image and the CT image by using a self-attention mechanism;
step S3, using a self-attention mechanism to perform cross-modal fusion on the single-modal characteristics in the PET image and the CT image to obtain cross-modal fusion image characteristics;
step S4, segmenting the tumor region based on the cross-modality fusion image features.
2. The PET-CT based cross-modal attention tumor segmentation method of claim 1, wherein: in step S1, the registered PET image and CT image are acquired using a PET/CT scanner.
3. The PET-CT based cross-modal attention tumor segmentation method of claim 1, wherein: in step S2, the PET image and the CT image acquired in step S1 are respectively segmented, and the segmented images are transformed from a matrix to a vector form to obtain vectors
Figure 638790DEST_PATH_IMAGE001
(ii) a Amount of coincidence
Figure 149406DEST_PATH_IMAGE001
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image.
4. The PET-CT based cross-modal attention tumor segmentation method of claim 1, wherein: in step S3, the single-mode image features of the PET image obtained in step S2 are linearly transformed
Figure 397985DEST_PATH_IMAGE002
、
Figure 386669DEST_PATH_IMAGE003
And
Figure 404304DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 820242DEST_PATH_IMAGE005
、
Figure 556117DEST_PATH_IMAGE006
And
Figure 20596DEST_PATH_IMAGE007
express the vector
Figure 282950DEST_PATH_IMAGE005
、
Figure 745155DEST_PATH_IMAGE008
And
Figure 827381DEST_PATH_IMAGE007
rewriting to matrix form Q 1 、K 1 And V 1 Then, the matrix form is obtained by self-attention calculation
Figure 33234DEST_PATH_IMAGE009
：
Figure 150095DEST_PATH_IMAGE010
Performing linear transformation on the single-mode image characteristics of the CT image obtained in step S2
Figure 845518DEST_PATH_IMAGE002
、
Figure 555986DEST_PATH_IMAGE003
And
Figure 424584DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 271318DEST_PATH_IMAGE005
、
Figure 199959DEST_PATH_IMAGE006
And
Figure 194460DEST_PATH_IMAGE011
express the vector
Figure 742116DEST_PATH_IMAGE005
、
Figure 833569DEST_PATH_IMAGE012
And
Figure 808478DEST_PATH_IMAGE011
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self-attention calculation
Figure 352592DEST_PATH_IMAGE013
：
Figure 703939DEST_PATH_IMAGE014
Form of matrix
Figure 649898DEST_PATH_IMAGE009
In matrix form
Figure 858026DEST_PATH_IMAGE013
Are superposed and fused to obtain a matrix form C, i.e.
Figure 764802DEST_PATH_IMAGE015
Then, linear transformation is performed on the matrix form C
Figure 778894DEST_PATH_IMAGE002
、
Figure 720305DEST_PATH_IMAGE003
And
Figure 427230DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 555723DEST_PATH_IMAGE005
、
Figure 373507DEST_PATH_IMAGE016
And
Figure 903845DEST_PATH_IMAGE011
express the vector
Figure 453775DEST_PATH_IMAGE005
、
Figure 459777DEST_PATH_IMAGE006
And
Figure 956618DEST_PATH_IMAGE011
rewriting to matrix form Q 3 、K 3 And V 3 Then, the matrix form is obtained by self-attention calculation
Figure 466096DEST_PATH_IMAGE017
，
Figure 124611DEST_PATH_IMAGE017
Namely, the cross-modal fusion image characteristics:
Figure 617909DEST_PATH_IMAGE018
wherein the content of the first and second substances,
Figure 652861DEST_PATH_IMAGE019
is to fuse image features
Figure 220109DEST_PATH_IMAGE020
T is the transpose of the matrix,
Figure 174158DEST_PATH_IMAGE002
、
Figure 30119DEST_PATH_IMAGE003
and
Figure 727816DEST_PATH_IMAGE004
is a learnable parameter matrix.
5. A PET-CT based cross-modal attention tumor segmentation system, comprising:
the image acquisition module is used for acquiring a PET image and a CT image;
the characteristic extraction module is used for respectively extracting the characteristics in the PET image and the CT image acquired by the image acquisition module by using a self-attention mechanism;
the characteristic fusion module is used for extracting and obtaining single-mode characteristics in the PET image and the CT image by using the self-attention mechanism cross-mode fusion characteristic extraction module to obtain cross-mode fusion image characteristics;
and the tumor segmentation module is used for segmenting the tumor region based on the cross-modal fusion image features obtained by the feature fusion module.
6. The PET-CT based cross-modal attention tumor segmentation system of claim 5, wherein the image acquisition module acquires the registered PET image and CT image using a PET/CT scanner.
7. The PET-CT-based cross-modal attention tumor segmentation system of claim 5, wherein the feature extraction module is used for respectively segmenting the PET image and the CT image acquired by the image acquisition module, and transforming the segmented images from a matrix to a vector form to obtain a vector
Figure 87254DEST_PATH_IMAGE001
(ii) a Amount of coincidence
Figure 477784DEST_PATH_IMAGE001
And carrying out nonlinear transformation to obtain the single-mode image characteristics of the PET image and the CT image.
8. The PET-CT-based cross-modal attention tumor segmentation system of claim 5, wherein the feature fusion module performs linear transformation on the single-modal image features of the PET image obtained by the feature extraction module
Figure 555461DEST_PATH_IMAGE002
、
Figure 322429DEST_PATH_IMAGE003
And
Figure 598689DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 35487DEST_PATH_IMAGE005
、
Figure 725094DEST_PATH_IMAGE006
And
Figure 905540DEST_PATH_IMAGE011
express the vector
Figure 98624DEST_PATH_IMAGE005
、
Figure 706323DEST_PATH_IMAGE012
And
Figure 883226DEST_PATH_IMAGE011
rewriting to matrix form Q 1 、K 1 And V 1 Then, the matrix form is obtained by self attention calculation
Figure 398521DEST_PATH_IMAGE009
：
Figure 852636DEST_PATH_IMAGE010
The characteristic fusion module carries out linear transformation on the single-mode image characteristics of the CT image obtained by the characteristic extraction module
Figure 755870DEST_PATH_IMAGE002
、
Figure 295436DEST_PATH_IMAGE003
And
Figure 676739DEST_PATH_IMAGE004
to obtain different vector expressions
Figure 985360DEST_PATH_IMAGE005
、
Figure 731599DEST_PATH_IMAGE008
And
Figure 617516DEST_PATH_IMAGE011
express the vector
Figure 943455DEST_PATH_IMAGE005
、
Figure 231217DEST_PATH_IMAGE016
And
Figure 86040DEST_PATH_IMAGE011
rewriting to matrix form Q 2 、K 2 And V 2 Then, the matrix form is obtained by self attention calculation
Figure 724832DEST_PATH_IMAGE013
：
Figure 385620DEST_PATH_IMAGE014
Feature fusion module recouplingMatrix form
Figure 403255DEST_PATH_IMAGE009
In matrix form
Figure 84772DEST_PATH_IMAGE021
Are superposed and fused to obtain a matrix form C, namely
Figure 820647DEST_PATH_IMAGE015
Then, linear transformation is performed on the matrix form C
Figure 347443DEST_PATH_IMAGE002
、
Figure 219584DEST_PATH_IMAGE003
And
Figure 744106DEST_PATH_IMAGE022
to obtain different vector expressions
Figure 91911DEST_PATH_IMAGE005
、
Figure 32185DEST_PATH_IMAGE023
And
Figure 149046DEST_PATH_IMAGE024
express the vector
Figure 47732DEST_PATH_IMAGE005
、
Figure 289357DEST_PATH_IMAGE025
And
Figure 423535DEST_PATH_IMAGE024
rewriting to matrix form Q 3 、K 3 And V 3 Then, the matrix form is obtained by self-attention calculation
Figure 270269DEST_PATH_IMAGE017
，
Figure 198910DEST_PATH_IMAGE017
Namely, the cross-modal fusion image characteristics:
Figure 193411DEST_PATH_IMAGE018
wherein the content of the first and second substances,
Figure 741067DEST_PATH_IMAGE019
is to fuse image features
Figure 832520DEST_PATH_IMAGE020
T is the transpose of the matrix,
Figure 807429DEST_PATH_IMAGE002
、
Figure 351543DEST_PATH_IMAGE003
and
Figure 702890DEST_PATH_IMAGE004
is a learnable parameter matrix.
CN202210807701.0A 2022-07-11 2022-07-11 Cross-modal attention tumor segmentation method and system based on PET-CT Pending CN114862881A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210807701.0A CN114862881A (en) 2022-07-11 2022-07-11 Cross-modal attention tumor segmentation method and system based on PET-CT

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210807701.0A CN114862881A (en) 2022-07-11 2022-07-11 Cross-modal attention tumor segmentation method and system based on PET-CT

Publications (1)

Publication Number Publication Date
CN114862881A true CN114862881A (en) 2022-08-05

Family

ID=82627039

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210807701.0A Pending CN114862881A (en) 2022-07-11 2022-07-11 Cross-modal attention tumor segmentation method and system based on PET-CT

Country Status (1)

Country Link
CN (1) CN114862881A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109503590A (en) * 2018-11-22 2019-03-22 四川大学华西医院 Using 7-deazaadenine base as mother nucleus18F-PET/CT tracer agent and preparation method thereof
US20190114773A1 (en) * 2017-10-13 2019-04-18 Beijing Curacloud Technology Co., Ltd. Systems and methods for cross-modality image segmentation
CN113035334A (en) * 2021-05-24 2021-06-25 四川大学 Automatic delineation method and device for radiotherapy target area of nasal cavity NKT cell lymphoma
CN113496495A (en) * 2021-06-25 2021-10-12 华中科技大学 Medical image segmentation model building method capable of realizing missing input and segmentation method
CN113951866A (en) * 2021-10-28 2022-01-21 北京深睿博联科技有限责任公司 Deep learning-based uterine fibroid diagnosis method and device
WO2022032823A1 (en) * 2020-08-10 2022-02-17 中国科学院深圳先进技术研究院 Image segmentation method, apparatus and device, and storage medium
CN114266726A (en) * 2021-11-22 2022-04-01 中国科学院深圳先进技术研究院 Medical image segmentation method, system, terminal and storage medium
US20220108478A1 (en) * 2020-10-02 2022-04-07 Google Llc Processing images using self-attention based neural networks
CN114359642A (en) * 2022-01-12 2022-04-15 大连理工大学 Multi-modal medical image multi-organ positioning method based on one-to-one target query Transformer

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190114773A1 (en) * 2017-10-13 2019-04-18 Beijing Curacloud Technology Co., Ltd. Systems and methods for cross-modality image segmentation
CN109503590A (en) * 2018-11-22 2019-03-22 四川大学华西医院 Using 7-deazaadenine base as mother nucleus18F-PET/CT tracer agent and preparation method thereof
WO2022032823A1 (en) * 2020-08-10 2022-02-17 中国科学院深圳先进技术研究院 Image segmentation method, apparatus and device, and storage medium
US20220108478A1 (en) * 2020-10-02 2022-04-07 Google Llc Processing images using self-attention based neural networks
CN113035334A (en) * 2021-05-24 2021-06-25 四川大学 Automatic delineation method and device for radiotherapy target area of nasal cavity NKT cell lymphoma
CN113496495A (en) * 2021-06-25 2021-10-12 华中科技大学 Medical image segmentation model building method capable of realizing missing input and segmentation method
CN113951866A (en) * 2021-10-28 2022-01-21 北京深睿博联科技有限责任公司 Deep learning-based uterine fibroid diagnosis method and device
CN114266726A (en) * 2021-11-22 2022-04-01 中国科学院深圳先进技术研究院 Medical image segmentation method, system, terminal and storage medium
CN114359642A (en) * 2022-01-12 2022-04-15 大连理工大学 Multi-modal medical image multi-organ positioning method based on one-to-one target query Transformer

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
BICAO LI等: "CSpA-DN:Channel and Spatial attention Dense Network for Fusing PET AND MRI images", 《2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION》 *
LEI Y等: "Low dose PET imaging WITH CT-aided cycle-consistent adversarial networks", 《MEDICAL IMAGING 2020:PHYSICS OF MEDICAL IMAGING》 *
WANG JIANYONG等: "A new delay connection for long-short Term memory networks", 《INTERNATIONAL JOURNAL OF NEURAL SYSTEMS》 *
冯诺等: "特征重用和注意力机制下肝肿瘤自动分类", 《中国图象图形学报》 *
朱晨光: "《机器阅读理解算法与实践》", 30 April 2020, 机械工业出版社 *
李林等: "PET/CT真正全身显像在结外NK/T细胞淋巴瘤（鼻型）患者中的应用价值", 《四川大学学报（医学版）》 *
石磊等: "自然语言处理中的注意力机制研究综述", 《数据分析与知识发现》 *

Similar Documents

Publication Publication Date Title
Wang et al. 3D auto-context-based locality adaptive multi-modality GANs for PET synthesis
Wang et al. Machine learning in quantitative PET: A review of attenuation correction and low-count image reconstruction methods
Wang et al. Predicting standard-dose PET image from low-dose PET and multimodal MR images using mapping-based sparse representation
Spuhler et al. Full‐count PET recovery from low‐count image using a dilated convolutional neural network
Kang et al. Prediction of standard‐dose brain PET image by using MRI and low‐dose brain [18F] FDG PET images
Lin et al. Deep learning based automatic segmentation of metastasis hotspots in thorax bone SPECT images
Cheng et al. Applications of artificial intelligence in nuclear medicine image generation
WO2016033458A1 (en) Restoring image quality of reduced radiotracer dose positron emission tomography (pet) images using combined pet and magnetic resonance (mr)
Chowdhury et al. Concurrent segmentation of the prostate on MRI and CT via linked statistical shape models for radiotherapy planning
Yang et al. A hybrid approach for fusing 4D‐MRI temporal information with 3D‐CT for the study of lung and lung tumor motion
Xie et al. Anatomically aided PET image reconstruction using deep neural networks
CN110415310A (en) Medical scanning imaging method, device, storage medium and computer equipment
Jin et al. Registration of PET and CT images based on multiresolution gradient of mutual information demons algorithm for positioning esophageal cancer patients
Ao et al. Improved dosimetry for targeted radionuclide therapy using nonrigid registration on sequential SPECT images
Tong et al. Disease quantification on PET/CT images without explicit object delineation
Lin et al. Classifying functional nuclear images with convolutional neural networks: a survey
Toyonaga et al. Deep learning–based attenuation correction for whole-body PET—a multi-tracer study with 18F-FDG, 68 Ga-DOTATATE, and 18F-Fluciclovine
Torkaman et al. Direct image-based attenuation correction using conditional generative adversarial network for SPECT myocardial perfusion imaging
Bauer et al. Automated measurement of uptake in cerebellum, liver, and aortic arch in full‐body FDG PET/CT scans
Amirkolaee et al. Development of a GAN architecture based on integrating global and local information for paired and unpaired medical image translation
Sanaat et al. A cycle-consistent adversarial network for brain PET partial volume correction without prior anatomical information
Li et al. Adaptive 3D noise level‐guided restoration network for low‐dose positron emission tomography imaging
Wang et al. 3D multi-modality Transformer-GAN for high-quality PET reconstruction
CN114862881A (en) Cross-modal attention tumor segmentation method and system based on PET-CT
Lei et al. Estimating standard-dose PET from low-dose PET with deep learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20220805

RJ01 Rejection of invention patent application after publication