CN110288609B - Multi-modal whole-heart image segmentation method guided by attention mechanism - Google Patents

Abstract

The invention discloses a multi-modal whole heart image segmentation method guided by an attention mechanism. And generating images corresponding to other modalities from the current modality through Cycle-GAN to expand the training set, and then performing image segmentation on the original image and the corresponding generated images through a semi-twin network. The semi-twin network comprises two independent encoders and a shared decoder, wherein the encoders are used for learning the characteristics private to the modes, the characteristics are subjected to characteristic fusion through an attention mechanism module, and then the characteristics common to the modes are extracted through the decoder for final segmentation. The invention fully utilizes the modal shared information and the private information and improves the segmentation precision.

Description

Multi-modal whole-heart image segmentation method guided by attention mechanism

Technical Field

The invention belongs to the field of medical images, and particularly relates to a multi-mode whole heart image segmentation method.

Background

According to the American Heart Association (AHA)2019 statistical report of heart disease and stroke, approximately 1,055,000 people suffered from coronary heart disease in 2019 in the united states, including 720,000 new and 335,000 recurrent coronary artery cases. In this sense, early diagnosis and treatment play an important role in reducing mortality and morbidity of cardiovascular disease. During early diagnosis, physicians often collect imaging information from different modalities (e.g., MR and CT) for comprehensive investigation, with one important prerequisite being to accurately segment cardiac substructures from different modality images. However, conventional manual segmentation is very time consuming and laborious. Therefore, it is urgent to develop a method for automatic segmentation of the whole heart.

Although methods based on deep convolutional neural networks have been widely used to segment other organs, the application of these methods to the multi-modal whole-heart segmentation task remains limited because: 1) modal inconsistency: images from different modalities have significant appearance differences; 2) the complicated results are: different cardiac substructures are connected and sometimes even overlap; 3) the heart of each patient also has differences in appearance.

In recent years, there have been several attempts at multi-modal whole-heart segmentation. For example, sinoqi et al propose an unsupervised cross-domain generation framework with counterlearning cross-modality medical image segmentation. Zhengzheng et al propose a method of simultaneous learning transformation and segmentation of medical 3D images that can learn unpaired datasets and keep their anatomical structure unchanged. Regarding image generation, CycleGAN can migrate and generate styles of unpaired images from one domain to another, but it has the problem of lacking labels for deformation constraints. Ronneberger et al inspired by the full convolution neural network FCN, proposed a "U-net" structure, which includes a contraction path to capture the upper and lower literature information and a symmetric expansion path to obtain accurate local information, and is commonly used for medical image segmentation. However, none of the above methods can fully utilize the information or correlation that can be shared between the two modalities, and cannot effectively overcome the above-mentioned limitations.

Disclosure of Invention

In order to solve the technical problems mentioned in the background art, the invention provides a multi-modal whole heart image segmentation method guided by an attention mechanism, which makes full use of modal shared information and private information and improves the segmentation precision.

In order to achieve the technical purpose, the technical scheme of the invention is as follows:

an attention mechanism guided multi-modal whole heart image segmentation method comprises the following steps:

(1) generating a cross-modal image:

introducing a generation countermeasure network, wherein the network comprises 2 generators and 2 discriminators, the generators respectively correspond to the CT images and the MRI images, and the original CT images and the original MRI images are respectively input into the corresponding generators to respectively generate images corresponding to another modality;

(2) cross-modal feature learning and image segmentation:

constructing a semi-twin network, wherein the network comprises 2 independent encoders and 1 shared decoder, and an attention mechanism module is arranged between the encoders and the decoders; the method comprises the steps of inputting an original CT image and an MRI image generated by the original CT image into one encoder, inputting the original MRI image and the CT image generated by the original MRI image into the other encoder, wherein the encoder comprises a plurality of layers of down-sampling layers, 2 encoders output 2 characteristic spectrograms proprietary to modalities, an attention mechanism module fuses the 2 characteristics proprietary to the modalities and sends the characteristics to a shared decoder, and the decoder outputs the segmentation result of the image.

Further, in generating the countermeasure network, a cycle consistency loss function L is defined as follows_cyc(G_A,G_B)：

In the above formula, x_A、x_BRespectively an original CT image sample and an MRI image sample,

to obey p_d(x_A) X of distribution_AIn the expectation that the position of the target is not changed,

to obey p_d(x_B) X of distribution_BExpectation of (1), G_A、G_BA generator for corresponding CT images and MRI images, respectively;

in generating a countermeasure network, a segmentation loss function L is defined as follows_seg(S_A/B,G_A,G_B)：

In the above equation, S is mapped_A/BA → Y, B → Y, A represents CT modality, B represents MRI modality, Y represents segmentation label, i represents a training sample,n is the total number of training samples, y_A、y_BThe real segmentation results in the A mode and the B mode are respectively.

Further, the cyclic consistency loss function and the segmentation loss function are integrated to define a total loss function L (G)_A,G_B,D_A,D_B,S_A/B)：

L(G_A,G_B,D_A,D_B,S_A/B)＝L_GAN(G_A,D_A)+L_GAN(G_B,D_B)+λL_cyc(G_A,G_B)+γL_seg(S_A/B,G_A,G_B)

In the above formula, L_GAN(G_A,D_A) And L_GAN(G_B,D_B) To generate a pairwise loss-resistance function, D_A、D_BCorresponding to the discriminators of the CT image and the MRI image, lambda and gamma are weight coefficients of the loss function.

Further, in a semi-twin network, the encoder localizes high-resolution features and captures more accurate information; the decoder propagates contextual information to higher resolution layers and learns higher level semantic information.

Further, the flow of the attention mechanism module is as follows:

firstly, connecting feature maps output by 2 encoders through a channel layer to obtain a primary fusion map, carrying out matrix recombination on the primary fusion map to obtain a map 1, sequentially carrying out matrix recombination and transposition on the primary fusion map to obtain a map 2, carrying out vector product on the map 1 and the map 2 to obtain an attention map through a softmax function, carrying out element-by-element summation on the result of vector product of the attention map and the primary fusion map to obtain a final feature fusion map.

Adopt the beneficial effect that above-mentioned technical scheme brought:

the invention adopts the improved cycleGAN to generate the cross-modal image to expand the training set, and reduces the image inconsistency of the modal layer; the invention provides a novel cross-modal attention mechanism guided semi-twin network, which is used for learning the characteristics of modality sharability and modality privacy and carrying out multi-modal full heart image segmentation. The method effectively solves the problems that the number of marked 3D full-heart CT and MRI images is small, cross-modal related information is not fully utilized in the prior art, and the like, and improves the segmentation precision, so the method has higher application value.

Drawings

FIG. 1 is a flow chart of a method of the present invention;

FIG. 2 is a network architecture diagram of the present invention;

FIG. 3 is a flow diagram of an attention mechanism module of the present invention;

FIG. 4 is a diagram of the segmentation result of the embodiment, which includes two subgraphs (a) and (b).

Detailed Description

The technical scheme of the invention is explained in detail in the following with the accompanying drawings.

The invention designs a multi-modal whole heart image segmentation method guided by an attention mechanism, which comprises the following steps as shown in figures 1-2:

1. generating a cross-modal image:

a generative countermeasure network was introduced comprising 2 generators and 2 discriminators, corresponding to CT images and MRI images respectively, which, by gaming each other during learning, could get a very good output. The original CT image and the MRI image are input to respective generators, and images corresponding to the other modalities are generated.

To solve the problem that the generator is to resist learning from unpaired images, the invention uses samples G that are forced to be generated_A(G_B(x_A) And G)_B(G_A(x_B) Constraint that remains consistent with the original image, thus defining a circular consistency loss function:

in the above formula, x_A、x_BRespectively an original CT image sample and an MRI image sample,

to obey p_d(x_A) X of distribution_AIn the expectation that the position of the target is not changed,

to obey p_d(x_B) X of distribution_BExpectation of (1), G_A、G_BCorresponding to the CT image and MRI image, respectively.

If the transformations learned by two generators are invertible to each other, then the two generators will pass the cycle-consistent test without any penalty. To prevent this problem of data deformation, another auxiliary mapping S is defined_A/BA → Y, B → Y, wherein A represents the CT modality, B represents the MR modality, Y represents the segmentation label, and the cross entropy is taken as a constrained segmentation loss function between the segmentation result and the true segmentation result:

in the above formula, i represents a training sample, N is the total number of training samples, y_A、y_BThe real segmentation results in the A mode and the B mode are respectively.

And integrating the cycle consistency loss function and the segmentation loss function to define a total loss function:

L(G_A,G_B,D_A,D_B,S_A/B)＝L_GAN(G_A,D_A)+L_GAN(G_B,D_B)+λL_cyc(G_A,G_B)+γL_seg(S_A/B,G_A,G_B)

in the above formula, L_GAN(G_A,D_A) And L_GAN(G_B,D_B) To generate a pairwise loss-resistance function, D_A、D_BCorresponding to the discriminators of the CT image and the MRI image, lambda and gamma are weight coefficients of the loss function.

2. Cross-modal feature learning and image segmentation:

a semi-twin network was constructed comprising 2 independent encoders and 1 shared decoder with an attention mechanism module between the encoders and decoders. The method comprises the steps of inputting an original CT image and an MRI image generated by the original CT image into one encoder, inputting the original MRI image and the CT image generated by the original MRI image into the other encoder, wherein the encoder comprises a plurality of layers of down-sampling layers, 2 encoders output 2 characteristic spectrograms proprietary to modalities, an attention mechanism module fuses the 2 characteristics proprietary to the modalities and sends the characteristics to a shared decoder, and the decoder outputs the segmentation result of the image.

The role of the encoder is to localize the high resolution features and capture more accurate information; the role of the decoder is to propagate contextual information to higher resolution layers and to learn higher level semantic information.

The invention designs a channel-wise attention mechanism module between an encoder and a decoder. The method takes a characteristic map with boundary information of each heart substructure in two modes as input, and outputs a new characteristic map. Note that the flow of the force mechanism module is shown in fig. 3. Firstly, the feature maps (with the assumed size of C) output by 2 encoders are obtained₁XHxW) to obtain a preliminary fusion map (size (C)₁+C₂) xHxW), performing matrix recombination on the primary fusion map to obtain a map 1, sequentially performing matrix recombination and transposition on the primary fusion map to obtain a map 2, and performing a vector product on the map 1 and the map 2 to obtain an attention map (with the size of (C) through a softmax function₁+C₂)×(C₁+C₂) Performing element-by-element addition on the result of vector multiplication of the attention map and the primary fusion map to obtain a final feature fusion map (with the size of (C))₁+C₂)×H×W)。

Since both modalities are used to describe the same heart organ, it is assumed that the two modalities may contain many associated features, which are in the off-diagonal blocks C of the attention map₁×C₂And C₂×C₁Part can beIs embodied. The modes also include some characteristics unique to the modes, which is in the diagonal block C₁×C₁And C₂×C₂Portions may also be embodied.

The effectiveness of the present invention is verified by simulation experiments below.

Random gradient descent is carried out by adopting an Adam method, and the learning rate is 2 multiplied by 10^-4Other settings refer entirely to the settings of CycleGAN to train the generator and arbiter. To accelerate the training process, a split pre-training G is selected_A/BAnd D_A/BAnd then the end-to-end split network is trained completely.

The method is applied to a public data set provided by the Challenge competition Multi-modification white Heart search of the flagship conference MICCAI 2017 in the field of medical images. The data set contained 3D images of 20 MRI and 20 CT unpaired. The substructures of the data set have all been labeled by the radiologist. The goal of segmentation is to segment 7 substructures of the heart including: left ventricle, left atrium, right ventricle, right atrium, aorta, pulmonary artery, and myocardium. In the training process, the data set is divided into a training set (10 samples) and a test set (10 samples), and a two-fold cross-validation is performed. Let the CT modality be a and the MRI modality be B.

All sample orientations are changed to coronal positions by the software ITK-SNAP because the direction of data acquisition is different. All sections with valid tags are cut out and then cut into 2534 CT and 2208 MRI slices in 2D. These differently shaped slices are then resized to a size of 128 x 128.

In order to measure the difference between the segmentation result and the actual result, a Dice coefficient is used as an evaluation index. Dice is used to measure the proportion of coincidence between the real mark and the segmentation result. Higher Dice indicates higher segmentation accuracy.

The visualization contrast result of heart segmentation is shown in fig. 4, where (a) in fig. 4 is the visualization result of heart segmentation from CT to MRI, and (b) is the visualization result of heart segmentation from MRI to CT, and Ours in the figure is the segmentation result obtained by the segmentation method of the present invention. It can be seen that the resulting image is very similar to the original image and does not have any significant distortion and segmentation errors.

The results of the segmentation using the different methods are shown in table 1. As can be seen from the table, the present invention successfully extracts features shared between modalities to improve the segmentation accuracy of MRI images. The invention firstly evaluates the result of the full convolution neural network (FCN) respectively applied to the two modal segmentations as a reference line. U-net was then evaluated for application to both modalities, respectively. To further verify the validity of the method of the present invention, only the CycleGAN method was used for cross-modality image generation on this data set. A comparison experiment shows that the method has a not obvious effect on improving the CT segmentation precision, but the obvious improvement of the MRI segmentation precision shows that the attention mechanism is effective. Also, the attention mechanism effectively avoids the risk of "bad" MRI images carrying "good" CT images.

TABLE 1

Method	Aorta	Left atrium	Left ventricle	Cardiac muscle	Right ventricle	Right atrium	Pulmonary artery	Average
									FCN_CT	0.8863	0.8163	0.8838	0.8541	0.7885	0.7940	0.7758	0.8284
FCN_MRI	0.6931	0.6882	0.8006	0.7161	0.6759	0.7593	0.6693	0.7146
									Unet_CT	0.8992	0.7704	0.8381	0.8162	0.7643	0.8003	0.7947	0.8119
Unet_MRI	0.7719	0.6942	0.7751	0.6961	0.6983	0.7829	0.7171	0.7336
									CycleGAN_CT	0.9407	0.8277	0.8362	0.7942	0.8064	0.8134	0.8103	0.8327
CycleGAN_MRI	0.7686	0.6555	0.7612	0.7038	0.6637	0.7658	0.6973	0.7165
									Ours_CT	0.9282	0.8131	0.8497	0.7869	0.8066	0.8255	0.8391	0.8356
Ours_MRI	0.7875	0.6940	0.8031	0.7189	0.6733	0.7967	0.7147	0.7412

The embodiments are only for illustrating the technical idea of the present invention, and the technical idea of the present invention is not limited thereto, and any modifications made on the basis of the technical scheme according to the technical idea of the present invention fall within the scope of the present invention.

Claims (5)

1. An attention mechanism guided multi-modal whole heart image segmentation method is characterized by comprising the following steps:

(1) generating a cross-modal image:

introducing a generation countermeasure network, wherein the network comprises 2 generators and 2 discriminators, one set of generators and discriminators corresponds to CT images, and the other set of generators and discriminators corresponds to MRI images; inputting the original CT image and the MRI image into corresponding generators respectively, and generating images corresponding to the other modalities respectively;

(2) cross-modal feature learning and image segmentation:

constructing a semi-twin network, wherein the network comprises 2 independent encoders and 1 shared decoder, and an attention mechanism module is arranged between the encoders and the decoders; the method comprises the steps of inputting an original CT image and an MRI image generated by the original CT image into one encoder, inputting the original MRI image and the CT image generated by the original MRI image into the other encoder, wherein the encoder comprises a plurality of layers of down-sampling layers, 2 encoders output 2 characteristic spectrograms proprietary to modalities, an attention mechanism module fuses the 2 characteristics proprietary to the modalities and sends the characteristics to a shared decoder, and the decoder outputs the segmentation result of the image.

2. The attention mechanism-guided multi-modal whole-heart image segmentation method according to claim 1, wherein in generating the countermeasure network, a cyclic consistency loss function L is defined as follows_cyc(G_A,G_B)：

In the above formula, x_A、x_BRespectively an original CT image sample and an MRI image sample,

to obey p_d(x_A) X of distribution_AIn the expectation that the position of the target is not changed,

to obey p_d(x_B) X of distribution_BExpectation of (1), G_A、G_BA generator for corresponding CT images and MRI images, respectively;

in generating a countermeasure network, a segmentation loss function L is defined as follows_seg(S_A/B,G_A,G_B)：

In the above equation, S is mapped_A/BA → Y, B → Y, A represents CT mode, B represents MRI mode, Y represents segmentation label, i represents a training sample, N is total number of training samples, Y represents_A、y_BThe real segmentation results in the A mode and the B mode are respectively.

3. The attention mechanism-guided multi-modal whole-heart image segmentation method according to claim 2, wherein the loop consistency loss function is synthesizedNumber and division loss function, defining a total loss function L (G)_A,G_B,D_A,D_B,S_A/B)：

L(G_A,G_B,D_A,D_B,S_A/B)＝L_GAN(G_A,D_A)+L_GAN(G_B,D_B)+λL_cyc(G_A,G_B)+γL_seg(S_A/B,G_A,G_B)

In the above formula, L_GAN(G_A,D_A) And L_GAN(G_B,D_B) To generate a pairwise loss-resistance function, D_A、D_BCorresponding to the discriminators of the CT image and the MRI image, lambda and gamma are weight coefficients of the loss function.

4. The attention mechanism-guided multi-modal whole-heart image segmentation method as claimed in claim 1, wherein in a semi-twin network, the encoder localizes high-resolution features and captures more accurate information; the decoder propagates contextual information to higher resolution layers and learns higher level semantic information.

5. The attention mechanism-guided multi-modal whole-heart image segmentation method according to claim 1, wherein the flow of the attention mechanism module is as follows:

firstly, connecting feature maps output by 2 encoders through a channel layer to obtain a primary fusion map, carrying out matrix recombination on the primary fusion map to obtain a map 1, sequentially carrying out matrix recombination and transposition on the primary fusion map to obtain a map 2, carrying out vector product on the map 1 and the map 2 to obtain an attention map through a softmax function, carrying out element-by-element summation on the result of vector product of the attention map and the primary fusion map to obtain a final feature fusion map.

Images