CN112348786A - One-shot brain image segmentation method based on bidirectional correlation - Google Patents

One-shot brain image segmentation method based on bidirectional correlation Download PDF

Info

Publication number
CN112348786A
CN112348786A CN202011186634.2A CN202011186634A CN112348786A CN 112348786 A CN112348786 A CN 112348786A CN 202011186634 A CN202011186634 A CN 202011186634A CN 112348786 A CN112348786 A CN 112348786A
Authority
CN
China
Prior art keywords
image
generator
atlas
discriminator
reconstructed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011186634.2A
Other languages
Chinese (zh)
Other versions
CN112348786B (en
Inventor
王连生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xiamen University
Original Assignee
Xiamen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xiamen University filed Critical Xiamen University
Priority to CN202011186634.2A priority Critical patent/CN112348786B/en
Publication of CN112348786A publication Critical patent/CN112348786A/en
Application granted granted Critical
Publication of CN112348786B publication Critical patent/CN112348786B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30016—Brain

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)

Abstract

The invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which comprises the following steps of: constructing an image transformation model comprising a generator GFGenerator GBAnd two discriminators D for inputting the atlas x and the unlabelled image y into the generator GFShunting processing to obtain forward mapping delta pFAnd the reconstructed images are distinguished by a discriminator D
Figure DDA0002751566780000015
Obtaining a reconstructed image with the unmarked image y
Figure DDA0002751566780000011
Will reconstruct the image
Figure DDA0002751566780000013
And atlas x input generator GBGet backward mapping Δ pBAnd the reconstructed images are distinguished by a discriminator D
Figure DDA0002751566780000014
And obtaining a reconstructed image with the atlas x
Figure DDA0002751566780000016
Through generator GFDiscriminator D and generator GBMutually constraining to obtain a final forward mapping delta pF and obtaining a marked reconstructed image through warp operation
Figure DDA0002751566780000012
The method simultaneously learns the forward mapping from the image set x to the unlabelled image y and the backward mapping from the unlabelled image y to the image set x through the image transformation model, and restricts the forward mapping through the backward mapping, so that the accuracy of the forward mapping is improved.

Description

One-shot brain image segmentation method based on bidirectional correlation
Technical Field
The invention relates to the technical field of medical image segmentation, in particular to a one-shot brain image segmentation method based on bidirectional correlation.
Background
A common method for brain anatomy segmentation is segmentation by conventional machine learning, but conventional machine learning relies on manually extracted features, which have limited feature representation and generalization capabilities, so Convolutional Neural Network (CNN) learning has been developed, because it is completely data-driven and can automatically retrieve hierarchical features using self-learned advanced features, eliminating the limitations of manual features in conventional machine learning methods, with fully labeled data, the convolutional neural network has a better effect in a fully supervised segmentation task, using a segmentation algorithm based on forward correlation, i.e. improving the segmentation network to learn the forward mapping of an atlas x to an unlabeled image y, applying the learned correlation mapping to the labeling of the atlas, to obtain the labeling of the unlabeled image y, but such a method learns only the forward mapping of the atlas x to the unlabeled image y, the forward mapping is constrained only by similarity loss and smoothness loss, and the mapping is highly difficult to control, resulting in low accuracy of mapping learning.
Disclosure of Invention
The invention aims to provide a one-shot brain image segmentation method based on bidirectional correlation, which is used for constructing an image transformation model, learning forward mapping from an atlas x to an unlabelled image y and backward mapping from the unlabelled image y to the atlas x through the image transformation model at the same time, and constraining forward mapping through the backward mapping to improve the accuracy of the forward mapping.
In order to achieve the purpose, the invention adopts the following technical scheme:
a one-shot brain image segmentation method based on bidirectional correlation comprises the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into an atlas x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, a generator GFGenerator GBAll match a discriminator D, generator GFAnd generator GBThe structure is the same and comprises a twin coder and a decoder;
s3, input generator G for image set x and unlabelled image yFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF；
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Figure BDA0002751566760000021
Differentiating the reconstructed images by a discriminator D
Figure BDA0002751566760000022
The unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
Figure BDA0002751566760000023
S5, reconstructing the image
Figure BDA0002751566760000024
And atlas x input generator GBBy means of a generator GBThe obtained twin encoder extracts the relevant characteristic diagram, fuses the characteristic diagram and inputs the characteristic diagram into a decoder, and the decoder is connected with the twin encoderMatching to obtain a reconstructed image
Figure BDA0002751566760000025
Backward mapping Δ p to atlas xB；
S6, reconstructing the image
Figure BDA0002751566760000026
Obtaining reconstructed images by warp operation
Figure BDA0002751566760000027
Differentiating the reconstructed images by a discriminator D
Figure BDA0002751566760000028
And atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
Figure BDA0002751566760000029
S7, reconstructing the image
Figure BDA00027515667600000210
Similarity to atlas x, so that Generator GFDiscriminator D and generator GBMutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Figure BDA00027515667600000211
Further, the generator GFAnd generator GBThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unmarked image y/reconstructed image are processed in a shunting manner through the encoding sub-modules
Figure BDA00027515667600000212
And a drawing set x, inputting the extracted related characteristic drawing into a double-attention module, and dividing the characteristic drawing into a plurality of groups through the double-attention moduleAnd (3) learning the spatial information and the channel information of the relevant characteristic diagram and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
Furthermore, the coding submodule has 5 coding submodules which are respectively a first coding submodule, a second coding submodule, a third coding submodule, a fourth coding submodule and a fifth coding submodule and forms 1 processing stream, and the atlas x and the unmarked image y/reconstructed image are respectively processed by 2 processing streams
Figure BDA00027515667600000213
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs an image set x to a forward mapping delta p marked by the image of the non-submoduleFReconstruction of images
Figure BDA0002751566760000031
Backward mapping Δ p to atlas xB。
Further, the double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new feature map.
Further, the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Further, the arbiter D adopts a PatchGAN arbiter.
Further, the image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and the generator G is constrained through the similarity lossFTo obtain similar reconstructed images
Figure BDA0002751566760000037
And an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Constraint generator G through spatial cyclic consistency lossBTo obtain similar reconstructed images
Figure BDA0002751566760000036
And atlas x; the discriminator D is constrained by the penalty on antagonism.
Further, the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
Figure BDA0002751566760000032
where t represents a voxel point in the image, fy(t) and
Figure BDA0002751566760000033
respectively representing the calculation of the unmarked image y and the reconstruction image
Figure BDA0002751566760000034
Local mean intensity function:
Figure BDA0002751566760000035
tidenotes a volume around t of l3Coordinates within the range.
After adopting the technical scheme, compared with the background technology, the invention has the following advantages:
1. the invention is obtained byClassifying the brain anatomical structure image to obtain an atlas x with labels and an unlabelled image y, and constructing an image transformation model with 2 generators and 2 discriminators D, wherein the 2 generators are generators G respectivelyFGenerator GBInputting the atlas x and the unlabelled image y into a generator GFThe twin encoder and decoder performs forward mapping to obtain a forward mapping Δ pFThrough a discriminator D and a generator GFPerforming countermeasure, so that the atlas x is subjected to warp operation to obtain a reconstructed image similar to the unlabelled image y
Figure BDA0002751566760000041
Will reconstruct the image
Figure BDA0002751566760000042
Input generator GBThe twin encoder and decoder performs backward mapping to obtain backward mapping delta pBBy means of a discriminator D and a generator GBConfrontation is carried out to obtain a reconstructed image similar to the atlas x
Figure BDA0002751566760000043
From the reconstructed image
Figure BDA0002751566760000044
And (3) carrying out circulation to enable backward mapping to restrict forward mapping so as to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation, and obtaining a labeled reconstructed image
Figure BDA0002751566760000045
Through generator GFGenerator GBRespectively confronted with a discriminator D, so that the image change model obtains forward mapping delta p with the highest accuracyFAnd with the label to reconstruct the image
Figure BDA0002751566760000046
2. The invention introduces loss modules, wherein the loss modules comprise similarity loss, smooth loss and nullCyclic consistency loss and antagonism loss, through different losses to generator GFGenerator GBAnd 2 discriminators D carry out constraint to improve the accuracy of the image transformation model.
3. The discriminator D selects the PatchGAN discriminator, the PatchGAN discriminator can better discriminate the local part of the image, each patch is judged true and false by dividing the image into a plurality of patches, and finally the judgment of the image level is obtained, and the accuracy and the performance are superior to those of a common discriminator.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
FIG. 2 is a schematic diagram of the overall structure of an image transformation model according to the present invention;
FIG. 3 is a schematic diagram of the operation of a twin encoder and decoder according to the present invention;
FIG. 4 is a schematic diagram of the decoder operation of the present invention;
FIG. 5 is a schematic structural diagram of a dual-note module of the present invention;
FIG. 6 is a schematic structural diagram of a discriminator D according to the invention;
FIG. 7 is a diagram illustrating the segmentation result of ICGAN forward mapping according to the present invention;
FIG. 8 is a graph showing the comparison of the segmentation results of the SiamENet and the ICGAN according to the present invention;
FIG. 9 is a diagram illustrating the segmentation results of the ICGAN forward mapping and backward mapping according to the present invention;
FIG. 10 is a graph comparing the results of the visual segmentation of the Simeneet, ICGAN and RCGAN of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Examples
As shown in fig. 1 to 7, the present invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which includes the following steps:
and S1, acquiring and classifying the brain anatomical structure image to obtain a labeled image and an unlabeled image y, and dividing the labeled image into an atlas x.
S2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, a generator GFGenerator GBAll match a discriminator D, generator GFAnd generator GBAll structurally identical comprise a twin encoder and a decoder.
S3, input generator G for image set x and unlabelled image yFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF。
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Figure BDA0002751566760000051
Differentiating the reconstructed images by a discriminator D
Figure BDA0002751566760000052
The unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
Figure BDA0002751566760000053
S5, reconstructing the image
Figure BDA0002751566760000054
And atlas x input generator GBBy means of a generator GBExtracting relevant characteristic graphs by the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed image
Figure BDA0002751566760000055
Backward mapping Δ p to atlas xB。
S6, reconstructing the image
Figure BDA0002751566760000056
Obtaining reconstructed images by warp operation
Figure BDA0002751566760000057
Differentiating the reconstructed images by a discriminator D
Figure BDA0002751566760000061
And atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
Figure BDA0002751566760000062
S7, reconstructing the image
Figure BDA0002751566760000063
Similarity to atlas x, so that Generator GFDiscriminator D and generator GBMutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Figure BDA0002751566760000064
Referring to fig. 2, ICGAN adds antagonism to the basis of the conventional GAN model, allowing generators and discriminators within the GAN model to compete, yielding the best results; the image transformation model of this embodiment uses CycleGAN as a basic frame, and adds cycle consistency on the basis of ICGAN, and proposes rcgan (reversible coresponsiveness gan), that is, an image transformation model, and simultaneously learns the mapping from an atlas x to an unlabelled image y and the backward mapping from an unlabelled image y to an atlas x, where the backward mapping can be used to constrain forward mapping, and the obtained final forward mapping is applied to the labels of the atlas to obtain the labels of the unlabelled images.
As shown in fig. 3 to 5, the generator GFAnd generator GBAlso includes double attentionThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unlabeled image y/reconstructed image are processed in a streaming way through the encoding sub-modules
Figure BDA0002751566760000065
And an image set x, inputting the extracted relevant feature map into a double-attention module, respectively learning the spatial information and the channel information of the relevant feature map through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
The coding submodule is provided with 5 coding submodules which are respectively a first coding submodule, a second coding submodule, a third coding submodule, a fourth coding submodule and a fifth coding submodule and forms 1 processing stream, and the picture set x and the unmarked image y/reconstructed image are respectively processed through 2 processing streams
Figure BDA0002751566760000066
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodule of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodule of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodule of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs a forward mapping delta p from the image set x to the unmarked image yFBackward mapping Δ p of reconstructed image y to atlas xB。
The double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
The coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Referring to fig. 6, the discriminator D adopts a patch gan discriminator, and a general discriminator directly judges whether an input image is a real image or a reconstructed image at an image level, and directly outputs a vector representative, which is or is not, but the high-frequency portion of the image has poor recovery capability, and in order to better locally judge the image, the patch gan discriminator divides the image into N × N patches and judges whether each patch is true or false; inputting a 160 × 160 × 128 three-dimensional image, generating a feature map with a size of 10 × 10 × 8 after passing through another convolution block with a convolution kernel size of 4 × 4 × 4 and a step size of 2, wherein each pixel represents a patch with a size of 16 × 16 × 16 on an original image, after passing through another convolution layer with a convolution kernel size of 4 × 4 × 4 and a step size of 1, judging whether each patch is true or false by using an activation function Sigmoid, a normalization layer of a PatchGAN discriminator uses BatchNormalization, and the other layer activation functions use LeakyReLU except the last layer activation function which uses Sigmoid.
The image transformation model further comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and a constraint generator G is constrained by the similarity lossFTo obtain similar reconstructed images
Figure BDA0002751566760000071
And an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Consistency loss constraint generator G by spatial loopBTo obtain similar reconstructed images
Figure BDA0002751566760000072
And atlas x; the discriminator D is constrained by the penalty on antagonism.
The similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
Figure BDA0002751566760000073
where t represents a voxel point in the image, fy(t) and
Figure BDA0002751566760000074
respectively representing the calculation of the unmarked image y and the reconstruction image
Figure BDA0002751566760000081
Local mean intensity function:
Figure BDA0002751566760000082
tidenotes a volume around t of l3Coordinates within the range, where l is preferably 3.
The smoothing loss that the different generators have respectively is defined as:
Figure BDA0002751566760000083
and
Figure BDA0002751566760000084
wherein t ∈ Ω denotes Δ PFOr Δ PBAll position spaces in, LsmoothApproximation using spatial gradients between neighboring pixels along the x, y, z directions, respectively
Figure BDA0002751566760000085
And
Figure BDA0002751566760000086
finally, the smoothness penalty for the image variation model is:
Lsmooth(ΔpF,ΔpB)＝Lsmooth(ΔpF)+Lsmooth(ΔpB)
in addition, the image change model can reconstruct the image, and the reconstruction effect is similar to the original atlas x or the unlabeled image y, so the embodiment adopts the L1 loss to strengthen the real atlas x and the reconstructed atlas x
Figure BDA0002751566760000087
The consistency between them is defined as:
Figure BDA0002751566760000088
the resistance loss is defined as:
Figure BDA0002751566760000089
wherein D (y) and D
Figure BDA00027515667600000810
Judging the unmarked image y and the reconstructed image for the discriminator D respectively
Figure BDA00027515667600000811
And (4) judging a result.
The cycleGAN is mapped through two reversible forward and backward learning processes, and a generator GBLearning x → y mappings and generating reconstructed images
Figure BDA00027515667600000812
The discriminator D reconstructs the image by training
Figure BDA00027515667600000813
The same distribution as the unlabelled imagery y, but because the supervision in the learning process is set-level, learning-only forward conversion will map the input images all to the same output image, so by another generator GFLearn y → x mapping, Generator GBAnd generator GFThe learned mapping is an idea and there is a cyclic consistency F (G (x)) x, the image transformation model learns to be invertibleIs mapped once as
Figure BDA0002751566760000091
Similarly, first pass through the generator GFLearn the mapping of y → x, then pass through the generator GBLearning the x → y mapping, once the invertible mapping learned by the image transformation model is
Figure BDA0002751566760000092
Wherein,
Figure BDA0002751566760000093
and
Figure BDA0002751566760000094
namely two sets of reversible mapping, the CycleGAN introduces the following cycle consistency loss formula of the image:
Figure RE-GDA0002882081280000095
in summary, the loss module of the image transformation model can be expressed as:
Figure RE-GDA0002882081280000096
wherein λ is1＝1，λ2＝3，λ3＝10。
Evaluation of experiments
As shown in fig. 8-10, data collected from The Child and Adolescent neurodevelopmental program (CANDI) at The medical institute of massachusetts university disclose a series of brain structure images as images of experimental examples and MRBrainS18 data published by The MICCAI2018 race.
The evaluation index of the experimental evaluation adopts a Dice similarity coefficient to evaluate the segmentation accuracy of the model, and the accuracy is used for measuring the similarity between the manual labeling and the prediction result:
Figure BDA0002751566760000097
wherein y issRepresenting a manual annotation of the test data,
Figure BDA0002751566760000098
the experimental evaluation takes the average Dice coefficient and the standard deviation of Dice as the evaluation standard, reflects the discrete degree of the prediction result of the measured data, and is defined as:
Figure BDA0002751566760000101
where n denotes the number of test data, diceiThe Dice value representing the ith test datum,
Figure BDA0002751566760000102
the average Dice of all test data is shown, and the smaller the standard deviation is, the more stable the performance of the model is.
Verifying the effectiveness of the countermeasure thought of the image transformation model, comparing the segmentation results of the SiamENet and the ICGAN, wherein the SiamENet is a segmentation model without antagonism and cyclic consistency, the main structure of the segmentation model is the same as that of the image transformation model, the ICGAN is based on the traditional GAN model and antagonism, and the GAN model generator and the discriminator are subjected to countermeasure, and the results are shown in table 1:
Figure BDA0002751566760000103
table 1 comparison of the results of the segmentation of the SiamENet and IGGAN network models
The average partition Dice of ICGAN on CANDI test set was 78.1%, which is 1.7% higher than SiamENet, while the variance was also increased from 5.2 to 3.1. It is noted that the Dice index of the worst case in the test set is improved from 70.4% to 72.4%, and the best case is also improved to some extent. On the MRBrains18 data set, the average Dice is improved from 76.8% to 79%, the result shows the positive effect of the countermeasures in learning the correlation mapping, and it can be seen that after the countermeasures are added, the network can learn the segmentation result more accurately, and the countermeasures can be verified to restrict the learning of the correlation.
As shown in fig. 7 and 8, the validity of the bidirectional reversible correlation mapping of the image transformation model is verified, the image transformation model is RCGAN, and the result of comparing RCGAN with ICGAN can directly indicate whether the learning backward mapping is valid, and the results are shown in tables 2 and 3:
Figure BDA0002751566760000104
Figure BDA0002751566760000111
TABLE 2 comparison of results for RCGAN and ICGAN
Figure BDA0002751566760000112
TABLE 3 average Dice coefficient tables for ICGAN and RCGAN
As shown in tables 2 and 3, in the CANDI data set, compared with the segmentation result of ICGAN, the average Dice of RCGAN on the test set is 1.1% higher than that of ICGAN, and the variance is also increased from 3.1 to 2.8; in addition, compared with the SimENet, the average Dice of RCGAN is improved by 79.2% from 76.4%, is improved by 2.8 percentage points, and the variance is also improved by 2.4; on the MRBrains dataset, the average Dice of RCGAN was 1.2% higher than ICGAN and 3.4% higher than SimENet; table 3 details the Dice comparison of SiamENet, RCGAN, and RCGAN segmenting each type of brain anatomical structure, and it can be seen from the table that RCGAN can segment most of brain anatomical structures more accurately, due to the mutual constraints of the two-way mapping, the segmentation result is more accurate.
Referring to FIG. 9, RCGAN training is shownThe first group of pictures represents the forward mapping, and the four columns respectively show the atlas x, the unlabeled image y, and the forward mapping Δ PBAnd reconstructing the image
Figure BDA0002751566760000113
The second group of pictures represent backward mapping and respectively show a picture set x, an unmarked image y and backward mapping delta PFAnd reconstructing the image
Figure BDA0002751566760000121
The training goal of the forward mapping process is to map the image set x to the unlabeled image y, and the goal of the backward training is to map the unlabeled image y to the image set x.
Referring to fig. 10, the results of segmenting the brain anatomical structures by using SiamENet, ICGAN and RCGAN are visualized respectively, and compared with SiamENet and ICGAN, the results of RCGAN of the image transformation model are smoother and the segmentation results are more accurate.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (8)

1. A one-shot brain image segmentation method based on bidirectional correlation is characterized by comprising the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into an atlas x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, generators GFGenerator GBAll match a discriminator D, generator GFAnd generator GBThe structure is the same and comprises a twin coder and a decoder;
s3, drawing set x andunmarked image y input generator GFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF；
S4, obtaining a reconstructed image by the aid of warp operation of the atlas x
Figure FDA0002751566750000011
Differentiating the reconstructed images by a discriminator D
Figure FDA0002751566750000012
The unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
Figure FDA0002751566750000013
S5, reconstructing the image
Figure FDA0002751566750000014
And atlas x input generator GBBy means of a generator GBExtracting relevant characteristic graphs from the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed image
Figure FDA0002751566750000015
Backward mapping Δ p to atlas xB；
S6, reconstructing the image
Figure FDA0002751566750000016
Obtaining reconstructed images by warp operation
Figure FDA0002751566750000017
Differentiating the reconstructed images by a discriminator D
Figure FDA0002751566750000018
And atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
Figure FDA0002751566750000019
S7, reconstructing the image
Figure FDA00027515667500000110
Similarity to atlas x, so that Generator GFDiscriminator D and generator GBMutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Figure FDA00027515667500000111
2. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the generator GFAnd generator GBThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unmarked image y/reconstructed image are processed in a shunting manner through the encoding sub-modules
Figure FDA00027515667500000112
And an image set x, inputting the extracted relevant characteristic diagram into a double-attention module, respectively learning the spatial information and the channel information of the relevant characteristic diagram through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
3. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 2, wherein: the coding submodule comprises 5 first, second, third and fourth coding submodulesThe block and the fifth coding sub-module are combined into 1 processing stream, and the atlas x and the unmarked image y/reconstructed image are respectively processed by 2 processing streams
Figure FDA0002751566750000021
And the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule comprises 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs forward mapping delta p from the image set x to the unmarked image yFReconstruction of images
Figure FDA0002751566750000024
Backward mapping Δ p to atlas xB。
4. The method for one-shot brain image segmentation based on bi-directional correlation as claimed in claim 3, wherein: the double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
5. The method for one-shot brain image segmentation based on bi-directional correlation as claimed in claim 3, wherein: the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
6. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the discriminator D adopts a PatchGAN discriminator.
7. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and a generator G is constrained through the similarity lossFTo obtain similar reconstructed images
Figure FDA0002751566750000022
And an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Constraint generator G through spatial cyclic consistency lossBTo obtain similar reconstructed images
Figure FDA0002751566750000023
And atlas x; the discriminator D is constrained by the penalty on antagonism.
8. The one-shot brain image segmentation method based on bidirectional correlation as claimed in claim 7, wherein: the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
Figure FDA0002751566750000031
where t represents a voxel point in the image, fy(t) and
Figure FDA0002751566750000032
respectively representing the calculation of the unmarked image y and the reconstruction image
Figure FDA0002751566750000033
Local mean intensity function:
Figure FDA0002751566750000034
tidenotes t surrounding volume as l3Coordinates within the range.
CN202011186634.2A 2020-10-29 2020-10-29 One-shot brain image segmentation method based on bidirectional correlation Active CN112348786B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011186634.2A CN112348786B (en) 2020-10-29 2020-10-29 One-shot brain image segmentation method based on bidirectional correlation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011186634.2A CN112348786B (en) 2020-10-29 2020-10-29 One-shot brain image segmentation method based on bidirectional correlation

Publications (2)

Publication Number Publication Date
CN112348786A true CN112348786A (en) 2021-02-09
CN112348786B CN112348786B (en) 2022-09-13

Family

ID=74356592

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011186634.2A Active CN112348786B (en) 2020-10-29 2020-10-29 One-shot brain image segmentation method based on bidirectional correlation

Country Status (1)

Country Link
CN (1) CN112348786B (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108062753A (en) * 2017-12-29 2018-05-22 重庆理工大学 The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study
CN109559332A (en) * 2018-10-31 2019-04-02 浙江工业大学 A kind of sight tracing of the two-way LSTM and Itracker of combination
CN110084863A (en) * 2019-04-25 2019-08-02 中山大学 A kind of multiple domain image conversion method and system based on generation confrontation network
CN111047594A (en) * 2019-11-06 2020-04-21 安徽医科大学 Tumor MRI weak supervised learning analysis modeling method and model thereof
CN111402259A (en) * 2020-03-23 2020-07-10 杭州健培科技有限公司 Brain tumor segmentation method based on multi-level structure relation learning network

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108062753A (en) * 2017-12-29 2018-05-22 重庆理工大学 The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study
CN109559332A (en) * 2018-10-31 2019-04-02 浙江工业大学 A kind of sight tracing of the two-way LSTM and Itracker of combination
CN110084863A (en) * 2019-04-25 2019-08-02 中山大学 A kind of multiple domain image conversion method and system based on generation confrontation network
CN111047594A (en) * 2019-11-06 2020-04-21 安徽医科大学 Tumor MRI weak supervised learning analysis modeling method and model thereof
CN111402259A (en) * 2020-03-23 2020-07-10 杭州健培科技有限公司 Brain tumor segmentation method based on multi-level structure relation learning network

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
GUHA BALAKRISHNAN ET AL.: "VoxelMorph: A Learning Framework for Deformable Medical Image Registration", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》 *
SHUXIN WANG ET AL.: "Conquering Data Variations in Resolution: A Slice-Aware Multi-Branch Decoder Network", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》 *
SHUXIN WANG ET AL.: "LT-Net: Label Transfer by Learning Reversible Voxel-Wise Correspondence for One-Shot Medical Image Segmentation", 《2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》 *
何剑华等: "基于改进的CycleGAN模型非配对的图像到图像转换", 《玉林师范学院学报》 *

Also Published As

Publication number Publication date
CN112348786B (en) 2022-09-13

Similar Documents

Publication Publication Date Title
Zhang et al. Learning 3d human shape and pose from dense body parts
Song et al. Region-based quality estimation network for large-scale person re-identification
Li et al. Beyond single reference for training: Underwater image enhancement via comparative learning
CN113283444B (en) Heterogeneous image migration method based on generation countermeasure network
Qin et al. Geotransformer: Fast and robust point cloud registration with geometric transformer
Tu et al. Consistent 3d hand reconstruction in video via self-supervised learning
Liu et al. Oiflow: Occlusion-inpainting optical flow estimation by unsupervised learning
EP4246458A1 (en) System for three-dimensional geometric guided student-teacher feature matching (3dg-stfm)
Bartoccioni et al. Lara: Latents and rays for multi-camera bird’s-eye-view semantic segmentation
CN111445548A (en) Multi-view face image generation method based on non-paired images
Chen et al. Transformer-based 3d face reconstruction with end-to-end shape-preserved domain transfer
Wang et al. Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation
CN114283265A (en) Unsupervised face correcting method based on 3D rotation modeling
CN112348786B (en) One-shot brain image segmentation method based on bidirectional correlation
Xu et al. Correlation via synthesis: end-to-end nodule image generation and radiogenomic map learning based on generative adversarial network
Xu et al. Unsupervised learning of depth estimation and camera pose with multi-scale GANs
Cao et al. Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis
Teng et al. Unimodal face classification with multimodal training
CN115631155A (en) Bone disease screening method based on space-time self-attention
CN112308833B (en) One-shot brain image segmentation method based on circular consistent correlation
Ren et al. Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images
Li et al. Geometry to the rescue: 3d instance reconstruction from a cluttered scene
Lu Basketball Motion Recognition Model Analysis Based on Perspective Invariant Geometric Features in Skeleton Data Extraction.
Bouafif et al. Hybrid approach for 3d head reconstruction: Using neural networks and visual geometry
Wang et al. Neighborhood Multi-compound Transformer for point cloud registration

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant