CN112348786A - One-shot brain image segmentation method based on bidirectional correlation - Google Patents
One-shot brain image segmentation method based on bidirectional correlation Download PDFInfo
- Publication number
- CN112348786A CN112348786A CN202011186634.2A CN202011186634A CN112348786A CN 112348786 A CN112348786 A CN 112348786A CN 202011186634 A CN202011186634 A CN 202011186634A CN 112348786 A CN112348786 A CN 112348786A
- Authority
- CN
- China
- Prior art keywords
- image
- generator
- atlas
- discriminator
- reconstructed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 210000004556 brain Anatomy 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000003709 image segmentation Methods 0.000 title claims abstract description 16
- 230000002457 bidirectional effect Effects 0.000 title claims abstract description 14
- 238000013507 mapping Methods 0.000 claims abstract description 76
- 230000009466 transformation Effects 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 19
- 238000010586 diagram Methods 0.000 claims description 21
- 230000008485 antagonism Effects 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 6
- 125000004122 cyclic group Chemical group 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 description 18
- 238000012360 testing method Methods 0.000 description 7
- 238000011156 evaluation Methods 0.000 description 5
- 238000013256 Gubra-Amylin NASH model Methods 0.000 description 4
- 230000002441 reversible effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 235000009508 confectionery Nutrition 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000001123 neurodevelopmental effect Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30016—Brain
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which comprises the following steps of: constructing an image transformation model comprising a generator GFGenerator GBAnd two discriminators D for inputting the atlas x and the unlabelled image y into the generator GFShunting processing to obtain forward mapping delta pFAnd the reconstructed images are distinguished by a discriminator DObtaining a reconstructed image with the unmarked image yWill reconstruct the imageAnd atlas x input generator GBGet backward mapping Δ pBAnd the reconstructed images are distinguished by a discriminator DAnd obtaining a reconstructed image with the atlas xThrough generator GFDiscriminator D and generator GBMutually constraining to obtain a final forward mapping delta pF and obtaining a marked reconstructed image through warp operationThe method simultaneously learns the forward mapping from the image set x to the unlabelled image y and the backward mapping from the unlabelled image y to the image set x through the image transformation model, and restricts the forward mapping through the backward mapping, so that the accuracy of the forward mapping is improved.
Description
Technical Field
The invention relates to the technical field of medical image segmentation, in particular to a one-shot brain image segmentation method based on bidirectional correlation.
Background
A common method for brain anatomy segmentation is segmentation by conventional machine learning, but conventional machine learning relies on manually extracted features, which have limited feature representation and generalization capabilities, so Convolutional Neural Network (CNN) learning has been developed, because it is completely data-driven and can automatically retrieve hierarchical features using self-learned advanced features, eliminating the limitations of manual features in conventional machine learning methods, with fully labeled data, the convolutional neural network has a better effect in a fully supervised segmentation task, using a segmentation algorithm based on forward correlation, i.e. improving the segmentation network to learn the forward mapping of an atlas x to an unlabeled image y, applying the learned correlation mapping to the labeling of the atlas, to obtain the labeling of the unlabeled image y, but such a method learns only the forward mapping of the atlas x to the unlabeled image y, the forward mapping is constrained only by similarity loss and smoothness loss, and the mapping is highly difficult to control, resulting in low accuracy of mapping learning.
Disclosure of Invention
The invention aims to provide a one-shot brain image segmentation method based on bidirectional correlation, which is used for constructing an image transformation model, learning forward mapping from an atlas x to an unlabelled image y and backward mapping from the unlabelled image y to the atlas x through the image transformation model at the same time, and constraining forward mapping through the backward mapping to improve the accuracy of the forward mapping.
In order to achieve the purpose, the invention adopts the following technical scheme:
a one-shot brain image segmentation method based on bidirectional correlation comprises the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into an atlas x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, a generator GFGenerator GBAll match a discriminator D, generator GFAnd generator GBThe structure is the same and comprises a twin coder and a decoder;
s3, input generator G for image set x and unlabelled image yFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF;
S4, obtaining a reconstructed image by the aid of warp operation of the atlas xDifferentiating the reconstructed images by a discriminator DThe unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
S5, reconstructing the imageAnd atlas x input generator GBBy means of a generator GBThe obtained twin encoder extracts the relevant characteristic diagram, fuses the characteristic diagram and inputs the characteristic diagram into a decoder, and the decoder is connected with the twin encoderMatching to obtain a reconstructed imageBackward mapping Δ p to atlas xB;
S6, reconstructing the imageObtaining reconstructed images by warp operationDifferentiating the reconstructed images by a discriminator DAnd atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
S7, reconstructing the imageSimilarity to atlas x, so that Generator GFDiscriminator D and generator GBMutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Further, the generator GFAnd generator GBThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unmarked image y/reconstructed image are processed in a shunting manner through the encoding sub-modulesAnd a drawing set x, inputting the extracted related characteristic drawing into a double-attention module, and dividing the characteristic drawing into a plurality of groups through the double-attention moduleAnd (3) learning the spatial information and the channel information of the relevant characteristic diagram and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
Furthermore, the coding submodule has 5 coding submodules which are respectively a first coding submodule, a second coding submodule, a third coding submodule, a fourth coding submodule and a fifth coding submodule and forms 1 processing stream, and the atlas x and the unmarked image y/reconstructed image are respectively processed by 2 processing streamsAnd the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs an image set x to a forward mapping delta p marked by the image of the non-submoduleFReconstruction of imagesBackward mapping Δ p to atlas xB。
Further, the double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new feature map.
Further, the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Further, the arbiter D adopts a PatchGAN arbiter.
Further, the image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and the generator G is constrained through the similarity lossFTo obtain similar reconstructed imagesAnd an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Constraint generator G through spatial cyclic consistency lossBTo obtain similar reconstructed imagesAnd atlas x; the discriminator D is constrained by the penalty on antagonism.
Further, the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
where t represents a voxel point in the image, fy(t) andrespectively representing the calculation of the unmarked image y and the reconstruction imageLocal mean intensity function:tidenotes a volume around t of l3Coordinates within the range.
After adopting the technical scheme, compared with the background technology, the invention has the following advantages:
1. the invention is obtained byClassifying the brain anatomical structure image to obtain an atlas x with labels and an unlabelled image y, and constructing an image transformation model with 2 generators and 2 discriminators D, wherein the 2 generators are generators G respectivelyFGenerator GBInputting the atlas x and the unlabelled image y into a generator GFThe twin encoder and decoder performs forward mapping to obtain a forward mapping Δ pFThrough a discriminator D and a generator GFPerforming countermeasure, so that the atlas x is subjected to warp operation to obtain a reconstructed image similar to the unlabelled image yWill reconstruct the imageInput generator GBThe twin encoder and decoder performs backward mapping to obtain backward mapping delta pBBy means of a discriminator D and a generator GBConfrontation is carried out to obtain a reconstructed image similar to the atlas xFrom the reconstructed imageAnd (3) carrying out circulation to enable backward mapping to restrict forward mapping so as to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation, and obtaining a labeled reconstructed imageThrough generator GFGenerator GBRespectively confronted with a discriminator D, so that the image change model obtains forward mapping delta p with the highest accuracyFAnd with the label to reconstruct the image
2. The invention introduces loss modules, wherein the loss modules comprise similarity loss, smooth loss and nullCyclic consistency loss and antagonism loss, through different losses to generator GFGenerator GBAnd 2 discriminators D carry out constraint to improve the accuracy of the image transformation model.
3. The discriminator D selects the PatchGAN discriminator, the PatchGAN discriminator can better discriminate the local part of the image, each patch is judged true and false by dividing the image into a plurality of patches, and finally the judgment of the image level is obtained, and the accuracy and the performance are superior to those of a common discriminator.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
FIG. 2 is a schematic diagram of the overall structure of an image transformation model according to the present invention;
FIG. 3 is a schematic diagram of the operation of a twin encoder and decoder according to the present invention;
FIG. 4 is a schematic diagram of the decoder operation of the present invention;
FIG. 5 is a schematic structural diagram of a dual-note module of the present invention;
FIG. 6 is a schematic structural diagram of a discriminator D according to the invention;
FIG. 7 is a diagram illustrating the segmentation result of ICGAN forward mapping according to the present invention;
FIG. 8 is a graph showing the comparison of the segmentation results of the SiamENet and the ICGAN according to the present invention;
FIG. 9 is a diagram illustrating the segmentation results of the ICGAN forward mapping and backward mapping according to the present invention;
FIG. 10 is a graph comparing the results of the visual segmentation of the Simeneet, ICGAN and RCGAN of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Examples
As shown in fig. 1 to 7, the present invention discloses a one-shot brain image segmentation method based on bidirectional correlation, which includes the following steps:
and S1, acquiring and classifying the brain anatomical structure image to obtain a labeled image and an unlabeled image y, and dividing the labeled image into an atlas x.
S2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, a generator GFGenerator GBAll match a discriminator D, generator GFAnd generator GBAll structurally identical comprise a twin encoder and a decoder.
S3, input generator G for image set x and unlabelled image yFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF。
S4, obtaining a reconstructed image by the aid of warp operation of the atlas xDifferentiating the reconstructed images by a discriminator DThe unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
S5, reconstructing the imageAnd atlas x input generator GBBy means of a generator GBExtracting relevant characteristic graphs by the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed imageBackward mapping Δ p to atlas xB。
S6, reconstructing the imageObtaining reconstructed images by warp operationDifferentiating the reconstructed images by a discriminator DAnd atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
S7, reconstructing the imageSimilarity to atlas x, so that Generator GFDiscriminator D and generator GBMutually constraining to obtain final forward mapping delta pF, applying the forward mapping delta pF to the label of the atlas x through warp operation to obtain a labeled reconstructed image
Referring to fig. 2, ICGAN adds antagonism to the basis of the conventional GAN model, allowing generators and discriminators within the GAN model to compete, yielding the best results; the image transformation model of this embodiment uses CycleGAN as a basic frame, and adds cycle consistency on the basis of ICGAN, and proposes rcgan (reversible coresponsiveness gan), that is, an image transformation model, and simultaneously learns the mapping from an atlas x to an unlabelled image y and the backward mapping from an unlabelled image y to an atlas x, where the backward mapping can be used to constrain forward mapping, and the obtained final forward mapping is applied to the labels of the atlas to obtain the labels of the unlabelled images.
As shown in fig. 3 to 5, the generator GFAnd generator GBAlso includes double attentionThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unlabeled image y/reconstructed image are processed in a streaming way through the encoding sub-modulesAnd an image set x, inputting the extracted relevant feature map into a double-attention module, respectively learning the spatial information and the channel information of the relevant feature map through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
The coding submodule is provided with 5 coding submodules which are respectively a first coding submodule, a second coding submodule, a third coding submodule, a fourth coding submodule and a fifth coding submodule and forms 1 processing stream, and the picture set x and the unmarked image y/reconstructed image are respectively processed through 2 processing streamsAnd the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule is provided with 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is respectively in long connection with the fourth coding submodule of 2 processing streams, the third decoding submodule receives the second decoding submodule and is respectively in long connection with the third coding submodule of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is respectively in long connection with the second coding submodule of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs a forward mapping delta p from the image set x to the unmarked image yFBackward mapping Δ p of reconstructed image y to atlas xB。
The double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
The coding sub-module is composed of ResNet-34 stacked by basic residual modules.
Referring to fig. 6, the discriminator D adopts a patch gan discriminator, and a general discriminator directly judges whether an input image is a real image or a reconstructed image at an image level, and directly outputs a vector representative, which is or is not, but the high-frequency portion of the image has poor recovery capability, and in order to better locally judge the image, the patch gan discriminator divides the image into N × N patches and judges whether each patch is true or false; inputting a 160 × 160 × 128 three-dimensional image, generating a feature map with a size of 10 × 10 × 8 after passing through another convolution block with a convolution kernel size of 4 × 4 × 4 and a step size of 2, wherein each pixel represents a patch with a size of 16 × 16 × 16 on an original image, after passing through another convolution layer with a convolution kernel size of 4 × 4 × 4 and a step size of 1, judging whether each patch is true or false by using an activation function Sigmoid, a normalization layer of a PatchGAN discriminator uses BatchNormalization, and the other layer activation functions use LeakyReLU except the last layer activation function which uses Sigmoid.
The image transformation model further comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and a constraint generator G is constrained by the similarity lossFTo obtain similar reconstructed imagesAnd an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Consistency loss constraint generator G by spatial loopBTo obtain similar reconstructed imagesAnd atlas x; the discriminator D is constrained by the penalty on antagonism.
The similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
where t represents a voxel point in the image, fy(t) andrespectively representing the calculation of the unmarked image y and the reconstruction imageLocal mean intensity function:tidenotes a volume around t of l3Coordinates within the range, where l is preferably 3.
The smoothing loss that the different generators have respectively is defined as:
wherein t ∈ Ω denotes Δ PFOr Δ PBAll position spaces in, LsmoothApproximation using spatial gradients between neighboring pixels along the x, y, z directions, respectively
finally, the smoothness penalty for the image variation model is:
Lsmooth(ΔpF,ΔpB)=Lsmooth(ΔpF)+Lsmooth(ΔpB)
in addition, the image change model can reconstruct the image, and the reconstruction effect is similar to the original atlas x or the unlabeled image y, so the embodiment adopts the L1 loss to strengthen the real atlas x and the reconstructed atlas xThe consistency between them is defined as:
the resistance loss is defined as:
wherein D (y) and DJudging the unmarked image y and the reconstructed image for the discriminator D respectivelyAnd (4) judging a result.
The cycleGAN is mapped through two reversible forward and backward learning processes, and a generator GBLearning x → y mappings and generating reconstructed imagesThe discriminator D reconstructs the image by trainingThe same distribution as the unlabelled imagery y, but because the supervision in the learning process is set-level, learning-only forward conversion will map the input images all to the same output image, so by another generator GFLearn y → x mapping, Generator GBAnd generator GFThe learned mapping is an idea and there is a cyclic consistency F (G (x)) x, the image transformation model learns to be invertibleIs mapped once asSimilarly, first pass through the generator GFLearn the mapping of y → x, then pass through the generator GBLearning the x → y mapping, once the invertible mapping learned by the image transformation model isWherein,andnamely two sets of reversible mapping, the CycleGAN introduces the following cycle consistency loss formula of the image:
in summary, the loss module of the image transformation model can be expressed as:
wherein λ is1=1,λ2=3,λ3=10。
Evaluation of experiments
As shown in fig. 8-10, data collected from The Child and Adolescent neurodevelopmental program (CANDI) at The medical institute of massachusetts university disclose a series of brain structure images as images of experimental examples and MRBrainS18 data published by The MICCAI2018 race.
The evaluation index of the experimental evaluation adopts a Dice similarity coefficient to evaluate the segmentation accuracy of the model, and the accuracy is used for measuring the similarity between the manual labeling and the prediction result:
wherein y issRepresenting a manual annotation of the test data,the experimental evaluation takes the average Dice coefficient and the standard deviation of Dice as the evaluation standard, reflects the discrete degree of the prediction result of the measured data, and is defined as:
where n denotes the number of test data, diceiThe Dice value representing the ith test datum,the average Dice of all test data is shown, and the smaller the standard deviation is, the more stable the performance of the model is.
Verifying the effectiveness of the countermeasure thought of the image transformation model, comparing the segmentation results of the SiamENet and the ICGAN, wherein the SiamENet is a segmentation model without antagonism and cyclic consistency, the main structure of the segmentation model is the same as that of the image transformation model, the ICGAN is based on the traditional GAN model and antagonism, and the GAN model generator and the discriminator are subjected to countermeasure, and the results are shown in table 1:
table 1 comparison of the results of the segmentation of the SiamENet and IGGAN network models
The average partition Dice of ICGAN on CANDI test set was 78.1%, which is 1.7% higher than SiamENet, while the variance was also increased from 5.2 to 3.1. It is noted that the Dice index of the worst case in the test set is improved from 70.4% to 72.4%, and the best case is also improved to some extent. On the MRBrains18 data set, the average Dice is improved from 76.8% to 79%, the result shows the positive effect of the countermeasures in learning the correlation mapping, and it can be seen that after the countermeasures are added, the network can learn the segmentation result more accurately, and the countermeasures can be verified to restrict the learning of the correlation.
As shown in fig. 7 and 8, the validity of the bidirectional reversible correlation mapping of the image transformation model is verified, the image transformation model is RCGAN, and the result of comparing RCGAN with ICGAN can directly indicate whether the learning backward mapping is valid, and the results are shown in tables 2 and 3:
TABLE 2 comparison of results for RCGAN and ICGAN
TABLE 3 average Dice coefficient tables for ICGAN and RCGAN
As shown in tables 2 and 3, in the CANDI data set, compared with the segmentation result of ICGAN, the average Dice of RCGAN on the test set is 1.1% higher than that of ICGAN, and the variance is also increased from 3.1 to 2.8; in addition, compared with the SimENet, the average Dice of RCGAN is improved by 79.2% from 76.4%, is improved by 2.8 percentage points, and the variance is also improved by 2.4; on the MRBrains dataset, the average Dice of RCGAN was 1.2% higher than ICGAN and 3.4% higher than SimENet; table 3 details the Dice comparison of SiamENet, RCGAN, and RCGAN segmenting each type of brain anatomical structure, and it can be seen from the table that RCGAN can segment most of brain anatomical structures more accurately, due to the mutual constraints of the two-way mapping, the segmentation result is more accurate.
Referring to FIG. 9, RCGAN training is shownThe first group of pictures represents the forward mapping, and the four columns respectively show the atlas x, the unlabeled image y, and the forward mapping Δ PBAnd reconstructing the imageThe second group of pictures represent backward mapping and respectively show a picture set x, an unmarked image y and backward mapping delta PFAnd reconstructing the imageThe training goal of the forward mapping process is to map the image set x to the unlabeled image y, and the goal of the backward training is to map the unlabeled image y to the image set x.
Referring to fig. 10, the results of segmenting the brain anatomical structures by using SiamENet, ICGAN and RCGAN are visualized respectively, and compared with SiamENet and ICGAN, the results of RCGAN of the image transformation model are smoother and the segmentation results are more accurate.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (8)
1. A one-shot brain image segmentation method based on bidirectional correlation is characterized by comprising the following steps:
s1, acquiring and classifying brain anatomical structure images to obtain labeled images and unlabeled images y, and dividing the labeled images into an atlas x;
s2, constructing an image transformation model, wherein the image transformation model comprises a generator GFGenerator GBAnd two discriminators D, generators GFGenerator GBAll match a discriminator D, generator GFAnd generator GBThe structure is the same and comprises a twin coder and a decoder;
s3, drawing set x andunmarked image y input generator GFShunting treatment by generator GFThe twin encoder extracts the relevant characteristic diagram and inputs the characteristic diagram into the decoder after fusion, and the decoder is matched with the twin encoder to obtain the forward mapping delta p from the image set x to the unmarked image yF;
S4, obtaining a reconstructed image by the aid of warp operation of the atlas xDifferentiating the reconstructed images by a discriminator DThe unlabelled image y, the discriminator D and the generator GFMake a countermeasure, so that the generator GFGenerating a reconstructed image similar to the unmarked image y
S5, reconstructing the imageAnd atlas x input generator GBBy means of a generator GBExtracting relevant characteristic graphs from the obtained twin encoder, fusing the characteristic graphs, inputting the fused characteristic graphs into a decoder, and matching the decoder with the twin encoder to obtain a reconstructed imageBackward mapping Δ p to atlas xB;
S6, reconstructing the imageObtaining reconstructed images by warp operationDifferentiating the reconstructed images by a discriminator DAnd atlas x, discriminator D and generator GBMake a countermeasure, so that the generator GBGenerating a reconstructed image similar to atlas x
2. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the generator GFAnd generator GBThe twin encoder comprises a plurality of encoding sub-modules for extracting shallow features of the image, and the image set x and the unmarked image y/reconstructed image are processed in a shunting manner through the encoding sub-modulesAnd an image set x, inputting the extracted relevant characteristic diagram into a double-attention module, respectively learning the spatial information and the channel information of the relevant characteristic diagram through the double-attention module, and transmitting the spatial information and the channel information to a decoder, wherein the decoder comprises decoding sub-modules matched with the number of the encoder sub-modules.
3. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 2, wherein: the coding submodule comprises 5 first, second, third and fourth coding submodulesThe block and the fifth coding sub-module are combined into 1 processing stream, and the atlas x and the unmarked image y/reconstructed image are respectively processed by 2 processing streamsAnd the x, 2 processing streams of the graph set are simultaneously connected with a fifth coding submodule, and the fifth coding submodule is connected with the double-attention module; the decoder submodule comprises 5 first decoding submodules, a second decoding submodule, a third decoding submodule, a fourth decoding submodule and a fifth decoding submodule, wherein the first decoding submodule is connected with the double attention module, the second decoding submodule receives the first decoding submodule and is in long connection with the fourth coding submodules of 2 processing streams, the third decoding submodule receives the second decoding submodule and is in long connection with the third coding submodules of 2 processing streams, the fourth decoding submodule receives the third decoding submodule and is in long connection with the second coding submodules of 2 processing streams, the fifth decoding submodule receives the fourth decoding submodule, and the fifth decoding submodule outputs forward mapping delta p from the image set x to the unmarked image yFReconstruction of imagesBackward mapping Δ p to atlas xB。
4. The method for one-shot brain image segmentation based on bi-directional correlation as claimed in claim 3, wherein: the double attention module comprises a space attention module and a channel module, information is captured in the space dimension and the channel dimension respectively, and the results of the space attention module and the channel attention module are added to obtain a new characteristic diagram.
5. The method for one-shot brain image segmentation based on bi-directional correlation as claimed in claim 3, wherein: the coding sub-module is composed of ResNet-34 stacked by basic residual modules.
6. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the discriminator D adopts a PatchGAN discriminator.
7. The one-shot brain image segmentation method based on the bidirectional correlation as claimed in claim 1, wherein: the image transformation model also comprises a loss module for supervising the image transformation model, wherein the loss module comprises similarity loss, smooth loss, space cycle consistency loss and antagonism loss, and a generator G is constrained through the similarity lossFTo obtain similar reconstructed imagesAnd an unlabelled image y; constraining generator G by smoothing lossFTo obtain a smoothed forward mapping Δ pFAnd backward mapping Δ pB(ii) a Constraint generator G through spatial cyclic consistency lossBTo obtain similar reconstructed imagesAnd atlas x; the discriminator D is constrained by the penalty on antagonism.
8. The one-shot brain image segmentation method based on bidirectional correlation as claimed in claim 7, wherein: the similarity loss employs a local normalized correlation loss for ensuring local consistency, and the formula is as follows:
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011186634.2A CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011186634.2A CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112348786A true CN112348786A (en) | 2021-02-09 |
CN112348786B CN112348786B (en) | 2022-09-13 |
Family
ID=74356592
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011186634.2A Active CN112348786B (en) | 2020-10-29 | 2020-10-29 | One-shot brain image segmentation method based on bidirectional correlation |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112348786B (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
CN109559332A (en) * | 2018-10-31 | 2019-04-02 | 浙江工业大学 | A kind of sight tracing of the two-way LSTM and Itracker of combination |
CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
CN111047594A (en) * | 2019-11-06 | 2020-04-21 | 安徽医科大学 | Tumor MRI weak supervised learning analysis modeling method and model thereof |
CN111402259A (en) * | 2020-03-23 | 2020-07-10 | 杭州健培科技有限公司 | Brain tumor segmentation method based on multi-level structure relation learning network |
-
2020
- 2020-10-29 CN CN202011186634.2A patent/CN112348786B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
CN109559332A (en) * | 2018-10-31 | 2019-04-02 | 浙江工业大学 | A kind of sight tracing of the two-way LSTM and Itracker of combination |
CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
CN111047594A (en) * | 2019-11-06 | 2020-04-21 | 安徽医科大学 | Tumor MRI weak supervised learning analysis modeling method and model thereof |
CN111402259A (en) * | 2020-03-23 | 2020-07-10 | 杭州健培科技有限公司 | Brain tumor segmentation method based on multi-level structure relation learning network |
Non-Patent Citations (4)
Title |
---|
GUHA BALAKRISHNAN ET AL.: "VoxelMorph: A Learning Framework for Deformable Medical Image Registration", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》 * |
SHUXIN WANG ET AL.: "Conquering Data Variations in Resolution: A Slice-Aware Multi-Branch Decoder Network", 《IEEE TRANSACTIONS ON MEDICAL IMAGING》 * |
SHUXIN WANG ET AL.: "LT-Net: Label Transfer by Learning Reversible Voxel-Wise Correspondence for One-Shot Medical Image Segmentation", 《2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》 * |
何剑华等: "基于改进的CycleGAN模型非配对的图像到图像转换", 《玉林师范学院学报》 * |
Also Published As
Publication number | Publication date |
---|---|
CN112348786B (en) | 2022-09-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Zhang et al. | Learning 3d human shape and pose from dense body parts | |
Song et al. | Region-based quality estimation network for large-scale person re-identification | |
Li et al. | Beyond single reference for training: Underwater image enhancement via comparative learning | |
CN113283444B (en) | Heterogeneous image migration method based on generation countermeasure network | |
Qin et al. | Geotransformer: Fast and robust point cloud registration with geometric transformer | |
Tu et al. | Consistent 3d hand reconstruction in video via self-supervised learning | |
Liu et al. | Oiflow: Occlusion-inpainting optical flow estimation by unsupervised learning | |
EP4246458A1 (en) | System for three-dimensional geometric guided student-teacher feature matching (3dg-stfm) | |
Bartoccioni et al. | Lara: Latents and rays for multi-camera bird’s-eye-view semantic segmentation | |
CN111445548A (en) | Multi-view face image generation method based on non-paired images | |
Chen et al. | Transformer-based 3d face reconstruction with end-to-end shape-preserved domain transfer | |
Wang et al. | Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation | |
CN114283265A (en) | Unsupervised face correcting method based on 3D rotation modeling | |
CN112348786B (en) | One-shot brain image segmentation method based on bidirectional correlation | |
Xu et al. | Correlation via synthesis: end-to-end nodule image generation and radiogenomic map learning based on generative adversarial network | |
Xu et al. | Unsupervised learning of depth estimation and camera pose with multi-scale GANs | |
Cao et al. | Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis | |
Teng et al. | Unimodal face classification with multimodal training | |
CN115631155A (en) | Bone disease screening method based on space-time self-attention | |
CN112308833B (en) | One-shot brain image segmentation method based on circular consistent correlation | |
Ren et al. | Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images | |
Li et al. | Geometry to the rescue: 3d instance reconstruction from a cluttered scene | |
Lu | Basketball Motion Recognition Model Analysis Based on Perspective Invariant Geometric Features in Skeleton Data Extraction. | |
Bouafif et al. | Hybrid approach for 3d head reconstruction: Using neural networks and visual geometry | |
Wang et al. | Neighborhood Multi-compound Transformer for point cloud registration |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |