CN110544275B - Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags - Google Patents
Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags Download PDFInfo
- Publication number
- CN110544275B CN110544275B CN201910764408.9A CN201910764408A CN110544275B CN 110544275 B CN110544275 B CN 110544275B CN 201910764408 A CN201910764408 A CN 201910764408A CN 110544275 B CN110544275 B CN 110544275B
- Authority
- CN
- China
- Prior art keywords
- code
- mode
- encoder
- decoder
- random
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/38—Registration of image sequences
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10088—Magnetic resonance imaging [MRI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a method, a system and a medium for generating registered multi-mode MRI with a focus segmentation label, wherein the method comprises the steps of obtaining a normally distributed random matrix and inputting the matrix into a decoder for generating trained structural features in a countermeasure network to decode and generate a structural feature graph; the structural characteristic graph and the randomly selected focus segmentation label graph are fused through random input to obtain a fusion result, and the fusion result is input to a trained random encoder in a generated countermeasure network to obtain a code; and generating decoders for resisting each trained mode in the network by using the coded input, and respectively generating the multi-mode MRI after registration. The generator in the generation countermeasure network is modularized into the encoder and the decoder, and through the combined training of a plurality of groups of encoders, decoders and discriminators, a random input meeting the design specification can be received to generate a group of registered multi-mode MRI images with focus segmentation labels, so that the method can be widely applied to the field of medical imaging.
Description
Technical Field
The invention relates to an image generation technology in the medical field, in particular to a method, a system and a medium for generating a multi-modality MRI with a lesion segmentation tag, which are used for acquiring a multi-modality MRI image with a lesion segmentation tag according to given random input meeting the specification.
Background
With the development of deep learning, more and more fields begin to adopt deep neural networks to perform image processing tasks. However, training of deep neural networks requires a large amount of data, but in many fields, construction of data sets is very difficult. Therefore, the image generation technology has important applications in image intelligent processing scenes in many fields, such as medical images and biological cell images. In medical image intelligent processing, there are many modalities for medical images, such as Magnetic Resonance Imaging (MRI) X-ray, CT, and so on. When different modalities are obtained from the same part of the same patient by different imaging techniques, the modalities are considered to be registered if the imaging positions and viewing angles coincide. Compared with single-mode data, the registered multi-mode image data can provide more information, can support more complex application scenes, meets the requirement of training data of a deep neural network, and is beneficial to providing more efficient and reliable intelligent diagnosis service. However, medical image collection is very difficult, especially for rare diseases, making medical image data sets both scarce and small, which makes many training tasks impossible. Naturally, registered multi-modality image data is more scarce. Therefore, by applying image generation techniques, the generation of registered multimodal images has a wide range of uses and profound significance.
The generation of a countermeasure network (GAN) is a flexible deep neural network that can be subjected to unsupervised training and also supervised training, and has been widely used in the field of computer vision. The generation of a countermeasure network generally includes a generator that can generate realistic images by accepting random input and a discriminator that distinguishes between real images and generated images by learning them and thereby directs the generator to generate more realistic images. However, how to acquire a multi-modality MRI image with a lesion segmentation tag according to a given random input meeting the specification when generating the multi-modality MRI with the lesion segmentation tag, is still a key technical problem to be solved urgently.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the invention discloses a method, a system and a medium for generating registered multi-modal MRI with focus segmentation labels, aiming at the problems in the prior art, the invention modularizes a generator in a generation countermeasure network into an encoder and a decoder, can receive a random input meeting design specifications through the combined training of a plurality of groups of encoders, decoders and discriminators so as to generate a set of registered multi-modal MRI images with focus segmentation labels, and can be widely applied to the field of medical images.
In order to solve the technical problems, the invention adopts the technical scheme that:
a method of generating registered multi-modality MRI with lesion segmentation tags, the implementation steps comprising:
1) from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RM;
2) Code random matrixF,RMTrained structural feature Decoder in input generation countermeasure networkFDecoding to generate a structural feature map FRM;
3) Structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;
4) inputting the fusion result into a trained random Encoder Encoder in the countermeasure networkRMObtaining a CodeRM;
5) Code to CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRIig。
Optionally, step 2) is preceded by training generationDecoder for structural features in countermeasure networksFThe detailed steps comprise:
A1) randomly selecting a mode, obtaining a graph n from the mode, extracting structural features to obtain a structural feature graph F, and extracting a Mask to obtain a corresponding Mask;
A2) encoder for generating structural features in countermeasure networksFCoding the structural feature graph F to obtain a coding mean matrix CodeF,meanAnd variance matrix CodeF,logvarFrom a normal distribution N (0, 1)2) To obtain random noise CodeeFrom the mean matrix CodeF,meanAnd variance matrix CodeF,logvarRandom noise CodeeThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noiseF;
A3) Decoder with maskMaskNormal distribution matrix CodeFDecoding to obtain reconstructed Maskr(ii) a Decoder using structural featuresFTo CodeFDecoding to obtain a reconstructed structural feature map Fr;
A4) Random generation of a normal distribution N (0, 1)2) Matrix Code ofF,RMDecoder using structural featuresFCode matrix pairF,RMDecoding to obtain a generated random structure characteristic diagram FRMUsing a mask DecoderMaskCode matrix pairF,RMDecoding to obtain the generated random MaskRM;
A5) Using a structural feature map DiscriminatorFRespectively to (F, Mask) and (F)RM,MaskRM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, FRMIs a random structural feature map, MaskRMIs a random mask; using a structural feature identifier FeatureDiscrimidatorFRespectively to CodeFAnd CodeF,RMPerforming identification to identify the former as false and the latter as true, wherein the CodeFTo approximate a normal distribution matrix, Code, after the addition of noiseF,RMFor random generation of a conforming normal distribution N (0, 1)2) A matrix of (a);
A6) calculating loss according to the output result of each step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter;
A7) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step A1); otherwise, exiting.
Optionally, step a2) is integrated to obtain an approximate normal distribution matrix Code after noise is addedFThe functional expression of (a) is:
CodeF=CodeF,mean+exp(0.5*CodeF,logvar)*Codee;
in the above formula, CodeF,meanEncoder for structural featuresFMean matrix, Code, obtained by encoding the structural feature map FF,logvarEncoder for structural featuresFCode obtained by coding the structural feature map FeIs normally distributed with N (0, 1)2) To obtain random noise.
Optionally, the detailed steps of step 3) include:
3.1) randomly selecting a focus segmentation label graph L, converting the focus segmentation label graph L containing a plurality of categories into a one-hot matrix to obtain a multi-dimensional label matrix with the same channel number and category number, wherein only part of pixels in each channel are effective, the rest parts are filled 0, and the non-0 pixel regions are registered with each segmentation region in the focus segmentation label graph;
3.2) dividing the multi-dimensional label matrix and the structural feature map FRMStacking is carried out in the channel dimension together, and a matrix fusing two input sources is obtained as a fusion result.
Optionally, the step 4) is preceded by an i-mode Decoder for training each mode iiI-mode Encoder EncoderiAnd lesion segmentation label solutionDecoderLAnd training the random Encoder EncoderRMThe detailed steps comprise:
B1) i-modal Decoder for training each modal iiI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLTraining random EncoderRM;
B2) Calculating loss according to the output result of each training step and the corresponding loss function, calling an optimizer to conduct derivation on the loss function to obtain the gradient of the model parameter in each component, and then performing difference calculation on each parameter and the corresponding gradient to complete updating of the network parameter;
B3) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step B1); otherwise, exiting.
Optionally, training the i-mode Decoder of each mode i in step B1)iI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLThe detailed steps comprise:
step 1, inputting an original image i of a random modality i;
step 2, using an i-mode Encoder EncoderiEncoding the original image i to obtain an encoded Codei;
Step 3, using i-mode DecoderiCode pairiDecoding to obtain a reconstructed picture ir(ii) a Decoder for focus segmentation labelLCode pairiDecoding to obtain a focus segmentation label map Li,f(ii) a Meanwhile, for any other mode j, firstly using a j-mode DecoderjCode pairiDecoding to obtain a j mode conversion graph j of the original graph itReuse the j mode Encoder EncoderjJ mode conversion chart j for original image itCoding is carried out to obtain a Codej,tReuse the i-mode DecoderiCode pairj,tDecoding to obtain a cyclic reconstruction image i of the original image i with j mode as an intermediate modec;
Step 4, respectively passing through a mode DiscriminatorxI-mode conversion diagram i for converting original image i and each mode j into mode itDiscrimination is performed to discriminate the former as true and the latter as false.
Optionally, training the random Encoder in step B1)RMThe detailed steps comprise:
step 1, randomly selecting a mode, and acquiring a graph n and a corresponding lesion segmentation label graph L from the modenObtaining a structural feature map F by using a structural feature extraction method1Obtaining a corresponding Mask by using a Mask extraction method; using lesion segmentation label map LnRemoving and extracting to obtain a structural feature map F1Obtaining a structural characteristic diagram F without focus information;
step 2, the structural feature graph F and the randomly input focus segmentation label graph L are randomly input and fused to obtain a fusion result FRM,expand;
Step 3, fusing the result FRM,expandSending into a random Encoder EncoderRMEncoding into CodeRM;
Step 4, Code is codedRMDecoder for input focus segmentation labelLDecoding a reconstructed focus segmentation label map Lr(ii) a Simultaneously for each modality i: code to CodeRMI-mode Decoder for input mode iiGet i Modal to generate graph igGenerating the i mode into a graph igExtracting structural features to obtain a structural feature map Fi,gGenerating the i mode into a graph igInput i-mode Encoder EncoderiObtaining a Codei,gCode to Codei,gDecoder for input focus segmentation labelLDecode Ly,gA Decoder of j mode for inputting other modes jjObtaining corresponding j mode to generate focus segmentation label graph jg,t;
Step 5, aiming at each mode i, i mode Discriminator of each mode iiGenerating a graph i for the original image n and i of the modegPerforming identification on the formerThe identification is true, the latter is false; respectively pairing the Code codes through a feature discriminatorRMAnd Code of each modality iiPerforming identification to obtain CodeRMCode of each mode i identified as falseiThe authentication is true.
Further, the present invention provides a system for generating registered multi-modality MRI with lesion segmentation tags, comprising:
a random matrix generation program unit for generating a random matrix from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RM;
A structural feature extraction program unit for extracting a random matrix CodeF,RMTrained structural feature Decoder in input generation countermeasure networkFDecoding to generate a structural feature map FRM;
Structural feature fusion program unit for fusing a structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;
a random coding program unit for inputting the fusion result into a trained random coder Encoder in the countermeasure networkRMObtaining a CodeRM;
A registration structure feature map generation program unit for encoding the CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRIig。
Furthermore, the present invention also provides a system for generating a registered lesion segmentation tagged multi-modality MRI, comprising a computer device programmed or configured to perform the steps of the method for generating a registered lesion segmentation tagged multi-modality MRI, or a computer program stored on a storage medium of the computer device and programmed or configured to perform the method for generating a registered lesion segmentation tagged multi-modality MRI.
Furthermore, the invention also provides a computer readable storage medium having stored thereon a computer program programmed or configured to perform the generating of the registered multi-modality MRI with lesion segmentation tags.
Compared with the prior art, the invention has the following advantages:
1. the generator in the generation countermeasure network is modularized into the encoder and the decoder, and through the combined training of a plurality of groups of encoders, decoders and discriminators, a random input meeting the design specification can be received to generate a group of registered multi-mode MRI images with focus segmentation labels, so that the method can be widely applied to the field of medical imaging.
2. The data used by the training of the invention does not need to be registered, is unsupervised learning, can realize the generation of multi-mode registration MRI, and the generated data is labeled, thus having no limit on the number of modes.
3. The invention adopts the modularized design, can conveniently carry out the mode expansion, leads the model training to be more flexible, leads the training to be carried out independently or synchronously, and leads the trained modules to be combined and reused when in use.
Drawings
FIG. 1 is a schematic diagram of the basic principle of the method according to the embodiment of the present invention.
FIG. 2 is a schematic diagram of a basic flow of a method according to an embodiment of the present invention.
FIG. 3 is a diagram of a training architecture generated from a structural feature diagram according to an embodiment of the present invention.
Fig. 4 is a diagram of an auxiliary training architecture in a modality registration image generation training according to an embodiment of the present invention.
Fig. 5 is a generation training architecture diagram in a modality registration image generation training of an embodiment of the present invention.
Fig. 6 is a main flow chart of a modality registration image generation training according to an embodiment of the present invention.
Detailed Description
The method, system and medium for generating registered lesion segmentation tagged multi-modality MRI of the present invention will be described in further detail below, using x and y modalities as an example. However, it should be noted that the method, system and medium for generating registered multi-modality MRI with lesion segmentation tags according to the present invention are not limited to two-modality registration image generation, but can be applied to three-modality and more registered multi-modality MRI generation.
As shown in fig. 1 and 2, the implementation steps of the method for generating registered multi-modality MRI with lesion segmentation tags of the present embodiment include:
1) from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RMCan be represented as N (0, 1)2)→CodeF,RM;
2) Code random matrixF,RMTrained structural feature Decoder in input generation countermeasure networkF(abbreviated as DC)F) Decoding to generate a structural feature map FRM;
3) Structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L; referring to fig. 2, the randomly selected lesion segmentation label map L is obtained by randomly performing simple image transformation operations such as translation, flipping, rotation, scaling and the like on a real lesion label map;
4) inputting the fusion result into a trained random Encoder Encoder in the countermeasure networkRM(abbreviation EC)R) Obtaining a CodeRM;
5) Code to CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRIig. Referring to FIG. 1, taking x and y two modes as examples, Code will be encodedRMInput generation resists the x modal Decoder of the good modal x of training in the networkx(abbreviated as DC)x) Respectively generating registered i-mode generation structure feature maps xg(ii) a Code to CodeRMInput generation confronts the y modal Decoder of good modal y in the networky(abbreviated as DC)y) Respectively generating registered i-mode generation structure feature map yg。
The present embodiment has the following random input specifications for the method of generating registered lesion segmentation tagged multi-modality MRI: the random input comprises a random structural feature map and a random segmentation label map. The random structural feature map is generated by decoding from a standard normal distribution matrix by a decoder. The real structural feature map is extracted from a real image by a structural feature map extraction method based on a Sobel operator. The random segmentation label map is a segmentation label of each part in a randomly selected real mode image. The present embodiment is capable of receiving random input that meets this specification, generating a set of segmentation tagged, registered multi-modality MRI data.
The method also comprises training a Decoder for generating structural features in the countermeasure network before the step 2) of the embodimentF(abbreviated as DC)F) Training the Decoder for generating structural features in the countermeasure networkF(abbreviated as DC)F) The step (a) can be independently trained, and after the training is completed, the generated random structural feature map is used for further generating a group of registered multi-modality MRI data with a lesion segmentation label. As shown in FIG. 3, the Decoder for structure feature in training generative countermeasure network in this embodimentF(abbreviated as DC)F) The detailed steps comprise:
A1) randomly selecting a mode, obtaining a graph n from the mode, performing structural feature extraction (in the embodiment, a Sobel operator is adopted, so that the Sobel operator is used for representing and extracting structural features in the graph) to obtain a structural feature graph F, and performing Mask extraction (represented by Mask in the graph) to obtain a corresponding Mask;
A2) encoder for generating structural features in countermeasure networksFCoding the structural feature graph F to obtain a coding mean matrix CodeF,meanAnd variance matrix CodeF,logvarCan be expressed as CodeF,mean,CodeF,logvar=EncoderF(F) From a normal distribution N (0, 1)2) To obtain random noise CodeeCan be represented as N (0, 1)2)→CodeeFrom the mean matrix CodeF,meanAnd variance matrix CodeF,logvarRandom noise CodeeThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noiseF(ii) a Following trainingIncrease in number of iterations, CodeFWill be closer and closer to the normal distribution.
A3) Decoder with maskMaskFor approximate normal distribution matrix CodeFDecoding to obtain reconstructed MaskrCan be expressed as Maskr=DecoderMask(CodeF) (ii) a Decoder using structural featuresFTo CodeFDecoding to obtain a reconstructed structural feature map FrCan be represented as Fr=DecoderF(CodeF);
A4) Random generation of a normal distribution N (0, 1)2) Matrix Code ofF,RMDecoder using structural featuresF(abbreviated as DC)F) Code matrix pairF,RMDecoding to obtain a generated random structure characteristic diagram FRMCan be represented as FRM=DecoderF(CodeF,RM) (ii) a Decoder with maskMask(abbreviated as DC)Mask) Code matrix pairF,RMDecoding to obtain the generated random MaskRMCan be expressed as MaskRM=DecoderMask(CodeF,RM);
A5) Using a structural feature map DiscriminatorF(abbreviation D)F) Respectively to (F, Mask) and (F)RM,MaskRM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, FRMIs a random structural feature map, MaskRMIs a random mask; using a structural feature identifier FeatureDiscrimidatorF(abbreviation FD)F) Respectively to CodeFAnd CodeF,RMPerforming identification to identify the former as false and the latter as true, wherein the CodeFTo approximate a normal distribution matrix, Code, after the addition of noiseF,RMFor random generation of a conforming normal distribution N (0, 1)2) A matrix of (a);
A6) calculating loss according to the output result of each step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter;
A7) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step A1); otherwise, exiting.
In the training process of the foregoing steps a1) -a 7), it is desirable that the structural feature map decoded by the random normal distribution matrix in this embodiment is more realistic, so that the structural feature map DiscriminatorFPerforming countermeasure learning on the structural features and masks extracted from the original image and the structural features and masks decoded by the random normal distribution matrix, and adding a generation countermeasure network GAN of coding features: through structural feature discriminator FeatureDiscrimidatorFCode for encoding result of real structure characteristic diagramFAnd a matrix Code following a standard normal distributionF,RMPerforming counterlearning to make Encoder EncoderFThe structural feature map F can be coded as a result of following a standard normal distribution.
In this embodiment, the corresponding loss functions in step a6) include the following four loss functions (1) to (4):
(1) make the structural feature map code obey the normally distributed antagonism loss:
in the above formula, lossFeatureDiscriminator,1Representing the loss of classification training of the feature discriminator, FeatureDiscrimidatorF(CodeF,RM) Expressed in CodeF,RMAs the result of discrimination output of the input feature discriminator, FeatureDiscrimatorF(CodeF) Expressed in CodeFAs the result of discrimination output of the input feature discriminator, omegaiFor the weight lost for each term (the same below), 0 and 1 indicate true or false, lossGenerator,1Representing the penalty of antagonism provided by the feature discriminator to the structural feature map encoder.
(2) The intermediate coding result of the structural feature graph obeys the normally distributed self-supervision loss:
in the above formula, lossSupervision,1Representing constrained CodeFSupervised loss, mean (Code) obeying normal distributionF,mean) Represents CodeF,meanMean of (mean of) (Code)F,logvar) Represents CodeF,logvarIs measured.
(3) So that the structural characteristic graph decoded by the random normal distribution matrix is more vivid in adversity loss:
in the above formula, lossDiscriminator,1Representing the loss of classification training of the structural feature map discriminator,
DiscriminatorF(F, Mask) represents the true and false identification output of the structural characteristic diagram identifier with F, Mask as the input,
DiscriminatorF(FRM,MaskRM) Is represented by FRM,MaskRMFor true and false discrimination output of the input structural feature map discriminator, lossGenerator,2Representing the impedance penalty that the structural feature map evaluator provided to the structural feature map and the mask generation component.
(4) And (3) performing pairwise self-supervision consistency loss on the fused structural feature graph and mask with the original structural feature graph and mask:
in the above formula, lossSupervision,2Representing the reconstructed auto-supervised loss of the texture feature map and the mask and the loss of consistency of the texture feature map and the mask, F representing the input texture feature map, FrRepresenting the reconstructed structural features, Mask representing the input Mask, MaskrMask representing the reconstruction, FRMRepresentation from random generationThe structural characteristic graph, Mask, obtained by decoding the normal distribution matrixRMRepresenting a mask decoded from a randomly generated normal distribution matrix.
Thus, the loss of the generator component is:
lossGenerator,F=lossGenerator,1+lossSupervision,1+lossGenerator,2+lossSupervision,2
in the above formula, lossGenerator,FTo generate the total loss of components, lossGenerator,1Loss of antagonism for the feature discriminator to provide to the texture feature map encoderSupervision,1Representing constrained CodeFLoss of supervision following a normal distributionGenerator,2Loss of contrast, which represents the structural feature pattern provided by the structural feature pattern discriminator to the structural feature pattern and mask generation componentSupervision,2Representing the reconstructed self-monitoring loss of the structural feature map and the mask and the consistency loss of the structural feature map and the mask.
The loss of the discriminator component is:
lossDiscriminator,F=lossFeatureDiscriminator,1+lossDiscriminator,1
in the above formula, lossDiscriminator,FFor total loss of discriminator components, lossFeatureDiscriminator,1Loss of class representing feature discriminatorsDiscriminator,1The classification training loss _ discriminator component and the generator component, which represent the structural feature map discriminator, are each updated separately. In this embodiment, the corresponding loss function in step a6) may also adopt other loss functions as needed.
It is well known that medical images generated directly from random noise by generating a counter-training are often difficult to train and to generate true structural information. In this embodiment, the image providing the basic contour structure information in the medical image is referred to as its structure feature map, for example, the retinal blood vessel distribution map can be regarded as the structure feature map of the retinal image. The structural feature map can provide necessary guiding information for the synthesis of medical images, for example, some studies obtain basic structural information from brain segmentation tag maps when synthesizing brain MRI images. However, the conventional structural feature maps such as the retinal vessel distribution map and the brain segmentation label map require additional data and training to extract the structural feature map from the original image. Therefore, the method for directly extracting the structural feature map is designed in the embodiment, and the method has the advantages of being fast in operation, free of training, free of additional data and the like. In the field of traditional digital image processing, people adopt Roberts operators, Prewitt operators, Sobel operators and the like which are all excellent edge detection operators. The Prewitt operator and Sobel operator are both templates of 3x3, and the resulting partial derivatives approximation is more accurate than that obtained with the templates of Roberts operator 2x 2. Compared with the Prewitt operator, the Sobel operator weights the influence of the position of the pixel, so that noise can be better suppressed, the edge blurring degree is reduced, and the effect is better. The Sobel operator is commonly used for processing medical images, and two gray level images are output after processing.
The detailed steps of extracting the structural features in step a1) in this embodiment are as follows: (1) inputting a real image n, wherein beta is a set pixel threshold; (2) respectively carrying out bitwise maximum value reduce _ max () on two outputs of the sobel operator to obtain f2 and bitwise minimum value reduce _ min () to obtain f 1; (3) difference values (mean (f1) -f1, f2-mean (f2) are obtained by respectively solving f1 and f2 and the mean pixel, and mean is a mean function), so that a low pixel edge feature map and a high pixel edge feature map are obtained; (4) and (3) performing binarization processing on the low pixel edge feature map and the high pixel edge feature map by using a pixel threshold value beta, and finally adding the two binary maps and performing binarization with a threshold value of 0 to obtain a clear structural feature map f. In this embodiment, the method for extracting the structural feature in step a1) is expressed as F ═ get _ F (n) using a function.
The detailed steps of mask extraction in step a1) in this embodiment are as follows: (1) firstly, carrying out binarization processing on an image n with a pixel threshold value of 0 to obtain a standard mask; (2) according to the size of the obtained input image, expanding p pixels of the original image size of the mask by adopting the nearest neighbor difference value, and grinding and amplifying the whole image; (3) and cutting the length and width of p pixels on the outermost layer to keep the size of the final output mask consistent with that of the original input image. In this embodiment, the method for performing Mask extraction in step a1) is expressed as Mask _ Mask (n) using a function.
In this embodiment, step a2) is performed to obtain an approximate normal distribution matrix Code after adding noiseFThe functional expression of (a) is:
CodeF=CodeF,mean+exp(0.5*CodeF,logvar)*Codee;
in the above formula, CodeF,meanEncoder for structural featuresFMean matrix, Code, obtained by encoding the structural feature map FF,logvarEncoder for structural featuresFCode obtained by coding the structural feature map FeIs normally distributed with N (0, 1)2) To obtain random noise.
In this embodiment, the detailed steps of step 3) include:
3.1) randomly selecting a focus segmentation label graph L, converting the focus segmentation label graph L containing 5 categories into a one-hot matrix to obtain a multi-dimensional label matrix with the same channel number and category number, wherein only part of pixels in each channel are effective, the rest part of the channels are filled with 0, and the non-0 pixel regions are registered with each segmentation region in the focus segmentation label graph;
3.2) dividing the multi-dimensional label matrix and the structural feature map FRMStacking is carried out in the channel dimension together, and a matrix fusing two input sources is obtained as a fusion result.
In this embodiment, before step 4), an i-mode Decoder for training each mode i is further includediI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLAnd training the random Encoder EncoderRMThe detailed steps comprise:
B1) i-modal Decoder for training each modal iiI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLTraining random EncoderRM;
B2) Calculating loss according to the output result of each training step and the corresponding loss function, calling an optimizer to conduct derivation on the loss function to obtain the gradient of the model parameter in each component, and then performing difference calculation on each parameter and the corresponding gradient to complete updating of the network parameter;
B3) judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step B1); otherwise, exiting.
Referring to fig. 4, i-mode Decoder for training each mode i in step B1)iI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLFor additional training, i-mode Decoder for training each mode i in step B1)iI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLThe detailed steps comprise:
step 1, inputting an original image i of a random modality i;
step 2, using an i-mode Encoder EncoderiEncoding the original image i to obtain an encoded CodeiIn this embodiment, Codex=Encoderx(x),Codey=Encodery(y);
Step 3, using i-mode DecoderiCode pairiDecoding to obtain a reconstructed picture irIn this embodiment, xr=Decoderx(Codex),yr=Decodery(Codey) (ii) a Decoder for focus segmentation labelLCode pairiDecoding to obtain a focus segmentation label map Li,fIn this embodiment, Lx,f=DecoderL(Codex),Ly,f= DecoderL(Codey) (ii) a Meanwhile, for any other mode j, firstly using a j-mode DecoderjCode pairiDecoding to obtain a j mode conversion graph j of the original graph itIn this embodiment, yt=Decodery(Codex),xt=Decoderx(Codey) (ii) a Encoder using j-mode EncoderjJ mode conversion chart j for original image itCoding is carried out to obtain a Codej,tIn this embodiment, Codex,t=Encoderx(xt),Codey,t=Encodery(yt) Reuse the i-mode DecoderiCode pairj,tDecoding to obtain j-mode cyclic conversion graph i of original graph icIn this embodiment, xc=Decoderx(Codey,t),yc=Decodery(Codex,t);
Step 4, respectively passing through a mode DiscriminatorxI-mode conversion diagram i for converting original image i and each mode j into mode itDiscrimination is performed to discriminate the former as true and the latter as false.
As shown in fig. 4, in the present embodiment, taking two modalities as an example, in a specific training process for generating images of the registered X, Y modalities from random input meeting the specification, the x-modality Decoder of the training modality xxX-mode Encoder, EncoderxAnd a lesion segmentation tag DecoderLThe detailed steps comprise:
s1, inputting random X-mode image X;
s2 Encoder using x modexOriginal image x is coded to obtain CodexDecoder for reusexTo CodexDecoding to obtain a reconstructed picture xr;
S3 Decoder for focus segmentation labelLTo CodexDecoding to obtain a segmentation label graph Lx,f;
S4 Decoder using y modeyTo CodexDecoding to obtain a conversion graph yt;
S5 Encoder using y modeyFor conversion chart ytCode is obtained by encodingy,tReuse of x-mode DecoderxTo Codey,tDecoding to obtain cyclic conversion chart xc;
S6 mode DiscriminatorxFor x and x respectivelytPerforming identification to identify the former asTrue, the latter is identified as false.
The training process of the mode Y is the same as that of the mode Y, and the synchronous auxiliary training is mainly used for training the Encoderx、Encodery、 Decoderx、DecoderyAnd DecoderLModule for training the random Encoder Encoder in step B1)RMThe generation training of (2) is easier to learn.
Referring to fig. 5 and 6, step B1) train the random EncoderRMThe detailed steps comprise:
step 1, randomly selecting a mode, and acquiring a graph n and a corresponding lesion segmentation label graph L from the modenObtaining a structural feature map F by using a structural feature extraction method1Obtaining a corresponding Mask by using a Mask extraction method; using lesion segmentation label map LnRemoving and extracting to obtain a structural feature map F1Obtaining a structural characteristic diagram F without focus information;
step 2, the structural feature graph F and the randomly input focus segmentation label graph L are randomly input and fused to obtain a fusion result FRM,expand;
Step 3, fusing the result FRM,expandSending into a random Encoder EncoderRM(abbreviation EC)RM) Encoding into CodeRM;
Step 4, Code is codedRMDecoder for input focus segmentation labelL(abbreviated as DC)L) Decoding a reconstructed focus segmentation label map Lr(ii) a Simultaneously for each modality i: code to CodeRMI-mode Decoder for input mode ii(abbreviated as DC)i) Get i Modal to generate graph igGenerating the i mode into a graph igExtracting structural features to obtain a structural feature map Fi,gGenerating the i mode into a graph igInput i-mode Encoder Encoderi(abbreviation EC)i) Obtaining a Codei,gCode to Codei,gDecoder for input focus segmentation labelL(abbreviated as DC)L) Decode Ly,gA Decoder of j mode for inputting other modes jj(abbreviated as DC)j) Obtaining corresponding j mode to generate focus segmentation label graph jg,t;
Step 5, aiming at each mode i, i mode Discriminator of each mode ii(abbreviation D)i) Generating a graph i for the original image n and i of the modegCarrying out identification, wherein the former is identified as true, and the latter is identified as false; respectively encoding Code by Feature Discriminator (FD)RMAnd Code of each modality iiPerforming identification to obtain CodeRMCode of each mode i identified as falseiThe authentication is true.
As shown in fig. 5 and fig. 6, in this embodiment, a specific generation training process of X, Y modality images and segmentation labels registered by inputting the random structure feature map and the random segmentation labels L is as follows:
s1, randomly selecting one mode at a time, and acquiring a graph n and a corresponding segmentation label graph L from the modenObtaining a structural feature map F by using a structural feature extraction method1In the present example, it is represented as F1Obtaining a corresponding Mask by using a Mask extraction method, which is denoted as Mask _ Mask (n) in the embodiment;
s2, utilizing the segmentation label graph LnRemoving and extracting to obtain a structural feature map F1The lesion information of (a) is obtained as a structural feature map F without lesion information, which is expressed as F (remove _ L) in the present embodimentn,F1) Wherein removeL()A simple function for binarizing the segmentation label into a mask and then performing product calculation with the structural feature map so as to eliminate the focus part pixels;
s3, fusing the structural feature graph F with the randomly input segmentation label graph L to obtain FRM,expandIn the present example, it is represented as FRM,expandConcat (one ot (L), F), where concat () is a channel splicing function and one ot () is the aforementioned one-hot vector extension method;
s4, adding FRM,expandSending into a random Encoder EncoderRMCoded as CodeRMIn this example, Code is shownRM=EncoderRM(FRM,expand);
S5, CodeRMRespectively to X-mode decodersxDecoder of Y modeyAnd a Decoder for segmenting the label graphLGenerating a diagram X by decoding X modes respectivelygAnd a generation diagram Y of the Y modegReconstructing the segmentation label map LrIn this embodiment, x is showng=Decoderx(CodeRM),yg=Decodery(CodeRM),Lr=DecoderL(CodeRM);
S6, extracting structural features and generating a graph X from the X modegAnd generation diagram Y of the Y modegExtract characteristic graph Fx,g、 Fy,gThis embodiment can be expressed as Fx,g=get_F(xg),Fy,g=get_F(yg) Get _ F () is a structural feature map extraction method based on the sobel operator;
s7 Encoder using X modexGeneration diagram X for X modalitygCoding to obtain Codex,gIn this example, Code is shownx,g=Encoderx(xg),Codey,g=Encodery(yg);
S8、Codex,gBy a DecoderLDecoding to obtain Lx,gIn this example, it is represented as Lx,g=DecoderL(Codex,g),Ly,g=DecoderL(Codey,g);
S9、Codex,gBy a DecoderyDecoding to obtain yg,tIn the present embodiment, is represented by yg,t=Decodery(Codex,g),xg,t=Decoderx(Codey,g);
S10, similarly, the same operation is carried out on the Y mode, and an Encoder Encoder is usedyGenerate the graph ygCoding to obtain Codey,g;
S11, sending to DecoderLDecoding to obtain Ly,g;
S12、Codey,gSent to decodingDecoderxDecoding to obtain xg,t。
S13 mode X DiscriminatorxFor x and x respectivelygDiscrimination is performed to discriminate the former as true and the latter as false. Modal Y DiscriminatoryFor y and y respectivelygDiscrimination is performed to discriminate the former as true and the latter as false. Feature discriminator for CodeRM、CodexAnd CodeyPerforming identification to obtain CodeRMThe identification is false, the latter two are identified as true. In the above image generation training process, the structural feature map F1Is extracted from the random image n, the extracted structural features may contain focus structural information, which may interfere with the focus information in the random label L and affect the image generated after fusion, so F1The lesion information needs to be eliminated before being fused with the random label L to obtain a structural feature map F without lesion information, so that the lesion information of the generated image is only from the label L.
In this embodiment, the loss functions of the multimodal map generation training and the auxiliary training process include the following loss functions (5) to (15):
(5): the antagonism loss that makes the X, Y modal graph generated by the random structural feature graph more realistic:
in the above formula, lossDiscriminator,2Representing a loss of classification training of the modality Discriminator on the x-modality, Discriminatorx(x) Representing the output of a modality discriminator with x as input, lossGenerator,3Representing the loss of antagonism, loss, of the x-modality provided to the generating component by the modality discriminatorDiscriminator,3Representing a loss of classification training of the modality Discriminator on the y-modality, Discriminatory(y) represents the output of the modality discriminator with y as input, lossGenerator,4Representing the antagonism loss of the y-modality provided to the generating component by the modality discriminator.
(6): the encoding result of the random structure characteristic diagram is more approximate to the antagonism loss of the encoding result of the real mode diagram (so as to reduce the decoding difficulty of the decoder and ensure that the decoder can successfully decode the mode diagram).
In the above formula, lossFeatureDiscriminator,2Loss of class training representing the feature discriminatorGenerator,5Representing the penalty provided by the feature discriminator to the texture feature map encoder to direct the encoder to encode the input fused map into an encoding result that approximates the encoding result of the real mode map, FeatureDiscriminitor (Code)RM) Expressed in CodeRMThe true and false authentication output of the input feature authenticator.
(7) Reconstruction of the input structural feature map self-supervision loss:
in the above formula, losssupervision,3Represents the reconstruction of the structural feature map and the loss of the self-supervision, and removeL (L, F) represents the structural feature of the disease-free information obtained by removeL () of L and FFeature output, removeL (L, F)x,g) Denotes that L and Fx,gStructural feature map output of lesion-free information by remove (), where removeL()A simple function for binarizing the segmentation label into a mask and then performing product calculation with the structural feature map so as to eliminate the focus part pixels;
(8) and (3) reconstructing the input focus segmentation label graph after being fused with the input structural feature graph, wherein the reconstruction loss is as follows:
in the above formula, losssupervision,4Representing reconstructed unsupervised loss of a lesion segmentation label map, L being an input label, Lx,gFor generation of tags for x modality, LrIs a directly reconstructed tag.
(9) X, Y supervised loss of modal graph segmentation training:
in the above formula, losssupervision,5Supervised loss, L, training for X, Y mode graph segmentationxTrue tags for the x modality, Lx,fTags are generated for the x modality.
(10) Converting the generated X-mode and Y-mode graphs to obtain a conversion graph and generating the self-supervision loss of the graph:
in the above formula, losssupervision,6Conversion graph obtained by converting generated X-mode and Y-mode graphs and self-supervision loss, X, of generated graphsgFor generating an MRI of modality x, xg,tIn order to convert the generated y-mode MRI into the x-mode MRI, the registration constraint of the generated multi-mode MRI is realized through the loss, and when the generated multi-mode MRI is expanded to more modes, the generated graph of each mode is similar to different modesThe mean square error loss is calculated from the conversion map of the mode into which the state generation map is converted.
(11) Supervision loss limiting the pixel generation range to the range of the organ body mask:
in the above formula, losssupervision,7And the Mask is a binary Mask map, the organ range of the organ described by the Mask map is 0 value, and the range outside the organ described by the Mask map is 1 value.
(12) The self-supervision loss of the original image and a reconstructed image obtained by reconstructing the X-mode image and the Y-mode image is as follows:
in the above formula, losssupervision,8The self-supervision loss of the original image and the reconstructed image obtained by reconstructing the X-mode image and the Y-mode image is shown.
(13) And (3) a loop conversion graph obtained by converting the X-mode graph and the Y-mode graph and an original graph have supervision loss:
in the above formula, losssupervisin,cIndicating the supervised loss, X, of the original graph and the cyclic conversion graph obtained by converting the X-mode graph and the Y-mode graphcAnd converting the MRI in the x mode into the MRI in the y mode, and then converting the MRI in the y mode into the circular reconstruction image of the MRI in the x mode, wherein when the X mode is expanded to more modes, the mean square error loss is calculated by the original image and the circular reconstruction images in different intermediate modes.
(14) Generating the self-supervision semantic consistency loss of the codes of the X mode and the Y mode graph and the codes obtained by the X mode and the Y mode graph through the encoder by the decoder:
in the above formula, lossSupervision,9Code representing the loss of self-supervised semantic consistency between the Code generated by the decoder into the X-mode and Y-mode maps and the Code obtained by the X-mode and Y-mode maps through the encoderx,gAnd recoding the obtained coding result for the generated graph of the x mode.
(15) Self-supervised semantic consistency loss of X-modality and Y-modality graph coding:
in the above formula, lossSupervision,10Representing the loss of self-supervised semantic consistency of true X-mode and Y-mode graph coding, CodexCode representing the result of MRI encoding of the true x-modey,tAnd when the MRI representing the real x modality is converted into the recoding result after the MRI of the y modality is expanded to more modalities, the mean square error loss is calculated by the original image coding result and the recoding result of different intermediate modalities.
The generating component in the multi-modal graph generating training and auxiliary conversion training process is a common component, and synchronously trains and updates.
In the multi-modal graph generation training and auxiliary training process of the embodiment, the total loss of the discriminator component is as follows:
lossDiscriminator=∑i=2lossDiscriminator,i+∑i=2lossFeatureDiscriminator,i
in the above formula, lossDiscriminatorRepresents the loss of the total discriminator component, Σi=2lossDiscriminator,iRepresents the total loss, Σ, of the mode discriminatori=2lossFeatureDiscriminator,iRepresenting the total loss of the feature discriminator.
The total loss of the resulting component is:
lossGenerator=∑i=3lossGenerator,i+∑i=3lossSupervision,i
in the above formula, lossGeneratorRepresents the total loss of the generator component, Σi=3lossGenerator,iiRepresents the total antagonism loss, Σ, provided by the discriminatori=3losssupervision,iThe overall lesion signature is represented by a supervised loss and an individual self-supervised loss.
The above loss functions are not illustrated by mode expansion, and can directly increase the loss terms of the corresponding modes according to the number of the generated and converted modes, and are not limited to the two modes x and y in this embodiment.
For the training of a plurality of modes, only the mode registration image generation training method is needed to supplement the mode conversion training.
When training n (n >1) modes, the auxiliary training needs to perform n-mode training, each mode comprises n-1 steps S4-S5, and the training is respectively corresponding to the cycle conversion training of the current mode and other n-1 modes; in the generating training step S5, a label graph and a generating graph of n modalities are generated, each modality includes n-1 steps S9, and the conversion training is respectively corresponding to the current modality and other n-1 modalities.
The generation countermeasure network of this embodiment has an encoder for receiving random input, a decoder for decoding an input segmentation label map from the encoded result, and a feature discriminator for discriminating the encoded result from true or false, and then, for each mode, there are an encoder, a decoder, and a mode discriminator. In addition, the embodiment includes a feature extraction module for extracting structural features from a real image, an encoder for encoding the structural feature map into a normal distribution, a decoder for decoding the structural feature map from the normal distribution, a true and false structural feature map discriminator for guiding the reconstruction of the structural feature map, and a structural feature discriminator for performing true and false discrimination on the encoding result. After the training of the steps A) -A7) and the steps B1) -B3) is completed, only part of module components in the steps A) -A7) and the steps B1) -B3) need to be recombined, and then a large number of multi-modal registration images can be generated conveniently and quickly. From a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RMBy usingTrained structural feature DecoderFDecoding to generate a structural feature map FRM,FRMFusing with randomly selected segmentation label graph L, and inputting the fused result into the trained random input Encoder EncoderRMTo obtain a CodeRMFinally, the Decoder of different modes is trainedx、DecoderyTo CodeRMDecoding to generate a registered X mode image XgY mode diagram Yg。
Compared with the prior art, the method for generating the registered multi-modality MRI with the lesion segmentation tag in the embodiment has the following advantages: 1) data used for training is not required to be registered, unsupervised learning is adopted, multi-mode registration MRI images can be generated, generated data are labeled, and the number of modes is not limited. 2) The modularized design can conveniently carry out mode expansion, and make model training more flexible, training can be independently carried out or can be carried out synchronously, and the trained modules are combined and reused when in use. 3) The method for extracting the structural features is improved based on the traditional Sobel operator method, the Sobel operator is used for extracting the features and then further processing is carried out, a maximum value graph and a minimum value graph are obtained, and finally, the structural feature graphs are obtained through fusion, and enough structural information is reserved.
In the above formulas of this embodiment, x, y, and xr、yr、xt、yt、xc、yc、xg、yg、xg,t、yg,tX, Y original image, reconstruction graph, conversion graph, circulation conversion graph, generation graph and generation conversion graph, Encoderx、 Encodery、Decoderx、DecoderyCoder and decoder, respectively, representing a modality X, Yx、Codey、Codex,t、 Codey,t、Codex,g、Codey,gRepresents pairs x, y, xt、yt、xg、ygBy corresponding Encoder Encoderx、EncoderyAnd respectively coding the obtained characteristic results. F. Fx,g、Fy,gRepresenting the structural feature extraction method get _ F on any modal graph n and xg、ygExtracting the structure characteristic diagram, generating the structure characteristic diagram in an X mode, generating the structure characteristic diagram in a Y mode, wherein the Mask represents a Mask extracted by a Mask extraction method get _ Mask, and a DecoderF、DecoderMask、DecoderL、EncoderRMRespectively, a structural feature decoder, a mask decoder, a segmentation tag decoder, and a random input encoder. CodeF,mean、CodeF,logvarRepresentation pair F passes through Encoder EncoderFCharacteristic result obtained by encoding, Codee、CodeF,RMRepresents a distribution from normal N (0, 1)2) Randomly acquired matrix, CodeFRepresenting the normal distribution matrix after the addition of noise, Fr、FRMRepresent respective pair CodeF、CodeF,RMBy DecoderFThe reconstructed structure characteristic diagram, the random structure characteristic diagram and the Mask are obtained by decodingr、MaskRMRepresent respective pair CodeF、CodeF,RMBy DecoderMaskA reconstruction mask, a random mask, F, obtained by decodingRM,expandIs represented by FRMA feature tag map, Code, obtained by fusing with the segmentation tag map LRMRepresents a pair FRM,expandBy Encoder EncoderRMThe characteristic result obtained by coding, Lx,f、Ly,f、Lr、Lx,g、Ly,gRepresent respective pair Codex、Codey、CodeRM、Codex,g、 Codey,gBy DecoderLDecoding obtained X-mode segmentation label graph, Y-mode segmentation label graph and reconstruction segmentation label graph LrAnd generating a segmentation label map in an X mode and generating a segmentation label map in a Y mode. Discriminator, also mentioned earlier in relation to the training methodFIs a structural feature diagram discriminator, FeatureDiscrimidatorFIs a structural feature identifier, Discriminatorx、 DiscriminatoryShown is a discriminator for modality X, Y, FeatureDiscrimidator is a multipleA feature discriminator common to the modalities.
Further, the present embodiments also provide a system for generating registered multi-modality MRI with lesion segmentation tags, comprising:
a random matrix generation program unit for generating a random matrix from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RM;
A structural feature extraction program unit for extracting a random matrix CodeF,RMTrained structural feature Decoder in input generation countermeasure networkFDecoding to generate a structural feature map FRM;
Structural feature fusion program unit for fusing a structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;
a random coding program unit for inputting the fusion result into a trained random coder Encoder in the countermeasure networkRMObtaining a CodeRM;
A registration structure feature map generation program unit for encoding the CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRIig。
Furthermore, the present embodiment also provides a system for generating registered lesion segmentation tagged multi-modality MRI, comprising a computer device programmed or configured to execute the steps of the aforementioned method for generating registered lesion segmentation tagged multi-modality MRI, or a storage medium of the computer device having stored thereon a computer program programmed or configured to execute the aforementioned method for generating registered lesion segmentation tagged multi-modality MRI.
Furthermore, the present embodiments also provide a computer readable storage medium having stored thereon a computer program programmed or configured to perform the aforementioned method of generating registered lesion segmentation tagged multi-modality MRI.
The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and embellishments within the scope of the invention may occur to those skilled in the art without departing from the principle of the invention, and are considered to be within the scope of the invention.
Claims (10)
1. A method of generating registered multi-modality MRI with lesion segmentation tags, characterized by the implementation steps comprising:
1) from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RM;
2) Code random matrixF,RMTrained structural feature Decoder in input generation countermeasure networkFDecoding to generate a structural feature map FRM;
3) Structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;
4) inputting the fusion result into a trained random Encoder Encoder in the countermeasure networkRMObtaining a CodeRM;
5) Code to CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRI ig。
2. The method of generating registered multi-modality MRI with lesion segmentation tags according to claim 1, wherein step 2) is preceded by training a structure feature Decoder in a generation countermeasure networkFThe detailed steps comprise:
A1) randomly selecting a mode, obtaining a graph n from the mode, extracting structural features to obtain a structural feature graph F, and extracting a Mask to obtain a corresponding Mask;
A2) encoder for generating structural features in countermeasure networksFCoding the structural feature graph F to obtain a coding mean matrix CodeF,meanAnd variance matrix CodeF,logvarFrom a normal distribution N (0),12) To obtain random noise CodeeFrom the mean matrix CodeF,meanAnd variance matrix CodeF,logvarRandom noise CodeeThe three codes are synthesized to obtain an approximate normal distribution matrix Code added with noiseF;
A3) Decoder with maskMaskFor the approximately normal distribution matrix Code after adding noiseFDecoding to obtain reconstructed Maskr(ii) a Decoder using structural featuresFTo CodeFDecoding to obtain a reconstructed structural feature map Fr;
A4) Random generation of a normal distribution N (0, 1)2) Matrix Code ofF,RMDecoder using structural featuresFCode matrix pairF,RMDecoding to obtain a generated random structure characteristic diagram FRMUsing a mask DecoderMaskCode matrix pairF,RMDecoding to obtain the generated random MaskRM;
A5) Using a structural feature map DiscriminatorFRespectively to (F, Mask) and (F)RM,MaskRM) Carrying out identification, wherein the former is identified as true, and the latter is identified as false; wherein F is a structural feature diagram, Mask is a Mask, FRMIs a random structural feature map, MaskRMIs a random mask; using a structural feature identifier FeatureDiscrimidatorFRespectively to CodeFAnd CodeF,RMPerforming identification to identify the former as false and the latter as true, wherein the CodeFFor normal distribution matrices, Code, after the addition of noiseF,RMFor random generation of a conforming normal distribution N (0, 1)2) A matrix of (a);
A6) calculating loss according to the output result of each step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter; each component comprises a structural feature Encoder EncoderFDecoder for maskMaskDecoder for decoding structural characteristicsFStructural feature diagram DiscriminatorFAnd a structural feature identifier FeatureDiscrimidatorF;
A7) Judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step A1); otherwise, exiting.
3. The method for generating registered multi-modality MRI with lesion segmentation labels as claimed in claim 2, wherein the step A2) is integrated to obtain an approximately normal distribution matrix Code with noise addedFThe functional expression of (a) is:
CodeF=CodeF,mean+exp(0.5*CodeF,logvar)*Codee;
in the above formula, CodeF,meanEncoder for structural featuresFMean matrix, Code, obtained by encoding the structural feature map FF,logvarEncoder for structural featuresFCode obtained by coding the structural feature map FeIs normally distributed with N (0, 1)2) To obtain random noise.
4. The method of generating registered lesion segmentation tagged multi-modality MRI as set forth in claim 1, wherein the detailed steps of step 3) include:
3.1) randomly selecting a focus segmentation label graph L, converting the focus segmentation label graph L containing a plurality of categories into a one-hot matrix to obtain a multi-dimensional label matrix with the same channel number and category number, wherein only part of pixels in each channel are effective, the rest parts are filled 0, and the non-0 pixel regions are registered with each segmentation region in the focus segmentation label graph;
3.2) dividing the multi-dimensional label matrix and the structural feature map FRMStacking is carried out in the channel dimension together, and a matrix fusing two input sources is obtained as a fusion result.
5. The banding of generating registration of claim 1The multi-mode MRI method of focus segmentation labels is characterized in that the step 4) is preceded by an i-mode Decoder for training each mode iiI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLAnd training the random Encoder EncoderRMThe detailed steps comprise:
B1) i-modal Decoder for training each modal iiI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLTraining random EncoderRM;
B2) Calculating loss according to the output result of each training step and the corresponding loss function, calling an optimizer to derive the loss function to obtain the gradient of the model parameter in each component, and then calculating the difference between each parameter and the corresponding gradient to complete the update of the network parameter; the components include i-mode Decoder of i-modeiI-mode Encoder EncoderiFocus segmentation label DecoderLAnd a random Encoder EncoderRM;
B3) Judging whether a preset iteration ending condition is met, wherein the iteration ending condition is that the loss function value is lower than a set threshold value or the iteration frequency reaches a set step number, and if not, skipping to execute the step B1); otherwise, exiting.
6. Method for generating registered lesion segmentation tagged multi-modality MRI according to claim 5, characterized in that in step B1) an i-mode Decoder of each modality i is trainediI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLTraining the i-mode Decoder of a certain mode iiI-mode Encoder EncoderiAnd a lesion segmentation tag DecoderLThe detailed steps comprise:
step 1, inputting an original image i of a random modality i;
step 2, using an i-mode Encoder EncoderiEncoding the original image i to obtain an encoded Codei;
Step 3, usingDecoder of i-modeiCode pairiDecoding to obtain a reconstructed picture ir(ii) a Decoder for focus segmentation labelLCode pairiDecoding to obtain a focus segmentation label map Li,f(ii) a Meanwhile, for any other mode j, firstly using a j-mode DecoderjCode pairiDecoding to obtain a j mode conversion graph j of the original graph itReuse the j mode Encoder EncoderjJ mode conversion chart j for original image itCoding is carried out to obtain a Codej,tReuse the i-mode DecoderiCode pairj,tDecoding to obtain a cyclic reconstruction image i of the original image i with j mode as an intermediate modec;
Step 4, respectively passing through a mode DiscriminatorxI-mode conversion diagram i for converting original image i and each mode j into mode itDiscrimination is performed to discriminate the former as true and the latter as false.
7. Method for generating registered lesion segmentation tagged multi-modality MRI according to claim 5, characterized in that a stochastic Encoder Encoder is trained in step B1)RMThe detailed steps comprise:
step 1, randomly selecting a mode, and acquiring a graph n and a corresponding lesion segmentation label graph L from the modenObtaining a structural feature map F by using a structural feature extraction method1Obtaining a corresponding Mask by using a Mask extraction method; using lesion segmentation label map LnRemoving and extracting to obtain a structural feature map F1Obtaining a structural characteristic diagram F without focus information;
step 2, the structural feature graph F and the randomly input focus segmentation label graph L are randomly input and fused to obtain a fusion result FRM,expand;
Step 3, fusing the result FRM,expandSending into a random Encoder EncoderRMEncoding into CodeRM;
Step 4, Code is codedRMDecoder for input focus segmentation labelLDecoding out of weightEstablishing a focus segmentation label chart Lr(ii) a Simultaneously for each modality i: code to CodeRMI-mode Decoder for input mode iiGet i Modal to generate graph igGenerating the i mode into a graph igExtracting structural features to obtain a structural feature map Fi,gGenerating the i mode into a graph igInput i-mode Encoder EncoderiObtaining a Codei,gCode to Codei,gDecoder for input focus segmentation labelLDecode Ly,gA Decoder of j mode for inputting other modes jjObtaining corresponding j mode to generate focus segmentation label graph jg,t;
Step 5, aiming at each mode i, i mode Discriminator of each mode iiGenerating a graph i for the original image n and i of the modegCarrying out identification, wherein the former is identified as true, and the latter is identified as false; respectively pairing the Code codes through a feature discriminatorRMAnd Code of each modality iiPerforming identification to obtain CodeRMCode of each mode i identified as falseiThe authentication is true.
8. A system for generating registered multi-modality MRI with lesion segmentation tags, comprising:
a random matrix generation program unit for generating a random matrix from a normal distribution N (0, 1)2) Obtaining random matrix CodeF,RM;
A structural feature extraction program unit for extracting a random matrix CodeF,RMTrained structural feature Decoder in input generation countermeasure networkFDecoding to generate a structural feature map FRM;
Structural feature fusion program unit for fusing a structural feature map FRMObtaining a fusion result by random input fusion with a randomly selected focus segmentation label graph L;
a random coding program unit for inputting the fusion result into a trained random coder Encoder in the countermeasure networkRMObtaining a CodeRM;
A registration structure feature map generation program unit for encoding the CodeRMI-modal Decoder for generating trained individual modal i in countermeasure networkiSeparately generating registered i-mode MRIig。
9. A system for generating registered lesion segmentation tagged multi-modality MRI comprising a computer device, characterized in that the computer device is programmed or configured to perform the steps of the method for generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1 to 7, or a storage medium of the computer device has stored thereon a computer program programmed or configured to perform the method for generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1 to 7.
10. A computer readable storage medium having stored thereon a computer program programmed or configured to perform the method of generating registered lesion segmentation tagged multi-modality MRI of any one of claims 1-7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910764408.9A CN110544275B (en) | 2019-08-19 | 2019-08-19 | Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910764408.9A CN110544275B (en) | 2019-08-19 | 2019-08-19 | Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110544275A CN110544275A (en) | 2019-12-06 |
CN110544275B true CN110544275B (en) | 2022-04-26 |
Family
ID=68711559
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910764408.9A Active CN110544275B (en) | 2019-08-19 | 2019-08-19 | Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110544275B (en) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111062432B (en) * | 2019-12-13 | 2022-03-25 | 华中科技大学 | Semantically multi-modal image generation method |
CN111104997B (en) * | 2019-12-25 | 2023-05-23 | 青岛创新奇智科技集团股份有限公司 | Commodity two-dimensional code generation method and system based on deep learning |
CN111383215A (en) * | 2020-03-10 | 2020-07-07 | 图玛深维医疗科技(北京)有限公司 | Focus detection model training method based on generation of confrontation network |
CN111489802B (en) * | 2020-03-31 | 2023-07-25 | 重庆金域医学检验所有限公司 | Report coding model generation method, system, equipment and storage medium |
CN112529949A (en) * | 2020-12-08 | 2021-03-19 | 北京安德医智科技有限公司 | Method and system for generating DWI image based on T2 image |
CN112651916A (en) * | 2020-12-25 | 2021-04-13 | 上海交通大学 | Method, system and medium for pre-training of self-monitoring model |
CN114283151A (en) * | 2021-08-16 | 2022-04-05 | 腾讯科技(深圳)有限公司 | Image processing method, device, equipment and storage medium for medical image |
CN114266777A (en) * | 2021-12-16 | 2022-04-01 | 推想医疗科技股份有限公司 | Segmentation model training method, segmentation device, electronic device, and medium |
CN114638745B (en) * | 2022-03-16 | 2023-08-18 | 江南大学 | Medical image intelligent conversion method based on multi-borrowing information |
CN116503505B (en) * | 2023-06-20 | 2024-04-05 | 中山大学肿瘤防治中心(中山大学附属肿瘤医院、中山大学肿瘤研究所) | Artifact removal method, device, equipment and medium for CBCT image |
CN117011718B (en) * | 2023-10-08 | 2024-02-02 | 之江实验室 | Plant leaf fine granularity identification method and system based on multiple loss fusion |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017223560A1 (en) * | 2016-06-24 | 2017-12-28 | Rensselaer Polytechnic Institute | Tomographic image reconstruction via machine learning |
CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
CN109325931A (en) * | 2018-08-22 | 2019-02-12 | 中北大学 | Based on the multi-modality images fusion method for generating confrontation network and super-resolution network |
CN109493308A (en) * | 2018-11-14 | 2019-03-19 | 吉林大学 | The medical image synthesis and classification method for generating confrontation network are differentiated based on condition more |
CN109523584A (en) * | 2018-10-26 | 2019-03-26 | 上海联影医疗科技有限公司 | Image processing method, device, multi-mode imaging system, storage medium and equipment |
CN110047056A (en) * | 2018-01-16 | 2019-07-23 | 西门子保健有限责任公司 | With the cross-domain image analysis and synthesis of depth image to image network and confrontation network |
CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
-
2019
- 2019-08-19 CN CN201910764408.9A patent/CN110544275B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017223560A1 (en) * | 2016-06-24 | 2017-12-28 | Rensselaer Polytechnic Institute | Tomographic image reconstruction via machine learning |
CN108062753A (en) * | 2017-12-29 | 2018-05-22 | 重庆理工大学 | The adaptive brain tumor semantic segmentation method in unsupervised domain based on depth confrontation study |
CN110047056A (en) * | 2018-01-16 | 2019-07-23 | 西门子保健有限责任公司 | With the cross-domain image analysis and synthesis of depth image to image network and confrontation network |
CN109325931A (en) * | 2018-08-22 | 2019-02-12 | 中北大学 | Based on the multi-modality images fusion method for generating confrontation network and super-resolution network |
CN109523584A (en) * | 2018-10-26 | 2019-03-26 | 上海联影医疗科技有限公司 | Image processing method, device, multi-mode imaging system, storage medium and equipment |
CN109493308A (en) * | 2018-11-14 | 2019-03-19 | 吉林大学 | The medical image synthesis and classification method for generating confrontation network are differentiated based on condition more |
CN110084863A (en) * | 2019-04-25 | 2019-08-02 | 中山大学 | A kind of multiple domain image conversion method and system based on generation confrontation network |
Non-Patent Citations (5)
Title |
---|
Extended Modality Propagation: Image Synthesis of Pathological Cases;Nicolas Cordier et al.;《IEEE Transactions on Medical Imaging》;20160711;第2598-2608页 * |
Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks;Salman Ul Hassan Dar;《IEEE Transactions on Medical Imaging》;20190226;第2375-2388页 * |
SkrGAN: Sketching-rendering Unconditional Generative Adversarial Networks for Medical Image Synthesis;Tianyang Zhang et al.;《arXiv.org》;20190806;第1-9页 * |
Unsupervised domain adaptation in brain lesion segmentation with adversarial networks;Konstantinos Kamnitsas et al.;《arXiv.org》;20161228;第1-13页 * |
生成对抗网络在医学图像处理中的应用;陈锟等;《生命科学仪器》;20181025(第Z1期);第71-80、91页 * |
Also Published As
Publication number | Publication date |
---|---|
CN110544275A (en) | 2019-12-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110544275B (en) | Methods, systems, and media for generating registered multi-modality MRI with lesion segmentation tags | |
Lu et al. | Deep spine: automated lumbar vertebral segmentation, disc-level designation, and spinal stenosis grading using deep learning | |
Liu et al. | Learning disentangled representations in the imaging domain | |
Wu et al. | Unsupervised brain tumor segmentation using a symmetric-driven adversarial network | |
CN110559009B (en) | Method for converting multi-modal low-dose CT into high-dose CT based on GAN | |
CN110689561B (en) | Conversion method, system and medium of multi-modal MRI and multi-modal CT based on modular GAN | |
CN110544239A (en) | Multi-modal MRI conversion method, system and medium for generating countermeasure network based on conditions | |
CN108985345B (en) | Detection apparatus based on lung medical image fusion classification | |
KR102359474B1 (en) | Method for missing image data imputation using neural network and apparatus therefor | |
JP2019075108A (en) | Information processing method and device and information detection method and device | |
US11360180B2 (en) | Methods, systems, and computer readable media for using a trained adversarial network for performing retrospective magnetic resonance imaging (MRI) artifact correction | |
Wang et al. | MDU‐Net: A Convolutional Network for Clavicle and Rib Segmentation from a Chest Radiograph | |
CN111563427A (en) | Method, device and equipment for editing attribute of face image | |
WO2021184195A1 (en) | Medical image reconstruction method, and medical image reconstruction network training method and apparatus | |
Zhao et al. | MPSHT: multiple progressive sampling hybrid model multi-organ segmentation | |
CN112862727B (en) | Cross-modal image conversion method and device | |
Chen et al. | Sjdl-vehicle: Semi-supervised joint defogging learning for foggy vehicle re-identification | |
van Tulder et al. | Representation learning for cross-modality classification | |
CN112819689A (en) | Training method of face attribute editing model, face attribute editing method and equipment | |
CN115761371A (en) | Medical image classification method and device, storage medium and electronic equipment | |
CN113689435B (en) | Image segmentation method, device, electronic equipment and storage medium | |
CN113327221A (en) | Image synthesis method and device fusing ROI (region of interest), electronic equipment and medium | |
Xu et al. | Correlation via synthesis: End-to-end image generation and radiogenomic learning based on generative adversarial network | |
Sirati-Amsheh et al. | AE-BoNet: A deep learning method for pediatric bone age estimation using an unsupervised pre-trained model | |
CN114373532B (en) | Multi-mode medical image translation method for generating countermeasure network based on target perception |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |