CN116310604B - Placenta implantation parting assessment tool and method - Google Patents

Abstract

The invention relates to a placenta implantation parting assessment tool and a placenta implantation parting assessment method, wherein the tool comprises an interested region preprocessing input module, a multi-stream depth model module and a multi-stream model prediction fusion output module, wherein the interested region preprocessing input module is used for extracting an interested region to obtain whole ROI region labeling information; the multi-stream depth model module is used for performing independent training on the FIESTA sequence, the SSFSE sequence, the FIESTA ROI sequence and the SSFSE ROI sequence by adopting the same 18-layer 3D residual error network; the multi-stream model prediction fusion output module is used for averaging the prediction probability addition of the four streams FIESTA stream, SSFSE stream, FIESTA ROI stream and SSFSE ROI stream of the multi-stream depth model for different placenta implantation types to obtain a final fusion prediction probability value, and taking the placenta implantation type corresponding to the highest probability value as the final prediction output of the multi-stream model.

Description

Placenta implantation parting assessment tool and method

Technical Field

The invention relates to the technical field of image classification, in particular to a placenta implantation typing assessment tool and method based on region-of-interest multi-mode MRI image fusion.

Background

The placenta-implanted disease (placenta accrete spectrum disorders, PAS) is a generic term for a series of diseases caused by the penetration of placenta fluff into the myometrium to varying degrees, and is classified into an adhesion type placenta implant (PA), an implantation type Placenta Implant (PI), and a penetration type placenta implant (placenta percreta, PP) according to the penetration depth of the placenta. With the rising proportion of pregnancy again after cesarean section in China, the incidence rate of placenta implantation complications is continuously increased, and the main risk problems are fatal postpartum hemorrhage and high hysterectomy rate, and serious patients even lead to death of pregnant and lying-in women. The literature reports that the occurrence rate of hysterectomy in the perinatal period of women is 0.024% -0.87%, and the occurrence rate of hysterectomy in the perinatal period of women is as high as 73.3% (Wang Yuanyuan, volume 36, 5 of the journal of Chinese practical gynecology and obstetrics 2020, judging bad pregnancy outcome of placenta-implanted patients by using imaging indexes). Therefore, how to accurately evaluate the implantation type before operation, predict the bleeding possibility during operation, explore the innovative operation mode of reserving uterus and reducing bleeding, so as to finally reduce the fatal postpartum hemorrhage rate, the hysterectomy rate of young women and the mortality rate of pregnant and lying-in women, avoid the waste of blood sources, and reduce the economic burden of patients and countries is a practical problem to be urgently solved in clinic (see Chinese patent document CN 201610794451.6).

The pathological principle and mechanism of placenta implantation are not clear, and it is considered that the present study is related to decidua defect, and the decidua tissue of uterus at the attachment place of villus is partially lost, so that placenta villus is implanted into the myometrium even serosa layer or adjacent viscera. The gold standard for placenta implantation is to observe whether implantation occurs by pathological analysis of placenta and myometrium sample cells after operation. The method is characterized by the need to be performed post-operatively and by the need for an assay time of 2-3 days. But has high diagnosis accuracy, is a post-test method and is used for verifying and researching the cause of the implantation pathology. The preoperative diagnostic evaluation mode of placenta implantation mainly comprises two modes of ultrasound and nuclear magnetic imaging (MRI). Ultrasound is a common mode used by most pregnant women, and is usually required to be periodically examined by the pregnant women due to its characteristics of low price, convenient operation, etc. Nuclear magnetic imaging has attracted increasing attention as a means commonly used in recent years. The nuclear magnetic imaging is characterized by high definition, and can see many details of far ultrasonic imaging such as placenta texture, myometrium thickness and the like, thereby becoming an important auxiliary means.

The technical proposal disclosed in the Chinese patent document CN201610794451.6 is that placenta implantation is divided into (ultrasonic) light type, medium type and heavy type according to ultrasonic imaging characteristics, and the placenta implantation is respectively used for predicting adhesion type, implantation type and penetration type; the ultrasonic grading scale is established by performing image processing on the image data of the B ultrasonic, 9 indexes (placenta position, placenta thickness, post-placenta hypoechoic zone, bladder line, placenta pit, placenta basal blood flow, cervical blood sinus, cervical morphology and caesarean section history) are selected, and the typing prediction is performed on the clinically suspicious placenta implanted patient according to the grading scale. Although this patent document proposes a B-ultrasonic evaluation method for predicting and quantifying placenta implantation, the selection of the 9 indexes, the preprocessing of the B-ultrasonic images of the 9 indexes, the artificial weighting of the correlation between each index and the result, and other works mainly depend on the problems of more subjective experiences of doctors, weak generalization ability, differences in different hospital institution standards, and the like.

The inventor of the application proposes a new invention to overcome the defects of the prior art, the new invention is a placenta implantation typing assessment method based on multi-mode MRI image fusion of a region of interest, is a non-invasive method end-to-end for assessing the existence or severity of uterus placenta implantation, and provides reference for subsequent pregnancy operation.

Disclosure of Invention

Placenta implantation typing is mainly classified into 3 types: the invention aims to provide a placenta implantation typing assessment tool and method based on region-of-interest multi-mode MRI image fusion, which aims to solve the technical problems including at least how to fully utilize complementarity among modes under the multi-mode condition, fuse information of different modes and improve accuracy and robustness of placenta implantation typing prediction.

In order to achieve the above purpose, the invention provides a placenta implantation typing assessment tool based on region-of-interest multi-mode MRI image fusion, which comprises a region-of-interest preprocessing input module, a multi-stream depth model module and a multi-stream model prediction fusion output module, wherein the region-of-interest preprocessing input module is used for extracting a region of interest to obtain whole ROI region labeling information; adopting a classical medical image segmentation U-Net model to train and learn the whole manually marked ROI region; until the segmentation prediction result of each MRI image in the training set can reach a preset degree of fit with the golden standard manual labeling result of the MRI image; after training, implementing automatic integral ROI region segmentation on new sample data; the multi-stream depth model module is used for independently training a FIESTA sequence, an SSFSE sequence, a FIESTA ROI sequence and an SSFSE ROI sequence by adopting a 3D residual error network with the same depth of 18 layers, selecting cubic blocks with the size of D multiplied by H multiplied by W voxels from four MRI mode image sequences in a patch random segmentation mode as input, inputting the cubic blocks obtained by random segmentation into corresponding 3D residual error networks for training, calculating a loss value through a loss function according to the obtained predicted value and the true value, updating the parameter weight of the network through back propagation according to the loss value, repeating training in the mode until the network converges, and finally obtaining the optimal parameter weight; the multi-stream model prediction fusion output module is used for averaging the prediction probability addition of the four streams FIESTA streams, SSFSE streams, FIESTA ROI streams and SSFSE ROI streams of the multi-stream depth model for different placenta implantation types to obtain a final fusion prediction probability value, and taking the placenta implantation category corresponding to the highest probability value as the final prediction output of the multi-stream model.

Preferably, the region of interest is the four regions of uterus, placenta, bladder and cervical os as a whole ROI region.

Preferably, the specific method for extracting the region of interest firstly adopts open source labeling software ITK-Snap to label the sample, and obtains the labeling information of the whole ROI region.

Preferably, the specific method for training and learning the artificially marked whole ROI by adopting the classical medical image segmentation U-Net model comprises the following steps: respectively performing downsampling for 4 times and upsampling for 4 times on the MRI image; the input is an MRI image, two convolution layers are firstly passed through, and the convolution kernel numbers are 64; then carrying out pooling downsampling for the 1 st time, enabling the image size to be half of the original size, and then enabling the convolution kernel number to be 128 after two layers of convolution layers to be used for further extracting image features; the latter downsampling process is similar, each layer being convolved twice to extract image features; each layer of downsampling reduces the image by half, the number of convolution kernels is doubled, and finally the number of convolution kernels is 1024; the 4-time up-sampling process comprises the steps of firstly carrying out 1 st deconvolution up-sampling, changing the size of an image into twice of the original size, adding a feature layer by adopting a method of directly splicing the down-sampled image after cutting the down-sampled image into the same size, and then carrying out convolution to extract the features; then, through two convolution layers, the convolution kernel number is reduced from 1024 to 512; then up-sampling is carried out again, and the process is repeated; each layer is convolved twice to extract the characteristics, and each layer is sampled, the image is doubled, and the number of convolution kernels is reduced by half; the final convolution kernel after 4 upsamples drops to 64; the final step selects two 1 x 1 convolution kernels to change 64 feature channels into 2, dividing the image into background and objectA category; finally, outputting a predicted segmentation result and a real result, calculating a loss value through a loss function, and recording the predicted segmentation result asM ^pred The real result of manual marking isM ^gt The array shapes of the predicted segmented result and the artificially annotated real result are (D, H, W), where D, H, W represent depth, height and width, respectively, and both (i.e., the predicted segmented resultM ^pred And manually annotated real resultsM ^gt ) The range of values at each location is 0,1,G}, whereinGRepresenting the number of categories, the following loss function equation (1) is calculated:

(1)

wherein the method comprises the steps ofIs a kronecker function, which,grepresenting class values, M _i,j,k Voxels representing the depth i, height j and width k positions in the MRI image; returning the gradient to each parameter in the U-Net model through a back propagation algorithm according to the obtained loss value, so that each parameter updates the weight value of the parameter; the training process is repeated until the segmentation prediction result of each MRI image in the training set can reach a preset degree of fit with the golden standard manual labeling result of the MRI image; after training, automatic integral ROI region segmentation is realized on the new sample data.

Preferably, the MRI image input is a FIESTA modality MRI image or an SSFSE modality MRI image.

Preferably, the region of interest preprocessing input module provides four MRI modalities as inputs, including: FIESTA modality, SSFSE modality, FIESTA ROI modality, and SSFSE ROI modality.

Preferably, the depth 18 layer 3D residual network comprises 17 convolution layers and 1 full connection layer, and the structure is recorded as: ((Conv 1, resBlock1×2, resBlock2×2, resBlock3×2, resBlock4×2, FC), where Conv1 represents convolutional layers, resBlock1, resBlock2, resBlock3, and ResBlock4 represent convolutional residual blocks, each of which contains two layers of convolutional layers; the model randomly divides and selects cubic blocks with the size of D multiplied by H multiplied by W from a FIESTA sequence, an SSFSE sequence, a FIESTA ROI sequence and an SSFSE ROI sequence respectively, inputs the cubic blocks into corresponding 3D residual error networks for training and predicting, and the shapes of the cubic blocks are (1, D, H and W), wherein D, H and W are the depth, the height and the width of the cubic blocks respectively, and '1' represents the number of channels; the cube is first passed through a first convolution layer Conv1 to obtain an intermediate feature 1 whose shape is (64, D, H/2, W/2), where "64" indicates the number of channels, then down-sampled through a pooling layer to obtain a shape of (64, D/2, H/4, W/4), then passed through a first convolution residual block ResBlock1 to obtain an intermediate feature 2 whose shape is (64, D/2, H/4, W/4), as before, unchanged, then passed through a second convolution residual block ResBlock1 to obtain an intermediate feature 3 whose shape is still (64, D/2, H/4, W/4), then passed through a third convolution residual block ResBlock2 to obtain an intermediate feature 4 whose shape is (128, D/4, H/8,W/8), then passed through a fourth convolution residual block ResBlock2 to obtain an intermediate feature 5 whose shape is (128, D/4, H/8,W/8), and so on, then, a fifth convolution residual block ResBlock3 and a sixth convolution residual block ResBlock3 are adopted to obtain an intermediate feature 6 and an intermediate feature 7, wherein the shapes of the intermediate feature 6 and the intermediate feature 7 are 256, D/8,H/16 and W/16; then, obtaining an intermediate feature 8 and an intermediate feature 9 through a seventh convolution residual block ResBlock4 and an eighth convolution residual block ResBlock4, wherein the shapes of the intermediate feature 8 and the intermediate feature 9 are (512, D/16, H/32, W/32); then the shape is obtained through a pooling layer (512,1,1,1), then the shape is obtained through a full-connection layer (1024), namely a 1024-dimensional vector, and finally the final output shape is obtained through a softmax layer (3), namely a 3-dimensional vector, corresponding to the 3 classification problem: no adhesion/adhesion type, implantation type and penetration type.

Preferably, the multi-stream depth model module selects a cross entropy loss function to measure an error between a predicted value and a true value of the 3D residual network, where the cross entropy loss function is:

（2）

wherein the method comprises the steps ofLRepresents the cross entropy loss function, x represents the sample, y represents the true value,representing a predicted value, N representing the number of training samples; the FIESTA flow, SSFSE flow, FIESTA ROI flow and SSFSE ROI flow all employ the cross entropy loss functions described above as the loss functions used in their own training.

The invention also provides a placenta implantation typing evaluation method based on the region-of-interest multi-mode MRI image fusion, which comprises the following steps:

s1, regarding two sagittal-plane MRI sequence image data of pregnant women provided by a hospital, wherein the areas related to symptoms are uterus, placenta, bladder and cervical orifice, taking four related areas of uterus, placenta, bladder and cervical orifice as an integral ROI area, manually labeling a sample, then training and learning by using a medical image segmentation model, and automatically segmenting, extracting and preprocessing unlabeled new sample data to obtain an MRI mode based on the ROI;

s2, inputting four MRI modes of a FIESTA stream, an SSFSE stream, a FIESTA ROI stream and an SSFSE ROI stream into four independent 3D residual error networks, and independently training and learning characteristic information of the MRI mode image data input by each stream, and respectively predicting and outputting placenta implantation typing;

s3, based on the output of the step S2, each stream predicts the implantation type of the placenta, a group of probability values are output respectively, each group of probability values comprises 3 probabilities, each group of probability values corresponds to 3 types of placenta implantation, then the four groups of prediction probability values are averaged to be used as the final prediction, and the placenta implantation type with the highest probability is used as the final prediction output.

Preferably, the two sagittal MRI sequence image data include FIESTA sequences and SSFSE sequences.

Advantageous effects

Compared with the prior art, the invention has the beneficial effects that:

the placenta implantation parting assessment tool and method provided by the invention are placenta implantation parting assessment methods based on multi-mode MRI image fusion of a region of interest, are non-invasive methods end-to-end for assessing the existence or severity of uterus placenta implantation, and can provide references for subsequent pregnancy operations.

The feature information between the FIESTA flow and the SSFSE flow has complementarity, and the accuracy of the model can be provided after fusion; in addition, the ROI information provided by the FIESTA ROI stream and the SSFSE ROI stream can respectively supplement and strengthen the FIESTA stream and the SSFSE stream, and the characteristic information of the region of interest (ROI) is emphasized and learned, so that the robustness of the model can be improved.

Drawings

The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.

Fig. 1 is a general frame diagram of a placenta implantation typing evaluation tool according to the present invention.

Fig. 2 is a raw MRI image of the placental uterus provided by the present invention.

Fig. 3 is a view showing the whole labeling of the whole ROI area such as the placenta uterus manually on the original MRI image shown in fig. 2.

FIG. 4 is a diagram of the improved U-Net model network architecture of the present invention.

FIG. 5 is an exemplary view of a manually labeled ROI area binarized segmented image of a given test specimen in accordance with the present invention.

FIG. 6 is a schematic diagram of the result of automatic ROI segmentation predicted by the U-Net model for medical image segmentation corresponding to FIG. 5 on a test specimen.

Fig. 7 is a 3D residual network configuration diagram provided by the present invention.

Detailed Description

The present invention is described in more detail below to facilitate an understanding of the present invention.

As shown in fig. 1, the placenta implantation typing evaluation tool of the present invention includes a region of interest preprocessing input module, a multi-stream depth model module, and a multi-stream model prediction fusion output module.

First region of interest (region of interest, ROI) preprocessing input module

The format of the example data is two MRI image sequences, from FIESTA and SSFSE modalities, respectively. Extracting a region of interest (four regions of uterus, placenta, bladder and cervical os are taken as an integral ROI region): specifically, the open source labeling software ITK-Snap is adopted to label a few samples, and the whole ROI region labeling information is obtained, as shown in figure 3. The original MRI image of the placental uterus provided by the present invention is shown in fig. 2. Fig. 3 is a view showing the whole labeling of the whole ROI area such as the placenta uterus manually on the original MRI image shown in fig. 2.

The classical medical image segmentation U-Net model is adopted to train and learn the whole ROI areas such as the placenta uterus marked by manpower, the network structure diagram is shown in figure 4, and the MRI image is respectively subjected to 4 downsampling and 4 upsampling. The input is an MRI image (FIESTA mode MRI image or SSFSE mode MRI image), and the convolution kernels are 64 from the leftmost part through two convolution layers; then carrying out pooling downsampling for the 1 st time, enabling the image size to be half of the original size, and then enabling the convolution kernel number to be 128 after two layers of convolution layers to be used for further extracting image features; the same applies to the subsequent downsampling process, and each layer is subjected to convolution twice to extract image features; each downsampling layer reduces the image by half, the number of convolution kernels is doubled, and the number of convolution kernels is increased to 1024. The right part is in 4 up-sampling processes from bottom to top, the 1 st deconvolution up-sampling is firstly carried out from the rightmost bottom corner, the image size is doubled as the original image size, the image can only be enlarged due to deconvolution and the image can not be restored, in order to reduce the data loss, a method of cutting the image in the left down-sampling process into the same size and then directly splicing the image is adopted to add a characteristic layer, and then the deconvolution is carried out to extract the characteristics; then, through two convolution layers, the convolution kernel number is reduced from 1024 to 512; then up-sampling is performed again, and the above-described process is repeated. Each layer is convolved twice to extract features, and each up-sampling layer doubles the image and reduces the number of convolution kernels by half. The final convolution kernel after 4 upsamples drops to 64.

The final step selects two 1 x 1 convolution kernels to change the 64 feature channels to 2, which in effect becomes a two-classification problem, separating the image into background and object two categories. Finally, outputting a predicted segmentation result and a real result, calculating a loss value through a loss function, and recording the predicted segmentation result asM ^pred The real result of manual marking isM ^gt The arrays are (D, H, W), wherein D, H, W represent depth, height and width, respectively (i.e. the predicted segmentation result isM ^pred And the real result of manual marking isM ^gt ) The range of values at each location is 0,1,G}, hereGRepresenting the number of categories, the present invention calculates the following loss function equation (1):

(1)

wherein the method comprises the steps ofIs a kronecker function, G represents a class value, the value range is {1,2, …, G }, G represents a background (the background is represented by class 0), and a certain G value in the G values {1,2, …, G } represents a corresponding G-th different class (in the present application, represents different types of placenta implantation, in the present application, there are 3 classes, namely G=3, the value range of G is {0,1,2,3 });

M _i,j,k representing voxels of depth i, height j and width k locations in the MRI image. And returning the gradient to each parameter in the U-Net model through a back propagation algorithm according to the obtained loss value, so that each parameter updates the weight value of the parameter.

The training process is repeated until the segmentation prediction result of each MRI image in the training set can reach higher fitness with the golden standard artificial labeling result of the image. The given test sample manual labeling ROI region binarization segmentation image is shown in figure 5, automatic integral ROI region segmentation is realized on new sample data after training is completed, and the result of automatic integral ROI region segmentation is shown in figure 6. Thus, finally, four MRI modalities may be provided as inputs: FIESTA modality, SSFSE modality, FIESTA ROI modality, and SSFSE ROI modality.

(II) Multi-stream deep model Module

The module adopts four 3D residual error networks, specifically adopts a 3D residual error network with depth of 18 layers, as shown in fig. 7, and comprises 17 convolution layers and 1 full connection layer, and the structure is recorded as follows: (Conv 1, resBlock1×2, resBlock2×2, resBlock3×2, resBlock4×2, FC), where Conv1 represents convolutional layers, resBlock1, resBlock2, resBlock3, and ResBlock4 represent convolutional residual blocks, each of which contains two layers of convolutional layers. The model randomly cuts and selects cubes with the size of D multiplied by H multiplied by W from the FIESTA sequence, the SSFSE sequence, the FIESTA ROI sequence and the SSFSE ROI sequence respectively, inputs the cubes into corresponding 3D residual error networks for training and prediction, and the shapes of the cubes are (1, D, H and W), wherein D, H and W are the depth, the height and the width of the cubes respectively, and '1' represents the number of channels. The cube first passes through the first convolution layer Conv1 to obtain an intermediate feature 1, the shape of which is (64, D, H/2, W/2), where "64" represents the number of channels; then down-sampling is carried out through the pooling layer, and the shape is changed into (64, D/2, H/4, W/4); then, the intermediate feature 2 is obtained through a first convolution residual block ResBlock1, the shape of the intermediate feature 2 is (64, D/2, H/4, W/4) which is the same as the previous shape and has no change, and then the intermediate feature 3 is obtained through a second convolution residual block ResBlock1, and the shape of the intermediate feature 3 is still (64, D/2, H/4, W/4); then, a third convolution residual block ResBlock2 is passed to obtain an intermediate feature 4, the shape of which is (128, D/4, H/8,W/8), and then a fourth convolution residual block ResBlock2 is passed to obtain an intermediate feature 5, the shape of which is still (128, D/4, H/8,W/8); and so on, then passing through a fifth convolution residual block ResBlock3 and a sixth convolution residual block ResBlock3 to obtain an intermediate feature 6 and an intermediate feature 7, wherein the shapes of the intermediate feature 6 and the intermediate feature 7 are (256, D/8,H/16, W/16); then, obtaining an intermediate feature 8 and an intermediate feature 9 through a seventh convolution residual block ResBlock4 and an eighth convolution residual block ResBlock4, wherein the shapes of the intermediate feature 8 and the intermediate feature 9 are (512, D/16, H/32, W/32); then the shape is obtained through a pooling layer (512,1,1,1), then the shape is obtained through a full-connection layer (1024), namely a 1024-dimensional vector, and finally the final output shape is obtained through a softmax layer (3), namely a 3-dimensional vector, corresponding to the 3 classification problem: no adhesion/adhesion, implantation type, penetration type.

Training process: the FIESTA sequence, the SSFSE sequence, the FIESTA ROI sequence and the SSFSE ROI sequence all adopt the same 18-layer 3D residual error network, but are trained independently, a cube with the size of D multiplied by H multiplied by W voxels is selected from the four MRI mode image sequences in a patch-based random segmentation mode to serve as input, the method is an effective method for data augmentation, the cube obtained by random segmentation is input into the corresponding 3D residual error network to be trained, a loss value is calculated through a loss function according to the obtained predicted value and the obtained true value, then the parameter weight of the network is updated through back propagation according to the loss value, and in the method, a large number of repeated training networks are used until the network converges, and finally the optimal parameter weight is obtained. Wherein the cross entropy loss function is selected to measure the error between the predicted value and the true value of the 3D residual network as shown in the following formula (2):

（2）

wherein the method comprises the steps ofLRepresents the cross entropy loss function, x represents the sample, y represents the true value,representing a predicted value, N representing the number of training samples; the FIESTA flow, SSFSE flow, FIESTA ROI flow and SSFSE ROI flow all employ the cross entropy loss functions described above as the loss functions used in their own training.

(III) multi-stream model prediction fusion output module

The four streams FIESTA, SSFSE, FIESTA and SSFSE ROI of the multi-stream depth model are all trained separately, the output is the probability of each prediction of the type of placenta implantation, in the example shown in fig. 1, the probability of prediction of FIESTA for different types of placenta implantation is (0.2,0.3,0.5), the probability of prediction of SSFSE for different types of placenta implantation is (0.1,0.3,0.6), the probability of prediction of FIESTA ROI for different types of placenta implantation is (0.2,0.1,0.7), and the probability of prediction of SSFSE ROI for different types of placenta implantation is (0.1,0.1,0.8); then, in the multi-stream output fusion block, a method of averaging the addition of four stream output prediction probabilities is adopted to obtain a final fusion prediction probability value (0.15,0.20,0.65), and the placenta implantation type 3 corresponding to 0.65 with the highest probability value is taken as the final prediction output of the multi-stream model, so that in the example shown in fig. 1, the final prediction result is penetration type placenta implantation.

The invention also provides a placenta implantation typing evaluation method based on the region-of-interest multi-mode MRI image fusion, which comprises the following steps:

s1, preprocessing an input module of a region of interest (region of interest, ROI): two sagittal MRI sequence image data for pregnant women provided at the hospital: FIESTA sequences and SSFSE sequences. The areas related to the symptoms are uterus, placenta, bladder and cervical orifice, the four related areas are taken as an integral ROI area, a small number of samples are marked manually, then medical image segmentation model training learning is utilized, automatic segmentation, extraction and pretreatment are carried out on unlabeled new sample data, and an MRI mode based on the ROI is obtained;

s2, a multi-stream depth model module: the module is composed of four independent 3D residual networks, corresponding to the four MRI modality inputs provided in step S1, namely FIESTA flow, SSFSE flow, FIESTA ROI flow and SSFSE ROI flow, respectively. Each stream is independently trained and learns the characteristic information of the MRI modal image data input by each stream, and the placenta implantation typing is respectively predicted;

s3, a multi-stream model prediction fusion output module: the module predicts the implantation type of the placenta by each stream based on the output of the multi-stream depth model in the step S2, respectively outputs a group of probability values, wherein each group of probability values comprises 3 probabilities which respectively correspond to 3 types of placenta implantation, averages the four groups of prediction probability values to be used as the final prediction, and takes the placenta implantation type with the highest probability as the final prediction output. The feature information between the FIESTA stream and the SSFSE stream has complementarity, and the accuracy of the model can be provided after fusion; in addition, the ROI information provided by the FIESTA ROI stream and the SSFSE ROI stream can respectively supplement and strengthen the FIESTA stream and the SSFSE stream, and the characteristic information of the region of interest (ROI) is emphasized and learned, so that the robustness of the model can be improved.

The foregoing describes preferred embodiments of the present invention, but is not intended to limit the invention thereto. Modifications and variations to the embodiments disclosed herein may be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims (6)

1. The placenta implantation parting assessment tool based on the region-of-interest multi-mode MRI image fusion is characterized by comprising a region-of-interest preprocessing input module, a multi-stream depth model module and a multi-stream model prediction fusion output module, wherein the region-of-interest preprocessing input module is used for extracting a region of interest to obtain whole ROI region labeling information; adopting a classical medical image segmentation U-Net model to train and learn the whole manually marked ROI region; the method comprises the following steps: the predictive segmentation result is recorded asThe true result of the manual annotation is +.>The array shape of the predicted segmentation result and the artificially marked real result are +.>Wherein->Representing depth, height and width, respectively, and predicted segmentation result +.>And artificially annotated real results->The value range in each position is +.>Wherein->Representing the number of categories, the following loss function equation (1) is calculated:

;

wherein the method comprises the steps ofIs a Croneck function, g represents the class value,/->Voxels representing the depth i, height j and width k positions in the MRI image; returning the gradient to each parameter in the U-Net model through a back propagation algorithm according to the obtained loss value, so that each parameter updates the weight value of the parameter;

until the segmentation prediction result of each MRI image in the training set can reach a preset degree of fit with the golden standard manual labeling result of the MRI image; after training, implementing automatic integral ROI region segmentation on new sample data;

the multi-stream depth model module is used for performing independent training on the FIESTA sequence, the SSFSE sequence, the FIESTA ROI sequence and the SSFSE ROI sequence by adopting a 3D residual error network with the same depth of 18 layers, and selecting from four MRI mode image sequences in a patch-based random segmentation modeCube of voxel sizeInputting the cube blocks obtained by random segmentation into corresponding respective 3D residual error networks for training, calculating a loss value through a loss function according to the obtained predicted value and the true value, updating the parameter weight of the network through back propagation according to the loss value, and repeating training in the mode until the network converges to finally obtain the optimal parameter weight; the multi-stream model prediction fusion output module is used for averaging the prediction probability addition of four streams of FIESTA streams, SSFSE streams, FIESTA ROI streams and SSFSE ROI streams of the multi-stream depth model for different placenta implantation types to obtain a final fusion prediction probability value, and taking the placenta implantation category corresponding to the highest probability value as the final prediction output of the multi-stream model;

the depth is 18 layers 3D residual error network includes 17 convolution layers and 1 full connection layer, and the structure is recorded as:wherein->Representing a convolution layer->Representing convolution residual blocks, each convolution residual block containing two layers of convolution layers; the models are respectively from->Sequence random segmentation selection +.>The cubic blocks with the size of the voxels are input into corresponding 3D residual error networks for training prediction, and the shape of the cubic blocks is +.>,/>Depth, height and width of the cubes, respectively, "1" tableShowing the number of channels; the cube first passes through the first convolution layer +.>An intermediate feature 1 is obtained whose shape is (64, D, H/2, W/2), where "64" represents the number of channels; then down-sampling is carried out through the pooling layer, and the shape is changed into (64, D/2, H/4, W/4); then, the intermediate feature 2 is obtained through a first convolution residual block ResBlock1, the shape of the intermediate feature is (64, D/2, H/4, W/4), and then the intermediate feature 3 is obtained through a second convolution residual block ResBlock1, and the shape of the intermediate feature is still (64, D/2, H/4, W/4); then, a third convolution residual block ResBlock2 is passed to obtain an intermediate feature 4, the shape of which is (128, D/4, H/8,W/8), and then a fourth convolution residual block ResBlock2 is passed to obtain an intermediate feature 5, the shape of which is still (128, D/4, H/8,W/8); then, a fifth convolution residual block ResBlock3 and a sixth convolution residual block ResBlock3 are adopted to obtain an intermediate feature 6 and an intermediate feature 7, wherein the shapes of the intermediate feature 6 and the intermediate feature 7 are 256, D/8,H/16 and W/16; then, obtaining an intermediate feature 8 and an intermediate feature 9 through a seventh convolution residual block ResBlock4 and an eighth convolution residual block ResBlock4, wherein the shapes of the intermediate feature 8 and the intermediate feature 9 are (512, D/16, H/32, W/32); then the shape (512,1,1,1) is obtained through a pooling layer, then the vector with 1024 dimensions is obtained through a full-connection layer, and finally the final vector with 3 dimensions is obtained through a softmax layer, and the problem of 3 classification is solved: non-adhesion/adhesion type, implantation type and penetration type;

the multi-stream depth model module selects a cross entropy loss function to measure the error between the predicted value and the true value of the 3D residual error network, and the cross entropy loss function is as follows:

;

wherein the method comprises the steps ofLRepresents the cross entropy loss function, x represents the sample, y represents the true value,representing pre-emphasisMeasuring a value, wherein N represents the number of training samples; the FIESTA flow, the SSFSE flow, the FIESTA ROI flow and the SSFSE ROI flow all adopt the cross entropy loss function as the loss function used in self training;

the input MRI image is a FIESTA mode MRI image or an SSFSE mode MRI image;

the region of interest preprocessing input module provides four MRI modalities as inputs, including: FIESTA modality, SSFSE modality, FIESTA ROI modality, and SSFSE ROI modality.

2. The tool of claim 1, wherein the region of interest is a region of interest comprising four regions of uterus, placenta, bladder and cervical os as a single ROI.

3. The placenta implantation typing evaluation tool based on the region-of-interest multi-mode MRI image fusion of claim 1, wherein the specific method for extracting the region-of-interest is to label the sample by using open-source labeling software ITK-Snap to obtain the labeling information of the whole ROI region.

4. The placenta implantation typing evaluation tool based on region-of-interest multi-mode MRI image fusion according to claim 1, wherein the specific method for training and learning the artificially labeled whole ROI region by using a classical medical image segmentation U-Net model is as follows: respectively performing downsampling for 4 times and upsampling for 4 times on the MRI image; the input is an MRI image, two convolution layers are firstly passed through, and the convolution kernel numbers are 64; then carrying out pooling downsampling for the 1 st time, enabling the image size to be half of the original size, and then enabling the convolution kernel number to be 128 after two layers of convolution layers to be used for further extracting image features; the following downsampling process is: each layer is subjected to convolution twice to extract image characteristics; each layer of downsampling reduces the image by half, the number of convolution kernels is doubled, and finally the number of convolution kernels is 1024; the 4-time up-sampling process comprises the steps of firstly carrying out 1 st deconvolution up-sampling, changing the size of an image into twice of the original size, adding a feature layer by adopting a method of directly splicing the down-sampled image after cutting the down-sampled image into the same size, and then carrying out convolution to extract the features; then, through two convolution layers, the convolution kernel number is reduced from 1024 to 512; then up-sampling is carried out again, and the process is repeated; each layer is convolved twice to extract the characteristics, and each layer is sampled, the image is doubled, and the number of convolution kernels is reduced by half; the final convolution kernel after 4 upsamples drops to 64; the final step selects two 1 x 1 convolution kernels to change 64 feature channels into 2, and divides the image into two categories of background and target; and finally, outputting a predicted segmentation result and a real result, and calculating through a loss function to obtain a loss value.

5. The method for evaluating a tool for evaluating the type of placental implantation based on region-of-interest multi-modality MRI image fusion according to any one of claims 1 to 4, characterized by comprising the steps of:

s1, regarding two sagittal-plane MRI sequence image data of pregnant women provided by a hospital, wherein the areas related to symptoms are uterus, placenta, bladder and cervical orifice, taking four related areas of uterus, placenta, bladder and cervical orifice as an integral ROI area, manually labeling a sample, then training and learning by using a medical image segmentation model, and automatically segmenting, extracting and preprocessing unlabeled new sample data to obtain an MRI mode based on the ROI;

s2, inputting four MRI modes of a FIESTA stream, an SSFSE stream, a FIESTA ROI stream and an SSFSE ROI stream into four independent 3D residual error networks, and independently training and learning characteristic information of the MRI mode image data input by each stream, and respectively predicting and outputting placenta implantation typing;

s3, based on the output of the step S2, each stream predicts the implantation type of the placenta, a group of probability values are output respectively, each group of probability values comprises 3 probabilities, each group of probability values corresponds to 3 types of placenta implantation, then the four groups of prediction probability values are averaged to be used as the final prediction, and the placenta implantation type with the highest probability is used as the final prediction output.

6. The method of claim 5, wherein the two sagittal MRI sequence image data comprise FIESTA sequences and SSFSE sequences.