Multi-modal image registration method based on synthetic ultrasound image
Technical Field
The invention relates to the field of multi-modal image registration, in particular to a multi-modal image registration method based on a synthesized ultrasonic image.
Background
Both ultrasound and magnetic resonance images are currently widely used in the diagnosis of various medical cases, for example, to detect infarcts and tumors in the head, to detect acute and chronic changes in the liver, and to navigate through various clinical procedures. Real-time imaging of the liver is essential for detecting lesions or clinical treatment, the requirement of real-time imaging can be met by utilizing an ultrasonic probe to scan and image, and the ultrasonic imaging is less harmful to a human body due to non-invasive imaging. However, compared with the magnetic resonance image, the ultrasound image has lower imaging quality, and the magnetic resonance image can provide more anatomical detail information to better assist diagnosis and treatment. However, the magnetic resonance image can only be acquired before operation, and cannot be adjusted in real time according to changes such as real-time pose of a patient during operation, so that the effect of providing two images for auxiliary treatment simultaneously in the clinical operation process can be better, and therefore, the registration, fusion and display of the two images in the operation process are very necessary.
It is difficult to directly register a magnetic resonance image to an ultrasound image using conventional methods because the two images are very different. The method is characterized in that ultrasonic synthesis is carried out through a magnetic resonance image, and the synthesis ultrasonic and the real ultrasonic are fused, so that the method is one way in a registration fusion technology, and the registration of the images of two different modes is converted into the image registration in the same mode, so that the registration difficulty is reduced, and the registration precision is improved.
In recent years, many scholars have been engaged in simulation studies of ultrasound images. These studies mainly involve simulating ultrasound images from both computed tomography images and magnetic resonance images to address the registration problem of different modality images prior to intervention. To simulate an ultrasound image, an automated image registration algorithm is presented using an ultrasound physics-based model and measuring linear correlation using a linearly combined correlation such that images of two different modalities complete spatial alignment. However, the simulation of the ultrasound image takes a lot of time, so that this method cannot be used in a real-time surgical navigation system, and cannot accurately simulate the information of the tumor region, and the lack of the tumor can cause unstable image registration before and after the resection. The registration fusion technology based on ultrasonic simulation becomes a research hotspot and achieves certain results, but the following defects still exist: the simulation of a three-dimensional magnetic resonance image into a three-dimensional ultrasonic image consumes a lot of time and is not suitable for a real-time image-guided surgery navigation system; when a tumor is included in the magnetic resonance image, an ultrasound image of the tumor at the corresponding position cannot be correctly simulated.
Therefore, we propose an ultrasound image synthesis technique based on deep learning and apply the technique to multi-modal image registration. The liver multi-modality image registration technology based on the magnetic resonance image synthesis ultrasonic image must meet the following conditions: (1) synthesizing the magnetic resonance image into an ultrasonic image in real time; (2) an ultrasound image containing the tumor can be synthesized.
In view of the above, it is a technical problem to be solved in the art to provide a new multi-modal image registration method based on a synthesized ultrasound image, which overcomes the above drawbacks in the prior art.
Disclosure of Invention
The present invention is directed to overcoming the above-mentioned drawbacks of the prior art and providing a multi-modal image registration method based on a synthesized ultrasound image.
The object of the invention can be achieved by the following technical measures:
the invention provides a multi-modal image registration method based on a synthesized ultrasonic image, which comprises the following steps:
s1, acquiring a plurality of three-dimensional magnetic resonance images of the same part and real ultrasonic images corresponding to the three-dimensional magnetic resonance images as training samples;
s2, constructing a generation countermeasure network, wherein the generation countermeasure network comprises a generator and a discriminator;
s3, inputting the acquired three-dimensional magnetic resonance image into a generator to obtain an output result, inputting the output result of the generator and the corresponding real ultrasonic image into a discriminator to train the generation countermeasure network, and generating a corresponding composite ultrasonic image of the three-dimensional magnetic resonance image by using the generator in the trained generation countermeasure network;
s4, registering and fusing the synthesized ultrasonic image and a real ultrasonic image corresponding to the three-dimensional magnetic resonance image to obtain registration parameters, and registering the three-dimensional magnetic resonance image and the real ultrasonic image according to the registration parameters.
Further, in step S3, the step of "training the generated countermeasure network" includes:
obtaining an L1 loss function of the generator according to the output result of the generator and the corresponding real ultrasonic image;
based on the least square loss function of the generator and the judger in the generation impedance network, obtaining the total loss function of the generator according to the output result of the judger and the L1 loss function of the real ultrasonic image, and obtaining the total loss function of the judger according to the output result of the judger;
and respectively updating parameters in the network structures of the arbiter and the generator according to the total loss function of the arbiter and the total loss function of the generator until the generation of the confrontation network convergence.
Further, in step S3, the step of "training the generated countermeasure network" further includes:
different learning rates are set for the generator and the discriminator.
Further, the arbiter comprises a local arbiter and a global arbiter, and the local arbiter comprises a first local arbiter and a second local arbiter.
Further, the L1 loss function is defined as
Wherein, I
MRRepresenting a magnetic resonance image, G (I)
MR) Representing a composite ultrasound image, I
USFor the input true ultrasound image, p (I)
US) For true ultrasound data distribution, p (I)
MR) Is a magnetic resonance data distribution;
the least squares loss function of the generator is
The least squares loss function of the discriminator is
Wherein, IMRRepresenting a magnetic resonance image, G (I)MR) Representing a composite ultrasound image, IUSFor the input real ultrasound image, a and b are labels of the generated data and the real data, respectively, and c is a label of the data considered as false by the generator and the discriminator;
setting a to 0 and b to c to 1, and substituting them into the above formula, the overall loss function of the generator can be obtained as follows:
the overall loss function of the judger is
Further, the step S4 includes
Setting a pyramid into multiple layers by utilizing a pyramid algorithm, wherein each layer corresponds to one scale, and sampling the synthesized ultrasonic image and the first real ultrasonic image in a layering manner respectively;
initializing deformation parameters from the lowest layer, superposing the deformation parameters on the real ultrasonic image, and calculating the similarity measure and the deformation field of the synthesized ultrasonic image and the real ultrasonic image under the scale corresponding to the same layer;
superposing the calculated deformation field to the upper layer of the pyramid as the deformation parameter of the layer, and continuing to perform similarity measurement and optimized deformation calculation on the synthesized ultrasonic image and the real ultrasonic image of the layer until the last layer of the pyramid to obtain a registration parameter;
and directly applying the registration parameters obtained by registering the synthesized ultrasonic image and the real ultrasonic image to the three-dimensional magnetic resonance image, and finishing registration fusion of the three-dimensional magnetic resonance image and the real ultrasonic image after the three-dimensional magnetic resonance image is transformed by the registration parameters.
Further, the step of calculating the "similarity measure" includes: and characterizing the same layer of the synthesized ultrasonic image and the real ultrasonic image structure by using a neighborhood description operator MIND, and then obtaining the similarity measure of the two images by using the difference square sum SSD of the characterization results as a registration measure.
Further, in the step of calculating the deformation field method, the optimized deformation function of the same layer is obtained by using a method of Gaussian Newton gradient descent.
Further, the MIND calculation is defined as:
wherein n is a normalized vector, R belongs to R as a search area, and the distance D
p(I,x
1,x
2)=∑
p∈P(I(x
1+p)-I(x
2+p))
2All patches within the search region R and in two voxels x are calculated
1And x
2SSD within a centered patch, the similarity measure being
An optimization function of the optimal deformation field is
Wherein, u ═ w (u, v, w)
TRepresenting the deformation field.
Further, the pyramid layer number is 3, and the sampling ratio is 2 × 2 × 2.
The invention has the beneficial effects that the invention provides a multi-mode image registration method based on a synthesized ultrasonic image, a magnetic resonance image and a real ultrasonic image at the same position corresponding to the magnetic resonance image are given, a simulated synthesized ultrasonic image is generated by constructing a generation countermeasure network of a generator and a discriminator, the synthesized ultrasonic image and the real ultrasonic image are registered to obtain a registration parameter, the registration parameter is applied to the magnetic resonance image, and the final registration fusion of the magnetic resonance image and the ultrasonic image is completed, compared with the prior art, the method has the following advantages:
1. the ultrasonic image can be synthesized from the magnetic resonance image in real time, and the requirement of real-time image-guided surgery is met;
2. the synthesized ultrasonic image is closer to a real ultrasonic image, the image quality is higher, and important detail information is better saved;
3. when the magnetic resonance image contains the tumor, the simulation ultrasonic image can be accurately synthesized;
4. the final registration technology does not need a complex registration algorithm, and can achieve a good registration effect only by a traditional simple registration algorithm.
Drawings
Fig. 1 is a flow chart of a multi-modality image registration method based on a synthesized ultrasound image according to an embodiment of the present invention.
FIG. 2 is a flow chart of a composite ultrasound image of an embodiment of the present invention.
Figure 3 is a flow chart of the registration of a magnetic resonance image and a true ultrasound image of an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In order to make the description of the present disclosure more complete and complete, the following description is given for illustrative purposes with respect to the embodiments and examples of the present invention; it is not intended to be the only form in which the embodiments of the invention may be practiced or utilized. The embodiments are intended to cover the features of the various embodiments as well as the method steps and sequences for constructing and operating the embodiments. However, other embodiments may be utilized to achieve the same or equivalent functions and step sequences.
Referring to fig. 1, fig. 1 is a flowchart illustrating a multi-modality image registration method for a magnetic resonance image synthesis ultrasound image according to an embodiment of the invention, and the invention is specifically explained with reference to fig. 1.
In step S1, acquiring a plurality of three-dimensional magnetic resonance images of the same part of the same patient and a real ultrasound image corresponding to each three-dimensional magnetic resonance image as a training sample;
in step S2, a generation countermeasure network including a generator and a discriminator is constructed;
in the step S3, inputting the acquired three-dimensional magnetic resonance image into the generator to obtain an output result, and inputting the output result of the generator and the corresponding real ultrasound image into the discriminator to train the generation countermeasure network, and generating a corresponding composite ultrasound image of the three-dimensional magnetic resonance image by using the generator in the generation countermeasure network after training;
in step S4, the synthetic ultrasound image and the real ultrasound image corresponding to the three-dimensional magnetic resonance image are registered and fused to obtain registration parameters, and the three-dimensional magnetic resonance image and the real ultrasound image are registered according to the registration parameters.
According to the method, a magnetic resonance image and a real ultrasonic image at the same position corresponding to the magnetic resonance image are given, a simulated synthetic ultrasonic image is generated by constructing a generator and a generation countermeasure network of a discriminator, registration parameters are obtained by registering the simulated synthetic ultrasonic image and the real ultrasonic image, the registration parameters are applied to the magnetic resonance image, and the final registration fusion of the magnetic resonance image and the ultrasonic image is completed.
Referring to fig. 2, fig. 2 is a flow chart illustrating the generation of a composite ultrasound image according to an embodiment of the present invention, and the generation process of the composite ultrasound image is explained in detail with reference to fig. 2.
The generation type countermeasure network comprises a generator and a discriminator, wherein the discriminator comprises a local discriminator and a global discriminator, the generator is used for generating a synthetic image which is approximate to a target ultrasonic image from a magnetic resonance image, the synthetic image is input into the discriminator, the discriminator cannot distinguish the true and false of the synthetic image and the target image as far as possible, and the discriminator is used for distinguishing the target image and the synthetic image as far as possible and transmitting false information to the generator for the generator to update parameters. The discriminator comprises a global discriminator and a local discriminator, the global discriminator ensures that the synthesized image and the target image are similar as much as possible in global structure information, the local discriminator ensures that the synthesized image and the target image are similar as much as possible in local detail information, and the local discriminator comprises a first local discriminator and a second local discriminator.
The process of deep network learning is a process of continuously iterating and optimizing a loss function, in order to ensure that a synthetic image is as close to a real image as possible, similarity calculation is carried out on the synthetic ultrasonic image and the real ultrasonic image, and the L1 loss of the synthetic ultrasonic image and the real ultrasonic image is calculated, the L1 loss can stabilize training of the whole network while ensuring low-frequency characteristic similarity, and the loss of the synthetic image generated by a generator is calculated by the L1 loss:
wherein, IMRRepresenting a magnetic resonance image, G (I)MR) Representing a composite ultrasound image, IUSFor a true ultrasound image p (I)US) For true ultrasound data distribution, p (I)MR) Is a magnetic resonance data distribution; the generation of the antagonistic network is further generated by deep learning of the generator and the discriminator, and in order to ensure the stability of network training, least square loss is used as a loss function of the generation of the antagonistic network, so that the problem of gradient disappearance in the network training process can be effectively avoided, and the network is easier to converge and more stable. The loss function of the least squares generated impedance network is defined as:
wherein, minVLSGAN(G) To the least-squares loss function of the generator, minVLSGAN(D) Least squares loss function as a discriminant, IMRRepresenting a magnetic resonance image, G (I)MR) Representing a composite ultrasound image, IUSFor a true ultrasound image, a and b are labels of the generated data and the true data, respectively, and c represents a label of the data that the generator and the discriminator consider false. In the training process, we use the rule of 0-1 coding to set c-b-1 and a-0, so the loss function is set as:
the total loss of the generator in the network is the sum of the least square loss of the generator in the generation countermeasure network and the L1 loss of the synthesized ultrasonic image generated by the generator relative to the real ultrasonic image, the total loss of the judger in the network is the sum of the least square loss of the global judger, the first local judger and the second local judger in the generation countermeasure network, therefore, the total loss function loss of the network is:
where minL (G) is the overall loss function of the generators in the network, and minL (D) is the overall loss function of the generators in the network.
Therefore, the training process for generating the countermeasure network is to obtain an L1 loss function of the real ultrasound image according to the output result of the generator and the corresponding real ultrasound image; based on the least square loss function of a generator and a discriminator in the generated countermeasure network, obtaining the total loss function of the generator according to the output result of the discriminator and the L1 loss function of the real ultrasonic image, and obtaining the total loss function of the discriminator according to the output result of the discriminator; and respectively updating parameters in the network structures of the arbiter and the generator according to the total loss function of the arbiter and the total loss function of the generator until the generation of the confrontation network convergence.
In training the network, different learning rates may be set for the generator and the arbiter in order to balance the training speeds of both the generator and the arbiter.
Generating an anti-type network through the deep network learning method, inputting a three-dimensional magnetic resonance image to a generator of the trained anti-type network, and then generating a synthetic ultrasonic image corresponding to the magnetic resonance image, wherein the synthetic ultrasonic image is a highly simulated ultrasonic image of a real ultrasonic image corresponding to the magnetic resonance image.
Referring to fig. 3, fig. 3 is a flowchart illustrating registration of a magnetic resonance image and a real ultrasound image according to an embodiment of the present invention, and the following explains fig. 3 in detail.
The pyramid algorithm is utilized to input a fixed image (a synthesized ultrasonic image) and a floating image (a real ultrasonic image), sampling is respectively carried out according to the ratio of 2 multiplied by 2, the pyramid layer number is set to be 3, and the pyramid layer number is divided into 3 different scales to carry out similarity measurement and deformation field calculation on the fixed image and the floating image of each layer.
Starting from the lowest layer, the deformation field parameters are initialized and the deformation field is superimposed on the floating image.
The size of the neighborhood descriptor, the Modality Index Neighboring Descriptor (MIND), of the layer of fixed images and floating images is calculated, respectively. MIND calculation is defined as:
wherein n is a normalized vector, R belongs to R as a search area, and the distance Dp(I,x1,x2)=∑p∈P(I(x1+p)-I(x2+p))2All patches within the search region R and in two voxels x are calculated1And x2The difference of the Sum of Squares (SSD) of all voxels within the central patch.
Calculating the difference between MIND of the fixed image and MIND of the floating image by using the SSD, and obtaining the similarity measure of the two images.
And optimizing the energy function by using a Gauss-Newton gradient descent method to obtain the optimal deformation field of the layer, wherein the optimization function is as follows:
wherein u ═(u,v,w)TRepresenting the deformation field.
After the optimal deformation field of the layer is obtained, the calculated deformation field is superposed to the upper layer of the pyramid to be used as the deformation parameters of the layer, and the calculation is continued to the upper layer of the pyramid until the optimal deformation field of the pyramid at the uppermost layer is obtained, wherein the optimal deformation field is the final registration parameter; and (4) directly acting the registration parameters obtained by registering the synthesized ultrasonic image and the real ultrasonic image on the magnetic resonance image, namely finishing the registration fusion of the final magnetic resonance image and the ultrasonic image.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.