CN115457020B - 2D medical image registration method fusing residual image information - Google Patents

2D medical image registration method fusing residual image information Download PDF

Info

Publication number
CN115457020B
CN115457020B CN202211201681.9A CN202211201681A CN115457020B CN 115457020 B CN115457020 B CN 115457020B CN 202211201681 A CN202211201681 A CN 202211201681A CN 115457020 B CN115457020 B CN 115457020B
Authority
CN
China
Prior art keywords
image
registration
coarse
registered
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211201681.9A
Other languages
Chinese (zh)
Other versions
CN115457020A (en
Inventor
丁熠
卜君健
曹明生
邓伏虎
赵洋
周尔强
秦臻
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Electronic Science and Technology of China
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN202211201681.9A priority Critical patent/CN115457020B/en
Publication of CN115457020A publication Critical patent/CN115457020A/en
Application granted granted Critical
Publication of CN115457020B publication Critical patent/CN115457020B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a 2D medical image registration method fusing residual image information, which relates to the technical field of medical images and comprises the following steps: constructing a medical image registration model; training a model; application of models in image registration. According to the method, the residual image is introduced when the MSE similarity is calculated, the local pixel information is effectively fused, and the problems of pixel dislocation, deformation and folding are solved; based on the locality of the convolutional neural network and the global of the multi-head attention mechanism in Vision Transformer, a basic registration network is designed, jump connection of fusion residual image information is innovatively used in the basic registration network, the problem that MSE only calculates pixel values and feature matching between pixels cannot be found accurately is solved, and the generalization performance of a registration model is improved effectively; the multi-resolution progressive registration strategy is provided, the accuracy of registration is improved, and the topology maintainability is enhanced in the deformation process.

Description

2D medical image registration method fusing residual image information
Technical Field
The invention relates to the technical field of medical images, in particular to a 2D medical image registration method for fusing residual image information.
Background
Image registration has numerous applications of practical value in medical image processing and analysis. With the advancement of medical imaging devices, images containing accurate anatomical information such as CT, MRI can be acquired for the same patient; at the same time, images containing functional information such as SPECT can also be acquired. However, diagnosis by observing different images requires spatial imagination and subjective experience of a doctor. By adopting a correct image registration method, various information can be accurately fused into the same image, so that doctors can observe focuses and structures from various angles more conveniently and accurately. Meanwhile, through the registration of dynamic images acquired at different moments, the change conditions of focuses and organs can be quantitatively analyzed, so that medical diagnosis, operation planning and radiotherapy planning are more accurate and reliable.
1. Traditional iterative optimization algorithm
Traditional registration algorithms such as demons, syN.
Demons regards the image contour to be aligned as a diffusion model, applies a devitrification force (demon force) to each pixel point on the contour, and each contour point is subjected to displacement diffusion under the action of the devitrification force. The devil force of each point is calculated according to the numerical gradient of the pixels around the image, and the disadvantage is that the matching of the semantic features of the image is lacking between the aligned pixels.
The SyN method extends the Lagrangian type two registration technique described by Avants et al (2006 a). This new formula has the symmetry properties required for registering the two images I and J, guaranteeing symmetry in the space of the differential transformation, regardless of the similarity measure chosen. Symmetrical differential embryo-like mapping ensures the properties inherent in both registrations: 1) The registration from i to j and the registration from j to i should not be affected by similarity metrics or optimization parameters when calculating 2) the image registration needs symmetry and the registration result is not affected by human mind, i.e. the input order of the two images as "fixed images" or "moving images" is determined. Is the best method in the traditional method at present, but has the defect that symmetry in the registration process is emphasized excessively, and registration accuracy cannot be guaranteed at the same time sometimes.
One common disadvantage of the conventional methods is that: conventional registration methods solve the problem of optimizing each volume pair by aligning voxels with similar appearance while imposing constraints on the registration map, which can be computationally intensive and therefore slow in practice. For example, the most advanced algorithms running on a CPU may take tens of minutes to hours to register a pair of high precision scans.
2. Existing deep learning-based method
In order to improve the image registration efficiency, many registration methods based on deep learning have been proposed. These methods can be categorized into supervised learning and unsupervised learning according to the training patterns of the network.
In the supervised method, deformation field truth value or anatomical truth value labels are needed, and the model takes the deformation field truth value labels as optimization targets to extract features according to input image pairs. Sokooti et al propose a Convolutional Neural Network (CNN) to directly rely on artificially generated displacement vector fields to directly estimate the deformation field. Cao et al have developed a deformable inter-modality image registration method that estimates the deformation field of inter-modality image registration using a deep neural network with intra-modality similarity supervision. A disadvantage of these methods is that the registration performance of these methods depends to a large extent on the distortion field truth values that are often difficult to obtain in clinical situations.
As for the unsupervised methods, they do not require true values of the deformation field, and the model aims to minimize the image variability as an optimization objective from the input image pair. Balakrishenan et al propose a 3D medical image registration model VoxelMorph that uses CNN reconstruction registration results with spatial transformation layers. Zhao et al have designed a convolutional neural network (VTN) comprising cascaded subnetworks to recursively improve registration performance. Kim et al have proposed a cyclically identical deformable image approval method called Cyclemorph that can improve topology retention by introducing cyclic consistency loss into the network, enhancing registration performance. A disadvantage is that although these methods are improving topology retention, they still have difficulty guaranteeing preservation of image topology during registration, which often results in loss of structural information leading to incorrect registration.
Summary of the prior art:
1) The method is used for 2D image registration, and has the advantages of poor effect, low registration accuracy and poor topology maintenance. The conventional method is inefficient in registration and unstable in effect when used for 2D images. Most of the existing deep learning algorithm registration algorithms are focused on solving the problem of 3D image registration, and are used for poor effect in 2D image registration.
2) There are problems of pixel misalignment and deformation folding during registration. The existing method takes the minimized similarity loss of the image pair as a main optimization target, which can lead to the lack of characteristic matching relation among 2D image pixels and the excessive fitting of training set data, and can lead to the problems of pixel misalignment, deformation and folding.
3) The generalization performance is limited. The traditional method is independently iterated for each pair of input images, but the efficiency is low and the registration effect cannot be ensured. The existing deep learning method has poor generalization performance on images outside the data set where the training set is located.
Disclosure of Invention
The invention aims to provide a 2D medical image registration method for fusing residual image information, which can alleviate the problems.
In order to alleviate the problems, the technical scheme adopted by the invention is as follows:
the invention provides a 2D medical image registration method fusing residual image information, which comprises the following steps:
s100, constructing a medical image registration model, wherein the image registration model comprises a first stacking module C1, a second stacking module C2, a coarse registration branch network and a fine registration branch network, and the coarse registration branch network and the fine registration branch network comprise registration networks constructed based on a convolutional neural network, a flow prediction network and a multi-head attention network in a vision converter;
s200, selecting a plurality of data pairs to form a training data pair set, wherein each data pair comprises a medical training image to be registered and a reference medical training image, the medical image registration model is trained for a plurality of times by utilizing the training data pair set, each training process uses a unused data pair, and each training process specifically comprises the following steps:
s210, selecting a data pair from the training data pair set, and registering the medical training image I to be registered according to the first stacking module C1, the coarse registration branch network and the currently selected data pair moving1 And reference medical training image I fixed1 Obtaining a large displacement deformation field V coarse And coarse registration image I coarse ；
S220, according to the reference medical training image I fixed1 Coarse registration image I coarse Second stacked module C2, large displacement deformation field V coarse Medical training image I to be registered moving1 And fine registering the branch network to obtain a complete deformation field V full And final registered image I moved1 ；
S230, calculating the coarse registration image I by using MSE coarse And reference medical training image I fixed1 Similarity loss L between mse1 Using a multi-resolution residual image similarity pyramid module to register the image I according to the coarse coarse And reference medical training image I fixed1 Calculating a similarity loss L of residual images res1 The square of the L-2 norm is used as the large displacement deformation field V coarse Is a smoothness regularization loss L of (2) reg1 According to the super-parameters and the similarity loss L mse1 Similarity loss L res1 And cross entropy loss L reg1 Calculate the total loss L of the coarse registration stage total1 ；
S240, calculating the final registered image I by using MSE moved1 And reference medical training image I fixed1 Similarity loss L between mse2 Using a multi-resolution residual image similarity pyramid module to obtain a final registered image I moved1 And reference medical training image I fixed1 Calculating a similarity loss L of residual images res2 The square of the L-2 norm is used as the complete deformation field V full Is a smoothness regularization loss L of (2) reg2 According to the super-parameters and the similarity loss L mse2 Similarity loss L res2 And cross entropy loss L reg2 Calculate the total loss L of fine registration stage total2 ；
S250, utilizing the total loss L of the coarse registration stage total1 Calculating the gradient of each neuron in the coarse registration branch network, carrying out gradient feedback, and updating network parameters;
s260, utilizing the total loss L of the fine registration stage total2 Calculating the gradient of each neuron in the medical image registration model, carrying out gradient feedback, updating network parameters, and finishing the current training of the medical image registration model;
s300, medical image I to be registered moving2 And reference medical image I fixed2 Inputting the trained medical image registration model, and outputting to obtain a registered image I moved2 。
In a preferred embodiment of the present invention, the step S210 specifically includes the following steps:
s211, medical training image I to be registered moving1 And reference medical training image I fixed1 Stacking by a first stacking module C1;
s212, inputting the stacked images in the step S211 into the coarse registration branch network, and outputting to obtain a large displacement deformation field V coarse And coarse registration image I coarse 。
In a preferred embodiment of the present invention, in step S212, a large displacement deformation field V is obtained through the coarse registration branch network coarse And coarse registration image I coarse The method of (1) comprises:
downsampling the stacked images in step S211 to obtain an image I 1 ；
Image I 1 Inputting a registration network, and outputting to obtain a large displacement deformation field V;
to large displacement deformation field VUp-sampling and up-sampling the same multiplying power value and amplifying to obtain large displacement deformation field V coarse ；
Using large displacement deformation fields V coarse Medical training image I to be registered through STN module moving1 Deformed into coarse registration image I coarse 。
In a preferred embodiment of the present invention, the step S220 specifically includes the following steps:
s221, registering the rough image I coarse And reference medical training image I fixed1 Stacking by a second stacking module C2;
s222, transforming the large displacement deformation field V coarse Medical training image I to be registered moving1 And the stacked images in the step S221 are input into the fine registration branch network and output to obtain a complete deformation field V full And final registered image I moved1 。
In a preferred embodiment of the present invention, in step S222, the complete deformation field V is obtained through the fine registration branch network full And final registered image I moved1 The method of (1) comprises:
inputting the stacked images in step S221 into a registration network, and outputting to obtain a fine deformation field V fine ；
Deformation field V with large displacement coarse And a fine deformation field V fine The complete deformation field V is obtained after combination full ；
Using the complete deformation field V full Medical training image I to be registered through STN module moving1 Deformed into final registered image I moved1 。
In a preferred embodiment of the present invention, the method for acquiring a deformation field by the registration network through the stacked images includes:
extracting the stacked image I by a convolution patch embedding layer input Is a local feature information of (1);
position embedding is carried out on the local characteristic information by using a position code which can be learned;
inputting the local feature information after position embedding into a multi-head attention network, matching global features, and outputting to obtain a feature matrix;
convolving and upsampling a feature matrix output by a multi-head attention network;
for the stacked images I input Downsampling and upsampling are sequentially performed to obtain an image
Image is formedWith the original image I input Calculating residual to obtain residual image->
Residual image utilization by convolution blockObtaining a probability value mask;
multiplying the feature matrix after convolution up-sampling with the probability value mask to obtain a new matrix;
and obtaining a deformation field according to the new matrix by using a stream prediction network.
In a preferred embodiment of the invention, when a medical image I to be registered is acquired moving2 After the true value to be registered of the region of interest in the image is obtained, the STN module is utilized to act on the true value to be registered through using the deformation field obtained by the output of the registration network to obtain the medical image I to be registered moving2 Registration truth value of the region of interest in the image is a medical image I to be registered moving2 True value tags for regions of interest.
In a preferred embodiment of the present invention, in step S230, the total loss L of the coarse registration stage total1 And total loss of fine registration stage L total2 The calculation formulas of (a) are respectively as follows:
L total1 ＝αL mse1 +βL reg1 +γL res1
L total2 ＝αL mse2 +βL reg2 +γL res2
where α, β, γ are hyper-parameters for adjusting the specific gravity of each loss.
In a preferred embodiment of the present invention, in the training process, the formula for calculating the similarity loss of the residual image by using the multi-resolution residual image similarity pyramid module is as follows:
wherein L is res For similarity loss of residual images, i represents pyramid level number, K is pyramid total layer number, F i Reference medical image input for pyramid ith layer, F i r To F pair i Image obtained by downsampling and upsampling in sequence, M i For the image to be registered input to the ith layer of the pyramid,for M i Sequentially performing downsampling and upsampling to obtain an image; for the layers behind the first layer of the pyramid, the input image is the image obtained after downsampling of the layer above the pyramid.
Compared with the prior art, the invention has the beneficial effects that:
aiming at the problems that NCC and NMI similarity measurement commonly used for 3D registration is not good in performance on a 2D image, and pixel characteristics between image pairs cannot be matched accurately by MSE measurement with the best performance on the 2D image, so that pixels are out of position and deformed and folded, the invention provides a residual image pyramid module, wherein a residual image is introduced when MSE similarity is calculated, local pixel information is effectively fused, and the problems of pixel out of position and deformed and folded are solved;
the invention designs a basic registration network by utilizing the locality of a convolutional neural network and the global nature of a multi-head attention mechanism in Vision Transformer (a visual converter is abbreviated as ViT), creatively uses jump connection (an operation process in a dotted line frame in fig. 4) of fused residual image information in the basic registration network, solves the problem that MSE only calculates pixel values and cannot accurately find feature matching among pixels, and effectively improves the generalization performance of a registration model;
aiming at the problems that the existing image registration method is poor in effect when used for 2D image registration, low in registration accuracy and poor in topology maintenance, the invention provides a multi-resolution progressive registration strategy, coarse registration is firstly carried out on coarse resolution images to predict a large displacement field, coarse registration is favorable for solving complex dislocation between input image pairs in the subsequent stage, so that the complexity of the problem of the subsequent high stage is reduced, fine small displacement fields are predicted on a full resolution image and a reference image after coarse registration, and finally the large displacement field and the small displacement fields are overlapped to form a complete displacement field, so that the registration accuracy is improved, and the topology maintenance is enhanced in the deformation process.
In order to make the above objects, features and advantages of the present invention more comprehensible, embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a general flow chart of a medical image registration method according to the present invention;
FIG. 2 is a flow chart of each training of the medical image registration model according to the present invention;
FIG. 3 is a schematic diagram of the architecture of a medical image registration model according to the present invention;
FIG. 4 is a schematic diagram of the architecture and extension of the registration network according to the present invention;
fig. 5 is a diagram illustrating an example of the structure of the multi-resolution residual image similarity pyramid module according to the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, 2 and 3, the invention discloses a 2D medical image registration method for fusing residual image information, the basic idea is to register an image to be registered to a reference image aiming at an image registration task, and to use a deep neural network to fit and solve an optimization function of a deformation field for aligning the image to be registered to the reference image.
The method is a non-rigid registration model of an unsupervised 2D medical image based on learning, the learning means that the model needs to be trained on a certain amount of data sets (such as a Stanford echo Net echocardiogram data set, a CAMUS heart ultrasound data set and an ACDC heart nuclear magnetic data set), and the unsupervised means that the method does not need to be supervised by true values of deformation fields in the training process.
The invention comprises the following steps:
s100, constructing a medical image registration model, wherein the image registration model comprises a first stacking module C1, a second stacking module C2, a coarse registration branch network and a fine registration branch network, and the coarse registration branch network and the fine registration branch network comprise registration networks constructed based on a convolutional neural network, a flow prediction network and a multi-head attention network in a vision converter. Wherein the stream prediction network is formed by a convolution block.
The invention designs a registration network by utilizing the locality of a convolutional neural network and the global property of ViT, and designs a whole medical image registration model by utilizing a two-stage registration strategy from thick to thin so as to improve the accuracy of registration and enhance the topology maintenance in the deformation process. The constructed medical image registration model architecture is shown in fig. 3. The structure within the dashed box in fig. 4, and the embedding of the convolution patch into the stream prediction module, constitute a registration network.
S200, selecting a plurality of data pairs to form a training data pair set, wherein each data pair comprises a medical training image to be registered and a reference medical training image, training the medical image registration model for a plurality of times by utilizing the training data pair set, using an unused data pair in each training process, and each training process is shown in fig. 2 and specifically comprises the following steps:
s210, selecting a data pair from the training data pair set, and registering the medical training image I to be registered according to the first stacking module C1, the coarse registration branch network and the currently selected data pair moving1 And reference medical training image I fixed1 Obtaining a large displacement deformation field V coarse And coarse registration image I coarse The specific process comprises the following steps:
s211, medical training image I to be registered moving1 And reference medical training image I fixed1 Stacking is performed by the first stacking module C1.
S212, inputting the stacked images in the step S211 into a coarse registration branch network, and outputting to obtain a large displacement deformation field V coarse And coarse registration image I coarse The method specifically comprises the following steps:
downsampling the stacked images in step S211 to obtain an image I 1 ；
Image I 1 Inputting a registration network, and outputting to obtain a large displacement deformation field V;
upsampling the large displacement deformation field V and amplifying the upsampled same-multiplying power value to obtain the large displacement deformation field V coarse ；
Using large displacement deformation fields V coarse Medical training image I to be registered is distorted by an STN module (spatial warping module for applying a deformation field to the image) moving1 Deformed into coarse registration image I coarse (Coarse Moved in FIG. 3).
In the present invention, as shown in fig. 3 and 4, the registration network passes through the stacked image I 1 The method for acquiring the large displacement deformation field V specifically comprises the following steps:
extracting the stacked image I by a convolution patch embedding layer 1 Is a local feature information of (1);
position embedding is carried out on the local characteristic information by using a position code which can be learned;
inputting the local feature information after position embedding into a multi-head attention network, matching global features, and outputting to obtain a feature matrix;
convolving and upsampling a feature matrix output by a multi-head attention network;
for the stacked images I 1 Downsampling and upsampling are sequentially performed to obtain an image
Image is formedWith the original image I 1 Calculating residual to obtain residual image->
Residual image utilization by convolution blockObtaining a probability value mask, namely a logic mask in fig. 4;
multiplying the feature matrix after convolution up-sampling with the probability value mask to obtain a new matrix;
and obtaining a large displacement deformation field V according to the new matrix by using a flow prediction network.
S220, training the image I according to the reference medicine fixed1 Coarse registration image I coarse Second stacked module C2, large displacement deformation field V coarse Medical training image I to be registered moving1 And fine registering the branch network to obtain a complete deformation field V full And final registered image I moved1 The specific process comprises the following steps:
s221, registering the rough image I coarse And reference medical training image I fixed1 Stacking is performed by the second stacking module C2.
S222, transforming the large displacement deformation field V coarse Medical training image I to be registered moving1 And step S221, inputting the stacked images into a fine registration branch network, and outputting to obtain a complete deformation field V full And final registered image I moved1 The method specifically comprises the following steps:
inputting the stacked images in step S221 into a registration network, and outputting to obtain a fine deformation field V fine ；
Deformation field V with large displacement coarse And a fine deformation field V fine The complete deformation field V is obtained after combination full The calculation formula is
Wherein, as the warp operation by the STN module;
using the complete deformation field V full Medical training image I to be registered through STN module moving1 Deformed into final registered image I moved1 。
In the present invention, as shown in fig. 3 and 4, the registration network acquires the fine deformation field V from the stacked images in step S221 fine The method of (1) comprises the following steps:
extracting local feature information of the images stacked in the step S221 through the convolution patch embedding layer;
position embedding is carried out on the local characteristic information by using a position code which can be learned;
inputting the local feature information after position embedding into a multi-head attention network, matching global features, and outputting to obtain a feature matrix;
convolving and upsampling a feature matrix output by a multi-head attention network;
downsampling and upsampling the stacked images in step S221 in order to obtain an image
Image is formedResidual error is calculated with the stacked images in the original step S221 to obtain residual error image +.>
Residual image utilization by convolution blockObtaining a probability value mask, namely a logic mask in fig. 4;
multiplying the feature matrix (the feature matrix after the size is recovered) after the convolution up-sampling with a probability value mask to obtain a new matrix;
obtaining a fine deformation field V according to a new matrix by using a stream prediction network fine 。
S230, calculating a coarse registration image I by using MSE coarse And reference medical training image I fixed1 Similarity loss L between mse1 Using a multi-resolution residual image similarity pyramid module, registering image I from coarse coarse And reference medical training image I fixed1 Calculating a similarity loss L of residual images res1 The square of the L-2 norm is used as the large displacement deformation field V coarse Is a smoothness regularization loss L of (2) reg1 According to the super-parameters and the similarity loss L mse1 Similarity loss L res1 And cross entropy loss L reg1 Calculate the total loss L of the coarse registration stage total1 The calculation formula is as follows:
L total1 ＝αL mse1 +βL reg1 +γL res1
wherein, alpha, beta and gamma are manually adjusted parameters according to manual experience, and a plurality of attempts are made to find the most suitable super parameters for adjusting the specific gravity of each loss.
S240, calculating final registered image I by using MSE moved1 And reference medical training image I fixed1 Similarity loss L between mse2 Using a multi-resolution residual image similarity pyramid module, according to the final registered image I moved1 And reference medical training image I fixed1 Calculating a similarity loss L of residual images res2 The square of the L-2 norm is used as the complete deformation field V full Is a smoothness regularization loss L of (2) reg2 According to the super-parameters and the similarity loss L mse2 Similarity loss L res2 And cross entropy loss L reg2 In the present invention, the total loss L of the fine registration stage is calculated total2 The calculation formula is as follows:
L total2 ＝αL mse2 +βL reg2 +γL res2
where α, β, γ are hyper-parameters for adjusting the specific gravity of each loss.
In the present invention, both downsampling and upsampling are achieved using quadratic linear interpolation, using a downsampling ratio of 0.5 and an upsampling ratio of 2. Each pixel after up-sampling and restoring the original resolution aggregates the information of adjacent pixels, so that the target model can pay more attention to the residual texture information of the image pairs and eliminate the possible brightness difference between the image pairs.
In the invention, the calculation of introducing the residual image into the MSE similarity applies an implicit regularization to the solved target model, prevents the model from being over fitted on the training set, and enhances the generalization of the model.
In steps S230 and S240, the method for calculating the similarity loss of the residual image using the multi-resolution residual image similarity pyramid module is the same, as shown in the following formula:
wherein L is res For similarity loss of residual images, i represents pyramid level number, K is pyramid total layer number, F i Reference medical image input for pyramid ith layer, F i r To F pair i Image obtained by downsampling and upsampling in sequence, M i For the image to be registered input to the ith layer of the pyramid,for M i Sequentially performing downsampling and upsampling to obtain an image; for the layers behind the first layer of the pyramid, the input image is the image obtained after downsampling of the layer above the pyramid.
In step S230, the image I is coarsely registered coarse Namely M 1 For the input image of the first layer of the pyramid corresponding to the image to be registered, reference is made to the medical training image I fixed1 Namely F 1 Is the input image of the first layer of the pyramid corresponding to the reference image.
In step S240, final post-registration image I moved1 Namely M 1 For the input image of the first layer of the pyramid corresponding to the image to be registered, reference is made to the medical training image I fixed1 Namely F 1 Is the input image of the first layer of the pyramid corresponding to the reference image.
The multi-resolution residual image similarity pyramid module shown in fig. 5 is 3 layers, i.e. the total number of pyramid layers k=3, and so on, and there may be more layers according to the actual accuracy requirement.
S250, utilizing the total loss L of the coarse registration stage total1 Calculating the gradient of each neuron in the coarse registration branch network, carrying out gradient feedback, and updating network parameters;
s260, utilizing the total loss L of the fine registration stage total2 Calculating gradients for each neuron in the whole medical image registration model network, carrying out gradient feedback, updating network parameters and completing medical image registrationTraining a model;
s250, utilizing the total loss L of the coarse registration stage total1 The gradient of each neuron in the rough registration branch network is calculated, gradient return is carried out, and network parameters are updated, wherein the specific process is as follows:
in the coarse registration stage, our coarse registration branch network is abstracted into an expression:
I coarse ＝f coarse (concat(I moving1 ,I fixed1 ))，
wherein f coarse The function represents the role of the coarse registration branch, including the trainable parameters contained in the convolutional layers, the full-join layers, contained in each module, the concat, which is the C1 stacking module in fig. 3.
Gradient of each parameter in coarse registration branch networkAs shown in the expression:
wherein the method comprises the steps ofRepresents f coarse Then we update the parameters of the network using Adam optimizer: />
S260, utilizing the total loss L of the fine registration stage total2 And (3) calculating gradients of each neuron in the medical image registration model, carrying out gradient feedback, updating network parameters, and finishing the current training of the medical image registration model, wherein the method comprises the following steps of:
in the fine registration stage, our fine registration branch network is abstracted into an expression:
I moved1 ＝f fine (concat(I coarse ,I fixed1 ))，
wherein f fine The function represents the role of the fine registration branch, including the trainable parameters contained in the convolutional layers, the full-join layers, contained in each module, here the concat, the C2 stacking module in fig. 3.
Gradient of each parameter in fine registration branch networkAs shown in the expression:
wherein the method comprises the steps ofRepresents f fine Is provided.
Due to I coarse From the coarse registration branch we can further derive the gradient of each parameter in the coarse registration branchAs shown in the expression:
wherein the method comprises the steps ofRepresents f coarse Is used to determine the parameters of the trainable parameters,
we then update the parameters of the network using Adam optimizer:
s300, medical image I to be registered moving2 And reference medical training image I fixed2 Inputting the trained medical image registrationOutputting the model to obtain registered image I moved2 Complete deformation field V of intermediate result full2 。
In the present invention, referring to fig. 4, when a medical image I to be registered is acquired moving2 After the true value to be registered of the region of interest in the image is obtained, the STN module is utilized to act on the true value to be registered through using the deformation field obtained by the output of the registration network to obtain the medical image I to be registered moving2 Registration truth value of the region of interest in the image is medical image I to be registered moving2 True value tags for regions of interest.
Experiments on three data sets, namely a Stanford echo Net echocardiogram data set, a CAMUS heart ultrasound data set and an ACDC heart nuclear magnetic data set, of the medical image registration model obtained through training of the invention show that the method is superior to the existing registration method based on learning in the aspects of 2D medical image registration accuracy, robustness and generalization, and the operation time advantage of the learning-based method is reserved. Compared with the prior art, the method has the advantages that:
1) The method is a first 2D image special deep learning registration algorithm aiming at the 2D image, uses a registration strategy from coarse registration to fine registration, and shows good registration effect;
2) A basic registration network is innovatively designed by combining the locality of CNN and the global novelty of ViT, and meanwhile, jump connection of fusion residual images is innovatively added, so that the problem of deformation folding is effectively reduced, and the generalization of registration is improved;
3) The residual image pyramid similarity module is innovatively provided, and the problem that the MSE measurement lacks feature matching relations between pixels on a 2D image is effectively solved.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (8)

1. A 2D medical image registration method fusing residual image information, comprising the steps of:
s100, constructing a medical image registration model, wherein the image registration model comprises a first stacking module C1, a second stacking module C2, a coarse registration branch network and a fine registration branch network, and the coarse registration branch network and the fine registration branch network comprise registration networks constructed based on a convolutional neural network, a flow prediction network and a multi-head attention network in a vision converter;
s200, selecting a plurality of data pairs to form a training data pair set, wherein each data pair comprises a medical training image to be registered and a reference medical training image, the medical image registration model is trained for a plurality of times by utilizing the training data pair set, each training process uses a unused data pair, and each training process specifically comprises the following steps:
s210, selecting a data pair from the training data pair set, and registering the medical training image I to be registered according to the first stacking module C1, the coarse registration branch network and the currently selected data pair moving1 And reference medical training image I fixed1 Obtaining a large displacement deformation field V coarse And coarse registration image I coarse ；
S220, according to the reference medical training image I fixed1 Coarse registration image I coarse Second stacked module C2, large displacement deformation field V coarse Medical training image I to be registered moving1 And fine registering the branch network to obtain a complete deformation field V full And final registered image I moved1 ；
S230, calculating the coarse registration image I by using MSE coarse And reference medical training image I fixed1 Similarity loss L between mse1 Using a multi-resolution residual image similarity pyramid module to register the image I according to the coarse coarse And reference medical training image I fixed1 Calculating a similarity loss L of residual images res1 The square of the L-2 norm is used as the large displacement deformation field V coarse Is positive in smoothness of (2)Then change loss L reg1 According to the super-parameters and the similarity loss L mse1 Similarity loss L res1 And cross entropy loss L reg1 Calculate the total loss L of the coarse registration stage total1 ；
S240, calculating the final registered image I by using MSE moved1 And reference medical training image I fixed1 Similarity loss L between mse2 Using a multi-resolution residual image similarity pyramid module to obtain a final registered image I moved1 And reference medical training image I fixed1 Calculating a similarity loss L of residual images res2 The square of the L-2 norm is used as the complete deformation field V full Is a smoothness regularization loss L of (2) reg2 According to the super-parameters and the similarity loss L mse2 Similarity loss L res2 And cross entropy loss L reg2 Calculate the total loss L of fine registration stage total2 ；
S250, utilizing the total loss L of the coarse registration stage total1 Calculating the gradient of each neuron in the coarse registration branch network, carrying out gradient feedback, and updating network parameters;
s260, utilizing the total loss L of the fine registration stage total2 Calculating the gradient of each neuron in the medical image registration model, carrying out gradient feedback, updating network parameters, and finishing the current training of the medical image registration model;
s300, medical image I to be registered moving2 And reference medical training image I fixed2 Inputting the trained medical image registration model, and outputting to obtain a registered image I moved2 ；
The formula for calculating the similarity loss of the residual image by adopting the multi-resolution residual image similarity pyramid module is as follows:
wherein L is res For similarity loss of residual images, i represents pyramid level number, and K is gold wordTotal number of layers of tower, F i Reference medical image input for pyramid ith layer, F i r To F pair i Image obtained by downsampling and upsampling in sequence, M i For the image to be registered input to the ith layer of the pyramid,for M i Sequentially performing downsampling and upsampling to obtain an image; for the layers behind the first layer of the pyramid, the input image is the image obtained after downsampling of the layer above the pyramid.
2. The method according to claim 1, wherein step S210 specifically comprises the steps of:
s211, medical training image I to be registered moving1 And reference medical training image I fixed1 Stacking by a first stacking module C1;
s212, inputting the stacked images in the step S211 into the coarse registration branch network, and outputting to obtain a large displacement deformation field V coarse And coarse registration image I coarse 。
3. The method according to claim 2, wherein in step S212, a large displacement deformation field V is acquired through the coarse registration branch network coarse And coarse registration image I coarse The method of (1) comprises:
downsampling the stacked images in step S211 to obtain an image I 1 ；
Image I 1 Inputting a registration network, and outputting to obtain a large displacement deformation field V;
upsampling the large displacement deformation field V and amplifying the value with the same multiplying power as the upsampling to obtain the large displacement deformation field V coarse ；
Using large displacement deformation fields V coarse Medical training image I to be registered through STN module moving1 Deformed into coarse registration image I coarse 。
4. A method according to claim 3, wherein step S220 comprises the steps of:
s221, registering the rough image I coarse And reference medical training image I fixed1 Stacking by a second stacking module C2;
s222, transforming the large displacement deformation field V coarse Medical training image I to be registered moving1 And the stacked images in the step S221 are input into the fine registration branch network and output to obtain a complete deformation field V full And final registered image I moved1 。
5. The method according to claim 4, wherein in step S222, a complete deformation field V is acquired through the fine registration branch network full And final registered image I moved1 The method of (1) comprises:
inputting the stacked images in step S221 into a registration network, and outputting to obtain a fine deformation field V fine ；
Deformation field V with large displacement coarse And a fine deformation field V fine The complete deformation field V is obtained after combination full ；
Using the complete deformation field V full Medical training image I to be registered through STN module moving1 Deformed into final registered image I moved1 。
6. The method of claim 5, wherein the method of the registration network to acquire a deformation field from the stacked images comprises:
extracting the stacked image I by a convolution patch embedding layer input Is a local feature information of (1);
position embedding is carried out on the local characteristic information by using a position code which can be learned;
inputting the local feature information after position embedding into a multi-head attention network, matching global features, and outputting to obtain a feature matrix;
convolving and upsampling a feature matrix output by a multi-head attention network;
for the stacked images I input Downsampling and upsampling are sequentially performed to obtain an image
Image is formedWith the original image I input Calculating residual to obtain residual image->
Residual image utilization by convolution blockObtaining a probability value mask;
multiplying the feature matrix after convolution up-sampling with the probability value mask to obtain a new matrix;
and obtaining a deformation field according to the new matrix by using a stream prediction network.
7. The method according to claim 6, wherein when a medical image I to be registered is acquired moving2 After the true value to be registered of the region of interest in the image is obtained, the STN module is utilized to act on the true value to be registered through using the deformation field obtained by the output of the registration network to obtain the medical image I to be registered moving2 Registration truth value of the region of interest in the image is medical image I to be registered moving2 True value tags for regions of interest.
8. The method according to claim 6, wherein in step S230, the coarse registration stage total loss L total1 And total loss of fine registration stage L total2 The calculation formulas of (a) are respectively as follows:
L total1 ＝αL mse1 +βL reg1 +γL res1
L total2 ＝αL mse2 +βL reg2 +γL res2
where α, β, γ are hyper-parameters for adjusting the specific gravity of each loss.
CN202211201681.9A 2022-09-29 2022-09-29 2D medical image registration method fusing residual image information Active CN115457020B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211201681.9A CN115457020B (en) 2022-09-29 2022-09-29 2D medical image registration method fusing residual image information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211201681.9A CN115457020B (en) 2022-09-29 2022-09-29 2D medical image registration method fusing residual image information

Publications (2)

Publication Number Publication Date
CN115457020A CN115457020A (en) 2022-12-09
CN115457020B true CN115457020B (en) 2023-12-26

Family

ID=84308126

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211201681.9A Active CN115457020B (en) 2022-09-29 2022-09-29 2D medical image registration method fusing residual image information

Country Status (1)

Country Link
CN (1) CN115457020B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116402865B (en) * 2023-06-06 2023-09-15 之江实验室 Multi-mode image registration method, device and medium using diffusion model
CN117474963B (en) * 2023-10-18 2024-04-19 南京国础科学技术研究院有限公司 Multi-source satellite image registration method, system, storage medium and electronic equipment
CN117935060B (en) * 2024-03-21 2024-05-28 成都信息工程大学 Flood area detection method based on deep learning

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113870327A (en) * 2021-09-18 2021-12-31 大连理工大学 Medical image registration method based on multi-level deformation field prediction
CN114373004A (en) * 2022-01-13 2022-04-19 强联智创（北京）科技有限公司 Unsupervised three-dimensional image rigid registration method based on dynamic cascade network
CN114820733A (en) * 2022-04-21 2022-07-29 北京航空航天大学 Interpretable thermal infrared visible light image registration method and system
WO2022193750A1 (en) * 2021-03-16 2022-09-22 深圳先进技术研究院 Breast image registration method based on deep learning

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022193750A1 (en) * 2021-03-16 2022-09-22 深圳先进技术研究院 Breast image registration method based on deep learning
CN113870327A (en) * 2021-09-18 2021-12-31 大连理工大学 Medical image registration method based on multi-level deformation field prediction
CN114373004A (en) * 2022-01-13 2022-04-19 强联智创（北京）科技有限公司 Unsupervised three-dimensional image rigid registration method based on dynamic cascade network
CN114820733A (en) * 2022-04-21 2022-07-29 北京航空航天大学 Interpretable thermal infrared visible light image registration method and system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
《A Coarse-to-Fine Deformable Transformation Framework for Unsupervised Multi-Contrast MR Image Registration With Dual Consistency Constraint》;Weijian Huang 等;《IEEE Transactions on Medical Imaging》;第40卷(第10期);2589 - 2599 *
《基于深度学习的仰卧位和俯卧位的乳腺图像配准》;欧阳效芸;《中国优秀硕士学位论文全文数据库》;1-58 *
《基于深度学习的医学图像配准与分割技术研究》;王烈基;《中国优秀硕士学位论文全文数据库》;1-69 *

Also Published As

Publication number Publication date
CN115457020A (en) 2022-12-09

Similar Documents

Publication Publication Date Title
CN115457020B (en) 2D medical image registration method fusing residual image information
Fan et al. BIRNet: Brain image registration using dual-supervised fully convolutional networks
Mahapatra et al. Training data independent image registration using generative adversarial networks and domain adaptation
CN110363802B (en) Prostate image registration system and method based on automatic segmentation and pelvis alignment
CN111524170B (en) Pulmonary CT image registration method based on unsupervised deep learning
CN112465827A (en) Contour perception multi-organ segmentation network construction method based on class-by-class convolution operation
CN113298855B (en) Image registration method based on automatic delineation
WO2024021523A1 (en) Graph network-based method and system for fully automatic segmentation of cerebral cortex surface
He et al. Few-shot learning for deformable medical image registration with perception-correspondence decoupling and reverse teaching
Hering et al. Memory-efficient 2.5 D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans
Sokooti et al. Hierarchical prediction of registration misalignment using a convolutional LSTM: Application to chest CT scans
CN115512110A (en) Medical image tumor segmentation method related to cross-modal attention mechanism
Ahmad et al. Surface-constrained volumetric registration for the early developing brain
CN113269774B (en) Parkinson disease classification and lesion region labeling method of MRI (magnetic resonance imaging) image
Zheng et al. Multi-strategy mutual learning network for deformable medical image registration
Shi et al. Dual dense context-aware network for hippocampal segmentation
Hu et al. Unsupervised computed tomography and cone-beam computed tomography image registration using a dual attention network
CN112990359B (en) Image data processing method, device, computer and storage medium
CN117333750A (en) Spatial registration and local global multi-scale multi-modal medical image fusion method
Chang et al. Cascading affine and B-spline registration method for large deformation registration of lung X-rays
Ou et al. HCS-Net: Multi-level deformation strategy combined with quadruple attention for image registration
Young et al. Supervision by Denoising for Medical Image Segmentation
Fourcade et al. Deformable image registration with deep network priors: a study on longitudinal PET images
Ellingsen et al. Robust deformable image registration using prior shape information for atlas to patient registration
Zheng et al. SurfNN: Joint reconstruction of multiple cortical surfaces from magnetic resonance images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant