CN111599448A - Multi-view shape constraint system and method for specific coronary artery calcification analysis - Google Patents
Multi-view shape constraint system and method for specific coronary artery calcification analysis Download PDFInfo
- Publication number
- CN111599448A CN111599448A CN202010533043.1A CN202010533043A CN111599448A CN 111599448 A CN111599448 A CN 111599448A CN 202010533043 A CN202010533043 A CN 202010533043A CN 111599448 A CN111599448 A CN 111599448A
- Authority
- CN
- China
- Prior art keywords
- unit
- slice
- view
- expansion
- unet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 21
- 238000004458 analytical method Methods 0.000 title claims description 19
- 206010003211 Arteriosclerosis coronary artery Diseases 0.000 title description 6
- 230000002308 calcification Effects 0.000 claims abstract description 39
- 230000003902 lesion Effects 0.000 claims abstract description 12
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 13
- 210000001367 artery Anatomy 0.000 claims description 12
- 230000011218 segmentation Effects 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 11
- OYPRJOBELJOOCE-UHFFFAOYSA-N Calcium Chemical compound [Ca] OYPRJOBELJOOCE-UHFFFAOYSA-N 0.000 claims description 10
- 229910052791 calcium Inorganic materials 0.000 claims description 10
- 239000011575 calcium Substances 0.000 claims description 10
- 238000010586 diagram Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 9
- 230000010339 dilation Effects 0.000 claims description 8
- 238000005070 sampling Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 4
- 230000001902 propagating effect Effects 0.000 claims description 3
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims 2
- 208000004434 Calcinosis Diseases 0.000 description 33
- 208000005475 Vascular calcification Diseases 0.000 description 7
- 238000011002 quantification Methods 0.000 description 5
- 238000012512 characterization method Methods 0.000 description 3
- 238000002591 computed tomography Methods 0.000 description 3
- 210000004351 coronary vessel Anatomy 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 230000000747 cardiac effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003278 mimic effect Effects 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 208000024172 Cardiovascular disease Diseases 0.000 description 1
- 201000000057 Coronary Stenosis Diseases 0.000 description 1
- 208000031481 Pathologic Constriction Diseases 0.000 description 1
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 208000029078 coronary artery disease Diseases 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000013077 scoring method Methods 0.000 description 1
- 208000037804 stenosis Diseases 0.000 description 1
- 230000036262 stenosis Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
- G16H10/20—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for electronic clinical trials or questionnaires
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Pathology (AREA)
- Databases & Information Systems (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Image Analysis (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
The invention discloses a multi-view shape constraint system, comprising: the slicing unit is used for slicing the received image to obtain a coronal slice, a sagittal slice and an axial slice; the first expansion Unet unit, the second expansion Unet unit and the residual expansion network RDN unit are used for respectively receiving the coronal slice, the sagittal slice and the axial slice, the first expansion Unet unit and the second expansion Unet unit are used for modeling the coronal slice and the sagittal slice and obtaining prediction graphs of the coronal view and the sagittal view as supplementary information of the calcified lesion, and the residual expansion network RDN unit is used for modeling the axial slice and outputting the prediction graph of the axial slice; the multi-view shape constraint unit integrates different view information through multi-view attention learning to generate multi-view attention characteristics; and a multitask learning unit for estimating a relevant calcification index using an output of the multiview shape constraint unit.
Description
Technical Field
The present invention relates to the field of medical image processing, and more particularly, to a multi-view shape constraint system and method for specific coronary artery calcification analysis.
Background
Specific coronary artery calcification analysis provides a comprehensive judgment of calcification in different coronary vessels. The location and extent of each coronary stenosis is revealed by simultaneous detection and direct quantification of specific arterial calcifications. The detection of calcium can capture the distribution information, geometric and structural characteristics of calcium focus, and provide higher diagnostic value. Quantitative measurement of calcification of specific arteries can provide the degree of calcification of individual arteries, which is closely related to the severity of stenosis of these vessels.
However, the problem of specific coronary calcification analysis aims at obtaining a discriminating signature representation of a multi-type of high-dimensional task. First, there is a dimensional difference between two-dimensional plane detection and one-dimensional linear quantization index. This dimensional difference results in a complex relationship of these multi-type coronary calcification indicators, increasing the difficulty of multi-type task dependent learning and commonality mining. Second, there are differences in the distribution between different calcium fractions. In particular, the argston score correlates with the peak CT intensity and area of each calcified lesion, while the mass score depends on the average CT intensity and volume of each lesion. Perspective point diversity results in distribution diversity and more complex characterization for different quantitative scores. Third, detection and quantification of Coronary Artery Calcification (CAC) embedded in high dimensional space is combined with estimation. Such intrinsic features pose difficulties for discriminative characterization of such multi-type tasks.
Currently, there are many automated methods for the quantification of coronary calcification, but they do not allow for the differential characterization of specific arterial calcification analyses due to their inherent structural limitations. These methods are only used for classification or regression of CAC, so they are not suitable for task-dependent learning and commonality mining for multi-type tasks in specific arterial calcification analysis. Existing methods cannot learn the joint distribution of multiple regression tasks because they only focus on one score at a time. In addition, due to the multi-type high-dimensional index estimation of specific artery calcification analysis, the existing multi-task learning method in other fields cannot provide satisfactory effects.
Disclosure of Invention
The present invention provides a multi-view shape constraint (MVSC) framework that we propose to explore the discriminating signature representation of automatic simultaneous detection and direct quantification of specific arterial CAC through multitask learning. First, an advanced Residual Dilation Network (RDN) module is proposed for learning CAC-expressing features for multi-type calcification tasks that seamlessly integrate residual networks and dilation convolutions. Second, calcified lesions usually occupy adjacent slices. In order to extract global semantic features and mimic the review procedure of a reporting clinician, the present invention integrates a multi-view shape constraint mechanism with RDN through residual attention learning to form a multi-view tri-encoder attention module, thereby enhancing the resolvability of RDN features. Third, a decoder model for semantic segmentation of Left Anterior Descending (LAD), Left Circumcision (LCX), Right Coronary Artery (RCA) and full heart calcification is built, as well as a regression model for specific arterial calcium scores, combined for task-dependent learning and mutual reinforcement. Artery-specific analysis can be effectively performed by combining the multi-view tri-encoder attention module and the discriminating feature representation of the multitask module.
According to an embodiment of the present invention, there is provided a multi-view shape constraint system including:
a slicing unit that slices the received image to obtain a coronal slice, a sagittal slice, and an axial slice;
the device comprises a first expansion Unet unit, a second expansion Unet unit and a residual expansion network RDN unit, wherein the first expansion Unet unit, the second expansion Unet unit and the residual expansion network RDN unit respectively receive a coronal slice, a sagittal slice and an axial slice, the first expansion Unet unit and the second expansion Unet unit model the coronal slice and the sagittal slice and obtain prediction pictures of the coronal view and the sagittal view as supplementary information of calcification lesions, and the residual expansion network RDN unit models the axial slice and outputs the prediction pictures of the axial slice;
a multi-view shape constraint unit which receives outputs of the first expansion Unet unit, the second expansion Unet unit and the residual expansion network RDN unit, integrates different view information through multi-view attention learning, and generates a multi-view attention feature; and
a multitask learning unit that estimates a relevant calcification index using an output of the multiview shape constraining unit.
In one embodiment of the invention, the multi-view shape constraint system further comprises: a down-sampling unit that down-samples the input image and supplies the down-sampled image to the slicing unit.
In one embodiment of the invention, the goal of the multi-view shape constraint system is defined as:
where t ∈ { seg, as, vs, ms } indicates a particular target task,/tA loss function representing task t, | W |2L2 normalization, which represents a network parameter.
In one embodiment of the invention, the residual dilation network RDN comprises a shrink path and an extension path,
in the systolic path, the input axial slice is coded using an extensive depth residual network ResNet with 4 residual blocks, consisting of 3, 4, 6, 3 residual blocks, each residual block having three consecutive convolutional layers with kernel sizes 1 × 1, 3 × 3, 1 × 1, each convolutional layer being followed by a batch normalization layer (BN) for normalizing the output feature map and a modified linear unit (ReLU) for improving sparsity, the last residual block replacing the conventional convolution with an expanded convolution so that the effective acceptance domain of the output feature map is larger than the original input image;
in the extended path, the dimension of the feature map is restored using an upsampling operation, followed by a concatenation with the corresponding level feature map of the contracted path and two 3 × 3 convolutions each with 16 kernels;
for a jump connection between the systolic path and the expanded path, three successive expansion convolutional layers are combined to assemble the feature map from the systolic path, rather than propagating them directly into the expanded path.
In one embodiment of the invention, the first expanded Unet unit and the second expanded Unet unit replace the hopping connection of U-Net by expanded convolution to combine the advantages of U-Net and expanded convolution, and at the end of the first expanded Unet unit and the second expanded Unet unit, their outputs are normalized to a range of [0,1] using a Sigmoid layer.
In one embodiment of the invention, the residual attention learning is defined as:
AMmv(xi,c)＝(1+Mcv(xi,c)+Msv(xi,c))*Fav(xi,c)
where i and c indicate the indices, x, of all spatial locations and channels, respectivelyi,cA feature vector indicating the ith spatial position the c-th channel; mcv(xi,c) And Msv(xi,c) Prediction maps representing coronal and sagittal slices, respectively; fav(xi,c) An output prediction graph representing the RDN; AM (amplitude modulation)mv(xi,c) Features representing multi-view attention model learning.
In one embodiment of the invention, the multitask learning unit comprises a cascade unit, a segmentation unit and a regression unit,
wherein, by the cascade unit, the image received by the previous slice unit is combined with the multi-view attention characteristic for accurate specific artery calcification detection in the segmentation unit, the convolution layer is followed by a BN layer and a ReLU layer, the number of channels is converted into 4, and the upsampling layer restores the characteristic diagram to the original image size;
in the regression unit, three fully connected layers of size 128 × 128 × 16, 64, 12 are applied, with a discard layer and addition operation and a first dimension, back and forth to the calcium score.
According to another embodiment of the present invention, there is provided a multi-view shape constraint method for specific coronary artery calcification analysis, including:
down-sampling an input image;
slicing the down-sampled image to obtain a coronal slice, a sagittal slice and an axial slice;
modeling the coronal slice and the sagittal slice through a first expansion Unet unit and a second expansion Unet unit, obtaining prediction graphs of the coronal view and the sagittal view as supplementary information of the calcified lesion, and modeling the axial slice through a residual expansion network RDN unit to obtain the prediction graph of the axial slice;
integrating the prediction maps of coronal, sagittal, and axial slices through multi-view attention learning to generate multi-view attention features; and
the output of the multi-view shape constraint unit is used by a multitask learning unit to estimate the relevant calcification index.
In another embodiment of the present invention, the goal of the multi-view shape constraint method is defined as:
where t ∈ { seg, as, vs, ms } indicates a particular target task,/tA loss function representing task t, | W |2L2 regularization representing network parameters.
In another embodiment of the invention, the residual attention learning is defined as:
AMmv(xi,c)＝(1+Mcv(xi,c)+Msv(xi,c))*Fav(xi,c)
where i and c indicate the indices, x, of all spatial locations and channels, respectivelyi,cA feature vector indicating the ith spatial position the c-th channel; mcv(xi,c) And Msv(xi,c) Prediction maps representing coronal and sagittal slices, respectively; fav(xi,c) An output prediction graph representing the RDN; AM (amplitude modulation)mv(xi,c) Features representing multi-view attention model learning.
The invention has the advantages that the specific artery calcification is directly quantified and simultaneously detected; the proposed multi-view three-encoder attention module provides a discriminating feature representation for the multitasking of specific arterial calcification analysis.
Drawings
To further clarify the above and other advantages and features of embodiments of the present invention, a more particular description of embodiments of the invention will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. In the drawings, the same or corresponding parts will be denoted by the same or similar reference numerals for clarity.
Fig. 1 shows an architecture diagram of a multi-view shape constraint (MVSC) system according to one embodiment of the invention.
Fig. 2 shows a schematic block diagram of a multi-view shape constraining unit according to an embodiment of the present invention.
FIG. 3 shows a schematic structural diagram of a multitasking learning unit according to one embodiment of the present invention.
Fig. 4 shows a flow diagram of a multi-view shape constraint method of specific coronary calcification analysis according to an embodiment of the invention.
Detailed Description
In the following description, the invention is described with reference to various embodiments. One skilled in the relevant art will recognize, however, that the embodiments may be practiced without one or more of the specific details, or with other alternative and/or additional methods, materials, or components. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of embodiments of the invention. Similarly, for purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the embodiments of the invention. However, the invention may be practiced without specific details. Further, it should be understood that the embodiments shown in the figures are illustrative representations and are not necessarily drawn to scale.
Reference in the specification to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.
Specific artery calcification analysis can provide a comprehensive assessment of the level of calcification in different coronary arteries, and is of clinical significance for the effective diagnosis of cardiovascular disease (CVD). The present invention proposes a multi-view shape constraint (MVSC) framework for detecting and quantifying specific arterial calcifications. MVSC models different input views with residual expanded networks and expanded unets for feature representation. Then, different view information is integrated through residual attention learning, so that the MVSC has strong feature recognition. In addition, MVSC combines segmentation and regression models through multitask learning to obtain efficient calcification detection and accurate calcium scores. It was found by comparison that the 232 non-contrast cardiac scan images collected from two centers were used to verify the performance of MVSCs, which showed very good performance for a variety of evaluation metrics. MVSCs enable reliable specific coronary calcification analysis.
The MVSC framework proposed by the present invention uses multi-view learning to mimic the reporting clinician's CAD review procedure, which focuses on planar information for 3D CT scans, typically by looking at axial slices, and collects ancillary information from coronal and sagittal views. The multi-view information is then integrated through a shape constraint mechanism. Finally, the segmentation model and the regression model are combined for specific artery calcification analysis based on the indivisible relationship of calcification detection and calcium score.
An overview of MVSCs is shown in fig. 1. Fig. 1 shows an architecture diagram of a multi-view shape constraint (MVSC) system according to one embodiment of the invention.
The multi-view shape constraint (MVSC) system proposed by the present invention treats specific artery calcification analysis as a multitasking learning problem. From a given CT scan image { XkIn the analysis of calcification of specific arteries, segmentation tasks are performed simultaneously
And regression tasks
Where K1 … K indicates the object index, as, vs, ms represent the argton score, the volume score and the volume score, respectively.
In one embodiment of the present invention, the goal of a multiview shape constraint (MVSC) system may be defined as follows:
where t ∈ { seg, as, vs, ms } indicates a particular target task,/tA loss function representing task t, W being a set of all parameters of the entire network, | W |2L2 regularization (i.e., sum of euclidean distances) representing network parameters; λ is the hyper-parameter of the L2 regularization term, the purpose of this hyper-parameter λ is to control the magnitude of the regularization.
The function f is used to refer to the whole network architecture, i.e. the input of the function f is XkW, minimizing the output of function f and the golden standard by optimization
The distance between them.
As shown in fig. 1, the multi-view shape constraint (MVSC) system includes a down-sampling unit 110, a slicing unit 120, a first expanded Unet unit 131, a second expanded Unet unit 132, a residual expanded network (RDN) unit 133, a multi-view shape constraint unit 140, and a multitask learning unit 150.
A multi-view shape constraint (MVSC) system receives an input image and extracts high resolution features therefrom. The input image may first be downsampled by the downsampling unit 110. For example, in one embodiment, the size of the input image is 64 × 512 × 512, and the downsampling unit 110 downsamples the input image to 64 × 128 × 128 by 7 × 7 convolution with a step size of 2 and 3 × 3 maximum pooling with a step size of 2. In other embodiments of the present invention, the down-sampling unit 110 may perform down-sampling in other manners to output down-sampled images.
The slicing unit 120 slices the downsampled image to obtain a coronal slice, a sagittal slice, and an axial slice.
The coronal, sagittal and axial slices are delivered to the first, second and RDN expansion uet units 131, 132, 133, respectively.
The axial slices are modeled by an RDN unit 133. In one embodiment of the present invention, Residual Dilation Network (RDN) unit 133 includes a shrink path and an expand path. In the systolic path, the input axial slice is encoded using an extensive depth residual network (ResNet) with 4 residual blocks. These residual blocks consist of 3, 4, 6, 3 residual blocks, respectively, each with three consecutive convolutional layers, with kernel sizes of 1 × 1, 3 × 3, 1 × 1. Each convolutional layer is followed by a batch normalization layer (BN) for normalizing the output feature map and a modified linear unit (ReLU) for improving sparsity. Furthermore, the last residual block replaces the conventional convolution with a dilation convolution (dilation rate set to 2) to make the effective acceptance domain of the output feature map larger than the original input image. Between each two adjacent residual blocks, a max-pooling operation is applied to reduce the number of dimensions, while reducing the memory requirements. In the extended path, the dimension of the feature map is restored using an upsampling operation, followed by a concatenation with the corresponding level feature map of the contracted path and two 3 × 3 convolutions each with 16 kernels.
For a jump connection between the systolic path and the expanded path, RDN unit 133 incorporates three successive expanded convolutional layers to assemble the feature map from the systolic path, rather than propagating them directly into the expanded path. Each dilated convolution contains 16 kernels of 3 x 3 with a dilation rate of 2. The expansion convolution reduces the number of channels and also enlarges the effective acceptance range of the jump connection characteristic, thereby generating the effective global semantic characteristic.
For coronal and sagittal views, the multi-view shape constraint (MVSC) system of the present invention models it using the first and second expanded Unet units 131, 132 and obtains complementary information of calcified lesions. The first expanded Unet unit 131 and the second expanded Unet unit 132 replace the hopping connection of U-Net by expanded convolution to combine the advantages of U-Net and expanded convolution. At the end of the expanded UNet, its output is normalized to a range of [0,1] using the Sigmoid layer.
The outputs of the first expanded Unet unit 131, the second expanded Unet unit 132, the RDN unit 133 enter the multi-view shape constraint unit 140. Since calcified lesions are usually small, irregular, spanning multiple axial slices; therefore, the multi-view shape constraint unit 140 employs multi-view learning and attention models with shape constraints (multi-view attention models for short) to help the models identify lesions. The prediction graph, which is used as an attention mask, is integrated with the RDN as a backbone branch to form the attention model. The mask branches may enhance the desired features and suppress noise, which may improve the distinguishability of the RDN. Thus, the attention model is defined as:
AMmv(xi,c)＝(1+Mcv(xi,c)+Msv(xi,c))*Fav(xi,c) (2)
where i and c indicate the indices, x, of all spatial locations and channels, respectivelyi,cA feature vector indicating the ith spatial position the c-th channel. Mcv(xi,c) And Msv(xi,c) The prediction maps for coronal and sagittal slices are shown, respectively. Fav(xi,c) An output prediction graph of the RDN is shown. AM (amplitude modulation)mv(xi,c) Features representing multi-view attention model learning.
Fig. 2 shows a schematic block diagram of the multi-view shape constraint unit 140 according to an embodiment of the present invention. As shown in fig. 2, the multi-view shape constraint unit includes a first addition unit 210, a multiplication unit 220, and a second addition unit 230. The first addition unit 210 adds the prediction maps output by the first expanded Unet unit 131 and the second expanded Unet unit 132, and outputs the result to the multiplication unit 220. The multiplication unit 220 multiplies the output result of the first addition unit 210 by the prediction map output by the RDN unit 133, and outputs the result to the second addition unit 230. The second addition unit 230 adds the output result of the multiplication unit 220 and the prediction map output by the RDN unit 133, and outputs the result to the multitask learning unit 150.
The multitask learning unit 150 utilizes the output of the multiview attention model in combination with the segmentation unit and the regression unit to estimate the associated calcification index. FIG. 3 shows a schematic structural diagram of a multitasking learning unit according to one embodiment of the present invention. As shown in fig. 3, the multitask learning unit includes a concatenation unit 310, a segmentation unit 320, and a regression unit 330.
The high resolution features from the previous downsampling unit 110 are combined with the multi-view attention features by the concatenation unit 310 for accurate specific arterial calcification detection in the segmentation unit 320. Next, the convolutional layer is followed by the BN layer and the ReLU layer, the number of channels is converted to 4, and the upsampling layer restores the feature map to the original image size.
In the regression unit 330, three fully connected layers of size 128 × 128 × 16, 64, 12 are applied, with a discard layer and addition operation and a first dimension, returning the calcium score. For the loss function in equation (1), the Dice loss and Mean Absolute Error (MAE) are used for the segmentation and regression tasks, respectively.
Fig. 4 shows a flow diagram of a multi-view shape constraint method of specific coronary calcification analysis according to an embodiment of the invention.
The method of fig. 4 is accomplished using the multi-view shape constraint system described in the previous embodiment.
First, in step 410, an input image is downsampled. In an embodiment of the present invention, the downsampling process is accomplished by downsampling unit 110.
Next, at step 420, the downsampled image is sliced to obtain a coronal slice, a sagittal slice, and an axial slice.
At step 430, prediction maps for coronal, sagittal, and axial slices are obtained.
In the embodiment of the invention, the first expansion Unet unit and the second expansion Unet unit are used for modeling the coronal slice and the sagittal slice and obtaining the prediction maps of the coronal view and the sagittal view as the supplementary information of the calcification lesion, and the residual expansion network RDN unit is used for modeling the axial slice to obtain the prediction map of the axial slice.
At step 440, a multi-view attention feature is obtained. The prediction maps for coronal, sagittal, and axial slices are integrated by multi-view attention learning to generate multi-view attention features.
At step 450, a relevant calcification index is obtained based on the multi-view attention feature. The output of the multi-view shape constraint unit is used by a multitask learning unit to estimate the relevant calcification index.
The present invention proposes a multi-view shape constrained framework system for simultaneous detection and quantification of specific arterial calcifications. The MVSC method of the invention integrates multi-view learning, attention model and multi-task learning seamlessly. The assessment of MVSCs disclosed in the present invention was performed in 232 non-contrast enhanced cardiac CT scans collected from two centers. Compared with other automatic calcium scoring methods, the MVSC of the invention has convincing performance. In conclusion, the MVSC provided by the invention can be used for effectively and reliably analyzing the calcification of specific arteries, so that the clinical diagnosis of coronary heart disease can be assisted.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be apparent to persons skilled in the relevant art that various combinations, modifications, and changes can be made thereto without departing from the spirit and scope of the invention. Thus, the breadth and scope of the present invention disclosed herein should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
Claims (10)
1. A multi-view shape constraint system, comprising:
a slicing unit that slices the received image to obtain a coronal slice, a sagittal slice, and an axial slice;
the device comprises a first expansion Unet unit, a second expansion Unet unit and a residual expansion network RDN unit, wherein the first expansion Unet unit, the second expansion Unet unit and the residual expansion network RDN unit respectively receive a coronal slice, a sagittal slice and an axial slice, the first expansion Unet unit and the second expansion Unet unit model the coronal slice and the sagittal slice and obtain prediction pictures of the coronal view and the sagittal view as supplementary information of calcification lesions, and the residual expansion network RDN unit models the axial slice and outputs the prediction pictures of the axial slice;
a multi-view shape constraint unit which receives outputs of the first expansion Unet unit, the second expansion Unet unit and the residual expansion network RDN unit, integrates different view information through multi-view attention learning, and generates a multi-view attention feature; and
a multitask learning unit that estimates a relevant calcification index using an output of the multiview shape constraining unit.
2. The multi-view shape constraint system of claim 1, further comprising: a down-sampling unit that down-samples the input image and supplies the down-sampled image to the slicing unit.
3. The multi-view shape constraint system of claim 1, wherein the goal of the multi-view shape constraint system is defined as:
where t ∈ { seg, as, vs, ms } indicates a particular target task,/tA loss function representing task t, W being a set of all parameters of the entire network, | W |2L2 regularization representing network parameters, λ is a super-parameter of the L2 regularization term,
is gold standard.
4. The multi-view shape constraint system of claim 1, characterized in that the residual dilation network RDN comprises a shrink path and an expand path,
in the systolic path, the input axial slice is coded using an extensive depth residual network ResNet with 4 residual blocks, consisting of 3, 4, 6, 3 residual blocks, each residual block having three consecutive convolutional layers with kernel sizes 1 × 1, 3 × 3, 1 × 1, each convolutional layer being followed by a batch normalization layer (BN) for normalizing the output feature map and a modified linear unit (ReLU) for improving sparsity, the last residual block replacing the conventional convolution with an expanded convolution so that the effective acceptance domain of the output feature map is larger than the original input image;
in the extended path, the dimension of the feature map is restored using an upsampling operation, followed by a concatenation with the corresponding level feature map of the contracted path and two 3 × 3 convolutions each with 16 kernels;
for a jump connection between the systolic path and the expanded path, three successive expansion convolutional layers are combined to assemble the feature map from the systolic path, rather than propagating them directly into the expanded path.
5. The multi-view shape constraint system of claim 1, characterized in that the first expanded Unet unit and the second expanded Unet unit replace the hopping connection of U-Net by expanded convolution to combine the advantages of U-Net and expanded convolution, at the end of which the Sigmoid layer is used to normalize their output to a range of [0,1 ].
6. The multi-view shape constraint system of claim 1, characterized in that the residual attention learning is defined as:
AMmv(xi,c)＝(1+Mcv(xi,c)+Msv(xi,c))*Fav(xi,c)
where i and c indicate the indices, x, of all spatial locations and channels, respectivelyi,cA feature vector indicating the ith spatial position the c-th channel; mcv(xi,c) And Msv(xi,c) Prediction maps representing coronal and sagittal slices, respectively; fav(xi,c) An output prediction graph representing the RDN; AM (amplitude modulation)mv(xi,c) Features representing multi-view attention model learning.
7. The multi-view shape constraint system of claim 1, characterized in that the multitasking learning unit comprises a concatenation unit, a segmentation unit and a regression unit,
wherein, by the cascade unit, the image received by the previous slice unit is combined with the multi-view attention characteristic for accurate specific artery calcification detection in the segmentation unit, the convolution layer is followed by a BN layer and a ReLU layer, the number of channels is converted into 4, and the upsampling layer restores the characteristic diagram to the original image size;
in the regression unit, three fully connected layers of size 128 × 128 × 16, 64, 12 are applied, with a discard layer and addition operation and a first dimension, back and forth to the calcium score.
8. A multi-view shape-constrained method of specific coronary calcification analysis, comprising:
down-sampling an input image;
slicing the down-sampled image to obtain a coronal slice, a sagittal slice and an axial slice;
modeling the coronal slice and the sagittal slice through a first expansion Unet unit and a second expansion Unet unit, obtaining prediction graphs of the coronal view and the sagittal view as supplementary information of the calcified lesion, and modeling the axial slice through a residual expansion network RDN unit to obtain the prediction graph of the axial slice;
integrating the prediction maps of coronal, sagittal, and axial slices through multi-view attention learning to generate multi-view attention features; and
the output of the multi-view shape constraint unit is used by a multitask learning unit to estimate the relevant calcification index.
9. The multi-view shape constraint method of claim 8, characterized in that the goal of the multi-view shape constraint method is defined as:
where t ∈ { seg, as, vs, ms } indicates a particular target task,/tA loss function representing task t, W being a set of all parameters of the entire network, | W |2L2 regularization representing network parameters, λ is a super-parameter of the L2 regularization term,
is gold standard.
10. The multi-view shape constraint method of claim 8, characterized in that the residual attention learning is defined as:
AMmv(xi,c)＝(1+Mcv(xi,c)+Msv(xi,c))*Fav(xi,c)
where i and c indicate the indices, x, of all spatial locations and channels, respectivelyi,cA feature vector indicating the ith spatial position the c-th channel; mcv(xi,c) And Msv(xi,c) Prediction maps representing coronal and sagittal slices, respectively; fav(xi,c) An output prediction graph representing the RDN; AM (amplitude modulation)mv(xi,c) Features representing multi-view attention model learning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010533043.1A CN111599448B (en) | 2020-06-12 | 2020-06-12 | Multi-view shape constraint system and method for specific coronary artery calcification analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010533043.1A CN111599448B (en) | 2020-06-12 | 2020-06-12 | Multi-view shape constraint system and method for specific coronary artery calcification analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111599448A true CN111599448A (en) | 2020-08-28 |
| CN111599448B CN111599448B (en) | 2022-06-10 |
Family
ID=72192530
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010533043.1A Expired - Fee Related CN111599448B (en) | 2020-06-12 | 2020-06-12 | Multi-view shape constraint system and method for specific coronary artery calcification analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111599448B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113409323A (en) * | 2021-06-18 | 2021-09-17 | 推想医疗科技股份有限公司 | Coronary artery segmentation method and device, computer-readable storage medium and electronic equipment |
| CN116030326A (en) * | 2023-01-18 | 2023-04-28 | 山东省人工智能研究院 | Multi-view fused coronary plaque identification method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030176780A1 (en) * | 2001-11-24 | 2003-09-18 | Arnold Ben A. | Automatic detection and quantification of coronary and aortic calcium |
| US20080279435A1 (en) * | 2001-11-24 | 2008-11-13 | Image Analysis, Inc. | Automated Calcium Scoring of the Aorta |
| US20110118595A1 (en) * | 2009-11-16 | 2011-05-19 | Peter Aulbach | Method and device for identifying and assigning coronary calcification to a coronary vessel and computer program product |
| US20120243764A1 (en) * | 2009-12-03 | 2012-09-27 | Cedars-Sinai Medical Center | Method and system for plaque characterization |
| CN103201768A (en) * | 2010-11-12 | 2013-07-10 | 皇家飞利浦电子股份有限公司 | Identifying individual sub-regions of the cardiovascular system for calcium scoring |
| CN109598722A (en) * | 2018-12-10 | 2019-04-09 | 杭州帝视科技有限公司 | Image analysis method based on recurrent neural network |
| CN110490863A (en) * | 2019-08-22 | 2019-11-22 | 北京红云智胜科技有限公司 | Whether there is or not the systems of total occlusion for detection coronary angiography based on deep learning |
| CN110647596A (en) * | 2018-06-11 | 2020-01-03 | 百度在线网络技术（北京）有限公司 | Map data processing method and device |
-
2020
- 2020-06-12 CN CN202010533043.1A patent/CN111599448B/en not_active Expired - Fee Related
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030176780A1 (en) * | 2001-11-24 | 2003-09-18 | Arnold Ben A. | Automatic detection and quantification of coronary and aortic calcium |
| US20080279435A1 (en) * | 2001-11-24 | 2008-11-13 | Image Analysis, Inc. | Automated Calcium Scoring of the Aorta |
| US20110118595A1 (en) * | 2009-11-16 | 2011-05-19 | Peter Aulbach | Method and device for identifying and assigning coronary calcification to a coronary vessel and computer program product |
| US20120243764A1 (en) * | 2009-12-03 | 2012-09-27 | Cedars-Sinai Medical Center | Method and system for plaque characterization |
| CN103201768A (en) * | 2010-11-12 | 2013-07-10 | 皇家飞利浦电子股份有限公司 | Identifying individual sub-regions of the cardiovascular system for calcium scoring |
| CN110647596A (en) * | 2018-06-11 | 2020-01-03 | 百度在线网络技术（北京）有限公司 | Map data processing method and device |
| CN109598722A (en) * | 2018-12-10 | 2019-04-09 | 杭州帝视科技有限公司 | Image analysis method based on recurrent neural network |
| CN110490863A (en) * | 2019-08-22 | 2019-11-22 | 北京红云智胜科技有限公司 | Whether there is or not the systems of total occlusion for detection coronary angiography based on deep learning |
Non-Patent Citations (8)
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113409323A (en) * | 2021-06-18 | 2021-09-17 | 推想医疗科技股份有限公司 | Coronary artery segmentation method and device, computer-readable storage medium and electronic equipment |
| CN116030326A (en) * | 2023-01-18 | 2023-04-28 | 山东省人工智能研究院 | Multi-view fused coronary plaque identification method |
| CN116030326B (en) * | 2023-01-18 | 2023-09-08 | 山东省人工智能研究院 | Multi-view fused coronary plaque identification method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111599448B (en) | 2022-06-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112927240B (en) | CT image segmentation method based on improved AU-Net network | |
| CN111539930A (en) | Dynamic ultrasonic breast nodule real-time segmentation and identification method based on deep learning | |
| CN111476796B (en) | Semi-supervised coronary artery segmentation system and segmentation method combining multiple networks | |
| WO2010100858A1 (en) | Image processing device and method, and program | |
| US20220284583A1 (en) | Computerised tomography image processing | |
| CN111599448B (en) | Multi-view shape constraint system and method for specific coronary artery calcification analysis | |
| CN101517614A (en) | Advanced computer-aided diagnosis of lung nodules | |
| CN111612756B (en) | Coronary artery specificity calcification detection method and device | |
| CN116739985A (en) | Pulmonary CT image segmentation method based on transducer and convolutional neural network | |
| CN116862877A (en) | Scanning image analysis system and method based on convolutional neural network | |
| CN115512182A (en) | CT angiography intelligent imaging method based on focused learning | |
| CN115018863A (en) | Image segmentation method and device based on deep learning | |
| CN114998292A (en) | Cardiovascular calcified plaque detection system based on residual double attention mechanism | |
| JP5364009B2 (en) | Image generating apparatus, image generating method, and program thereof | |
| CN113947593B (en) | Segmentation method and device for vulnerable plaque in carotid ultrasound image | |
| Denzinger et al. | CAD-RADS scoring using deep learning and task-specific centerline labeling | |
| CN116433586A (en) | Mammary gland ultrasonic tomography image segmentation model establishment method and segmentation method | |
| CN116228690A (en) | Automatic auxiliary diagnosis method for pancreatic cancer and autoimmune pancreatitis based on PET-CT | |
| Sha et al. | A robust segmentation method based on improved U-Net | |
| WO2022272160A1 (en) | A method for cerebral vessel calcification detection and quantification, using machine learning | |
| Mani | Deep learning models for semantic multi-modal medical image segmentation | |
| CN114565626A (en) | Lung CT image segmentation algorithm based on PSPNet improvement | |
| Baral et al. | A review on machine learning for arterial extraction and quantitative assessment on invasive coronary angiograms | |
| CN114926475A (en) | Automatic segmentation and scoring method and system for focus of cerebral stem ischemic stroke | |
| CN117788495A (en) | Lower limb artery CTA blood vessel and plaque segmentation method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220610 |