CN110020987A - A kind of medical image super resolution ratio reconstruction method based on deep learning - Google Patents

A kind of medical image super resolution ratio reconstruction method based on deep learning Download PDF

Info

Publication number
CN110020987A
CN110020987A CN201910224792.3A CN201910224792A CN110020987A CN 110020987 A CN110020987 A CN 110020987A CN 201910224792 A CN201910224792 A CN 201910224792A CN 110020987 A CN110020987 A CN 110020987A
Authority
CN
China
Prior art keywords
data
resolution
medical image
coordinate
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910224792.3A
Other languages
Chinese (zh)
Other versions
CN110020987B (en
Inventor
刘蓬博
王瑾
朱青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN201910224792.3A priority Critical patent/CN110020987B/en
Publication of CN110020987A publication Critical patent/CN110020987A/en
Application granted granted Critical
Publication of CN110020987B publication Critical patent/CN110020987B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00ICT specially adapted for the handling or processing of medical images
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00Road transport of goods or passengers
    • Y02T10/10Internal combustion engine [ICE] based vehicles
    • Y02T10/40Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Evolutionary Computation (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Artificial Intelligence (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Image Processing (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

The present invention provides a kind of method of medical image super-resolution rebuilding based on deep learning, medical image is directly inputted in system, correct super-resolution result can be obtained, network is trained by a large amount of high quality medical image, using the location information for introducing lesion in medical image in the training stage, the location information refer to lesion centre coordinate and size or lesion it is fine edge mark, by the way that these prior informations are added, can solve other methods medical image super-resolution network instruct it is askew caused by medical image after super-resolution the problem of being lost script imaging value.

Description

A kind of medical image super resolution ratio reconstruction method based on deep learning
Technical field
The invention belongs to technical field of computer vision, more particularly to a kind of medical image super-resolution based on deep learning Rate method for reconstructing.
Background technique
With the development of deep learning, computer vision and the horizontal of image procossing are constantly breaking through in some scenes The bottleneck of conventional method.Image super-resolution is same to flourish as the important technology in a kind of computer vision.Natural field Super-resolution method under scape based on deep learning emerges one after another, and has also derived a variety of different network design modes, network Depth is continuously increased, and feature multiplexing capacity constantly enhances, and the PSNR index under different scale is also rising steadily, and is supervised in video It has a wide range of applications in the actual scenes such as control, video restoration, digital high-definition and satellite image.
The limitation of dosage when limitation or medical imaging due to medical instrument hardware are shot, super-resolution is in medical image Also there is very big development space in field, and there are also researchers at present directly migrates to medical image for the method in natural image On, although directly will be in natural image in PSNR index and visual effect is as also obtaining some promotions, in this Method moves on more serious medical image, and the way not adjusted accordingly according to the specific features of medical image Leave a question open.Different with natural image, there is positive region (lesion) and negative areas (normal tissue) in medical image, Method of the conventional natural image image super-resolution directly by image block training can be caused due to yin-yang data extremely imbalance Model is askew to going in normal tissue, can cause destruction to the medical image information with imaging value in this way.And An evaluation index of the PSNR as an image global information signal level only uses PSNR as measurement medical image super-resolution The evaluation index of rate effect be also it is incomplete, the super-resolution efect that can not evaluate really important region is good on earth not Good, the signal of focal area may be buried in a large amount of normal tissue regions signal.
Summary of the invention
For the problem to be solved in the present invention as mentioned in background technique, the present invention provides a kind of medicine shadow based on deep learning As super resolution ratio reconstruction method, main contents are as follows:
(1) medical image features are directed to, the additional markup information of medical data collection is utilized, proposes new data set The method to solve the problems, such as yin-yang sample imbalance is referred to as PN-sample hereinafter by production and Training strategy balance。
(2) due to generally acknowledging effective medical image super-resolution rebuilding Indexes of Evaluation Effect, this hair there is no one at present The bright evaluation index for proposing to use CAD system combination PSNR association evaluation medical image super-resolution rebuilding effect.
The introducing of CAD system, which can make up PSNR, can not measure the focal area whether super-resolution is really paid close attention to doctor The defect of effective enhancing is carried out.Only super-resolution technique is carried out towards being correctly oriented, and the promotion of the index of PSNR is also It is meaningful.
Opposite with the prior art, the present invention has following clear superiority:
1, the PN-sample balance method proposed can effectively guarantee model training stability, guarantee yin-yang area Domain can access the learning effect preferably balanced.
2, the association evaluation index proposed can effectively identify that the promotion of major part negative areas causes when avoiding network instruction askew Whole PSNR promoted the case where.It can ensure the validity of medical image super-resolution efect.
Detailed description of the invention:
Fig. 1 is flow chart of the present invention
Fig. 2 is the 3D dense network structure that the present invention uses;
Fig. 3 is the network structure of Dense Block and Compressor that the present invention designs;
Fig. 4 validation PSNR performance, left figure are existing method, the method that right figure is proposed for the present invention 1;
Fig. 5 is that the present invention 2 verifies, CAD system effect contrast figure;
Specific embodiment
In order to enable the objectives, technical solutions, and advantages of the present invention are more clearly understood, below in conjunction with attached drawing 1-5 and reality Example is applied, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only to explain this hair It is bright, it is not intended to limit the present invention.In addition, technical characteristic involved in each embodiment of present invention described below is only Not constituting a conflict with each other can be combined with each other.
The method and system of the present invention provides a set of medical image super-resolution rebuilding based on deep learning.By medicine Image, which is directly inputted in system, can be obtained correct super-resolution result.By a large amount of high quality medical image come Training network neutralizes other methods the difference is that we believe in the position that the training stage introduces lesion in medical image Breath, but do not needed when test, meet actual demand.Herein location information refer to lesion centre coordinate and size or lesion it is fine Edge mark.By the way that these prior informations are added, it is askew that we can solve other methods medical image super-resolution network instruction Medical image after caused super-resolution is lost the problem of script imaging value, our the method are referred to as PN-sample Balance method.And propose that CAD system joint PSNR jointly really effectively comments the progress of medical image super-resolution performance Estimate, super-resolution method should be promoted under the premise of ensuring CAD system performance boost PSNR just it is significant, if super-resolution The PSNR that data afterwards cause the decline of CAD system performance to be promoted is lost meaning, may be to carry out to some extraneous areas Enhancing, and the shortcomings that our the focal area information of real concern have been destroyed, this is also PSNR, CAD system Introducing can effectively make up this disadvantage.
In the embodiment of the present invention, step (1) specifically includes that the acquisition and lesions position information of high quality medical image It introduces and handles.
Specifically, step (1-1), collects the medical images data sets of high quality, and clean to data, by data set In poor data delete.Since our task is super-resolution rebuilding, so the high-resolution in tri- directions z, y, x is It is contemplated that primary factor, the spacing that we are typically chosen three directions be respectively less than 1mm be quality data.Spacing The spatial information being meant that in a voxel is included in medical image actual physics space, the spacing in three directions Smaller, the object of actual size will be made of more voxels, that is, resolution ratio is higher.And we need to collect data The prior information that the location information of middle lesion additionally considers when training as us.
Step (1-2), we will start from super-resolution training dataset after gathering data, in order to higher suitable With property, we are by the way of multiple dimensioned while training, first to quality data collection on the target direction in three directions The down-sampling operation of corresponding scale is carried out, then original resolution is interpolated by bicubic interpolation method or Cubic Spline Method, The data and initial data of low resolution as simulation form low resolution (ILR) and high-resolution (IHR) data pair (data-pairs).To lesions position information, we by Vo xel coords=(World coords-Origin)/ Spacing converts true coordinate information to and the matched voxel coordinate information of 3-D image.Coordinate is according to specific doctor The dimension for treating image is specifically divided into two-dimensional coordinate (y, x) or three-dimensional coordinate (z, y, x).Vo xel coords is medical image square Corresponding coordinate in battle array, World coords are the world coordinate system of lesion, and origin is the coordinate origin of filming instrument, The spatial information that spacing then includes by a voxel mentioned above.We need to record and image used in us Matched Voxel coords is saved in memory for instructing sampling policy when training.
The training and optimization of the main packet network of step (2).
It is used to train specifically, data are directly cut block before training different from other methods by step (2-1), we adopt With location information instruct online stripping and slicing strategy, every time obtain training data when according to the yin-yang ratio data of artificial settings come Generate random number, it is assumed that it is considered that yin-yang is a reasonable ratio than 2: 1, then it can first generate random number 0,1,2, if 0 is generated, then goes to take a coordinate information at random in the positive region coordinate information of preservation in memory from step (1-2), and The disturbance of 20 voxels, by coordinate centered on the coordinate after disturbance, the institute in step (1-2) are added in each dimension in space The low resolution high-resolution data stated is to youngster (ILR、IHRData-pairs a biggish data block is intercepted in) respectively, herein " biggish data block " is larger when referring to than formal training, cuts off extra part after augmentation again, prevents training data from doing Edge is caused to have the case where padding after complete online data augmentation.By carrying out to training data block: flip horizontal hangs down Straight overturning, front and back overturning, translation, scaling, margins of excision redundance is to 42*42* after the random augmentation operation such as multi-angle rotary The size of 42 (being herein 3 dimension datas, 2-D data is then 42*42) is used for network training.If random number is 1 or 2, scheming Then the random active position sampling as in is carried out at the identical stripping and slicing augmentation operation of positive region, for training.
The batch of data block for meeting yin-yang ratio that sampling in step (1-2) obtains is sent into subsequent net by step (2-2) It goes to be trained in network.High-definition picture (the I of reconstruction after obtaining super-resolutionSR), the I that will be obtainedSRWith as goldstandard IHRCarry out MSE lossIt calculates Loss is rebuild, then network parameter is optimized by back-propagation algorithm (back propagation).I in formula, j, k are respectively three The coordinate serial number in direction.It is subsequent use that suitable super-resolution network model is chosen after training is steady.
The mode of the outstanding densenet of our service performances of network intensively connected carrys out planned network herein, to low layer height Layer feature carries out multiplexing fusion, with reach it is better as a result, as shown in Fig. 2, and the parameter amount that is used under same performance it is smaller, Can the characterization ability to limited parameter maximumlly developed, be illustrated in figure 3 we dense block and The basic structure of compressor, dense block are dense connection type, complete main feature extraction work, Compressor is mainly in the effect for wherein playing Data Dimensionality Reduction, because 3D data are very easy to cause the deficiency of video memory.
So far our first inventive point statement finishes, when without reference to our second inventive point, i.e. oversubscription net Whether network correctly enhances medical image, only in PSNR index we to have also exceeded traditional training method very much, Fig. 4 is our existing Comparative results.Though not considering that yin-yang sample imbalance problem present in medical image also has larger It is promoted, but limited capacity.And the present invention 2 in it can be found that existing oversubscription method there are the problem of.
Step 3, existing super-resolution method is verified in the present invention 2, that verifies super-resolution method has conscientiously Effect property.The validity of the present invention 1 can be verified simultaneously.Note that the present invention 2 is verified in real data, that is, test Used in high-resolution data be the high-resolution data really acquired, and it is inartificial by sampling obtained false low point Resolution data.Wherein:
Step (3-1) first carries out the training and test of CAD system in the data operated without any oversubscription, as This batch data attainable performance baseline.
True high-resolution data is carried out oversubscription operation using the model of training in our steps (2) by step (3-2) Enhanced to obtain ISR, and use ISRInstead of original high-resolution data, according to identical parameter training CAD system algorithm, Obtain the performance indicator of the present invention 1.
Step (3-3) uses other configurations all the same in step (2) in addition to the equilibrium between yin and yang method in the present invention 1, into Row training, represents as existing methods.
Step (3-4) compares experiment three times, it is found that existing oversubscription method will lead to the appearance of a large amount of false positives, Also mean that the medical information in influencing has been destroyed.And the method in the present invention 1 is then it is possible to prevente effectively from this problem, and Performance is promoted, as shown in Figure 5.
Step (3-5) to sum up illustrates the problem of present invention 2 can find out the study of mistake existing for existing oversubscription method, can More fully evaluate the performance of medical image super-resolution.And the present invention 1 then can effectively solve the above problems, correctly to doctor It learns image feature to be learnt, to achieve the effect that correctly to promote medical image quality.
Above embodiments are only exemplary embodiment of the present invention, are not used in the limitation present invention, protection scope of the present invention It is defined by the claims.Those skilled in the art can within the spirit and scope of the present invention make respectively the present invention Kind modification or equivalent replacement, this modification or equivalent replacement also should be regarded as being within the scope of the present invention.

Claims (1)

1. a kind of medical image super resolution ratio reconstruction method based on deep learning characterized by comprising
The acquisition of step 1, high quality medical image
Step (1-1), the medical images data sets for collecting high quality, and data are cleaned;
Step (1-2) makes super-resolution training dataset, by the way of multiple dimensioned while training, first to high quality number The down-sampling operation of corresponding scale is carried out on the target direction in three directions according to collection, then by bicubic interpolation method or Cubic Spline Method is interpolated into original resolution, and the data and initial data of the low resolution as simulation form low resolution (ILR) and high-resolution (IHR) data pair (data-pairs) passes through Voxel coords=to lesions position information (World coords-Origin)/Spacing converts true coordinate information to and sits with the matched voxel of 3-D image Mark information.Coordinate is specifically divided into two-dimensional coordinate (y, x) or three-dimensional coordinate (z, y, x) according to the dimension of specific medical image, Voxel coords is corresponding coordinate in medical image matrix, and World coords is the world coordinate system of lesion, origin The spatial information for then including by a voxel mentioned above for the coordinate origin of filming instrument, spacing;
The introducing and processing of step 2, lesions position information
Data are cut block and are used to train by step (2-1) before training, the online stripping and slicing strategy instructed using location information, Random number is generated according to the yin-yang ratio data of artificial settings when obtaining training data every time, it is assumed that yin-yang ratio is 2:1, then may be used First to generate random number 0,1,2, if generating 0, go to save positive region coordinate information in memory from step (1-2) In take a coordinate information at random, and in each dimension in space be added 20 voxels disturbance, using the coordinate after disturbance as Centre coordinate, the low resolution high-resolution data described in step (1-2) is to youngster (ILR、IHRData-pairs in) respectively Intercept a biggish data block;By carrying out to training data block: flip horizontal, flip vertical, front and back overturn, translate, put The size of margins of excision redundance to 42*42*42 are used for network training after contracting, the random augmentation operation of multi-angle rotary;If with Machine number is 1 or 2, then random active position sampling in the picture, is then carried out at the identical stripping and slicing augmentation operation of positive region, For training;
Step (2-2), sampling obtains in step (1-2) the data block for meeting yin-yang ratio is sent into subsequent network go into Row training, the high-definition picture (I of the reconstruction after obtaining super-resolutionSR), the I that will be obtainedSRWith the I as goldstandardHRIt carries out It calculates and rebuilds damage It loses, then network parameter is optimized by back-propagation algorithm (back propagation), wherein i, j, k are respectively three directions Coordinate serial number.
CN201910224792.3A 2019-03-24 2019-03-24 Medical image super-resolution reconstruction method based on deep learning Active CN110020987B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910224792.3A CN110020987B (en) 2019-03-24 2019-03-24 Medical image super-resolution reconstruction method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910224792.3A CN110020987B (en) 2019-03-24 2019-03-24 Medical image super-resolution reconstruction method based on deep learning

Publications (2)

Publication Number Publication Date
CN110020987A true CN110020987A (en) 2019-07-16
CN110020987B CN110020987B (en) 2023-06-30

Family

ID=67189877

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910224792.3A Active CN110020987B (en) 2019-03-24 2019-03-24 Medical image super-resolution reconstruction method based on deep learning

Country Status (1)

Country Link
CN (1) CN110020987B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110610459A (en) * 2019-09-17 2019-12-24 中国气象局公共气象服务中心(国家预警信息发布中心) Image processing method and device
CN111192255A (en) * 2019-12-30 2020-05-22 上海联影智能医疗科技有限公司 Index detection method, computer device, and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654425A (en) * 2015-12-07 2016-06-08 天津大学 Single-image super-resolution reconstruction method applied to medical X-ray image
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN108615236A (en) * 2018-05-08 2018-10-02 上海商汤智能科技有限公司 A kind of image processing method and electronic equipment
CN109360152A (en) * 2018-10-15 2019-02-19 天津大学 3 d medical images super resolution ratio reconstruction method based on dense convolutional neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654425A (en) * 2015-12-07 2016-06-08 天津大学 Single-image super-resolution reconstruction method applied to medical X-ray image
US20180075581A1 (en) * 2016-09-15 2018-03-15 Twitter, Inc. Super resolution using a generative adversarial network
CN108615236A (en) * 2018-05-08 2018-10-02 上海商汤智能科技有限公司 A kind of image processing method and electronic equipment
CN109360152A (en) * 2018-10-15 2019-02-19 天津大学 3 d medical images super resolution ratio reconstruction method based on dense convolutional neural networks

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
刘蓬博: "基于深度学习的医学图像增强算法的研究" *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110610459A (en) * 2019-09-17 2019-12-24 中国气象局公共气象服务中心(国家预警信息发布中心) Image processing method and device
CN111192255A (en) * 2019-12-30 2020-05-22 上海联影智能医疗科技有限公司 Index detection method, computer device, and storage medium
CN111192255B (en) * 2019-12-30 2024-04-26 上海联影智能医疗科技有限公司 Index detection method, computer device, and storage medium

Also Published As

Publication number Publication date
CN110020987B (en) 2023-06-30

Similar Documents

Publication Publication Date Title
US9317970B2 (en) Coupled reconstruction of hair and skin
CN109934832A (en) Liver neoplasm dividing method and device based on deep learning
CN105453139A (en) Sparse GPU voxelization for 3D surface reconstruction
CN107464216A (en) A kind of medical image ultra-resolution ratio reconstructing method based on multilayer convolutional neural networks
Zhang et al. Data-driven synthetic modeling of trees
Zhang et al. Adaptive and quality 3d meshing from imaging data
Luo et al. Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images
CN109285225A (en) A kind of method for building up of the virtual reality auxiliary operation based on medical image
CN107341846A (en) A kind of method and apparatus of the extensive three-dimensional reconstruction scene of real-time display
CN112785692B (en) Single-view-angle multi-person human body reconstruction method based on depth UV prior
CN110020987A (en) A kind of medical image super resolution ratio reconstruction method based on deep learning
Muqeet et al. Hybrid residual attention network for single image super resolution
Zhang et al. Scenewiz3d: Towards text-guided 3d scene composition
Son et al. Singraf: Learning a 3d generative radiance field for a single scene
CN110322548B (en) Three-dimensional grid model generation method based on geometric image parameterization
Schirmacher et al. Efficient Free Form Light Field Rendering.
CN116801115A (en) Sparse array camera deployment method
CN112002019B (en) Method for simulating character shadow based on MR mixed reality
Jäger et al. A comparative Neural Radiance Field (NeRF) 3D analysis of camera poses from HoloLens trajectories and Structure from Motion
Wang et al. GScream: Learning 3D Geometry and Feature Consistent Gaussian Splatting for Object Removal
Zeng et al. 3D plants reconstruction based on point cloud
WO2024156209A1 (en) Security boundary generation method and apparatus, device, storage medium and program product
Tanay et al. Global Latent Neural Rendering
Hesse et al. A Quantitative Analysis of Culling Techniques for Real-time Rendering of Digital Elevation Models
Westerteiger Virtual Reality Methods for Research in the Geosciences

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant