CN111461990B - Method for realizing super-resolution imaging step by step based on deep learning - Google Patents
Method for realizing super-resolution imaging step by step based on deep learning Download PDFInfo
- Publication number
- CN111461990B CN111461990B CN202010261626.3A CN202010261626A CN111461990B CN 111461990 B CN111461990 B CN 111461990B CN 202010261626 A CN202010261626 A CN 202010261626A CN 111461990 B CN111461990 B CN 111461990B
- Authority
- CN
- China
- Prior art keywords
- resolution
- resolution image
- images
- low
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 115
- 238000003384 imaging method Methods 0.000 title claims abstract description 96
- 238000013135 deep learning Methods 0.000 title claims abstract description 48
- 238000013528 artificial neural network Methods 0.000 claims abstract description 99
- 238000012549 training Methods 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 17
- 230000000694 effects Effects 0.000 abstract description 13
- 238000006243 chemical reaction Methods 0.000 abstract description 10
- 230000000903 blocking effect Effects 0.000 abstract description 3
- 238000012360 testing method Methods 0.000 description 31
- 238000007796 conventional method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 239000000835 fiber Substances 0.000 description 8
- 230000006872 improvement Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000008045 co-localization Effects 0.000 description 4
- 229920002401 polyacrylamide Polymers 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000000386 microscopy Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- TXBBUSUXYMIVOS-UHFFFAOYSA-N thenoyltrifluoroacetone Chemical class FC(F)(F)C(=O)CC(=O)C1=CC=CS1 TXBBUSUXYMIVOS-UHFFFAOYSA-N 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000000799 fluorescence microscopy Methods 0.000 description 1
- 238000004020 luminiscence type Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for realizing super-resolution imaging step by step based on deep learning, which comprises the following steps: s1, inputting the low-resolution image to be processed into a pre-trained neural network to obtain an intermediate-resolution image; s2, cutting the intermediate resolution image after amplifying the intermediate resolution image by k times to ensure that the resolution of the intermediate resolution image is matched with the resolution of the low resolution image to be processed; wherein k is the ratio of the pixel resolution of the low-resolution image to be processed to the pixel resolution of the intermediate-resolution image; and S3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the spliced images to the original 1/k to obtain high-resolution images. According to the invention, by improving the resolution of the image in a blocking and step-by-step manner, the problem of low performance of a neural network for image conversion caused by overlarge difference between a low-resolution image and a high-resolution image is solved, and the obtained super-resolution image has a good effect.
Description
Technical Field
The invention belongs to the field of image processing, and particularly relates to a method for realizing super-resolution imaging step by step based on deep learning.
Background
The traditional super-resolution imaging technology needs to acquire a large amount of data for generating a final super-resolution image, so that the problems of long acquisition time, more acquired samples and the like often exist. In recent years, with the rapid development of machine learning technology, many image conversion technologies based on deep learning are also used in super-resolution imaging, which can greatly reduce the number of pictures to be acquired, thereby improving the efficiency of super-resolution imaging, and therefore, there is an important significance in researching a method for realizing super-resolution imaging based on deep learning.
In the existing method for realizing super-resolution imaging based on deep learning, a neural network is always obtained by extracting corresponding features from an input picture and comparing the corresponding features with a target picture, so that the required type of image conversion can be realized. However, when the relevance and similarity between the input picture and the target picture are low, that is, when the difference between the high-resolution picture and the low-resolution picture is too large, the super-resolution imaging effect is poor.
Disclosure of Invention
Aiming at the defects or improvement requirements of the prior art, the invention provides a method for realizing super-resolution imaging step by step based on deep learning, which is used for solving the technical problem of poor super-resolution imaging effect caused by the fact that the existing method does not consider the larger difference between a low-resolution picture and a high-resolution picture when a neural network is trained.
In order to achieve the above object, in a first aspect, the present invention provides a method for implementing super-resolution imaging step by step based on deep learning, including the following steps:
s1, inputting the low-resolution image to be processed into a pre-trained neural network to obtain an intermediate-resolution image;
s2, cutting the intermediate resolution image after amplifying the intermediate resolution image by k times to ensure that the resolution of the intermediate resolution image is matched with the resolution of the low resolution image to be processed; wherein k is the ratio of the pixel resolution of the low-resolution image to be processed to the pixel resolution of the intermediate-resolution image;
s3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the spliced images to 1/k of the original images to obtain high-resolution images;
wherein the resolution of the intermediate resolution image is greater than the resolution of the low resolution image and less than the resolution of the high resolution image.
Further preferably, the neural network is a cGAN network.
Further preferably, the training method of the neural network includes: respectively processing images in a preset image set to obtain a corresponding intermediate resolution image set and a corresponding low resolution image set; and training the neural network by taking the low-resolution image set as input and the middle-resolution image set as output.
Further preferably, the images in the preset image set are processed by respectively adopting Gaussian models with the radius of the first pixel and the radius of the second pixel to obtain a low-resolution image and a middle-resolution image; wherein the first pixel is larger than the second pixel.
Further preferably, the method for realizing super-resolution imaging step by step based on deep learning provided by the first aspect of the present invention is further used for realizing time super-resolution imaging.
In a second aspect, the present invention provides a method for realizing super-resolution imaging step by step based on deep learning, including:
inputting a low-resolution image to be processed into a pre-trained neural network to obtain a high-resolution image;
the neural network comprises n cascaded neural networks, and the n cascaded neural networks are respectively marked as a first neural network, a second neural network and an nth neural network according to the direction from image input to image output; n is an integer of 2 or more; and each neural network is trained respectively;
the first neural network is used for processing the low-resolution image F to be processed0Conversion into an intermediate resolution image F1；
The ith neural network is used for converting the intermediate resolution image Fi-1Conversion into an intermediate resolution image Fi；
The nth neural network is used for converting the intermediate resolution image Fn-1Conversion into a high resolution image Fn；
Wherein, F0Resolution of < F1Resolution of < Fi-1Resolution of < FiResolution of < FnI is more than or equal to2 and less than n.
Further preferably, the n neural networks are cGAN networks.
Further preferably, the n neural networks are trained respectively; the method specifically comprises the following steps:
processing each image in the preset image set respectively to obtain corresponding low-resolution images F0Intermediate resolution image F1Intermediate resolution image Fi-1Intermediate resolution image FiAnd a high resolution image Fn；
The resulting low resolution images F0Intermediate resolution image F1Intermediate resolution image Fi-1Intermediate resolution image FiAnd a high resolution image FnRespectively forming low resolution image sets S0Intermediate resolution image set S1Intermediate resolution image set Si-1Intermediate resolution image set SiAnd a high resolution image set Sn；
With low resolution image set S0For input, a set S of intermediate resolution images1Training a first neural network for output; at intermediate resolution image Si-1For input, with intermediate resolution image SiTraining an ith neural network for output; at intermediate resolution image Sn-1For input, with a high resolution image set SnTraining an nth neural network for the output; wherein i is more than or equal to2 and less than n.
Further preferably, for the images in the preset image set, the radius is taken as a pixel p respectively0Pixel pi-1Pixel piAnd a pixel pnThe Gaussian model is processed to obtain a low-resolution image F0Intermediate resolution image Fi-1Intermediate resolution image FiAnd a high resolution image Fn(ii) a Wherein p is0＞pi-1＞pi＞pnAnd i is more than or equal to2 and less than n.
Further preferably, the method for realizing super-resolution imaging step by step based on depth learning provided by the second aspect of the present invention is further used for realizing time super-resolution imaging.
Generally, by the above technical solution conceived by the present invention, the following beneficial effects can be obtained:
1. the invention provides a method for realizing super-resolution imaging step by step based on deep learning, which comprises the steps of converting a low-resolution image into an intermediate-resolution image, amplifying and cutting the image, sequentially inputting the image into the same neural network, blocking to improve the resolution of the image, splicing the obtained images according to the original cutting sequence, and reducing the images into the original size to obtain a high-resolution image. According to the invention, by improving the resolution of the image in a blocking and step-by-step manner, the problem of low performance of a neural network for image conversion caused by overlarge difference between a low-resolution image and a high-resolution image is solved, the conversion accuracy of the low-resolution image to the high-resolution image is obviously improved, and the obtained super-resolution image has a good effect.
2. According to the method for realizing super-resolution imaging step by step based on deep learning, provided by the invention, the same neural network is used for multiple times to improve the resolution, so that the training cost is reduced, and the utilization rate of the neural network is greatly improved.
3. According to the method for realizing super-resolution imaging step by step based on deep learning, which is provided by the second aspect of the invention, a plurality of serially connected neural networks are adopted, a low-resolution image is converted into an intermediate-resolution image, and then the intermediate-resolution image is converted into a high-resolution image.
3. According to the method for realizing super-resolution imaging step by step based on deep learning, provided by the invention, in the neural network training stage, the resolution difference between input and a target is reduced, the training difficulty is obviously reduced, and meanwhile, the accuracy of neural network processing is also improved.
4. When training samples are collected, if a photographing mode is directly adopted for collection, the difficulty of the collection process is high due to the fact that the number of required pictures is large, and images with different resolutions are required; in addition, because the images collected by the method have other interference factors and influence the reliability of the training set, when the training samples are collected, the images with different resolutions are obtained by simulation according to Gaussian models with different radiuses, so that the limitation of collecting the images with different resolutions by actual photographing is eliminated, and the method is more flexible; and the neural network obtained by training the obtained training sample has higher accuracy and is more reliable.
5. The method provided by the invention is also suitable for time super-resolution imaging, and longer exposure time is needed because fluorescent molecules with longer service life are often used in time super-resolution imaging. The fluorescence intensity of the long-life fluorescent molecule in a short time is low, and the fluorescence signal captured by the traditional super-resolution method utilizing the fluorescence switch is too weak to be suitable for the long-life fluorescent molecule. The time-resolved imaging obtained by the invention can reduce scattered light and short-life fluorescence interference, has high signal-to-noise ratio and good super-resolution image effect. In addition, the method for realizing the time super-resolution imaging is not influenced by the training sample, and because the low resolution of the time super-resolution imaging is limited by the point spread function, the low resolution image and the high resolution in the training set can be obtained by simulating by the Gaussian function, so the method is suitable for the neural network obtained by training by using the common image and has universality.
Drawings
FIG. 1 is a flowchart of a method for implementing super-resolution imaging step by step based on deep learning according to a first aspect of the present invention;
fig. 2 is a flowchart of a method for implementing super-resolution imaging step by step based on deep learning according to embodiment 1 of the present invention;
fig. 3 is a high-resolution image obtained by processing images in a low-resolution image test set by using the method provided in embodiment 1 of the present invention and the conventional method for implementing super-resolution imaging based on deep learning, respectively; wherein, the graph (a) is an original low-resolution image, the graph (b) is an ideal high-resolution image, the graph (c) is a high-resolution image obtained by adopting the existing method for realizing super-resolution imaging based on depth learning, and the graph (d) is a high-resolution image obtained by adopting the method provided by the embodiment 1 of the invention;
FIG. 4 is a schematic diagram of the results of time super-resolution imaging implemented by the method according to the first aspect of the present invention; the image (a) is a low-resolution image obtained by adopting a long-life phosphorescent complex Eu (TTA) 3to dye polyacrylamide fibers and imaging on a time-resolved microscope; FIG. (b) is a high resolution image obtained by processing the low resolution image of FIG. (a) by the method of the first aspect of the present invention;
FIG. 5 is a flowchart of a method for implementing super-resolution imaging step by step based on deep learning according to a second aspect of the present invention;
FIG. 6 is a flowchart of a method for training neural network diagrams according to embodiment 4 of the present invention;
fig. 7 is a high-resolution image obtained by processing images in a low-resolution image test set by using the method provided in embodiment 4 of the present invention and the conventional method for implementing super-resolution imaging based on deep learning, respectively; wherein, the graph (a) is an original low-resolution image, the graph (b) is an ideal high-resolution image, the graph (c) is a high-resolution image obtained by adopting the existing method for realizing super-resolution imaging based on depth learning, and the graph (d) is a high-resolution image obtained by adopting the method provided by the embodiment 4 of the invention;
fig. 8 is a high-resolution image obtained by processing images in a low-resolution image test set by using the method provided in embodiment 5 of the present invention and the conventional method for implementing super-resolution imaging based on deep learning, respectively; wherein, the graph (a) is an original low-resolution image, the graph (b) is an ideal high-resolution image, the graph (c) is a high-resolution image obtained by adopting the existing method for realizing super-resolution imaging based on depth learning, and the graph (d) is a high-resolution image obtained by adopting the method provided by the embodiment 5 of the invention;
FIG. 9 is a diagram illustrating the result of time super-resolution imaging implemented by the method according to the second aspect of the present invention; wherein, plot (a) is a low resolution image taken at a chopper frequency of 600Hz using a self-mode-locked phase time resolution system; fig. (b) is a high resolution image obtained by processing the low resolution temporal image in fig. (a) by the method provided by the second aspect of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
In order to achieve the above object, in a first aspect, the present invention further provides a method for implementing super-resolution imaging step by step based on deep learning, as shown in fig. 1, including the following steps:
s1, inputting the low-resolution image to be processed into a pre-trained neural network to obtain an intermediate-resolution image;
s2, cutting the intermediate resolution image after amplifying the intermediate resolution image by k times to ensure that the resolution of the intermediate resolution image is matched with the resolution of the low resolution image to be processed; wherein k is the ratio of the pixel resolution of the low-resolution image to be processed to the pixel resolution of the intermediate-resolution image;
s3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the spliced images to 1/k of the original images to obtain high-resolution images;
wherein the resolution of the intermediate resolution image is greater than the resolution of the low resolution image and less than the resolution of the high resolution image.
The details are given below with reference to the examples:
examples 1,
A method for realizing 4 times resolution improvement step by step based on deep learning is disclosed, the number n of cascaded neural networks in the embodiment is 2, and the method specifically comprises the following steps:
generating a training set: 520 scattered dot patterns distributed like fibers were randomly generated, and each dot had a different brightness. The 520 pictures are respectively processed by point diffusion models (B, Zhang, J.Zerubia, and J.Olivo-Marin, Gaussian improvements of fluorescence point-spread functions models, appl.Opt.46(10), 1819 and 1829 (2007)) with different radiuses, so as to obtain pictures with different resolutions. In order to simplify the model, in this embodiment, the point spread model is a gaussian model, and the 520 images are processed by using gaussian models with radii of 1 pixel, 2 pixels, and 4 pixels, respectively, to obtain corresponding images with a resolution of 1 pixel, 2 pixels, and 4 pixels. Since the smaller the pixels that can be resolved, the higher the resolution, the images of 1-pixel resolution, 2-pixel resolution, and 4-pixel resolution are taken as high-resolution, intermediate-resolution, and low-resolution images, respectively, to obtain a high-resolution image set, an intermediate-resolution image set, and a low-resolution image set. The images in each image set are 520, and the high-resolution images, the middle-resolution images and the low-resolution images correspond to one another.
Generating a test set: 100 scattered dot patterns distributed like fibers were randomly generated, and each dot had a different brightness. And respectively processing the 100 images by adopting Gaussian models with the radiuses of 4 pixels and 1 pixel to obtain corresponding images with the resolutions of 4 pixels and 1 pixel, thereby obtaining a low-resolution image test set and a high-resolution image test set.
Training a neural network: the neural network G42 is trained using the low resolution image set as input and the intermediate resolution image set as output. The neural network G42 is a cGAN network, and compared with other neural network models, images obtained by adopting the cGAN network are clearer.
Realizing super-resolution imaging, as shown in fig. 2, comprises the following steps:
s1, respectively inputting the images in the low-resolution image test set into the pre-trained neural network G42 to obtain an intermediate-resolution image; the pixel resolution of the low-resolution image is 4, and the pixel resolution of the intermediate-resolution image is 2.
S2, amplifying the obtained intermediate resolution image by 2 times, and then cutting to enable the resolution to be matched with the resolution of the low-resolution image to be processed; specifically, the obtained intermediate-resolution image was enlarged by 2 times to have a resolution of 4 pixels, and then cut into 4 parts.
And S3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the spliced images into the original 1/2 to obtain high-resolution images. Specifically, the obtained cut images are sequentially input into the pre-trained neural network G42, and the obtained images are spliced according to the original cutting sequence and then reduced by 2 times to obtain a high-resolution image.
In order to verify the method provided by the invention, the method provided by the embodiment and the existing method for realizing super-resolution imaging based on depth learning are respectively adopted to process the images in the low-resolution image test set so as to obtain the high-resolution images. As shown in fig. 3, where (a) is an original low-resolution image, (b) is an ideal high-resolution image, fig. (c) is a high-resolution image obtained by using a conventional method for implementing super-resolution imaging based on depth learning, and fig. (d) is a high-resolution image obtained by using the method provided in this embodiment, it can be seen from the drawings that, compared with the conventional method, the high-resolution image obtained by using the method provided in this embodiment is closer to the ideal high-resolution image, and the super-resolution imaging effect is better. Further, the Structural Similarity value (SSIM) and the co-localization Correlation Coefficient (PCC) between each high-resolution image obtained by using the method provided by the present embodiment and the existing method for realizing super-resolution imaging based on depth learning and the corresponding image in the high-resolution image test set are calculated respectively, so as to obtain the results shown in table 1. The SSIM and PCC parameters are two standard parameters widely applied to the field of microscopic imaging and picture processing and are used for comparing the similarity of two pictures, the numerical value of the SSIM and PCC parameters is in direct proportion to the similarity, and the larger the SSIM and PCC values are, the better the high-resolution imaging effect is. As can be seen from table 3, both SSIM and PCC of the method provided by this embodiment are higher than those of the existing method for implementing super-resolution imaging based on deep learning, and the effect of high-resolution imaging is better.
TABLE 1
| Method | SSIM | PCC |
| Existing methods | 0.9236-0.9532(0.9396) | 0.8369-0.8987(0.8801) |
| Example 4 the procedure provided | 0.9315-0.9773(0.9522) | 0.8510-0.9053(0.8812) |
Examples 2,
A method for realizing 2.25 times resolution improvement step by step based on deep learning specifically comprises the following steps:
generating a training set: 550 pictures consisting of lines distributed like fibers were randomly generated, and each line had a different brightness. And processing the 550 images by adopting Gaussian models with the radii of 2 pixels, 3 pixels and 4.5 pixels respectively to obtain the corresponding images with the resolution of 2 pixels, the images with the resolution of 3 pixels and the images with the resolution of 4.5 pixels. And respectively taking the obtained images with the 2-pixel resolution, the 3-pixel resolution and the 4.5-pixel resolution as high-resolution images, middle-resolution images and low-resolution images, so as to obtain a high-resolution image set, a middle-resolution image set and a low-resolution image set. The images in each image set are 550 images, and the high-resolution images, the middle-resolution images and the low-resolution images correspond to one another.
Generating a test set: and randomly generating 200 pictures which are distributed like fibers and are composed of lines, wherein each line has different brightness, and the density of the lines in 100 images is high and is recorded as a high-density resolution image. And respectively processing the 200 images by adopting Gaussian models with the radiuses of 4.5 pixels and 2 pixels to obtain images with the corresponding resolutions of 4.5 pixels and 2 pixels, thereby obtaining a low-resolution image test set and a high-resolution image test set.
Training a neural network: similarly, the neural network g4.5to3 is trained with the low resolution image set as input and the intermediate resolution image set as output. Wherein, the neural network G4.5to3 is a cGAN network.
Realizing super-resolution imaging, comprising the following steps:
s1, respectively inputting the images in the low-resolution image test set into the pre-trained neural network G4.5to3 to obtain an intermediate-resolution image; the pixel resolution of the low-resolution image is 4.5, and the pixel resolution of the intermediate-resolution image is 3.
S2, amplifying the obtained intermediate resolution image by 1.5 times, and then cutting to enable the resolution to be matched with the resolution of the low-resolution image to be processed; specifically, the obtained intermediate-resolution image is enlarged by 1.5 times to have a resolution of 4.5 pixels, and then is subjected to a certain cutting.
And S3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the obtained spliced images by 1.5 times to obtain high-resolution images. Specifically, the obtained cut images are sequentially input into the pre-trained neural network G4.5to3, and the obtained images are spliced according to the original cutting sequence and then reduced by 1.5 times to obtain a high-resolution image.
Similarly, in order to verify the method proposed by the present invention, the following is compared with the existing method for implementing super-resolution imaging based on deep learning. In the existing method for realizing super-resolution imaging based on deep learning, a low-resolution image set is used as input, and a high-resolution image set is used as output to train a neural network G4.5to2. And respectively inputting the images in the low-resolution image test set into the pre-trained neural network G4.5to2 to obtain high-resolution images. The method provided by the present embodiment and the existing method for realizing super-resolution imaging based on deep learning are respectively used for processing the high-density image and the non-high-density (normal-density) image in the low-resolution image test set, and the structural similarity value SSIM and the co-localization correlation coefficient PCC between each high-resolution image obtained by calculation and the corresponding image in the high-resolution image test set are obtained, so as to obtain the results shown in table 2. As can be seen from Table 2, both the SSIM and the PCC obtained by the method provided by the invention are higher than those obtained by the existing method, and the super-resolution imaging effect is better no matter the images are complex images with high line density or images with relatively low line density.
TABLE 2
Examples 3,
The method for realizing super-resolution imaging step by step based on deep learning provided by the first aspect of the invention can also be used for realizing time super-resolution imaging. In this embodiment, after a long-life phosphorescent complex Eu (TTA)3(TTA is deprotonated alpha-thenoyltrifluoroacetone) is adopted to dye a polyacrylamide fiber, imaging is performed on a Time-resolved microscope, and the obtained Time-resolved imaging result is shown in a (a) diagram in FIG. 4, wherein the Time-resolved microscope adopts a microscope provided in a reference (Jin D, Piper JA. Time-weighted luminescence microscopy imaging of blue-dependent microscopy in background-free conditioning, analytical Chemistry (2011) 83: 2294-; then, the method for realizing super-resolution imaging step by step based on deep learning provided by the first aspect of the invention is further adopted to carry out super-resolution imaging on the image, and the result as shown in (b) of fig. 4 is obtained. Since longer-lived fluorescent molecules are often used in time-resolved imaging, longer exposure times are required. The fluorescence intensity of the long-life fluorescent molecule in a short time is low, and the fluorescence signal captured by the traditional super-resolution method utilizing the fluorescence switch is too weak to be suitable for the long-life fluorescent molecule. Compared with the conventional steady-state fluorescence imaging, the time-resolved imaging obtained by the invention can reduce scattered light and short-life fluorescence interference, and has higher signal-to-noise ratio. In addition, the method for realizing the time super-resolution imaging is not influenced by the training sample, and because the low resolution of the time super-resolution imaging is limited by the point spread function, the low resolution image and the high resolution in the training set can be obtained by simulating by the Gaussian function, so the method is suitable for the neural network obtained by training by using the common image and has universality.
In a second aspect, the present invention provides a method for implementing super-resolution imaging step by step based on deep learning, as shown in fig. 5, including:
inputting a low-resolution image to be processed into a pre-trained neural network to obtain a high-resolution image;
the neural network comprises n cascaded neural networks, and the n cascaded neural networks are respectively marked as a first neural network, a second neural network, a. n is an integer of 2 or more; and each neural network is trained respectively; specifically, the number n of cascaded neural networks is determined according to specific conditions (such as selection of intermediate resolution).
The first neural network is used for processing the low-resolution image F to be processed0Conversion into an intermediate resolution image F1；
The ith neural network is used for converting the intermediate resolution image Fi-1Conversion into an intermediate resolution image Fi；
The nth neural network is used for converting the intermediate resolution image Fn-1 into a high resolution image Fn;
wherein the resolution of F0 is less than the resolution of F1 is less than the resolution of Fi-1 is less than the resolution of Fn, and i is more than or equal to2 and less than n.
The details are given below with reference to the examples:
examples 4,
A method for realizing 4 times resolution improvement step by step based on deep learning is disclosed, the number n of cascaded neural networks in the embodiment is 2, and the method specifically comprises the following steps:
the training set (high resolution image set, intermediate resolution image set, and low resolution image set) and the test set (low resolution image test set and high resolution image test set) were obtained in the same manner as in example 1.
Training a neural network: as shown in FIG. 6, a first neural network G42 is trained using the low resolution image set as input and the intermediate resolution image set as output. A second neural network G21 is trained using the intermediate resolution image set as input and the high resolution image set as output. The obtained first neural network G42 and the second neural network G21 are connected in series to obtain a pre-trained neural network. The first neural network G42 and the second neural network G21 are both cGAN networks, and compared with other neural network models, images obtained by adopting the cGAN networks are clearer.
Realizing super-resolution imaging: and respectively inputting the images in the low-resolution image test set into the pre-trained neural network, obtaining an intermediate-resolution image after the intermediate-resolution image passes through the first neural network G42, and obtaining a high-resolution image after the intermediate-resolution image passes through the second neural network G21, so that super-resolution imaging is realized.
In order to verify the method proposed by the present invention, the following is compared with the existing method for realizing super-resolution imaging based on deep learning. In the conventional method for implementing super-resolution imaging based on deep learning, as shown in fig. 6, a neural network G41 is trained with a low-resolution image set as an input and a high-resolution image set as an output. And respectively inputting the images in the low-resolution image test set into the pre-trained neural network G41 to obtain high-resolution images. The method provided by the embodiment and the existing method for realizing super-resolution imaging based on deep learning are respectively adopted to process the images in the low-resolution image test set to obtain high-resolution images. As shown in fig. 7, where (a) is an original low-resolution image, (b) is an ideal high-resolution image, (c) is a high-resolution image obtained by using a conventional method for implementing super-resolution imaging based on depth learning, and (d) is a high-resolution image obtained by using the method provided in this embodiment. As can be seen from the figure, compared with the existing method, the high-resolution image obtained by the method provided by the embodiment is closer to the ideal high-resolution image, and the super-resolution imaging effect is better. Further, the structural similarity value SSIM and the co-localization correlation coefficient PCC between each high-resolution image obtained by using the method provided by the present embodiment and the existing method for realizing super-resolution imaging based on deep learning, and the corresponding image in the high-resolution image test set are calculated respectively, and the results shown in table 3 are obtained. As can be seen from table 3, both SSIM and PCC of the method provided by this embodiment are higher than those of the existing method for implementing super-resolution imaging based on deep learning, and the effect of high-resolution imaging is better.
TABLE 3
| Method | SSIM | PCC |
| Existing methods | 0.9236-0.9532(0.9396) | 0.8369-0.8987(0.8801) |
| The method provided in example 1 | 0.9247-0.9728(0.9480) | 0.8535-0.9129(0.8886) |
Examples 5,
A method for realizing 2.25 times resolution improvement step by step based on deep learning is disclosed, the number n of cascaded neural networks in the embodiment is 2, and the method specifically comprises the following steps:
the training set (high resolution image set, intermediate resolution image set, and low resolution image set) and the test set (low resolution image test set and high resolution image test set) were obtained in the same manner as in example 2.
Training a neural network: and training a first neural network G4.5to3 by taking the low-resolution image set as input and the middle-resolution image set as output. A second neural network G3to2 is trained with the intermediate resolution image set as input and the high resolution image set as output. The obtained first neural network G4.5to3 and the second neural network G3to2 are connected in series to obtain a pre-trained neural network. Wherein, the first neural network G4.5to3 and the second neural network G3to2 are both cGAN networks.
Realizing super-resolution imaging: and respectively inputting the images in the low-resolution image test set into the pre-trained neural network, obtaining an intermediate-resolution image after the images pass through a first neural network G4.5to3, and obtaining a high-resolution image after the obtained intermediate-resolution image passes through a second neural network G3to2, thereby realizing super-resolution imaging.
In order to verify the method proposed by the present invention, the following is compared with the existing method for realizing super-resolution imaging based on deep learning. In the existing method for realizing super-resolution imaging based on deep learning, a low-resolution image set is used as input, and a high-resolution image set is used as output to train a neural network G4.5to2. And respectively inputting the images in the low-resolution image test set into the pre-trained neural network G4.5to2 to obtain high-resolution images. The method provided by the embodiment and the existing method for realizing super-resolution imaging based on deep learning are respectively adopted to process the images in the low-resolution image test set to obtain high-resolution images. As shown in fig. 8, a diagram (a) is an original low-resolution image, a diagram (b) is an ideal high-resolution image, a diagram (c) is a high-resolution image obtained by applying a conventional method for implementing super-resolution imaging based on depth learning to the original low-resolution image, and a diagram (d) is a high-resolution image obtained by applying the method provided in this embodiment to the original low-resolution image. As can be seen from the figure, compared with the existing method, the high-resolution image obtained by the method provided by the embodiment is closer to the ideal high-resolution image, and the super-resolution imaging effect is better. Further, the method provided in this embodiment and the existing method for realizing super-resolution imaging based on deep learning are respectively used to process the high-density images and the non-high-density (normal-density) images in the low-resolution image test set, and the structural similarity value SSIM and the co-localization correlation coefficient PCC between each high-resolution image obtained by calculation and the corresponding image in the high-resolution image test set are obtained as shown in table 4. As can be seen from Table 4, both SSIM and PCC obtained by the method provided by the invention are higher than those obtained by the existing method, and the super-resolution imaging effect is better no matter whether the images are complex images with high line density or images with relatively low line density.
TABLE 4
In this embodiment, the number n of the cascaded neural networks is 2, and when n is greater than 2, the operation method is similar to the above operation method, and is not described herein again.
Examples 6,
Similarly, the method for realizing super-resolution imaging step by step based on deep learning provided by the second aspect of the invention can also be used for realizing time super-resolution imaging. Similarly to example 3, in this example, the long-life phosphorescent complex eu (TTA)3(TTA is deprotonated α -thenoyltrifluoroacetone) is also used to dye the polyacrylamide fiber, then the dyed polyacrylamide fiber is photographed by using the auto-mode-locking time-resolved imaging system, a low-resolution picture obtained by photographing when the chopper frequency is 600Hz is shown as (a) in fig. 9, and then the super-resolution imaging is performed by using the method for realizing super-resolution imaging step by step based on depth learning provided by the second aspect of the present invention, so as to obtain the result shown as (b) in fig. 9.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (5)
1. A method for realizing super-resolution imaging step by step based on deep learning is characterized by comprising the following steps:
s1, inputting the low-resolution image to be processed into a pre-trained neural network to obtain an intermediate-resolution image;
s2, cutting the intermediate resolution image after amplifying the intermediate resolution image by k times to ensure that the resolution of the intermediate resolution image is matched with the resolution of the low resolution image to be processed; wherein k is the ratio of the pixel resolution of the low-resolution image to be processed to the pixel resolution of the intermediate-resolution image;
s3, sequentially inputting the cut images into the pre-trained neural network, splicing the obtained images according to the original cutting sequence, and reducing the spliced images to 1/k of the original images to obtain high-resolution images;
the resolution of the intermediate-resolution image is greater than the resolution of the low-resolution image and less than the resolution of the high-resolution image.
2. The method for realizing super-resolution imaging step by step based on deep learning of claim 1, wherein the neural network is a cGAN network.
3. The method for realizing super-resolution imaging step by step based on deep learning according to claim 1 or 2, wherein the training method of the neural network comprises the following steps: respectively processing images in a preset image set to obtain a corresponding intermediate resolution image set and a corresponding low resolution image set; and training the neural network by taking the low-resolution image set as input and the middle-resolution image set as output.
4. The method for realizing super-resolution imaging step by step based on deep learning according to claim 3, characterized in that the images in the preset image set are processed by adopting Gaussian models with the radius of a first pixel and a second pixel respectively to obtain a low-resolution image and a middle-resolution image; wherein the first pixel is larger than the second pixel.
5. The method for realizing super-resolution imaging step by step based on deep learning of claim 1, which is further used for realizing time super-resolution imaging.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010261626.3A CN111461990B (en) | 2020-04-03 | 2020-04-03 | Method for realizing super-resolution imaging step by step based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010261626.3A CN111461990B (en) | 2020-04-03 | 2020-04-03 | Method for realizing super-resolution imaging step by step based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111461990A CN111461990A (en) | 2020-07-28 |
| CN111461990B true CN111461990B (en) | 2022-03-18 |
Family
ID=71685921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010261626.3A Active CN111461990B (en) | 2020-04-03 | 2020-04-03 | Method for realizing super-resolution imaging step by step based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111461990B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| WO2020056274A1 (en) * | 2018-09-14 | 2020-03-19 | The Johns Hopkins University | Machine learning processing of contiguous slice image data |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10803378B2 (en) * | 2017-03-15 | 2020-10-13 | Samsung Electronics Co., Ltd | System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions |
| CN107203969B (en) * | 2017-06-13 | 2019-06-25 | 武汉大学 | A kind of high magnification image super-resolution rebuilding method of medium scale constraint |
| JP6915685B2 (en) * | 2017-06-20 | 2021-08-04 | 日本電気株式会社 | Learning device, super-resolution device, learning method, super-resolution method and program |
| US10692184B2 (en) * | 2018-07-05 | 2020-06-23 | SVXR, Inc. | Super-resolution X-ray imaging method and apparatus |
| CN110136062B (en) * | 2019-05-10 | 2020-11-03 | 武汉大学 | Super-resolution reconstruction method combining semantic segmentation |
-
2020
- 2020-04-03 CN CN202010261626.3A patent/CN111461990B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016019484A1 (en) * | 2014-08-08 | 2016-02-11 | Xiaoou Tang | An apparatus and a method for providing super-resolution of a low-resolution image |
| WO2020056274A1 (en) * | 2018-09-14 | 2020-03-19 | The Johns Hopkins University | Machine learning processing of contiguous slice image data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111461990A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ale et al. | Road damage detection using RetinaNet | |
| CN110210608B (en) | Low-illumination image enhancement method based on attention mechanism and multi-level feature fusion | |
| US9897792B2 (en) | Method and system for extended depth of field calculation for microscopic images | |
| CN108830208A (en) | Method for processing video frequency and device, electronic equipment, computer readable storage medium | |
| CN109903282B (en) | Cell counting method, system, device and storage medium | |
| CN109785252B (en) | Night image enhancement method based on multi-scale residual error dense network | |
| EP2926558B1 (en) | A method and system for extended depth of field calculation for microscopic images | |
| CN113159019B (en) | Dim light video enhancement method based on optical flow transformation | |
| CN111382743A (en) | License plate character recognition method based on data enhancement and data generation | |
| CN103778616A (en) | Contrast pyramid image fusion method based on area | |
| WO2023284637A1 (en) | Sequence image super-resolution correction method based on image gradient variance analysis | |
| CN111858340A (en) | Deep neural network test data generation method based on stability transformation | |
| CN111461990B (en) | Method for realizing super-resolution imaging step by step based on deep learning | |
| CN114596233A (en) | Attention-guiding and multi-scale feature fusion-based low-illumination image enhancement method | |
| CN115100578B (en) | Rotary kiln flame combustion state identification method | |
| CN117095248A (en) | Furnace flame oxygen concentration monitoring method based on generating convolutional neural network | |
| CN114972888A (en) | Communication maintenance tool identification method based on YOLO V5 | |
| CN113012074B (en) | Intelligent image processing method suitable for low-illumination environment | |
| CN115170811A (en) | Denoising method based on convolutional neural network | |
| CN111007020B (en) | Double-frame four-spectrum imaging method and application | |
| CN115100033B (en) | Fluorescent microscopic image super-resolution reconstruction method and device and computing equipment | |
| CN112200152B (en) | Super-resolution method for aligning face images based on residual back projection neural network | |
| WO2019140434A2 (en) | Overlapping pattern differentiation at low signal-to-noise ratio | |
| CN109993172B (en) | License plate recognition method for road traffic flow | |
| WO2022022310A1 (en) | Method and apparatus for identifying optical splitter ports |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |