CN111650738A - Fourier laminated microscopic image reconstruction method and device based on deep learning - Google Patents
Fourier laminated microscopic image reconstruction method and device based on deep learning Download PDFInfo
- Publication number
- CN111650738A CN111650738A CN202010378100.3A CN202010378100A CN111650738A CN 111650738 A CN111650738 A CN 111650738A CN 202010378100 A CN202010378100 A CN 202010378100A CN 111650738 A CN111650738 A CN 111650738A
- Authority
- CN
- China
- Prior art keywords
- sample
- fourier
- pgnn
- resolution
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000013135 deep learning Methods 0.000 title claims abstract description 17
- 230000004075 alteration Effects 0.000 claims abstract description 60
- 238000009826 distribution Methods 0.000 claims abstract description 55
- 230000003287 optical effect Effects 0.000 claims abstract description 50
- 238000013528 artificial neural network Methods 0.000 claims abstract description 28
- 238000005457 optimization Methods 0.000 claims abstract description 27
- 230000006870 function Effects 0.000 claims description 43
- 230000008569 process Effects 0.000 claims description 35
- 238000012546 transfer Methods 0.000 claims description 30
- 230000001427 coherent effect Effects 0.000 claims description 29
- 238000001000 micrograph Methods 0.000 claims description 18
- 238000001228 spectrum Methods 0.000 claims description 15
- 238000005286 illumination Methods 0.000 claims description 11
- 238000012937 correction Methods 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 claims description 10
- 238000012634 optical imaging Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 7
- 238000013136 deep learning model Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000000386 microscopy Methods 0.000 claims description 4
- 238000011084 recovery Methods 0.000 claims description 4
- 230000003595 spectral effect Effects 0.000 claims description 4
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 210000001747 pupil Anatomy 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000003062 neural network model Methods 0.000 abstract description 10
- 239000000523 sample Substances 0.000 description 47
- 238000003384 imaging method Methods 0.000 description 11
- 238000012549 training Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 201000009310 astigmatism Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000012472 biological sample Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
- G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0025—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for optical correction, e.g. distorsion, aberration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Optics & Photonics (AREA)
- Multimedia (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
A Fourier laminated microscopic image reconstruction method and device based on deep learning are disclosed, wherein high-resolution amplitude and phase distribution are reconstructed from a low-resolution intensity map sequence through a neural network PgNN based on a physical model based on an iterative optimization mode of a neural network. The invention uses a neural network model based on a physical model to realize unsupervised reconstruction of high-resolution amplitude and phase distribution of a sample from a data set of a single sample, and can correct complex optical aberration.
Description
Technical Field
The invention relates to Fourier Ptychographic Microscopy (FPM), in particular to a learning Fourier ptychographic Microscopy image reconstruction method.
Background
Fourier laminated microscopic imaging experimental device As shown in figure 1, only a programmable LED matrix is introduced as an illumination light source on the basis of a traditional biological microscope. In the FPM data acquisition process, the LED lamps at different positions on the LED matrix are sequentially lightened, a sample placed on the objective table is sequentially irradiated by inclined plane waves at different positions, emergent waves are received by the camera through the low-power objective lens and the lens cone lens, and a series of wide view field and low-resolution intensity images which change along with the illumination angle are generated. According to the Fourier transform property, Fourier domain frequency spectrum information of a two-dimensional thin object is translated along with an incident angle after the two-dimensional thin object is irradiated by oblique plane waves. Thus, high frequency information components that cannot be detected under a low numerical aperture objective lens are sequentially shifted to within the spectral passband of the microscope system and are thus imaged by the camera (the numerical aperture of the optical system is a dimensionless number that measures the angular range of light that the system can collect). Based on the fact, the FPM recovers the object amplitude and phase distribution with high resolution and wide field of view under the support domain constraint of the coherent transfer function of the Fourier domain objective lens and the amplitude constraint of the light intensity picture shot in the spatial domain based on the low-resolution intensity graph sequence of the laminated spliced sample in the Fourier domain of the computed imaging mechanism, and has deeper axial resolution.
The existing FPM deep learning method generally attempts to learn an underlying mapping relationship from an input picture sequence to an output high resolution picture from massive data by using a data-driven network model, such as a multi-level residual neural network model proposed by Zhang et al (Zhang J, Xu T, Shen Z, et al. fourier ptychographic microscopical connectivity with multiscale depth residual network [ J ]. Optics express, 2019, 27(6): 8612-8625), and the overall flow is summarized as follows: the method comprises the steps of acquiring resolution images of a plurality of biological samples by using a high-power objective lens in advance, and generating an FPM training data set by using a simulation algorithm based on the resolution images. In the reconstruction process, firstly, a traditional FPM algorithm is used for synthesizing a low-resolution intensity graph sequence of a single sample into two initial high-resolution pictures with amplitude and phase distribution, and then the initial distribution is input into a multi-stage residual error neural network model. The weights of the neural network are optimized by minimizing the loss of L1 between the output image and the high resolution image. After training is complete, the network can reconstruct high resolution amplitude and phase distributions from the sequence of FPM images.
FPM is an economic and effective calculation imaging technology, and can span the resolution limit of low-cost low-power objective lenses only by adding LED matrixes on a biological microscope, so that the resolution effect which can be achieved by high-power objective lenses is achieved. Therefore, the FPM can be easily applied to a laboratory and other scenes, and no standard FPM experimental device specification exists so far. In this context, the following problems exist in solving the FPM problem using a data-driven based deep learning model:
1) because the FPM image acquisition process needs to sequentially turn on the LED lamps at different positions for illumination and acquire more than 200 intensity pictures, acquiring a FPM image sequence of a single sample generally requires more than 3min, and this low efficiency restricts the acquisition of a large-scale FPM sample data set. In practice, the FPM data set disclosed so far usually contains only a single sample of the image sequence. The rarity of data sets enables a data-driven neural network model to be learned only from limited data, and therefore network performance is restricted, and development of related applications is also restricted.
2) Due to the simplicity of the FPM imaging device, there is no standard FPM experimental device specification. Under different laboratory collection conditions, the experimental devices including the LED matrix, the objective lens parameters and the like have common differences, so that FPM sample data collected by different laboratories are concentrated, and some key physical parameters have larger deviations. In this context, a data-driven class model that performs well in a certain laboratory environment is difficult to perform well on a new data set.
3) In the process of acquiring the FPM microscopic image sequence, the problem of inevitable optical aberration exists, and is mainly caused by lens defects and accurate focusing. Specifically, the method is limited by a processing technology, the non-paraxial ray tracing result in an actual optical system deviates from ideal Gaussian optics, the non-paraxial ray tracing result and the paraxial ray tracing result are difficult to image on a focal plane together, and the imaging is blurred and deformed after being transmitted by an optical surface; the accurate focusing is always a key factor for limiting the microscopic imaging quality, and the best focusing effect needs to be obtained by continuously adjusting the image plane during image acquisition, which is difficult to achieve in practical situations. In this case, the optical aberration in the FPM image sequence of different samples can have a great influence on the quality of the reconstructed image, and the influence of the optical aberration needs to be eliminated as much as possible. The data-driven neural network avoids some tiny optical aberration influences through learning, but the network itself has no mechanism aiming at the optical aberration, so that the data-driven neural network is difficult to deal with complicated and variable optical aberration scenes.
In summary, the data-driven neural network model commonly used at present is difficult to be widely applied to the FPM problem because it needs mass data for training and is difficult to overcome the problems of parameter conditions and optical aberrations of transformation under different laboratory acquisition conditions.
Disclosure of Invention
The main purpose of the present invention is to overcome at least one of the above technical drawbacks, and to provide a fourier stacked microscopy image reconstruction method based on deep learning.
In order to achieve the purpose, the invention adopts the following technical scheme:
a Fourier laminated microscopic image reconstruction method based on deep learning comprises the following steps:
s1, collecting an FPM image sequence of a single sample, an incident wave vector sequence and relevant physical parameters of an experimental device;
s2, reconstructing high-resolution amplitude and phase distribution from the low-resolution intensity graph sequence through a neural network PgNN based on a physical model in an iterative optimization mode based on a neural network;
the reconstruction process comprises initialization and iterative optimization; the initialization comprises the following steps: generating an initial high resolution sample distribution O from a pre-acquired FPM image sequence0Generating an initial coherent transfer function distribution C from the relevant physical parameters of the experimental device0(ii) a The iterative optimization comprises: using a series of PgNNs with identical neural network structureiI 1 … n, the initial high resolution estimate O generated by said initialization0And C0Input PgNN1Obtaining an updated high resolution estimate O through a parameter optimization process of the neural network1And C0Repeating the operation, and obtaining a high-resolution estimation O of the sample reconstructed by the network after n times of iteration processesnSum coherent transfer function estimation Cn。
Further:
in the initialization, an initial sample distribution O is generated0Directly carrying out ultrasplitting on a low-resolution intensity map acquired under the illumination of the central lamp to obtain high-resolution amplitude distribution of a sample, and setting a zero phase as high-resolution phase distribution of the sample; or averaging all the images and then performing overdivision to obtain sample initial amplitude distribution; generating an initial coherent transfer function distribution C0When, C0Determined directly by the numerical aperture of the microscopy apparatus and the wavelength of the illumination light source, appears as a standard circular pupil function in the image plane of the camera fourier domain.
Neural network PgNN based on FPM physical modeliWherein the optical imaging model of the FPM is as follows:
ΦIm(k)＝0(i)(k+km)·C(i)(k)
in the formula kmRecording position information of the illumination light source for the incident wave vector recorded during the data acquisition process, O(i)And C(i)Are respectively input PgNNiThe high-resolution complex amplitude distribution and the coherent transfer function of the sample to be reconstructed, and k refers to a Fourier domain coordinate; and introduces an alternate projection principle to realize the updating process of the network self-supervision parameters:
in the formula IlmIs and kmLow-resolution intensity maps in a one-to-one correspondence, wherein F represents Fourier transform;
integrating FPM optical imaging model and alternate projection principle, PgNNiIn the network structure of (1), a high-resolution image O to be reconstructediModeling network hidden layer parameters, fusing the parameters with an incident wave vector kmCombined to form Oi(k+km) Then via the point multiplication module and the coherent transfer function CiMultiplying to generate philm，ΦlmBy alternating projection processing, using IlmUpdating the spectral information to obtain phihm(ii) a Measuring phi by MSE loss functionlmAnd phihmThe difference between them, PgNNiUnsupervised to OiAnd CiUpdating parameters; extracting PgNN through an iterative optimization processiThe hidden layer parameters of the middle representative sample distribution and the coherent transfer function are obtained to obtain updated Oi+1And Ci+1。
The alternative projection is derived from a phase recovery algorithm, firstly, the Fourier domain specific frequency spectrum information is projected to a space domain, secondly, the acquired low-resolution intensity graph is used for constraining the space domain amplitude of the frequency spectrum information and keeping the phase unchanged, and finally, the updated space domain complex amplitude distribution is converted back to the Fourier domain for updating the frequency spectrum information; when the deep learning model converges, the frequency spectrum deviation before and after alternate projection tends to 0, so that the network can reconstruct the complex amplitude distribution of the sample without supervision.
The PgNNiIntroducing an aberration correction module to compensate the optical aberration, wherein the aberration correction module is an embedded optical aberration correction module based on an alternate updating process and a physical model; in the optical imaging model of the FPM, C is selected under consideration of optical aberrationi(k) Can represent the complex of all optical disturbance factors in the acquisition process by aligning C in the FPM reconstruction processi(k) Correcting the variable, and compensating optical aberration including defocusing aberration in the acquired image sequence; PgNN, a network-learnable parameter by directly modeling coherent transfer functionsiAn optical aberration variable is learned from a sequence of acquired images.
The alternating update process changes the information flow of the network gradient update so that the network focuses on updating the sample or aberration at the same time; wherein, in PgNN1In the iterative optimization process, the sample parameter O is firstly activated0To obtain updated O1And a coherent transfer function C0Keeping the same; at this point, the updated sample parameter O is considered1Ratio C0Closer to the optimal solution, the network will fix O1Go to update C0To obtain C1(ii) a And setting alternate updating of the number of rounds to reconstruct both the sample and the coherent transfer function.
The PgNNiIntroduction of Zernike polynomial mechanismPerforming phase compensation of optical aberration, replacing phase part of coherent transfer function C (k) with Zernike polynomial, introducing Zernike polynomial, PgNNiThe expression of (C), (k) is as follows:
C(k)＝|C(k)|·exp{i∠C(k)}
∠C(k)＝∑cl·Zl(k)
in the formula Zl(k) Zernike polynomials of different orders, clAre the corresponding coefficients.
The PgNNiIntroducing a total variation loss function aiming at FPM amplitude and phase distribution to optimize a network structure; introducing full variation terms to the sample amplitude and the phase distribution respectively, wherein the form of the full variation terms and the improved network loss function are as follows:
TV{o}＝∑(|ox+1，y-ox，y|2+|ox，y-1-ox，y|2)1/2
wherein the TV item takes o as an example to express a calculation mode; the network loss function is based on the original MSE and is used for F after the alternate projection process-1{ΦhmThe amplitude and phase components are separately computed to form total variational terms, α1And α2Respectively, are corresponding term coefficients.
A deep learning-based Fourier laminated microscopy image reconstruction device comprises a computer-readable storage medium and a processor, wherein the computer-readable storage medium stores an executable program, and the executable program is characterized in that when being executed by the processor, the method for reconstructing the deep learning-based Fourier laminated microscopy image is realized.
A computer-readable storage medium storing an executable program which, when executed by a processor, implements the deep learning-based fourier-stacked microscopy image reconstruction method.
The invention has the following beneficial effects:
the invention overcomes the difficulties of large data demand, difficult correction of optical aberration and the like of a data driving model in the FPM problem, reconstructs high-resolution amplitude and phase distribution from a low-resolution intensity map sequence through a neural network PgNN based on a physical model based on an iterative optimization mode of a neural network, designs a universal FPM deep learning model in different application scenes, and only needs an FPM image sequence of a single sample as network input, and models the amplitude and phase distribution of the sample to be reconstructed into network learnable parameters, thereby unsupervised solving the FPM problem through the iterative optimization mode. Meanwhile, the network can compensate the optical aberration, and normal work in a complex acquisition scene is realized.
Compared with the existing other deep learning methods for solving the problem of Fourier stack microscopic imaging, the method needs massive training data and high-resolution pictures for supervised learning, and is difficult to correct the optical aberration in the data acquisition process, the method uses the neural network model based on the physical model, realizes unsupervised reconstruction of high-resolution amplitude and phase distribution of the sample from the data set of the single sample, and can correct the complex optical aberration.
Drawings
FIG. 1 is a schematic diagram of a Fourier stacked microscopy imaging apparatus.
FIG. 2 is a block diagram of the overall framework (left) and PgNN structure (right) of a Fourier stacked microscopy image reconstruction method according to an embodiment of the present invention;
FIG. 3 is a PgNN in an embodiment of the present inventioniThe specific network structure of (1).
Detailed Description
The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary in nature and is not intended to limit the scope of the invention or its application.
The Fourier laminated microscopic image reconstruction method based on deep learning provided by the embodiment of the invention comprises the following specific processes:
and S1, acquiring parameters such as an FPM image sequence and an incident wave vector sequence of a single sample.
The LED lamps at different positions on the LED matrix as shown in fig. 1 are illuminated in sequence and a series of wide field, low resolution intensity maps varying with the illumination angle are acquired using a camera. In order to ensure the performance of the FPM reconstruction result, more than 200 pictures are generally required to be acquired for a single sample, and the corresponding LED incident wave vector sequence, that is, the LED lamp position information, is recorded. In addition, relevant physical parameters of the experimental device, such as numerical aperture, incident wave wavelength and the like, should be recorded in the data acquisition process.
S2, reconstructing high-resolution amplitude and phase distribution from the low-resolution intensity map sequence through the neural network PgNN based on the physical model in an iterative optimization mode based on the neural network.
Fig. 2 shows the overall framework (left) and PgNN structure (right) of a fourier stacked microscopy image reconstruction method according to an embodiment of the present invention.
Due to the particularity of the FPM application scene, the data acquisition efficiency is low, the data-driven deep learning model is low in practicability due to the diversity of the acquisition environment and the like. In this context, the present invention designs a PgNN (Physics-defined Neural Network) specific to the FPM scene based on an iterative optimization method of a Neural Network, and reconstructs high-resolution amplitude and phase distributions from a low-resolution intensity map sequence. As shown in fig. 2, the fourier stacked microscopy image reconstruction algorithm proposed by the present invention is composed of two main parts: an initialization module and an iterative optimization module.
The initialization module generates an initial high-resolution sample distribution O according to the FPM image sequence acquired in the early stage0Generating an initial coherent transfer function distribution C from the relevant physical parameters of the experimental device0. Generating an initial sample distribution O0During the process, a low-resolution intensity map acquired under the illumination of a central LED position is usually directly subjected to ultradifferentiation to obtain high-resolution amplitude distribution of a sample, and a zero phase is set as high-resolution phase distribution of the sample; or taking the mean value of all the pictures and then performing overdivision to obtain the initial amplitude distribution of the sample. Generating an initial coherent transfer function distribution C0When, C0By numerical aperture of the microscopic means and illumination sourceThe wavelength is directly determined and appears as a standard circular pupil function in the image plane of the camera fourier domain.
The iterative optimization module consists of a series of PgNNsiI is 1 … n, and PgNNiThe same neural network structure is provided. Initializing the initial high resolution estimate O generated by the module0And C0Input PgNN1Obtaining an updated high resolution estimate O through a parameter optimization process of the neural network1And C0. Repeating the operation, and obtaining a high-resolution estimation O of the sample of the network reconstruction by the iteration optimization module after n iterationsnSum coherent transfer function estimation CnOf note, PgNNiBesides the self network structure, the system also comprises three functional modules, namely an aberration correction module, a phase compensation module and a structure optimization module.
In a preferred embodiment, to compensate for the specificity of the FPM application scenario, PgNNiThe network structure is determined by a series of customized hidden layers to obtain an optimized FPM neural network model module PgNN based on a physical modeli。
In a preferred embodiment, to compensate for optical aberrations that may be present in the acquisition environment, PgNNiAn aberration correction module will be introduced to compensate for the error.
In a preferred embodiment, in order to improve the accuracy and network performance of aberration compensation, the invention introduces a Zernike polynomial mechanism to perform phase compensation of optical aberration.
In addition, in order to improve the network performance, in the preferred embodiment, the loss function is used to optimize the network structure.
1. Physical model-based FPM neural network model module
FIG. 3 shows the physical model-based FPM neural network model module PgNN in the preferred embodimentiThe network structure of (1). The network structure based on the physical model starts from a specific algorithm principle, and simulates the whole application scene by converting the mathematical principle into a customized network hidden layer step by step, and solves the image inverse problem based on an iterative optimization mode. In other words, the neural network is inherently strongThe strong prior information can more quickly and accurately mine potential information from the data set. Under the background, the invention builds a neural network framework PgNN based on an FPM physical modeli. The optical imaging model of the FPM is as follows:
ΦIm(k)＝O(i)(k+km)·C(i)(k)
in the formula kmRecording the incident wave vector recorded in the data acquisition process-recording the position information of the LED illumination light source, OiAnd CiAre respectively input PgNNiThe high-resolution complex amplitude distribution and the coherent transfer function of the sample to be reconstructed, and k refers to a Fourier domain coordinate. Furthermore, it is contemplated that the present invention will unsupervised reconstruction O from a single sample FPM image sequencenAnd CnOn the basis of the formula of the last step, an alternative projection principle is further introduced to realize the updating process of the network self-supervision parameters:
in the formula IlmIs and kmOne-to-one correspondence of low-resolution intensity maps, F stands for fourier transform. The alternate projection is derived from a phase recovery algorithm, the specific frequency spectrum information of a Fourier domain is projected to a space domain in the first step, the acquired low-resolution intensity graph is used for constraining the space domain amplitude of the frequency spectrum information in the second step, the phase is kept unchanged, and finally the updated space domain complex amplitude distribution is converted back to the Fourier domain for updating the frequency spectrum information. According to the phase recovery theory, after the target frequency spectrum information is converged, the sub-frequency spectrum information is kept unchanged before and after the projection process is alternated. Therefore, by introducing the principle, when the deep learning model converges, the spectrum deviation before and after the alternate projection tends to 0, and the network can reconstruct the complex amplitude distribution of the sample without supervision.
Integrating FPM optical imaging model and alternate projection principle, PgNNiThe network structure of (2) is shown in fig. 3. Network to reconstruct high-fraction image OiModeling network hidden layer parameters, fusing the parameters with an incident wave vector kmCombined to form Oi(k+km) Then, againVia a point multiplication module with a coherent transfer function CiMultiplying to generate philm。ΦlmBy alternating projection modules, using IlmUpdating the spectral information to obtain phihm. Measuring phi by MSE loss functionlmAnd phihmThe difference between them, PgNNiCan be unsupervised to OiAnd CiAnd updating the parameters. Extracting PgNN through an iterative optimization processiThe hidden layer parameters of the middle representative sample distribution and the coherent transfer function can obtain the updated Oi+1And Ci+1。
It is noted that the network is defined entirely in the complex domain, and all hidden layers contain real and imaginary components, such as OiBy
And
two parts are formed.
2. Embedded optical aberration correction module based on alternate updating process and physical model
Due to lens process limitations and differences in the acquisition environment, there are more or less inherent optical aberrations in the FPM image sequence. In terms of aberration compensation, physical model-based neural network models, such as PgNNiHas the natural advantages. In the FPM optical imaging model, Ci(k) This parameter represents the coherence transfer function of the microscope objective. However, considering the optical aberration, it can represent a complex of all optical disturbance factors in the acquisition process, and correcting this variable in the FPM reconstruction process can compensate the optical aberration including the defocus aberration in the acquired image sequence.
Thus, PgNN is a network-learnable parameter by directly modeling the coherent transfer functioniThe optical aberration variation can be learned from the sequence of acquired images to attenuate the error pair for reconstructing the high resolution image OnThe aberration disturbance factors which beset the FP deep learning extension can be well obtained from the imaging principleThe solution is that.
Meanwhile, the optical aberration and the sample pixel have large difference in magnitude (the aberration distribution interval is [ -pi, pi ] is considered]The sample pixel distribution interval is [0, 255 ]]) The simultaneous parameter update during the network training process may cause interference between the sample and the aberration, which is especially serious in the case of large aberration disturbance. Therefore, the present invention will additionally introduce an alternate update mechanism to change the information flow of the network gradient update so that the network focuses on updating the sample or the aberration at the same time. Specifically, the iterative optimization module shown in FIG. 2, at PgNN1In the iterative optimization process, the sample parameter O is firstly activated0To obtain updated O1And a coherent transfer function C0Keeping the same; at this point, the updated sample parameter O is considered1Ratio C0Closer to the optimal solution, the network will fix O1Go to update C0To obtain C1. Through alternate updating of a certain number of rounds, both the sample and the coherent transfer function can be perfectly reconstructed. Another benefit of the alternate update mechanism is that it can make the network focus more on the features of the current path, speeding up the network convergence speed.
3. Optical aberration phase compensation mechanism based on Zernike polynomial
In the embedded optical aberration compensation mechanism defined above, the phase part of the optical aberration is fitted by updating the overall parameters. In fact, due to the flexible and changeable characteristics of the physical model type network, a better solution exists, and a Zernike polynomial is one of the solutions.
Zernike polynomials, which are advanced theories in the optical field for describing plane wave wavefront distortion, are composed of a series of infinite series polynomials defined inside a unit circle and orthogonal to each other, each of which strictly corresponds to a certain specific type of wavefront distortion, such as that the 0 th term represents the mean optical path difference, the 3 rd term represents defocus, and the 4 th and 5 th terms represent astigmatism, which are common types of optical aberrations. According to research, the top 10 Zernike polynomials have been shown to be complete representations of the types of aberrations commonly found in optical systems. Zernike polynomials are classified into odd and even categories, even categories:
odd-numbered species:
wherein n is more than or equal to m and is a non-negative integer,
for azimuth, 0 ≦ ρ ≦ 1 is the radial distance, and the odd and even properties of the polynomial are determined by the parity with m. In the formula
When n-m is an even number:
in the formula
When n-m is an odd number:
by using Zernike polynomials to replace the phase part of the coherent transfer function C (k), the neural network can achieve the same effect as the original phase overall reconstruction by fitting the parameters with constant magnitude, and has higher precision and more accurate description. Thus, after introducing Zernike polynomials, PgNNiThe expression of (C) (k) is updated as follows:
C(k)＝|C(k)|·exp{i∠C(k)}
∠C(k)＝∑cl·Zl(k)
in the formula Zl(k) Zernike polynomials of different orders, clAre the corresponding coefficients.
4. Introducing a fully-variant loss function for FPM amplitude and phase distributions
In the aspect of network structure, the interpretability and expressibility of a physical model class network are greatly enhanced, and the phase part of optical aberration is compensated by introducing Zernike polynomials in the embodiment of the invention. In the aspect of loss functions, the single reconstruction loss MSE is not enough to represent the advantages of a neural network, according to the characteristics of an FPM application scene, an additional loss function is introduced to compensate, and a total variation component is a type of loss function suitable for the scene.
Total Variation (TV) is the sum of the vertical and horizontal gradients of an image to obtain the degree of noise interference on the image. By introducing the regularization term into the sample amplitude and phase distribution respectively, the aliasing of the amplitude and the phase caused by the loss of high-frequency information can be effectively inhibited, and the imaging quality is rapidly improved. The form of the total variation term and the modified network loss function are as follows:
TV{o}＝∑(|ox+1，y-ox，y|2+|ox，y-1-ox，y|2)1/2
wherein the TV item takes o as an example to express a calculation mode; the network loss function is based on the original MSE and is used for F after the alternate projection process-1{ΦhmThe amplitude and phase components are separately computed to form total variational terms, α1And α2Respectively, are corresponding term coefficients.
By introducing the total variation item to inhibit high-frequency noise, the embodiment of the invention can obtain better FPM reconstruction effect.
In the embodiment of the invention, a Fourier laminated microscopic image reconstruction method based on a deep learning framework is realized through a data acquisition link, a data initialization link, the iterative optimization process and an improvement module thereof. The method allows unsupervised reconstruction of high-resolution complex amplitude distributions of samples in a sequence of FPM images of a single sample and compensates for optical aberrations present in the data in a more robust and advanced manner.
In the preferred embodiment of the invention, a physical model-based PgNN is adoptediThe method comprises a network structure design scheme, an embedded optical aberration compensation scheme based on an alternate updating mechanism, a physical model and a Zernike polynomial and a design scheme of introducing a total variation term into the amplitude and phase part of an FPM reconstructed image. By modeling the FPM optical imaging model by means of a neural network, the method can compensate optical distortion from the imaging principle and reconstruct the sample complex amplitude distribution. In addition, the network structure based on the physical model contains strong prior information, and unsupervised parameter updating in the FPM data set of a single sample can be realized without any high-resolution reference picture. In order to further improve the performance of the method, the Zernike polynomial and the alternating updating mechanism are preferably introduced to further optimize the optical aberration, so that the method can be applied to complex and changeable practical scenes and can normally work under different laboratory acquisition conditions. In addition, by introducing the total variation terms to the sample amplitude and phase distribution, the visual resolution of the reconstructed image is greatly improved.
The background of the present invention may contain background information related to the problem or environment of the present invention and does not necessarily describe the prior art. Accordingly, the inclusion in the background section is not an admission of prior art by the applicant.
The foregoing is a more detailed description of the invention in connection with specific/preferred embodiments and is not intended to limit the practice of the invention to those descriptions. It will be apparent to those skilled in the art that various substitutions and modifications can be made to the described embodiments without departing from the spirit of the invention, and these substitutions and modifications should be considered to fall within the scope of the invention. In the description herein, references to the description of the term "one embodiment," "some embodiments," "preferred embodiments," "an example," "a specific example," or "some examples" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction. Although embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the scope of the claims.
Claims (10)
1. A Fourier laminated microscopic image reconstruction method based on deep learning is characterized by comprising the following steps:
s1, collecting an FPM image sequence of a single sample, an incident wave vector sequence and relevant physical parameters of an experimental device;
s2, reconstructing high-resolution amplitude and phase distribution from the low-resolution intensity graph sequence through a neural network PgNN based on a physical model in an iterative optimization mode based on a neural network;
the reconstruction process comprises initialization and iterative optimization; the initialization comprises the following steps: generating an initial high resolution sample distribution O from a pre-acquired FPM image sequence0Generating an initial coherent transfer function distribution C from the relevant physical parameters of the experimental device0(ii) a The iterative optimization comprises: using a series of PgNNs with identical neural network structureiN, the initial high resolution estimate O generated by the initialization is estimated0And C0Input PgNN1Obtaining an updated high resolution estimate O through a parameter optimization process of the neural network1And C0Repeating the operation, and obtaining a high-resolution estimation O of the sample reconstructed by the network after n times of iteration processesnSum coherent transfer function estimation Cn。
2. The fourier stacked microscopy image reconstruction method of claim 1, wherein in the initializing, an initial sample distribution O is generated0Directly carrying out ultrasplitting on a low-resolution intensity map acquired under the illumination of the central lamp to obtain high-resolution amplitude distribution of a sample, and setting a zero phase as high-resolution phase distribution of the sample; or averaging all the images and then performing overdivision to obtain sample initial amplitude distribution; generating an initial coherent transfer function distribution C0When, C0Determined directly by the numerical aperture of the microscopy apparatus and the wavelength of the illumination light source, appears as a standard circular pupil function in the image plane of the camera fourier domain.
3. The fourier stacked microscopy image reconstruction method of claim 1 or 2, wherein the neural network PgNN based on the FPM physical modeliWherein the optical imaging model of the FPM is as follows:
inlm(k)＝O(i)(k+km)·C(i)(k)
In the formula kmRecording position information of the illumination light source for the incident wave vector recorded during the data acquisition process, O(i)And C(i)Are respectively input PgNNiThe high-resolution complex amplitude distribution and the coherent transfer function of the sample to be reconstructed, and k refers to a Fourier domain coordinate; and introduces an alternate projection principle to realize the updating process of the network self-supervision parameters:
in the formula IlmIs and kmLow-resolution intensity maps in a one-to-one correspondence, wherein F represents Fourier transform;
integrating FPM optical imaging model and alternate projection principle, PgNNiIn the network structure of (1), a high-resolution image O to be reconstructediModeling network hidden layer parameters, fusing the parameters with an incident wave vector kmCombined to form Oi(k+km) Then via the point multiplication module and the coherent transfer function CiMultiplying to generate philm，ΦlmBy alternating projection processing, using IlmUpdating the spectral information to obtain phihm(ii) a Measuring phi by MSE loss functionlmAnd phihmThe difference between them, PgNNiUnsupervised to OiAnd CiUpdating parameters; extracting PgNN through an iterative optimization processiThe hidden layer parameters of the middle representative sample distribution and the coherent transfer function are obtained to obtain updated Oi+1And Ci+1。
4. The reconstruction method of the Fourier laminated microscopic image according to claim 3, wherein the alternate projection is derived from a phase recovery algorithm, in the first step, Fourier domain specific frequency spectrum information is projected to a space domain, in the second step, the acquired low-resolution intensity map is used for constraining the space domain amplitude of the frequency spectrum information and keeping the phase unchanged, and finally, the updated space domain complex amplitude distribution is converted back to the Fourier domain for updating the frequency spectrum information; when the deep learning model converges, the frequency spectrum deviation before and after alternate projection tends to 0, so that the network can reconstruct the complex amplitude distribution of the sample without supervision.
5. The method of reconstructing a fourier stacked microscopy image of any one of claims 1 to 4, wherein the PgNN is configured to reconstruct a fourier transform of the image of the objectiIntroducing an aberration correction module to compensate the optical aberration, wherein the aberration correction module is an embedded optical aberration correction module based on an alternate updating process and a physical model; in the optical imaging model of the FPM, C is selected under consideration of optical aberrationi(k) Can represent the complex of all optical disturbance factors in the acquisition process by aligning C in the FPM reconstruction processi(k) Correcting the variable, and compensating optical aberration including defocusing aberration in the acquired image sequence; PgNN, a network-learnable parameter by directly modeling coherent transfer functionsiAn optical aberration variable is learned from a sequence of acquired images.
6. The method of fourier stacked microscopy image reconstruction of claim 5, wherein the fourier stacked microscopy image reconstruction is performedThe alternating update process changes the information flow of the network gradient update so that the network focuses on updating the sample or aberration at the same time; wherein, in PgNN1In the iterative optimization process, the sample parameter O is firstly activated0To obtain updated O1And a coherent transfer function C0Keeping the same; at this point, the updated sample parameter O is considered1Ratio C0Closer to the optimal solution, the network will fix O1Go to update C0To obtain C1(ii) a And setting alternate updating of the number of rounds to reconstruct both the sample and the coherent transfer function.
7. The method of reconstructing a fourier stacked microscopy image of any one of claims 1 to 6, wherein the PgNN is configured to reconstruct a fourier transform of the image of the objectiIntroducing Zernike polynomial mechanism to compensate optical aberration phase, replacing coherent transfer function C (k) phase part with Zernike polynomial, introducing Zernike polynomial, PgNNiThe expression of (C), (k) is as follows:
C(k)＝|C(k)|·exp{i∠C(k)}
∠C(k)＝∑cl·Zl(k)
in the formula Zl(k) Zernike polynomials of different orders, clAre the corresponding coefficients.
8. The method of reconstructing a fourier stacked microscopy image of any one of claims 1 to 6, wherein the PgNN is configured to reconstruct a fourier transform of the image of the objectiIntroducing a total variation loss function aiming at FPM amplitude and phase distribution to optimize a network structure; introducing full variation terms to the sample amplitude and the phase distribution respectively, wherein the form of the full variation terms and the improved network loss function are as follows:
TV{o}＝∑(|ox+1，y-ox，y|2+|ox，y-1-ox，y|2)1/2
where TV items are exemplified by o, tableThe calculation mode is described; the network loss function is based on the original MSE and is used for F after the alternate projection process-1{ΦhmThe amplitude and phase components are separately computed to form total variational terms, α1And α2Respectively, are corresponding term coefficients.
9. A deep learning-based fourier stack microscopy image reconstruction apparatus comprising a computer readable storage medium and a processor, the computer readable storage medium storing an executable program, wherein the executable program, when executed by the processor, implements the deep learning-based fourier stack microscopy image reconstruction method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing an executable program, wherein the executable program, when executed by a processor, implements the method of deep learning based fourier-stack microscopy image reconstruction according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010378100.3A CN111650738A (en) | 2020-05-07 | 2020-05-07 | Fourier laminated microscopic image reconstruction method and device based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010378100.3A CN111650738A (en) | 2020-05-07 | 2020-05-07 | Fourier laminated microscopic image reconstruction method and device based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111650738A true CN111650738A (en) | 2020-09-11 |
Family
ID=72352008
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010378100.3A Pending CN111650738A (en) | 2020-05-07 | 2020-05-07 | Fourier laminated microscopic image reconstruction method and device based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111650738A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465703A (en) * | 2020-12-01 | 2021-03-09 | 大连海事大学 | Fourier laminated imaging depth learning image reconstruction method containing spectral offset estimation |
| CN112819082A (en) * | 2021-02-09 | 2021-05-18 | 南京邮电大学 | Satellite spectrum sensing data reconstruction method based on deep learning |
| CN113281979A (en) * | 2021-05-20 | 2021-08-20 | 清华大学深圳国际研究生院 | Lensless laminated diffraction image reconstruction method, system, device and storage medium |
| CN114677447A (en) * | 2022-03-21 | 2022-06-28 | 中国科学院西安光学精密机械研究所 | Optical microscopic imaging redundant information quantitative measurement method |
| CN114972284A (en) * | 2022-06-08 | 2022-08-30 | 南京大学 | Lens-free microscopic imaging system and method based on self-supervision deep learning |
| CN116609942A (en) * | 2023-07-18 | 2023-08-18 | 长春理工大学 | Sub-aperture compressed sensing polarization super-resolution imaging system and method |
| WO2023221741A1 (en) * | 2022-05-19 | 2023-11-23 | 南京理工大学 | Transport of intensity diffraction tomography microscopic imaging method based on non-interferometric synthetic aperture |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180350459A1 (en) * | 2017-06-05 | 2018-12-06 | University Of Florida Research Foundation, Inc. | Methods and apparatuses for implementing a semantically and visually interpretable medical diagnosis network |
| CN110197493A (en) * | 2019-05-24 | 2019-09-03 | 清华大学深圳研究生院 | Eye fundus image blood vessel segmentation method |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
-
2020
- 2020-05-07 CN CN202010378100.3A patent/CN111650738A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180350459A1 (en) * | 2017-06-05 | 2018-12-06 | University Of Florida Research Foundation, Inc. | Methods and apparatuses for implementing a semantically and visually interpretable medical diagnosis network |
| CN110197493A (en) * | 2019-05-24 | 2019-09-03 | 清华大学深圳研究生院 | Eye fundus image blood vessel segmentation method |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
Non-Patent Citations (1)
| Title |
|---|
| YONGBING ZHANG ETC: ""PgNN：Physics-guided Neural Network for Fourier Ptychographic Microscopy"", 《ARXIV》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465703A (en) * | 2020-12-01 | 2021-03-09 | 大连海事大学 | Fourier laminated imaging depth learning image reconstruction method containing spectral offset estimation |
| CN112465703B (en) * | 2020-12-01 | 2024-03-29 | 大连海事大学 | Fourier stacked imaging deep learning image reconstruction method containing spectrum offset estimation |
| CN112819082A (en) * | 2021-02-09 | 2021-05-18 | 南京邮电大学 | Satellite spectrum sensing data reconstruction method based on deep learning |
| CN113281979A (en) * | 2021-05-20 | 2021-08-20 | 清华大学深圳国际研究生院 | Lensless laminated diffraction image reconstruction method, system, device and storage medium |
| CN113281979B (en) * | 2021-05-20 | 2022-04-19 | 清华大学深圳国际研究生院 | Lensless laminated diffraction image reconstruction method, system, device and storage medium |
| CN114677447A (en) * | 2022-03-21 | 2022-06-28 | 中国科学院西安光学精密机械研究所 | Optical microscopic imaging redundant information quantitative measurement method |
| CN114677447B (en) * | 2022-03-21 | 2023-05-12 | 中国科学院西安光学精密机械研究所 | Quantitative measurement method for redundant information of optical microscopic imaging |
| WO2023221741A1 (en) * | 2022-05-19 | 2023-11-23 | 南京理工大学 | Transport of intensity diffraction tomography microscopic imaging method based on non-interferometric synthetic aperture |
| CN114972284A (en) * | 2022-06-08 | 2022-08-30 | 南京大学 | Lens-free microscopic imaging system and method based on self-supervision deep learning |
| CN116609942A (en) * | 2023-07-18 | 2023-08-18 | 长春理工大学 | Sub-aperture compressed sensing polarization super-resolution imaging system and method |
| CN116609942B (en) * | 2023-07-18 | 2023-09-22 | 长春理工大学 | Sub-aperture compressed sensing polarization super-resolution imaging method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111650738A (en) | Fourier laminated microscopic image reconstruction method and device based on deep learning | |
| CN110441271B (en) | Light field high-resolution deconvolution method and system based on convolutional neural network | |
| Horstmeyer et al. | Overlapped Fourier coding for optical aberration removal | |
| Shajkofci et al. | Spatially-variant CNN-based point spread function estimation for blind deconvolution and depth estimation in optical microscopy | |
| US8254714B2 (en) | Methods and systems for designing electromagnetic wave filters and electromagnetic wave filters designed using same | |
| Korkiakoski et al. | Fast & Furious focal-plane wavefront sensing | |
| Zhang et al. | Hybrid particle swarm global optimization algorithm for phase diversity phase retrieval | |
| CN110378981A (en) | Fourier's lamination microscope pupil restoration methods neural network based | |
| JP2022514580A (en) | Optical correction by machine learning | |
| CN109407311A (en) | A kind of diffraction optics imaging system wavefront inversion algorithm based on phase difference method | |
| US7635832B2 (en) | Hybrid diversity method utilizing adaptive diversity function for recovering unknown aberrations in an optical system | |
| Chen et al. | Optimized denoising method for fourier ptychographic microscopy based on wirtinger flow | |
| Wu et al. | Adaptive correction method of hybrid aberrations in Fourier ptychographic microscopy | |
| Liaudat et al. | Rethinking data-driven point spread function modeling with a differentiable optical model | |
| Sweere et al. | Deep learning-based super-resolution and de-noising for XMM-newton images | |
| Zhang et al. | Neural network model assisted Fourier ptychography with Zernike aberration recovery and total variation constraint | |
| CN111694016A (en) | Non-interference synthetic aperture super-resolution imaging reconstruction method | |
| Zhou et al. | Analysis of postreconstruction digital refocusing in Fourier ptychographic microscopy | |
| Moore et al. | Ptychography for optical metrology with limited translation knowledge | |
| Archinuk et al. | Mitigating the nonlinearities in a pyramid wavefront sensor | |
| Zhang et al. | Alternative deep learning method for fast spatial-frequency shift imaging microscopy | |
| Zhang et al. | Pgnn: Physics-guided neural network for fourier ptychographic microscopy | |
| Lehmann et al. | Resolution and accuracy of nonlinear regression of point spread function with artificial neural networks | |
| Zhang et al. | The integration of neural network and physical reconstruction model for Fourier ptychographic microscopy | |
| Ye et al. | Towards practical single-shot phase retrieval with physics-driven deep neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |