CN108873302B - Automatic correction and refocusing method of light field microscopic image - Google Patents

Automatic correction and refocusing method of light field microscopic image Download PDF

Info

Publication number
CN108873302B
CN108873302B CN201810691605.8A CN201810691605A CN108873302B CN 108873302 B CN108873302 B CN 108873302B CN 201810691605 A CN201810691605 A CN 201810691605A CN 108873302 B CN108873302 B CN 108873302B
Authority
CN
China
Prior art keywords
image
images
light field
sub
algorithm
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810691605.8A
Other languages
Chinese (zh)
Other versions
CN108873302A (en
Inventor
李莉华
李荣彬
张昆霭
王孝忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hong Kong Polytechnic University HKPU
Original Assignee
Hong Kong Polytechnic University HKPU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hong Kong Polytechnic University HKPU filed Critical Hong Kong Polytechnic University HKPU
Priority to CN201810691605.8A priority Critical patent/CN108873302B/en
Publication of CN108873302A publication Critical patent/CN108873302A/en
Application granted granted Critical
Publication of CN108873302B publication Critical patent/CN108873302B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Optics & Photonics (AREA)
  • Theoretical Computer Science (AREA)
  • Microscoopes, Condenser (AREA)
  • Image Processing (AREA)

Abstract

The application provides an automatic correction and refocusing method of a light field microscopic image, (a) a captured light field image is automatically segmented into microimages through an automatic correction algorithm; (b) segmenting the microimage of step (a) into sub-images; (c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image; the color light field image is converted into a binary image by using an Ostu algorithm, so that the color light field image is not only used for detecting the defects of the light field image, but also the center of each micro lens and the radius of each micro image are easy to calculate, the time is saved, and the correction precision is improved; more noise is added to the out-of-focus area by the scrolling and adding algorithm, thereby providing more contrast between the in-focus and out-of-focus areas.

Description

Automatic correction and refocusing method of light field microscopic image
Technical Field
The invention relates to the technical field of light field microscopic images, in particular to a method for automatically correcting and refocusing an image.
Background
Conventional microscopes have some limitations, when using high power lenses or larger aperture lenses, the depth of field is very shallow, requiring an adjustable platform to move objects up and down to view different layers of objects, resulting in the organisms or photosensitive samples being difficult to observe or capture, and conventional optical microscopes have only one directional projection, and cannot provide 3D information.
Currently, light field microscopy imaging systems have been developed that capture information of light intensity, color, and depth using microlens arrays. To analyze light field information within images captured by a light field microscopy imaging system, an image processing method should be performed to calculate depth information and refocus the light field microscopy images; in the prior art, the micro lens is generally corrected first. The traditional correction method is to manually input the center and the radius of the micro lens, and because the micro lens array has hundreds of thousands of micro lenses, the manual input is time-consuming and inaccurate; after correcting the microlenses, the shift-add algorithm is used to calculate the depth information of the light field image, generating a refocused image, defects in the microlens array are ignored, they should be considered as noise, but generally they are not detected and still used to calculate the depth information.
A light field microscopic imaging system is composed of five basic components: microlens arrays, special microlenses, digital cameras, optical microscopes and LED illumination. The microlens array is mounted in front of the CMOS of the camera, fig. 1 is an image at F number of different microlens arrays in a light field microscope, fig. 1(a) a schematic of a light field microscope; FIG. 1(b) when the F number of the microlens array is lower than the F number of the image side of the objective lens, the image gap is relatively large; FIG. 1(c) when the F number of the microlens array is higher than the F number on the image side of the objective lens, the images overlap; fig. 1(d) is continuous between images when the F-number of the microlens array matches the image-side F-number of the objective lens. For the cases in fig. 1(b) and 1(d), if the center and radius of the microimage can be found, the microimage can be segmented; however, in the case of fig. 1(c), since the microimages overlap, it is impossible to correctly divide the microimages.
Disclosure of Invention
The invention provides an automatic correction and refocusing method of a light field microscopic image, which can detect image defects, automatically, quickly and accurately correct the image defects and refocus by using a superposition algorithm of the invention.
The invention relates to an automatic correction and refocusing method of a light field microscopic image, which comprises the following steps:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
Preferably, the automatic correction algorithm in step (a) is implemented by locating the center of the microimage and calculating the radius of the microimage.
Preferably, the step (a) includes converting the colored light field image into a binary image using Otsu's algorithm for locating the center of the microimage and calculating the radius of the microimage.
Preferably, the automatic correction algorithm in step (a) is used to detect defects in the image of the optical field.
Preferably, in step (b), the sub-image is a specific view of the light field image.
Preferably, in the step (b), the microimages are divided into sub-images according to the centers of the microimages and the radii of the microimages, and the relationship between the sub-images and the microimages is defined as follows:
Si，j(x，y)＝Mx，y(i，j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
Preferably, in the step (c), the number of the sub-images is n, and n is a positive integer.
Preferably, in step (c), the relationship between the sub-images and the refocused image is as follows:
Figure RE-GDA0002715595200000021
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively.
Preferably, the addition of noise outside the focus area is achieved by the refocused image calculated using a rolling addition algorithm.
Preferably, an image processing module comprises the automatic correction and refocusing method of light field microscopy images of the present application.
The beneficial effect of adopting above-mentioned technical scheme is:
according to the method, the color light field image is converted into the binary image by using the Otsu algorithm, the center of the microimage is positioned, the radius of the microimage is calculated, the defect of the light field image is detected, the center of the black hole is used, the center of each micro lens and the radius of the microimage are easily calculated, time is saved, correction precision is improved, the defect in the light field image is automatically detected, and more accurate depth information calculation is provided; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic of an image at F-numbers of different microlens arrays in a light field microscope;
FIG. 2 is a flow chart of a method of automatic correction and refocusing of a light field microscopic image in accordance with the present invention;
FIG. 3 is a white background image and partial view captured by a microlens array;
FIG. 4 is a schematic diagram of the binary image of FIG. 3 after Otsu's algorithm;
FIG. 5 is a schematic diagram of the generation of a sub-image;
FIG. 6 is a schematic illustration of a sub-image being re-focused by a rolling addition algorithm;
FIG. 7 is a schematic illustration of a light field microscopic image;
FIG. 8 is a schematic diagram of the detection results of black holes and defects in a light field microscopic image;
FIG. 9 is a schematic diagram of a microimage acquired after auto-calibration;
FIG. 10 is a schematic illustration of the generation of a sub-image after the microimage generation of FIG. 9;
FIG. 11 is a schematic illustration of a refocused image generated by the sub-image of FIG. 10.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
A method for automatic correction and refocusing of light field microscopy images, as shown in fig. 2, comprising the steps of:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (a), as shown in fig. 3a, a flat white paper is photographed using a light field microscope, wherein defects and black holes are present in the image, and some microlenses are defective in the upper left area of fig. 3a and should not be used to calculate depth information; the black holes in FIG. 3b show the non-overlapping areas of the microimages.
To find the center of the microimage, the black hole needs to be detected first. As shown in fig. 4, by converting the color image into a binary image using Otsu's algorithm, it is possible to highlight the black hole and the defective region, find the outline of each black region, and calculate the number of pixels of each region. The number of pixels of the black hole should be smaller than that of the defect, and therefore, all the regions having similar and relatively smaller number of pixels are located and are detected black holes, and using the center of the black hole, it is easy to calculate the center of each microlens and the radius of the microimage.
For step (b), the sub-image is a particular view of the light field image; dividing the micro image into sub images according to the center of the micro image and the radius of the micro image, wherein the relation between the sub images and the micro image is defined as follows:
Si，j(x，y)＝Mx，y(i，j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
As shown in fig. 5, it is shown how a sub-image S can be obtained from a micro-image calculated from a light field image consisting of 3 columns and 2 rows of micro-images, i.e. a light field image consisting of six micro-images, each taking the pixels of (-1, -1)-1,-1
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
Figure RE-GDA0002715595200000051
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; if the image acquired after the sub-image S is moved in the x direction is defined as S', and the image acquired after the sub-image S is moved in the xy direction is defined as S ″, it can be derived:
S″＝RollXY(S，i，j)＝RollY(S′，j)
S′＝RollX(S，i)
fig. 6 shows how the rolxy (S, 2, -1) is calculated from the sub-image S; moving 2 pixels in the x-direction yields S ', and moving S' 1 element in the y-direction yields the sub-image S ". The roll and add algorithm, which is different from the shift and add method, fills the pixels that move out of the region with empty pixels instead of with a value of 0, by which noise is added to the incoherent pixels, making the objects on the focal plane clearer and the objects on the non-focal plane more blurred, thereby providing more contrast between the in-focus and out-of-focus regions.
Example 2
In order to further verify the technical effect of the scheme, further experiments are carried out on an automatic correction and refocusing method of a light field microscopic image.
Automatically segmenting a captured light field image into microimages through an automatic correction algorithm; as shown in fig. 7, a light field image captured by a light field microscopy imaging system; fig. 9 is a micro image segmented by an auto-correction algorithm.
Step (b) segmenting the microimage of step (a) into sub-images; by converting the color image into a binary image using Otsu's algorithm, as shown in fig. 8, it is possible to highlight black holes and defective regions, find the outline of each black region, and calculate the number of pixels of each region; FIG. 8 shows the detection result of black holes and the defects of the light field image; the black dots are detected black holes, and the result shows that the detection is very accurate, the detection speed is very high, and 5184 multiplied by 3456 resolution images with 120 ten thousand micro images can be detected in 2 seconds. The center of each microlens and the radius of the microimage are easily calculated by using the center of the black hole; the sub-images are segmented according to the center of the micro-image and the radius of the micro-image as shown in fig. 10.
Step (c) refocuses the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
Figure RE-GDA0002715595200000061
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, RollX and RollY is let sub-picture Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; as shown in fig. 11, a refocused image calculated by the scroll and add algorithm is displayed; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (2)

1. A method for automatically correcting and refocusing a light field microscopic image, comprising the steps of:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm; the automatic correction algorithm is to convert the color image into a binary image by using an Otsu algorithm, find the outline of each black area, calculate the number of pixels of each black area, detect a black hole, and calculate the center of each microlens and the radius of the microimage by using the center of the black hole;
(b) segmenting the microimage of step (a) into sub-images;
the relationship of the sub-image to the microimage is defined as follows:
Si,j(x,y)＝Mx,y(i,j)
wherein S is a sub-image, M is a micro-image, i and j are pixel coordinates in the micro-image M, x is x columns of micro-images in the light field image, and y is y rows of micro-images in the light field image;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image;
the number of the sub-images is n, n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
Figure FDA0002715595190000011
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively;
the pixels that move out of the region are filled with empty pixels by a scroll and add algorithm by which noise is added to the incoherent pixels.
2. An image processing module comprising the method of automatic correction and refocusing of light field microscopy images of claim 1.
CN201810691605.8A 2018-06-28 2018-06-28 Automatic correction and refocusing method of light field microscopic image Active CN108873302B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810691605.8A CN108873302B (en) 2018-06-28 2018-06-28 Automatic correction and refocusing method of light field microscopic image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810691605.8A CN108873302B (en) 2018-06-28 2018-06-28 Automatic correction and refocusing method of light field microscopic image

Publications (2)

Publication Number Publication Date
CN108873302A CN108873302A (en) 2018-11-23
CN108873302B true CN108873302B (en) 2021-02-05

Family

ID=64296789

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810691605.8A Active CN108873302B (en) 2018-06-28 2018-06-28 Automatic correction and refocusing method of light field microscopic image

Country Status (1)

Country Link
CN (1) CN108873302B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11030776B2 (en) * 2019-02-01 2021-06-08 Molecular Devices (Austria) GmbH Calibration of a light-field imaging system
CN113219643B (en) * 2021-05-11 2022-07-12 浙江大学 Optical microscope focusing stabilization method and system based on incoherent imaging edge blurring

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013003826A1 (en) * 2011-06-29 2013-01-03 The Regents Of The University Of Michigan Analysis of temporal changes in registered tomographic images
DE102011054106A1 (en) * 2011-09-30 2013-04-04 Jörg Piper Method for producing variable phase contrast and/or dark-field imaging, using microscope, involves enabling light rings to make the lights pass through object, so that the summation image is arranged in the plane of intermediate image
CN105683815A (en) * 2013-09-04 2016-06-15 卢门科有限责任公司 Servo system, and encoder
CN104732563B (en) * 2015-04-01 2017-06-27 河南理工大学 Circle ring center's line detecting method based on dual range conversion and feature distribution in image

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013003826A1 (en) * 2011-06-29 2013-01-03 The Regents Of The University Of Michigan Analysis of temporal changes in registered tomographic images
DE102011054106A1 (en) * 2011-09-30 2013-04-04 Jörg Piper Method for producing variable phase contrast and/or dark-field imaging, using microscope, involves enabling light rings to make the lights pass through object, so that the summation image is arranged in the plane of intermediate image
CN105683815A (en) * 2013-09-04 2016-06-15 卢门科有限责任公司 Servo system, and encoder
CN104732563B (en) * 2015-04-01 2017-06-27 河南理工大学 Circle ring center's line detecting method based on dual range conversion and feature distribution in image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Near-field scanning optical microscopy（MSOM）;E.Betziq,et al.;《Scanning Microscopy Technologies and Applications》;19881231;全文 *
基于光场成像的三维测量方法的研究;王宇等;《仪器仪表学报》;20160615;全文 *

Also Published As

Publication number Publication date
CN108873302A (en) 2018-11-23

Similar Documents

Publication Publication Date Title
CA3040006C (en) Device and method for obtaining distance information from views
JP5446797B2 (en) Imaging device
JP5173954B2 (en) Image processing apparatus and image processing method
JP5552214B2 (en) Focus detection device
US9088729B2 (en) Imaging apparatus and method of controlling same
US8730374B2 (en) Focus detection apparatus
RU2525034C2 (en) Image capturing device
US10321044B2 (en) Image pickup apparatus and image pickup system with point image intensity distribution calculation
CN102663732A (en) Relative radiometric calibration method for light field camera
JP6381266B2 (en) IMAGING DEVICE, CONTROL DEVICE, CONTROL METHOD, PROGRAM, AND STORAGE MEDIUM
CN107960120B (en) Image processing apparatus, image capturing apparatus, image processing method, and storage medium
JP2016038414A (en) Focus detection device, control method thereof, and imaging apparatus
JP6700986B2 (en) Image processing device, imaging device, image processing method, and program
CN108873302B (en) Automatic correction and refocusing method of light field microscopic image
CN107249089A (en) Filming apparatus and method, video camera and logistics capturing system
JP2015194736A (en) Imaging device and method for controlling the same
TWI604221B (en) Method for measuring depth of field and image pickup device using the same
US20140347548A1 (en) Method and system for rendering an image from a light-field camera
JP2015210285A (en) Imaging device, manufacturing method of the same, program thereof and recording medium
CN112866547B (en) Focusing method and device, electronic equipment and computer readable storage medium
JP2018133064A (en) Image processing apparatus, imaging apparatus, image processing method, and image processing program
US10200622B2 (en) Image processing method, image processing apparatus, and imaging apparatus
JP5018932B2 (en) Imaging device
CN113596431B (en) Image processing apparatus, image capturing apparatus, image processing method, and storage medium
He et al. Point Spread Function Estimation of Defocus

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant